The multi-objective neural architecture search (NAS) can automatically realize the network design for high accuracy and high hardware performance for varied applications. However, the existing methods usually need to sample a large number of networks in the search process to guide the controller’s search behavior. As a result, the entire search process requires a huge search time overhead. We propose a NAS framework that can perform dynamic adaptive network sampling that regulated by the latency requirements for specific devices and application scenarios. In the layer-wise network sampling process, the sampling probability for each layer is adjusted dynamically according to the current remaining available inference latency space. So that the latency of the sampled network will close to the required latency. Thereby, reducing useless network sampling and improving the search efficiency. Experimental results show that the search efficiency has a 2.35x speedup.