Machine vision applications running on resource-constrained embedded systems can face contentions by coexisting applications. This can cause increase in inference delay which can be unacceptable for time-critical tasks. We propose an adaptive model selection framework to isolate machine vision application from system contention and prevent unexpected inference delay increase by trading off the accuracy of the application minimally. The framework uses a set of hierarchical deep learning models for image classification and selects the optimal model for each frame considering the system contention. Our framework on average improves the accuracy while decreasing the inference delay violations compared to other approaches.