A low power real-time visual object tracking (VOT) processor using the siamese network (SiamNet) is proposed for mobile devices. Two key features enable a real-time VOT with low power consumption on mobile devices. First, correlation-based spatial early stopping (CSES) is proposed to reduce the computational workload. CSES reduces ~56.8% of the overall computation of SiamNet by gradually eliminating the background. Second, the dual mode reuse core is proposed for supporting both the convolution layer and the cross-correlation layer with high core utilization. Finally, the proposed VOT processor is implemented in 28 nm CMOS technology and occupies 0.42 mm2. Compared to the previous VOT hardware, the proposed processor shows state-of-the-art performance while showing lower power consumption. The proposed processor achieves 64.1 mW power consumption at 32.1 frame-per-second real-time VOT on mobile devices.