Detection, description, and matching of keypoints are commonly used as a first step in embedded real-time vision applications such as tracking, 3D reconstruction, visual odometry, or SLAM. This paper presents the architecture of a hardware accelerator for keypoint extraction and matching designed to meet low latency and memory footprint constraints while being scalable and generic. We present two implementations with two well known methods (Harris detector and U-SURF descriptor) to show the performance reached by our approach.