Optical flow (OF) using frame-based cameras is speed limited due to the frame rate or model size of the network. Event camera provides asynchronous event stream circumventing the frame-rate limitation. But algorithms for processing this either borrow frame like setup limiting the speed or suffer from lower accuracy. We fuse the complementary accuracy and speed advantages of the frame and event-based pipelines to provide high-speed optical flow while maintaining a low error rate to mitigate the accuracy latency trade-off. We validate the method on the MVSEC dataset and apply it to a real flying drone scenario to demonstrate speed advantage.