A hybrid capsule network-based deep learning framework for deciphering ancient scripts with scarce annotations is presented. To verify the feasibility of our proposed framework, the Phoenician epigraphy is used as a case study. A corpus of labeled data of Phoenician alphabets that covers all different styles and stages is presented. This corpus can help in contributing to the digitization process of the Phoenician culture. Our model achieves a state-of-the-art performance in recognizing handwritten characters with an overall accuracy of 0.9891 and loss of 0.021. Therefore, our model can help develop an automated deciphering system to save epigraphists\' valuable time and effort in deciphering the Phoenician epigraphy in a short period. Moreover, this work can be replicated for any other ancient scripts with minor modifications considering the systematic methodology that we proposed since it has proven its effectiveness in deciphering the Phoenician epigraphy. Our model can be employed as a transfer learning backbone for recognizing other existing alphabets which suffer from lack of annotated data.