Remote photoplethysmography (RPPG) is a camerabased optical technique for detecting volumetric changes of organs. This technique enables the non-contact measurement of respiration and pulse. Monitoring newborn infants is a challenging task, due to the weak pulse signals, the rather irregular respiratory pattern, and the frequent movements. Therefore, heavy optimization of the sensing and evaluation process is a must in a resource-limited embedded vision system. This is a two-faceted study, with a special focus on low computational complexity. In the field of respiration monitoring, the paper introduces an optimized convolutional neural network (CNN) and a novel, light Long Short-term Memory (LSTM) motion classifier with a narrow CNN layer. From heart rate measurement point of view, a skin segmentation based algorithm is presented. The performance of each algorithm is evaluated on a database collected at the Ist Dept. of Neonatology of Pediatrics, Dept of Obstetrics and Gynecology, Semmelweis University, Budapest, Hungary.