Measuring the respiratory rate (RR) in a hospital setting involves wearing bulky uncomfortable sensors. Accurate measurement can be performed by extracting respiratory modulations from Photoplethysmogram (PPG) signal obtained from a pulse oximeter indirectly. Respiratory rate estimates from derived modulations are fused to get robust results. However, all the three extracted modulations are not true representative of the respiration activity all the time, subject to the patient\'s health condition and body position. Therefore, we propose novel modulation quality indices (MQI) to check the quality of the extracted modulations before computing the RR from it. We take the mean of only those estimates which pass an empirical quality threshold. This approach increases the robustness of the mean fusion methodology. We have validated our algorithm on a publicly available dataset: benchmark dataset CapnoBase. The proposed approach outperforms the current state-of-the-art with mean absolute errors (median, 25th-75th percentiles with 32-sec window size) of 0.4 (0.1-0.7) without discarding any PPG window, thus enabling accurate RR estimates.