The matrix decomposition (MD) based FIR filter design technique can synthesize a traditional FIR filter with much less implementation complexity. A scheme for obtaining more sparse coefficients of a MD-FIR filter is proposed, which consists of two parts. The first part is the previous procedure of designing a MD-FIR filter (i.e., design an initial MD-FIR filter using a certain MD method and optimize its coefficients). The second part is the proposed procedure of obtaining more sparse coefficients, where the MD-FIR filters’ coefficients also need to be optimized. The performances of the various initial MD-FIR filters, which are obtained based on the various MD methods, in the implementation of this scheme are experimentally compared. The MD-FIR filter’s coefficients can be effectively optimized by the trust-region iterative-gradient-searching (TR-IGS) algorithm. We present further results on the TR-IGS.