This contribution presents a new Modified Compressed Sparse Row (MCSR) format designed for multi-core FPGA-based hardware accelerators for the calculation of Sparse Matrix Vector Multiplication (SMVM). Current state-of-the-art methods suffer from memory bandwidth bottlenecks affecting the speed of parallel SMVM. The salient feature of the proposed MCSR format is that an extension of Hu’s scheduling algorithm has been adopted to relocate the non-zero entries of the sparse matrix to ensure a continuous flow of data at each core, thus eliminating redundant memory requests and unnecessary intermediate result storage. Consequently, effects of the memory bandwidth bottleneck are reduced. Dedicated Multiplication and Accumulation (MAC) cores have been used in the experiments presented in this paper. The correct functionality of the proposed method is verified through Modelsim simulations. A thorough investigation of the effectiveness of the new format was carried out using a number of case studies with variable sparsities and matrix sizes, resulting to a speedup that reaches up to four orders of magnitude compared with an Intel i7 CPU.