In recent years, the architecture based on Resistive Processing Unit (RPU) has become a hot topic due to its potential to accelerate training of a Neural Network (NN). However, attempts to realize the RPU concept based on non-volatile memory technology face a myriad of technological and physical constraints. The theoretical concept of crossbar array is nearly impossible to implement in the real world without certain tweaks. Hence, we propose an Voltage output Complementary Metal Oxide Semiconductor (CMOS)-based RPU design VRPU, which is used to build a voltage-based neural network. We also introduce a novel diode-based circuit to behave as a non-linear activation function, which consists of a single diode (D) and diode pair (DP). The proposed VRPU design when tested with MNIST dataset for hidden layer and output layer combinations of ReLU+Sigmoid, D+Sigmoid, ReLU+DP, D+DP (low temperature) and D+DP (high temperature) resulted in accuracies of $94.29\\%$, $95.90\\%$, $95.53\\%$, $96.75\\%$ and $96.57\\%$ respectively corroborating the merits of the proposed design.