Recently, due to the high fault tolerance and low hardware cost, stochastic computing (SC) based-neural network (NN) accelerator has been widely developed. One of the big challenges of the SC-based NN accelerator is the implementation of accumulation and activation function, which has a significant impact on the accuracy of the network. However, the existing implementation scheme has the problems of low accuracy and high energy consumption. In this paper, based on the thermometer coding in parallel SC, we propose an accurate implementation of accumulation and non-linear function for SC, which is called the non-linear adder. A dedicated design for the commonly used activation functions of neural network, i.e, hyperbolic tangent (tanh), logistic (or sigmoid), and rectified linear units (ReLU), is proposed using the bitonic sort network and the selected internet.