Based on two-player two-action and three-action game models, this paper studies the dynamics of Q-learning and Frequency Adjusted Q-(FAQ-) learning algorithms in multi-agent systems, and discloses the underlying mechanisms of these algorithms through the perspective of evolutionary dynamics. It is showed that the dynamics of FAQ-learning or Q-learning with Boltzmann exploration mechanism corresponds to the evolutionary dynamics of selection mechanism with the linear or super-exponential growth, respectively. Hence, FAQ-learning algorithm can converge to the equilibrium state of a game model, whereas, the convergence of Q-learning algorithm is related with the initial states of the population. Therefore, the continuous evolutionary dynamics with selection mechanism can predict the learning process of discrete Q-learning like algorithms well.