An evaluative analysis of particle swarm optimization for reinforcement learning in pendulum task

Abstract

Applying swarm intelligence algorithms to reinforcement learning of neural networks is practical because they do not rely on gradients. Particle swarm optimization (PSO) is a representatives of swarm algorithms. In this paper, the author experimentally evaluates the effectiveness of PSO in the reinforcement learning of multilayer perceptrons (MLPs), using a pendulum control task. Experimental results demonstrated the successful training of an MLP with 8 hidden units, enabling rapid uprighting of the pendulum. Notably, it was found that increasing the population size rather than the number of iterations allowed PSO to discover better solutions. In PSO, increasing the population size promotes global exploration in the early stages, while increasing the number of iterations enhances local exploitation in the later stages. Based on the results of this experiment, it is evident that in this learning task, early-stage global exploration is more important.