Implemented Q-learning with a Q-table to achieve desired position for simple pendulum
Simulation of Simple Pendulum
The inverted pendulum has a limit on the maximum torque it can apply, therefore it is necessary for the pendulum to do a few "back and forth" motions to be able to reach the inverted position (
Pendulum Model
In the following,the vector of states of the system is given by:
We will also work with time-discretized dynamics, and refer to
We want to minimize the following discounted cost function
This cost mostly penalizes deviations from the inverted position but also encourages small velocities and control.
The Q-learning algorithm is implemented with a table. For the action value function given in equation,
Q-table
Q-table gives the quality of the state and action pair, where value of
To obtain next state in above equation,
The action-value function combines all results of the single-stage predictive search. For each state-action pair, the optimal expected long-term return is displayed, allowing the selection of optimal actions without the knowledge of future states and their value, and thus without knowing anything about the dynamics of the environment.
Important for Reinforcement is that both, policy, as well as value function, can be learned and lead to a close optimal behavior.
Epsilon-Greedy is a simple method to balance exploration and exploitation by choosing between exploration and exploitation randomly. In this method, epsilon refers to the probability of choosing to explore, exploits most of the time with a small chance of exploring given by:
Q-learning finds an optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps, starting from the current state. Temporal difference error is calculate over eachstep size using equation,
and the value of Q is updated by adding the product of learning rate and tempral difference error to Q as shown in equation,
By taking control
Plot of cost vs episodes
The pendulum took 2 swings to reach the inverted position in almost 3 seconds which can be observed from the plots of states w.r.t. time
Plot of states vs time
From the control plots, it can be observed that for the values of control in a given range it is taking more time to be at the same value of control during that time the pendulum is trying to attain momentum to reach the swing upward condition. When it is close to the desired condition it is taking less time to be at the same control value and after reaching the inverted position the control is oscillating at a high rate between an upper and lower limit which is equivalent to zero control.
To understand it in a better way we can compare the plot of omega with the plot of controls and understand that whenever the value of omega is positive, the control is positive and vice versa.
Plot of control vs time
The plot of value function with respect to states can be observed that the value function is maximum when
![]() |
![]() |
---|---|
Plot of Value Function | Plot of Policy |