Actions That Do Nothing: Stationary Actions as an RL Pitfall

Authors Ali Backour and Sarah Mokhtar
Published May 2026
Kind Course project note
Outline
The pitfall Stationary actions Why they break learning Why penalties failed Action masking Results Limitations

The pitfall

When building a reinforcement learning environment, it is tempting to expose a fixed action space and let the agent learn everything from rewards. In 2048, the fixed action space is tiny: up, down, left, and right. But depending on the board, some of these actions do absolutely nothing.

For example, if every row is already pushed as far left as possible and no adjacent tiles can merge, then the action left leaves the board unchanged. The environment accepts the action, but the state does not move.

Small lesson: actions that leave the state unchanged are not neutral. In value-based RL, they can create self-reinforcing loops.

This post is a short case study from a course project on RL for 2048. The goal is not to present a state-of-the-art 2048 agent. The useful lesson is narrower: stationary actions can quietly break training, and simply giving them negative reward did not fix the problem in our experiments. The cleaner solution was to mask them out.

Short version: if an action deterministically keeps the agent in the same state, do not just hope the reward function teaches the agent to avoid it. If you can identify it, mask it.
This is best read as an implementation/diagnostic note: a small environment-design issue that created a surprisingly large learning problem.

What is a stationary action?

In our 2048 environment, a state is a $4 \times 4$ board and the action set is

$$\mathcal{A}=\{\text{up},\text{down},\text{left},\text{right}\}.$$

An action $a$ is stationary at state $s$ if taking it leaves the environment unchanged:

$$\Pr(S_{t+1}=s \mid S_t=s, A_t=a)=1.$$

This is not a rare corner case. As a 2048 board fills up, it becomes more constrained: fewer directions are able to slide or merge tiles. In random play, the fraction of stationary actions tends to increase as the game progresses.

Example 2048 board
Example 2048 board
Percentage of stationary actions during random play
Stationary actions become more common
Figure 1: 2048 naturally creates state-dependent action availability. The action set is always displayed as four moves, but the set of moves that actually change the board depends on the state. As the board becomes more congested, useless moves become more frequent.
This is the key distinction: the action is not syntactically illegal, but it is dynamically useless in the current state.

Why can this break value-based learning?

At first, stationary actions seem harmless. If an action gives no reward and changes nothing, shouldn't the agent eventually learn not to take it? In tabular RL, often yes. But with deep Q-learning, function approximation, bootstrapping, exploration, and replay buffers make the failure mode more subtle.

For a stationary action, the agent observes a transition of the form

$$(s,a,r,s).$$

The next state is the same as the current state. Therefore the Bellman target for a Q-learning style update bootstraps from the same state:

$$y = r + \gamma \max_{a'} Q_{\theta^-}(s,a').$$

This creates a self-referential update. If $Q(s,a)$ for the stationary action is overestimated, the policy may choose it again. That produces another transition from $s$ back to $s$, which adds another nearly identical sample to the replay buffer. The agent can waste many updates learning from transitions that contain no new state information.

The paper version wrote the intuition using the simplified stationary-action TD update

$$\theta_{t+1} = \theta_t + \eta\bigl(R-(1-\gamma)Q_\theta(s,a)\bigr) \nabla_\theta Q_\theta(s,a).$$

The term $R-(1-\gamma)Q_\theta(s,a)$ is the important part. If the action is stationary, lowering its value through reward penalties requires $R$ to be sufficiently negative relative to the current value estimate. If the penalty is too small, the update may not push the action down enough. If the penalty is very large, it can dominate the loss and destabilize training.

Intuition: stationary actions can become sticky. The agent chooses them because they are overestimated, then the resulting transition sends it back to the same state, creating more data that reinforces the same local failure mode.
The issue is not only that stationary actions are bad decisions. They also create bad training data: repeated transitions with no movement through the state space.

Why not just give negative rewards?

The obvious fix is to penalize stationary actions. We tried that first. We trained DDQN while sweeping the reward for stationary actions from 0 down to -1000.

This did not cleanly solve the problem. Across the penalty sweep, the agent still struggled to learn meaningful behavior and repeatedly fell into loops of stationary actions. A random agent achieved an average score around 1000, while the unmasked DDQN variants remained poor and unstable.

DDQN performance with different stationary-action penalties
Figure 2: Penalizing stationary actions did not fix the failure mode. We swept negative rewards for stationary actions from 0 to -1000. The distributions remained clustered at weak performance, showing that simply making useless moves more expensive was not enough to make DDQN learn a good policy.

The reason is that a penalty is only an indirect signal. The agent still has to explore the bad action, store the transition, and learn from the target. If the penalty is mild, the bootstrapped value can still make the action look acceptable. If the penalty is extreme, the learning signal can become harsh and unstable under function approximation.

In other words, negative rewards ask the agent to learn a fact we already know: this action cannot change the state. When the environment can compute that fact exactly, it is cleaner to remove the action from consideration.

Main lesson from the ablation: reward shaping was a brittle way to handle stationary actions. The action set itself was wrong for the current state.
This is the central figure of the post. It is what makes the lesson more than “invalid actions are bad”: the naive penalty fix did not work.

The solution: action masking

Instead of penalizing stationary actions, we mask them. At each state, the environment computes whether each action changes the board:

$$m(s,a)= \begin{cases} 1, & f_a(s)\neq s,\\ 0, & f_a(s)=s. \end{cases}$$

For DQN, we can assign masked actions a value of $-\infty$ before choosing an action:

$$\widetilde{Q}(s,a)= \begin{cases} Q(s,a), & m(s,a)=1,\\ -\infty, & m(s,a)=0. \end{cases}$$

Then the greedy action is selected only among moves that actually change the state:

$$a_t \in \arg\max_a \widetilde{Q}(s_t,a).$$

This changes the learning problem in a simple but important way. The agent no longer spends exploration or replay-buffer capacity on transitions of the form $(s,a,r,s)$ when $a$ was known in advance to do nothing. It learns among the meaningful actions available at that state.

Masking is not just a performance trick: it encodes the state-dependent action set. The environment says, “these actions are possible in the API, but they are not meaningful here.”
In this setting, masking is more principled than reward penalties because stationarity is exactly detectable before taking the action.

What happened after masking?

After adding action masking, DDQN started to learn meaningful behavior. With one million iterations, the masked agent reached an average reward around 2417 in evaluation. With longer training, around twenty million iterations, this increased to about 3153.

Masked DQN training and evaluation distribution
Figure 3: Masking enabled DDQN to learn instead of looping on stationary actions. Left: reward improves over training once stationary actions are excluded. Right: the evaluation distribution shifts upward when training is extended from one million to twenty million iterations.

These numbers are not meant to compete with optimized 2048 solvers. The useful empirical point is simpler: the unmasked penalty-based approach failed to produce healthy learning, while masking the stationary actions produced a much cleaner training problem.

Main empirical result: the fix was not “find the right negative reward.” The fix was to prevent the agent from selecting actions that provably cannot change the state.
This is why I would present the project as an RL environment-design lesson rather than as a 2048-performance result.

Limitations

This is a small course-project case study. We did not build a state-of-the-art 2048 solver, and the results should not be interpreted that way. The value of the experiment is diagnostic: it shows how a seemingly harmless environment design choice can make learning much harder.

The conclusion also depends on being able to identify stationary actions exactly. In 2048 that is easy: simulate the move and check whether the board changes. In other environments, invalid or useless actions may be harder to detect, and one might need to learn an action-elimination model instead.

Conclusion

Actions that do nothing are not always harmless. In 2048, they become more common as the board fills up, and they can create self-reinforcing loops for value-based agents. Penalizing them with negative reward was brittle in our DDQN experiments. Masking them out was the clean intervention: it removed uninformative transitions, made the action set state-dependent, and allowed learning to proceed.

The broader lesson is simple: when designing an RL environment, do not only ask what actions are syntactically allowed. Ask which actions are meaningful in the current state. If the environment knows an action cannot matter, the agent may not need to waste samples discovering that from scratch.

References

[1] H. van Hasselt, A. Guez, and D. Silver. Deep Reinforcement Learning with Double Q-Learning. AAAI, 2016.

[2] T. Zahavy, M. Haroush, N. Merlis, D. J. Mankowitz, and S. Mannor. Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning. arXiv, 2018.

The blog framing matters: this is not “we solved 2048,” but “here is a clean RL pitfall and the fix that mattered.”