What question did this study set out to answer?

The study aims to optimize chemical dosing in wastewater treatment using model-based reinforcement learning techniques.

June 3, 2026Processes0 citationsOpen Access

Model-Based Reinforcement Learning for Chemical Dosing Optimization in a Municipal Wastewater Treatment Plant: A Comparative Study of Three Actor–Critic Algorithms

Key Points

The study aims to optimize chemical dosing in wastewater treatment using model-based reinforcement learning techniques.
A multilayer perceptron (MLP) model was developed using 227 historical operation records to train the three algorithms.
Three actor–critic algorithms—SAC, TD3, and PPO—were compared over 50,000 training steps and evaluated over 50 test episodes.
A composite reward function was created to balance effluent quality against operational costs.
SAC achieved a mean score of 70.23 (95% CI: [67.94, 72.52]), TD3 scored 70.33 (95% CI: [68.02, 72.64]), and PPO scored 68.78 (95% CI: [66.47, 71.09]).
The proportional control method scored 61.01 (95% CI: [58.23, 63.80]), while random dosing achieved 45.87 (95% CI: [41.29, 50.45]).
Both SAC and TD3 demonstrated equivalent control performance (p = 0.0631), leading to low-dosage strategies.

Abstract

Chemical dosing control in wastewater treatment plants (WWTPs) necessitates a dynamic equilibrium between effluent compliance and operational costs, exemplifying a typical multi-objective sequential decision problem. Given the significant operational and compliance risks associated with online trial-and-error methods in full-scale plants, this study introduces a model-based reinforcement learning (MBRL) framework aimed at optimizing dosing. Utilizing 227 historical operation records, we construct a multilayer perceptron (MLP) virtual WWTP that maps 15 process states and 2 dosing actions to 6 effluent indicators, thereby providing a secure training environment for policy development. A composite reward function is formulated to incorporate penalties for effluent quality, constraints for abnormal conditions, and terms related to chemical costs. Three actor–critic algorithms—Soft Actor–Critic (SAC), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Proximal Policy Optimization (PPO)—are trained for 50,000 steps and evaluated over 50 test episodes against both a random baseline and a practically deployed proportional feedforward control baseline. All three reinforcement learning methods yield significantly higher rewards: SAC achieves a mean score of 70.23 (95% CI: 67.94, 72.52), TD3 scores 70.33 (95% CI: 68.02, 72.64), and PPO scores 68.78 (95% CI: 66.47, 71.09), compared to 61.01 (95% CI: 58.23, 63.80) for proportional control and 45.87 (95% CI: 41.29, 50.45) for random dosing. Notably, both off-policy agents (SAC and TD3) demonstrate statistically equivalent, state-of-the-art control performance (paired Wilcoxon, p = 0.0631) and converge towards highly economical, low-dosage strategies. This study validates the feasibility of data-driven virtual commissioning for WWTP dosing optimization and supports the advancement of hybrid intelligent control systems that incorporate mechanistic constraints. In addition to algorithm comparisons, this framework can be further refined to serve as a practical, data-driven decision support tool for wastewater treatment plant (WWTP) operators. It enables them to formulate daily chemical dosing plans while adhering to compliance and cost constraints.

Ask AI

Helpful

Bookmark

View Full Paper