Towards automated clash resolution of reinforcing steel design in reinforced concrete frames via Q-learning and building information modeling

The design of reinforcing steel bars (rebars) is critical to reinforced concrete (RC) structures. Generally, a good number of rebars are required by a design code, particularly at member connections. As such, rebar clashes (i.e., collisions and congestions) would be inevitable. It would be impractical, labor-intensive, and error-prone to avoid all possible clashes manually or even using standard design software. The building information modeling (BIM) technology has been utilized by the present architecture, engineering, and construction (ACE) industry for clash-free rebar designs. However, most existing BIM-based approaches offer the clash resolution strategy for moving components with an optimization algorithm, and are only applicable to the RC structures with regular shapes. In particular, the optimized path of rebars cannot be adjusted to avoid the obstacles, thus limiting the practical applications. Furthermore, most existing studies lack the learning from design code and constructibility constraints to realize automatic and intelligent arrangement and adjustment of rebars for avoiding the obstacles encountered in complex RC joints and frame structures. Considering these shortcomings, the authors have recently proposed an immediate reward-based multi-agent reinforcement learning (MARL) system with BIM, towards automatic clash-free rebar designs of RC joints without clashes. However, as the immediate reward is required in the MARL system for guiding the learning of a rebar design, it will not succeed in clash-free rebar designs of complex RC structures where immediate reward is often unavailable. In this study, this study further extends the previous work with Q-learning (a model-free reinforcement learning algorithm) for more realistic path planning considering both immediate and delayed rewards in clash-free rebar designs for real-world RC structures. In particular, the rebar design problem is treated as a path-planning problem of multi-agent system, where each rebar is deemed as an intelligence reinforcement learning agent. Next, by employing the Q-learning as the reinforcement learning engine, the particular form of state, action, and immediate and delayed rewards for the reinforcement MARL for automatic rebar designs considering more actual constructible constraints and design codes can be developed. Comprehensive experiments on three typical beam-column joints and a two-story RC building frame were conducted to evaluate the efficiency of the proposed method. The study results of paths of rebar designs, success rates, and average time confirm that the proposed framework with MARL and BIM is effective and efficient.