Table Structure Recognition (TSR) aims to reconstruct the logical structure of a table to understand semantic information ordered in the table. Many approaches to modeling the TSR problem have been proposed and have achieved promising results. However, most heavy models or complex post-processing approaches require much time and data consumption for inference and training progress. This paper proposes a new TSR approach called RTSR, a robust simplifying modeling. RTSR includes a lightweight backbone and a module to enhance contextual information between rows/columns. We combine two stages in a split-and-merge manner into only one step by reconstructing a table with horizontal and vertical separators. Specifically, we redesign the split stage to identify grid and spanning cells. Our RTSR can run on average at 38.1 FPS while achieving comparable performance with state-of-the-art methods on several benchmark datasets, including SciTSR, PubTabNet, FinTabNet, and WTW.