Human-Centered Automation in Software Engineering Bridging Evaluation, Comprehension, and Practice

Devjeet Raj Roy

doi:10.7273/000008341

The rapid expansion in scale and complexity of software systems over the past two decades has necessitated the development of advanced software engineering (SE) technologies aimed at enhancing developer productivity and reducing maintenance costs. A central challenge in this evolution is ensuring that these automated systems align with human factors--developer perceptions, needs, and workflows--to be effective and widely adopted. Misalignment can lead to tools that, despite technical sophistication, fail to deliver practical benefits to their intended users. This work focuses on bridging the gap between automated SE technologies and human factors, emphasizing the importance of a human-centric approach in the development and evaluation of these tools. We examine how reliance on automatic evaluation metrics as proxies for human assessment can create a disconnect between SE technologies and developers across multiple domains. In particular, we study code summarization, where automatic metrics are pivotal benchmarks yet may not reflect developer preferences, and code readability assessment, where metrics intended for direct use by developers may not align with their perceptions of readability--especially when code evolves through incremental changes. Our findings reveal that commonly used automatic metrics often fail to accurately represent human assessments, underscoring the need for evaluation methods that better reflect developer judgments in practice. Next, we explore how integrating insights from human factors research, such as program comprehension studies, can enhance the effectiveness of SE technologies. First, we propose human-centered interventions in automated testing--an underutilized domain with a wealth of literature that has not seen large-scale adoption. By using semantically meaningful test names and natural-language commentary, the intervention is designed to align more closely with real-world workflows and is preferred by practitioners over traditional methods. Second, we develop an approach for large language model applications in SE that calibrates model outputs to meet target confidence levels in practice, enabling the system to provide answers when reliable and defer when not, with the goal of increasing developers’ trust in the system. This work underscores the necessity of aligning automated SE technologies with human factors at every stage--from evaluation to system design. By placing human comprehension at the forefront of computation, this work contributes to the development of more effective, adoptable, and trustworthy tools in software engineering, ultimately bridging the divide between automated systems and their human users.

Human-Centered Automation in Software Engineering Bridging Evaluation, Comprehension, and Practice

(1)