From Assistants to Agents: Evaluating Autonomous LLM Agents in Real-World DevOps Pipeline

Abstract

Software development and deployment automation has undergone a paradigm shift with the incorporation of autonomous large language model (LLM) agents into the DevOps process. Autonomous LLM agents are evaluated in this article for their performance across the DevOps lifecycle. This includes development, testing, deployment, and monitoring. The goals are to compare the agents' performance to that of traditional DevOps tools and to assess the operational efficacy, scalability, and decision-making quality of LLM agents. Key performance variables like deployment time, mistake rates, and automation efficiency are examined in the study using case studies and an intensive data collection approach. Based on the findings, LLM agents should be able to maximize decision-making in CI/CD pipelines, speed up automation, and significantly reduce human intervention. A framework for evaluating LLM agents and recommendations for improving their integration and practical implementation are the paper's primary contributions.