Large Language Models (LLMs) have demonstrated transformative potential, with promising insights that could potentially enhance clinical practice. Their application in overall healthcare practices is increasingly noted, but their cumulative usage in the field of urology is less explored. Our study aimed to systematically review the current evidence on the usage of LLMs in urology and perform a quantitative analysis of comparable studies.

We searched electronic databases namely, Pubmed, Embase, Web of Science, and Scopus to include studies involving urological data or patients in which LLMs were utilized in clinical management or other relevant applications. The studies were grouped based on the application of the LLM that was being studied as: 1) Answering FAQs, 2) Patient Information Materials, 3) Clinical Practice, 4) Medical examination and 5) Other applications. The quality assessment was done utilising the Newcastle-Ottawa Scale.

A total of 814 articles were found, and after the removal of duplicates and screening of the articles, 39 studies were included in our review. Various applications of LLMs in different domains were listed, and details regarding the training, outcomes, and limitations were studied. ChatGPT was the most commonly studied LLM. The pooled accuracy of the output of various LLMs on various medical examinations was found to be 63.24 (CI 53.69 – 72.72), and a forest plot was constructed. Studies analysing the output of LLMs in answering frequently asked questions reported an accuracy of 67.08% to 100%, with ChatGPT – 4 outperforming ChatGPT – 3.5.

The findings of this systematic review and meta-analysis underscore the growing role of LLMs in urology, highlighting their potential to enhance clinical practice across diverse applications. The pooled accuracy of 63.24% for LLM performance on medical examinations reflects moderate reliability, with variability likely influenced by differences in training data, model architecture, and evaluation methods. Notably, ChatGPT emerged as the most frequently studied LLM, with its latest version (ChatGPT-4) demonstrating superior accuracy (67.08% to 100%) in answering frequently asked questions compared to its predecessor, ChatGPT-3.5. This suggests that advancements in LLM iterations can significantly improve performance, particularly in patient-facing tasks. However, the results also reveal a lack of standardization in evaluation metrics and methodologies across studies, which poses challenges for consistent benchmarking and integration into clinical workflows.

LLMs' applications are diverse and have augmented urological practice. However, uniform evaluation methods and performance metrics for assessing the output generated for various purposes are needed to further streamline the use and synchronous incorporation of LLM in regular urological practice.