Abstract:AIM: To evaluate the accuracy and appropriateness of responses of different large language models (LLMs) to ophthalmology case-based questions. METHODS: Prospective, observational, cross-sectional study conducted in 2025 in Ontario, Canada. Three LLMs (Gemini Advanced 2.5 Pro, ChatGPT 4.5, and DeepSeek DeepThink R1) were asked to respond to all “Clinical Challenge” cases published in The Journal of the American Medical Association Ophthalmology from 2021 to 2024 (135 multiple-choice questions). They were evaluated for accuracy and response appropriateness. Chi-square and Kruskal-Wallis tests were used to compare their performances. RESULTS: The average accuracy across all LLMs was 303/405 (74.8%), with Gemini leading at 106/135 (78.5%), followed by ChatGPT (99/135; 73.3%) and DeepSeek (98/135; 72.6%). The pairwise differences among the LLMs were not statistically significant. The average appropriateness score of all LLMs together was 4.44 out of 6. Gemini, ChatGPT, and DeepSeek had average appropriateness scores of 4.6, 4.2, and 4.5, respectively (P=0.007). CONCLUSION: These LLMs reveal a moderate-to-high level of accuracy and appropriateness of reasoning when answering complex ophthalmology case-based questions. While ChatGPT and DeepSeek differ by one correct answer, Gemini outperforms them by over 5%. When using LLMs for clinical decision-making, the model’s ability to provide quality explanations for the selected course of action is an asset. Although Gemini is the overall leader in appropriateness of reasoning, the fact that DeepSeek outperforms ChatGPT in appropriateness, combined with its open-source, free-to-use, and offline-capable features, makes it an exciting prospect for future use in ophthalmology.