Interobserver variability in LOCS III cataract grading across experience levels: implications for AI training label quality
Author:
Corresponding Author:

Xu-Dong Song and Jing Tao. Beijing Tongren Eye Center, Beijing Tongren Hospital, Beijing Ophthalmology and Visual Science Key Lab, Capital Medical University, No.1 Dongjiaominxiang Street, Dongcheng District, Beijing 100730, China. drxdsong@sina.com; taojing@mail.ccmu.edu.cn

Affiliation:

Clc Number:

Fund Project:

Supported by Beijing Nova Program (No.20240484601); Beijing Natural Science Foundation (No.7242038; No.7264252).

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
  • |
  • Comments
    Abstract:

    AIM: To evaluate the variability in cataract grading based on the Lens Opacities Classification System III (LOCS III) among ophthalmologists with different experience levels, and identify which type of annotated data may serve as high-quality labels for artificial intelligence (AI) model development. METHODS: Ophthalmologists were grouped by clinical experience (resident, senior, expert) and independently graded nuclear, cortical, and posterior subcapsular cataract (PSC) according to LOCS III. Consistency was analyzed at multiple levels: label consistency (presence/absence of opacity), grading consistency (strict and tolerant), and cross-subtype consistency using presence scores (agreement on cataract detection) and pairwise exact-match proportions (agreement on identical LOCS III grade). Intragroup reliability was assessed using intraclass correlation coefficients (ICCs), while intergroup and cross-subtype comparisons were conducted using Friedman and Kruskal-Wallis tests with Bonferroni-corrected post-hoc analyses. RESULTS: A total of 138 participants with a mean age of 66.64±10.09y (77 females) diagnosed with cataracts were included in the study, yielding 644 anterior segment slit-lamp photographs under mydriatic conditions. Each image was annotated independently by six ophthalmologists, including two residents, two attending-level seniors, and two experts. Experts achieved the highest intragroup agreement (ICC=0.780), while senior ophthalmologists and residents showed comparable moderate agreement. Intergroup comparison showed significantly higher label and grading consistency between senior-expert pairs than between resident-senior or resident-expert pairs, especially for cortical cataracts and PSC (P<0.05). Among subtypes, nuclear cataracts achieved the highest overall consistency, while cortical cataracts and PSC showed lower agreement, likely due to diffuse or obscured opacities and image quality variability (P<0.001). CONCLUSION: Consistency in LOCS III-based cataract grading varies with clinical experience and cataract subtype. Expert and senior ophthalmologists provide more stable annotations, and nuclear cataract images demonstrate superior interobserver agreement. These findings emphasize the value of expert consensus and subtype-specific data curation for improving AI-based cataract grading models.

    Reference
    Related
    Cited by
Get Citation

Zhen-Yu Wang, Xi-Jin Wu, Ran Zhang, et al. Interobserver variability in LOCS III cataract grading across experience levels: implications for AI training label quality. Int J Ophthalmol, 2026,19(12):2313-2320

Copy
Article Metrics
  • Abstract:
  • PDF:
Publication History
  • Received:April 17,2026
  • Revised:July 27,2026
  • Adopted:
  • Online: November 16,2026
  • Published: