Abstract:AIM: To evaluate the variability in cataract grading based on the Lens Opacities Classification System III (LOCS III) among ophthalmologists with different experience levels, and identify which type of annotated data may serve as high-quality labels for artificial intelligence (AI) model development. METHODS: Ophthalmologists were grouped by clinical experience (resident, senior, expert) and independently graded nuclear, cortical, and posterior subcapsular cataract (PSC) according to LOCS III. Consistency was analyzed at multiple levels: label consistency (presence/absence of opacity), grading consistency (strict and tolerant), and cross-subtype consistency using presence scores (agreement on cataract detection) and pairwise exact-match proportions (agreement on identical LOCS III grade). Intragroup reliability was assessed using intraclass correlation coefficients (ICCs), while intergroup and cross-subtype comparisons were conducted using Friedman and Kruskal-Wallis tests with Bonferroni-corrected post-hoc analyses. RESULTS: A total of 138 participants with a mean age of 66.64±10.09y (77 females) diagnosed with cataracts were included in the study, yielding 644 anterior segment slit-lamp photographs under mydriatic conditions. Each image was annotated independently by six ophthalmologists, including two residents, two attending-level seniors, and two experts. Experts achieved the highest intragroup agreement (ICC=0.780), while senior ophthalmologists and residents showed comparable moderate agreement. Intergroup comparison showed significantly higher label and grading consistency between senior-expert pairs than between resident-senior or resident-expert pairs, especially for cortical cataracts and PSC (P<0.05). Among subtypes, nuclear cataracts achieved the highest overall consistency, while cortical cataracts and PSC showed lower agreement, likely due to diffuse or obscured opacities and image quality variability (P<0.001). CONCLUSION: Consistency in LOCS III-based cataract grading varies with clinical experience and cataract subtype. Expert and senior ophthalmologists provide more stable annotations, and nuclear cataract images demonstrate superior interobserver agreement. These findings emphasize the value of expert consensus and subtype-specific data curation for improving AI-based cataract grading models.