J Med Internet Res. 2026 Jul 10;28:e85290. doi: 10.2196/85290.
ABSTRACT
BACKGROUND: Online health communities (OHCs) are important channels for families of children with autism spectrum disorder to obtain health information and psychosocial support. Differences between an open forum platform and the physician-patient consultation platforms may shape caregiver decisions, yet comparative evidence from China remains limited. A large language model (LLM) provides a scalable approach for systematic content annotation in large OHC datasets.
OBJECTIVE: This study proposes and validates a standardized LLM-assisted annotation framework under a unified classification schema and compares topic distributions and poster identities across an open forum platform (Baidu Tieba) and physician-patient consultation platforms (Chunyu Doctor and Haodf).
METHODS: We implemented an LLM-assisted annotation framework. A unified taxonomy of topics and poster identities was first developed through human open coding. Poster identities in Baidu Tieba were annotated through a double-blind manual procedure. For topic classification, interannotator and human-LLM agreement were evaluated on a manually labeled subset to benchmark models of varying sizes. The best-performing LLM was selected for full-dataset topic annotation, followed by statistical and cross-platform analysis.
RESULTS: When metrics were arithmetically averaged across all annotation tasks, the best-performing LLM achieved agreement levels comparable to human annotation (accuracy=79.18%, SD 0.20%; κ=0.736, SD 0.003; F1-score=0.727, SD 0.006), approaching interannotator agreement (accuracy=81.65%; κ=0.767; F1-score=0.758), demonstrating strong stability and scalability. Full-dataset analysis yielded 3 main findings. First, model performance increased with parameter scale but plateaued beyond 14B, indicating diminishing marginal returns from further scaling. Second, clear cross-platform differences in poster identity were observed: the open forum platform was dominated by family members of patients (caregivers: 2377/3516, 67.61%), with substantial participation from commercial rehabilitation practitioners (commercial posters: 427/3516, 12.14%), resulting in a more heterogeneous participation structure. Third, topic distributions reflected both shared high demand for resource-related information and differentiated help-seeking pathways: both platform types demonstrated consistently high demand for resource recommendation and evaluation; the open forum platform was primarily characterized by diagnosis-related discussions (1183/7535, 15.70%), whereas the physician-patient consultation platforms were centered on intervention-related consultation (2864/7687, 37.26%).
CONCLUSIONS: The LLM-assisted annotation framework proposed in this study enables reliable large-scale annotation of OHC data while maintaining high human-LLM agreement and operational stability. Midsized models (eg, 14B) demonstrated favorable cost-performance efficiency. The findings reveal 2 key aspects: the open forum platform exhibits a complex participation structure, and the influence of commercially affiliated actors should not be overlooked; users on both platform types show sustained demand for resource-related information but follow different help-seeking pathways, emphasizing diagnostic exploration and professional intervention, respectively. These results suggest that platform structure and governance mechanisms may shape caregivers’ information access and decision-making. The framework provides a transparent, reproducible, and cost-effective approach for OHC research. All data were deidentified and handled in accordance with relevant platform policies and ethical standards.
PMID:42430199 | DOI:10.2196/85290
