Summary
Introduction: Computed tomography (CT) has approximately 98% sensitivity for Coronavirus disease-2019 (COVID-19). Various algorithms were designed using CT images. However, the interobserver agreement of different radiological classifications of COVID-19 is not yet known. Thus, this study aimed to investigate the interobserver agreement of different radiological classifications of COVID-19.
Materials and Methods: This study included 212 patients who were positive on the polymerase chain reaction test and eligible for CT. Four radiologists examined all CT images simultaneously. They reached a consensus that CT images can provide definite findings of COVID-19. The Radiological Society of North America (RSNA) consensus statement, the British Society of Thoracic Imaging (BSTI) structured reporting statement, and COVID-19 Reporting and Data System (CO-RADS) were used. Fleiss’ Kappa was used to detect interobserver agreement. Kappa values of 0.00-0.20 were considered as slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0.81-1.00 as near-perfect agreement, and p<0.05 was accepted as significant.
Results: A total of 137 patients did not have any pathological CT findings. The most prevalent radiological findings were ground-glass opacities and consolidations. The agreements on all classifications were at near-perfect levels: RSNA, 0.86 (0.82-0.90); BSTI, 0.83 (0.79-0.87), and CO-RADS, 0.82 (0.79-0.86). The RSNA classification has the highest consistency rate, followed by BSTI and CO-RADS. However, substantial and moderate agreements were found in the subcategories of each classification.
Conclusion: In this study, some subcategories had a lower agreement, despite the high consistency rates for COVID-19 radiological classification systems in the literature. Therefore, improving the items without consensus can lead to the development of better radiological diagnostic approaches.
Introduction
Severe acute respiratory syndrome Coronavirus-2, a novel Coronavirus that spread from Wuhan, China, has become one of the biggest threats that the world has recently faced[1]. Common symptoms of the resultant infection, Coronavirus disease-2019 (COVID-19), include fever and respiratory symptoms such as cough and dyspnea[2]. Pneumonia, severe acute respiratory infection, kidney failure, and even death may occur in more severe cases[2].
Although the real-time reverse transcription-polymerase chain reaction (RT-PCR) test for viral nucleic acids is the gold standard in the diagnosis of COVID-19, it may reveal false-negative results; thus, computed tomography (CT) has taken over this diagnostic role[3].
Chest radiography has low sensitivity, and images without pathological signs cannot exclude the diagnosis. Thus, chest CT is considered to obtain detailed information[4]. Recent studies have shown that CT has reached a sensitivity of 98%[5, 6]. The typical CT findings of COVID-19 pneumonia are multifocal ground-glass areas with peripheral and posterior distribution. They are predominantly found in the lower lobe, mostly bilaterally located, and may be accompanied by focal consolidations. Various algorithms have been created on CT, thanks to the more widespread recognition of its findings[7]. To date, various structured reporting systems and classifications can be used to create a common language in reporting CT examinations to facilitate standardization and patient management. To the best of our knowledge, this standardization has not been fully achieved yet, and no study has compared these classifications. Thus, this study aimed to evaluate imaging findings of patients who were positive on the PCR test for COVID-19 and to assess the consistency and efficiency of classifications in the literature.
Methods
This retrospective study was conducted in Kırıkkale University Faculty of Medicine and Kırıkkale Yüksek İhtisas Hospital according to the principles of the Declaration of Helsinki. The relevant ethical approval was taken from the Ethics Committee of Kırıkkale University (approval no.: 2020.05.07, date: 20.05.2020) and the COVID-19 Scientific Research Committee of the Ministry of Health.
Subjects
From March 2020 to July 2020, 328 patients diagnosed with COVID-19 by PCR test in the Kırıkkale province were examined, but 116 patients without a chest CT result were excluded from the study. Finally, the remaining 212 patients were included in the study.
Although the patients have taken follow-up CT during the study, only their initial chest CT scans were evaluated. The time that has elapsed between the initial appearance of symptoms and the date of the initial chest CT was noted for each patient. The chest CT images were obtained from the Picture Archiving and Communication Systems.
CT Imaging and Analysis
The images of 41 patients who underwent CT with 3-mm-thick slices using a Brilliance 64 scanner (Philips Medical System, Best, Netherlands) were obtained from Kırıkkale University Faculty of Medicine. The images of 171 patients who underwent CT with 3-mm-thick slices using an Alexion 16 scanner (Toshiba Medical Systems, Japan) were obtained from Kırıkkale Yüksek İhtisas Hospital. All scans were obtained with a routine chest CT imaging procedure without radiocontrast agents and sedation. The images were transferred to a workstation.
For each patient, chest CT images were evaluated for the following characteristics: presence, amount, and distribution pattern of ground-glass opacities, consolidation, air bronchogram, number of lobes affected where ground-glass or consolidative opacities are present, nodules, pleural effusion, thoracic lymphadenopathy (defined as lymph node size >10 mm in short-axis size), airway abnormalities (including airway wall thickening, bronchiectasis, and endoluminal secretions), and underlying lung diseases, such as emphysema or fibrosis. Moreover, opacities with a crazy-paving pattern, reversed halo sign, rounded morphology, and intralesional cavitation and linear opacities were noted.
All CT images were simultaneously examined by four radiologists (A.Ö., S.Y., S.P.K.E., and A.G.) with approximately 10, 27, 5, and three years of experience, respectively, using an imaging console. The radiologists reached a consensus that CT images can provide definite findings of COVID-19.
According to the Radiological Society of North America (RSNA) consensus statement, the images were separated into four groups, namely, typical appearance (imaging features with greater specificity for COVID-19 pneumonia), indeterminate appearance (non-specific imaging features of COVID-19 pneumonia), atypical appearance (uncommonly or not reported features of COVID-19 pneumonia), and negative for pneumonia (no features of pneumonia)[8].
Using the structured reporting statement of the British Society of Thoracic Imaging (BSTI), CT images were evaluated within four groups: classic COVID-19, probable COVID-19, indeterminate, and non-COVID[9].
The COVID Working Group of the Dutch Radiological Society described the COVID-19 Reporting and Data System (CO-RADS)[10]. This classification consists of seven subgroups. CO-RADS 0 represents incomplete testing, while CO-RADS 6 represents the PCR-positive group. Categories 0 and 6 were excluded from the study because there was no bad image, and all participating patients were positive in the PCR test.
The images were evaluated separately by four radiologists according to the BSTI, RSNA, and CO-RADS classifications. The discriminatory power and consistency of the classifications were then compared.
Statistical Analysis
IBM Statistical Package for the Social Sciences 26.0 (IBM Corp., NY, USA, Armonk) was used for all statistical analyses. Categorical variables were given as numbers and percentages (%). Values with normal distribution were presented as median (interquartile range [IQR]). Fleiss’ kappa was used to detect interobserver agreement. Kappa values were interpreted as follows: 0.00-0.20, slight agreement; 0.21-0.40, fair agreement; 0.41-0.60, moderate agreement; 0.61-0.80, substantial agreement; 0.81-1.00, near-perfect agreement[11]; p<0.05 was accepted as significant in all statistical analyses.
Results
The median age of the patients was 39 (IQR, 26) years. While 108 of the participants were female (50.9%), 104 were male (49.1%). The most common symptom was fever (n=75). Sixty-six patients experienced a single symptom, while 37 patients were asymptomatic. Symptoms and comorbidities on admission are presented in Table 1.
The most common radiological findings were ground-glass opacities and consolidations. In total, 137 patients had no radiological findings, 75 patients had involvements of one or more lobes, and 30 patients had involvement of all lobes
(Table 2).
During the disease course, among the patients with normal CT findings, 28 were on days 0-4, 87 on days 4-8, and 22 on days 9-14. Among those with pathological CT findings, 11 patients were found to have such chest CT findings on days 0-4, 51 on days 4-8, and 13 on days 9-14.
The agreements on all classifications were at near-perfect levels: RSNA, 0.86 (0.82-0.90); BSTI, 0.83 (0.79-0.87), and CO-RADS, 0.82 (0.79-0.86). Substantial and moderate agreements were also found for each classification.
In RSNA subcategories, typical appearance demonstrated near-perfect agreements, but negative agreements were found for pneumonia categories, while substantial agreements were observed for indeterminate appearance and atypical appearance. CORADS 3 and 4 demonstrated moderate agreement, CORADS 2 showed substantial agreement, and CORADS 1 and 5 demonstrated near-perfect agreement. For BSTI classification, a near-perfect agreement was observed in the classic COVID-19 and indeterminate subgroups, while moderate agreement was observed in probable COVID-19 and non-COVID groups (Table 3).
Discussion
Previous studies have suggested that fever and cough are the most common symptoms of COVID-19 and diabetes and hypertension are the most common comorbidities[2, 12, 13]. In this study, the most common symptoms were fever, cough, and shortness of breath, in this order, while the most common comorbidities were diabetes and hypertension, which agrees with previous studies.
Although chest CT findings of COVID-19 may partially overlap those of other diseases, particularly other types of viral infections, COVID-19 may have specific CT characteristics that are less commonly observed in other diseases[14]. In recent studies, the most frequently observed characteristic findings of COVID-19 include bilateral lung involvement, peripheral ground-glass appearances, and peripheral consolidation[2, 12, 13, 15]. Similarly, the most common findings in our patients were bilateral lung involvement, peripheral ground-glass opacities, peripheral consolidation, and reversed halo sign.
Radiologists and clinicians must be familiar with the imaging findings of COVID-19 pneumonia because of the increasing concerns regarding the COVID-19 pandemic. Thus, classifications can be used to create a common language and provide standardization in reporting of CT findings. Several recommendations have been published to standardize structured CT reports for suspected COVID-19 cases. Moreover, it is essential to reveal which of the classifications in the literature are more effective and consistent. In this respect, BSTI, RSNA, and CO-RADS classifications were assessed in this study. The internal consistencies of the three classifications were significant. The RSNA classification was found to have the highest consistency rate, followed by BSTI and CO-RADS.
Considering the groups that decreased the consistency rates of CO-RADS and BSTI classifications, the consistency rates of CO-RADS 3 and 4 and probable COVID-19 and non-COVID groups were lower than those in the other groups. The presence of single, few, or unilateral lesions, which are not specified in the classifications and generally included in indeterminate groups, may lead to confusion. Thus, assigning patients to a group will become a major problem. This uncertainty may cause difficulties in the diagnosis and management of the COVID-19 pandemic.
Moreover, RSNA classification described indeterminate appearance as “unilateral ground-glass opacity with or without consolidation lacking a specific distribution and are non-rounded or non-peripheral” and “few tiny ground-glass opacity with a non-rounded and non-peripheral distribution”[16]. However, in this study, 10 patients who were positive in the PCR test had unilateral, single, and/or few ground-glass appearances. These results showed that COVID-19 could present unilaterally in the form of a single lesion.
False-negative RT-PCR test results have been reported in patients diagnosed with COVID-19 based on CT findings and ultimately tested positive with serial sampling[5]. Therefore, physicians perform chest CTs for emergency diagnosis in daily practice[17]. In this study, 137 patients were diagnosed with COVID-19 despite having normal CT findings, and this suggests that chest CT scan may be also used in daily practice even if it is not recommended as a screening tool or as a first-line test to diagnose COVID-19. Thus it should be considered that COVID-19 can present with any imaging finding.
This study has several limitations. First, images were evaluated by four radiologists employed in the same center. Second, the sample size was small. The population consisted only of patients who were positive on the PCR test, so the predictive values for diagnosis were not evaluated. To the best of our knowledge, this is the first study evaluating different radiological classification systems of COVID-19.
Conclusion
The use of structured reporting templates facilitate accurate radiological diagnosis and reduce variability in chest CT interpretation. This may help reduce the workload and combat the COVID-19 pandemic effectively. Therefore, the most consistent and effective reporting system should be used. In conclusion, although the consistency rates of radiological classification systems of COVID-19 in the literature were high, they might decrease in the subcategories. Therefore, improving the items without consensus can lead to the development of better radiological diagnostic approaches.
Ethics
Ethics Committee Approval: The relevant ethical approval was taken from the Ethics Committee of Kırıkkale University (approval no.: 2020.05.07, date: 20.05.2020) and the COVID-19 Scientific Research Committee of the Ministry of Health.
Informed Consent: Retrospective study.
Peer-review: Externally peer-reviewed.
Authorship Contributions
Surgical and Medical Practices: A.Ö., S.Y., Ö.Ö.E., S.K., A.G., S.P.K.E., İ.K., B.K., S.G., Concept: A.Ö., S.Y., Ö.Ö.E., S.K., A.G., S.P.K.E., İ.K., B.K., S.G., Design: A.Ö., S.Y., Ö.Ö.E., S.K., A.G., S.P.K.E., İ.K., B.K., S.G., Data Collection or Processing: A.Ö., S.Y., Ö.Ö.E., S.K., A.G., S.P.K.E., İ.K., B.K., S.G., Analysis or Interpretation: A.Ö., S.Y., S.K., Literature Search: A.Ö., A.G., S.P.K.E., İ.K., Writing: A.Ö., İ.K.
Conflict of Interest: No conflict of interest was declared by the authors.
Financial Disclosure: The authors declared that this study received no financial support.
