Purpose
To develop a robust algorithm for classification of musculoskeletal x-ray images in the most common projections of predefined body regions, and to evaluate the performance of the algorithm. Classifying radiographs is important for applying region-specific algorithms. As there is currently no large data set with x-ray projections, one had to be created first. Our goal was also to gain insight into the decision of the network.
Methods
Musculoskeletal radiographs from our department from 2018-2019 were classified into predefined body parts and projections. The dataset consisted of anonymized studies from adult patients of all genders with various pathologies from which the most common 15 body parts and the most common 30 radiographic projections were selected. 14100 images were annotated by a medical student, followed by a junior and a senior radiologist. Data leakage was excluded by separating the data groups on a patient level. First a group of patients containing 500 images were randomly selected and separated for the network testing. The remaining patients were split 70:30 resulting in 9492 images for the training dataset and 4108 for the validation dataset. Data annotation was performed using our scientific medical imaging platform Nora. As a Deep Learning framework Tensorflow 2.4.0 with the included Keras framework was used. For robust results an established convolutional neural network, Inception-v3 by C. Szegedy et al., was selected. As Inception-v3 was optimized for 1000 classes the network top layer was removed and a new network top was added. It consists of a pooling layer, a dropout layer for preventing fast overfitting and a dense layer for selecting the output of the 30 classes. As X-ray images differ from Imagenet images we decided to leave the whole network fully retrainable and used the Imagenet weights as an optimized starting point for the base network. The input images were optimized as two color-channels would not be used by a grayscale image. These were filled using the inverted image and an edge-optimized image. The network input size was set to 256*256 pixels and the X-ray images were accordingly resized. On each training and validation step the corresponding datasets were randomized, homogeneously balanced, and mildly augmented. For augmentation a possible vertical flip, a mild random cropping and a small degree of random rotation and contrast changes were applied. The amount of training epochs was set to 200, using a batch size of 50 with 100 steps of batching per epoch. The initial learning rate was set to 0.1 and decayed steadily during training to 0.05. Training was performed with a server graphics unit (Nvidia Tesla K80). Scikit-learn was used to calculate the data statistics. For visualization the software package to tf-explain was used, especially the implementation of Gradient-weighted Class Activation Mapping (Grad-CAM). It uses the gradients of the final convolutional layer to produce a heat map of the important regions in the image.
Results
The CNN training took 2:32h. Image processing of all 500 test-images took 31 seconds. The overall accuracy of the separate test sample was 97,6%. The f1-score of each class ranged from 0.67 to 1,00. Rare body projections were in the lower range, e.g. the AP view of the hip with 0,67 and the largest class of the AP view and the lateral view of the knee achieved an excellent result each of 1,00 and 0,97. However, classes with unique anatomical appearance could show good results even with a reduced number of cases. Noticeable errors were shown between the groups of the oblique view of the forefoot to the oblique view of the foot, or the AP view of the hip to the Lauenstein view of the hip where the overlap in clinical routine is often also present. The results for all classes are visualized in fig. 1 using a normalized confusion matrix.
Discussion
Even though the number of images used corresponds to only a medium-sized dataset, very good results were achieved for all classes. The mechanisms of the modified network top and the augmentation prevented a relevant overfitting and in the test dataset excellent results could be shown. An explanation and promising result for the very good results could be shown by the visualization using GradCam, the network showed a focusing on the persistent anatomical structures in image projections. Only in a few cases no explanation for the misclassification could be surely determined.
The number of image examples per class was partly reflected in the accuracy. This can probably be compensated by an increase of the rare classes by extending the dataset. The developed algorithm has so far only been tested on an internal data set of a single clinic in a retrospective classification. Furthermore, the data set has been limited to musculoskeletal radiographs and adult patients. In order to evaluate a potential real-life implementation of the developed algorithm, the performance of the prospective classification of external radiographs would still need to be assessed.
Conclusions
The algorithm demonstrated an excellent classification rate of MSK radiographs in the most common projections. Classification of radiographs into predefined body regions and projection using the presented approach will enable an automated use of AI-based algorithms with more specialized region-specific networks in clinical workflow.

**Purpose**
To develop a robust algorithm for classification of musculoskeletal x-ray images in the most common projections of predefined body regions, and to evaluate the performance of the algorithm. Classifying radiographs is important for applying region-specific algorithms. As there is currently no large data set with x-ray projections, one had to be created first. Our goal was also to gain insight into the decision of the network.
**Methods**
Musculoskeletal radiographs from our department from 2018-2019 were classified into predefined body parts and projections. The dataset consisted of anonymized studies from adult patients of all genders with various pathologies from which the most common 15 body parts and the most common 30 radiographic projections were selected. 14100 images were annotated by a medical student, followed by a junior and a senior radiologist. Data leakage was excluded by separating the data groups on a patient level. First a group of patients containing 500 images were randomly selected and separated for the network testing. The remaining patients were split 70:30 resulting in 9492 images for the training dataset and 4108 for the validation dataset. Data annotation was performed using our scientific medical imaging platform Nora. As a Deep Learning framework Tensorflow 2.4.0 with the included Keras framework was used. For robust results an established convolutional neural network, Inception-v3 by C. Szegedy et al., was selected. As Inception-v3 was optimized for 1000 classes the network top layer was removed and a new network top was added. It consists of a pooling layer, a dropout layer for preventing fast overfitting and a dense layer for selecting the output of the 30 classes. As X-ray images differ from Imagenet images we decided to leave the whole network fully retrainable and used the Imagenet weights as an optimized starting point for the base network. The input images were optimized as two color-channels would not be used by a grayscale image. These were filled using the inverted image and an edge-optimized image. The network input size was set to 256*256 pixels and the X-ray images were accordingly resized. On each training and validation step the corresponding datasets were randomized, homogeneously balanced, and mildly augmented. For augmentation a possible vertical flip, a mild random cropping and a small degree of random rotation and contrast changes were applied. The amount of training epochs was set to 200, using a batch size of 50 with 100 steps of batching per epoch. The initial learning rate was set to 0.1 and decayed steadily during training to 0.05. Training was performed with a server graphics unit (Nvidia Tesla K80). Scikit-learn was used to calculate the data statistics. For visualization the software package to tf-explain was used, especially the implementation of Gradient-weighted Class Activation Mapping (Grad-CAM). It uses the gradients of the final convolutional layer to produce a heat map of the important regions in the image.
**Results**
The CNN training took 2:32h. Image processing of all 500 test-images took 31 seconds. The overall accuracy of the separate test sample was 97,6%. The f1-score of each class ranged from 0.67 to 1,00. Rare body projections were in the lower range, e.g. the AP view of the hip with 0,67 and the largest class of the AP view and the lateral view of the knee achieved an excellent result each of 1,00 and 0,97. However, classes with unique anatomical appearance could show good results even with a reduced number of cases. Noticeable errors were shown between the groups of the oblique view of the forefoot to the oblique view of the foot, or the AP view of the hip to the Lauenstein view of the hip where the overlap in clinical routine is often also present. The results for all classes are visualized in fig. 1 using a normalized confusion matrix.
**Discussion**
Even though the number of images used corresponds to only a medium-sized dataset, very good results were achieved for all classes. The mechanisms of the modified network top and the augmentation prevented a relevant overfitting and in the test dataset excellent results could be shown. An explanation and promising result for the very good results could be shown by the visualization using GradCam, the network showed a focusing on the persistent anatomical structures in image projections. Only in a few cases no explanation for the misclassification could be surely determined.
The number of image examples per class was partly reflected in the accuracy. This can probably be compensated by an increase of the rare classes by extending the dataset. The developed algorithm has so far only been tested on an internal data set of a single clinic in a retrospective classification. Furthermore, the data set has been limited to musculoskeletal radiographs and adult patients. In order to evaluate a potential real-life implementation of the developed algorithm, the performance of the prospective classification of external radiographs would still need to be assessed.
**Conclusions**
The algorithm demonstrated an excellent classification rate of MSK radiographs in the most common projections. Classification of radiographs into predefined body regions and projection using the presented approach will enable an automated use of AI-based algorithms with more specialized region-specific networks in clinical workflow.

Deep learning for classification of musculoskeletal x-ray images.

[Networking hour](https://app.wonder.me/?spaceId=fe093845-759f-448f-9f55-01d3adccd820)
[COIs](https://drive.google.com/drive/folders/1nbfwAa9lf9Kz-wAZ7iLKhfpBFjpN_gkn?usp=sharing)