WO2022129628A1

WO2022129628A1 - Computer-implemented method for image analysis of medical images by neural network supported region-of-interest segmentation

Info

Publication number: WO2022129628A1
Application number: PCT/EP2021/086681
Authority: WO
Inventors: Pavel Klastrup LISOUSKI; Eric Navarro COMES; Martin Christian AXELSEN; Mads Jarner BREVADT
Original assignee: Radiobotics Aps
Priority date: 2020-12-18
Filing date: 2021-12-18
Publication date: 2022-06-23

Abstract

The present disclosure detail a computer-implemented method for determining a region-of-interest in medical images having a first and high resolution by downscaling the image for a first segmentation and region-of-interest identification and subsequently, the region of interest upscaled for final neural network supported image segmentation to produce a segmentation output image using a single neural network for both segmentations.

Description

TITLE OF INVENTION

Computer-implemented method for image analysis of medical images by neural network supported region-of-interest segmentation .

TECHNICAL FIELD

In the field of image analysis of biomedical images , a computer-implemented method for image analysis of X-ray images by region-of-interest segmentation is proposed for use in deep neural network-supported X-ray analysis .

BACKGROUND

Automated or computer assisted analysis of medical images are computationally hard problems requiring considerable calculational power of the computer systems implementing the assisted analysis , in particular since , with the improvement in imaging technology over the last decades , the medical images often have very high resolutions ( often more than several thousands of pixels in each dimension) . Typical image resolutions for e . g . , an x-ray image range from 1000 x 1000 up to 4000 x 4000 pixels . However, depending on information content and density per pixel , file format , and data storage mode , the storage capacity needed for storing a 16-megapixel medical image on a non-transitory computer-readable storage medium will be larger, often signi ficantly larger than the image resolution, and the corresponding transitory computer- readable storage requirements during computer program execution will be considerable .

For this reason, many applications rely on resi zing or downscaling of the information content comprised in a medical image prior to subsequent data processing, usually by datapoint exclusion, i . e . , removal of datapoints from the original image , thereby lowering the information content of the image .

However, as resi zing or downscal ing images by data point or pixel exclusion to smaller si zes for analysis is often inappropriate ( since , ultimately, the high resolution can be crucial for the final analysis of the medical image by clinicians ) , no simple method of information content retention is currently available , which at the same time lowers the requirements on the computational power of a computer system performing data analysis on the medical image .

Present Applicant markets a CE-marked clinical decision tool (RBknee™) for automated identi fication of knee osteoarthritis based on a proprietary neural network supported computer-implemented decision algorithm .

A problem when using deep neural networks and convolutional neural networks for image analysis is , that such networks are computationally limited by the processing power and memory capacity of the computer systems implementing the respective neural networks making it di f ficult to train neural networks both on low and on high resolution medical images .

However, as analysis for speci fic locali zed conditions per se do not require applications using arti ficial intelligence , deep learning or convolutional neural networks to analyze the entire medical image resulting from a medical procedure , typically only speci fic regions of an image like j oints between bones for osteoarthritis analysis or fractures in specific regions are of interest, computational power can be liberated for analysis by appropriate region of interest selection prior to subsequent adaptive segmentation and analysis of the medical images in question.

For this reason, numerous approaches to adaptive segmentation of medical images have been proposed, both in the scientific literature, c.f. e.g., C. Kokkotis et al. in Osteoarthritis and Cartilage Open 2 (2020) 100069 or N. Beyramoglu et al. in J. Osteoarthritis and Cartilage, V28, 17, pp941-952, July 01, 2020, as well as in patent literature.

E.g., in US 7,539, 332 Bl to to Al-Dayeh and Bi there is detailed a system and a method for automatically identifying at least one region of interest comprising a first type of bone tissue from a target bone of interest, prior to submitting the at least one region of interest comprising the first type of bone tissue to further analysis.

In WO 2015/077813, a method is detailed for early recognition of bone and joint diseases, such as osteoarthritis, by radiographic analysis, in which a digital radiographic image of the bone in the area of the joint head and/or joint socket is taken and the fractal dimension of at least one image zone is determined, and a bone structural value is calculated on the basis of the fractal dimension of the at least one image zone and used for assessing the state of health of the joint is disclosed.

In US 20190333222, an ab-initio two-stage computer- implemented region-of-interest segmentation method is disclosed, wherein a plurality of trained neural networks provides first a coarse segmentation and second a patch-based segmentation, wherein the coarse segmentation provided by at least a first trained neural network, but usually by a plurality of di f ferently trained neural networks , provides global information about the image structure and general location of labels of interest , which stage is followed by a patch based segmentation using a second and di f ferently trained neural network compared to the first trained neural network .

A limitation of the methods disclosed in the prior art is that the full capacity for pattern recognition of trained neural networks is not exploited to its full . Rather, a certain element of operator bias is retained in the methods of the prior art , which can limit the methods employment in clinic .

For this reason and others , the present inventors propose in accordance with the present disclosure , a generali zed "region-of-interest"-based segmentation approach, which speeds up the training and inference of deep convolutional neural networks for medical image analysis , while retaining the desired high resolution of the image segmentation performed by the neural networks .

Another problem in the art is , that contextual information in medical images is being underused in relation to region of interest identi fication and the present inventors show how, training of neural networks and of the identi fication of regions of interest can be sped up using the present methods of the invention, as the contextual information can be included in the methods for eliminating time and complexity of the region of interest analysis by a neural network . In accordance with the invention, the medical images containing a predetermined skeletal bone member o f interest to be analyzed are downscaled for a first segmentation and region-of-interest identi fication using a neural network trained for identi fying said predetermined skeletal bone member of interest and subsequently, the region of interest is upscaled for a final neural network supported image segmentation using the same neural network trained for identi fying the aforementioned predetermined skeletal bone member of interest to produce a segmentation output image .

Thereby, the present methods di f fer from current ab-ini ti o or prior-knowledge independent approaches known in the art . Further elements and embodiments of the present invention are detailed herein below .

DEFINITIONS

In the context o f the present di sclosure and in accordance with contemporary usage , a pixel is the smallest content bearing image element or unit in a two-dimensional image , i . e . the smallest addressable point or picture element in any given 2D image .

As such, a 4000x4000 pixel medical image wil l have 16 megapixels of content bearing image elements , each pixel normally corresponding to an image sensor element comprised in a detection unit of a medical device for creating the D- medical image .

In the context o f the present di sclosure and in accordance with contemporary usage , a voxel is the smallest content bearing image element or unit in a three-dimensional image , i . e . , the smallest addressable point or picture element in any given 3D image .

As such, a 1000x1000x1000 voxel medical image will have 1 giga voxels of content bearing image elements , each voxel normally corresponding to an image sensor element comprised in a detection unit of a medical device for creating the 3D- medical image .

As pixel and voxel is defined and used herein, no fixed length scale is a pri ori assigned for spatial resolution between two respective either pixels or voxels .

In the context of the present invention, a medical image showing a predetermined skeleton bone member of interest is used according to common usage as meaning a viewer of the medical image will see by looking at the medical image a two- dimensional proj ection onto the medical image of the three- dimensional predetermined skeleton bone member of interest imaged on the medical image .

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 : Flow diagram for the generali zed ROI-method

Figure 2 : Method - 1^st resi zing

Figure 3 : Computation gain ratio for image analysis

Figure 4 : Final region of interest image

Figure 5 : Information loss comparison

Figure 6 : Method - 1^st segmentation

Figure 7 : Region of interest identi fication

Figure 8 : Method - 2^nd resi zing

Figure 9 : Resolution limits for bounding box approach

Figure 10 : Method - 2^nd segmentation

Figure 11 : Flow diagram sliced tomographic images Figure 12A: Example of final selection - knee anterior

Figure 12B: Example of final selection - knee lateral

Figure 12C: Example of final selection - hip joint lateral

Figure 13: Examples of segmentation masks

Figure 14: Size determination for ROI

Figure 15: Size determination for ROI

Figure 16: Alternative size determination for ROI

Figure 17 : Diagram for NN-training elements

Figure 18: Joint space width JSW-determination

Figure 19: Training model for JSW-model

Figure 20: ROI-size determination for JSW-model

It is to be understood, that the embodiments shown in the figures are for illustration of the present invention and cannot be construed as being limiting on the present invention. Unless otherwise indicated, the drawings are intended to be read (e.g., cross-hatching, arrangement of parts, proportion, degree, etc.) together with the specification, and are to be considered a portion of the entire written description of this disclosure.

DETAILED DESCRIPTION

The methods of the present invention are based on the often- overlooked fact in the present art of computer-implemented methods of medical image analysis by neural networks, that medical images are created within a clinical and medical context and therefore come with certain elements of information already contained in the context of why a given medical image was acquired and what it contains.

E.g., when a medical practitioner requests an X-ray image of a knee (or any other predetermined skeleton bone member of interest (8) ) , the radiographic department at the hospital or clinic will return with an X-ray of the patient's knee (8) to the requestor for further medical examination and diagnosis .

Depending on the resolution of the medical image (2) , the region of interest (5) showing the predetermined skeletal bone member of interest (8) may fill the entire medical image (2) (highest magnification) , but normally and for many reasons such as e.g., for compensating patient-to-patient variations in physiology or medical condition, or variations in magnification levels in the available medical equipment, the medical image (2) are created at standard magnifications which are lower than the maximum magnification.

This leads to excess information (background) being stored in the medical image (2) , which the skilled medical practitioner by training knows to visually discard in the evaluation of the medical image (2) , which however in automated medical examination e.g., in computer-assisted medical examination, must be discarded by identifying suitable regions of interest and reducing the set of data for use in the computer-assisted medical examination to only the data contained in the suitable region of interest.

A medical image (2) in clinical context accordingly a priori contains the information that it is a medical image (2) e.g., an X-ray image (2) , showing a predetermined skeleton bone member of interest (8) , e.g., a knee (8) , in a least a region of interest (5) inside the acquired medical image (2) .

The present inventors have realized that by utilizing the information contained a priori in a medical image, improvements to region-of-interest identification and reductions of computational complexity can be obtained. In accordance with the invention, the present computer- implemented method (1) for image analysis of a medical image (2) containing at least a region of interest (5) showing a predetermined skeletal bone member of interest (8) by neural network supported region-of-interest segmentation comprises: i. performing a process (10) of downscaling of the input medical image (2) containing at least the region of interest (5) showing the predetermined skeletal bone member of interest (8) from a first resolution to a second resolution smaller than the first resolution for obtaining a downscaled medical image (3) ; ii. performing a first process (20) of neural network supported image segmentation (4) using a neural network trained for identifying said predetermined skeletal bone member of interest (8) on the downscaled medical image (3) for identifying (30) the region of interest (5) showing the predetermined skeletal bone member of interest ( 8 ) ; iii. performing a process (40) of rescaling of the downscaled medical image (3) on the identified region of interest

(5) showing the predetermined skeleton bone member of interest (8) from the second resolution to a final resolution smaller than or equal to the first resolution for obtaining a rescaled medical image (6) consisting of the identified region of interest (5) ; iv. performing a second process (50) of neural network supported image segmentation using said neural network trained for identifying the predetermined skeletal bone member of interest (8) on the rescaled medical image

(6) for obtaining a segmentation output image (7) at the final resolution. In the present methods, there is accordingly made use, contrary to the ab initio approaches in the prior art, of the a priori fact that the medical images (2) obtained from the clinic all contain at least one region of interest (5) showing a predetermined skeletal bone member of interest (8) . This fact is exploited to simplify the analysis for the region of interest (5) in the medical image (2) as the analysis for the region of interest now can be performed using the same neural network at all resolutions of interest, and that therefore the neural network only has to be trained to identify the predetermined skeleton bone member of interest (8) but not any other objects in the medical image (2) for a correct region of interest determination.

Consequently, the first region of interest identification or segmentation at step ii. can be performed even at the resolution limit of the neural network as trained, significantly increasing the pixel reduction ratio possible to use while still obtaining a clinically meaningful result, thereby at the same time increasing the segmentation speed and reducing the requirements on computational power for performing the segmentation. In Figure 5D is shown how the present neural network of the inventors is still fully suitable for identifying the predetermined skeleton bone member of interest (8) , here a knee, at only 5% of the original information content held in the X-ray showing the knee .

It is possible to use a higher pixel reduction ratio, i.e., removing more data from the initial image, however a balance must be struck between the benefits of reducing the information content of the downscaled medical image (3) versus any safety margins added to the determined region of interest (5) (see below) for compensating for influences on the si ze of the region of interest ( 5 ) originating with bone member determination arti facts in the downscaled image ( 3 ) . Practically, the present inventors have found that reducing the image si ze of the medical image ( 2 ) below 5% of the original si ze does not provide any reasonable benefits to computational time in light of present-day computational systems and computing power .

Accordingly, in embodiments of the present invention, the second resolution can be e . g . , 25% or lower of the first resolution, 20% or lower of the first resolution, preferably 15% or lower of the first resolution, 10% or lower of the first resolution, or more preferably 5% or lower of the first resolution . In further embodiments , the second resolution is 25% of the first resolution, 20% of the first resolution, preferably 15% of the first resolution, 10% of the first resolution, or more preferably 5% of the first resolution .

In terms of product economy having to train only a single neural network signi ficantly reduces development costs , but also signi ficantly increases the resolution limit of the neural network, as only one neural network has to be trained, allowing also less accurate sets of training elements to be included and adj usted for during the training of the neural network, additionally obtaining an increased elimination of errors originating with the training and adj usting of a further neural network .

In an embodiment of the computer-implemented method ( 1 ) for image analysis of a medical image ( 2 ) by neural network supported region-of-interest segmentation, a resolution is a resolution in pixels or in voxels . In an embodiment of the computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region-of-interest segmentation, the final resolution is equal to the first resolution. This will normally be the case in the embodiments of the invention, since for the clinician a maximum information is provided at full resolution of the resulting region of interest. However, in some embodiments a lower resolution may be suitable and can be included into the resulting image, e.g., by choosing a suitable lower final resolution at initiation of the present methods.

In an embodiment of the computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region-of-interest segmentation, the process (10) of downscaling comprises a linear compression of at least one axis of the input medical image (2) . Other methods of downscaling are known, such as e.g., cubic interpolation or dark pixel cropping, but the present invention has been illustrated using the preferred present embodiment of linear compression .

In an embodiment of the computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region-of-interest segmentation, the region of interest (5) showing the predetermined skeletal bone member of interest (8) resulting from the first process (20) of neural network supported image segmentation is a bounding box ( 5 ) .

In an embodiment of the computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region-of-interest segmentation, the obtained segmentation output image (7) is submitted to a subsequent process of neural network supported bone condition identification for identifying a predetermined bone condition. It is a clear benefit of the present invention that since the region of interest identification is performed at high accuracy, the accuracy of further computer implemented neural network supported analysis are increased, while reducing computational time and effort.

As will be detailed below, in an embodiment of the computer- implemented method (1) for image analysis of a medical image (2) by neural network supported region-of-interest segmentation, the method (1) further comprises a step of generating a training set for a neural network for arranging the neural network suitable for performing a region-of- interest segmentation on a medical image (2) for identifying a region of interest (5) showing a predetermined skeletal bone member of interest (8) .

In an embodiment of the computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region-of-interest segmentation, the method (1) further comprises training at least one classifier of the neural network on the training set for obtaining the aforementioned neural network suitable for performing a region-of-interest segmentation on a medical image (2) for identifying a region of interest (5) showing the predetermined skeletal bone member of interest (8) .

In an embodiment of the computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region-of-interest segmentation, the input medical image (2) showing said predetermined skeletal bone member of interest (8) is an X-ray image. in an embodiment of the computer-implemented method ( 1 ) for image analysis of a medical image ( 2 ) by neural network supported region-of-interest segmentation disclosed herein the output of the first segmentation is a multi-class segmentation mask each segmentation mask containing information of a predetermined bone forming part of the predetermined skeleton bone member of interest ( 8 ) .

In an embodiment of the computer-implemented method ( 1 ) for image analysis of a medical image ( 2 ) by neural network supported region-of-interest segmentation disclosed herein, the said predetermined skeleton bone member of interest ( 8 ) is a knee .

In an aspect of the invention there is detailed, a non- transitory computer-readable storage medium is detailed, comprising a computer program stored thereon for executing by a computer system a computer-implemented method ( 1 ) according to any of the embodiments detailed herein .

In an aspect of the invention there is detailed, a computer system comprising a non-transitory computer-readable storage medium according to the embodiments detailed herein, for executing a computer program stored on the computer readable storage medium, wherein the computer program comprises instructions for executing a computer-implemented method ( 1 ) according to any of the embodiments detailed herein .

In an aspect of the invention there is detailed a medical imaging processing system comprising a computer system according to any aspect or embodiment disclosed herein .

In general , it i s considered that the skilled person within the field of computer-implemented methods of neural network supported image analysis is capable of compiling the computer hardware and software environment for operating the computer hardware and this aspect of the present invention has not been further illustrated beyond the present embodiments for making the computer system suitable for implementing the methods of the invention on said computer system. In a preferred embodiment of the computer system, the computer system is a stand-alone system with a dedicated graphics card .

RO -based Method Step by Step

The region-of-interest based computer-implemented method (1) of the invention is illustrated below (c.f. Figure 1) and will be explained in further details in the next sections. In Figure 1, showing a flow diagram for the method (1) of the invention, the method (1) of the invention is illustrated using a 1000x2000 pixel X-ray input medical image (2) .

In accordance with the present computer-implemented method

(1) an:

1) Input medical image consists of a medical 2D image which could be an x-ray with a high resolution (here 1000x2000 pixels ) .

2) The input image is downscaled to a smaller image (512x512 pixels) .

3) A first segmentation is performed on the smaller image.

4) A specific region of interest (ROI) is selected based on the segmentation result of the first segmentation.

5) The smaller image is upscaled in the relevant region of interest to the original resolution.

6) A second segmentation of the region of interest (ROI) is performed at the original resolution 7) High resolution segmentation output is created based on the segmentation of the ROI, preferably a high resolution segmentation output suitable for further image analysis.

Resize Input Image

In accordance with the invention, in a first step (i) of the computer-implemented method (1) there is performed a process (10) of downscaling of an input medical image (2) from a first resolution to a second resolution smaller than the first resolution for obtaining a downscaled medical image (3) , c.f. Figures 1 and 2.

The resizing of the image is a down-sampling of the original input image with dimensions (xfull * yfull) to a smaller size/resolution (xres * yres) e.g., resized from original size of 1000x2000 to 512x512. The size of the resized image can be adjusted according to a specific application and the processing capability available.

The resize ratio for width and height can be defined as r_w = ^wfuii/^wres and r_h = w_full/w_res, respectively, c.f. Figure 3.

In one embodiment of the present invention, downscaling is by pixel exclusion. Other methods of downscaling are known, such as e.g., cubic interpolation or dark pixel cropping, but the present invention has been illustrated using the preferred present embodiment of linear compression.

In an embodiment thereof, the pixel exclusion is by linear compression, wherein pixels to be excluded are evenly spaced along an axis of the medical image with a linear resize ratio as given above for a respective axis. Original vs. ROI size

The pixel ratio, which is the ratio between the number of pixels in the original image (imgf) and ROI image (imgr) can be calculated as follows:

The Pixel Ratio increases exponentially as the object, or the ROI size decreases (c.f. Figure 3) . While the size of the original medical image can vary from detector unit to detector unit, in the example, the original size is fixed to be 4000x4000 pixels for illustration of how the computational complexity will decrease exponentially as the size of the object and thereby the size of region of interest decreases as shown in Figure 3.

The computer-implemented method is therefore especially applicable when the object of interest is significantly smaller than the size of the image itself as this will give rise to the largest exponential reduction in computational complexity .

As an example (c.f. Figure 4) , a knee x-ray image is shown with the dimensions (2048x1265 pixels) . The actual region of interest (5) is where the knee joint is which is in this case can be meaningfully comprised in a region of interest (5) with dimensions around 450x450 pixels. This results in a pixel ratio of 12.7. between the two images. Therefore, instead of performing the second segmentation (30) on the entire image, a full second segmentation can be applied on one twelfth of the number of pixels for the final segmentation of the image prior to further analysis.

Limitations of the size of the full segmentation

The limit of the resizing factor of the full-scale image in the initial step is dependent on the size of the object in the image. Examples of different resizing factors are shown in the images comprised in Figure 5, where the previous image with dimensions (2048x1265) is used. The bones tibia and femur are clearly defined and visible when resizing the image to 20% of the full size as well as 10% of the size, however, when the image is resized to less than 10%, it is very hard to distinguish the two bones from each other.

However, as shown, the neural network is still capable of correctly identifying the two respective bones even at a reduction to only 5% of the original size, such that a correct region of interest (5) can be determined. In accordance with Formula (1) , a size reduction to only 5% of the original corresponds to a factor 400 reduction in computational complexity for performing the first segmentation (20) yet retaining the ability to correctly identify (30) a meaningful region of interest (5) for the second segmentation (40) .

Segmentation of Resized Image

In accordance with the invention, in a second step (ii) of the computer-implemented method (1) there is performed a process (20) of neural network supported image segmentation on the downscaled medical image (3) for identifying (30) a region of interest (5) comprising information on a predetermined skeletal bone member of interest (8) , c.f. Figure 6. The segmentation of the downscaled image (3) employs a deep learning and/or convolution neural network model, which takes an image as input and returns segmentations of objects in the image as illustrated e.g., in Figure 6. Since the downscaled image (3) is much smaller than the original image (2) , the second resolution of the segmentation image (4) is low, i.e., below 30% of the first resolution (c.f. Figure 5) and depends on the size of the downscaled image (3) . Although the second resolution is "low", it can be used to specify a region of interest (5) depending on the location of the segmented objects.

Selection of Region of Interest (ROI)

After the first process (20) of neural network supported image segmentation (4) on the downscaled medical image (3) , objects of interest (8) , such as a predetermined bone member of interest (8) , in the downscaled image (3) can be identified (30) and a specific "region of interest" (ROI) (5) can be identified based on the location of the specified object of interest (8) .

In Figure 7, the region of interest (5) is defined by the joint between the two bones (femur and tibia) . In the example, the region of interest (5) is thereby predetermined (in the example based on the location and the rotation of the specified object segmented) .

In an embodiment of the invention, when the region of interest (5) is identified (30) (c.f. Figure 8) in the downscaled and segmented image (4) , the corresponding ROI coordinates in the original full-size input medical image (2) can be found based on the resize ratio between the original (2) and the downscaled medical image (3) . A cropped "ROI" image with a high resolution can be processed further.

Precision and limit of ROI coordinates

However, the precision of the ROI coordinates is dependent on how much the original image is resized in the first segmentation step (c.f. Figure 9) . In the figure is illustrated how the ROI precision will change with three different resolution sizes of the original image (in decreasing order of resolution) . The figure illustrates how the optimal (middle box - labels 5a-c' ' ) region of interest (5) size for each level (A-C) of resolution becomes consecutively larger as the resolution decreases. This in consequence will lead to the final region of interest (5) being larger than necessary if the resolution applied during the first segmentation process (20) becomes too low.

Accordingly, a balance exists between the gain in computational time by having a data size of the downscaled image (3) low for faster first segmentation (20) and the increase in computational time by the resulting increased data size for the rescaled medical image (6) at the final resolution consisting of the established region of interest (5) .

From Figures 3, 5 and 9 it can be understood that an optimal resolution for the second resolution for providing a reasonable balance between the abovementioned conflicting requirements in some embodiments is from 10% to 30% of the first resolution, preferably from 15% to 25% of the first resolution, more preferably is 20% of the first resolution. Segmentation of the Region of Interest

In accordance with the invention, in a fourth step (iv) of the computer-implemented method (1) there is performed a second process (50) of neural network supported image segmentation on the rescaled medical image (6) for obtaining a segmentation output image (7) at the final resolution (c.f. Figure 10)

The ROI image cropped from the full-size image has the original resolution, but the size of the image is much smaller than the original full-size image. It is possible to apply deep learning / convolutional neural networks on the image to segment the objects in a much higher resolution in that specific region as illustrated below.

The final outcome is a specific image region, which has a high-resolution segmentation.

Segmentation masks

In relation to the present invention, a predetermined skeleton bone member (8) may comprise more than one skeleton bone. E.g., the present invention is illustrated also with respect to a knee. A knee, however, although it is a skeleton bone member, is constituted from several bones, including femur and tibia, wherein each bone carry part of the skeletal features, that together forms the knee.

In such cases the resulting segmentations in the methods of the present invention will result in a multi-class segmentation mask, wherein each respective segmentation mask corresponds to a respective skeletal bone comprised in the predetermined skeletal bone member (8) . In relation to the present invention as illustrated for a knee in the examples presented herein, not all bones necessary for forming a knee are included into the neural network supported segmentation, since for a knee , the femur and the tibia are diagnostically determining, hence it is suf ficient to provide the segmentation mas ks for respectively femur and tibia for identi fication of a knee in a medical image ( 2 ) , simpli fying analysis as detailed herein .

Segmentation of a resized original image

The first step in the present method is segmentation of the original input image ( 2 ) , which is resi zed down to a speci fic resolution . In most embodiments of the present invention, the standard resolution is set to 512x512 pixel s , but as discussed herein, the aforementioned second resolution may also be dynamically determined, rather than a constant si ze . However, having the aforementioned second resolution be of a constant si ze simpli fies programming, but at the cost of potential over- or undersampling of data .

The output of the original segmentation model after step ii . may in some embodiments of the invention be a multi-class segmentation mask, preferably being a multi-class segmentation mask having a constant si ze , such as a si ze of 512x512 pixels . In Figure 13 , the segmentation mask for the femur and the segmentation mask for the tibia are shown within the constant si zed 512x512 pixels resulting medical image .

Standardized ROI selection based on a priori knowledge

Based on the segmentation mask and the a -pri ori knowledge of the must be presence of the predetermined bone member ( 8 ) in these speci fic radiographs , it is possible in accordance with embodiments of the invention to speci fy very speci fic regions of interest ( 5 ) to be segmented in step iv .

Here , a region of interest ( 5 ) is speci fied that will extract the knee j oint , which will be analyzed further to detect tibiofemoral osteoarthritis in accordance with further embodiments of the invention .

It is preferably in the embodiments of the invention to employ a dedicated region-of- interest detector (ROI- detector ) subroutine for using the resulting multi-class segmentation masks for the to find the minimum and maximum value of the x-coordinates of the contained segmentation masks ( femur and tibia ) of the example . The coordinates are speci fied as :

• ^xfmin and xf_maxas the minimum and maximum values of the lower part of the femur bone , respectively .

• x min and xt_maxas the minimum and maximum values of the lower part of the femur bone , respectively .

• The minimum x-value can be calculated as x_s =

• x-value can be calculated as x_e =

• The width, ROI_W, can be calculated as ROI_W = x_e — x_s

• The center x-coordinate can be calculated as : x_c = (x_e + x_s)/2

A y-coordinate that determines the mean y-coordinate ( y_c) between the two segmentation masks can then be determined by the maximum y-coordinate of the femur mask and minimum y- coordinate of the tibia mask as illustrated in Figures 14 and 15. Based on the center coordinates x_c and y_c, the region of interest (5) for the knee joint can be determined. A margin scale is added in the x- and y-direction, which defines the ROI width and height (the ROI is quadratic if the margin is the same in both directions) :

• Upper left corner:

• Lower right corner

In that way, it is possible to automatically extract a standardized ROI of the knee joint based on the segmentation.

Final ROI is selected based on the detected ROI and a margin so that extra area is cropped around the bone as shown in the figures below (c.f., Figure 14 and Figure 15) .

In Figure 16 a simpler approach according to some embodiments of the invention, to the region of interest extraction is illustrated; wherein minimum and maximum x-values are calculated based on the two separate segmentation masks. The mean y-value is calculated and based on this information, and a standardized region of interest (5) can then be calculated around the knee joint.

Method / Algorithm Limits of the computer-implemented method

The limits of this approach depend highly on the following variables :

• Size of the input image (hfull, wfull)

• Size of the ROI object (hres, wres) to be determined compared to the full image size

• Needed precision of the ROI coordinates Depending on the application, the size of the original image can change, and the size of the ROI can change. The computer- implemented method is most beneficial when there is a high ratio between the original medical image (2) and the rescaled ROI image ( 6 ) .

Training Process and Method

Full bone segmentation

The present invention is illustrated with respect to a region of interest identification for the knee as the predetermined skeleton bone member (8) of the invention.

In accordance with the above comments on the knee in relation to femur and tibia it is sufficient for a full bone segmentation, that the neural network is able to detect the femur and tibia bone masks.

The algorithm is implemented using a U-net architecture for semantic segmentation. For feature extraction, different backbones are used, ranging from MobileNet, to SeResNet and ResNets. The loss function used is the dice loss (c.f., Figure 17 ) .

The network input is a grayscale image, and the output is a three-class segmentation map (tibia, femur and background) . The original network is described in further detail in the article: "U-Net: Convolutional Networks for Biomedical Image Segmentation" - https://arxiv.org/abs/1505.04597 Training of the full segmentation

This section will briefly describe how the segmentation network has been trained .

The segmentation model is trained on 1746 PA knee images . On each training image the femur and tibia have been manually segmented to establish a mask used as the target label for training the network .

The network is trained by performing random search in a set of defined hyperparameters , such as loss functions ( dice , j accard, categorical cross entropy) , optimi zers ( adam, RMSprop ) , batch si zes , etc . Training is stopped under the conditions of early stopping, and model selection is performed by looking at the dice score in the validation set . The training and validation data is sampled di f ferently for each experiment run .

Real time augmentations are applied to the input images during training (not validation) by using the augmentations library . To apply those augmentations , a DataGenerator class was defined . Some of the augmentations covered are :

• Dropout - drop a certain number of pixels ( salt pepper ) .

• Sharpen

• Rotation and shearing

• Hori zontal flip

• Small crop

• Gaussian blur / Average blur / Median blur

• Gaussian noise

• Brightness change

• Contrast change Training of the ROI segmentation

The same architecture and code were used for segmenting the ROI bones of tibia and femur . The segmentation was trained on automatically extracted region of interests . In that way, the j oint space region of the bones is segmented with much higher resolution . The knee-ROI segmentation can be applied on the extract ROI from the ROI-detector .

Training of the ROI segmentation is done completely in the same manner as the full bone segmentation .

The ROI segmentation is done on the same 1746 PA knee images as above , with the same manual segmentations . With the exception that the tibia and femur are now detected as two separate classes , allowing for unique identi fication of each bone .

Knee joint space width ROI Segmentation based on prior knowledge

The method described above is used to determine a fast and precise segmentation of a standardi zed region of interest in a radiograph . This method can be used in several applications and in several iterations .

A primary finding of osteoarthriti s is j oint space narrowing which can be measured as j oint space width ( JSW) between the femur and tibia bones in a PA knee radiograph . Measurement of the j oint space width can be automated using neural network-based systems by segmenting the femur and tibia bones in the knee radiograph and measuring the width between the two bones in very speci fic and standardi zed regions (medial and lateral compartment of the tibiofemoral j oint ) as illustrated in Figure 18 .

The precise segmentation of the two bones can then be obtained with the same methodology as described above as illustrated in Figure 19 .

The standardi zed selection of the region of interest can again be incorporated in the pipeline using the prior knowledge of where the useful information needs to be extracted from (medial and lateral condyles in the tibiofemoral knee j oint ) .

The standardi zed extraction is described below (here , for the lateral part of the j oint - the methodology is the same for the medial ) :

• The masks , femur and tibia, generated in the first iteration are used for the calculation of the standardi zed ROT

• First , the width of the femur mask is calculated by finding the start (xf_s ) and end xf_e ) point (x- coordinate ) of the femur mask . This can be used to calculate the width, w, of the femur mask which is used as the standardi zation reference

• The center y-coordinate of the region of interest is detected by calculating the mean of the maximum y- coordinate of the femur mask (yf_max') and the minimum y- coordinarte of the tibia mask (yt_min) ; y_c = yfmax + yt_min)/2

A region of interest can now be created; centered around the y_c coordinate in the y-direction and with the xf_s coordinate as the reference for the x-direction . The height and width of the region of interest can then be calculated relative ( standardi zed) to the femur width to a final result as shown in Figure 20 .

Training of the joint space width ROI segmentation

The JSW measurements are based on segmentation of the medial and lateral compartments of the knee as shown in the image below . The ROI segmentation i s used to determine the compartment patch, which is again segmented to gain a betterquality segmentation .

The segmentation model is segmenting both the medial and lateral patches - only the medial area is visuali zed in Figure 20 .

The ROI segmentation used for measurement of the JSW is then refined ( area replaced) by the compartment segmentation .

Training of the compartment segmentation is done completely in the same manner as the ROI bone segmentation .

The training is done on the same 1746 PA knee images , with the same manual segmentations .

Applications

Different 2D applications

It is an advantage of the present computer-implemented method ( 1 ) that it is easily adaptable to 3D-imaging and generally within 2D x-ray imaging - especially when a region of interest ( 5 ) is known and well-defined . A few examples are shown below : 3D applications

The same computer-implemented method can be applied to 3D applications such as MRI or CT, where regions of interests are found per slice as shown in Figure 11, wherein the method (1) of the invention is repeatedly applied for each slice to identify a 3D-dimensional body part in the overlay of slices.

Other Use Cases

Region of Interest Overlay Output

A general overlay structure / outputs in present Applicant's products is a Region of Interest (ROI) of the analyzed area(s) as illustrated below. This automatic "zoom" and combination of images into one overlay makes it easier for the doctor to review the overlays. The "region of Interest based" computer-implemented method described above can be used to identify specific regions which are used in the output of a system.

In Figure 12A-C, it is illustrated how the regions of interest can be used with present Applicant's RBknee™ product, after the knee regions have been identified using the "region of Interest" based computer-implemented method, analyzed by the system and then outputted with overlays as a secondary capture. Here are depicted 3 examples of bone conditions subsequently identified on medical images (7) consisting of identified regions of interest (5) according to the present method (1) . Automatic placement of information to avoid occlusion based on RO I segmentation

A need from doctors is that the legends / information does not occlude the bones in the overlays . A solution is to analyze automatically where a "non-bone" area exists and place the legends in that region automatically . The "region of interest-based" computer-implemented method of the invention automatically makes it possible to identi fy and segment the bone obj ects in the image , and thereby place information in the "non-bone" area outside the identi fied region of interest ( 5 ) in the original medical image ( 2 ) .

CLOS ING COMMENTS

Although the present invention has been described in detail for purpose of illustration, it is understood that such detail is solely for that purpose , and variations can be made therein by those skilled in the art in practicing the claimed subj ect matter, from a study of the drawings , the disclosure , and the appended claims .

The term " comprising" as used in the claims does not exclude other elements or steps . The indefinite article "a" or "an" as used in the claims does not exclude a plurality . A single processor or other unit may ful fill the functions of several means recited in the claims . A reference sign used in a claim shall not be construed as limiting the scope .

Claims

1. A computer-implemented method (1) for image analysis of a medical image (2) containing at least a region of interest (5) showing a predetermined skeletal bone member of interest (8) by neural network supported region-of- interest segmentation, the method (1) comprising: i. performing a process (10) of downscaling of said input medical image

(2) containing at least the region of interest (5) showing the predetermined skeletal bone member of interest (8) from a first resolution to a second resolution smaller than said first resolution for obtaining a downscaled medical image

( 3 ) ; ii. performing a first process (20) of neural network supported image segmentation

(4) using a neural network trained for identifying said predetermined skeletal bone member of interest (8) on said downscaled medical image (3) for identifying (30) said region of interest (5) showing said predetermined skeletal bone member of interest (8) ; iii. performing a process (40) of rescaling of said downscaled medical image (3) on said identified region of interest (5) showing said predetermined skeleton bone member of interest (8) from said second resolution to a final resolution smaller than or equal to said first resolution for obtaining a rescaled medical image (6) consisting of said identified region of interest

(5) ; iv. performing a second process (50) of neural network supported image segmentation using said neural network trained for identifying said predetermined skeletal bone member of interest (8) on said rescaled medical image

(6) for obtaining a segmentation output image (7) at said final resolution. The computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region- of-interest segmentation according to claim 1, wherein a resolution is a resolution in pixels or in voxels. The computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region- of-interest segmentation, according to either claim 1 or claim 2, wherein said final resolution is equal to said first resolution. The computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region- of-interest segmentation according to any of the preceding claims, wherein said process (10) of downscaling comprises a linear compression of at least one axis of said input medical image (2) . The computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region- of-interest segmentation according to any of the preceding claims, wherein said region of interest (5) showing said predetermined skeletal bone member of interest (8) resulting from said first process (20) of said neural network supported image segmentation is a bounding box (5) . The computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region- of-interest segmentation according to any of the preceding claims, wherein said obtained segmentation output image (7) is submitted to a subsequent process of neural network supported bone condition identification for identifying a predetermined bone condition.

7. The computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region- of-interest segmentation according to any of the preceding claims, further comprising a step of generating a training set for a neural network for arranging said neural network for performing a region-of-interest segmentation on a medical image (2) for identifying a region of interest (5) showing a predetermined skeletal bone member of interest ( 8 ) .

8. The computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region- of-interest segmentation according to claim 7, further comprising training at least one classifier of said neural network on said training set for obtaining said neural network for performing a region-of-interest segmentation on a medical image (2) for identifying a region of interest (5) showing said predetermined skeletal bone member of interest (8) .

9. The computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region- of-interest segmentation according to any of the preceding claims, wherein said input medical image (2) containing said region of interest (5) showing said predetermined skeletal bone member of interest (8) is an X-ray image.

10. The computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region-of-interest segmentation according to any of the preceding claims, wherein the output of the first segmentation is a multi-class segmentation mask each segmentation mask containing information of a predetermined bone forming part of the predetermined skeleton bone member of interest (8) . The computer-implemented method (1) for image analysis of a medical image (2) by neural network supported region-of-interest segmentation according to any of the preceding claims, wherein said predetermined skeleton bone member of interest (8) is a knee. A non-transitory computer-readable storage medium comprising a computer program stored thereon for executing by a computer system a computer-implemented method (1) for image analysis of a medical image (2) containing said region of interest (5) showing said predetermined skeletal bone member of interest (8) by neural network supported region-of-interest segmentation according to any of the claims 1 to 11.