AU2021236468A1

AU2021236468A1 - Image Analysis Method and System

Info

Publication number: AU2021236468A1
Application number: AU2021236468A
Authority: AU
Inventors: Yu Peng
Original assignee: Curvebeam AI Ltd
Current assignee: Curvebeam AI Ltd
Priority date: 2019-06-21
Filing date: 2021-09-21
Publication date: 2021-10-21
Also published as: AU2019204376B1; AU2020230279A1; AU2023270223A1

Abstract

: The invention provides an image analysis method and system, the method comprising: quantifying one or more features segmented and identified from a medical image of or including bone of a subject; extracting non-image or video data pertaining to the subject and pertinent to musculoskeletal disease, decay, loss or deterioration from one or more sources of non-image or video data; extracting clinically relevant features from the non image orvideo data pertainingto the subject; assessing bone fragility or decay, comprising assessing the quantified features segmented from the medical image and the features extracted from the non-image or video data with a trained machine learning model; and outputting one or more results of the assessing of the bone fragility or decay. FIG.1 1/8 10 Image analysis controller 12 Processor 18 Memory 20 22---- Image data Program 38 processor code 24 Segmenter Image data 40 & identifier Feature Non-image data 42 25'-'quantifier Training data 44 26,- Non-image Machine learning 46 data processor models 28 Feature Domain 48 assessor knowledge 29s-- Precedent comparator Report generator 32 Machine I/O Interface 34 30,- learning model trainer Results output 36 User interface 14- GU--16 Figure 1

Description

1/8

Image analysis controller 12 Processor 18 Memory 20

22---- Image data Program 38 processor code

24 Segmenter Image data 40 & identifier

Feature Non-image data 42 25'-'quantifier Training data 44

26,- Non-image Machine learning 46 data processor models

28 Feature Domain 48 assessor knowledge

29s-- Precedent comparator Report generator 32

Machine I/O Interface 34 30,- learning model trainer Results output 36

User interface 14- GU--16

Figure 1

Image Analysis Method and System

Related Application

This application is divided from and claims the benefit of the filing and priority date of application no. 2020230279 filed 9 September 2020, which was divided from application no. 2019204376 filed 21 June 2019, the contents of both of which as filed are incorporated herein by reference in their entirety.

Field of the Invention

The present invention relates to an image analysis method and system, of particular but by no means exclusive application in medical imaging, such as to diagnose and monitor diseases or conditions in (e.g. musculoskeletal) imaging. The region of interest may be a 2D region or3D volume in such medical images. Possible medical imaging applications include Computed Tomography (CT), Magnetic Resonance (MR), Ultrasound, and Pathology Scanner imaging.

Background

The morbidity, mortality and costs associated with the occurrence of musculoskeletal related diseases are increasing, in part because longevity increases the proportion of the old population. Early detection of musculoskeletal disease facilitates intervention that may reduce disease progression or minimize negative consequences (such as bone fractures).

In the field of musculoskeletal imaging, various approaches are used to detect musculoskeletal disease. For example, bone biopsy is used to detect bone cancer or other bone diseases, and can measure the characteristic of tissues accurately. However, bone biopsies are invasive and can cause pain and scaring.

In other approaches, musculoskeletal diseases are assessed by analysing musculoskeletal images to identify or quantify abnormalities, the images collected with various imaging modalities-including DXA (Dual-energy X-ray Absorptiometry), CT (Computed Tomography), MRI (Magnetic Resonance Imaging) and X-ray scanner. Each modality has specific advantages in the screening and evaluation of musculoskeletal diseases (such as bone fragility, osteoarthritis, rheumatoid arthritis, osteomalacia and bone deformity).

For example, HRpQCT (High-Resolution peripheral Quantitative Computed Tomography) is

1 P0091AUO2 high resolution, low radiation and non-invasive; it is used to assess three important determinants of bone quality: microarchitecture, mineralisation and biomechanical properties.

There exist manual, semi-manual, and automated techniques for assessing musculoskeletal diseases, involving various imaging modalities. For example, the BMB (Bone Marrow Burden) score isan MRI scoring method for manually assessing the extent of bone marrow involvement in Gaucher disease. Radiologists measure a BMB score with an MRI image of lumbar spine and femur, using signal intensity and distribution according to ranking criteria. For example, one known score ranges from 0 to 8 forthe lumbar spine and from 0 to 8 forthe femurs, for an overall score of 0 to 16. A highertotal BMB score indicates more severe bone marrow involvement.

Measuring bone mineral density using DXA images is performed semi-manually. The spine or hip are scanned by DXA. The radiologist or doctor selects the region of interest (such as different spine sections on the spine scan and femoral neck on the hip scan). The bone mineral density of the selected regions is determined based on a predefined density calibration formula. The measured density value is converted to a T score by comparing it with measurements from a population of young adults of the same sex with peak bone mass. A T score of > -1 is considered normal; a T score -2.5 < T < -1 is classified as osteopenia; a T score of -2.5 is defined as osteoporosis. The T score is considered by a clinician when hershe isassessingthe riskof fracture development and whether treatment should be recommended.

However, major concerns in manual or semi-manual analyses of musculoskeletal imaging include labour-intensiveness and reproducibility. Accuracy and reproducibility cannot be guaranteed, owingto the subjectivity in the measurements and their evaluation.

One existing automated method for assessing musculoskeletal images is disclosed in U.S. Patent No. 906432, entitled "Method and system for image analysis of selected tissue structures". This method automatically analyses and assesses a musculoskeletal image, such as a wrist CT scan. The method can be used automatically to extract the radius bone from the wrist CT scan, segment the radius bone into microstructures (viz. compact cortex, transitional zone and trabecular region), and quantify the cortical porosity and trabecular density.

2 P0091AUO2

Known methods have focused primarily on each measurement individually, which is not helpful in understanding whether certain attributes of musculoskeletal can discriminate between those with and without disease. For example, the BMB score assesses the involvement of bone marrow only in Gaucher disease; the BMD score evaluates the contributions of bone mineral density only to the fracture risk; cortical porosity measures an important attribute of the bone but does not provide complete information on bone microstructure.

Summary

According to a first aspect of the present invention, there is provided a computer implemented image analysis method, comprising:

quantifying one or more features (comprising, for example, structures or materials) segmented and identified from a medical image of or including bone of a subject;

extracting non-image or video data pertaining to the subject and pertinent to musculoskeletal disease, decay, loss or deterioration from one or more sources of non image or video data;

extracting clinically relevant features from the non-image or video data pertaining to the subject;

assessing bone fragility or decay, comprising assessing the quantified features segmented from the medical image and the features extracted from the non-image or video data with a trained machine learning model; and

outputting one or more results of the assessing of the bone fragility or decay.

Thus, by combining a plurality of features extracted from both image and non-image or video data using machine learning algorithms (such as deep learning algorithms), the present invention can be used to diagnose and monitor diseases or conditions in medical imaging, such as musculoskeletal images.

In an embodiment, the sources of non-image or video data comprise:

video; and/or

voice; and/or

3 P0091AUO2 recorded voice from subject interview data; and/or video of the subject while walking; and/or video of the subject while walking wherein the method comprises extracting one or more gait features.

In an embodiment, extracting non-image or video data includes extracting structured data and unstructured data from sources of structured data and unstructured data respectively, and/or converting unstructured data into structured data.

In an embodiment, the method includes receiving the image, segmenting one or more features from the image, and identifying the features segmented from the image.

In an embodiment, the method includes receiving the image with features segmented therefrom, and identifying the features segmented from the image.

In an embodiment, the segmenting and identifying are implemented with a machine learning algorithm trained segmentation and identification model configured to segment and identify the features from the image. In an example, the trained segmentation and identification model comprises a deep convolutional neural network trained model.

In an embodiment, the trained machine learning model is a disease classification model.

In an embodiment, the trained machine learning model is a model trained using features extracted from patient data and labels or annotations indicating disease or non-disease.

In an embodiment, the trained machine learning model comprises a deep learning neural network or other machine learning algorithms (such as a SupportVector Machine, Decision Tree or AdaBoost).

In an embodiment, the trained machine learning model is a model trained to diagnose and/or monitor one or more (e.g. musculoskeletal) diseases.

In an embodiment, the method further comprises (i) training the trained machine learning model, and/or (ii) updating (such as continually) the trained machine learning model with additional labelled data derived from new or newly analysed subject data.

In an embodiment, the results comprise one or more disease classifications and/or probabilities.

4 P0091AUO2

In an embodiment, the method further comprises generating a report based on at least the results. In an example, the method includes generating the report based additionally on information from a domain knowledge database.

In an embodiment, quantifying the features comprises determining any one or more of: volumetric bone mineral density of the bone, cortical porosity, transitional region volumetric bone mineral density, trabecular region volumetric bone mineral density, matrix mineralization level, marrow adiposity, average thickness of the compactcortex, trabecular tissue separation, trabecular density, and bone volume fraction.

In an embodiment, the results include one or more fracture risk scores.

According to this aspect, there is also provided a computer-implemented method of diagnosing, monitoring, screening for or evaluating a musculoskeletal disease orcondition, the method comprising:

quantifying one or more features segmented and identified from a medical image of a subject;

extracting non-image or video data pertaining to the subject and pertinent to the musculoskeletal disease orcondition from one or more sources of non-image orvideo data;

extracting clinically relevant features from the non-image or video data;

diagnosing, monitoring, screening for or evaluating the musculoskeletal disease or condition, comprising assessing the quantified features segmented from the medical image and the features extracted from the non-image or video data with a trained machine learning model trained to diagnose, monitor, screen for or evaluate one or more musculoskeletal diseases or conditions; and

outputting one or more results of the diagnosing, monitoring, screening for or evaluating the musculoskeletal disease or condition.

According to this aspect, there is also provided a computer-implemented method of determining a treatment, or producing guidelines for treatment of, a musculoskeletal disease or condition, the method comprising

5 P0091AUO2 extracting non-image or video data pertaining to the subject and pertinent to the musculoskeletal disease or condition from one or more sources of non-image or video data; extracting clinically relevant features from the non-image or video data; determining a treatment, or producing guidelines for treatment of, the musculoskeletal disease or condition, comprising assessing the quantified features segmented from the medical image and the features extracted from the non-image or video data with a trained machine learning model trained to determine a treatment, or produce guidelines for treatment of, one or more musculoskeletal diseases or conditions; and outputting one or more results of the determining a treatment, or producing guidelines for treatment of, the musculoskeletal disease or condition.

The disease or condition may be arthritis, bone fracture, or osteomalacia.

According to a second aspect of the present invention, there is provided an image analysis system, comprising:

a feature quantifier configured to quantify one or more features (structures or materials) segmented and identified from a medical image of or including bone a subject;

a non-image or video data processor configured to extract non-image or video data pertaining to the subject and pertinent to musculoskeletal disease, decay, loss or deterioration from one or more sources of non-image or video data and to extract clinically relevant features from the non-image or video data pertaining to the subject;

a feature assessor configured to assess bone fragility or decay, comprising assessing the quantified features segmented from the medical image and the features extracted from the non-image or video data with a trained machine learning model; and

an output configured to output one or more results of the assessing of the bone fragility or decay.

In an embodiment, the sources of non-image or video data comprise:

video; and/or

voice; and/or

recorded voice from subject interview data; and/or

6 P0091AUO2 video of the subject while walking; and/or video of the subject while walking wherein the non-image or video data processor is further configured to extract one or more gait features.

In an embodiment, the non-image or video data processor is further configured: to extract structured data and unstructured data from sources ofstructured data and unstructured data respectively, and/orto convert unstructured data intostructured data.

In an embodiment, the system further comprises a segmenter and identifier configured to receive the image, segment one or more features from the image, and identify the features segmented from the image. In one example, the segmenter and identifier comprises a segmentation and identification model (that is, a segmentation and identification model trained usinga machine learning algorithm) configured tosegmentand identify the features from the image. The trained segmentation and identification model may comprise a deep convolutional neural network trained model.

In an embodiment, the trained machine learning model comprises a deep learning neural network or other machine learning algorithms.

In an embodiment, the system further comprises a machine learning model trainer configured to update (such as continually) the trained machine learning model with additional labelled data derived from new or newly analysed subject data.

In an embodiment, the system further comprises a report generator configured to generate a report based on at least the results. In an example, the report generator is configured to generate the report based additionally on information from a domain knowledge database (such as of the system).

7 P0091AUO2

In an embodiment, the feature quantifier is configured to determine any one or more of: volumetric bone mineral density of the bone, cortical porosity, transitional region volumetric bone mineral density, trabecular region volumetric bone mineral density, matrix mineralization level, marrow adiposity, average thickness of the compactcortex, trabecular tissue separation, trabecular density, and bone volume fraction.

In an embodiment, the results include one or more fracture risk scores.

According to this aspect, there is also provided a system for diagnosing, monitoring, screeningfor or evaluating a musculoskeletal disease or condition, the system comprising:

a feature quantifier configured to quantify one or more features segmented and identified from a medical image of a subject;

a non-image orvideo data processor configured to extract non-image orvideo data pertaining to the subject and pertinent to the musculoskeletal disease or condition from one or more sources of non-image or video data and to extract clinically relevant features from the non-image or video data;

a feature assessor configured to diagnose, monitor, screen for or evaluate the musculoskeletal disease or condition, comprising assessing the quantified features segmented from the medical image and the features extracted from the non-image or video data with a trained machine learning model trained to diagnose, monitor, screen for or evaluate one or more musculoskeletal diseases or conditions; and

an output configured to output one or more results of diagnosing, monitoring, screening for or evaluating the musculoskeletal disease or condition.

According to this aspect, there is also provided a system for determining a treatment, or producing guidelines for treatment of, a musculoskeletal disease or condition, the system comprising:

a non-image orvideo data processor configured to extract non-image orvideo data pertaining to the subject and pertinent to the musculoskeletal disease or condition from one or more sources of non-image or video data and to extract clinically relevant features

8 P0091AUO2 from the non-image or video data; a feature assessor configured to determine a treatment, or produce guidelines for treatmentof, the musculoskeletal disease orcondition, comprising assessing the quantified features segmented from the medical image and the features extracted from the non-image or video data with a trained machine learning model trained to determine a treatment, or produce guidelines for treatment of, one or more musculoskeletal diseases or conditions; and an output configured to output one or more results of the determining a treatment, or producing guidelines for treatment of, the musculoskeletal disease or condition.

The disease or condition maybe arthritis, bone fracture, or osteomalacia.

According to a third aspect of the present invention, there is provided computer program code comprising instructions configured, when executed byone or more computing device, to implementthe method of thefirst aspect. This aspect also provides a computer-readable medium (which may be non-transitory) comprising such computer program code.

Itshould be noted that any of the various individual features of each of the above aspects of the invention, and any of the various individual features of the embodiments described herein including in the claims, can be combined as suitable and desired.

Brief Description of the Drawing

In order that the invention may be more clearly ascertained, embodiments will now be described by way of example with reference to the following drawing, in which:

Figure 1 is a schematic view of a medical image analysis system according to an embodiment of the present invention;

Figures 2A and 2B are schematic views of exemplary image data, as stored in the memory of the medical image analysis system of Figure 1;

Figures 3A and 3B depict schematically examples of the non-image data store, unpopulated and populated respectively, of the medical image analysis system of Figure 1;

Figure 4 is a schematic representation of the operation of the image data processor of the medical image analysis system of Figure 1, in a bone fragility application;

9 P0091AUO2

Figure 5 is a schematic representation of the operation of the non-image data processor of the medical image analysis system of Figure 1, on exemplary non-image data that includes structured and unstructured data;

Figure 6 is a schematic view of the offline trainingof the machine learning model (or models) of the feature assessor of the medical image analysis system of Figure 1;

Figure 7 is a schematic view of the report generator of the medical image analysis system of Figure 1;

Figure 8 illustrates an exemplary operation of the medical image analysissystem of Figure 1 in a bone fragility application, with image data comprising a wrist HRpQCT scan and non-image data comprising basic patient information; and

Figure 9 is an exemplary report, generated by the report generator of the medical image analysis system of Figure 1.

Detailed description

Figure 1 is a schematic view of an image analysis system in the form of a medical image analysissystem 10 accordingto an embodiment of the present invention.

Referring to Figure 1, system 10 comprises an image analysis controller 12 and a user interface 14 (includinga GUI 16). User interface 14 typically comprises one or more displays (on one or more of which may be displayed the GUI 16), a keyboard and a mouse, and optionally a printer. Image analysis controller 12 includes at least one processor 18 and a memory20. System 10 maybe implemented, for example, as a combination ofsoftwareand hardware on a computer (such as a personal computer or mobile computing device), or as a dedicated image segmentation system. System 10 may optionally be distributed; for example, some or all of the components of memory 20 may be located remotely from processor 18; user interface 14 may be located remotely from memory 20 and/or from processor 18 and, indeed, may comprise a web browser and a mobile device application.

Memory 20 is in data communication with processor 18, and typically comprises both volatile and non-volatile memory (and may include more than one of each type of memory), including RAM (Random Access Memory), ROM and one or more mass storage devices.

As is discussed in greater detail below, processor 18 includes an image data processor 22

10 P0091AUO2

(which includes a segmenter and identifier 24 and a feature quantifier25), a non-image data processor 26, a feature assessor 28 (which includes a precedent comparator 29), a machine learning model trainer 30, a report generator 32, an 1/O interface 34 and an output in the form of a results output36. Memory20 includes program code38, image data 40, non-image data 42 (which, in this embodiment, may include video data), training data 44, trained machine learning models 46, and domain knowledge 48. Image analysis controller 12 is implemented, at least in part, by processor 18 executing program code38 from memory20.

In broad terms, 1/O interface 34 is configured to read or receive image data and non-image data (such as in DICOM format, and which may include video data) pertainingto subjects or patients into image data 40 and non-image data 42 of memory 20 for analysis. Image data processor 22, using segmenter and identifier 24 and feature quantifier 25, segments (i.e. extracts) and quantifies features from the image data, non-image data processor26 extracts features from the non-image data, feature assessor 28 processes the features with one or more of machine learning models46, then 1/O interface 34 outputsthe resultof the analysis to, for example, results output 36 and/orto GUI 16.

System 10 employs artificial intelligence (such as a deep neural network), other machine learning algorithms and computer vision algorithms, to automate-accurately and reproducibly-feature extraction and quantification of imagefeatures, of particularvalue in analysing musculoskeletal images. The results are suitable for subsequent use in locating sites of musculoskeletal disease, for diagnosing musculoskeletal disease, and for monitoring disease progression.

Referringto Figure 1, system 10 is configured to receive two types of subject or patient data: image data (e.g. medical images acquired by one or more image modalities at one or more anatomical sites) and non-image data (structured patient data, such as clinical history data and questionnaire data, and unstructured patient data, such as doctors' notes and voice records). System 10 stores these data in image data 40 and non-image data 42, respectively. The different data forms and structures of the non-image data determine the operation of non-image data processor 26.

Image data processor 22 comprises two components: segmenter and identifier 24 is configured to perform segmentation and identification, and feature quantifier 25 configured to perform feature quantification. The image data 40 are processed by image data processor 22 which, using segmenter and identifier 24, automatically segments and

11 P0091AUO2 identifies clinically relevant structures, features or materials from, in this embodiment, the medical image of the subject or patient, then feature quantifier25 quantifies the segmented and identified clinically relevant structures, features, materials or combinations thereof. It should be noted, however, that in some embodiments system 10 may be configured to receive images that have already been segmented (with, optionally, the segmented features already identified), such that segmenter and identifier 24, or its use, may be omitted.

In this embodiment, the segmentation and identification is implemented with a conventional (i.e. non-machine learning) image processing method, such as thresholding, contour detection, or blob detection, or according to the method disclosed in U.S. Patent No. 9,064,320 (entitled "Method and system for image analysis of selected tissue structures").

However, in some other embodiments, segmenter and identifier 24 comprises a machine learning algorithm trained segmentation and identification model configured to segment and identify the structures or features of interest from the image. For example, in a bone fragility application, such a model-trained using a deep convolutional neural network may be used to segment and identify radius bone from a wrist HRpQCT scan; the training data may be annotated wrist HRpQCT scans that distinguish the voxels of the radius bone, ulna bone, and surrounding material.

The non-image data are processed by non-image data processor 26, which extracts clinically relevant features from the non-image data.

The quantitative features obtained from both the image data and non-image data are inputted into a machine learning model 46 byfeatures assessor28. Machine learning model 46 is pre-trained using training data with labels, that is, annotations that constitute the 'ground truth'for machine learning. For example, to train a bone segmentation model, the

training data typically comprises original images and the corresponding ground truth, hence the images in which the bones have been annotated. To train a disease classification model, the training data typically comprises the features extracted from patient data and the corresponding ground truth, which are thelabels/annotations indicating disease or non-disease. (The terms'label'and 'annotation'are essentially interchangeably, but'label' is generally reserved herein for a disease or condition, with 'annotation' generally reserved for image annotation in image segmentation/ identification machine learning models.)

12 P0091AUO2

In this embodiment, processor 18includes a machine learning model trainer 30, configured to train machine learning models 46 (and to retrain or update train machine learning models 46, as discussed below) using training data 44 (which may include new subject data). In other embodiments, however, machine learning model trainer 30 may be configured or used only to retrain or update train machine learning models 46.

Various machine learning algorithms (using features from both image and non-image data) may be used to train the machine learning models 46 employed in this embodiment to diagnose and monitor musculoskeletal diseases, etc., such as a deep learning neural network (which is preferred) or other machine learning algorithms such as Support Vector Machine, Decision Tree and AdaBoost.

In one implementation, a pre-trained machine learning model 46 is updated continually with additional labelled data derived from newly analysed patient data.

Thus, feature assessor 28 evaluate and assesses the features using a machine learning model 46, and then outputs results (in the form of, for example, classifications and probabilities) to report generator32. Report generator32 generates a report based on these results. Report generator 32 may also pull additional information from domain knowledge 48, and combine that information with the results, both being presented in the report.

For example, in an embodiment of bone fragility assessment application, feature assessor 28-using a machine learning model 46-may output a result comprising a fracture risk score. Report generator 32 may be configured to pull information indicated by this score from domain knowledge 48, such as information to assist in the interpretation of the score. Report generator 32 can then present both the score and the information in a report.

The ultimate report is outputted via results output 36 and/or user interface 14.

Figures 2A and 2B are schematic views of exemplary image data, as stored in image data 40. The image data comprise clinically relevant medical images of the subject or patient, typically images acquired by one or more imaging modalities at one or more anatomical sites.

For example, in the example of bone fragility, the input image data may comprise a wrist HRPpQCT scan 50, as illustrated in Figure 2A. In another example, it may be desired to analyse both the peripheral and central skeletal so, as depicted in Figure 2B, the input

13 P0091AUO2 image data may comprise a wrist HRpQCT scan 50, a leg HRpQCT scan 52, a spine DXA scan 54, and a hip DXA scan 56 (and, if bone marrow information is optionally incorporated, a hip MRI scan 58).

Figures 3A and 3B depict schematically examples of non-image data 42, unpopulated and populated respectively. The non-image data received bysystem 10 and stored in non-image data 42 may include a significant quantity of useful data indicative of musculoskeletal diseases. As shown in Figure 3A, the non-image data may include patient information from various sources of structured and unstructured data, which are collected-for example during a subject or patent's medical consultations and treatment. The structured data may include basic patient information, such as sex, age, weight, height, laboratory test results (such as blood test results and DNA test results), treatment data (such as types of medication and dosage), and questionnaire data (such as smoking and drinking habits and fracture history). The unstructured data may include text documents of laboratory results, doctors' notes and radiological reports.

As illustrated in Figure 3A, the non-image data in non-image data 42 may be in a variety of formats, such asstructured video 62,text64, numbers 66, and/orvoice 68, and unstructured

video 72, text 74, numbers 76 and/or voice 78. Figure 3B depicts an exemplary scenario, in which the non-image data comprises structured text 64 in the form of clinical information and history 80, structured numbers 66 in the form of patient measurements 82,

unstructured video 72 in the form of video of the patientwhile walking 84, unstructured text 74 in the form of doctors' notes 86, and unstructured voice 78 in the form of patient interviews 88.

Non-image data processor 26 is configured with and employs different data processing and feature extraction techniques in processing the non-image data, in each case according to the structure and form of each portion of that data.

Figure 4 is a schematic representation 90 of the operation of image data processor 22, in a bone fragility application. In this example, the radius bone is identified and segmented 92 from a wrist HRpQCT scan using a pre-trained deep neural network model. The radius bone is then segmented and identified by segmenter and identifier 24, into compact cortex 94, transitional zone 96, and trabecular region 98. Next, the attributes of different structures or combination of structures are quantified byfeature quantifier25 into the following features: vBMD (the volumetric bone mineral density of the entire radius bone) 102, cortical porosity

14 P0091AUO2

(the percentages of pores in the bone volume) 104,transitional region vBMD (the volumetric bone mineral density of the transitional region) 106,trabecular regionvBMD (the volumetric bone mineral density of the trabecular region) 108, matrix mineralization level (the average percentage of the mineralized material) 110, marrow adiposity (the density of marrow adiposity in the trabecular region) 112, cortical thickness (the average thickness of compact cortex) 114,and trabecular tissue separation (the average separation of trabecular tissues) 116. These features are chosen because they have been shown to be indicative of bone fragilityand riskof bonefracture. In other applications, features pertinent to the condition of interestwould likewise beselected.

Figure 5 a schematic representation 120 of the operation of non-imagedata processor26on exemplary non-image data 42 that includes structured and unstructured data. Non-image data processor 26 extracts features from the non-image data with a variety of tools each adapted to extracttypes of non-image data of different structure and form.

Structured data 122 are typically stored and maintained in structured data storage such as database tables, .json files, .xml files and .csv files. Non-image data processor 26 extracts features from the structured data by querying 124 the required parameters and attributes from the sources of the data, and thereby extracting the information of interest from the structured data sources. It should be noted that, in some cases, the structured data may be of complete in itself and of interest, such that no querying is required.

Unstructured data 126 may comprise doctors' notes, voice record and graphic reports, etc. so, before feature extraction, non-image data processor 26 typically converts 128 the unstructured data 126 into structured data. The conversion method employed by non image data processor26 isspecificto each source ortype of unstructured data. Forexample, to convert doctors' notes to structured data, non-image data processor 26 may include or employ and employ a trained model of optical character recognition (OCR) to convert 128 the notes to text recognisable by system 10. Non-image data processor 26 then parses the text (using keywords such as "fractures", "pain", "fall", etc).

In another example, non-image data processor 26 may comprise or employ a trained model of speech recognition to convert 128 the recorded voice from an interview to text recognisable by system 10. Non-image data processor 26 then segments and organises the converted text into structured data by referring to the original interview questions.

15 P0091AUO2

In another exemplary embodiment, the non-image data may include a video as, for example, studies have shown that patterns of walking gait are associated with fracture risk. Non-imagedata processor 26 processes the video and extracts 130 gait features therefrom.

Once converted 128 into structured data, non-image data processor 26 extracts features from the (previously unstructured) structured data by querying124 the required parameters and attributes from the sources of the data. As discussed above, however, if the now structured data is complete in itself and of interest, no querying will be required; in the above example of video of walking, after converting the video into gait features-which constitute structured data-no such query 124 is required.

Ultimately, non-image data processor 26 extracts features 130 such as sex 132, age 134, smoking habit 136, fracture history 138, treatment history 140 and gait 142.

Figure 6 is a schematic view 144 of the (offline) training of the model or models in machine learning models46. Firstly, atstep 146, training data are accessed orselected for processing from a database of training data 44. The training data may comprise subject image data and/or subject non-image data. Data with various attributes are selected for training to represent the target population. The training data are processed by, at step 148, extracting features and, at step 150, quantifying the extracted features, using the methods described above. The training data then comprises a set of quantitative features.

At step 152, the training data are annotated (by a human operator) with the correct diagnosis results. At step 154, the training data represented by the extracted features combined with the diagnosis labels are fed into one or more machine learning algorithms of machine learning model trainer 30 to train one or more model(s). It is to be appreciated that various machine learning algorithms may be suitable to train the model(s). It may be a black-box algorithm (e.g. a neural network) such that the process of decision making is not human-readable. It may alternatively be a white-box algorithm, (e.g. a decision tree, support vector machine, or linear discriminant analysis) such that a human can interpret the decision-making process of the model. It may also comprise a hybrid algorithm that combines both white-box and black-box algorithms.

At step 156, the model or models-now trained-are deployed for use, typically by being stored in machine learning models 46.

In use, the features extracted from new patient data are fed into the now trained machine

16 P0091AUO2 learning model 46. As described above, a machine learning model 46 assesses the features and outputs one or more results. The results may be binary classifications, scores, or probabilities. In some embodiments, the new patient data may be added (at optional step 158) to training data 44, to be used in retraining or updating model 46.

The model training and its utility is specific to the domain; hence, so are each of the resulting machine learning models 46. Each of model 46 depends on the data and targeted conditions (e.g. a condition, such as bone fragility; a disease, such as osteoporosis, an infection, such as a particular bacterial infection). In one embodiment, one of models 46 is trained and used to diagnose one or more diseases or conditions, by generating one or more disease probability scores. In another embodiment, a plurality of models 46 may be trained and used to diagnose respective diseases. Likewise, in another embodiment, one of models 46 may be trained to generate a prediction of the likelihood of the expression of one or more symptoms (e.g. bone fracture), or a plurality of models 46 may be trained and used to generate respective predictions of the likelihood of the expression of the respective symptoms.

Figure 7 is a schematic view of report generator 32, which generates reports based on at least the results 162 obtained from machine learning model 46. Report generator 32 may also, in generating the report, use information stored in domain knowledge 48. This information may include, for example, information of value to a doctor in interpreting the results outputted by features assessor 28 (such as diagnosis and treatment guidelines). For example, in the application of system 10 to the assessment of bone fractures, a doctor may be presented with treatment guidelines pertinent to different levels of bone fracture risk. The report may then include both a plurality of results generated by features assessor 28 and characterizing a bone of a subject or patient (such as a fragility score, a porosity score and a trabecular score), but also with information of use in interpreting those scores and in selecting a treatment.

This information may optionally include the diagnoses of a past cohort of (one or more) subjects or patientswith comparable scores, and/orthe treatment ortherapy prescribed for that cohort. Indeed, in some embodiments, features assessor 28 includes an optional precedent comparator 29, configured to compare the results (in particular numerical results such as scores) generated by machine learning model 46 with such information from domain knowledge 48, to automatically form a diagnosis (by assigning a probability-or

17 P0091AUO2 respective probabilities-that the subject has the disease or diseases) and therapy/treatment recommendation based on that comparison, and to output these to report generator 32 for inclusion in the report.

The report may also include the results of the monitoring of disease progression and the efficacy of treatment. For example, the effectiveness of the treatment may be shown if the treated subject or patient's bone fracture risk is seen to decrease in a follow-up test.

The report may also display similar or opposite cases. Presenting similar or opposite cases can be of value to a medical practitioner in assessing prognosis and determining treatment. The presentation of similar or opposite cases can be useful in training inexperienced users, or assist users to confirm the results generated from model 46. Moreover, by studying similar or opposite cases to see how past subjects or patients responded to different treatment options, a medical practitioner can assess the efficacy of a treatment (or likely efficacy of a proposed treatment) on the current patient.

All of these results and information 164 are included in the report 166 by report generator 32, which then delivers the report 166 (via results output 36) to user interface 14 (such as a web browser, a PC application or a mobile device application) for perusal by a user or users.

Figure 8 illustrates an exemplary operation of system 10 in a bone fragility application. The image data comprises a wrist HRpQCT scan 172 and the non-image data comprises basic patient information 174. As shown in Figure 8, segmenter and identifier 24-using a trained segmentation and identification model 46' from machine learning models 46-identifies and extracts 176 the radius bone and segments 178 it into structures: compact cortex, transitional zone, and trabecular region. Based on this structural segmentation, feature quantifier 25 determines cortical porosity 180 and trabecular density 182.

Non-image data processor 26 employs a query 184 to extract sex 186 and age 188 from basic patient information 174 of the non-image data. Feature assessor 28 receives the four features cortical porosity 180, trabecular density 182, sex 186 and age 188. Feature assessor 28 employs anothertrained model 46" (in the form of, in this example, a bone decay model) to generate a structural fragility score 190 by assessing the four aforementioned features 180, 182, 186, 188. Alternatively, bone decay model 46' may be trained using an algorithm such as that described by Zebaze et al. in "Increased Cortical Porosity and Reduced Trabecular Density Are Not Necessarily Synonymous With Bone Loss and Microstructural

18 P0091AUO2

Deterioration" (JBMR Plus (2018)), and with training data 44 representing the four aforementioned features 180, 182, 186, 188 and annotated with the correct bone fragility diagnosis.

Fragility score 190 and other information are used by report generator 32 to generate a report 166, which is ultimately outputted to a user interface 14 in the form of, in this example, a web browser.

Figure 9 is an exemplary report 200, generated by report generator 32 of system 10. Report 200 includesseveral registers: a subject/patient details register202, a BoneVolume Fraction Map register 204, a Scores register 206 and a Score Graphs register 208. Subject/patient details register 202 typically includes biographical details (such as name, date of birth, sex and age), doctor's name (or doctors'names), the date of the acquisition of the data, and the date of the processing of the data (which is also the date of the report 200).

Bone Volume Fraction Map register 204 includes a false colour 3D reconstruction 210 of, in this example, the patient's radius bone volume, a false colour 3D reconstruction 212 of a corresponding reference (and typically healthy) radius bone volume, and a false colour key 214. (Note that, in figure 9, the colours are reproduced as a greyscale.) The false colour is used to indicate "bone volume fraction", which is the volume of mineralised bone per unit volume of the sample. The report 200 presents the reconstructions 210, 212 adjacentto one another so that a medical practitioner can readily evaluate the extent and distribution of in this example-bone loss.

Scores register206 includesscores generated byfeatures assessor28. In this example, these scores are a fragility score 216, a porosity score 218 and a trabecular score 220, expressed as percentages. Fragility score 216 is indicative of the level of coexisting cortical and trabecular deterioration. Porosity score 218 is indicative of the porosity of cortex bone. Trabecular score 220 is a score indicative of the density of the trabecular region.

Optionally, these scores 216, 218, 220 may be presented graphically on respective bars 222, 224, 226 with the score represented-for example-as a slider 228, 230, 232. Each bar, in this example, is labelled with three values: low and high values indicating the scores of the ends of the bar, and a third value indicating a division between lower and higher risk (or normal and abnormal) ranges. For example, these three values are, in the illustrated example of bar 222 of fragility score 216, respectively 20%, 90% and 70%. In the illustrated example of bar

19 P0091AUO2

224 of porosity score 218, respectively 20%, 60% and 46%, and in the illustrated example of bar 226 of trabecular score 220, respectively 0%, 10% and 2%.

These three values (low end, high end, and the division between lower and higher risk) are determined from precedent data, such as that contained in training data 44 or domain knowledge 48. Alternatively, some or all of the precedent data may be accessed by system 10 from one or more external databases.

20 P0091AUO2

The bars 222,224,226 may optionally be coloured or shaded to indicate the regions of lower risk 234 and higher risk 236, so that it is immediately visually clear whether a respective slider 228, 230, 232 (and hence score 216, 218, 220) corresponds to precedent data associated with high or low risk subjects. In Figure 9, a denser shading is used to indicate the region of higher risk.

Thus, in scores register 206, the report200 gives a fragilityscore 216 of 75% (indicated to be higher risk), a porosity score of 37% (indicated to be lower risk), and a trabecular score of 1% (indicated to be higher risk). These scores, neither individually nor in combination, constitute a diagnosis, as they do not identify a specific disease or affliction. However, they give the user a useful indication of symptoms and of how pronounced are those symptoms.

Score Graphs register 208 presents plots 240, 242, 244 of fragility score (FS((%)), porosity score (PS((%)), and trabecular score (TS((%)), respectfully, evolving over time T. In the illustrated example, the time axis divisions are typically a day or two, but in general each division will represent the time between scans (which also means that the divisions in a single graph may not represent a constant time difference). In Figure 9, plots 240, 242, 244 first indicate the scores of Scores register 206, followed in each case by subsequent scores obtained in a series of follow-up consultations; plots 240, 242, 244 thereby allow a medical practitioner to quickly assess changes in the subject, whether spontaneous, in response to treatment, or otherwise.

It will be understood to persons skilled in the art of the invention that many modifications may be made without departing from the scope of the invention, in particular it will be apparent that certain features of embodiments of the invention can be employed to form further embodiments.

It is to be understood that, if any prior art is referred to herein, such reference does not constitute an admission that the prior art forms a part of the common general knowledge in the art in any country.

In the claims which follow and in the preceding description of the invention, except where the context requires otherwise due to express language or necessary implication, the word "comprise" orvariations such as"comprises" or"comprising" is used in an inclusive sense, i.e. to specify the presence of the stated features but not to preclude the presence or addition of furtherfeatures in various embodiments of the invention.

21 P0091AUO2

Claims

CLAIMS:

1. A computer-implemented image analysis method, comprising:

quantifying one or more features segmented and identified from a medical image of or including bone of a subject;

outputting one or more results of the assessing of the bone fragility or decay.

2. A computer-implemented method of diagnosing, monitoring, screening for or evaluating a musculoskeletal disease or condition, the method comprising:

extracting non-image or video data pertaining to the subject and pertinent to the musculoskeletal disease or condition from one or more sources of non-image orvideo data;

extracting clinically relevant features from the non-image or video data;

diagnosing, monitoring, screening for or evaluating the musculoskeletal disease or condition, comprising assessingthe quantified features segmented from the medical image and the features extracted from the non-image or video data with a trained machine learning model trained to diagnose, monitor, screen for or evaluate one or more musculoskeletal diseases or conditions; and

22 P0091AUO2

3. A computer-implemented method of determining a treatment, or producing guidelines for treatment of, a musculoskeletal disease or condition, the method comprising

extracting non-image or video data pertaining to the subject and pertinent to the musculoskeletal disease or condition from one or more sources of non-image or video data;

extracting clinically relevant features from the non-image or video data;

determining a treatment, or producing guidelines for treatment of, the musculoskeletal disease or condition, comprising assessing the quantified features segmented from the medical image and thefeatures extracted from the non-image orvideo data with a trained machine learning model trained to determine a treatment, or produce guidelines for treatment of, one or more musculoskeletal diseases or conditions; and

outputting one or more results of the determining a treatment, or producing guidelines for treatment of, the musculoskeletal disease or condition.

4. A method as claimed in any one of the preceding claims, wherein the sources of non image or video data comprise:

video; and/or

voice; and/or

recorded voice from subject interview data; and/or

video of the subject while walking; and/or

video of the subject while walking wherein the method comprises extracting one or more gait features.

5. A method as claimed in any one of the preceding claims, wherein extracting non-image or video data includes extracting structured data and unstructured data from sources of structured data and unstructured data respectively, and/or converting unstructured data into structured data.

6. A method as claimed in any one of the preceding claims, wherein the segmenting and identifying are implemented with (i) a machine learning algorithm trained segmentation

23 P0091AUO2 and identification model configured to segment and identify the features from the image, or (ii) a machine learning algorithm trained segmentation and identification model that comprises a deep convolutional neural networktrained model and is configured tosegment and identify the features from the image.

7. A method as claimed in any one of the preceding claims, wherein the trained machine learning model further comprises any one or more of:

(a) a disease classification model;

(b) a model trained using features extracted from patient data and labels or annotations indicating disease or non-disease;

(c) a deep learning neural network or other machine learning algorithms; and

(d) a model trained to diagnose and/or monitor one or more musculoskeletal diseases.

8. A method as claimed in any one of the preceding claims, further comprising (i) training the trained machine learning model, and/or (ii) updating the trained machine learning model with additional labelled data derived from new or newly analysed subject data.

9. A method as claimed in any one of the preceding claims, wherein the results comprise any one of more of (i) one or more disease classifications; (ii) one or more disease probabilities; (iii) a bone fracture risk; (iv) a fragility score; (v) a porosity score; (vi) a trabecular score; (vii) a plotof fragilityscore evolving overtime; (viii) a plotof porosityscore evolving overtime; (ix) a plotof trabecularscore evolving overtime; (x) disease progression; (xi) decrease in bone fragility; and (xii) one or more fracture risk scores.

10. A method as claimed in any one of the preceding claims, wherein quantifying the features comprises determining any one or more of: volumetric bone mineral density of the bone, cortical porosity, transitional region volumetric bone mineral density, trabecular region volumetric bone mineral density, matrix mineralization level, marrow adiposity, average thickness of the compact cortex, trabecular tissue separation, trabecular density, and bone volume fraction.

11. A method as claimed in either claim 2 or 3, wherein the disease or condition is arthritis, bone fracture, orosteomalacia.

24 P0091AUO2

12. An image analysis system, comprising:

a feature quantifier configured to quantify one or more features segmented and identified from a medical image of or including bone of a subject;

a non-image data processor configured to extract non-image or video data pertaining to the subject and pertinent to musculoskeletal disease, decay, loss or deterioration from one or more sources of non-image or video data and to extract clinically relevant features from the non-image orvideo data pertainingto the subject;

a feature assessor configured to assess bone fragility or decay, comprising assessing the quantified features segmented from the medical image and the features extracted from the non-image orvideo data with a trained machine learning model; and

13. A system for diagnosing, monitoring, screening for or evaluating a musculoskeletal disease or condition, the system comprising:

a non-image or video data processor configured to extract non-image or video data pertaining to the subject and pertinent to the musculoskeletal disease or condition from one or more sources of non-image or video data and to extract clinically relevant features from the non-image or video data;

a feature assessor configured to diagnose, monitor, screen for or evaluate the musculoskeletal disease or condition, comprising assessing the quantified features segmented from the medical image and thefeatures extracted from the non-image orvideo data with a trained machine learning model trained to diagnose, monitor, screen for or evaluate one or more musculoskeletal diseases or conditions; and

14. A system for determining a treatment, or producing guidelines for treatment of, a musculoskeletal disease or condition, the system comprising:

25 P0091AUO2 a feature quantifier configured to quantify one or more features segmented and identified from a medical image of a subject; a non-image orvideo data processor configured to extract non-image or video data pertaining to the subject and pertinent to the musculoskeletal disease or condition from one or more sources of non-image or video data and to extract clinically relevant features from the non-image or video data; a feature assessor configured to determine a treatment, or produce guidelines for treatmentof, the musculoskeletal disease orcondition, comprising assessing the quantified features segmented from the medical image and the features extracted from the non-image or video data with a trained machine learning model trained to determine a treatment, or produce guidelines for treatment of, one or more musculoskeletal diseases or conditions; and an output configured to output one or more results of the determining a treatment, or producing guidelines for treatment of, the musculoskeletal disease or condition.

15. A system as claimed in any one of claims 12 to 14, wherein the sources of non-image or video data comprise:

video; and/or

voice; and/or

recorded voice from subject interview data; and/or

video of the subject while walking; and/or

video of the subject while walking wherein the non-image or video data processor is further configured to extract one or more gait features.

16. A system as claimed in any one of claims 12 to 14, wherein the non-image or video data processor is further configured: to extract structured data and unstructured data from sources of structured data and unstructured data respectively, and/or to convert unstructured data into structured data.

17. A system as claimed in any one of claims 12 to 16, wherein the trained machine learning model further comprises any one or more of:

(a) a disease classification model;

26 P0091AUO2

(c) a deep learning neural network or other machine learning algorithms; and/or

18. Asystem as claimed in any one of claims 12 to 17, further comprising a machine learning model trainer configured to continually update the trained machine learning model with additional labelled data derived from new or newly analysed subject data, comprising adding new patient data as training data for updating the trained machine learning model.

19. A system as claimed in anyone of claims 12 to 18, wherein the results comprise anyone or more of: (i) one or more disease classifications; (ii) one or more disease probabilities; (iii) a bone fracture risk; (iv) a fragility score; (v) a porosity score; (vi) a trabecular score; (vii) a plot of fragility score evolving overtime; (viii) a plot of porosity score evolving overtime; (ix) a plot of trabecular score evolving over time; (x) disease progression; (xi) decrease in bone fragility; and (xii) one or more fracture risk scores.

20. A system as claimed in any one of claims 12 to 19, wherein the feature quantifier is configured to determine any one or more of: volumetric bone mineral density of the bone, cortical porosity, transitional region volumetric bone mineral density, trabecular region volumetric bone mineral density, matrix mineralization level, marrow adiposity, average thickness of the compact cortex, trabecular tissue separation, trabecular density, and bone volume fraction.

21. A system as claimed in either claim 13 or 14, wherein the disease or condition is arthritis, bone fracture, or osteomalacia.

22. Computer program code comprising instructions configured, when executed by one or more computing devices, to implement the method of any one of claims 1 to 11.

23. A computer-readable medium comprising computer program code as claimed in claim 22.

27 P0091AUO2