CN114287915A - Noninvasive scoliosis screening method and system based on back color image - Google Patents
Noninvasive scoliosis screening method and system based on back color image Download PDFInfo
- Publication number
- CN114287915A CN114287915A CN202111629135.0A CN202111629135A CN114287915A CN 114287915 A CN114287915 A CN 114287915A CN 202111629135 A CN202111629135 A CN 202111629135A CN 114287915 A CN114287915 A CN 114287915A
- Authority
- CN
- China
- Prior art keywords
- human body
- color image
- image
- training
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 206010039722 scoliosis Diseases 0.000 title claims abstract description 51
- 238000012216 screening Methods 0.000 title claims abstract description 46
- 238000012549 training Methods 0.000 claims abstract description 72
- 238000002372 labelling Methods 0.000 claims abstract description 29
- 230000011218 segmentation Effects 0.000 claims abstract description 24
- 238000012545 processing Methods 0.000 claims abstract description 10
- 238000013145 classification model Methods 0.000 claims abstract description 7
- 238000013527 convolutional neural network Methods 0.000 claims description 12
- 238000005452 bending Methods 0.000 claims description 10
- 230000005856 abnormality Effects 0.000 claims description 6
- 201000010099 disease Diseases 0.000 claims description 5
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 claims description 5
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 238000012552 review Methods 0.000 claims description 4
- 101000911772 Homo sapiens Hsc70-interacting protein Proteins 0.000 claims description 3
- 210000000746 body region Anatomy 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 claims description 3
- 238000011161 development Methods 0.000 claims description 3
- 230000018109 developmental process Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 230000007935 neutral effect Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 9
- 238000013473 artificial intelligence Methods 0.000 abstract description 4
- 208000022567 adolescent idiopathic scoliosis Diseases 0.000 description 10
- 238000003745 diagnosis Methods 0.000 description 4
- 239000011436 cob Substances 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 210000000988 bone and bone Anatomy 0.000 description 2
- 230000006378 damage Effects 0.000 description 2
- 238000004441 surface measurement Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 208000017667 Chronic Disease Diseases 0.000 description 1
- 206010033799 Paralysis Diseases 0.000 description 1
- 208000019155 Radiation injury Diseases 0.000 description 1
- 208000020307 Spinal disease Diseases 0.000 description 1
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 210000003484 anatomy Anatomy 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002612 cardiopulmonary effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000002559 palpation Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 238000012106 screening analysis Methods 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a non-invasive scoliosis screening method and a system based on a back color picture, wherein the method comprises the following steps: s1, collecting back human body RGB-D images, and storing the back human body RGB-D images as a depth image and a color image; s2, performing Mask-RCNN network training to obtain a human body segmentation model; s3, segmenting the human body to obtain a human body color image without a background; s4, training a YOLOv5 network to obtain a back recognition model; s5, intercepting the back area of the human body to obtain a back color image; s6, obtaining a human body back depth map, and calculating a maximum ATR angle value; and S7, formulating a classification standard, labeling a class label, and inputting the back color image and the class label into an EfficientNet network for training to obtain the spine classification model. The invention combines the artificial intelligence technology, the image processing technology and the biomechanics technology, and can automatically, efficiently and stably evaluate whether an input image has the risk of suffering from scoliosis or not through the screening of the scoliosis in four stages.
Description
Technical Field
The invention belongs to the technical field of image processing and artificial intelligence, and particularly relates to a non-invasive scoliosis screening method and system based on a back color image.
Background
Adolescent Idiopathic Scoliosis (AIS) is the most common spinal disease in adolescents with a global prevalence of 0.5-5.2%. The AIS, without intervention, develops before the skeleton matures, affecting the physical appearance, affecting cardiopulmonary function, and even causing paralysis. But as a chronic disease, it is usually found early in the disease, and can be effectively controlled and corrected through timely periodic review and health education. Therefore, prevention of such diseases is considered to be far more important than later treatment.
AIS is clinically commonly diagnosed by Axial Trunk Rotation (ATR) angle or by calculating Cobb angle. Although the Cobb angle is more universal in domestic AIS diagnosis, it cannot be accurately calculated from human appearance data and requires additional invasive radiographic examinations to expose the full spine features to be obtained. Therefore, in the field of AIS screening, ATR directly available from human body surfaces has become a more internationally popular and universal standard.
Currently, conventional AIS screening procedures include primary screening, outpatient screening, and instrumental screening. Common approaches to primary and outpatient screening include general examination, anteflexion testing, and scoliometer examination. General examination and anteflexion tests require the subject to expose the back, stand naturally or make a standard anteflexion position, and be diagnosed by visual observation by the examiner. Their accuracy is highly dependent on the diagnostic experience of the clinician, leading to subjectivity of the screening. And the scoliosis meter detects the ATR angle on the basis of the anterior flexion posture through the scoliosis meter. Although this is cheap, readily available and non-damaging, the complex diagnostic procedure and crude examination instrumentation make it difficult to ensure standardization of the diagnosis. Meanwhile, the diagnosis method needs to be assisted by means of palpation in the process of proceeding, which brings ethical problems. And the instrumental screening is usually specified in X-ray film examination. The method has speciality and authority in the aspect of diagnosis accuracy, can accurately calculate the Cobb angle of the spine, but has the following limitations that: firstly, the X-ray equipment can cause radiation damage to the patient, and in addition, the X-ray examination is expensive, strict requirements on the shooting environment are imposed, and professional technicians are required for operation. These drawbacks limit the usability of X-rays in routine screening.
To solve these problems, many non-invasive AIS evaluation methods have been proposed, such as by moire patterns or parallel light to show back surface morphology, but strict device usage conditions have prevented their popularization and widespread use. Another method of non-radiative injury assessment by ultrasound was introduced for imaging the shape of the spine, but requires application of a medium on the back of the human body, requires professional physician manipulation, and requires contact with the back of the human body, which is very inconvenient and inefficient for screening during covid epidemics. In addition to this, optical non-invasive surface measurement systems [9] developed based on high precision surface measurement devices enable three-dimensional reconstruction of the back or the entire torso, but this is usually very expensive. Therefore, there is an urgent need to develop a non-invasive, non-contact, low-cost method to achieve the primary screening analysis of AIS and subsequent follow-up of spinal rehabilitation.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides a noninvasive scoliosis screening method based on a back color image, which can automatically, efficiently and stably evaluate whether an input image has the risk of suffering scoliosis or not through four-stage scoliosis screening by combining an artificial intelligence technology, an image processing technology and a biomechanics technology, and provides a noninvasive scoliosis screening system.
The purpose of the invention is realized by the following technical scheme: a noninvasive scoliosis screening method based on a back color image comprises the following steps:
s1, collecting RGB-D images of human bodies on the back of people in different regions, different ages, different sexes and different scoliosis degrees, and respectively storing the images as a depth map and a color image to obtain a corresponding depth image data set and a corresponding color image data set;
s2, marking a human body region in the color image by adopting a manual marking mode, inputting the original color image and a marking file into a Mask-RCNN network for training to obtain a human body segmentation model;
s3, inputting the color image to be segmented into the human body segmentation model trained in the step S2 to segment the human body, and filling the background into black to obtain a human body color image without the background;
s4, selecting a back area between the seventh cervical vertebra and the sacrum in the human body color image in a frame mode through manual marking, training a YOLOv5 network by using the human body color image and a marking file, and obtaining a back recognition model capable of recognizing the back area between the seventh cervical vertebra and the sacrum in the human body image after training;
s5, inputting the human body color image to be recognized into the back recognition model trained in the step S4, recognizing the back area of the human body, intercepting the back area of the human body in the image, and filling other positions with black to obtain a back color image;
s6, taking the color image of the back of the human body obtained in the S5 as a template, and extracting a back area of the depth image data according to the corresponding relation between the depth image and the color image to obtain a depth image of the back of the human body; obtaining a back point cloud picture through internal and external parameters of an RGB-D camera and a human back depth picture, performing three-dimensional reconstruction on the back shape of a human body by using the back point cloud picture, then extracting medical anatomical feature points of the back of the human body, finding out the positions of spinous process points, extracting all cross sections where the positions of the spinous process points are located, and calculating a maximum ATR angle value;
and S7, formulating a classification standard and marking a class label according to the maximum ATR angle value calculated in the step S6, inputting the back color image and the class label into an EfficientNet network for training, and obtaining a spine classification model capable of judging whether the spine is normal or not after training is finished.
Further, the specific implementation method of step S2 is as follows: labeling a human body part in a point labeling format in an original color image by using Labelme software, naming a labeling area as Person, and storing coordinate information and naming of each point in a labeling file; then uploading all the original color images and the labeled files to computing equipment to train a Mask-RCNN network; the data processing process of Mask-RCNN is as follows: inputting the image into a constructed Mask-RCNN network structure, and extracting image features by using a Convolutional Neural Network (CNN); then generating N recommendation windows for each image by using FPN; mapping the suggestion window to the last layer convolution feature map of the CNN; then enabling each RoI to generate a feature map with a fixed size through a RoI Align layer; and finally, classifying the human body and the background by utilizing full connection, and returning the position of the marking frame.
Further, the specific implementation method of step S4 is as follows: in a human body color image, a Back area between the seventh cervical vertebra and the sacrum is tightly framed by a rectangular frame by using LabelImg software, the rectangular frame is named as Back, and vertex coordinate information and the name of a labeling frame are stored in a labeling file with a specific format; then uploading all the human body color images and the labeled files to a computing device to train a YOLOv5 network; the data processing flow of YOLOv5 is as follows: inputting the image into a built YOLOv5 network structure, and extracting image features through a CSPDarknet53 structure and a Focus structure; then, an SPP module and an FPN + PAN module are used in the neutral network to further improve the diversity and robustness of the characteristics; and finally, outputting the marking frame through regression, and outputting the category of the target framed by the marking frame through classification.
Further, in step S6, a specific implementation method for calculating the maximum ATR angle value includes:
s61, calculating the average curvature k1 and Gaussian curvature k2 of the back of the human body based on the three-dimensional point cloud image of the back of the human body, and summarizing the characteristics of the back of the human body into a paraboloid, a concave-convex surface and a saddle surface according to curvature information: the paraboloid is formed when k1 is 0 or k2 is 0; k2 is convex when < 0; k1 is concave when > 0; k2 is more than 0 and is more than k1, the saddle surface is a saddle surface; then marking and positioning the carina, the sacrum, the left and right iliac posterior superior spines and the spinous process line by combining the position and the characteristics of the medical anatomical point of the back of the human body; the labeling rule is: the carina is on the convex surface of the cervical vertebra part, namely the place where k2 is greater than 0; the sacral point is located at the lowest concavity of the human hip, where k1> 0; the left and right posterior superior iliac spines are positioned at the concave position above the hip, namely k1 is greater than 0; the spinous process point is the position with the minimum curvature difference between the left and the right of the spinal point on the cross section of each spinal point on the back, and a connection line formed by the spinous process points forms a spinous process line;
s62, extracting a three-dimensional human back cross-section curve corresponding to the spinous point of 18 spinal points between the carina and the L5 lumbar vertebra; on a cross section curve, 20 units of left and right target points are respectively extracted by taking the spinous process point position as the center, three-dimensional position information of the left and right target points is recorded, the rotation angle of the back of a human body at the position of the vertebra is calculated according to the relative position of the left and right target points, the rotation angles of 18 vertebrae on the cross section are respectively calculated, and the maximum rotation angle is defined as the maximum ATR angle value.
Further, in step S7, the classification criteria include the following two schemes:
in the first scheme, according to the maximum ATR angle value, the back color images are divided into two types: when the maximum ATR angle value is less than 5 degrees, the spine of the current back image is considered to have no abnormality or slight posture abnormality, and only regular review and health education are needed; when the maximum ATR angle value is greater than or equal to 5 degrees, the suspected scoliosis of the back image is considered, and further outpatient screening and instrument screening are needed to be carried out, and intervention and treatment are carried out in time; according to the classification scheme, the back color image with the maximum ATR angle value less than 5 degrees is marked as normal, and the label value is 0; the back color image with the maximum ATR angle value larger than or equal to 5 degrees is marked as abnormal, and the label value is 1;
and according to the maximum ATR angle value, dividing the back color images into three categories: when the maximum ATR angle value is less than or equal to 4 degrees, the spine of the current back image is considered to be basically normal, and a good habit is kept; when the maximum ATR angle value is between 4 degrees and 7 degrees, the current back image spine is considered to have a certain risk of lateral bending, but the risk degree is low, and the disease development process needs to be observed and monitored further; when the maximum ATR angle value is larger than 7 degrees, the current back image spine is considered to have higher risk of lateral bending, and professional medical measures are required to be taken for treatment in time; according to this classification scheme, the back color image with a maximum ATR angle value less than or equal to 4 degrees is labeled as normal with a label value of 0; the back color image with the maximum ATR angle value between 4 degrees and 7 degrees is marked as low risk of scoliosis, and the label value is 1; marking the back color image with the maximum ATR angle value larger than 7 degrees as the high risk of scoliosis, and the label value is 2;
EfficientNet: inputting the image into a well-built EfficientNet network structure, extracting image characteristics through CNN, and then outputting the image category through a classification layer.
Another object of the present invention is to provide a non-invasive scoliosis screening system based on a back color image, comprising the following modules:
an image acquisition module: collecting original color data of back human body images in different regions, different ages, different sexes and different lateral bending degrees, and respectively storing the original color data as a depth map and a color image to obtain a depth image data set and a color image data set which correspond to each other one by one;
a human body segmentation module: the system comprises a Mask-RCNN network, a human body segmentation model and a human body segmentation model, wherein the Mask-RCNN network is used for training a Mask-RCNN network by using an annotated original color image to obtain a human body segmentation model;
human back recognition module: the method comprises the steps of training a YOLOv5 network by using a labeled human body color image to obtain a back recognition model;
a spine assessment module: and the spine classification model is obtained by training the EfficientNet network by using the labeled back color image according to the classification standard.
Specifically, the non-invasive scoliosis screening system further comprises the following modules:
an original data labeling module: the system is used for manually marking the collected original color image, adopting pixel-level marking during marking, and marking out a human body and a background along the contour of the human body;
a first training data saving module: the system is used for storing an original color image which is artificially marked out of a human body as first training data;
the human body color data labeling module: the marking frame and the name are arranged at the back position between the seventh cervical vertebra and the sacrum on the image during marking;
the second training data storage module: the human body color image after being manually marked out of the back is used as second training data to be stored;
the classification label storage module: for storing the image name, the calculated maximum ATR angle value and the value of the classification label under different schemes in an Excel file.
Specifically, the non-invasive scoliosis screening system based on the back color image further comprises the following modules:
Mask-RCNN network training module: the system comprises a master-RCNN network, a first training data acquisition unit and a second training data acquisition unit, wherein the master-RCNN network is used for training a Mask-RCNN network through the first training data;
YOLO v5 network training module: for training the YOLO v5 network with second training data;
maximum ATR angle value calculation module: calculating a maximum ATR angle value through the depth map information of the back of the human body;
an EffcientNet network training module: the system is used for training the EfficientNet network through the data in the classification label storage module.
The invention has the beneficial effects that: the invention provides a noninvasive scoliosis screening method and system based on a back color image by combining an artificial intelligence technology, an image processing technology and a biomechanics technology, wherein a scoliosis screening task is completed by four stages, the background of an original color image is removed in the first stage, and the color image only comprising a human body region is reserved; the second stage identifies the back area of the human body in the color image of the human body; in the third stage, automatically calculating a back maximum ATR angle value as a label truth value based on the three-dimensional point cloud image; the fourth stage assesses scoliosis. Namely, the invention can automatically, efficiently and stably evaluate whether an input image has the risk of suffering from the scoliosis or not through the four-stage scoliosis screening. Therefore, the invention can realize the preliminary screening of the scoliosis diseased condition in a non-invasive, non-contact and low-cost way, completely avoid the radiation injury, effectively solve the ethical problem and greatly reduce the examination cost of the patient.
Drawings
FIG. 1 is a flow chart of a non-invasive scoliosis screening method based on a back color image of the present invention;
fig. 2 is a diagram illustrating an example of the original color image and the image mask obtained by inputting the human body segmentation model and the result of the human body color image after the segmentation in step S3;
fig. 3 is a diagram of the recognition result of the back of the human body and an example of a back color image in which only the back is left at step S5;
FIG. 4 is an exemplary graph of images of different classes under two classification criteria.
Detailed Description
ATR angle and back information are highly relevant and the currently predominant pre-screening approach is screening by ATR. Therefore, a convenient, rapid and noninvasive AIS preliminary screening method is developed based on RGB images of the back surface of a human body, people can finish preliminary assessment work of scoliosis risks to teenagers at home through common equipment such as mobile phones, and social medical expenditure and scoliosis morbidity are reduced. The technical scheme of the invention is further explained by combining the attached drawings.
As shown in fig. 1, the non-invasive scoliosis screening method based on a back color image of the present invention includes the following steps:
s1, collecting RGB-D images of human bodies on the back of people in different regions, different ages, different sexes and different scoliosis degrees, and respectively storing the images as a depth map and a color image to obtain a corresponding depth image data set and a corresponding color image data set;
the essence of an RGB-D image is the combination of two images: one is a color image with three color channels red (R), green (G) and blue (B) (i.e. the RGB part in RGB-D) and one is a single channel Depth image (i.e. the D part in RGB-D) that only records the actual distance (Depth) of the sensor from the object. Wherein the color image can describe the appearance, color and texture information of the object, and the depth image can describe the shape, scale and geometric space information of the object. The RGB-D image can be separated into a color image and a depth image, and the color image and the depth image are stored respectively.
The method comprises the steps of using a non-invasive and non-contact type camera device, such as a Kinect sensor, to shoot back human body RGB-D images, wherein the shooting is required to be conducted from the right back of a subject, no clothing is covered, and medical anatomical marks on the back of the human body can be completely exposed. The shot images are saved on a local computer, and the depth map and the color image are required to be separated when the shot images are saved, but the shot images are required to have correlation when the shot images are named, so that the depth map and the color image can be in one-to-one correspondence.
S2, marking the human body area in the color image by adopting a manual marking mode, inputting the original color image and the marking file into a Mask-RCNN network for training to obtain a human body segmentation model capable of segmenting the human body in the image;
the specific implementation method comprises the following steps: labeling a human body part in a point labeling format in an original color image by using Labelme software, naming a labeling area as Person, and storing coordinate information and naming of each point in a labeling file; then uploading all the original color images and the labeled files to computing equipment to train a Mask-RCNN network; the data processing process of Mask-RCNN is as follows: inputting the image into a constructed Mask-RCNN network structure, and extracting image features by using a Convolutional Neural Network (CNN); then generating N recommendation windows for each image by using FPN; mapping the suggestion window to the last layer convolution feature map of the CNN; then enabling each RoI to generate a feature map with a fixed size through a RoI Align layer; and finally, classifying the human body and the background by utilizing full connection, and returning the position of the marking frame.
S3, inputting the color image to be segmented into the human body segmentation model trained in the step S2 to segment the human body, and filling the background into black to obtain a human body color image without the background;
all original color images are input into the trained human body segmentation model, and then the mask of the image can be output, wherein the human body area is white, the corresponding R, G, B values are 255, the background area is black, and the corresponding R, G, B values are 0. And comparing the value of each pixel in the mask with the value of the pixel at the corresponding position of the original color image, and taking the smaller value to obtain the human body color image with black background filling. Fig. 2 shows an example of an image mask obtained in the present embodiment and a result obtained after segmenting a human body.
S4, selecting a back area between the seventh cervical vertebra and the sacrum in the human body color image in a frame mode through manual marking, training a YOLOv5 network by using the human body color image and a marking file, and obtaining a back recognition model capable of recognizing the back area between the seventh cervical vertebra and the sacrum in the human body image after training;
the specific implementation method comprises the following steps: in a human body color image, a Back area between the seventh cervical vertebra and the sacrum is tightly framed by a rectangular frame by using LabelImg software, the rectangular frame is named as Back, and vertex coordinate information and the name of a labeling frame are stored in a labeling file with a specific format; then uploading all the human body color images and the labeled files to a computing device to train a YOLOv5 network; the data processing flow of YOLOv5 is as follows: inputting the image into a built YOLOv5 network structure, and extracting image features through a CSPDarknet53 structure and a Focus structure; then, an SPP module and an FPN + PAN module are used in the neutral network to further improve the diversity and robustness of the characteristics; and finally, outputting the marking frame through regression, and outputting the category (namely the back area of the human body) of the target framed by the marking frame through classification.
S5, inputting the human body color image to be recognized into the back recognition model trained in the step S4, recognizing the back area of the human body, intercepting the back area of the human body in the image, and filling other positions with black to obtain a back color image;
inputting all human body color images into the trained back recognition model, and outputting the image marked with the back, the confidence coefficient that the target framed by the marking frame is regarded as the back of the human body, and a text document named by the image name and written in the coordinates of the center point of the marking frame and the width and height of the frame. And calculating values of an upper vertical coordinate and a lower vertical coordinate of the labeling frame according to the document information, filling pixels smaller than the lower vertical coordinate and pixels larger than the upper vertical coordinate into black, and obtaining a back color image only retaining the back. Fig. 3 shows an example of a back recognition result map obtained in this embodiment and a back color image in which only the back is left.
S6, taking the color image of the back of the human body obtained in the S5 as a template, and extracting a back area of the depth image data according to the corresponding relation between the depth image and the color image to obtain a depth image of the back of the human body; obtaining a back point cloud picture through internal and external parameters of an RGB-D camera and a human back depth picture, performing three-dimensional reconstruction on the back shape of a human body by using the back point cloud picture, then extracting medical anatomical feature points of the back of the human body, finding out the positions of spinous process points, extracting all cross sections where the positions of the spinous process points are located, and calculating a maximum ATR angle value; medically, the spinous process refers to a portion of the spinal anatomy. The spine is a section of irregular bone, the round structure with thicker front part is a vertebral body, the vertebral canal is arranged at the rear part of the vertebral body, a bony structure which protrudes towards the rear part is arranged at the rear part of the vertebral canal and is called a spinous process, and the bony structures protruding at the two sides of the vertebral canal like the tips of bones are transverse processes. The spinous process is therefore a convex posterior bony prominence of the entire spine, at the posterior-most aspect of the spine.
The specific implementation method for calculating the maximum ATR angle value comprises the following steps:
s61, calculating the average curvature k1 and Gaussian curvature k2 of the back of the human body based on the three-dimensional point cloud image of the back of the human body, and summarizing the characteristics of the back of the human body into a paraboloid, a concave-convex surface and a saddle surface according to curvature information: the paraboloid is formed when k1 is 0 or k2 is 0; k2 is convex when < 0; k1 is concave when > 0; k2 is more than 0 and is more than k1, the saddle surface is a saddle surface; then marking and positioning the spine (cervical vertebra No. 7C 7), sacrum, left and right iliac posterior superior spine and spinous process line (line composed of spinous process point of each vertebra) by combining the position and characteristics of medical anatomical point of human back; the labeling rule is: the carina is on the convex surface of the cervical vertebra part, namely the place where k2 is greater than 0; the sacral point is located at the lowest concavity of the human hip, where k1> 0; the left and right posterior superior iliac spines are positioned at the concave position above the hip, namely k1 is greater than 0; the spinous process point is the position with the minimum curvature difference between the left and the right of the spinal point on the cross section of each spinal point on the back, and a connection line formed by the spinous process points forms a spinous process line;
s62, extracting a three-dimensional human back cross-section curve corresponding to the spinous point of 18 spinal points between the carina and the L5 lumbar vertebra; on a cross section curve, 20 units of left and right target points are respectively extracted by taking the spinous process point position as the center, three-dimensional position information of the left and right target points is recorded, the rotation angle of the back of a human body at the position of the vertebra is calculated according to the relative position of the left and right target points, the rotation angles of 18 vertebrae on the cross section are respectively calculated, and the maximum rotation angle is defined as the maximum ATR angle value.
And S7, formulating a classification standard and marking a class label according to the maximum ATR angle value calculated in the step S6, inputting the back color image and the class label into an EfficientNet network for training, and obtaining a spine classification model capable of judging whether the spine is normal or not after training is finished.
Specifically, there are two schemes for classifying the images, and a schematic diagram of the images in different categories is shown in fig. 4.
In the first scheme, according to the maximum ATR angle value, the back color images are divided into two types: when the maximum ATR angle value is less than 5 degrees, the spine of the current back image is considered to have no abnormality or slight posture abnormality, and only regular review and health education are needed; when the maximum ATR angle value is greater than or equal to 5 degrees, the suspected scoliosis of the back image is considered, and further outpatient screening and instrument screening are needed to be carried out, and intervention and treatment are carried out in time; according to the classification scheme, the back color image with the maximum ATR angle value less than 5 degrees is marked as normal, and the label value is 0; the back color image with the maximum ATR angle value larger than or equal to 5 degrees is marked as abnormal, and the label value is 1;
and according to the maximum ATR angle value, dividing the back color images into three categories: when the maximum ATR angle value is less than or equal to 4 degrees, the spine of the current back image is considered to be basically normal, and a good habit is kept; when the maximum ATR angle value is between 4 degrees and 7 degrees, the current back image spine is considered to have a certain risk of lateral bending, but the risk degree is low, and the disease development process needs to be observed and monitored further; when the maximum ATR angle value is larger than 7 degrees, the current back image spine is considered to have higher risk of lateral bending, and professional medical measures are required to be taken for treatment in time; according to this classification scheme, the back color image with a maximum ATR angle value less than or equal to 4 degrees is labeled as normal with a label value of 0; the back color image with the maximum ATR angle value between 4 degrees and 7 degrees is marked as low risk of scoliosis, and the label value is 1; marking the back color image with the maximum ATR angle value larger than 7 degrees as the high risk of scoliosis, and the label value is 2;
EfficientNet: inputting the image into a well-built EfficientNet network structure, extracting image features through CNN, and then outputting the image category through a classification layer (namely, judging whether the image is normal or abnormal).
According to this classification scheme, the back color image with a maximum ATR angle value less than or equal to 4 degrees is labeled as normal with a label value of 0; the back color image with the maximum ATR angle value between 4 degrees and 7 degrees is marked as low risk of scoliosis, and the label value is 1; the back color image with a maximum ATR angle value greater than 7 degrees is labeled as high risk of scoliosis with a label value of 2.
Through the trained EfficientNet network, the risk degree of scoliosis can be judged for a back color image. In this embodiment, the back color image labeled with the lateral curvature risk degree of the spine based on the maximum ATR angle value is used as training data of the spine assessment module, and the image data is uploaded to a computing device with high-performance computing capability for training. The deep learning algorithm used for training is an Efficientnet network, and a model for spinal column assessment is obtained through training. By using a back color image as an input of the model, the risk degree of lateral bending of the spine in the image can be rapidly reported.
The invention discloses a non-invasive scoliosis screening system based on a back color image, which comprises the following modules:
an image acquisition module: collecting original color data of back human body images in different regions, different ages, different sexes and different lateral bending degrees, and respectively storing the original color data as a depth map and a color image to obtain a depth image data set and a color image data set which correspond to each other one by one;
a human body segmentation module: the system comprises a Mask-RCNN network, a human body segmentation model and a human body segmentation model, wherein the Mask-RCNN network is used for training a Mask-RCNN network by using an annotated original color image to obtain a human body segmentation model; the human body segmentation model is used for carrying out human body segmentation on an original color image to be segmented to obtain a background-free human body color image and storing the background-free human body color image;
human back recognition module: the method comprises the steps of training a YOLOv5 network by using a labeled human body color image to obtain a back recognition model; the back recognition model is used for carrying out back recognition on the human body color image to be recognized, reserving the recognized back, obtaining a back color image and storing the back color image;
a spine assessment module: the spine classification model is used for carrying out spine classification on the input back color images so as to evaluate the occurrence condition of scoliosis;
an original data labeling module: the system is used for manually marking the collected original color image, adopting pixel-level marking during marking, and marking out a human body and a background along the contour of the human body;
a first training data saving module: the system is used for storing an original color image which is artificially marked out of a human body as first training data;
the human body color data labeling module: the marking frame and the name are arranged at the back position between the seventh cervical vertebra and the sacrum on the image during marking;
the second training data storage module: the human body color image after being manually marked out of the back is used as second training data to be stored;
the classification label storage module: the method is used for storing the image name, the calculated maximum ATR angle value and the values of the classification labels under different schemes in an Excel file;
Mask-RCNN network training module: the system comprises a master-RCNN network, a first training data acquisition unit and a second training data acquisition unit, wherein the master-RCNN network is used for training a Mask-RCNN network through the first training data;
YOLO v5 network training module: for training the YOLO v5 network with second training data;
maximum ATR angle value calculation module: calculating a maximum ATR angle value through the depth map information of the back of the human body;
an EffcientNet network training module: the system is used for training the EfficientNet network through the data in the classification label storage module.
It will be appreciated by those of ordinary skill in the art that the embodiments described herein are intended to assist the reader in understanding the principles of the invention and are to be construed as being without limitation to such specifically recited embodiments and examples. Those skilled in the art can make various other specific changes and combinations based on the teachings of the present invention without departing from the spirit of the invention, and these changes and combinations are within the scope of the invention.
Claims (8)
1. A noninvasive scoliosis screening method based on a back color image is characterized by comprising the following steps:
s1, collecting RGB-D images of human bodies on the back of people in different regions, different ages, different sexes and different scoliosis degrees, and respectively storing the images as a depth map and a color image to obtain a corresponding depth image data set and a corresponding color image data set;
s2, marking a human body region in the color image by adopting a manual marking mode, inputting the original color image and a marking file into a Mask-RCNN network for training to obtain a human body segmentation model;
s3, inputting the color image to be segmented into the human body segmentation model trained in the step S2 to segment the human body, and filling the background into black to obtain a human body color image without the background;
s4, selecting a back area between the seventh cervical vertebra and the sacrum in the human body color image in a frame mode through manual marking, training a YOLOv5 network by using the human body color image and a marking file, and obtaining a back recognition model capable of recognizing the back area between the seventh cervical vertebra and the sacrum in the human body image after training;
s5, inputting the human body color image to be recognized into the back recognition model trained in the step S4, recognizing the back area of the human body, intercepting the back area of the human body in the image, and filling other positions with black to obtain a back color image;
s6, taking the color image of the back of the human body obtained in the S5 as a template, and extracting a back area of the depth image data according to the corresponding relation between the depth image and the color image to obtain a depth image of the back of the human body; obtaining a back point cloud picture through internal and external parameters of an RGB-D camera and a human back depth picture, performing three-dimensional reconstruction on the back shape of a human body by using the back point cloud picture, then extracting medical anatomical feature points of the back of the human body, finding out the positions of spinous process points, extracting all cross sections where the positions of the spinous process points are located, and calculating a maximum ATR angle value;
and S7, formulating a classification standard and marking a class label according to the maximum ATR angle value calculated in the step S6, inputting the back color image and the class label into an EfficientNet network for training, and obtaining a spine classification model capable of judging whether the spine is normal or not after training is finished.
2. The method for non-invasive scoliosis screening based on back color images according to claim 1, wherein the step S2 is implemented by: labeling a human body part in a point labeling format in an original color image by using Labelme software, naming a labeling area as Person, and storing coordinate information and naming of each point in a labeling file; then uploading all the original color images and the labeled files to computing equipment to train a Mask-RCNN network; the data processing process of Mask-RCNN is as follows: inputting the image into a constructed Mask-RCNN network structure, and extracting image features by using a Convolutional Neural Network (CNN); then generating N suggested windows for each image by using RPN; mapping the suggestion window to the last layer convolution feature map of the CNN; then enabling each RoI to generate a feature map with a fixed size through a RoI Align layer; and finally, classifying the human body and the background by utilizing full connection, and returning the position of the marking frame.
3. The method for non-invasive scoliosis screening based on back color images according to claim 1, wherein the step S4 is implemented by: in a human body color image, a Back area between the seventh cervical vertebra and the sacrum is tightly framed by a rectangular frame by using LabelImg software, the rectangular frame is named as Back, and vertex coordinate information and the name of a labeling frame are stored in a labeling file with a specific format; then uploading all the human body color images and the labeled files to a computing device to train a YOLOv5 network; the data processing flow of YOLOv5 is as follows: inputting the image into a built YOLOv5 network structure, and extracting image features through a CSPDarknet53 structure and a Focus structure; then, an SPP module and an FPN + PAN module are used in the neutral network to further improve the diversity and robustness of the characteristics; and finally, outputting the marking frame through regression, and outputting the category of the target framed by the marking frame through classification.
4. The method for non-invasive scoliosis screening based on back color images according to claim 1, wherein the step S6 is implemented by calculating the maximum ATR angle value:
s61, calculating the average curvature k1 and Gaussian curvature k2 of the back of the human body based on the three-dimensional point cloud image of the back of the human body, and summarizing the characteristics of the back of the human body into a paraboloid, a concave-convex surface and a saddle surface according to curvature information: the paraboloid is formed when k1 is 0 or k2 is 0; k2 is convex when < 0; k1 is concave when > 0; k2 is more than 0 and is more than k1, the saddle surface is a saddle surface; then marking and positioning the carina, the sacrum, the left and right iliac posterior superior spines and the spinous process line by combining the position and the characteristics of the medical anatomical point of the back of the human body; the labeling rule is: the carina is on the convex surface of the cervical vertebra part, namely the place where k2 is greater than 0; the sacral point is located at the lowest concavity of the human hip, where k1> 0; the left and right posterior superior iliac spines are positioned at the concave position above the hip, namely k1 is greater than 0; the spinous process point is the position with the minimum curvature difference between the left and the right of the spinal point on the cross section of each spinal point on the back, and a connection line formed by the spinous process points forms a spinous process line;
s62, extracting a three-dimensional human back cross-section curve corresponding to the spinous point of 18 spinal points between the carina and the L5 lumbar vertebra; on a cross section curve, 20 units of left and right target points are respectively extracted by taking the spinous process point position as the center, three-dimensional position information of the left and right target points is recorded, the rotation angle of the back of a human body at the position of the vertebra is calculated according to the relative position of the left and right target points, the rotation angles of 18 vertebrae on the cross section are respectively calculated, and the maximum rotation angle is defined as the maximum ATR angle value.
5. The method for non-invasive scoliosis screening based on back color images according to claim 1, wherein in the step S7, the classification criteria include the following two schemes:
in the first scheme, according to the maximum ATR angle value, the back color images are divided into two types: when the maximum ATR angle value is less than 5 degrees, the spine of the current back image is considered to have no abnormality or slight posture abnormality, and only regular review and health education are needed; when the maximum ATR angle value is greater than or equal to 5 degrees, the suspected scoliosis of the back image is considered, and further outpatient screening and instrument screening are needed to be carried out, and intervention and treatment are carried out in time; according to the classification scheme, the back color image with the maximum ATR angle value less than 5 degrees is marked as normal, and the label value is 0; the back color image with the maximum ATR angle value larger than or equal to 5 degrees is marked as abnormal, and the label value is 1;
and according to the maximum ATR angle value, dividing the back color images into three categories: when the maximum ATR angle value is less than or equal to 4 degrees, the spine of the current back image is considered to be basically normal, and a good habit is kept; when the maximum ATR angle value is between 4 degrees and 7 degrees, the current back image spine is considered to have a certain risk of lateral bending, but the risk degree is low, and the disease development process needs to be observed and monitored further; when the maximum ATR angle value is larger than 7 degrees, the current back image spine is considered to have higher risk of lateral bending, and professional medical measures are required to be taken for treatment in time; according to this classification scheme, the back color image with a maximum ATR angle value less than or equal to 4 degrees is labeled as normal with a label value of 0; the back color image with the maximum ATR angle value between 4 degrees and 7 degrees is marked as low risk of scoliosis, and the label value is 1; marking the back color image with the maximum ATR angle value larger than 7 degrees as the high risk of scoliosis, and the label value is 2;
EfficientNet: inputting the image into a well-built EfficientNet network structure, extracting image characteristics through CNN, and then outputting the image category through a classification layer.
6. The system for noninvasive scoliosis screening based on a back color image as claimed in any one of claims 1 to 5, characterized by comprising the following modules:
an image acquisition module: collecting original color data of back human body images in different regions, different ages, different sexes and different lateral bending degrees, and respectively storing the original color data as a depth map and a color image to obtain a depth image data set and a color image data set which correspond to each other one by one;
a human body segmentation module: the system comprises a Mask-RCNN network, a human body segmentation model and a human body segmentation model, wherein the Mask-RCNN network is used for training a Mask-RCNN network by using an annotated original color image to obtain a human body segmentation model;
human back recognition module: the method comprises the steps of training a YOLOv5 network by using a labeled human body color image to obtain a back recognition model;
a spine assessment module: and the spine classification model is obtained by training the EfficientNet network by using the labeled back color image according to the classification standard.
7. The back color image-based noninvasive scoliosis screening system of claim 6, further comprising the following modules:
an original data labeling module: the system is used for manually marking the collected original color image, adopting pixel-level marking during marking, and marking out a human body and a background along the contour of the human body;
a first training data saving module: the system is used for storing an original color image which is artificially marked out of a human body as first training data;
the human body color data labeling module: the marking frame and the name are arranged at the back position between the seventh cervical vertebra and the sacrum on the image during marking;
the second training data storage module: the human body color image after being manually marked out of the back is used as second training data to be stored;
the classification label storage module: for storing the image name, the calculated maximum ATR angle value and the value of the classification label under different schemes in an Excel file.
8. The back color image-based noninvasive scoliosis screening system of claim 7, further comprising the following modules:
Mask-RCNN network training module: the system comprises a master-RCNN network, a first training data acquisition unit and a second training data acquisition unit, wherein the master-RCNN network is used for training a Mask-RCNN network through the first training data;
YOLO v5 network training module: for training the YOLOv5 network with second training data;
maximum ATR angle value calculation module: calculating a maximum ATR angle value through the depth map information of the back of the human body;
an EffcientNet network training module: the system is used for training the EfficientNet network through the data in the classification label storage module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111629135.0A CN114287915B (en) | 2021-12-28 | 2021-12-28 | Noninvasive scoliosis screening method and system based on back color images |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111629135.0A CN114287915B (en) | 2021-12-28 | 2021-12-28 | Noninvasive scoliosis screening method and system based on back color images |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114287915A true CN114287915A (en) | 2022-04-08 |
| CN114287915B CN114287915B (en) | 2024-03-05 |
Family
ID=80971076
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111629135.0A Active CN114287915B (en) | 2021-12-28 | 2021-12-28 | Noninvasive scoliosis screening method and system based on back color images |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114287915B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115886787A (en) * | 2023-03-09 | 2023-04-04 | 深圳市第二人民医院(深圳市转化医学研究院) | Ground reaction force transformation method for disease screening, bone disease screening system and equipment |
| WO2024033897A1 (en) * | 2022-08-11 | 2024-02-15 | Momentum Health Inc. | Method and system for determining spinal curvature |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110458831A (en) * | 2019-08-12 | 2019-11-15 | 深圳市智影医疗科技有限公司 | A kind of scoliosis image processing method based on deep learning |
| CN111666890A (en) * | 2020-06-08 | 2020-09-15 | 平安科技（深圳）有限公司 | Spine deformation crowd identification method and device, computer equipment and storage medium |
| CN112258516A (en) * | 2020-09-04 | 2021-01-22 | 温州医科大学附属第二医院、温州医科大学附属育英儿童医院 | Method for generating scoliosis image detection model |
| CN112381757A (en) * | 2020-10-09 | 2021-02-19 | 温州医科大学附属第二医院、温州医科大学附属育英儿童医院 | System and method for measuring and calculating scoliosis Cobb angle through full-length X-ray film of spine based on artificial intelligence-image recognition |
| CN112734757A (en) * | 2021-03-29 | 2021-04-30 | 成都成电金盘健康数据技术有限公司 | Spine X-ray image cobb angle measuring method |
| CN113397485A (en) * | 2021-05-27 | 2021-09-17 | 上海交通大学医学院附属新华医院 | Scoliosis screening method based on deep learning |
-
2021
- 2021-12-28 CN CN202111629135.0A patent/CN114287915B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110458831A (en) * | 2019-08-12 | 2019-11-15 | 深圳市智影医疗科技有限公司 | A kind of scoliosis image processing method based on deep learning |
| CN111666890A (en) * | 2020-06-08 | 2020-09-15 | 平安科技（深圳）有限公司 | Spine deformation crowd identification method and device, computer equipment and storage medium |
| CN112258516A (en) * | 2020-09-04 | 2021-01-22 | 温州医科大学附属第二医院、温州医科大学附属育英儿童医院 | Method for generating scoliosis image detection model |
| CN112381757A (en) * | 2020-10-09 | 2021-02-19 | 温州医科大学附属第二医院、温州医科大学附属育英儿童医院 | System and method for measuring and calculating scoliosis Cobb angle through full-length X-ray film of spine based on artificial intelligence-image recognition |
| CN112734757A (en) * | 2021-03-29 | 2021-04-30 | 成都成电金盘健康数据技术有限公司 | Spine X-ray image cobb angle measuring method |
| CN113397485A (en) * | 2021-05-27 | 2021-09-17 | 上海交通大学医学院附属新华医院 | Scoliosis screening method based on deep learning |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024033897A1 (en) * | 2022-08-11 | 2024-02-15 | Momentum Health Inc. | Method and system for determining spinal curvature |
| CN115886787A (en) * | 2023-03-09 | 2023-04-04 | 深圳市第二人民医院(深圳市转化医学研究院) | Ground reaction force transformation method for disease screening, bone disease screening system and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114287915B (en) | 2024-03-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5603859B2 (en) | Method for controlling an analysis system that automatically analyzes a digitized image of a side view of a target spine | |
| CN108320288B (en) | Data processing method for idiopathic scoliosis image | |
| US10872408B2 (en) | Method and system for imaging and analysis of anatomical features | |
| CN112258516B (en) | Method for generating scoliosis image detection model | |
| CN108309334B (en) | Data processing method of spine X-ray image | |
| CN112734757B (en) | Spine X-ray image cobb angle measuring method | |
| CN111047572A (en) | Automatic spine positioning method in medical image based on Mask RCNN | |
| CN114287915B (en) | Noninvasive scoliosis screening method and system based on back color images | |
| Korez et al. | A deep learning tool for fully automated measurements of sagittal spinopelvic balance from X-ray images: performance evaluation | |
| Yang et al. | Automatic 3-D imaging and measurement of human spines with a robotic ultrasound system | |
| CN110223279B (en) | Image processing method and device and electronic equipment | |
| WO2020259600A1 (en) | A device, process and system for diagnosing and tracking of the development of the spinal alignment of a person | |
| CN112274164B (en) | Scoliosis prediction method, scoliosis prediction device, electronic device, and storage medium | |
| CHOI et al. | CNN-based spine and Cobb angle estimator using moire images | |
| CN112349391A (en) | Optimized rib automatic labeling method | |
| JP3234668U (en) | Image recognition system for scoliosis by X-ray | |
| CN115187606A (en) | Juvenile idiopathic scoliosis PUMC typing method | |
| Dempere-Marco et al. | Analysis of visual search patterns with EMD metric in normalized anatomical space | |
| CN115602320B (en) | Difficult airway assessment method and system | |
| CN116758087A (en) | Lumbar vertebra CT bone window side recess gap detection method and device | |
| US20230169644A1 (en) | Computer vision system and method for assessing orthopedic spine condition | |
| TWI759946B (en) | Spine Measurement and Status Assessment Methods | |
| Gabriel et al. | Development and clinical application of Vertebral Metrics: using a stereo vision system to assess the spine | |
| CHOI et al. | Measurement of vertebral rotation from moire image for screening of adolescent idiopathic scoliosis | |
| Subsol et al. | 3d image processing for the study of the evolution of the shape of the human skull: presentation of the tools and preliminary results |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |