CN113838027A

CN113838027A - Method and system for obtaining target image element based on image processing

Info

Publication number: CN113838027A
Application number: CN202111114031.6A
Authority: CN
Inventors: 刘金勇; 黄志俊; 钱坤; 张胜
Original assignee: Lancet Robotics Co Ltd
Current assignee: Lancet Robotics Co Ltd
Priority date: 2021-09-23
Filing date: 2021-09-23
Publication date: 2021-12-24

Abstract

The invention discloses a method and a system for acquiring target image elements based on image processing, wherein the method comprises the following steps: acquiring a to-be-processed image group associated with a target object, and performing data preprocessing on the to-be-processed image group to obtain a preprocessed image group; performing image recognition on the preprocessed image group to determine a plurality of image files, and performing image processing on the plurality of image files by using a neural network to obtain a plurality of target image areas; performing image filling on each target image area based on the structural features of the target object to obtain a mask image associated with each target image area; and performing a masking operation on the associated target image area based on each mask image, thereby obtaining a plurality of target image elements associated with the target object.

Description

Method and system for obtaining target image element based on image processing

Technical Field

The present invention relates to the field of image data processing technology, and more particularly, to a method and system for acquiring a target image element based on image processing, and a storage medium and an electronic device. The invention also relates to a system and a method for processing the image data for auxiliary diagnosis based on 3D deep learning.

Background

With the acceleration of computer data processing capabilities and the maturation of artificial intelligence techniques, physicians are increasingly using digital images for auxiliary diagnostic treatments. X-ray examination is a traditional method for effectively screening diseases, but because of the limitations of medical imaging technology and the specificity of the imaged object, and the inherent characteristics of medical images, such as uneven gray scale, overlapping images, high noise and blurred boundaries, the segmentation of medical images is challenging.

At present, the segmentation of the bone tissue of an X-ray film is mainly marked by hands of doctors, so that the time and the labor are consumed, the segmentation result is difficult to reproduce, and the accuracy is unstable. Moreover, a large number of medical image pictures generated by a patient in a hospital need to be analyzed carefully by a doctor, corresponding image data is obtained after the device scans a human body, and then the image data is printed on a film or output to a display screen, and the general processing method is that the doctor observes with naked eyes. On the one hand, the repetitive work is a tedious and difficult task for medical staff in an imaging department, and the domestic huge patient requirements are difficult to meet; on the other hand, the diagnosis result mainly judged by the naked eyes of the doctor is influenced by certain subjective factors and abilities, and particularly in some regions or primary hospitals with limited medical level, the clinical application of the medical imaging device is limited to a certain extent.

The computer technology enables a clinician to more conveniently and rapidly check lesion tissues and structures to help the physician analyze and judge the disease condition on one hand, and on the other hand, more accurate diagnosis reports are provided for the physician and the misdiagnosis rate is reduced to be possible.

With the continuous development of computer hardware, the operation speed of a processor is greatly improved, the artificial intelligence technology makes a breakthrough in the field of computer research, particularly, deep learning is widely applied to the fields of computer vision, computational audio analysis, natural language processing and the like, and a good opportunity is provided for the development of computer-aided diagnosis and treatment. The development of medical image processing as a research hotspot at home and abroad can solve the problem of difficult task of reading digital images by doctors in clinic to a great extent, thereby reducing the burden of reading the images by the doctors, improving the diagnosis accuracy of the doctors and reducing the misdiagnosis rate, and having very important research significance.

The important problems to be solved are that medical image segmentation, analysis and diagnosis of the patient's condition by doctors and the like depend on the accuracy of the segmentation, and not only is a key link in medical image processing, but also is the foundation and bottleneck of the medical image processing problem.

The structure of human body is very complicated, and there is also very big difference between everybody, and the segmentation tissue both is a challenging work, also has very important realistic meaning. Although some algorithms have been used to solve the problem of bone tissue segmentation on X-ray films, due to limitations of medical imaging techniques and particularity of the imaged object, and also due to inherent characteristics of medical images, such as uneven gray scale, overlapping images, large noise and blurred boundaries, the segmentation of medical images is challenging.

It can be seen that there is a need in the art for a solution for acquiring target image elements based on image processing.

Disclosure of Invention

In order to solve the problems in the prior art, the invention provides a medical image recognition deep learning computer system and a method, the system firstly designs an image area automatic segmentation overall framework from a macroscopic view, then specifically designs and realizes the image area automatic segmentation overall framework in steps, the system comprises three main steps of data preprocessing, neural network training and testing and image post-processing, and Pycharm software, MITK software, MatLab software and Mimics software are combined.

According to one aspect, the present invention provides a method for acquiring a target image element based on image processing, the method comprising:

acquiring a to-be-processed image group associated with a target object, and performing data preprocessing on the to-be-processed image group to obtain a preprocessed image group;

performing image recognition on the preprocessed image group to determine a plurality of image files, and performing image processing on the plurality of image files by using a neural network to obtain a plurality of target image areas;

performing image filling on each target image area based on the structural features of the target object to obtain a mask image associated with each target image area; and

masking the associated target image area based on each mask image to obtain a plurality of target image elements associated with the target object.

Preferably, the image group to be processed includes a plurality of image files, each of which has an image area and in which a target object is contained.

Preferably, the data preprocessing the image group to be processed to obtain a preprocessed image group includes:

performing data cleaning on the image group to be processed to obtain a data-cleaned image group;

carrying out normalization processing on the image group subjected to data cleaning to obtain an image group subjected to normalization processing; and

and performing data enhancement on the image group subjected to the normalization processing, thereby obtaining a preprocessed image group.

Preferably, the data washing of the image group to be processed includes:

converting the file format of the image group to be processed into a preset file format to obtain the image group with the preset file format;

carrying out image recognition on the image group in the preset file format to determine whether an image file which does not meet the preset requirement exists in the image group in the preset file format;

and if so, deleting the image files which do not meet the preset requirements to obtain the image group subjected to data cleaning.

Preferably, the normalizing the image group subjected to data cleaning comprises the following steps:

determining a resolution of each of a plurality of image files within the data-washed image set;

acquiring a preset standard resolution, and performing edge filling on image files with the resolution lower than the standard resolution by using preset contents so as to enable the resolution of each image file of a plurality of image files in the image group subjected to data cleaning to be the standard resolution;

the resolution of each image file is adjusted from the standard resolution to the compressed resolution.

Preferably, after obtaining the image group subjected to the normalization processing, the method further includes performing binarization processing on each image file in the image group subjected to the normalization processing.

Preferably, the data enhancement of the normalized image group includes:

performing contrast adjustment, brightness adjustment and/or chromaticity adjustment on each image file in the image group subjected to the normalization processing to obtain a plurality of image files subjected to data enhancement;

and forming a preprocessed image group by the plurality of image files subjected to data enhancement.

Preferably, each image file corresponds to a target image area.

Preferably, wherein the image filling of each target image area based on the structural features of the target object, the obtaining of the mask image associated with each target image area comprises:

determining a region contour of the target object in each target image region based on the structural features of the target object;

and filling the connected regions based on the region outline of the target object in each target image region to obtain a mask image of the target object associated with each target image region.

Preferably, before the image processing of the plurality of image files by the neural network, training the neural network is further included.

Preferably, wherein training the neural network comprises:

step 1, initializing a plurality of network parameters of a neural network;

step 2, acquiring a training sample set for training the neural network, dividing training samples in the training sample set into a plurality of training sample subsets based on the set batch size, and inputting each training sample subset into the neural network;

step 3, carrying out forward propagation on the training sample subset until the last layer of output of the neural network is subjected to iterative training, and calculating the output loss of the neural network;

step 4, determining whether the iteration times are equal to a time threshold value, if not, performing step 5, performing reverse propagation through a loss function, updating the weight and the deviation value of each network parameter layer by layer, and performing step 2;

if yes, go to step 6, obtain the neural network trained.

Preferably, before training the neural network, further comprising,

acquiring a plurality of sample image groups associated with a target object, and performing data preprocessing on each sample image group to obtain a plurality of preprocessed sample image groups;

performing image recognition on each preprocessed sample image group to determine a plurality of sample image files, and labeling each sample image file with a label to obtain a plurality of sample image files with labels, wherein the label comprises a training identifier and a testing identifier;

the plurality of sample image files are divided into a training sample set and a testing sample set based on the labels.

Preferably, the data preprocessing each sample image group to obtain a plurality of preprocessed sample image groups comprises:

performing data cleaning on each sample image group to obtain a sample image group subjected to data cleaning;

normalizing each sample image group subjected to data cleaning to obtain a normalized sample image group; and

and performing data enhancement on each sample image group subjected to the normalization processing, thereby obtaining a plurality of sample image groups subjected to preprocessing.

Preferably, wherein the data washing of each sample image group comprises:

converting the file format of each sample image group into a preset file format to obtain a sample image group with the preset file format;

carrying out image recognition on the sample image group in the preset file format to determine whether a sample image file which does not meet the preset requirement exists in the sample image group in the preset file format;

and if so, deleting the sample image files which do not meet the preset requirement to obtain the sample image group subjected to data cleaning.

Preferably, the normalizing each data-washed sample image group comprises:

determining a resolution of each sample image file of a plurality of sample image files within each data-washed sample image set;

acquiring a preset standard resolution, and performing edge filling on a sample image file with the resolution lower than the standard resolution by using preset content so as to enable the resolution of each sample image file of a plurality of sample image files in the sample image group subjected to data cleaning to be the standard resolution;

the resolution of each sample image file is adjusted from the standard resolution to the compressed resolution.

Preferably, after obtaining the sample image group subjected to the normalization processing, the method further includes performing binarization processing on each sample image file in the sample image group subjected to the normalization processing.

Preferably, the data enhancement of each normalized sample image group includes:

performing contrast adjustment, brightness adjustment and/or chroma adjustment on each sample image file in the sample image group subjected to the normalization processing to obtain a plurality of sample image files subjected to data enhancement;

and forming a preprocessed sample image group by the plurality of sample image files subjected to data enhancement.

Preferably, the method further comprises testing the trained neural network with a test sample set to obtain a tested neural network.

Preferably, the neural network is configured to acquire an input image file, construct a convolution image file of a predetermined size for output, wherein the output convolution image file of a predetermined size is an image file capable of being subjected to scaling processing and is an image file having the same length and width, and wherein a parameter of a batch size of an input layer of the neural network is an integer power of 2;

taking a convolution image file as input to construct an input convolution layer, carrying out filled convolution and batch standardization BN, then activating by using a leakage correction linear unit function LeakyReLU, and repeating twice; then carrying out convolution once again, stacking the convolution with the previous input, then carrying out downward maximum pooling with the step length of 2, and outputting a first convolution result; wherein the two-dimensional convolution is a 3 x 3 two-dimensional convolution;

constructing a first convolution layer by taking the first convolution result as input, performing filled convolution and batch standardization BN, then activating by using LeakyReLU, and repeating twice; then carrying out convolution once again, stacking the input and the input, then carrying out down maximum pooling, wherein the step length is 2, and outputting a second convolution result;

constructing a second convolution layer by taking the second convolution result as input, performing filled convolution and batch standardization BN, then activating by using LeakyReLU, and repeating twice; then carrying out convolution once again, stacking the input and the input, then carrying out down maximum pooling, wherein the step length is 2, and outputting a third convolution result;

constructing a third convolution layer by taking the third convolution result as input, performing filled convolution and batch standardization BN, then activating by using LeakyReLU, and repeating twice; then carrying out convolution once again, stacking the input and the input, then carrying out down maximum pooling, wherein the step length is 2, and outputting a fourth convolution result;

constructing a fourth convolution layer by taking the fourth convolution result as input, performing filled convolution and batch standardization BN, then activating by using LeakyReLU, and repeating twice; then carrying out convolution once again, stacking the input and the input, then carrying out down maximum pooling, wherein the step length is 2, and outputting a fifth convolution result;

performing up-sampling on the fifth convolution result, performing feature fusion on the fifth convolution result and the fourth convolution result to output as a first up-sample, performing filled convolution and batch standardization BN on the first up-sample, activating by using LeakyReLU, and repeating twice; then carrying out convolution once again, stacking the convolution and the first up-sampling, and outputting a sixth convolution result;

performing up-sampling on the sixth convolution result, performing feature fusion on the sixth convolution result and the third convolution result to output a second up-sample, performing filled convolution and batch standardization BN on the second up-sample, activating by using LeakyReLU, and repeating twice; then carrying out convolution once again, stacking the convolution and the second up-sampling, and outputting a seventh convolution result;

performing up-sampling on the seventh convolution result, performing feature fusion on the seventh convolution result and the second convolution result to output a third up-sample, performing filled convolution and batch standardization BN on the third up-sample, activating by using LeakyReLU, and repeating twice; then carrying out convolution once again, stacking the convolution and the third up-sampling, and outputting an eighth convolution result;

the eighth convolution result is upsampled and then feature fused with the first convolution result to output a fourth upsampled, which is then subjected to filled convolution and batch normalization BN, and then activated using a leakyreu, which is repeated twice. Then carrying out convolution once again, stacking the convolution and the fourth up-sampling, and outputting a ninth convolution result;

and taking the ninth convolution result as input to construct an output layer, performing 1x1 convolution once, activating by using sigmoid, and outputting an image file.

According to another aspect, the present invention provides a system for acquiring a target image element based on image processing, the system comprising:

the preprocessing device is used for acquiring a to-be-processed image group associated with a target object, and performing data preprocessing on the to-be-processed image group to obtain a preprocessed image group;

the recognition device is used for carrying out image recognition on the preprocessed image group to determine a plurality of image files, and carrying out image processing on the plurality of image files by utilizing a neural network to obtain a plurality of target image areas;

filling means for performing image filling on each target image area based on the structural features of the target object to obtain a mask image associated with each target image area; and

and processing means for performing a masking operation on the associated target image area based on each mask image, thereby obtaining a plurality of target image elements associated with the target object.

According to yet another aspect, the invention provides a computer-readable storage medium, characterized in that the storage medium stores a computer program for performing any of the methods described above.

According to yet another aspect, the present invention provides an electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

the processor is configured to read the executable instructions from the memory and execute the instructions to implement any of the methods described above.

Aiming at the problems that the time consumption and the accuracy are unstable when the X-ray slice bone tissue segmentation is mainly marked by doctors manually at present, the invention uses the R.U.Net neural network which combines the advantages of a depth residual error network and a U-Net architecture and applies the R.U.Net neural network to the automatic segmentation of the femoral region, provides and realizes the automatic segmentation framework of the femoral region, realizes the full automatic segmentation of the femoral region of the X-ray slice in an end-to-end batch manner, and improves the segmentation precision and efficiency.

Drawings

FIG. 1 shows a flow diagram of a method of acquiring a target image element based on image processing according to an embodiment of the invention;

FIG. 2 shows a flow diagram of a method of automatically segmenting an image according to an embodiment of the invention;

FIG. 3 shows a schematic diagram of image screening according to an embodiment of the invention;

FIG. 4 shows a schematic diagram of a processed image according to an embodiment of the invention;

FIG. 5 shows a schematic diagram of a network infrastructure according to an embodiment of the invention;

FIG. 6 shows a schematic diagram of a neural network, according to an embodiment of the invention;

FIG. 7 shows a schematic diagram of a training process according to an embodiment of the invention;

FIG. 8 shows a schematic diagram of an original image, a mask image, and a complete femur region image according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of a system for acquiring a target image element based on image processing according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention are described in detail below with reference to the accompanying drawings. The exemplary embodiments described below and illustrated in the figures are intended to teach the principles of the present invention and enable one skilled in the art to implement and use the invention in several different environments and for several different applications. The scope of the invention is, therefore, indicated by the appended claims, and the exemplary embodiments are not intended to, and should not be considered as, limiting the scope of the invention.

Convolutional networks have existed for a long time, but the training set data size and the depth of the network have also imposed significant limitations. In medicine, the number of samples is often difficult to meet the requirement of data, and technicians can well generate more accurate segmentation on the edge characteristics of an image by acquiring context information by using a network based on small sample training data, such as a U _ Net network. But at each convolution results in an incomplete edge pixel. With the iterative training of the network, the problem of network degradation exists, the neural network with deeper layers is not easy to train, and the residual learning can simplify the training of the network and enable the training to be easier to optimize, so that the network can reach deeper layers than before, and meanwhile, the network layers can be deepened to obtain better accuracy.

Fig. 1 shows a flow diagram of a method 100 of acquiring a target image element based on image processing according to an embodiment of the invention. The method 100 begins at step 101.

In step 101, a group of images to be processed associated with a target object is acquired, and the group of images to be processed is subjected to data preprocessing to obtain a preprocessed group of images. The image group to be processed comprises a plurality of image files, wherein each image file is provided with an image area and the target object is contained in the image area.

According to one embodiment, the pre-processing the image group to be processed to obtain a pre-processed image group comprises: performing data cleaning on the image group to be processed to obtain a data-cleaned image group; carrying out normalization processing on the image group subjected to data cleaning to obtain an image group subjected to normalization processing; and performing data enhancement on the image group subjected to the normalization processing, thereby obtaining a preprocessed image group.

According to one embodiment, the data washing of the image group to be processed includes: converting the file format of the image group to be processed into a preset file format to obtain the image group with the preset file format; carrying out image recognition on the image group in the preset file format to determine whether an image file which does not meet the preset requirement exists in the image group in the preset file format; and if so, deleting the image files which do not meet the preset requirements to obtain the image group subjected to data cleaning.

According to one embodiment, the normalizing the image group subjected to data cleaning comprises the following steps: determining a resolution of each of a plurality of image files within the data-washed image set; acquiring a preset standard resolution, and performing edge filling on image files with the resolution lower than the standard resolution by using preset contents so as to enable the resolution of each image file of a plurality of image files in the image group subjected to data cleaning to be the standard resolution; the resolution of each image file is adjusted from the standard resolution to the compressed resolution.

According to one embodiment, after obtaining the image group subjected to the normalization processing, the method further comprises performing binarization processing on each image file in the image group subjected to the normalization processing. The data enhancement of the image group subjected to the normalization processing comprises the following steps: performing contrast adjustment, brightness adjustment and/or chromaticity adjustment on each image file in the image group subjected to the normalization processing to obtain a plurality of image files subjected to data enhancement; and forming a preprocessed image group by the plurality of image files subjected to data enhancement.

At step 102, image recognition is performed on the preprocessed image set to determine a plurality of image files, and the plurality of image files are image processed using a neural network to obtain a plurality of target image regions. According to one embodiment, each image file corresponds to a target image area. The method further includes training the neural network prior to image processing the plurality of image files using the neural network.

According to one embodiment, training the neural network comprises:

step 1, initializing a plurality of network parameters of a neural network;

if yes, go to step 6, obtain the neural network trained.

According to one embodiment, before training the neural network, the method further comprises the steps of obtaining a plurality of sample image groups associated with the target object, and performing data preprocessing on each sample image group to obtain a plurality of preprocessed sample image groups; performing image recognition on each preprocessed sample image group to determine a plurality of sample image files, and labeling each sample image file with a label to obtain a plurality of sample image files with labels, wherein the label comprises a training identifier and a testing identifier; the plurality of sample image files are divided into a training sample set and a testing sample set based on the labels.

According to one embodiment, the data preprocessing of each sample image group to obtain a plurality of preprocessed sample image groups comprises: performing data cleaning on each sample image group to obtain a sample image group subjected to data cleaning; normalizing each sample image group subjected to data cleaning to obtain a normalized sample image group; and performing data enhancement on each sample image group subjected to the normalization processing, thereby obtaining a plurality of sample image groups subjected to preprocessing.

According to one embodiment, the data washing of each sample image group comprises: converting the file format of each sample image group into a preset file format to obtain a sample image group with the preset file format; carrying out image recognition on the sample image group in the preset file format to determine whether a sample image file which does not meet the preset requirement exists in the sample image group in the preset file format; and if so, deleting the sample image files which do not meet the preset requirement to obtain the sample image group subjected to data cleaning.

According to one embodiment, the normalization processing of each data-washed sample image group comprises: determining a resolution of each sample image file of a plurality of sample image files within each data-washed sample image set; acquiring a preset standard resolution, and performing edge filling on a sample image file with the resolution lower than the standard resolution by using preset content so as to enable the resolution of each sample image file of a plurality of sample image files in the sample image group subjected to data cleaning to be the standard resolution; the resolution of each sample image file is adjusted from the standard resolution to the compressed resolution.

According to one embodiment, after obtaining the sample image group subjected to the normalization processing, the method further comprises performing binarization processing on each sample image file in the sample image group subjected to the normalization processing. According to one embodiment, the data enhancement of each normalized sample image group includes: performing contrast adjustment, brightness adjustment and/or chroma adjustment on each sample image file in the sample image group subjected to the normalization processing to obtain a plurality of sample image files subjected to data enhancement; and forming a preprocessed sample image group by the plurality of sample image files subjected to data enhancement. According to one embodiment, the method further comprises testing the trained neural network with a set of test samples to obtain a tested neural network.

In step 103, each target image area is image-filled based on the structural features of the target object, and a mask image associated with each target image area is obtained. According to one embodiment, wherein image filling is performed on each target image area based on structural features of the target object, obtaining the mask image associated with each target image area comprises: determining a region contour of the target object in each target image region based on the structural features of the target object; and filling the connected regions based on the region outline of the target object in each target image region to obtain a mask image of the target object associated with each target image region.

At step 104, a masking operation is performed on the associated target image area based on each mask image, thereby obtaining a plurality of target image elements associated with the target object.

According to one embodiment, the neural network is used for acquiring an input image file, constructing a convolution image file with a preset size for output, wherein the output convolution image file with the preset size is an image file capable of being subjected to scaling processing and is an image file with the same length and width, and a parameter of a batch size of an input layer of the neural network is an integral power of 2;

In the application, the deep network training is simplified by the R _ U _ Net neural network constructed by combining U _ Net and residual fast. The use of residual units facilitates information propagation without degradation, and exhibits better performance on image segmentation. Wherein U _ Net employs a network structure comprising down-sampling and up-sampling.

The use of the R _ U _ Net neural network can not only simplify training and enable the network parameters to occupy only one fourth of U _ Net, but also improve the network performance to the greatest extent.

FIG. 2 shows a flow diagram of a method of automatically segmenting an image according to an embodiment of the invention. The invention realizes the automatic femur region segmentation framework shown in fig. 2, and applies the proposed R _ U _ Net neural network to the automatic femur region segmentation. The horizontal direction can be divided into two main stages, namely a neural network training stage and a batch automatic segmentation stage. The longitudinal direction can be divided into three main parts, namely data preprocessing, neural network training and testing and image post-processing. Firstly, an original image is preprocessed, then a target area is marked to produce a label graph, and data enhancement is carried out on a training set. And then inputting the parameters into an R _ U _ Net neural network to fully train and tune parameters, and storing the optimized network model. And finally, inputting the test image to be segmented into a stored network model to obtain the contour of the femoral region, filling the connected region, and then performing user-defined masking operation to obtain the segmentation result of the femoral region, thereby realizing the full-automatic segmentation of the end-to-end X-ray femur.

As shown in fig. 2, the overall framework is automatically segmented: firstly, carrying out normalization processing and binarization on original data, then carrying out data enhancement, inputting the data into a neural network for optimization training, and storing a trained model; and inputting an original image during automatic segmentation, performing data processing, inputting a network model stored before, performing open operation on the outline of a target area after training to obtain a mask image and an original image, performing mask operation, and finally segmenting and outputting.

The subsequent data preprocessing comprises: acquiring a grouped and classified original medical image group, wherein the original medical image group is an X-ray film; firstly, data cleaning is carried out on an original image of an X-ray film, then the whole data set is normalized to the same size, then binarization processing is carried out on the data set, secondly, a femur region marked with a part of samples is made into a label graph to form a training set, label graphs of the rest samples are marked to be used as a test set, and finally, data enhancement of the training set is realized to increase training samples.

Specifically, the data preprocessing includes:

2.1 data cleansing

The data of the X-ray film is a DICOM-format data file, and is first read and stored in jpg format. Some unsatisfactory pictures, such as pictures of nailers, etc., are then removed. Then, the pictures which are in accordance with the picture are screened out. As shown in fig. 3, the first row is a normal picture and the second row is a nailed picture. FIG. 3 shows a schematic diagram of image screening according to an embodiment of the invention.

2.2 normalization

The resolution of the original image is approximately (2048 × 2560), and in order not to change the aspect ratio of the original image, the image is first edge-filled with a background color, that is, black in a batch manner, and then the images are aligned to the (2560 × 2560) size resolution. For images that do not match the standard, zero padding of the edges into images with a resolution of (2560 × 2560) is performed. Then, to reduce the dimensionality of the neural network input, thereby alleviating the difficulty of network training and saving hardware resources, all images normalized to (2560 × 2560) resolution are scaled equally to (512 × 512) resolution in bulk.

After normalization, the picture is subjected to binary processing, and because of the limitation of computer processing performance, the image is simplified, the image characteristics of the region of interest are highlighted, and the image is subjected to binary processing. As shown in fig. 4, the first line is the normalized image, and the second line corresponds to the binarized image. FIG. 4 shows a schematic diagram of a processed image according to an embodiment of the invention

2.3 data enhancement

In order to enlarge the data set, random cropping, turning or mirroring, rotation, adjustment of contrast or brightness, chroma and other methods are used for data enhancement.

Subsequently, neural network training prediction is performed: and inputting the training set into the R _ U _ Net neural network, fully learning the mapping relation from input to output of the neural network, and continuously iteratively optimizing parameters of each network layer to store the optimized network model for later use.

Then, preprocessing any plurality of pre-segmentation images in the test set as training data, and inputting the preprocessed images into the stored R _ U _ Net neural network model to obtain the contour of the femoral region. Fig. 5 shows a schematic diagram of a network infrastructure according to an embodiment of the invention. The network of fig. 5 uses a combination of a residual block and a U _ Net network, and fig. 7 uses forward and backward propagation algorithms.

FIG. 7 shows a schematic diagram of a training process according to an embodiment of the invention. As shown in fig. 7, in the training, a picture is input, all network parameters are initialized, a sample with the size of batch is selected and input into the network for training, forward propagation is performed first until the last layer is output, whether the iteration number meets the requirement is judged, if the iteration number is smaller than a preset value, backward propagation is performed through loss, the parameter weight and the deviation are updated layer by layer, the iterative training is continued, until the iteration number is met, the network training is finished, and the trained network is stored. And then, calculating the output of the network according to the parameter forward propagation of the trained network in the test, and then comparing the output with the label to calculate the result.

FIG. 6 shows a schematic diagram of a neural network, according to an embodiment of the invention. The network of fig. 6 includes: the method comprises the steps of acquiring an input image, constructing an output convolution image, wherein the size of the output image is 512x512, the convolution image can be subjected to scaling processing, and an image with the same length and width is output, and an input layer batch size parameter is an integral power of 2, such as 16, 32, 64 and the like.

The input convolution layer was constructed with the convolved image as input, with filled convolution, bn (batch normalization), then activated with leakyreu, and repeated twice. And then carrying out convolution once again, stacking the convolution with the previous input, then carrying out maximum pooling downwards with the step size of 2, and outputting a convolution result I. The two-dimensional convolution in the present embodiment is 3 × 3 two-dimensional convolution (Conv2D) unless otherwise specified.

The first convolution layer is constructed with the convolution result one as input, and is subjected to filled convolution, bn (batch normalization), and then activated using leakyreu, which is repeated twice. Then carrying out convolution once again, stacking the input and the input, then carrying out pooling for the maximum downwards, wherein the step length is 2, and outputting a convolution result II;

and constructing a second convolution layer by taking the convolution result two as an input, performing filled convolution, namely BN (batch normalization), and then activating by using LeakyReLU, and repeating twice. Then carrying out convolution once again, stacking the input and the input, then carrying out pooling for the maximum downwards, wherein the step length is 2, and outputting a convolution result III;

the convolution result of three is used as input to construct a third convolution layer, and the convolution with padding, bn (batch normalization), is performed, and then activated by using leakyreu, and is repeated twice. Then carrying out convolution once again, stacking the input and the input, then carrying out pooling for the maximum downwards, wherein the step length is 2, and outputting a convolution result IV;

the fourth convolution layer was constructed with convolution result four as input, and was subjected to filled convolution, bn (batch normalization), and then activated using LeakyReLU, and repeated twice. Then carrying out convolution once again, stacking the input and the input, then carrying out pooling for the maximum downwards, wherein the step length is 2, and outputting a convolution result five;

and performing up-sampling on the convolution result five, performing feature fusion on the convolution result five, outputting a result as an up-sampling one, performing filled convolution on the up-sampling one, namely BN (batch normalization), activating by using LeakyReLU, and repeating twice. Then carrying out convolution once again, stacking the convolution and the first up-sampling, and outputting a convolution result six;

and performing up-sampling on the convolution result six, performing feature fusion on the convolution result six and the convolution result three, outputting an up-sampling two, performing filled convolution on the up-sampling two, namely BN (batch normalization), activating by using LeakyReLU, and repeating twice. Then carrying out convolution once again, stacking the convolution and the up-sampling two, and outputting a convolution result seven;

and performing up-sampling on the convolution result seven, performing feature fusion on the convolution result seven and the convolution result two, outputting an up-sampling result three, performing filled convolution on the up-sampling result three, BN (batch normalization), activating by using LeakyReLU, and repeating twice. Then carrying out convolution once again, stacking the convolution and the up-sampling three, and outputting a convolution result eight;

and performing up-sampling on the convolution result eight, performing feature fusion on the convolution result eight and the convolution result I, outputting an up-sampling result four, performing filled convolution on the up-sampling result four, BN (batch normalization), activating by using LeakyReLU, and repeating twice. Then carrying out convolution once again, stacking the convolution and the up-sampling four, and outputting a convolution result nine;

and constructing an output layer by taking the convolution result nine as an input, performing 1x1 convolution once, activating by using sigmoid, and outputting an image.

And finally, carrying out image post-processing: and after the test data set is tested by the neural network, opening the obtained contour of the femoral region to ensure that the continuity of the contour is better, refilling the inside of the connected region to obtain a mask image, and finally performing mask operation to obtain a final femoral region segmentation image.

As shown in fig. 8: the first row is the normalized original image, the second row is the mask image obtained in the previous step, and the third row can obtain the complete femur area image by performing the custom mask operation on the two.

Fig. 9 is a schematic structural diagram of a system for acquiring a target image element based on image processing according to an embodiment of the present invention. The system 900 includes: preprocessing means 901, recognition means 902, filling means 903 and processing means 904.

The preprocessing device 901 is configured to acquire a group of images to be processed associated with a target object, and perform data preprocessing on the group of images to be processed to obtain a preprocessed group of images. The image group to be processed comprises a plurality of image files, wherein each image file is provided with an image area and the target object is contained in the image area.

According to an embodiment, the preprocessing device 901 performing data preprocessing on the image group to be processed to obtain a preprocessed image group includes: performing data cleaning on the image group to be processed to obtain a data-cleaned image group; carrying out normalization processing on the image group subjected to data cleaning to obtain an image group subjected to normalization processing; and performing data enhancement on the image group subjected to the normalization processing, thereby obtaining a preprocessed image group.

According to an embodiment, the pre-processing device 901 performing data washing on the image group to be processed includes: converting the file format of the image group to be processed into a preset file format to obtain the image group with the preset file format; carrying out image recognition on the image group in the preset file format to determine whether an image file which does not meet the preset requirement exists in the image group in the preset file format; and if so, deleting the image files which do not meet the preset requirements to obtain the image group subjected to data cleaning.

According to one embodiment, the preprocessing device 901 performs normalization processing on the image group subjected to data cleaning, including: determining a resolution of each of a plurality of image files within the data-washed image set; acquiring a preset standard resolution, and performing edge filling on image files with the resolution lower than the standard resolution by using preset contents so as to enable the resolution of each image file of a plurality of image files in the image group subjected to data cleaning to be the standard resolution; the resolution of each image file is adjusted from the standard resolution to the compressed resolution.

According to one embodiment, the preprocessing means 901 further includes, after obtaining the group of normalized images, performing binarization processing on each image file in the group of normalized images. The data enhancement of the image group subjected to the normalization processing comprises the following steps: performing contrast adjustment, brightness adjustment and/or chromaticity adjustment on each image file in the image group subjected to the normalization processing to obtain a plurality of image files subjected to data enhancement; and forming a preprocessed image group by the plurality of image files subjected to data enhancement.

And an identifying device 902, configured to perform image identification on the preprocessed image group to determine a plurality of image files, and perform image processing on the plurality of image files by using a neural network to obtain a plurality of target image areas. According to one embodiment, each image file corresponds to a target image area. The method further includes training the neural network prior to image processing the plurality of image files using the neural network.

According to one embodiment, the training of the neural network by the recognition device 902 includes:

step 1, initializing a plurality of network parameters of a neural network;

if yes, go to step 6, obtain the neural network trained.

According to one embodiment, before training the neural network, the identifying device 902 further obtains a plurality of sample image sets associated with the target object, and performs data preprocessing on each sample image set to obtain a plurality of preprocessed sample image sets; performing image recognition on each preprocessed sample image group to determine a plurality of sample image files, and labeling each sample image file with a label to obtain a plurality of sample image files with labels, wherein the label comprises a training identifier and a testing identifier; the plurality of sample image files are divided into a training sample set and a testing sample set based on the labels.

According to one embodiment, the data preprocessing performed by the identifying means 902 on each sample image group to obtain a plurality of preprocessed sample image groups includes: performing data cleaning on each sample image group to obtain a sample image group subjected to data cleaning; normalizing each sample image group subjected to data cleaning to obtain a normalized sample image group; and performing data enhancement on each sample image group subjected to the normalization processing, thereby obtaining a plurality of sample image groups subjected to preprocessing.

According to one embodiment, the data washing of each sample image group by the recognition device 902 includes: converting the file format of each sample image group into a preset file format to obtain a sample image group with the preset file format; carrying out image recognition on the sample image group in the preset file format to determine whether a sample image file which does not meet the preset requirement exists in the sample image group in the preset file format; and if so, deleting the sample image files which do not meet the preset requirement to obtain the sample image group subjected to data cleaning.

According to one embodiment, the identification device 902 performs normalization processing on each sample image group subjected to data cleaning, including: determining a resolution of each sample image file of a plurality of sample image files within each data-washed sample image set; acquiring a preset standard resolution, and performing edge filling on a sample image file with the resolution lower than the standard resolution by using preset content so as to enable the resolution of each sample image file of a plurality of sample image files in the sample image group subjected to data cleaning to be the standard resolution; the resolution of each sample image file is adjusted from the standard resolution to the compressed resolution.

According to one embodiment, after obtaining the normalized sample image group, the identification means 902 performs binarization processing on each sample image file in the normalized sample image group. According to one embodiment, the data enhancement of each normalized sample image group includes: performing contrast adjustment, brightness adjustment and/or chroma adjustment on each sample image file in the sample image group subjected to the normalization processing to obtain a plurality of sample image files subjected to data enhancement; and forming a preprocessed sample image group by the plurality of sample image files subjected to data enhancement. According to one embodiment, the method further comprises testing the trained neural network with a set of test samples to obtain a tested neural network.

And a filling device 903, configured to perform image filling on each target image area based on the structural features of the target object, and obtain a mask image associated with each target image area. According to one embodiment, wherein image filling is performed on each target image area based on structural features of the target object, obtaining the mask image associated with each target image area comprises: determining a region contour of the target object in each target image region based on the structural features of the target object; and filling the connected regions based on the region outline of the target object in each target image region to obtain a mask image of the target object associated with each target image region.

And a processing device 904, configured to perform a masking operation on the associated target image area based on each mask image, thereby obtaining a plurality of target image elements associated with the target object.

While the invention has been described with reference to various specific embodiments, it should be understood that changes can be made within the spirit and scope of the inventive concepts described. Accordingly, it is intended that the invention not be limited to the described embodiments, but that it will have the full scope defined by the language of the following claims.

Claims

1. A method for acquiring a target image element based on image processing, the method comprising:

2. The method of claim 1, wherein the set of images to be processed comprises a plurality of image files, wherein each image file has an image area and the image area contains the target object.

3. The method of claim 1, wherein data pre-processing the set of images to be processed to obtain a pre-processed set of images comprises:

4. The method of claim 3, wherein data washing the set of images to be processed comprises:

5. The method of claim 3, wherein normalizing the set of data-washed images comprises:

6. The method according to claim 5, further comprising, after obtaining the normalized image group, performing binarization processing on each image file in the normalized image group.

7. The method of claim 3, wherein the data enhancing the normalized image group comprises:

8. The method of claim 1, wherein each image file corresponds to a target image area.

9. The method of claim 1, wherein image-filling each target image region based on structural features of the target object, obtaining a mask image associated with each target image region comprises:

10. The method of claim 1, further comprising, prior to image processing the plurality of image files with the neural network, training the neural network.

11. The method of claim 10, wherein training the neural network comprises:

step 1, initializing a plurality of network parameters of a neural network;

if yes, go to step 6, obtain the neural network trained.

12. The method of claim 10, further comprising, prior to training the neural network,

13. The method of claim 12, wherein pre-processing data for each sample image set to obtain a plurality of pre-processed sample image sets comprises:

14. The method of claim 13, wherein data washing each of the sets of sample images comprises:

15. The method of claim 13, wherein normalizing each set of data-washed sample images comprises:

16. The method according to claim 15, further comprising, after obtaining the set of normalized sample images, performing binarization processing on each sample image file in the set of normalized sample images.

17. The method of claim 13, wherein the data enhancing each group of normalized sample images comprises:

18. The method of claim 12, further comprising testing the trained neural network with a set of test samples to obtain a tested neural network.

19. The method according to claim 1 or 12, wherein the neural network is used for acquiring an input image file, constructing a convolution image file with a preset size for output, wherein the convolution image file with the preset size for output is an image file capable of being subjected to scaling processing and is an image file with the same length and width, and the parameter of the batch size of the input layer of the neural network is an integral power of 2;

20. A system for acquiring a target image element based on image processing, the system comprising:

21. A computer-readable storage medium, characterized in that the storage medium stores a computer program for performing the method of any of the preceding claims 1-19.

22. An electronic device, characterized in that the electronic device comprises:

a processor;

a memory for storing the processor-executable instructions;

the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any of claims 1-19.