WO2022126903A1

WO2022126903A1 - Method and device for image anomaly area detection, electronic device, and storage medium

Info

Publication number: WO2022126903A1
Application number: PCT/CN2021/083088
Authority: WO
Inventors: 李泽远; 王健宗
Original assignee: 平安科技（深圳）有限公司
Priority date: 2020-12-18
Filing date: 2021-03-25
Publication date: 2022-06-23
Also published as: CN112465819A; CN112465819B

Abstract

A method and device for image anomaly area detection, an electronic device, and a storage medium, related to the field of artificial intelligence. The method comprises: acquiring a diagnosis image and a diagnosis result from a local database, performing anomaly area positioning and 3D image construction with respect to diagnosis image to produce a 3D anomaly area image; performing feature extraction with respect to the 3D anomaly area image to produce a pathological feature, vectorizing the pathological feature and the diagnosis result and then selecting a shared sample therefrom; training an anomaly area detection model by utilizing the shared sample to produce an initial model gradient; uploading the initial model gradient to a server and receiving an updated model gradient returned by the server; and, by utilizing the anomaly area detection model updated on the basis of the updated model gradient, detecting for an anomaly area in an image to be inspected. Also related to the blockchain technology, the shared sample can be stored in a blockchain. The method increases the accuracy of image anomaly area detection.

Description

Image abnormal area detection method, device, electronic device and storage medium

This application claims the priority of the Chinese patent application filed on December 18, 2020 with the application number 202011508405.8 and the invention title is "Image Abnormal Area Detection Method, Device, Electronic Device and Storage Medium", the entire content of which is approved by Reference is incorporated in this application.

technical field

The present application relates to the field of artificial intelligence, and in particular, to a method, device, electronic device, and computer-readable storage medium for detecting abnormal image areas.

Background technique

The inventor realized that, in the medical field, the detection of abnormal areas in images is usually performed by using computed tomography (CT). The abnormal area detection of the tomographic image is used to track the dynamic changes of different tomographic images, which will lead to the low accuracy of the abnormal area detection in the image, and it is very difficult to complete the diagnosis of the disease by viewing the different tomographic images with the naked eye of the doctor. Inevitably there will be deviations in diagnostic results.

In addition, due to the privacy of medical data, the protection of relevant laws and regulations, and the competitiveness of various institutions, the medical data of various hospitals cannot be reasonably shared, which will also affect the detection of abnormal areas in medical images. accuracy.

SUMMARY OF THE INVENTION

A method for detecting an abnormal area in an image provided by this application, the method is applied to the client of one of the participants participating in the federated learning, including:

Obtaining diagnosis and treatment images and diagnosis and treatment results from a pre-built local database, and locating abnormal areas on the diagnosis and treatment images in the diagnosis and treatment images to obtain abnormal area images;

3D image construction is performed on the abnormal area image to generate a 3D abnormal area image;

Perform feature extraction on the 3D abnormal image to obtain pathological features, and perform vectorization processing on the pathological features and diagnosis and treatment results, respectively, to obtain a pathological feature vector and a diagnosis and treatment result vector;

A sample alignment method is used to select shared samples from the pathological feature vector and the diagnosis and treatment result vector;

Use the shared samples to train the pre-built abnormal region detection model to obtain the initial model gradient;

After encrypting the initial model gradient and adding noise, upload it to the server, and accept the updated model gradient returned by the server;

The model gradient of the abnormal area detection model is updated according to the updated model gradient to obtain an updated abnormal area detection model, and the abnormal area detection model is used to detect the abnormal area of the image to be inspected.

The present application also provides an image abnormal area detection device, the device includes:

The positioning module is used for obtaining the diagnosis and treatment images and diagnosis and treatment results from the pre-built local database, and locating the abnormal area on the diagnosis and treatment image to obtain the abnormal area image;

a building module for constructing a 3D image of the abnormal area image to generate a 3D abnormal area image;

an extraction module, configured to perform feature extraction on the 3D abnormal image to obtain pathological features, and perform vectorization processing on the pathological features and diagnosis and treatment results, respectively, to obtain a pathological feature vector and a diagnosis and treatment result vector;

a selection module, used for using a sample alignment method to select shared samples from the pathological feature vector and the diagnosis and treatment result vector;

A training module is used to train a pre-built abnormal region detection model using the shared sample to obtain an initial model gradient;

an update module, configured to encrypt the initial model gradient and add noise, upload it to the server, and accept the updated model gradient returned by the server;

The detection module is configured to update the model gradient of the abnormal area detection model according to the updated model gradient to obtain the updated abnormal area detection model, and use the updated abnormal area detection model to detect the abnormal area of the image to be inspected.

The present application also provides an electronic device, the electronic device comprising:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to implement the image abnormal area detection method as described below:

The present application also provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is executed by a processor in an electronic device to realize the abnormal image area as described below Detection method:

Description of drawings

FIG. 1 is a schematic flowchart of a method for detecting abnormal regions in an image provided by an embodiment of the present application;

FIG. 2 is a detailed schematic flowchart of one of the steps of the image abnormal area detection method provided in FIG. 1 in the first embodiment of the present application;

FIG. 3 is a detailed flowchart of another step of the image abnormal area detection method provided in FIG. 1 in the first embodiment of the present application;

FIG. 4 is a schematic block diagram of an image abnormal area detection apparatus provided by an embodiment of the present application;

5 is a schematic diagram of the internal structure of an electronic device for implementing a method for detecting abnormal image areas provided by an embodiment of the present application;

The realization, functional characteristics and advantages of the purpose of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

Embodiments of the present application provide a method for detecting abnormal regions in an image. The execution subject of the image abnormal area detection method includes, but is not limited to, at least one of electronic devices that can be configured to execute the method provided by the embodiments of the present application, such as a server and a terminal. In other words, the image abnormal area detection method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like. Referring to FIG. 1 , a schematic flowchart of a method for detecting abnormal regions in an image provided by an embodiment of the present application is shown. In the embodiment of the present application, the image abnormal area detection method is applied to the client of one of the participants participating in the federated learning, and includes:

S1. Acquire a diagnosis and treatment image and a diagnosis and treatment result from a pre-built local database, and locate an abnormal area on the diagnosis and treatment image to obtain an abnormal area image.

In the embodiment of the present application, the local database includes a medical database, and the medical database is established by accessing a pre-authorized hospital background database. Preferably, the medical database is an Oracle database. Further, in the embodiment of the present application, the medical database The medical database includes medical images and medical results.

Wherein, the diagnosis and treatment images include: lumbar spine cross-sectional CT images, chest CT images, tumor images, etc., and the diagnosis and treatment results include: the diagnosis results made by the doctor for the patient's condition and the treatment plan (drug name, drug name, drug name, drug name, etc.) made by the doctor for the condition of the patient. mode of administration, dosage, frequency).

Further, since there are abnormal areas and non-abnormal areas in the diagnosis and treatment images, the present application selects the abnormal area images by locating the abnormal areas in the diagnosis and treatment images. Optionally, in the embodiment of the present application, a fully convolutional neural network is used to predict the category of the pixel points of the diagnosis and treatment image to realize the location of abnormal regions. Optionally, the fully convolutional neural network is Fully Convolutional Networks for Semantic Segmentation (FCN for short).

In detail, as shown in FIG. 2 , the abnormal area location is performed on the diagnosis and treatment image to obtain the abnormal area image, including:

S10, using the convolution layer in the fully convolutional neural network to perform a convolution operation on each of the diagnosis and treatment images to obtain a characteristic diagnosis and treatment image;

S11, using the fusion layer in the fully convolutional neural network to fuse the underlying features of the diagnosis and treatment image with the characteristic diagnosis and treatment image to obtain a target characteristic image;

S12. Use the activation function in the fully convolutional neural network to output the detection result of the target feature image, and select an abnormal region of the target feature image according to the detection result to obtain an abnormal region image.

In the embodiment of the present application, the convolution layer performs a convolution operation on the image, which can realize image feature extraction. Optionally, the convolution operation may be implemented by performing a convolution operation on the tensor of the medical image.

The fusion layer in the fully convolutional neural network fuses the underlying features of the image into the extracted image features, which can reduce the influence on image grayscale changes caused by different gains. The underlying feature refers to the basic features of the diagnosis and treatment image, such as color, length, width, etc. Preferably, in the embodiment of the present application, the CSP (Cross-Stage-Partial-connections, Partial connection across stages) module implementation.

In an optional embodiment of the present application, the activation function includes:

Among them, s' represents the detection result of the target feature image, and s represents the target feature image.

Further, in the embodiment of the present application, the detection results include: x, y, height, width, category, etc., where x and y represent the center point of the target feature image, and the category represents whether the target feature image is an abnormal area, that is, Category 0 indicates that it is not an abnormal area, and category 1 indicates that the predicted area is an abnormal area. Therefore, in this embodiment of the present application, a target feature image of category 1 is selected as an abnormal area, so as to generate the abnormal area image.

S2. Perform 3D image construction on the abnormal area image to generate a 3D abnormal area image.

Due to the different positions of the abnormal area images in the abnormal area images, if the abnormal area images are observed one by one, it is easy to bring a large time cost, and at the same time, it is not conducive to tracking the abnormal area positions of the abnormal area images. changes, which will affect the training effect of the subsequent model. For example, there are 100 images of abnormal areas in the diagnosis and treatment images of patient A. If the 100 images of abnormal areas are observed one by one, it will bring a great time cost, and it is also difficult to track Changes in the position of the abnormal area to 100 abnormal area images. Therefore, the present application generates a 3D abnormal area image by constructing a 3D image of the abnormal image, so as to track the position change of the abnormal area in the abnormal area image, and improve the effect of subsequent model training.

In a preferred implementation of the present application, the 3D image construction is realized by 3D medical image reconstruction software.

S3. Perform feature extraction on the 3D abnormal image to obtain pathological features, and perform vectorization processing on the pathological features and diagnosis and treatment results, respectively, to obtain a pathological feature vector and a diagnosis and treatment result vector.

In the embodiment of the present application, the feature extraction is performed on the 3D abnormal image to obtain patient information of the corresponding patient, such as the position sequence of the damaged part of the patient, the severity information of the damaged part, and the like.

In detail, as shown in FIG. 3 , the pathological features obtained by feature extraction on the 3D abnormal image include:

S20, obtaining the regional coordinate frame of the 3D abnormal image, that is, the label value;

S21. Calculate the first pathological feature prediction coordinate frame of the 3D abnormal image, and calculate the first intersection ratio between the first pathological feature prediction coordinate frame and the region coordinate frame, and filter out the first intersection ratio A first pathological feature prediction coordinate frame larger than a preset first threshold is cropped, and a first pathological feature image is obtained by cropping the screened first pathological feature prediction coordinate frame;

S22. Calculate the second pathological feature prediction coordinate frame of the first pathological feature image, and calculate the second intersection ratio between the second pathological feature prediction coordinate frame and the region coordinate frame, and filter out the second intersection and comparing the second pathological feature prediction coordinate frame larger than the preset second threshold, cropping the screened second pathological feature prediction coordinate frame to obtain a second pathological feature image;

S23. Calculate a third pathological feature prediction coordinate frame of the second pathological feature image, and calculate a third intersection ratio between the third pathological feature prediction coordinate frame and the region coordinate frame, and filter out the third intersection and comparing a third pathological feature prediction coordinate frame larger than a preset third threshold, cropping the screened third pathological feature prediction coordinate frame to obtain a third pathological feature image;

S24. Extract pathological features from the third pathological feature image.

Wherein, the region coordinate frame of the 3D abnormal image is used to compare the pathological feature region coordinate frame manually marked in the 3D abnormal image with the subsequently predicted pathological feature prediction coordinate frame, so as to extract the corresponding pathological feature. Features, the intersection ratio is used to indicate the degree of overlap between the predicted pathological feature prediction coordinate frame and the labeled pathological feature prediction coordinate frame. The present application performs three predictions of the pathological feature prediction coordinate frame on the 3D abnormal image to improve the follow-up. The accuracy of pathological feature extraction.

In an optional embodiment of the present application, the following method is used to calculate the first pathological feature prediction coordinate frame of the 3D abnormal image:

Among them, T(r) represents the first pathological feature prediction coordinate frame, r represents the image grayscale order of the 3D abnormal image, and P _r (r) grayscale probability density function.

In one of the optional embodiments of the present application, the following method is used to calculate the first intersection ratio of the first pathological feature prediction coordinate frame and the region coordinate frame:

Among them, DICE(A, B) represents the intersection ratio, A represents the pathological feature prediction coordinate frame, and B represents the regional coordinates. Optionally, the preset threshold is 0.7.

In an optional embodiment of the present application, the first pathological feature prediction coordinate frame, the second pathological feature prediction coordinate frame, and the third pathological feature prediction coordinate frame are cropped by an image cropping tool, and the image cropping tool includes: Photoshop tool.

In an optional embodiment of the present application, a histogram algorithm is used to extract the pathological feature from the third pathological feature image, and the histogram algorithm includes: a histogram of Oriented Gradient (HOG) algorithm.

Further, the calculation method of the second pathological feature prediction coordinate frame and the third pathological feature prediction coordinate frame can refer to the calculation method of the above-mentioned first pathological feature prediction coordinate frame, and the second intersection ratio and the third intersection ratio are compared. For the calculation method of , please refer to the calculation method of the first cross-union ratio above. Optionally, the preset first threshold, the preset second threshold and the preset third threshold are 0.5, 0.6 and 0.7 respectively.

Further, in the embodiment of the present application, the Word2vec vector transformation method is used to perform vector transformation on the pathological feature and the diagnosis and treatment result to obtain a pathological feature vector and a diagnosis and treatment result vector, which are used to train an abnormal region detection model.

S4. Using a sample alignment method, a shared sample is selected from the pathological feature vector and the diagnosis and treatment result vector.

The embodiment of the present application is based on the premise of protecting the privacy of different medical data, from the pathological feature vector and the diagnosis and treatment result vector have the same characteristics of the image and diagnosis and treatment results as shared samples, preferably, the application uses sample alignment technology, from The pathological feature vector and the diagnosis and treatment result vector select shared samples. Using the sample alignment technology to obtain shared samples can align the sample numbers and time information, which is conducive to the accurate sharing of data between each participant participating in the federated training. Wherein, the sample alignment technology belongs to the currently known technology, and will not be further elaborated here.

Further, it should be emphasized that, in order to ensure the reusability of the shared samples, the shared samples can also be stored in a blockchain node.

S5. Use the shared samples to train the pre-built abnormal region detection model to obtain an initial model gradient.

In the embodiment of the present application, the abnormal area detection model is constructed by using the VGG deep learning network, wherein the abnormal area detection model includes: a convolution layer, a pooling layer, a fully connected layer, and the like.

Further, the step S5 includes: using the convolution layer to perform feature extraction on the shared samples to obtain feature samples; using the pooling layer to reduce the dimension of the feature samples to obtain dimensional samples; using the The fully connected layer outputs the training value of the dimensionality reduction sample, and uses the loss function in the abnormal area detection model to calculate the loss value of the training value and the shared sample label value; according to the loss value, adjust the abnormality The model gradient of the region detection model, until the loss value is smaller than the preset threshold, obtains the initial model gradient of the abnormal region detection model.

In an optional embodiment, the model gradient is adjusted by a stochastic gradient descent algorithm.

S6. After encrypting the initial model gradient and adding noise, upload it to the server, and accept the updated model gradient returned by the server.

Since the initial model gradient has a certain degree of privacy, in this embodiment of the present application, the initial model gradient is encrypted and processed by adding noise, and then uploaded to the server to ensure that the initial model gradient is not stolen during the transmission process. And to avoid the problem of model data being leaked when updating the model.

Wherein, the initial model gradient encryption is encrypted according to the public key pre-set in the server, and the added noise is realized by salt and pepper noise.

Further, in the embodiment of the present application, before the receiving the updated model gradient returned by the server includes: decrypting the initial model gradient by using the private key stored in the server to obtain the decrypted model gradient, and in the In the server, the gradient of the decryption model is weighted and averaged to obtain the gradient of the updated model.

Among them, the server refers to a security server that is jointly trusted by multiple participants participating in the federation training. By sharing the model training gradient on the server, it can ensure that the local data of each participant is not stolen, effectively The privacy of local data is guaranteed.

S7, update the model gradient of the abnormal area detection model according to the updated model gradient, obtain the updated abnormal area detection model, and utilize the updated abnormal area detection model to detect the abnormal area of the image to be inspected.

In the embodiment of the present application, the initial model gradient of the abnormal area detection model is updated according to the updated model gradient returned by the server to obtain the updated abnormal area detection model, and the abnormal area detection model is used to detect the abnormal area of the image to be inspected. , to obtain a detection result, wherein the image to be checked includes: a CT image of the lumbar spine, and the detection result includes: a fracture area and a treatment plan existing in the CT image of the lumbar spine.

In the embodiment of the present application, firstly, a diagnosis and treatment image and diagnosis and treatment results are obtained from a pre-built local database, an abnormal area is located on the diagnosis and treatment image to obtain an abnormal area image, and a 3D image is constructed on the abnormal area image to generate a 3D abnormal area. image, to track the position change of the abnormal area in the abnormal area image, and improve the follow-up model training effect; secondly, the embodiment of the present application performs feature extraction, vectorization processing and shared sample selection on the 3D abnormal image and the diagnosis and treatment results, The shared samples are obtained, which ensures the privacy between data, so that the data of the participants participating in the training can be well and reasonably shared; further, the embodiment of the present application uses the shared samples to train the pre-built abnormal area detection model, And use the trained abnormal area detection model to perform abnormal area detection on the image to be inspected to obtain a detection result. Therefore, the method for detecting abnormal areas in images proposed in this application can improve the accuracy of detecting abnormal areas in images.

As shown in FIG. 4 , it is a functional block diagram of the image abnormal area detection apparatus of the present application.

The image abnormal area detection apparatus 100 described in this application may be installed in an electronic device. According to the implemented functions, the image abnormal area detection apparatus may include a positioning module 101 , a construction module 102 , an extraction module 103 , a selection module 104 , a training module 105 , an update module 106 and a detection module 107 . The modules described in the present invention can also be called units, which refer to a series of computer program segments that can be executed by the electronic device processor and can perform fixed functions, and are stored in the memory of the electronic device.

In this embodiment, the functions of each module/unit are as follows:

The positioning module 101 is configured to obtain the diagnosis and treatment images and diagnosis and treatment results from a pre-built local database, and to locate the abnormal area on the diagnosis and treatment image to obtain the abnormal area image;

The construction module 102 is used to construct a 3D image of the abnormal area image to generate a 3D abnormal area image;

The extraction module 103 is configured to perform feature extraction on the 3D abnormal image to obtain pathological features, and perform vectorization processing on the pathological features and diagnosis and treatment results, respectively, to obtain a pathological feature vector and a diagnosis and treatment result vector;

The selection module 104 is used to select a shared sample from the pathological feature vector and the diagnosis and treatment result vector by adopting a sample alignment method;

The training module 105 is configured to use the shared samples to train the pre-built abnormal region detection model to obtain an initial model gradient;

The updating module 106 is configured to encrypt the initial model gradient and add noise processing, upload it to the server, and accept the updated model gradient returned by the server;

The detection module 107 is configured to update the model gradient of the abnormal area detection model according to the updated model gradient to obtain an updated abnormal area detection model, and use the updated abnormal area detection model to detect abnormal areas in the image to be inspected.

In detail, the modules in the image abnormal area detection apparatus 100 in the embodiments of the present application use the same technical means as the image abnormal area detection methods described in FIG. 1 to FIG. 3 , and can The same technical effect is produced, which is not repeated here.

As shown in FIG. 5 , it is a schematic structural diagram of an electronic device implementing the method for detecting abnormal image areas in the present application.

The electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as an abnormal image area detection program 12.

Wherein, the memory 11 includes at least one type of readable storage medium, and the readable storage medium may be volatile or non-volatile. Specifically, the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (eg, SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1 , such as a mobile hard disk of the electronic device 1 . In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a pluggable mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) equipped on the electronic device 1. card, flash memory card (FlashCard), etc. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software installed in the electronic device 1 and various types of data, such as codes for detecting abnormal image areas, etc., but also can be used to temporarily store data that has been output or will be output.

In some embodiments, the processor 10 may be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits packaged with the same function or different functions, including one or more integrated circuits. Central processing unit (Central Processing unit, CPU), microprocessor, digital processing chip, graphics processor and combination of various control chips, etc. The processor 10 is the control core (ControlUnit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device, by running or executing the program or module (for example, executing the image) stored in the memory 11. abnormal area detection, etc.), and call data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

The bus may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (extended industry standard architecture, EISA for short) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. The bus is configured to implement connection communication between the memory 11 and at least one processor 10 and the like.

FIG. 5 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. 5 does not constitute a limitation on the electronic device 1, and may include fewer or more components than those shown in the drawings. components, or a combination of certain components, or a different arrangement of components.

For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) for powering the various components, preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so that the power management The device implements functions such as charge management, discharge management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device 1 may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

Further, the electronic device 1 may also include a network interface, optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used in the electronic device 1 Establish a communication connection with other electronic devices.

Optionally, the electronic device 1 may further include a user interface, and the user interface may be a display (Display), an input unit (eg, a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. The display may also be appropriately called a display screen or a display unit, which is used for displaying information processed in the electronic device 1 and for displaying a visualized user interface.

It should be understood that the embodiments are only used for illustration, and are not limited by this structure in the scope of the patent application.

The image abnormal area detection 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions, and when running in the processor 10, can realize:

Obtaining diagnosis and treatment images and diagnosis and treatment results from a pre-built local database, and locating abnormal areas on the diagnosis and treatment images to obtain images of abnormal areas;

Specifically, for the specific implementation method of the above-mentioned instruction by the processor 10, reference may be made to the description of the relevant steps in the corresponding embodiment of FIG. 1, and details are not described herein.

Further, if the modules/units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. The computer-readable storage medium may be volatile or non-volatile. Specifically, the computer-readable storage medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read Only Memory) -Only Memory).

In the several embodiments provided in this application, it should be understood that the disclosed apparatus, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, but that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

Accordingly, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the application is to be defined by the appended claims rather than the foregoing description, which is therefore intended to fall within the scope of the claims. All changes within the meaning and scope of the equivalents of , are included in this application. Any reference signs in the claims shall not be construed as limiting the involved claim.

The blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

Furthermore, it is clear that the word "comprising" does not exclude other units or steps and the singular does not exclude the plural. Several units or means recited in the system claims can also be realized by one unit or means by means of software or hardware. Second-class terms are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present application.

Claims

A method for detecting abnormal areas in images, wherein the method is applied to a client of one of the participants participating in federated learning, including:

Obtaining diagnosis and treatment images and diagnosis and treatment results from a pre-built local database, and locating abnormal areas on the diagnosis and treatment images to obtain images of abnormal areas;

3D image construction is performed on the abnormal area image to generate a 3D abnormal area image;

Perform feature extraction on the 3D abnormal image to obtain pathological features, and perform vectorization processing on the pathological features and diagnosis and treatment results, respectively, to obtain a pathological feature vector and a diagnosis and treatment result vector;

A sample alignment method is used to select shared samples from the pathological feature vector and the diagnosis and treatment result vector;

Use the shared samples to train the pre-built abnormal region detection model to obtain the initial model gradient;

After encrypting the initial model gradient and adding noise, upload it to the server, and accept the updated model gradient returned by the server;

The model gradient of the abnormal area detection model is updated according to the updated model gradient to obtain an updated abnormal area detection model, and the abnormal area detection model is used to detect the abnormal area of the image to be inspected.
The method for detecting an abnormal area in an image according to claim 1, wherein the performing an abnormal area location on the diagnosis and treatment image to obtain an abnormal area image, comprising:

Using the convolution layer in the fully convolutional neural network to perform a convolution operation on the diagnosis and treatment image to obtain a characteristic diagnosis and treatment image;

Using the fusion layer in the fully convolutional neural network to fuse the underlying features of the diagnosis and treatment image with the characteristic diagnosis and treatment image to obtain a target feature image;

The detection result of the target feature image is output by using the activation function in the fully convolutional neural network, and according to the detection result, an abnormal region of the target feature image is selected to obtain an abnormal region image.
The method for detecting an abnormal area in an image according to claim 1, wherein the pathological feature obtained by feature extraction on the 3D abnormal image comprises:

obtaining the regional coordinate frame of the 3D abnormal image;

Calculate the first pathological feature predicted coordinate frame of the 3D abnormal image, and calculate the first intersection ratio of the first pathological feature predicted coordinate frame and the region coordinate frame, and filter out the first intersection ratio greater than the predicted coordinate frame. A first pathological feature prediction coordinate frame of a first threshold is set, and the screened first pathological feature prediction coordinate frame is cropped to obtain a first pathological feature image;

Calculate the second pathological feature prediction coordinate frame of the first pathological feature image, and calculate the second intersection ratio of the second pathological feature prediction coordinate frame and the region coordinate frame, and filter out the second intersection ratio a second pathological feature prediction coordinate frame larger than a preset second threshold, crop the screened second pathological feature prediction coordinate frame to obtain a second pathological feature image;

Calculate the third pathological feature prediction coordinate frame of the second pathological feature image, and calculate the third intersection and union ratio of the third pathological feature prediction coordinate frame and the region coordinate frame, and filter out the third intersection and union ratio A third pathological feature prediction coordinate frame larger than a preset third threshold is cropped, and a third pathological feature image is obtained by cropping the screened third pathological feature prediction coordinate frame;

A pathological feature is extracted from the third pathological feature image.
The image abnormal area detection method according to claim 3, wherein the calculating the first pathological feature prediction coordinate frame of the 3D abnormal image comprises:

The first pathological feature prediction coordinate frame of the 3D abnormal image is calculated by the following method:

Among them, T(r) represents the first pathological feature prediction coordinate frame, r represents the image grayscale order of the 3D abnormal image, and P r (r) grayscale probability density function.
The method for detecting an abnormal area in an image according to claim 1, wherein the use of the shared sample to train a pre-built abnormal area detection model to obtain an initial model gradient, comprising:

Use the convolution layer in the abnormal area detection model to perform feature extraction on the shared samples to obtain feature samples;

Use the pooling layer in the abnormal area detection model to reduce the dimension of the feature sample to obtain the dimension-reduced sample;

Use the fully connected layer in the abnormal area detection model to output the training value of the dimensionality reduction sample, and use the loss function in the abnormal area detection model to calculate the loss value of the training value and the shared sample label value;

According to the loss value, the model gradient of the abnormal area detection model is adjusted until the loss value is less than a preset threshold, and the initial model gradient of the abnormal area detection model is obtained.
The method for detecting abnormal image areas according to claim 1, wherein before receiving the updated model gradient returned by the server, the method further comprises: decrypting the initial model gradient by using the private key stored in the server, The decryption model gradient is obtained, and weighted average is performed on the decryption model gradient in the server to obtain the update model gradient.
The image abnormal area detection method according to any one of claims 1 to 6, wherein the image to be detected includes a lumbar spine CT image.
An image abnormal area detection device, wherein the device includes:

The positioning module is used for obtaining the diagnosis and treatment images and diagnosis and treatment results from the pre-built local database, and locating the abnormal area on the diagnosis and treatment image to obtain the abnormal area image;

a building module for constructing a 3D image of the abnormal area image to generate a 3D abnormal area image;

an extraction module, configured to perform feature extraction on the 3D abnormal image to obtain pathological features, and perform vectorization processing on the pathological features and diagnosis and treatment results, respectively, to obtain a pathological feature vector and a diagnosis and treatment result vector;

a selection module, used for using a sample alignment method to select shared samples from the pathological feature vector and the diagnosis and treatment result vector;

a training module, used for using the shared samples to train the pre-built abnormal region detection model to obtain the initial model gradient;

an update module, configured to encrypt the initial model gradient and add noise, upload it to the server, and accept the updated model gradient returned by the server;

The detection module is configured to update the model gradient of the abnormal area detection model according to the updated model gradient to obtain the updated abnormal area detection model, and use the updated abnormal area detection model to detect the abnormal area of the image to be inspected.
An electronic device, wherein the electronic device comprises:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores computer program instructions executable by the at least one processor, the computer program instructions being executed by the at least one processor to enable the at least one processor to execute an image anomaly region as described below Detection method:

Obtaining diagnosis and treatment images and diagnosis and treatment results from a pre-built local database, and locating abnormal areas on the diagnosis and treatment images to obtain images of abnormal areas;

3D image construction is performed on the abnormal area image to generate a 3D abnormal area image;

Perform feature extraction on the 3D abnormal image to obtain pathological features, and perform vectorization processing on the pathological features and diagnosis and treatment results, respectively, to obtain a pathological feature vector and a diagnosis and treatment result vector;

A sample alignment method is used to select shared samples from the pathological feature vector and the diagnosis and treatment result vector;

Use the shared samples to train the pre-built abnormal region detection model to obtain the initial model gradient;

After encrypting the initial model gradient and adding noise, upload it to the server, and accept the updated model gradient returned by the server;

The model gradient of the abnormal region detection model is updated according to the updated model gradient to obtain the updated abnormal region detection model, and the abnormal region detection model is used to detect the abnormal region of the image to be inspected.
The electronic device according to claim 9, wherein the performing abnormal area positioning on the diagnosis and treatment image to obtain an abnormal area image, comprising:

Using the convolution layer in the fully convolutional neural network to perform a convolution operation on the diagnosis and treatment image to obtain a characteristic diagnosis and treatment image;

Using the fusion layer in the fully convolutional neural network to fuse the underlying features of the diagnosis and treatment image with the characteristic diagnosis and treatment image to obtain a target feature image;

The detection result of the target feature image is output by using the activation function in the fully convolutional neural network, and according to the detection result, an abnormal region of the target feature image is selected to obtain an abnormal region image.
The electronic device according to claim 9, wherein the pathological features obtained by performing feature extraction on the 3D abnormal image include:

obtaining the regional coordinate frame of the 3D abnormal image;

Calculate the first pathological feature predicted coordinate frame of the 3D abnormal image, and calculate the first intersection ratio of the first pathological feature predicted coordinate frame and the region coordinate frame, and filter out the first intersection ratio greater than the predicted coordinate frame. A first pathological feature prediction coordinate frame of a first threshold is set, and the screened first pathological feature prediction coordinate frame is cropped to obtain a first pathological feature image;

Calculate the second pathological feature prediction coordinate frame of the first pathological feature image, and calculate the second intersection ratio of the second pathological feature prediction coordinate frame and the region coordinate frame, and filter out the second intersection ratio a second pathological feature prediction coordinate frame larger than a preset second threshold, crop the screened second pathological feature prediction coordinate frame to obtain a second pathological feature image;

Calculate the third pathological feature prediction coordinate frame of the second pathological feature image, and calculate the third intersection and union ratio of the third pathological feature prediction coordinate frame and the region coordinate frame, and filter out the third intersection and union ratio A third pathological feature prediction coordinate frame larger than a preset third threshold is cropped, and a third pathological feature image is obtained by cropping the screened third pathological feature prediction coordinate frame;

A pathological feature is extracted from the third pathological feature image.
The electronic device according to claim 11, wherein the calculating the first pathological feature prediction coordinate frame of the 3D abnormal image comprises:

The first pathological feature prediction coordinate frame of the 3D abnormal image is calculated by the following method:

Among them, T(r) represents the first pathological feature prediction coordinate frame, r represents the image grayscale order of the 3D abnormal image, and P r (r) grayscale probability density function.
The electronic device according to claim 9, wherein the training of a pre-built abnormal region detection model by using the shared samples to obtain an initial model gradient comprises:

Use the convolution layer in the abnormal area detection model to perform feature extraction on the shared samples to obtain feature samples;

Use the pooling layer in the abnormal area detection model to reduce the dimension of the feature sample to obtain the dimension-reduced sample;

Use the fully connected layer in the abnormal area detection model to output the training value of the dimensionality reduction sample, and use the loss function in the abnormal area detection model to calculate the loss value of the training value and the shared sample label value;

According to the loss value, the model gradient of the abnormal area detection model is adjusted until the loss value is less than a preset threshold, and the initial model gradient of the abnormal area detection model is obtained.
The electronic device according to claim 9, wherein before receiving the updated model gradient returned by the server, the method further comprises: decrypting the initial model gradient by using the private key stored in the server to obtain a decrypted model The gradient of the decryption model is weighted and averaged in the server to obtain the updated model gradient.
A computer-readable storage medium storing a computer program, wherein, when the computer program is executed by a processor, the following method for detecting an abnormal area in an image is implemented:

Obtaining diagnosis and treatment images and diagnosis and treatment results from a pre-built local database, and locating abnormal areas on the diagnosis and treatment images to obtain images of abnormal areas;

3D image construction is performed on the abnormal area image to generate a 3D abnormal area image;

Perform feature extraction on the 3D abnormal image to obtain pathological features, and perform vectorization processing on the pathological features and diagnosis and treatment results, respectively, to obtain a pathological feature vector and a diagnosis and treatment result vector;

A sample alignment method is used to select shared samples from the pathological feature vector and the diagnosis and treatment result vector;

Use the shared samples to train the pre-built abnormal region detection model to obtain the initial model gradient;

After encrypting the initial model gradient and adding noise, upload it to the server, and accept the updated model gradient returned by the server;

The model gradient of the abnormal area detection model is updated according to the updated model gradient to obtain an updated abnormal area detection model, and the abnormal area detection model is used to detect the abnormal area of the image to be inspected.
The computer-readable storage medium according to claim 15, wherein the performing abnormal region positioning on the diagnosis and treatment image to obtain the abnormal region image comprises:

Using the convolution layer in the fully convolutional neural network to perform a convolution operation on the diagnosis and treatment image to obtain a characteristic diagnosis and treatment image;

Using the fusion layer in the fully convolutional neural network to fuse the underlying features of the diagnosis and treatment image with the characteristic diagnosis and treatment image to obtain a target feature image;

The detection result of the target feature image is output by using the activation function in the fully convolutional neural network, and according to the detection result, an abnormal region of the target feature image is selected to obtain an abnormal region image.
The computer-readable storage medium of claim 15, wherein the pathological feature obtained by feature extraction on the 3D abnormal image comprises:

obtaining the regional coordinate frame of the 3D abnormal image;

Calculate the first pathological feature predicted coordinate frame of the 3D abnormal image, and calculate the first intersection ratio of the first pathological feature predicted coordinate frame and the region coordinate frame, and filter out the first intersection ratio greater than the predicted coordinate frame. A first pathological feature prediction coordinate frame of a first threshold is set, and the screened first pathological feature prediction coordinate frame is cropped to obtain a first pathological feature image;

Calculate the second pathological feature prediction coordinate frame of the first pathological feature image, and calculate the second intersection ratio of the second pathological feature prediction coordinate frame and the region coordinate frame, and filter out the second intersection ratio a second pathological feature prediction coordinate frame larger than a preset second threshold, crop the screened second pathological feature prediction coordinate frame to obtain a second pathological feature image;

Calculate the third pathological feature prediction coordinate frame of the second pathological feature image, and calculate the third intersection and union ratio of the third pathological feature prediction coordinate frame and the region coordinate frame, and filter out the third intersection and union ratio A third pathological feature prediction coordinate frame larger than a preset third threshold is cropped, and a third pathological feature image is obtained by cropping the screened third pathological feature prediction coordinate frame;

A pathological feature is extracted from the third pathological feature image.
The computer-readable storage medium of claim 17, wherein the calculating the first pathological feature prediction coordinate frame of the 3D abnormal image comprises:

The first pathological feature prediction coordinate frame of the 3D abnormal image is calculated by the following method:

Among them, T(r) represents the first pathological feature prediction coordinate frame, r represents the image grayscale order of the 3D abnormal image, and P r (r) grayscale probability density function.
The computer-readable storage medium according to claim 15, wherein the training of a pre-built abnormal region detection model by using the shared samples to obtain an initial model gradient comprises:

Use the convolution layer in the abnormal area detection model to perform feature extraction on the shared samples to obtain feature samples;

Use the pooling layer in the abnormal area detection model to reduce the dimension of the feature sample to obtain the dimension-reduced sample;

Use the fully connected layer in the abnormal area detection model to output the training value of the dimensionality reduction sample, and use the loss function in the abnormal area detection model to calculate the loss value of the training value and the shared sample label value;

According to the loss value, the model gradient of the abnormal area detection model is adjusted until the loss value is less than a preset threshold, and the initial model gradient of the abnormal area detection model is obtained.
The computer-readable storage medium according to claim 15, wherein before receiving the updated model gradient returned by the server, the method further comprises: decrypting the initial model gradient by using the private key stored in the server, The decryption model gradient is obtained, and weighted average is performed on the decryption model gradient in the server to obtain the update model gradient.