CN114841952B - Cloud-edge cooperative retinopathy of prematurity detection system and detection method - Google Patents
Cloud-edge cooperative retinopathy of prematurity detection system and detection method Download PDFInfo
- Publication number
- CN114841952B CN114841952B CN202210460497.XA CN202210460497A CN114841952B CN 114841952 B CN114841952 B CN 114841952B CN 202210460497 A CN202210460497 A CN 202210460497A CN 114841952 B CN114841952 B CN 114841952B
- Authority
- CN
- China
- Prior art keywords
- detection
- model
- cloud
- edge
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 198
- 206010038933 Retinopathy of prematurity Diseases 0.000 title claims description 31
- 210000001525 retina Anatomy 0.000 claims abstract description 58
- 230000002028 premature Effects 0.000 claims abstract description 48
- 230000003902 lesion Effects 0.000 claims abstract description 43
- 208000017442 Retinal disease Diseases 0.000 claims abstract description 20
- 206010038923 Retinopathy Diseases 0.000 claims abstract description 20
- 238000000034 method Methods 0.000 claims abstract description 19
- 230000005540 biological transmission Effects 0.000 claims abstract description 13
- 238000007781 pre-processing Methods 0.000 claims abstract description 11
- 230000004256 retinal image Effects 0.000 claims abstract description 8
- 230000009466 transformation Effects 0.000 claims abstract description 4
- 230000000007 visual effect Effects 0.000 claims abstract description 4
- 238000003062 neural network model Methods 0.000 claims description 35
- 238000012545 processing Methods 0.000 claims description 25
- 238000005265 energy consumption Methods 0.000 claims description 22
- 238000005457 optimization Methods 0.000 claims description 19
- 230000011218 segmentation Effects 0.000 claims description 12
- 238000012549 training Methods 0.000 claims description 12
- 238000011156 evaluation Methods 0.000 claims description 9
- 238000012360 testing method Methods 0.000 claims description 8
- 238000012795 verification Methods 0.000 claims description 8
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000003321 amplification Effects 0.000 claims description 5
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 5
- 230000036285 pathological change Effects 0.000 claims description 5
- 231100000915 pathological change Toxicity 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 230000008901 benefit Effects 0.000 claims description 3
- 238000005562 fading Methods 0.000 claims description 3
- 238000000638 solvent extraction Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 description 13
- 238000012986 modification Methods 0.000 description 6
- 230000004048 modification Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000003708 edge detection Methods 0.000 description 4
- 201000004569 Blindness Diseases 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000006735 deficit Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 229910052760 oxygen Inorganic materials 0.000 description 1
- 239000001301 oxygen Substances 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30041—Eye; Retina; Ophthalmic
Abstract
The invention discloses a cloud edge cooperative retinopathy detection system of premature infants, which comprises the following components: the system comprises premature infant retina image acquisition equipment, a detection application end and a cloud server; wherein: the premature infant retina image acquisition equipment is used for acquiring retina images of multiple visual angles of the premature infant and sending the retina images to the detection application end; the retinal image constitutes a dataset; the detection application end is arranged on the edge equipment, and performs preprocessing operation on the retina image of the premature infant through the edge equipment, wherein the preprocessing operation comprises geometric transformation and/or image enhancement; the detection application end comprises an image detection module, a medical science popularization module and an information exchange module; the cloud server comprises a user information database and a lesion detection model, and the content in the user information database can be sent to a detection application end through data transmission and rendered on a page of the detection application end. Corresponding methods, electronic devices, and computer-readable storage media are also disclosed.
Description
Technical Field
The invention belongs to the technical fields of computers, intelligent medical treatment and image processing, and particularly relates to a cloud-edge cooperative retinopathy detection system and a cloud-edge cooperative retinopathy detection method for premature infants.
Background
Retinopathy of prematurity (Retinopathy of Prematurity, ROP) is one of the most important causes of blindness and vision impairment in children, and timely screening, early identification and intervention can effectively prevent blindness caused by retinopathy of prematurity. Oxygen and retinopathy prevention and control guidelines for treatment of premature infants (revised edition) published in 2016 in China state: ROP screening must be performed on premature infants with birth ages of 34 weeks or less or with birth mass <2000 g. At present, the ROP screening work is mainly carried out by experienced ophthalmologists to carry out binocular indirect fundus examination, and the screening equipment and the experienced ROP screening ophthalmologists are indispensable. However, there are problems that occur when screening resources are unevenly distributed worldwide, so that premature infants in the basic level or remote area cannot get the condition of immediate eyeground screening aggravation or even blindness, for example, the screening problems are found in the research of the retinopathy prevention and control system of premature infants in China: ROP screening in various domestic areas is quite different, and personnel, equipment and technology restrict the development of ROP screening.
Artificial intelligence has been applied to the medical field, but there still exists a disadvantage in the application of screening retinopathy of prematurity, such as single data source, insufficient self-adaption and self-optimization capabilities, limited screening accuracy by equipment, insufficient protection of privacy information of patients, and the like.
Disclosure of Invention
The invention aims to provide a cloud-edge cooperative retinopathy detection system and a cloud-edge cooperative retinopathy detection method for premature infants, which are used for constructing Bian Yun cooperative framework for a premature infant ROP screening system, establishing a system self-adaptive self-optimization management mechanism, improving universality of an ROP intelligent detection model and enabling the ROP intelligent detection model to have self-adaption and self-optimization capabilities.
In one aspect, the invention provides a cloud-edge cooperative retinopathy of prematurity detection system, comprising: the system comprises premature infant retina image acquisition equipment, a detection application end and a cloud server; wherein:
The premature infant retina image acquisition equipment is used for acquiring retina images of multiple visual angles of the premature infant and sending the retina images to the detection application end; retinal images of multiple views of the premature infant acquired over a period of time form a dataset;
The detection application end is arranged on the edge equipment, and performs preprocessing operation on the premature infant retina image through the edge equipment, wherein the preprocessing operation comprises geometric transformation and/or image enhancement; the detection application end comprises an image detection module, a medical science popularization module and an information exchange module;
The cloud server comprises a user information database and a lesion detection model, namely the cloud server deploys the user information database and the trained lesion detection model, and the content in the user information database can be sent to the detection application end through data transmission and rendered on a page of the detection application end.
Preferably, the image detection module includes:
the edge end lesion detection model is used for locally executing the detection task of the retina image;
a transmitting unit, configured to transmit data to the cloud server, where the data includes data of the retinal image and an intermediate result in a detection process;
And the receiving unit is used for receiving the detection result of the retina image, which is transmitted back from the cloud.
Preferably, the image detection module includes three main detection modes of an edge detection mode, a collaborative detection mode and a cloud detection mode and an auxiliary retrieval mode, wherein: selecting one of three main detection modes according to the requirements of a user, and inputting the retina image into a trained neural network model for detection;
In the edge end detection mode, the retina image is only processed on edge equipment and is not uploaded to the cloud server;
In a collaborative detection mode, a trained neural network model is divided into a first part and a second part according to the current network condition and the current task quantity, the retina image is processed by the first part model on the edge equipment to obtain the intermediate result, the intermediate result is uploaded to the cloud server to finish the detection, the detection result is returned, and the segmentation position of the lesion detection model is determined through calculation of the task execution time delay;
In a cloud detection mode, uploading the retina image to the cloud server for processing, encrypting and uploading local data after removing user information, detecting an image by a pathological change detection model of the cloud, and updating the cloud model in time;
and aiming at the retina image with larger detection difficulty, the auxiliary search mode logs in a special account number of the detection application terminal by an expert to manually detect the retina image, and sends a detection result to a user corresponding to the retina image.
Preferably, the system further comprises a case report output module, wherein the case report output module is used for forming an auxiliary detection result according to the retinopathy analysis result of the retinopathy analysis module of the premature infant, and forming a detection report through the confirmation, modification and/or input of a doctor's advice.
Preferably, the information exchange module is connected with the cloud server, and the user publishes content through the information exchange module, wherein the content is stored in a user information database on the cloud server, so that information exchange and sharing among different users are realized.
Preferably, the lesion detection model is obtained by processing and analyzing a data set, and the establishing of the lesion detection model comprises the following steps:
Dividing the data set into a training set, a verification set and a test set;
Inputting the images in the training set into a neural network model, and adjusting a first parameter of the neural network model;
Inputting the verification set into a neural network model, and adjusting a second parameter of the neural network model;
Inputting the test set into a neural network model, and evaluating the neural network model to finally obtain the lesion detection model for the premature infant retina.
The neural network model is evaluated by adopting four performance evaluation indexes of Accuracy (Accuracy), precision (Precision), recall rate (Recall) and comprehensive evaluation index (F1-Measure), which are specifically defined as follows:
among them, TP (True Positive): predicting the positive class as a positive class number; TN (True Negative): predicting the negative class as a negative class number; FP (False Positive): predicting negative classes as positive class numbers, and misreporting; FN (FALSE NEGATIVE ): predicting positive class as negative class number, and missing report;
And applying a neural network model with good performance evaluation indexes as the retinopathy detection model of the premature infant.
The second aspect of the invention provides a cloud-edge cooperative retinopathy of prematurity detection method, comprising the following steps:
constructing a model, and taking the premature infant retina image acquired by the image acquisition equipment as a data set; carrying out data preprocessing on the data in the data set; the problem of data imbalance is solved for the preprocessed data through data amplification operation; inputting the data amplified into a neural network model for training to obtain a model for detecting the pathological changes of the premature retina, and deploying the model into an edge device and a cloud server;
the method comprises the steps of lesion detection, selecting a proper detection mode according to user requirements in consideration of detection time delay, energy consumption and user privacy requirements, selecting proper model segmentation points for Bian Yun collaborative detection, inputting a retina image of the premature infant to be detected into a trained lesion detection model for detection, and processing a case with high detection difficulty by an expert at a detection application end manually; comprising the following steps:
selecting the optimal model segmentation point, thereby maximizing the advantage of cooperative detection; the selection of the model partitioning points in the detection mode is based on the transmission delay and the energy consumption of the whole system, and when the task is executed, the processing delay is as follows:
wherein U represents the task data amount, p represents the CPU cycle number required by processing each bit of task, and f c represents the computing power of the edge device;
The data transmission rate r LU of the edge device sending the task to the cloud and the data transmission rate r LD of the cloud returning the result to the edge device are as follows:
Wherein B represents a bandwidth between the edge device and the cloud server, t -r represents a channel coefficient between the edge device and the cloud server, d represents a distance between the edge device and the cloud server, r represents a fading factor of the channel, and σ 2 represents noise power of the channel;
The time delay t U of the edge device uploading the task to the cloud, the time delay d D of the cloud returning the calculation result to the edge end and the processing time t E of the task are respectively as follows:
The total time for completing the task is as follows:
t＝tU+tD+tE
The total energy consumption generated by the whole system for processing user tasks is as follows:
E＝EL+EU
e L is energy consumption generated by an edge device CPU; e U is energy consumption generated when the edge device unloads the task to cloud processing;
Setting a problem optimization target as the minimum weighted sum of task completion time and energy consumption, obtaining an optimization problem as shown below, and selecting a proper model segmentation node according to the result of the optimization problem:
Wherein t is not less than t max,E≤Emax, and omega is not less than 0 and not more than 1.
As a preferred embodiment, further comprising:
model optimization, wherein the cloud server receives local data of a plurality of edge devices with user privacy information removed, continuously optimizes the lesion detection model based on the local data, analyzes and learns all retinopathy data characteristics by cloud, and makes self-adaption and self-optimization adjustment according to a general model after constructing the lesion detection model for each regional crowd.
A third aspect of the invention provides an electronic device comprising a processor and a memory, the memory storing a plurality of instructions, the processor being for reading the instructions and performing the method according to the first aspect.
A fourth aspect of the invention provides a computer readable storage medium storing a plurality of instructions readable by a processor and for performing the method of the first aspect.
The method, the device, the electronic equipment and the computer readable storage medium provided by the invention have the following beneficial technical effects:
According to the system and the method for detecting the retinopathy of the premature infant by the edge cloud cooperation, provided by the invention, an edge cloud cooperation framework mode of diagnosis and treatment of the retinopathy of the premature infant is constructed, and the self-adaptive capacity of the system is improved; the user can communicate with the detection personnel through the lesion detection application end; and the personal information of the user is removed by uploading the cloud data, so that the privacy of the user is protected to the greatest extent.
Drawings
Fig. 1 is a schematic block diagram of a cloud-side cooperative retinopathy of prematurity detection system according to a preferred embodiment of the present invention;
FIG. 2 is a flow chart illustrating an implementation of a cloud-side cooperative retinopathy of prematurity detection system in accordance with a preferred embodiment of the present invention;
FIG. 3 is a block diagram of the image acquisition module of the edge-cloud collaborative retinopathy of prematurity detection system according to a preferred embodiment of the present invention;
fig. 4 is a schematic illustration of a cooperative detection of a cloud-cooperative retinopathy of prematurity detection system, according to a preferred embodiment of the present invention;
FIG. 5 is a TensorFlow model conversion flow diagram of a side-cloud collaborative retinopathy of prematurity detection system according to a preferred embodiment of the present invention;
fig. 6 is a schematic structural diagram of an embodiment of an electronic device according to the present invention.
Detailed Description
The following describes in further detail the embodiments of the present invention with reference to the drawings and examples. The following examples are illustrative of the invention and are not intended to limit the scope of the invention.
Example 1
A cloud-edge coordinated retinopathy of prematurity detection system comprising: the system comprises premature infant retina image acquisition equipment, a detection application end, a cloud server and a case report output module; wherein:
The premature infant retina image acquisition equipment is used for acquiring retina images of multiple visual angles of the premature infant and sending the retina images to the detection application end; retinal images of multiple views of the premature infant acquired over a period of time form a dataset;
The detection application end is arranged on the edge equipment, and performs preprocessing operation on the premature infant retina image through the edge equipment, wherein the preprocessing operation comprises geometric transformation and/or image enhancement; the detection application end comprises an image detection module, a medical science popularization module and an information exchange module;
the cloud server comprises a user information database and a lesion detection model, namely the cloud server deploys the user information database and the trained lesion detection model, and the content in the user information database can be sent to the detection application end through data transmission and rendered on a page of the detection application end.
As a preferred embodiment, the image detection module includes:
the edge end lesion detection model is used for locally executing the detection task of the retina image;
a transmitting unit, configured to transmit data to the cloud server, where the data includes data of the retinal image and an intermediate result in a detection process;
And the receiving unit is used for receiving the detection result of the retina image, which is transmitted back from the cloud.
As a preferred embodiment, the image detection module includes three main detection modes including an edge detection mode, a collaborative detection mode and a cloud detection mode, and an auxiliary search mode, wherein: selecting one of three main detection modes according to the requirements of a user, and inputting the retina image into a trained neural network model for detection;
In the edge detection mode, the retina image is only processed on edge equipment and is not uploaded to the cloud server, and the mode is used for meeting the privacy requirements of users; the edge end detection mode periodically acquires required data and models from cloud resources to update and optimize the local model;
In a collaborative detection mode, a trained neural network model is divided into a first part and a second part according to the current network condition and the current task amount, the retina image is processed by the first part model on the edge equipment to obtain the intermediate result, the intermediate result is uploaded to the cloud server to finish the detection, and the detection result is returned, and the mode aims at minimum time delay and energy consumption, meets the requirement of a user on the detection speed, and reduces the energy consumption of a system; according to the cooperative detection mode, the segmentation position of the lesion detection model is determined through calculation of task execution time delay, so that edge cloud cooperation is realized;
In a cloud detection mode, uploading the retina image to the cloud server for processing, wherein the mode can make up for the deficiency of computing capacity of edge equipment and reduce energy consumption of the edge equipment; the cloud detection mode is used for encrypting and uploading the local data after removing the user information, the lesion detection model of the cloud detects the image, and meanwhile, the cloud model is updated in time, so that the data are effectively processed, and the privacy safety of the local data is also protected;
and aiming at the retina image with larger detection difficulty, the auxiliary search mode logs in a special account number of the detection application terminal by an expert to manually detect the retina image, and sends a detection result to a user corresponding to the retina image.
As a preferred embodiment, the system further comprises a case report output module, wherein the case report output module is used for forming an auxiliary detection result according to the retinopathy analysis result of the retinopathy analysis module of the premature infant, and forming a detection report through the confirmation, modification and/or input of a doctor's advice.
In an embodiment, the information exchange module is connected with the cloud server, and the user publishes content through the information exchange module, wherein the content is stored in a user information database on the cloud server, so that information exchange and sharing among different users are realized.
As a preferred embodiment, the lesion detection model is obtained by processing and analyzing a data set, in this example, retinopathy of prematurity images of premature infants obtained from a plurality of hospital databases for nearly five years are formed into the data set, and the establishing of the lesion detection model includes:
Dividing the data set into a training set, a verification set and a test set;
Inputting the images in the training set into a neural network model, and adjusting a first parameter of the neural network model;
Inputting the verification set into a neural network model, and adjusting a second parameter of the neural network model;
Inputting the test set into a neural network model, and evaluating the neural network model to finally obtain the lesion detection model for the premature infant retina.
As a preferred embodiment, the lesion detection model is built with TensorFlow Lite, tensorFlow Lite is a framework for a mobile device with TensorFlow, all tools required for converting TensorFlow the model and running TensorFlow the model on a mobile, embedded (embeded) and internet of things (IoT) devices are provided, a user is allowed to run TensorFlow the model on various devices, tensorFlow Lite to efficiently execute the model on various devices, a special format is adopted, tensorFlow the model must be converted into the format before being used by TensorFlow Lite, tensorFlow Lite the lesion detection model is converted into the format and loaded to the edge, and the lesion detection model can be locally run to perform image processing tasks.
Example two
As shown in fig. 2, a cloud-edge cooperative retinopathy of prematurity detection method includes:
Constructing a model, and taking the premature infant retina image acquired by the image acquisition equipment as a data set; carrying out data preprocessing on the data in the data set; the problem of data imbalance is solved for the preprocessed data through data amplification operation; inputting the data amplified into a neural network model for training to obtain a model for detecting the pathological changes of the premature retina, and deploying the model into an edge device and a cloud server; data amplification in order to prevent overfitting of the model and enhance generalization ability of the model, data is amplified offline during training to increase diversity of the data, and methods of data amplification include random inversion of all pixels, random up-down/left-right inversion, random gaussian blur, random translation, random rotation, random contrast enhancement, and application of mixup data enhancement algorithms.
The method comprises the steps of lesion detection, selecting a proper detection mode according to user requirements in consideration of detection time delay, energy consumption and user privacy requirements, selecting proper model segmentation points for Bian Yun collaborative detection, inputting a retina image of a premature infant to be detected into a trained lesion detection model for detection, and processing a case with high detection difficulty by an expert through manual work at a detection application end.
As a preferred embodiment, further comprising:
And model optimization, wherein the cloud server receives local data from which the user privacy information is removed, continuously optimizes the lesion detection model based on the local data, and performs self-adaption and self-optimization adjustment according to a general model after the edge equipment acquires the lesion detection model on the cloud service.
In this embodiment, the dataset is obtained from a hospital database, the dataset is divided into a training set, a verification set and a test set, the image in the training set is input into a neural network model to adjust parameters of the model, the verification set is input into the neural network model to adjust super parameters of the model, the test set is input into the neural network model to evaluate the model, and in order to quantitatively evaluate the performance of the neural network model, four evaluation indexes including an Accuracy (Accuracy), a Precision (Precision), a Recall (Recall) and a comprehensive evaluation index (F1-Measure) are adopted, wherein the four evaluation indexes are specifically defined as follows:
Among them, TP (True Positive): predicting the positive class as a positive class number; TN (True Negative): predicting the negative class as a negative class number; FP (False Positive): predicting negative classes as positive class numbers, and misreporting; FN (FALSE NEGATIVE ): and predicting positive class as negative class number, and missing report.
As a preferred embodiment, a model with good performance index is used as the retinopathy of prematurity detection model.
As a preferred implementation manner of this embodiment, the image acquisition device is used for acquiring retinal images of multiple views of the premature infant, and the data set of the detection application end is acquired from the device; the detection application end installed on the edge equipment comprises an image detection module, a medical science popularization module and an information exchange module; the cloud server deploys a user information database and a trained lesion detection model.
As a preferred implementation manner of this embodiment, as shown in fig. 3, the image detection module includes: the edge end lesion detection model is used for locally executing an image detection task; the sending unit is used for transmitting data to the cloud, including image data and intermediate results; and the receiving unit is used for receiving the detection result returned from the cloud.
As shown in fig. 4, different model segmentation points generate different time delays and energy consumption, so that an optimal model segmentation point needs to be selected, thereby maximizing the advantage of collaborative detection.
As a preferred implementation manner of this embodiment, the selection of the model splitting point in the cooperative detection mode considers the transmission delay and the energy consumption of the whole system, and when the task is executed, the processing delay is as follows:
Where U represents the amount of task data, p represents the number of CPU cycles required to process each bit of task, and f c represents the computing power of the edge device.
The data transmission rate r LU of the edge device sending the task to the cloud and the data transmission rate r LD of the cloud returning the result to the edge device are as follows:
Wherein B represents a bandwidth between the edge device and the cloud server, d -r represents a channel coefficient between the edge device and the cloud server, d represents a distance between the edge device and the cloud server, r represents a fading factor of the channel, and σ 2 represents noise power of the channel.
Therefore, the time delay t U of the edge device uploading the task to the cloud, the time delay t D of the cloud returning the calculation result to the edge end, and the processing time t E of the task are respectively:
The total time for completing the task is as follows:
t＝tU+tD+tE
The total energy consumption generated by the whole system for processing user tasks is as follows:
E＝EL+EU
E L is energy consumption generated by an edge device CPU; e U is energy consumption generated when the edge device offloads the task to the cloud processing.
As a preferred embodiment, the problem optimization target may be set to be that the weighted sum of the task completion time and the energy consumption is the smallest, so as to obtain an optimization problem as shown below, and a suitable model segmentation node is selected according to the result of the optimization problem:
Wherein t is not less than t max,E≤Emax, and omega is not less than 0 and not more than 1.
As a preferred embodiment, the lesion detection model of the edge detection mode is obtained by converting the lesion detection model through a TensorFlow Lite converter on the basis of a trained TensorFlow model. TensorFlow Lite is a framework for mobile devices TensorFlow, providing a translation TensorFlow model, and running all tools required for TensorFlow models on edge (mobile), embedded (embeded), and internet of things (IoT) devices, allowing users to run TensorFlow models on a variety of devices. TensorFlow Lite in order to efficiently execute models on a variety of devices, a special format is employed into which the TensorFlow model must be converted before it can be used by TensorFlow Lite.
As shown in FIG. 5, tensorFlow Lite provides a C++ application program interface on the Android, IOS, linux system, java application program interfaces on the Android and Linux systems, and Python application program interfaces on the Linux operating system. The model conversion process of the edge-end lesion diagnosis and treatment model firstly converts the trained TensorFlow model into a tflite format which is available in TensorFlow Lite, and then deploys a TensorFlow Lite format file into the edge equipment through an application program interface.
As a preferred embodiment, the edge device needs to perform self-adaptive and self-optimized adjustment on cloud resources and detection models, so that the utilization rate and universality of the cloud resources and models can be better improved, local data of a plurality of edge devices are uploaded to the cloud for further improving the sensitivity and specificity of intelligent detection of the edge device, all retinopathy data features are analyzed and learned by the cloud, a premature infant retinopathy detection model for each regional crowd is constructed, and optimization and improvement of system performance are realized.
The invention also provides a memory, which stores a plurality of instructions for implementing the method according to the second embodiment.
As shown in fig. 6, the present invention further provides an electronic device, including a processor 301 and a memory 302 connected to the processor 301, where the memory 302 stores a plurality of instructions, and the instructions may be loaded and executed by the processor, so that the processor can perform the method as described in the second embodiment.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention. It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (7)
1. A cloud-edge coordinated retinopathy of prematurity detection system, comprising: the system comprises premature infant retina image acquisition equipment, a detection application end and a cloud server; wherein:
the premature infant retina image acquisition equipment is used for acquiring retina images of multiple visual angles of the premature infant and sending the retina images to the detection application end; retinal images of the premature infant from multiple perspectives form a dataset;
The detection application end is arranged on the edge equipment, and performs preprocessing operation on the premature infant retina image through the edge equipment, wherein the preprocessing operation comprises geometric transformation and/or image enhancement; the detection application end comprises an image detection module, a medical science popularization module and an information exchange module;
The cloud server comprises a user information database and a lesion detection model, wherein the content in the user information database can be sent to the detection application end through data transmission and rendered on a page of the detection application end;
The image detection module includes:
the edge end lesion detection model is used for locally executing the detection task of the retina image;
a transmitting unit, configured to transmit data to the cloud server, where the data includes data of the retinal image and an intermediate result in a detection process;
The receiving unit is used for receiving the detection result of the retina image, which is transmitted back from the cloud;
the image detection module comprises three main detection modes of an edge end detection mode, a cooperative detection mode and a cloud detection mode and an auxiliary retrieval mode, wherein: selecting one of three main detection modes according to the requirements of a user, and inputting the retina image into a trained neural network model for detection;
In the edge end detection mode, the retina image is only processed on edge equipment and is not uploaded to the cloud server;
In a collaborative detection mode, a trained neural network model is divided into a first part and a second part according to the current network condition and the current task quantity, the retina image is processed by the first part model on the edge equipment to obtain the intermediate result, the intermediate result is uploaded to the cloud server to finish the detection, the detection result is returned, and the segmentation position of the lesion detection model is determined through calculation of the task execution time delay;
In a cloud detection mode, uploading the retina image to the cloud server for processing, encrypting and uploading local data after removing user information, detecting an image by a pathological change detection model of the cloud, and updating the cloud model in time;
The auxiliary retrieval mode is used for logging in a special account number of the detection application terminal by an expert for manually detecting the retina image with higher detection difficulty, and sending a detection result to a user corresponding to the retina image;
the lesion detection model is obtained by processing and analyzing a data set, and the establishment of the lesion detection model comprises the following steps:
Dividing the data set into a training set, a verification set and a test set;
inputting the images in the training set into a neural network model, and adjusting first parameters of the neural network model;
Inputting the verification set into a neural network model, and adjusting a second parameter of the neural network model;
inputting the test set into a neural network model, and evaluating the neural network model to finally obtain the lesion detection model for the premature infant retina;
The neural network model is evaluated by adopting four performance evaluation indexes of Accuracy (Accuracy), precision (Precision), recall rate (Recall) and comprehensive evaluation index (F1-Measure), which are specifically defined as follows:
Wherein, TP (TruePositive, true): predicting the positive class as a positive class number; TN (TrueNegative ): predicting the negative class as a negative class number; FP (FalsePositive ): predicting negative classes as positive class numbers, and misreporting; FN (FALSENEGATIVE ): predicting positive class as negative class number, and missing report;
and applying the neural network model with good performance evaluation index as a premature infant retinopathy detection model.
2. The cloud-edge collaborative retinopathy of prematurity detection system according to claim 1, further comprising a case report output module for forming an auxiliary detection result from the retinopathy analysis result of the retinopathy of prematurity analysis module, the detection report being formed by a doctor confirming, modifying and/or inputting orders.
3. The cloud-edge collaborative retinopathy of prematurity detection system according to claim 1, wherein the information exchange module is connected with the cloud server, and a user publishes content through the information exchange module, wherein the content is stored in a user information database on the cloud server, so that information exchange and sharing among different users are realized.
4. A cloud-edge cooperative retinopathy of prematurity detection method, implemented based on the system of any one of claims 1-3, comprising:
constructing a model, and taking the premature infant retina image acquired by the image acquisition equipment as a data set; carrying out data preprocessing on the data in the data set; the problem of data imbalance is solved for the preprocessed data through data amplification operation; inputting the data amplified into a neural network model for training to obtain a model for detecting the pathological changes of the premature retina, and deploying the model into an edge device and a cloud server;
the method comprises the steps of lesion detection, selecting a proper detection mode according to user requirements in consideration of detection time delay, energy consumption and user privacy requirements, selecting proper model segmentation points for Bian Yun collaborative detection, inputting a retina image of the premature infant to be detected into a trained lesion detection model for detection, and processing a case with high detection difficulty by an expert at a detection application end manually; comprising the following steps:
Selecting the optimal model partitioning point, so as to exert the advantage of cooperative detection to the maximum extent; the selection of the model partitioning points in the detection mode is based on the transmission delay and the energy consumption of the whole system, and when the task is executed, the processing delay is as follows:
wherein U represents the task data amount, p represents the CPU cycle number required by processing each bit of task, and f c represents the computing power of the edge device;
The data transmission rate r LU of the edge device sending the task to the cloud and the data transmission rate r LD of the cloud returning the result to the edge device are as follows:
wherein B represents a bandwidth between the edge device and the cloud server, d -r represents a channel coefficient between the edge device and the cloud server, d represents a distance between the edge device and the cloud server, r represents a fading factor of the channel, and σ 2 represents noise power of the channel;
The time delay t U of the edge device uploading the task to the cloud, the time delay t D of the cloud returning the calculation result to the edge end and the processing time t E of the task are respectively:
The total time for completing the task is as follows:
t＝tU+tD+tE
The total energy consumption generated by the whole system for processing user tasks is as follows:
E＝EL+EU
e L is energy consumption generated by an edge device CPU; e U is energy consumption generated when the edge device unloads the task to cloud processing;
Setting a problem optimization target as the minimum weighted sum of task completion time and energy consumption, obtaining an optimization problem as shown below, and selecting a proper model segmentation node according to the result of the optimization problem:
Wherein t is not less than t max,E≤Emax, and omega is not less than 0 and not more than 1.
5. The method of detecting according to claim 4, further comprising:
model optimization, wherein the cloud server receives local data of a plurality of edge devices with user privacy information removed, continuously optimizes the lesion detection model based on the local data, analyzes and learns all retinopathy data characteristics by cloud, and makes self-adaption and self-optimization adjustment according to a general model after constructing the lesion detection model for each regional crowd.
6. An electronic device comprising a processor and a memory, the memory storing a plurality of instructions, the processor configured to read the instructions and perform the method of any of claims 4-5.
7. A computer readable storage medium storing a plurality of instructions readable by a processor and for performing the method of any one of claims 4-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210460497.XA CN114841952B (en) | 2022-04-28 | 2022-04-28 | Cloud-edge cooperative retinopathy of prematurity detection system and detection method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210460497.XA CN114841952B (en) | 2022-04-28 | 2022-04-28 | Cloud-edge cooperative retinopathy of prematurity detection system and detection method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114841952A CN114841952A (en) | 2022-08-02 |
| CN114841952B true CN114841952B (en) | 2024-05-03 |
Family
ID=82567391
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210460497.XA Active CN114841952B (en) | 2022-04-28 | 2022-04-28 | Cloud-edge cooperative retinopathy of prematurity detection system and detection method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114841952B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110010219A (en) * | 2019-03-13 | 2019-07-12 | 杭州电子科技大学 | Optical coherence tomography image retinopathy intelligent checking system and detection method |
| CN111127425A (en) * | 2019-12-23 | 2020-05-08 | 北京至真互联网技术有限公司 | Target detection positioning method and device based on retina fundus image |
| CN111585916A (en) * | 2019-12-26 | 2020-08-25 | 国网辽宁省电力有限公司电力科学研究院 | LTE electric power wireless private network task unloading and resource allocation method based on cloud edge cooperation |
| CN112465789A (en) * | 2020-12-02 | 2021-03-09 | 智程工场(佛山)科技有限公司 | Detection method and system for retinopathy plus disease of premature infant |
| CN112801959A (en) * | 2021-01-18 | 2021-05-14 | 华南理工大学 | Auxiliary assembly system based on visual feature recognition |
| CN113067873A (en) * | 2021-03-19 | 2021-07-02 | 北京邮电大学 | Edge cloud collaborative optimization method based on deep reinforcement learning |
| CN113282348A (en) * | 2021-05-26 | 2021-08-20 | 浙江理工大学 | Edge calculation task unloading system and method based on block chain |
| CN113419867A (en) * | 2021-08-23 | 2021-09-21 | 浙大城市学院 | Energy-saving service supply method in edge-oriented cloud collaborative computing environment |
| CN114022723A (en) * | 2021-09-18 | 2022-02-08 | 华南理工大学 | Data set generation method and device for neural network training |
| CN114093505A (en) * | 2021-11-17 | 2022-02-25 | 山东省计算中心（国家超级计算济南中心） | Cloud-edge-end-architecture-based pathological detection system and method |
-
2022
- 2022-04-28 CN CN202210460497.XA patent/CN114841952B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110010219A (en) * | 2019-03-13 | 2019-07-12 | 杭州电子科技大学 | Optical coherence tomography image retinopathy intelligent checking system and detection method |
| CN111127425A (en) * | 2019-12-23 | 2020-05-08 | 北京至真互联网技术有限公司 | Target detection positioning method and device based on retina fundus image |
| CN111585916A (en) * | 2019-12-26 | 2020-08-25 | 国网辽宁省电力有限公司电力科学研究院 | LTE electric power wireless private network task unloading and resource allocation method based on cloud edge cooperation |
| CN112465789A (en) * | 2020-12-02 | 2021-03-09 | 智程工场(佛山)科技有限公司 | Detection method and system for retinopathy plus disease of premature infant |
| CN112801959A (en) * | 2021-01-18 | 2021-05-14 | 华南理工大学 | Auxiliary assembly system based on visual feature recognition |
| CN113067873A (en) * | 2021-03-19 | 2021-07-02 | 北京邮电大学 | Edge cloud collaborative optimization method based on deep reinforcement learning |
| CN113282348A (en) * | 2021-05-26 | 2021-08-20 | 浙江理工大学 | Edge calculation task unloading system and method based on block chain |
| CN113419867A (en) * | 2021-08-23 | 2021-09-21 | 浙大城市学院 | Energy-saving service supply method in edge-oriented cloud collaborative computing environment |
| CN114022723A (en) * | 2021-09-18 | 2022-02-08 | 华南理工大学 | Data set generation method and device for neural network training |
| CN114093505A (en) * | 2021-11-17 | 2022-02-25 | 山东省计算中心（国家超级计算济南中心） | Cloud-edge-end-architecture-based pathological detection system and method |
Non-Patent Citations (2)
| Title |
|---|
| Deep neural networks to predict diabetic retinopathy;Thippa Reddy Gadekallu 等;《Springer》;20200424;第5407页-5420页 * |
| Diagnosis of Diabetic Retinopathy Using Deep Neural Networks;ZHENTAO GAO等;《IEEE》;20190111;第3360页-第3370页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114841952A (en) | 2022-08-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3933551A1 (en) | Motor imagery electroencephalogram signal processing method, device, and storage medium | |
| KR20200005986A (en) | System and method for diagnosing cognitive impairment using face recognization | |
| US10360482B1 (en) | Crowd-sourced artificial intelligence image processing services | |
| WO2023178737A1 (en) | Spiking neural network-based data enhancement method and apparatus | |
| CN116631564B (en) | Emergency electronic medical record management system and management method | |
| Wang et al. | Fedacs: Federated skewness analytics in heterogeneous decentralized data environments | |
| CN113536697A (en) | Bearing residual life prediction method based on improved residual error network and WGAN | |
| CN116311539B (en) | Sleep motion capturing method, device, equipment and storage medium based on millimeter waves | |
| CN116664930A (en) | Personalized federal learning image classification method and system based on self-supervision contrast learning | |
| JP7400449B2 (en) | Analysis equipment, analysis method, and analysis program | |
| CN115168669A (en) | Infectious disease screening method and device, terminal equipment and medium | |
| Chen et al. | A new optimal diagnosis system for coronavirus (COVID-19) diagnosis based on Archimedes optimization algorithm on chest X-ray images | |
| CN114358250A (en) | Data processing method, data processing apparatus, computer device, medium, and program product | |
| CN114841952B (en) | Cloud-edge cooperative retinopathy of prematurity detection system and detection method | |
| CN116569194A (en) | Joint learning | |
| JP2020067680A (en) | Image diagnostic system of embryo, image diagnostic program of embryo and creating method of image diagnostic discriminator of embryo | |
| Serhani et al. | Dynamic Data Sample Selection and Scheduling in Edge Federated Learning | |
| CN111477336A (en) | Fusion method, system and storage medium for infectious disease diagnosis data | |
| Nam et al. | Selective prediction with long short-term memory using unit-wise batch standardization for time series health data sets: algorithm development and validation | |
| CN111680798A (en) | Joint learning model system and method, apparatus, and computer-readable storage medium | |
| CN115171896A (en) | System and method for predicting long-term death risk of critically ill patient | |
| JP2021068336A (en) | Image diagnostic system for embryo, image diagnosis program for embryo, and method of creating diagnostic imaging classifier for embryo | |
| Alattab et al. | A Novel Multi-Objective Learning Algorithm for Disease Identification and Classification in Electronic Healthcare System | |
| Karling et al. | Prediction of Breast Cancer Using Machine Learning Techniques for Health Data | |
| Vavekanand | SUBMIP: Smart Human Body Health Prediction Application System Based on Medical Image Processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |