CN110809037B - Internet of things skin mirror system based on depth multivariate features - Google Patents
Internet of things skin mirror system based on depth multivariate features Download PDFInfo
- Publication number
- CN110809037B CN110809037B CN201911034898.3A CN201911034898A CN110809037B CN 110809037 B CN110809037 B CN 110809037B CN 201911034898 A CN201911034898 A CN 201911034898A CN 110809037 B CN110809037 B CN 110809037B
- Authority
- CN
- China
- Prior art keywords
- skin
- dermatoscope
- skin disease
- module
- expert
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/44—Detecting, measuring or recording for evaluating the integumentary system, e.g. skin, hair or nails
- A61B5/441—Skin evaluation, e.g. for skin disorder diagnosis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H80/00—ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Abstract
The invention discloses an Internet of things dermatoscope system based on depth multivariate characteristics, which comprises a handheld dermatoscope terminal, a dermatoscope server, an online expert system and a user application module, wherein the handheld dermatoscope terminal realizes wireless communication by arranging a 4G Internet of things card in the handheld dermatoscope terminal, so that the use of equipment is not limited by a regional place, and a remote high-performance server is called for computer-assisted medical diagnosis; the dermatosis model based on the depth multivariate characteristics in the dermatoscope server realizes a higher-universality dermatosis detection method by combining the depth characteristics and the multivariate characteristics selected manually. The invention can solve the problems that the skin patient is inconvenient to go to the hospital for examination and the remote area where the hospital is located cannot be provided with high-performance skin mirror equipment, improves the effectiveness of computer-assisted medical diagnosis and integrates medical resources in different areas.
Description
Technical Field
The invention relates to the technical field of medical image processing, in particular to an Internet of things skin mirror system based on depth multivariate characteristics.
Background
With the development of deep learning in the field of medical image processing, hospitals and scientific research institutions gradually begin to use computer technology to assist medical diagnosis. In diagnostic means of skin diseases, there have been a lot of studies on intelligent identification based on a dermatoscope image. Due to limitations of the skin mirror examination device, some remote clinics cannot be equipped with the skin mirror examination device for skin mirror examination, and patients with inconvenient actions are difficult to go to hospitals for examination, so that the application of the skin mirror examination technology is limited.
Aiming at the problems of a skin mirror device, the mainstream devices at present are of two types, one type is a small handheld skin mirror, the size is small, the handheld skin mirror is convenient to carry, but only amplification and data acquisition are carried out on skin, the auxiliary diagnosis cannot be combined with a computer technology, and the instant diagnosis cannot be carried out; the other type is a large medical skin mirror which comprises a computer, a main control console, a handheld collector, a display and other modules, is inconvenient to move in a large range and can only be used in professional hospitals. There is therefore a need to provide a portable dermoscope system and computer-aided diagnosis via a remote high-performance server.
In addition, facing the problem of skin lesion classification, current classification is mainly based on two broad categories of features: the method comprises the steps of manually selecting multiple features and deeply learning extracted depth features. The manually selected multi-feature is obtained by a large number of experiments, research, summarization and extraction based on experts, and under the condition of having a large amount of prior knowledge, the proper feature can be selected in a targeted manner to achieve the classification effect; the depth features extracted by deep learning are trained through a depth network and a large amount of data, and the most suitable features are screened out through a large amount of calculation, so that the method is suitable for scenes with a large amount of data. For complex practical application scenarios, a single class of features cannot cope with numerous situations, and a method with higher compatibility needs to be provided for identification and classification.
In summary, the internet of things technology can be utilized to enable the handheld dermoscope device to access and use a remote high-performance server to assist diagnosis, and meanwhile, a more universal dermoscope detection technology is provided in the server by combining manual features and depth features.
Disclosure of Invention
The invention aims to overcome the defects and shortcomings of the prior art, provides an Internet of things dermatoscope system based on depth multivariate features, connects a handheld dermatoscope terminal and a dermatoscope server through the Internet of things technology, has the advantages of portability and computer-aided diagnosis, solves the problems that a single feature classification algorithm is limited in application range and cannot adapt to most production environments, and enables the computer-aided diagnosis technology to be widely applied.
In order to achieve the purpose, the technical scheme provided by the invention is as follows: an internet of things dermatoscope system based on depth multivariate features, comprising:
the handheld dermatoscope terminal is used for collecting dermatoscope images of skin patients for medical staff, is connected to the dermatoscope server and the online expert system through the Internet of things communication module, and can provide the medical staff with the operations of diagnosis confirmation, dermatoscope image collection, dermatoscope data uploading, online expert calling and diagnosis result checking;
the skin mirror server is set up by a server with the memory of not less than 32GB, the video memory of not less than 12GB and the CPU core number of not less than 8 cores and is used for storing and managing skin mirror image data acquired by collecting skin mirror images by the handheld skin mirror terminal, the skin mirror server can preprocess the skin mirror image data, detect skin diseases of the preprocessed skin mirror image data or train a skin disease model of the preprocessed skin mirror image data and provide a called API (application program interface) for the handheld skin mirror terminal;
the on-line expert system is used for managing the skin disease expert information, responding to an on-line expert call initiated by the handheld dermatoscope terminal, and providing a web interface for managing a diagnosis process and collecting and managing a local dermatoscope database;
and the user application module is used for managing the personal medical record of the patient, reserving the on-site examination, inquiring the information of the hospital and checking the diagnosis result.
Further, handheld dermatoscope terminal includes dermatoscope image acquisition unit, system application module, thing networking communication module, touch-sensitive screen, wherein:
the skin mirror image acquisition unit is used for acquiring a skin mirror image, consists of an optical amplification element, a camera and a digital image acquisition card, amplifies the surface of the skin through the optical amplification element, and acquires the skin mirror image into a digital image through the camera and the digital image acquisition card to form a skin mirror image;
the system application module consists of an operating system and application software, wherein the operating system is used for providing a calling interface of bottom hardware, and comprises a skin mirror image acquisition unit, an Internet of things communication module and a calling interface of a touch screen; the application software provides high-level application capability based on bottom hardware, has the flow control of the dermatoscope image acquisition, calls an API (application program interface) module of a dermatoscope server by utilizing an Internet of things communication module and initiates an online expert call;
the Internet of things communication module comprises a 4G Internet of things card and an Internet of things communication module, the 4G Internet of things card has a networking function, is connected to a wireless network through the Internet of things communication module, is connected to the dermatoscope server and the online expert system by using a wireless private network technology, can ensure that data transmission cannot be intercepted and stolen illegally by using the wireless private network technology, and simultaneously ensures the stability of transmission;
the touch screen is responsible for providing instruction input and display, and medical staff can carry out the operations of diagnosis confirmation, dermatoscope image acquisition, dermatoscope data uploading, online expert calling and diagnosis result viewing according to prompt information through the touch screen.
Further, the dermatoscope server comprises a skin disease model based on the depth multivariate features, an image preprocessing module, a skin disease detection module, a skin disease model training module, an API (application program interface) module and a database management module, wherein:
the skin disease model based on the depth multivariate characteristics is built by python language and is a model based on a depth neural network and combined with manually selected multivariate characteristics, the skin disease model comprises a model structure and a model parameter, the model structure comprises two inputs, the first is a three-channel image with fixed size, and the second is manually selected multivariate characteristics and comprises LBP (local binary pattern) map characteristics and gray level co-occurrence matrix characteristics; based on a deep neural network, the skin disease model can be used for rapidly converging model parameters by training and excavating depth characteristics and combining multiple characteristics selected manually, so that a usable prediction result is obtained when a few skin mirror images are available, and an accurate prediction result is obtained when a large number of skin mirror images are available; the skin disease model performs four times of convolution, activation and pooling on an input three-channel image with a fixed size to obtain a depth characteristic, fuses the depth characteristic with an LBP map characteristic in an input manually-selected multivariate characteristic, performs three times of convolution, activation and pooling again, fuses with a gray level co-occurrence matrix characteristic in the manually-selected multivariate characteristic to obtain a depth multivariate characteristic, and classifies the depth multivariate characteristic through an SVM classifier to obtain the output of the skin disease model based on the depth multivariate characteristic; the model parameters are obtained by performing skin disease model training through a skin disease model training module and are used for skin disease detection through a skin disease detection module;
the image preprocessing module is responsible for preprocessing a dermatoscope image acquired by a handheld dermatoscope terminal, and because the dermatoscope image acquired by the handheld dermatoscope terminal is not suitable for being directly used for skin disease detection, the dermatoscope image needs to be normalized into a three-channel image with fixed size suitable for computer processing, and LBP (local binary pattern) features and gray level co-occurrence matrix features of the three-channel image are extracted to form manually selected multi-element features;
the skin disease detection module is responsible for inputting a three-channel image with a fixed size obtained after pretreatment of the image pretreatment module and manually selected multivariate features into a skin disease model based on the depth multivariate features, executing skin disease detection, obtaining the output of the skin disease model based on the depth multivariate features as a prediction result, and storing the prediction result;
the skin disease model training module is responsible for inputting three-channel images with fixed sizes obtained after pretreatment of the image pretreatment module and manually selected multivariate features into a skin disease model based on the depth multivariate features, executing skin disease model training to obtain model parameters of the skin disease model based on the depth multivariate features, and storing the model parameters;
the API interface module is responsible for managing and providing API interfaces and providing services for the handheld dermatoscope terminal and the online expert system in a Web Service mode, the handheld dermatoscope terminal and the online expert system can upload dermatoscope data, start dermatosis detection and start dermatosis model training through the API interface module, and the API interface module can also record the times and time for each user to call each API interface;
the database management module is used for storing and managing the operation record of the image preprocessing module, storing and managing the operation record of the skin disease detection module, storing and managing the operation record of the skin disease model training module, storing and managing the skin mirror image collected by the handheld skin mirror terminal, storing and managing the three-channel image with fixed size and the manually selected multi-element characteristics which are obtained after the image preprocessing module is preprocessed.
Further, the online expert system comprises an expert management module, a diagnosis process management module and a skin disease database, wherein:
the expert management module manages a skin disease expert information database, realizes the real-name system of skin disease experts, records the diagnosis result of the skin disease experts, receives an online expert call initiated by a handheld dermatoscope terminal, pushes the online expert call to the skin disease experts in the skin disease expert information database, responds to the online expert call in an online order grabbing manner by the skin disease experts, and records the frequency of each skin disease expert responding to the online expert call as the reference basis for the performance of the skin disease experts;
the diagnosis process management module makes a skin disease diagnosis process, firstly, a treatment number is generated when a skin patient makes an on-line appointment or arrives at a hospital for skin disease detection, the skin patient performs skin disease detection according to the treatment number during treatment, after a skin mirror image is collected through a handheld skin mirror terminal, an on-line expert call is initiated, the on-line expert call is connected to a skin mirror server for skin mirror data uploading, if a skin disease expert responds, the on-line diagnosis is confirmed in combination with a prediction result of a skin disease detection module of the skin mirror server, and a diagnosis result is obtained; if no response of the skin disease expert is given within the set time, the prediction result of the skin disease detection module is used as an initial diagnosis result, the diagnosis result corresponding to the diagnosis number is updated after the skin disease expert in the skin disease expert information database confirms the initial diagnosis result, and the diagnosis result is pushed to the user application module;
the dermatosis database is used for recording the treatment number, the dermatoscope image and the incidence relation between the dermatosis expert and the diagnosis result, periodically uploading the dermatoscope image and the diagnosis result to the dermatoscope server, updating the dermatoscope image through the database management module of the dermatoscope server, and starting the dermatosis model training by using the API interface module of the dermatoscope server.
Further, the user application module is a mobile phone application program, including personal medical record management, appointment visiting inspection, hospital information inquiry and diagnosis result pushing, wherein:
the personal medical record management is used for checking the diagnosis result of the historical skin disease detection for the skin patient and providing the historical reference opinions for the skin disease expert;
the appointment in-door inspection is used for appointment of medical staff to go out to detect skin diseases, and is beneficial to skin patients who are inconvenient to move or go to a hospital in person to contact with the medical staff to see a doctor in time;
the hospital information inquiry provides nearby hospital guide for skin patients, checks related equipment of the hospital and checks the information of dermatologists;
and the diagnosis result pushing unit is responsible for pushing the diagnosis result of the skin disease detection and related treatment opinions to the skin patient.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the portability of the terminal equipment is guaranteed while the high-performance server is used for operation by utilizing the internet of things technology, the data transmission can be guaranteed not to be intercepted and stolen illegally by utilizing the wireless private network technology, and meanwhile, the transmission stability is guaranteed.
2. The skin disease detection and the skin disease model training are carried out by utilizing the special dermatoscope server, the service is provided for the handheld dermatoscope terminal and the online expert system in a WebService form, the training and the detection work can be quickly completed on the premise of not influencing the user experience, the feedback is given in time, the operation pressure of the terminal equipment can be effectively reduced, and the use experience of the handheld dermatoscope terminal is improved.
3. The skin disease model adopted by the invention is a model based on a deep neural network and combined with manually selected multivariate features, the skin disease model can rapidly converge model parameters by training and excavating the deep features and combined with the manually selected multivariate features, a usable prediction result is obtained when a few skin mirror images are available, an accurate prediction result is obtained when a large number of skin mirror images are available, the model can be suitable for most production environments on the premise of not changing the model, the rapid deployment of a system is convenient, and the construction cost is effectively reduced.
Drawings
Fig. 1 is a relationship diagram of each module of the internet of things skin mirror system in the embodiment.
Fig. 2 is a schematic diagram of a calling process of each module of the internet of things skin mirror system in the embodiment.
Fig. 3 is a schematic diagram of a skin disease model based on a depth multivariate feature.
Fig. 4 is a flow chart of a skin disease diagnosis.
Detailed Description
The present invention will be further described with reference to specific examples.
As shown in fig. 1 and fig. 2, the internet of things dermatoscope system based on depth multivariate features provided in this embodiment is an overall solution including hardware and software, a hospital headquarters deploys a high-performance server with a memory of not less than 32GB, a video memory of not less than 12GB, a CPU core number of not less than 8 cores, each hospital branch deploys an application server, and configures a plurality of handheld dermatoscope terminals according to the scale of the hospital where the hospital is located, the handheld dermatoscope terminals are connected to a network through an internet of things card, API interfaces of the dermatoscope server and an online expert system are called in a WebService form, and the dermatoscope server performs dermatosis detection by calling a Python program.
Specifically, thing networking dermatoscope system, including:
the handheld dermatoscope terminal is used for collecting dermatoscope images of skin patients for medical staff, is connected to the dermatoscope server and the online expert system through the Internet of things communication module, and can provide the medical staff with the operations of diagnosis confirmation, dermatoscope image collection, dermatoscope data uploading, online expert calling and diagnosis result checking;
the system comprises a dermatoscope server, a hospital headquarters is provided with a server with a memory of 32GB, a display memory of 12GB, an 8-core CPU, a 256GB system hard disk and a 2TB data hard disk, and the server is used for storing and managing dermatoscope image data acquired by collecting dermatoscope images of a handheld dermatoscope terminal;
the system comprises an online expert system, a server with a 32GB, 4-core CPU, 256GB system hard disk and 1TB data hard disk, which is configured in each hospital, is used for managing the skin disease expert information, responding to an online expert call initiated by a handheld dermatoscope terminal, and providing a web interface for managing a diagnosis process, collecting and managing a local dermatoscope database;
and the user application module provides services for the skin patient in an Android application program mode, and is used for managing personal medical records of the patient, reserving the on-site examination, inquiring hospital information and checking diagnosis results.
Handheld dermatoscope terminal includes dermatoscope image acquisition unit, system application module, thing networking communication module, touch-sensitive screen, wherein:
the skin mirror image acquisition unit consists of an optical amplification element, a CMOS camera with 200 ten thousand pixels and an electronic acquisition card, the surface of the skin is amplified by the optical amplification element and is acquired into a digital image by the camera and the digital image acquisition card to form a skin mirror image;
the system application module consists of an operating system and application software, wherein the operating system of the handheld dermatoscope terminal adopts a Linux kernel and is used for providing a calling interface of bottom hardware, and the Linux kernel comprises a dermatoscope image acquisition unit, an Internet of things communication module and a calling interface of a touch screen; the application software provides high-level application capability based on bottom hardware, has the flow control of the dermatoscope image acquisition, calls an API (application program interface) module of a dermatoscope server by utilizing an Internet of things communication module and initiates an online expert call;
the Internet of things communication module comprises a 4G Internet of things card and an Internet of things communication module, the Internet of things card adopts a China Mobile Internet of things card, a 6G flow monthly package is opened, the flows of the Internet of things cards in the same hospital are shared, the handheld dermatoscope terminal is connected to a wireless network through the Internet of things communication module, the handheld dermatoscope terminal is connected to a dermatoscope server and an online expert system through a wireless private network technology, the wireless private network technology can be used for ensuring that data transmission cannot be intercepted and stolen illegally, and meanwhile, the transmission stability is ensured;
the touch screen is responsible for providing instruction input and display, and medical staff can carry out the operations of diagnosis confirmation, dermatoscope image acquisition, dermatoscope data uploading, online expert calling and diagnosis result viewing according to prompt information through the touch screen.
The dermatoscope detection server module includes the skin disease model based on the many first characteristics of degree of depth, image preprocessing module, skin disease detection module, skin disease model training module, API interface module, database management module, wherein:
the skin disease model based on the depth multivariate characteristics is built by python language and is a model based on a depth neural network and combined with manually selected multivariate characteristics, the skin disease model comprises a model structure and a model parameter, the model structure is shown in figure 3 and comprises two inputs, the first is a three-channel image with fixed size, and the second is the manually selected multivariate characteristics and comprises LBP (local binary pattern) map characteristics and gray level co-occurrence matrix characteristics; based on a deep neural network, the skin disease model can be used for rapidly converging model parameters by training and excavating depth characteristics and combining multiple characteristics selected manually, so that a usable prediction result is obtained when a few skin mirror images are available, and an accurate prediction result is obtained when a large number of skin mirror images are available; the skin disease model performs four times of convolution, activation and pooling on an input three-channel image with a fixed size to obtain a depth characteristic, fuses the depth characteristic with an LBP map characteristic in an input manually-selected multivariate characteristic, performs three times of convolution, activation and pooling again, fuses with a gray level co-occurrence matrix characteristic in the manually-selected multivariate characteristic to obtain a depth multivariate characteristic, and classifies the depth multivariate characteristic through an SVM classifier to obtain the output of the skin disease model based on the depth multivariate characteristic; the model parameters are obtained by performing skin disease model training through a skin disease model training module and are used for skin disease detection through a skin disease detection module;
the image preprocessing module is responsible for preprocessing a dermatoscope image acquired by a handheld dermatoscope terminal, and because the dermatoscope image acquired by the handheld dermatoscope terminal is not suitable for being directly used for dermatosis detection, the dermatoscope image needs to be normalized into a three-channel image with the length of 150 pixels and the width of 150 pixels, and LBP (local binary pattern) map characteristics and gray level co-occurrence matrix characteristics of the three-channel image are extracted to form manually selected multi-element characteristics;
the skin disease detection module is responsible for inputting a three-channel image with a fixed size obtained after pretreatment of the image pretreatment module and manually selected multivariate features into a skin disease model based on the depth multivariate features, executing skin disease detection, obtaining the output of the skin disease model based on the depth multivariate features as a prediction result, and storing the prediction result;
the skin disease model training module is responsible for inputting three-channel images with fixed sizes obtained after pretreatment of the image pretreatment module and manually selected multivariate features into a skin disease model based on the depth multivariate features, executing skin disease model training to obtain model parameters of the skin disease model based on the depth multivariate features, and storing the model parameters;
the API interface module is responsible for managing and providing API interfaces and providing services for the handheld dermatoscope terminal and the online expert system in a Web Service mode, the handheld dermatoscope terminal and the online expert system can upload dermatoscope data, start dermatosis detection and start dermatosis model training through the API interface module, and the API interface module can also record the times and time for each user to call each API interface;
the database management module stores and maintains the operation record of the image preprocessing module, the operation record of the skin disease detection module and the operation record of the skin disease model training module by using an Oracle database; the skin mirror image collected by the handheld skin mirror terminal, the three-channel image with the fixed size obtained after pretreatment of the image pretreatment module and the manually selected multi-element features are stored in a data hard disk, and index information is stored and managed by using an Oracle database.
The online expert module comprises an expert management module, a diagnosis process management module and a skin disease database, wherein:
the expert management module manages a skin disease expert information database, realizes the real-name system of skin disease experts, records the diagnosis result of the skin disease experts, receives an online expert call initiated by a handheld dermatoscope terminal, pushes the online expert call to the skin disease experts in the skin disease expert information database, responds to the online expert call in an online order grabbing manner by the skin disease experts, and records the frequency of each skin disease expert responding to the online expert call as the reference basis for the performance of the skin disease experts;
the diagnosis process management module makes a skin disease diagnosis process, as shown in fig. 4, a skin patient generates a treatment number when making an online appointment through the user application module or performing skin disease detection in a hospital, the skin patient performs skin disease detection according to the treatment number when in treatment, medical staff initiates an online expert call to an online expert system after acquiring a skin mirror image through a handheld skin mirror terminal, and connects the acquired skin mirror image to a skin mirror server to upload skin mirror data, if the skin disease expert responds, the diagnosis is confirmed online by combining the prediction result of the skin disease detection module of the skin mirror server, and a diagnosis result is obtained; if no response of the skin disease expert is found in a short time, the prediction result of the skin disease detection module is used as an initial diagnosis result, the diagnosis result corresponding to the diagnosis number is updated after the skin disease expert in the skin disease expert information database confirms the initial diagnosis result, and the diagnosis result is pushed to the user application module;
the dermatosis database records the treatment number, the dermatoscope image and the incidence relation between the dermatoscope expert and the diagnosis result by using an Oracle database, the system defaults to upload the dermatoscope image and the diagnosis result to a dermatoscope server every month, administrators of hospital departments can manually upload the dermatoscope data to the dermatoscope server at any time, the dermatosis expert of the hospital headquarters verifies the reliability of newly uploaded data through a database management module of the dermatoscope server every month, the dermatoscope image is updated, and the API interface module of the dermatoscope server is used for starting the dermatosis model training.
The user application module is an Android application program written by Java language, and the user application module is used by skin patients through mobile phones and comprises personal medical record management, appointment visiting inspection, hospital information inquiry and diagnosis result pushing, wherein:
the personal medical record management is used for checking the diagnosis result of the historical skin disease detection for the skin patient, and providing the historical reference opinions for the skin disease expert when the skin patient is examined for the skin disease;
the appointment attendance check is used for reserving medical staff to attend to skin disease detection, and nearby hospitals can receive and arrange the medical staff and feed back secondary confirmation appointment conditions to skin patients, so that the skin patients who are inconvenient to move or go to the hospitals in person can contact the medical staff to see a doctor in time;
the hospital information inquiry provides nearby hospital guide for skin patients, checks related equipment of the hospital and checks the information of dermatologists;
and the diagnosis result pushing unit is responsible for pushing the diagnosis result of the skin disease detection and related treatment opinions to the skin patient.
The above-mentioned embodiments are merely preferred embodiments of the present invention, and the scope of the present invention is not limited thereto, so that the changes in the shape and principle of the present invention should be covered within the protection scope of the present invention.
Claims (4)
1. An internet of things skin mirror system based on depth multivariate features, comprising:
the handheld dermatoscope terminal is used for collecting dermatoscope images of skin patients for medical staff, is connected to the dermatoscope server and the online expert system through the Internet of things communication module, and can provide the medical staff with the operations of diagnosis confirmation, dermatoscope image collection, dermatoscope data uploading, online expert calling and diagnosis result checking;
the skin mirror server is set up by a server with the memory of not less than 32GB, the video memory of not less than 12GB and the CPU core number of not less than 8 cores and is used for storing and managing skin mirror image data acquired by collecting skin mirror images by the handheld skin mirror terminal, the skin mirror server can preprocess the skin mirror image data, detect skin diseases of the preprocessed skin mirror image data or train a skin disease model of the preprocessed skin mirror image data and provide a called API (application program interface) for the handheld skin mirror terminal;
the on-line expert system is used for managing the skin disease expert information, responding to an on-line expert call initiated by the handheld dermatoscope terminal, and providing a web interface for managing a diagnosis process and collecting and managing a local dermatoscope database;
the user application module is used for managing personal medical records of patients, reserving to check at home, inquiring hospital information and checking diagnosis results;
the dermatoscope server comprises a skin disease model based on the depth multivariate characteristics, an image preprocessing module, a skin disease detection module, a skin disease model training module, an API (application program interface) module and a database management module, wherein:
the skin disease model based on the depth multivariate characteristics is built by python language and is a model based on a depth neural network and combined with manually selected multivariate characteristics, the skin disease model comprises a model structure and a model parameter, the model structure comprises two inputs, the first is a three-channel image with fixed size, and the second is manually selected multivariate characteristics and comprises LBP (local binary pattern) map characteristics and gray level co-occurrence matrix characteristics; based on a deep neural network, the skin disease model can be used for rapidly converging model parameters by training and excavating depth characteristics and combining multiple characteristics selected manually, so that a usable prediction result is obtained when a few skin mirror images are available, and an accurate prediction result is obtained when a large number of skin mirror images are available; the skin disease model performs four times of convolution, activation and pooling on an input three-channel image with a fixed size to obtain a depth characteristic, fuses the depth characteristic with an LBP map characteristic in an input manually-selected multivariate characteristic, performs three times of convolution, activation and pooling again, fuses with a gray level co-occurrence matrix characteristic in the manually-selected multivariate characteristic to obtain a depth multivariate characteristic, and classifies the depth multivariate characteristic through an SVM classifier to obtain the output of the skin disease model based on the depth multivariate characteristic; the model parameters are obtained by performing skin disease model training through a skin disease model training module and are used for skin disease detection through a skin disease detection module;
the image preprocessing module is responsible for preprocessing a dermatoscope image acquired by a handheld dermatoscope terminal, and because the dermatoscope image acquired by the handheld dermatoscope terminal is not suitable for being directly used for skin disease detection, the dermatoscope image needs to be normalized into a three-channel image with fixed size suitable for computer processing, and LBP (local binary pattern) features and gray level co-occurrence matrix features of the three-channel image are extracted to form manually selected multi-element features;
the skin disease detection module is responsible for inputting a three-channel image with a fixed size obtained after pretreatment of the image pretreatment module and manually selected multivariate features into a skin disease model based on the depth multivariate features, executing skin disease detection, obtaining the output of the skin disease model based on the depth multivariate features as a prediction result, and storing the prediction result;
the skin disease model training module is responsible for inputting three-channel images with fixed sizes obtained after pretreatment of the image pretreatment module and manually selected multivariate features into a skin disease model based on the depth multivariate features, executing skin disease model training to obtain model parameters of the skin disease model based on the depth multivariate features, and storing the model parameters;
the API interface module is responsible for managing and providing API interfaces and providing services for the handheld dermatoscope terminal and the online expert system in a Web Service mode, the handheld dermatoscope terminal and the online expert system can upload dermatoscope data, start dermatosis detection and start dermatosis model training through the API interface module, and the API interface module can also record the times and time for each user to call each API interface;
the database management module is used for storing and managing the operation record of the image preprocessing module, storing and managing the operation record of the skin disease detection module, storing and managing the operation record of the skin disease model training module, storing and managing the skin mirror image collected by the handheld skin mirror terminal, storing and managing the three-channel image with fixed size and the manually selected multi-element characteristics which are obtained after the image preprocessing module is preprocessed.
2. The internet of things dermatome system based on the depth multivariate feature as claimed in claim 1, wherein: handheld dermatoscope terminal includes dermatoscope image acquisition unit, system application module, thing networking communication module, touch-sensitive screen, wherein:
the skin mirror image acquisition unit is used for acquiring a skin mirror image, consists of an optical amplification element, a camera and a digital image acquisition card, amplifies the surface of the skin through the optical amplification element, and acquires the skin mirror image into a digital image through the camera and the digital image acquisition card to form a skin mirror image;
the system application module consists of an operating system and application software, wherein the operating system is used for providing a calling interface of bottom hardware, and comprises a skin mirror image acquisition unit, an Internet of things communication module and a calling interface of a touch screen; the application software provides high-level application capability based on bottom hardware, has the flow control of the dermatoscope image acquisition, calls an API (application program interface) module of a dermatoscope server by utilizing an Internet of things communication module and initiates an online expert call;
the Internet of things communication module comprises a 4G Internet of things card and an Internet of things communication module, the 4G Internet of things card has a networking function, is connected to a wireless network through the Internet of things communication module, is connected to the dermatoscope server and the online expert system by using a wireless private network technology, can ensure that data transmission cannot be intercepted and stolen illegally by using the wireless private network technology, and simultaneously ensures the stability of transmission;
the touch screen is responsible for providing instruction input and display, and medical staff can carry out the operations of diagnosis confirmation, dermatoscope image acquisition, dermatoscope data uploading, online expert calling and diagnosis result viewing according to prompt information through the touch screen.
3. The internet of things dermatome system based on the depth multivariate feature as claimed in claim 1, wherein: the online expert system comprises an expert management module, a diagnosis process management module and a skin disease database, wherein:
the expert management module manages a skin disease expert information database, realizes the real-name system of skin disease experts, records the diagnosis result of the skin disease experts, receives an online expert call initiated by a handheld dermatoscope terminal, pushes the online expert call to the skin disease experts in the skin disease expert information database, responds to the online expert call in an online order grabbing manner by the skin disease experts, and records the frequency of each skin disease expert responding to the online expert call as the reference basis for the performance of the skin disease experts;
the diagnosis process management module makes a skin disease diagnosis process, firstly, a treatment number is generated when a skin patient makes an on-line appointment or arrives at a hospital for skin disease detection, the skin patient performs skin disease detection according to the treatment number during treatment, after a skin mirror image is collected through a handheld skin mirror terminal, an on-line expert call is initiated, the on-line expert call is connected to a skin mirror server for skin mirror data uploading, if a skin disease expert responds, the on-line diagnosis is confirmed in combination with a prediction result of a skin disease detection module of the skin mirror server, and a diagnosis result is obtained; if no response of the skin disease expert is given within the set time, the prediction result of the skin disease detection module is used as an initial diagnosis result, the diagnosis result corresponding to the diagnosis number is updated after the skin disease expert in the skin disease expert information database confirms the initial diagnosis result, and the diagnosis result is pushed to the user application module;
the dermatosis database is used for recording the treatment number, the dermatoscope image and the incidence relation between the dermatosis expert and the diagnosis result, periodically uploading the dermatoscope image and the diagnosis result to the dermatoscope server, updating the dermatoscope image through the database management module of the dermatoscope server, and starting the dermatosis model training by using the API interface module of the dermatoscope server.
4. The internet of things dermatome system based on the depth multivariate feature as claimed in claim 1, wherein: the user application module is a mobile phone end application program and comprises personal medical record management, appointment visiting inspection, hospital information inquiry and diagnosis result pushing, wherein:
the personal medical record management is used for checking the diagnosis result of the historical skin disease detection for the skin patient and providing the historical reference opinions for the skin disease expert;
the appointment in-door inspection is used for appointment of medical staff to go out to detect skin diseases, and is beneficial to skin patients who are inconvenient to move or go to a hospital in person to contact with the medical staff to see a doctor in time;
the hospital information inquiry provides nearby hospital guide for skin patients, checks related equipment of the hospital and checks the information of dermatologists;
and the diagnosis result pushing unit is responsible for pushing the diagnosis result of the skin disease detection and related treatment opinions to the skin patient.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911034898.3A CN110809037B (en) | 2019-10-29 | 2019-10-29 | Internet of things skin mirror system based on depth multivariate features |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911034898.3A CN110809037B (en) | 2019-10-29 | 2019-10-29 | Internet of things skin mirror system based on depth multivariate features |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110809037A CN110809037A (en) | 2020-02-18 |
| CN110809037B true CN110809037B (en) | 2020-09-22 |
Family
ID=69489376
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911034898.3A Active CN110809037B (en) | 2019-10-29 | 2019-10-29 | Internet of things skin mirror system based on depth multivariate features |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110809037B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111507414B (en) * | 2020-04-20 | 2022-02-08 | 安徽中科首脑智能医疗研究院有限公司 | Deep learning skin disease picture comparison and classification method, storage medium and robot |
| CN113362933A (en) * | 2021-07-02 | 2021-09-07 | 上海法路源医疗器械有限公司 | Skin optical imaging and remote consultation system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102938026A (en) * | 2012-11-28 | 2013-02-20 | 周纯栋 | Intelligent medical platform |
| CN108197320A (en) * | 2018-02-02 | 2018-06-22 | 北方工业大学 | Multi-view image automatic labeling method |
| CN109272507A (en) * | 2018-07-11 | 2019-01-25 | 武汉科技大学 | The layer dividing method of coherent light faultage image based on structure Random Forest model |
| CN109712705A (en) * | 2018-12-19 | 2019-05-03 | 中国石油大学(华东) | A kind of cholelithiasis intelligent diagnostics APP based on deep learning |
| CN109887589A (en) * | 2019-02-21 | 2019-06-14 | 吉林禾熙科技开发有限公司 | Artificial intelligence medical treatment big data system |
| CN109993103A (en) * | 2019-03-29 | 2019-07-09 | 华南理工大学 | A kind of Human bodys' response method based on point cloud data |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101903522B1 (en) * | 2015-11-25 | 2018-11-23 | 한국전자통신연구원 | The method of search for similar case of multi-dimensional health data and the apparatus of thereof |
| CN106682449A (en) * | 2017-03-01 | 2017-05-17 | 中国科学院苏州生物医学工程技术研究所 | Skin disease map auxiliary diagnosis system based on cloud database |
| US10824938B2 (en) * | 2017-04-24 | 2020-11-03 | Intel Corporation | Specialized fixed function hardware for efficient convolution |
| TWI614624B (en) * | 2017-04-24 | 2018-02-11 | 太豪生醫股份有限公司 | System and method for cloud medical image analyzing |
| US10729907B2 (en) * | 2017-10-20 | 2020-08-04 | Neuropace, Inc. | Systems and methods for clinical decision making for a patient receiving a neuromodulation therapy based on deep learning |
-
2019
- 2019-10-29 CN CN201911034898.3A patent/CN110809037B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102938026A (en) * | 2012-11-28 | 2013-02-20 | 周纯栋 | Intelligent medical platform |
| CN108197320A (en) * | 2018-02-02 | 2018-06-22 | 北方工业大学 | Multi-view image automatic labeling method |
| CN109272507A (en) * | 2018-07-11 | 2019-01-25 | 武汉科技大学 | The layer dividing method of coherent light faultage image based on structure Random Forest model |
| CN109712705A (en) * | 2018-12-19 | 2019-05-03 | 中国石油大学(华东) | A kind of cholelithiasis intelligent diagnostics APP based on deep learning |
| CN109887589A (en) * | 2019-02-21 | 2019-06-14 | 吉林禾熙科技开发有限公司 | Artificial intelligence medical treatment big data system |
| CN109993103A (en) * | 2019-03-29 | 2019-07-09 | 华南理工大学 | A kind of Human bodys' response method based on point cloud data |
Non-Patent Citations (3)
| Title |
|---|
| 《Generative Adversarial Network (GAN) Based Data Augmentation for Palmprint Recognition》;Gengxing Wang等;《2018 Digital Image Computing: Techniques and Applications (DICTA)》;20181213;1-7 * |
| 基于深度学习的皮肤镜图像黑色素瘤检测技术研究;柏朋成;《中国优秀硕士学位论文全文数据库 信息科技辑》;20181215;E072-609 * |
| 深度学习在胎儿超声图像与皮肤镜图像分类中的研究与应用;余镇;《中国优秀硕士学位论文全文数据库 信息科技辑》;20190715;A006-314 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110809037A (en) | 2020-02-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Thomaz et al. | Feasibility of identifying eating moments from first-person images leveraging human computation | |
| CN103690240B (en) | A kind of medical system | |
| CN103324831A (en) | Remote interrogation system | |
| CN103500270A (en) | Visual interaction tele-medicine consultative service system | |
| US9529968B2 (en) | System and method of integrating mobile medical data into a database centric analytical process, and clinical workflow | |
| CN208985713U (en) | Medical information system based on recognition of face | |
| CN110809037B (en) | Internet of things skin mirror system based on depth multivariate features | |
| Chakraborty et al. | Identification of chronic wound status under tele-wound network through smartphone | |
| CN113707264A (en) | Medicine recommendation method, device, equipment and medium based on machine learning | |
| Zuo et al. | Mobile teledermatology: a promising future in clinical practice | |
| US20230133330A1 (en) | Systems and methods for medical resource intelligence | |
| CN111863178A (en) | Method, device, medium and electronic device for issuing medical report | |
| Gezimati et al. | Internet of things enabled framework for terahertz and infrared cancer imaging | |
| CN110970137A (en) | Intelligent cloud nursing system and method based on 5G mobile communication | |
| CN205210954U (en) | Medical information management system | |
| CN102078186B (en) | Physiological parameter collection method, mobile phone and system | |
| CN206788862U (en) | Skin disease collection of illustrative plates assistant diagnosis system based on cloud database | |
| CN108550390A (en) | Intelligent medical treatment manages system and management method | |
| CN108154907A (en) | A kind of medical information input system and its method of work | |
| CN110074759B (en) | Voice data auxiliary diagnosis method, device, computer equipment and storage medium | |
| CN112309028A (en) | Novel self-service diagnosis and treatment machine and self-service diagnosis and treatment method thereof | |
| CN106951697A (en) | A kind of disease treatment scheme supplying system and application method based on cloud data platform | |
| CN105512452A (en) | Remote diagnosis and treatment system with knowledge base management function | |
| CN106709224A (en) | Diabetic retinopathy diagnosis system and method | |
| KR20220145558A (en) | A system, a device and a method of medical questionnaire for a patience |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |