CN113077874B - Intelligent auxiliary diagnosis and treatment system and method for rehabilitation of spine diseases based on infrared thermal images - Google Patents
Intelligent auxiliary diagnosis and treatment system and method for rehabilitation of spine diseases based on infrared thermal images Download PDFInfo
- Publication number
- CN113077874B CN113077874B CN202110293245.8A CN202110293245A CN113077874B CN 113077874 B CN113077874 B CN 113077874B CN 202110293245 A CN202110293245 A CN 202110293245A CN 113077874 B CN113077874 B CN 113077874B
- Authority
- CN
- China
- Prior art keywords
- treatment
- patient
- subsystem
- module
- diagnosis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011282 treatment Methods 0.000 title claims abstract description 190
- 238000003745 diagnosis Methods 0.000 title claims abstract description 65
- 238000000034 method Methods 0.000 title claims abstract description 29
- 201000010099 disease Diseases 0.000 title claims description 21
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 title claims description 21
- 238000011156 evaluation Methods 0.000 claims abstract description 70
- 208000029761 vertebral disease Diseases 0.000 claims abstract description 58
- 230000000694 effects Effects 0.000 claims abstract description 32
- 238000004458 analytical method Methods 0.000 claims abstract description 31
- 208000020307 Spinal disease Diseases 0.000 claims abstract description 9
- 238000001514 detection method Methods 0.000 claims description 24
- 230000007246 mechanism Effects 0.000 claims description 16
- 238000010586 diagram Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 14
- 238000011176 pooling Methods 0.000 claims description 11
- 239000011159 matrix material Substances 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 7
- 238000001931 thermography Methods 0.000 claims description 7
- 229940079593 drug Drugs 0.000 claims description 4
- 239000003814 drug Substances 0.000 claims description 4
- 230000001629 suppression Effects 0.000 claims description 4
- 238000001757 thermogravimetry curve Methods 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 abstract description 2
- 206010050031 Muscle strain Diseases 0.000 description 9
- 238000012549 training Methods 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 5
- 208000015122 neurodegenerative disease Diseases 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 206010052443 Myofascitis Diseases 0.000 description 4
- 230000008859 change Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000002757 inflammatory effect Effects 0.000 description 4
- 230000007850 degeneration Effects 0.000 description 3
- 230000007774 longterm Effects 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 230000036541 health Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 206010041591 Spinal osteoarthritis Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 208000036319 cervical spondylosis Diseases 0.000 description 1
- 230000005786 degenerative changes Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 210000004705 lumbosacral region Anatomy 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013021 overheating Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000005180 public health Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 206010039722 scoliosis Diseases 0.000 description 1
- 208000005801 spondylosis Diseases 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0033—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0033—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room
- A61B5/004—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room adapted for image acquisition of a particular organ or body part
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0062—Arrangements for scanning
- A61B5/0064—Body surface scanning
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0075—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Public Health (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Animal Behavior & Ethology (AREA)
- Heart & Thoracic Surgery (AREA)
- Veterinary Medicine (AREA)
- Surgery (AREA)
- Epidemiology (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Primary Health Care (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
The invention discloses an infrared thermal image-based intelligent auxiliary diagnosis and treatment system for rehabilitation of spinal diseases, which comprises a medical record management subsystem, an intelligent analysis subsystem, a treatment evaluation subsystem and a control center subsystem; the medical record management subsystem comprises a record uploading module, a database module and a search engine module; the medical record management subsystem, the intelligent analysis subsystem and the treatment evaluation subsystem are respectively connected with the control center subsystem. The invention also provides a using method of the intelligent auxiliary diagnosis and treatment system for rehabilitation of the vertebra diseases based on the infrared thermal image. According to the invention, the infrared thermal image of the vertebra part is analyzed by adopting a deep learning method, the rehabilitation condition of the vertebra disease reflected by the infrared thermal image of the vertebra part can be classified and evaluated, and the evaluation can assist doctors in diagnosis and treatment and provide objective treatment effect evaluation reports for patients.
Description
Technical Field
The invention relates to the field of image recognition and intelligent auxiliary diagnosis, in particular to an infrared thermal image-based intelligent auxiliary diagnosis and treatment system and method for rehabilitation of spinal diseases.
Background
The spine diseases refer to diseases related to spine lesions, 97% of middle-aged and elderly people in China have spine diseases according to related investigation, and in recent years, the tendency of younger is shown, and in people under 40 years old, more than 40% of human vertebrates have various diseases, and the incidence rate of the children scoliosis is more than 25%. Spinal diseases have become a public health problem seriously threatening the health of people in China, and rehabilitation process evaluation of patients with spinal diseases has a great influence on treatment effects.
Research shows that the characteristics of the infrared thermal image of the vertebra part can reflect the health condition of the vertebra, the infrared thermal image diagnosis of the vertebra part needs an experienced pathologist to carefully observe the image and make a judgment, the rehabilitation of the vertebra disease is a long-term process, and the infrared thermal image of different rehabilitation treatment stages of a patient is difficult to give objective and dynamic assessment to the long-term procedural treatment effect in a manual judgment mode; in addition, the existing diagnosis and treatment system contains long-term data of various diseases of patients, does not collect the diseases of the spine, is inconvenient in inquiring historical data and infrared thermal imaging images, and does not have the function of intelligent auxiliary diagnosis and treatment.
Accordingly, improvements in the art are needed.
Disclosure of Invention
The invention aims to provide an infrared thermal image-based intelligent auxiliary diagnosis and treatment system and an infrared thermal image-based intelligent auxiliary diagnosis and treatment method for rehabilitation of a spinal disease, which are used for realizing intelligent auxiliary diagnosis and classified assessment of rehabilitation conditions of the spinal disease.
In order to solve the technical problems, the invention provides an infrared thermal imaging-based intelligent auxiliary diagnosis and treatment system for rehabilitation of spinal diseases, which comprises the following components: the medical record management subsystem, the intelligent analysis subsystem, the treatment evaluation subsystem and the control center subsystem;
the medical record management subsystem comprises a record uploading module, a database module and a search engine module;
the intelligent analysis subsystem comprises a target detection module and a classification module; the target detection module comprises an SSD network, an attention mechanism module and a detection module, and an infrared thermogram of a patient obtains a cervical vertebra disease classification result and a lumbar vertebra disease classification result of the patient through the intelligent analysis subsystem;
the treatment evaluation subsystem comprises a single treatment rehabilitation evaluation module and a plurality of treatment process rehabilitation evaluation modules;
the medical record management subsystem, the intelligent analysis subsystem and the treatment evaluation subsystem are respectively connected with the control center subsystem.
As an improvement of the intelligent auxiliary diagnosis and treatment system for rehabilitation of the spine diseases based on infrared thermal imaging, the invention comprises the following steps:
the back of each convolution layer Conv4_3, conv7, conv8_2, conv9_2, conv10_2 and Conv11_2 of the SSD network is respectively connected with an attention mechanism model, then each attention mechanism model is connected with a detection module, and the detection module comprises a non-maximum value suppression method; feature map F E R output by each convolution layer of SSD network C×H×W Respectively inputting attention mechanism models corresponding to all convolution layers of an SSD network to obtain feature images F ', and then inputting all the feature images F ' into a detection module to obtain feature images S ' marked with cervical vertebra areas and lumbar vertebra areas of a patient;
the classification module is an improved AlexNet network obtained by replacing a convolution layer with a convolution kernel size of 5 multiplied by 5 in the AlexNet network by using the acceptance module, and the characteristic diagram S' marked with the cervical vertebra region and the lumbar vertebra region of the patient is subjected to the classification module to obtain the cervical vertebra disease classification result and the lumbar vertebra disease classification result of the patient.
The invention also provides a using method of the intelligent auxiliary diagnosis and treatment system for rehabilitation of the spine diseases based on the infrared thermal image, which comprises the following steps:
step S01, diagnosis and treatment information management
Step S101, recording and updating diagnosis and treatment information
Recording diagnosis and treatment information comprising patient information, infrared thermal image images before and after treatment of the patient's spine diseases, treatment time, treatment scheme, medication scheme and diagnosis description information through a recording and uploading module, and uploading and uniformly storing the information into a database module; all diagnosis and treatment information in the database module form an electronic medical record database, and the infrared thermal image of the patient forms an infrared thermal image set S i ;
The medical record management subsystem processes the diagnosis and treatment information, and establishes, adds or updates single or multiple treatment records of the patient in the database module;
step S102, diagnosis and treatment information searching
The user inputs keywords including 'patient name', 'patient visit card number', 'primary doctor name' or 'primary doctor number' to the search engine module through the control center subsystem;
the search engine module searches and matches each piece of electronic medical record data in the database module according to the input keywords and presents search results on the upper computer;
s02, treatment evaluation:
step S201, evaluation of Single treatment
Acquiring diagnosis and treatment information of a patient through the diagnosis and treatment information search of the step S102;
the single-treatment rehabilitation evaluation module selects a single-treatment record from diagnosis and treatment information of a patient through a control center subsystem, and gathers an infrared thermal image S in the single-treatment record i Inputting into an intelligent analysis subsystem to obtain a cervical vertebra disease classification result and a lumbar vertebra disease classification result of a patient, and then evaluating a single treatment effect:respectively converting the cervical vertebra disease classification result and the lumbar vertebra disease classification result of the patient into 0-4 five digital grades to obtain and present the result of single treatment effect evaluation of the patient on an upper computer;
step S202, multiple treatment evaluation
Acquiring diagnosis and treatment information of a patient through the diagnosis and treatment information search of the step S102;
the multi-treatment rehabilitation evaluation unit selects a plurality of treatment records from the diagnosis and treatment information of the patient through the control center subsystem, and sets an infrared thermal image S in each treatment record of the patient i Inputting into an intelligent analysis subsystem to obtain cervical vertebra disease classification results and lumbar vertebra disease classification results of patients after treatment for each time, and then evaluating the treatment effects for multiple times: and converting the cervical vertebra disease classification result and the lumbar vertebra disease classification result of the patient after each treatment into five digital grades of 0-4, generating a multi-treatment evaluation line graph, and obtaining and presenting the multi-treatment effect evaluation result of the patient on an upper computer.
The beneficial effects of the invention are mainly as follows:
1. according to the invention, the infrared thermal image of the vertebra part is analyzed by adopting a deep learning method, the rehabilitation condition of the vertebra disease reflected by the infrared thermal image of the vertebra part can be classified and evaluated, and the evaluation can assist doctors in diagnosis and treatment and provide objective treatment effect evaluation reports for patients;
2. the invention fully utilizes the characteristics of non-invasiveness, painless, noninvasive and wide applicable crowd of infrared thermal image examination, combines with advanced image processing technology and analysis mode to be suitable for realizing automatic auxiliary diagnosis of the rehabilitation treatment process of the vertebra diseases, thereby rapidly and effectively providing reference diagnosis opinion for doctors, providing objective treatment effect evaluation report for patients and improving the working efficiency of doctors.
Drawings
The following describes the embodiments of the present invention in further detail with reference to the accompanying drawings.
FIG. 1 is a schematic block diagram of an infrared thermal imaging-based intelligent auxiliary diagnosis and treatment system for rehabilitation of spinal diseases;
FIG. 2 is a schematic flow diagram of the intelligent analysis subsystem of FIG. 1;
FIG. 3 is a flow diagram of the treatment evaluation subsystem of FIG. 1;
FIG. 4 is a schematic diagram of a network model of the object detection module in FIG. 1;
FIG. 5 is a schematic diagram of a network model of the classification module of FIG. 1;
fig. 6 is a line graph of multiple treatment assessments.
Detailed Description
The invention will be further described with reference to the following specific examples, but the scope of the invention is not limited thereto:
the embodiment 1, an infrared thermal image-based intelligent auxiliary diagnosis and treatment system for rehabilitation of the spine diseases, as shown in fig. 1, comprises a medical record management subsystem, an intelligent analysis subsystem, a treatment evaluation subsystem and a control center subsystem, wherein a relevant algorithm model is designed according to the characteristics of an infrared thermal image through reasonable database design and system architecture design, and the infrared thermal image-based intelligent auxiliary diagnosis and treatment system for rehabilitation of the spine diseases is built in an upper computer, so that a system for providing tracking, comparison and evaluation functions of rehabilitation treatment process for the spine diseases patients through unified management of patient medical records is realized, and doctors are helped to provide objective rehabilitation evaluation;
the medical record management subsystem comprises a record uploading module, a database module and a search engine module; the record uploading module is used for recording electronic medical record data, wherein the electronic medical record data comprises patient information (including patient name, patient visit card number and the like), main doctor information (including main doctor name, main doctor number and the like), infrared thermography before and after treatment of spine (cervical vertebra and lumbar vertebra) diseases, treatment time, treatment scheme, medication scheme and diagnosis description information, and uploading and uniformly storing single or multiple treatment records (electronic medical record data) of a patient to the database module; the database module is used for storing all electronic medical record data, all diagnosis and treatment information in the database module form an electronic medical record database, and the infrared thermal image of the patient forms an infrared thermal image set S i (i is a patientA patient medical record number); the search engine module can search and match the input relevant information of the patient in the 'patient name' field, the 'patient visit card number' field, the 'mainly used doctor name' field and the 'mainly used doctor number' field of each piece of electronic medical record data in the database module;
the intelligent analysis subsystem comprises a target detection module and a classification module, wherein the target detection module is used for extracting the cervical vertebra part and the lumbar vertebra part in the infrared thermal image of the patient, and the classification module is used for classifying and evaluating the rehabilitation condition of the vertebra diseases;
the target detection module comprises a SSD (Single Shot MultiBox Detector) network, an attention mechanism module and a detection module, as shown in fig. 4, one attention mechanism model is respectively connected behind each convolution layer conv4_3, conv7, conv8_2, conv9_2, conv10_2 and conv11_2 of the SSD network, and then each attention mechanism model is connected with the detection module; the resolution of the input pictures of the SSD network is generally 300×300 or 512×512, the original infrared thermogram resolution of the patient is 256×324, the original infrared thermogram resolution is scaled to 300×300, and then the image is input to the SSD network, the SSD network detects the cervical vertebra area and the lumbar vertebra area by utilizing a multi-scale convolution detection structure, and the characteristic pictures F epsilon R output by each convolution layer (namely Conv4_3, conv7, conv8_2, conv9_2, conv10_2 and Conv11_2) of the SSD network C×H×W Respectively inputting the attention mechanism models corresponding to all convolution layers of the SSD network to obtain a feature map F ', and then inputting the feature map F' output by all the attention mechanism models into a detection module;
the detection module comprises a Non-maximum suppression (NMS) method, the obtained characteristic diagram F ' after passing through an SSD network and an attention mechanism module is provided with a plurality of target boundary frames, the boundary frames with higher overlapping degree are removed through the Non-maximum suppression in the detection module, and the characteristic diagram S ' marked with the cervical vertebra region and the lumbar vertebra region of the patient is obtained, namely the target detection module outputs the characteristic diagram S ' marked with the cervical vertebra region and the lumbar vertebra region of the patient;
SSD network directly detects after extracting features in several convolution layers, obtains the offset of a priori frame to determine the final edgeThe position of the bounding box is faster in operation speed, in addition, the back infrared thermal image is smaller in cervical vertebra area and larger in lumbar vertebra area, a small target and a large target are required to be detected simultaneously, the multi-scale convolution detection structure of the SSD can well meet the requirement of the point, although the SSD network model can extract multi-scale characteristics, the information extracted by each convolution layer has useful infrared image information and some useless background information, and the attention introducing module can further process convolution results to effectively eliminate interference information; in the attention mechanism model, the feature map F ε R C×H×W First pass through channel attention mapping matrix M c ∈R C×1×1 (wherein M c ∈R C×1×1 The calculation method of (1) is as follows: for the input feature map F ε R C×H×W Respectively carrying out maximum pooling treatment and average pooling treatment, then inputting the results of the two pooling treatments into the same multi-layer perceptron, adding the respective outputs, and obtaining a mapping matrix M of the spatial attention after Sigmoid activation function treatment c ∈R C ×1×1 ) The method comprises the steps of carrying out a first treatment on the surface of the By the formulaObtaining a feature map F ', carrying out maximum pooling treatment and average pooling treatment on the feature map F', carrying out convolution operation on the results of the maximum pooling treatment and the average pooling treatment to extract features, and obtaining a spatial attention mapping matrix M after Sigmiod activation function treatment s ∈R 1×H×W Then the feature map F' is passed through a space attention mapping matrix M s ∈R 1×H×W By the formula->Obtaining a characteristic diagram F';
the classification module is based on an improved AlexNet network built by the AlexNet network, the input of the classification module is a characteristic diagram S' marked with the cervical vertebra region and the lumbar vertebra region of a patient, and the output is a cervical vertebra disease classification result and a lumbar vertebra disease classification result (the disease type with the largest probability is used as the classification result), wherein the cervical vertebra disease classification result is sequentially classified into 5 types according to the increment of the severity: normal (j_a), fatigue (j_b), cervical muscle strain or inflammatory change (j_c), cervical overload (j_d), cervical degenerative disease (j_e), lumbar spine disease classification results are sequentially classified into 5 categories according to increasing severity: normal (y_a), fatigue (y_b), myofascitis (y_c), lumbar muscle strain (y_d), lumbar degenerative changes (y_e). For example, for a patient, the cervical vertebra disease classification result is normal (0.02), fatigue (0.02), cervical muscle strain or inflammatory change (0.03), cervical vertebra overload (0.09), cervical vertebra degeneration (0.84), the lumbar vertebra disease classification result is normal (0), fatigue (0.02), myofascitis (0.03), lumbar muscle strain (0.86), lumbar degeneration (0.09), the classification module outputs the cervical vertebra disease classification result of the patient as cervical vertebra degeneration (j_e), the lumbar vertebra disease classification result as lumbar muscle strain (y_d); besides global information, local overheating in infrared thermal imaging is also an important basis for judging disease conditions, so that the network also has the capability of extracting characteristics in multiple scales, an AlexNet network is further improved based on an acceptance module, the AlexNet network sequentially comprises 1 convolution layer with the size of 11 x 11 convolution cores, 1 convolution layer with the size of 5*5 convolution layers, 3 convolution layers with the size of 3*3 convolution layers and 3 full connection layers, and the acceptance module is utilized to replace the convolution layer with the size of 5 x 5 convolution cores in the AlexNet network so as to form a classification module, as shown in fig. 5; the acceptance module comprises a plurality of convolution kernels with different sizes, wherein convolution has two-point effect, namely, the combination of all characteristic information on a certain pixel point is realized, the nonlinearity of the characteristic is improved under the condition that the receptive field is kept unchanged, and the dimension reduction of any scale is carried out according to the number of the convolution kernels passing through a high-dimensional characteristic diagram of a previous layer, so that the operand is reduced; the convolution kernels with different sizes are used for extracting features with different scales, and when the features of the upper layer are input into the acceptance module, the network can automatically select and output information with what scale so as to achieve the effect of extracting the features with multiple scales;
training the intelligent analysis subsystem, configuring an upper computer used for training as a central Linux 7.5.1804 operating system, selecting a CPU with E5-2667 v4@3.20GHz and 8 cores, and 2 GPUs with model number Tesla P4; the training data set is a human body infrared thermal image data set collected in a trimethyl hospital, the resolution of an infrared thermal image is 256 multiplied by 324, the total number of infrared thermal image patterns of 5400 cases of 300 patients is 5400, the 5400 cases of infrared thermal image patterns are all marked by a professional doctor according to the actual condition setting of the patients, the classification of cervical vertebra diseases (normal (J_A), fatigue (J_B), cervical muscle strain or inflammatory change (J_C), cervical vertebra overload (J_D) and cervical vertebra degenerative disease (J_E)) and the classification of lumbar vertebra diseases (normal (Y_A), fatigue (Y_B), myofascitis (Y_C), lumbar muscle strain (Y_D) and lumbar vertebra degenerative disease (Y_E)), and then the marked infrared thermal image patterns are used for data set with the following weight ratio of 7:3 into a training set and a testing set, training an intelligent analysis subsystem by using the training set, setting the batch_size of an AlexNet network to be 32, setting the learning rate to be 5 multiplied by 10 < -5 >, and setting the iteration number to be 40000; setting the batch_size of the SSD network to be 32, setting the learning rate to be 0.0001, setting the iteration number to be 6000, and finally obtaining a trained intelligent analysis subsystem; and then inputting the test set into a trained intelligent analysis subsystem for verification, comparing the cervical vertebra disease classification result and the lumbar vertebra disease classification result of the patient output by the intelligent analysis subsystem with labels set by a professional doctor, and counting that the ratio of the number of correctly classified samples to the total number of samples is 94.69%, thereby meeting the requirement of actual use.
The treatment evaluation subsystem comprises a single treatment recovery evaluation module and a multi-treatment process recovery evaluation module, as shown in fig. 3, the single treatment recovery evaluation module queries the medical record management subsystem through the control center subsystem to obtain a single treatment record, then inputs an infrared thermal image in the single treatment record into the intelligent analysis subsystem to obtain a cervical vertebra disease classification result and a lumbar vertebra disease classification result of a patient, and then evaluates the single treatment effect, so that a single treatment effect evaluation result of the patient is obtained, namely the cervical vertebra disease classification result and the lumbar vertebra disease classification result of the patient are respectively converted into 0-4 five digital grades, and the single treatment effect evaluation results are respectively:
1) Cervical spondylosis: normal (j_a) =0, fatigue (j_b) =1, cervical muscle strain or inflammatory change (j_c) =2, cervical overload (j_d) =3, cervical degenerative disease (j_e) =4;
2) Lumbar disease state: normal (y_a) =0, fatigue (y_b) =1, myofascitis (y_c) =2, lumbar muscle strain (y_d) =3, lumbar degenerative disease (y_e) =4;
the multi-treatment process rehabilitation evaluation module inquires the medical record management subsystem through the control center subsystem to obtain multi-treatment process records, then inputs an infrared thermal image in the multi-treatment process records into the intelligent analysis subsystem to obtain cervical vertebra disease classification results and lumbar vertebra disease classification results of patients after each treatment, and then evaluates the multi-treatment effects of the patients to obtain multi-treatment effect evaluation results of the patients, namely, the cervical vertebra disease classification results and the lumbar vertebra disease classification results of the patients are converted into 0-4 digital grades (consistent with the definition of the five digital grades of the single treatment effect evaluation results), then the digital grade of each patient is converted into a multi-treatment evaluation line graph, the horizontal coordinate is the number of times of treatment of the patients, and the vertical coordinate is the evaluation effect digital grade, so that the evaluation effect of the patients for multiple treatments is displayed clearly.
The control center subsystem is a convergence point of the medical record management subsystem, the intelligent analysis subsystem and the treatment evaluation subsystem, and the medical record management subsystem, the intelligent analysis subsystem and the treatment evaluation subsystem are respectively connected with the control center subsystem.
The application method of the invention is as follows:
step S01, diagnosis and treatment information management
Step S101, recording and updating diagnosis and treatment information
Recording diagnosis and treatment information comprising patient information, infrared thermal image images before and after treatment of the patient's spine diseases, treatment time, treatment scheme, medication scheme and diagnosis description information through a recording and uploading module, and uploading and uniformly storing the information into a database module; all diagnosis and treatment information in the database module form an electronic medical record database, and the infrared thermal image of the patient forms an infrared thermal image set S i ;
The medical record management subsystem processes the diagnosis and treatment information, and establishes, adds or updates single or multiple treatment records of the patient in the database module;
step S102, diagnosis and treatment information searching
The user inputs keywords including 'patient name', 'patient visit card number', 'primary doctor name' or 'primary doctor number' to the search engine module through the control center subsystem;
the search engine module searches and matches each piece of electronic medical record data in the database module according to the input keywords and presents search results on the upper computer;
s02, treatment evaluation:
step S201, evaluation of Single treatment
Acquiring diagnosis and treatment information of a patient through the diagnosis and treatment information search of the step S102;
the single-treatment rehabilitation evaluation module selects a single-treatment record from diagnosis and treatment information of a patient through a control center subsystem, and gathers an infrared thermal image S in the single-treatment record i Inputting the data into an intelligent analysis subsystem to obtain a cervical vertebra disease classification result and a lumbar vertebra disease classification result of a patient, and then evaluating the single treatment effect to obtain and present a single treatment effect evaluation result of the patient on an upper computer;
step S202, multiple treatment evaluation
Acquiring diagnosis and treatment information of a patient through the diagnosis and treatment information search of the step S102;
the multi-treatment rehabilitation evaluation unit selects a plurality of treatment records from the diagnosis and treatment information of the patient through the control center subsystem, and sets an infrared thermal image S in each treatment record of the patient i Inputting the results into an intelligent analysis subsystem to obtain cervical vertebra disease classification results and lumbar vertebra disease classification results of patients after treatment, then evaluating the treatment effects for multiple times to obtain and presenting the results of the evaluation of the treatment effects for multiple times of the patients on an upper computer.
Experiment 1:
taking the training set and the testing set used for training in the embodiment 1 as the data set of the experiment 1, respectively taking the testing set as the input of the intelligent analysis subsystem and other three network models in the invention, and carrying out a comparison classification experiment; in the classification problem, for one of the classes, the class is called Positive, the class is not called Negative, true Positive (TP) represents the Positive classification of Positive, true Negative (TN) represents the Negative classification of Negative, false Positive (FP) represents the Positive classification of Negative errors, false Negative (FN) represents the Negative classification of Positive errors, for example for a 5-classification problem, the confusion matrix for class D is as follows:
accuracy (Accuracy) refers to the ratio of the number of correctly classified samples to the total number of samples,
precision refers to the ratio of the actual positive samples among all samples predicted to be positive.
Recall (Recall), also known as true positive, refers to the ratio of positive samples that are correctly predicted out of all true positive samples,
f1 is an index for comprehensively considering the accuracy and recall,
calculation of accuracy in this experiment: directly calculating the ratio of all the correctly classified samples to the total number of samples;
calculation of average accuracy: first according toCalculating the precision of each category, and then calculating the average value of the precision and the precision as the average precision;
calculation of average recall: first according toCalculating the recall rate of each category, and then calculating the average value of the recall rates as the average recall rate;
calculation of the average F1 value: first according toThe F1 value for each class is calculated and then their average value is calculated as the average F1 value.
Finally, the statistical comparison results are shown in the following table:
under the same experimental data and the same test method, the intelligent analysis subsystem has the highest test accuracy.
Experiment 2:
1. single treatment effect evaluation
Visual Analog Scoring (VAS), commonly used for clinical pain assessment, (patients scored from 0 to 10 according to their pain level, 0-2 for painless comfort, 3-4 for mild pain, 5-6 for pain evident, 7-8 for pain severe, and 9-10 for severe pain) the VAS Score of the patient was divided into 5 categories of numerical scale at intervals of 2: 0-2 is class 0, 3-4 is class 1, 5-6 is class 2, 7-8 is class 3, 9-10 is class 4, corresponding to 5 number classes of results of a single treatment assessment, respectively.
Selecting 300 patients, recording 900 times of treatment, scoring the patients according to actual experiences of the patients after each treatment, performing single treatment evaluation on infrared thermal image images shot after each treatment of the 300 patients to obtain digital grades corresponding to and converted from cervical vertebra disease classification results and lumbar vertebra disease classification results of the patients, performing comparison statistics with the VAS scores of the patients, and if the type of single treatment effect evaluation is consistent with the VAS score type of the patients, considering the patients as correct, otherwise, considering the patients as incorrect, and dividing the correct number of evaluation by the total number of the evaluation to obtain the accuracy: the total number of participation evaluations was 900, with the correct number of evaluations 819 and the accuracy 91%.
2. Multiple treatment effect assessment
The cervical vertebra disease classification result and the lumbar vertebra disease classification result of the patient are converted into digital grades, the digital grade of the patient is converted into a multi-treatment evaluation line graph, the abscissa is the number of times of treatment of the patient, the ordinate is the evaluation effect digital grade and the VAS score, and the trend of the evaluation effect digital grade line of the patient is consistent with the trend of the VAS score after the multi-treatment as shown in figure 6; experimental results demonstrate the effectiveness of the treatment evaluation subsystem.
Finally, it should also be noted that the above list is merely a few specific embodiments of the present invention. Obviously, the invention is not limited to the above embodiments, but many variations are possible. All modifications directly derived or suggested to one skilled in the art from the present disclosure should be considered as being within the scope of the present invention.
Claims (2)
1. The intelligent auxiliary diagnosis and treatment system for the rehabilitation of the vertebra diseases based on the infrared thermal images is characterized by comprising a medical record management subsystem, an intelligent analysis subsystem, a treatment evaluation subsystem and a control center subsystem;
the medical record management subsystem comprises a record uploading module, a database module and a search engine module;
the intelligent analysis subsystem comprises a target detection module and a classification module; the target detection module comprises an SSD network, an attention mechanism module and a detection module, and an infrared thermogram of a patient obtains a cervical vertebra disease classification result and a lumbar vertebra disease classification result of the patient through the intelligent analysis subsystem;
the treatment evaluation subsystem comprises a single treatment rehabilitation evaluation module and a plurality of treatment process rehabilitation evaluation modules;
the medical record management subsystem, the intelligent analysis subsystem and the treatment evaluation subsystem are respectively connected with the control center subsystem;
the back of each convolution layer Conv4_3, conv7, conv8_2, conv9_2, conv10_2 and Conv11_2 of the SSD network is respectively connected with an attention mechanism model, then each attention mechanism model is connected with a detection module, and the detection module comprises a non-maximum value suppression method; feature map F E R output by each convolution layer of SSD network C×H×W Respectively inputting attention mechanism models corresponding to all convolution layers of an SSD network to obtain feature images F ', and then inputting all the feature images F ' into a detection module to obtain feature images S ' marked with cervical vertebra areas and lumbar vertebra areas of a patient; in the attention mechanism model, the feature map F ε R C×H×W First pass through channel attention mapping matrix M c ∈R C×1×1 The method comprises the steps of carrying out a first treatment on the surface of the By the formulaObtaining a feature map F ', carrying out maximum pooling treatment and average pooling treatment on the feature map F by the feature map F', carrying out convolution operation on the results of the maximum pooling treatment and the average pooling treatment to extract features, and obtaining a spatial attention mapping matrix M after Sigmoid activation function treatment s ∈R 1×H×W Then the feature map F' is passed through a space attention mapping matrix M s ∈R 1×H×W By the formulaObtaining a characteristic diagram F';
the classification module is an improved AlexNet network obtained by replacing a convolution layer with a convolution kernel size of 5 multiplied by 5 in the AlexNet network by using the acceptance module, and the characteristic diagram S' marked with the cervical vertebra region and the lumbar vertebra region of the patient is subjected to the classification module to obtain the cervical vertebra disease classification result and the lumbar vertebra disease classification result of the patient.
2. The method for using the intelligent auxiliary diagnosis and treatment system for rehabilitation of the spinal diseases based on infrared thermal imaging as claimed in claim 1, which is characterized by comprising the following steps:
step S01, diagnosis and treatment information management
Step S101, recording and updating diagnosis and treatment information
Recording diagnosis and treatment information comprising patient information, infrared thermal image images before and after treatment of the patient's spine diseases, treatment time, treatment scheme, medication scheme and diagnosis description information through a recording and uploading module, and uploading and uniformly storing the information into a database module; all diagnosis and treatment information in the database module form an electronic medical record database, and the infrared thermal image of the patient forms an infrared thermal image set S i ;
The medical record management subsystem processes the diagnosis and treatment information, and establishes, adds or updates single or multiple treatment records of the patient in the database module;
step S102, diagnosis and treatment information searching
The user inputs keywords including 'patient name', 'patient visit card number', 'primary doctor name' or 'primary doctor number' to the search engine module through the control center subsystem;
the search engine module searches and matches each piece of electronic medical record data in the database module according to the input keywords and presents search results on the upper computer;
s02, treatment evaluation:
step S201, evaluation of Single treatment
Acquiring diagnosis and treatment information of a patient through the diagnosis and treatment information search of the step S102;
the single-treatment rehabilitation evaluation module selects a single-treatment record from diagnosis and treatment information of a patient through a control center subsystem, and gathers an infrared thermal image S in the single-treatment record i Inputting into an intelligent analysis subsystem to obtain a cervical vertebra disease classification result and a lumbar vertebra disease classification result of a patient, and then evaluating a single treatment effect: respectively converting the cervical vertebra disease classification result and the lumbar vertebra disease classification result of the patient into 0-4 five digital grades to obtain and present the result of single treatment effect evaluation of the patient on an upper computer;
step S202, multiple treatment evaluation
Acquiring diagnosis and treatment information of a patient through the diagnosis and treatment information search of the step S102;
the multi-treatment rehabilitation evaluation unit selects a plurality of treatment records from the diagnosis and treatment information of the patient through the control center subsystem, and sets an infrared thermal image S in each treatment record of the patient i Inputting into an intelligent analysis subsystem to obtain cervical vertebra disease classification results and lumbar vertebra disease classification results of patients after treatment for each time, and then evaluating the treatment effects for multiple times: and converting the cervical vertebra disease classification result and the lumbar vertebra disease classification result of the patient after each treatment into five digital grades of 0-4, generating a multi-treatment evaluation line graph, and obtaining and presenting the multi-treatment effect evaluation result of the patient on an upper computer.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110293245.8A CN113077874B (en) | 2021-03-19 | 2021-03-19 | Intelligent auxiliary diagnosis and treatment system and method for rehabilitation of spine diseases based on infrared thermal images |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110293245.8A CN113077874B (en) | 2021-03-19 | 2021-03-19 | Intelligent auxiliary diagnosis and treatment system and method for rehabilitation of spine diseases based on infrared thermal images |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113077874A CN113077874A (en) | 2021-07-06 |
CN113077874B true CN113077874B (en) | 2023-11-28 |
Family
ID=76612910
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110293245.8A Active CN113077874B (en) | 2021-03-19 | 2021-03-19 | Intelligent auxiliary diagnosis and treatment system and method for rehabilitation of spine diseases based on infrared thermal images |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113077874B (en) |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103020446A (en) * | 2012-12-10 | 2013-04-03 | 吴志宏 | Data processing system for information of chronicle spinal degenerative diseases and deformity |
WO2016101351A1 (en) * | 2014-12-26 | 2016-06-30 | 深圳市前海安测信息技术有限公司 | Network hospital based general practitioner auxiliary diagnosis and treatment system and method |
CN106897546A (en) * | 2017-01-13 | 2017-06-27 | 深圳市前海安测信息技术有限公司 | Medical information aids in doctor's diagnosis and therapy system and method |
KR20190059440A (en) * | 2017-11-23 | 2019-05-31 | 주식회사 아이엠투지 | System and method for diagnostic support through automatic search of similar patient |
CN111986177A (en) * | 2020-08-20 | 2020-11-24 | 太原科技大学 | Chest rib fracture detection method based on attention convolution neural network |
CN112488234A (en) * | 2020-12-10 | 2021-03-12 | 武汉大学 | End-to-end histopathology image classification method based on attention pooling |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180350459A1 (en) * | 2017-06-05 | 2018-12-06 | University Of Florida Research Foundation, Inc. | Methods and apparatuses for implementing a semantically and visually interpretable medical diagnosis network |
US20200397313A1 (en) * | 2017-10-06 | 2020-12-24 | Mayo Foundation For Medical Education And Research | Ecg-based cardiac ejection-fraction screening |
-
2021
- 2021-03-19 CN CN202110293245.8A patent/CN113077874B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103020446A (en) * | 2012-12-10 | 2013-04-03 | 吴志宏 | Data processing system for information of chronicle spinal degenerative diseases and deformity |
WO2016101351A1 (en) * | 2014-12-26 | 2016-06-30 | 深圳市前海安测信息技术有限公司 | Network hospital based general practitioner auxiliary diagnosis and treatment system and method |
CN106897546A (en) * | 2017-01-13 | 2017-06-27 | 深圳市前海安测信息技术有限公司 | Medical information aids in doctor's diagnosis and therapy system and method |
KR20190059440A (en) * | 2017-11-23 | 2019-05-31 | 주식회사 아이엠투지 | System and method for diagnostic support through automatic search of similar patient |
CN111986177A (en) * | 2020-08-20 | 2020-11-24 | 太原科技大学 | Chest rib fracture detection method based on attention convolution neural network |
CN112488234A (en) * | 2020-12-10 | 2021-03-12 | 武汉大学 | End-to-end histopathology image classification method based on attention pooling |
Non-Patent Citations (2)
Title |
---|
"基于红外热像的颈椎病变辅助诊疗系统研究与应用";李鑫;《中国优秀硕士学位论文全文数据库 医药卫生科技辑》;全文 * |
医学影像疾病诊断的残差神经网络优化算法研究进展;周涛;霍兵强;陆惠玲;师宏斌;;中国图象图形学报(10);全文 * |
Also Published As
Publication number | Publication date |
---|---|
CN113077874A (en) | 2021-07-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN101911077B (en) | For the method and apparatus of hierarchical search | |
Hsu et al. | SPIRS: a Web-based image retrieval system for large biomedical databases | |
US20040193036A1 (en) | System and method for performing probabilistic classification and decision support using multidimensional medical image databases | |
US20090082637A1 (en) | Multi-modality fusion classifier with integrated non-imaging factors | |
WO2010005656A2 (en) | Brain condition assessment | |
CN115131642B (en) | Multi-modal medical data fusion system based on multi-view subspace clustering | |
US11978208B2 (en) | Trained model, learning method, learning program, medical information acquisition device, medical information acquisition method, and medical information acquisition program | |
Mansourvar et al. | Automated web based system for bone age assessment using histogram technique | |
CN113782184A (en) | Cerebral apoplexy auxiliary evaluation system based on facial key point and feature pre-learning | |
CN112862749A (en) | Automatic identification method for bone age image after digital processing | |
Modi et al. | A comprehensive analysis of artificial intelligence techniques for the prediction and prognosis of lifestyle diseases | |
Miao et al. | Neural pattern recognition model for breast cancer diagnosis | |
Ji et al. | Detection of cervical spondylotic myelopathy based on gait analysis and deterministic learning | |
Cheng et al. | Report of clinical bone age assessment using deep learning for an Asian population in Taiwan | |
CN113077874B (en) | Intelligent auxiliary diagnosis and treatment system and method for rehabilitation of spine diseases based on infrared thermal images | |
Jayachandran et al. | Multi-dimensional cascades neural network models for the segmentation of retinal vessels in colour fundus images | |
Paunksnis et al. | The use of information technologies for diagnosis in ophthalmology | |
Jabbar et al. | Deep learning based classification of wrist cracks from X-ray imaging | |
Chang et al. | Using machine learning algorithms in medication for cardiac arrest early warning system construction and forecasting | |
Jai-Andaloussi et al. | Content Based Medical Image Retrieval based on BEMD: optimization of a similarity metric | |
Hsu et al. | Predicting thorax diseases with NIH chest X-rays | |
Chen et al. | Alzheimer's Disease Clinical Scores Prediction based on the Label Distribution Learning using Brain Structural MRI | |
Kaur et al. | An Accurate Integrated System to detect Pulmonary and Extra Pulmonary Tuberculosis using Machine Learning Algorithms | |
Hsu et al. | A comprehensive study of age-related macular degeneration detection | |
Korchiyne et al. | New Approach Based on Multifractal Spectrum Features for Detection of Osteoporosis |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |