CN112750531A - Automatic inspection system, method, equipment and medium for traditional Chinese medicine - Google Patents

Automatic inspection system, method, equipment and medium for traditional Chinese medicine Download PDF

Info

Publication number
CN112750531A
CN112750531A CN202110083762.2A CN202110083762A CN112750531A CN 112750531 A CN112750531 A CN 112750531A CN 202110083762 A CN202110083762 A CN 202110083762A CN 112750531 A CN112750531 A CN 112750531A
Authority
CN
China
Prior art keywords
image
inspection part
inspection
segmentation
landmark
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110083762.2A
Other languages
Chinese (zh)
Inventor
王峰
刘进辉
王晓洒
王宏武
潘观潮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong University of Technology
Original Assignee
Guangdong University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong University of Technology filed Critical Guangdong University of Technology
Priority to CN202110083762.2A priority Critical patent/CN112750531A/en
Publication of CN112750531A publication Critical patent/CN112750531A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85—Stereo camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20092—Interactive image processing based on input by user
    • G06T2207/20104—Interactive definition of region of interest [ROI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Medical Informatics (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Public Health (AREA)
  • Multimedia (AREA)
  • Molecular Biology (AREA)
  • Pathology (AREA)
  • Databases & Information Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The application discloses a system, a method, equipment and a medium for automatic inspection of traditional Chinese medicine, comprising: the system comprises an acquisition unit, a display unit and a control unit, wherein the acquisition unit is used for acquiring a inspection part image which comprises a face image, a tongue image, an ear image, an eye image and a human body shape image; the positioning and segmentation unit is used for inputting the inspection part image into a preset convolution expert network model for landmark point positioning and image segmentation and outputting the inspection part segmentation image; and the classification unit is used for inputting the inspection part segmentation image into a preset depth convolution neural network model for classification, or extracting the directional gradient histogram characteristics of the inspection part segmentation image, inputting the directional gradient histogram characteristics into a preset classifier for classification, and outputting the diagnosis result of the inspection part segmentation image. The method solves the technical problem that the accuracy of the diagnosis result is low because the prior traditional Chinese medicine inspection method directly extracts and analyzes the characteristics of the collected single facial image or single tongue image to obtain the final diagnosis result.

Description

Automatic inspection system, method, equipment and medium for traditional Chinese medicine
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an automated inspection system, method, device, and medium for traditional Chinese medicine.
Background
With the development of science and technology, the inspection methods of traditional Chinese medicine are increasing day by day, and regardless of image analysis or near infrared spectrum technology, the inspection methods of traditional Chinese medicine are going towards refinement, systematization and objectification. The traditional Chinese medicine inspection method directly extracts and analyzes the characteristics of the collected single facial image or single tongue image to obtain the final diagnosis result, and has the problem of low accuracy of the diagnosis result.
Disclosure of Invention
The application provides a traditional Chinese medicine automatic inspection system, a traditional Chinese medicine automatic inspection method, a traditional Chinese medicine automatic inspection device and a traditional Chinese medicine automatic inspection medium, which are used for solving the technical problem that the accuracy of a diagnosis result is low because the traditional Chinese medicine inspection method directly extracts and analyzes the characteristics of an acquired single facial image or a single tongue image to obtain a final diagnosis result.
In view of this, the first aspect of the present application provides an automated inspection system for traditional Chinese medicine, comprising:
the system comprises an acquisition unit, a display unit and a control unit, wherein the acquisition unit is used for acquiring a visual inspection part image which comprises a face image, a tongue image, an ear image, an eye image and a human body shape image;
the positioning and segmentation unit is used for inputting the inspection part image into a preset convolution expert network model for landmark point positioning and image segmentation and outputting an inspection part segmentation image;
and the classification unit is used for inputting the inspection part segmentation image into a preset depth convolution neural network model for classification, or extracting the directional gradient histogram characteristics of the inspection part segmentation image, inputting the directional gradient histogram characteristics into a preset classifier for classification, and outputting the diagnosis result of the inspection part segmentation image.
Optionally, the method further includes: a configuration unit;
the configuration unit specifically includes:
the acquisition subunit is used for acquiring a training image of the inspection part;
the training subunit is used for inputting the inspection part training image into a flexible object three-dimensional point distribution model for training to obtain flexible object three-dimensional point distribution model parameters corresponding to the inspection part training image;
the input subunit is used for inputting the inspection part training image and the flexible object three-dimensional point distribution model parameters corresponding to the inspection part training image into a convolution expert network to obtain the landmark alignment probability of each pixel position in the inspection part training image;
and the optimization subunit is used for carrying out network parameter optimization by combining the landmark alignment probability and the flexible object three-dimensional point distribution model parameters to obtain the preset convolution expert network model.
Optionally, the flexible object three-dimensional point distribution model is:
Figure BDA0002910064190000021
wherein the content of the first and second substances,
Figure BDA0002910064190000022
is the mean coordinate of the ith landmark point, phiiA principal component matrix of the ith landmark point, q is an m-dimensional vector for controlling non-rigid parameters, the rigid parameters comprise a scaling quantity s, and a translation quantity t is [ t [ ]x,ty]TAnd a three-dimensional rotation matrix R ═ Rx,ry,rz]T，R2DThe first two rows of parameters of the rotation matrix R.
Optionally, the optimized subunit is specifically configured to:
combining the landmark alignment probability and the flexible object three-dimensional point distribution model parameters to obtain a convolution expert constraint landmark model, wherein the convolution expert constraint landmark model is as follows:
Figure BDA0002910064190000023
wherein, P*For optimal parameter set for controlling landmark position, p ═ s, t, R, q]Three-dimensional point distribution model parameters of the flexible object, I training image of the inspection part, DiIth landmark position x in inspection part training image I output by convolution expert networkiN is the number of landmark positions,
Figure BDA0002910064190000024
a penalty function of a flexible object three-dimensional point distribution model is adopted, so that most of flexible objects without deformation can be accurately described by model points;
and carrying out optimization solution on the convolution expert constraint landmark model based on non-uniform regularization mean shift until the convolution expert constraint landmark model converges to obtain the preset convolution expert network model.
Optionally, the method further includes:
the preprocessing unit is used for preprocessing the image of the inspection part, and the preprocessing comprises color card color correction, image denoising or image alignment;
correspondingly, the positioning and segmentation unit is specifically configured to:
inputting the preprocessed inspection part image into a preset convolution expert network model for landmark point positioning and image segmentation, and outputting an inspection part segmentation image.
The second aspect of the present application provides an automated inspection method for traditional Chinese medicine, which is applied to any one of the automated inspection systems for traditional Chinese medicine of the first aspect, and comprises:
acquiring a inspection part image through an acquisition unit, wherein the inspection part image comprises a face image, a tongue image, an ear image, an eye image and a human body shape image;
inputting the inspection part image into a preset convolution expert network model for landmark point positioning and image segmentation through a positioning and segmentation unit, and outputting an inspection part segmentation image;
and inputting the inspection part segmentation image into a preset depth convolution neural network model for classification through a classification unit, or extracting the directional gradient histogram characteristics of the inspection part segmentation image, inputting the directional gradient histogram characteristics into a preset classifier for classification, and outputting the diagnosis result of the inspection part segmentation image.
Optionally, the method further includes:
the preset convolution expert network model is configured through a configuration unit, and the configuration unit specifically comprises the following steps:
acquiring a training image of the inspection part through an acquisition subunit;
inputting the inspection part training image into a flexible object three-dimensional point distribution model for training through a training subunit to obtain flexible object three-dimensional point distribution model parameters corresponding to the inspection part training image;
inputting the inspection part training image and the flexible object three-dimensional point distribution model parameters corresponding to the inspection part training image into a convolution expert network through an input subunit to obtain the landmark alignment probability of each pixel position in the inspection part training image;
and carrying out network parameter optimization by combining the landmark alignment probability and the flexible object three-dimensional point distribution model parameter through an optimization subunit to obtain the preset convolution expert network model.
Optionally, the positioning and segmenting unit inputs the image of the inspection part into a preset convolutional expert network model for landmark positioning and image segmentation, and outputs the segmented image of the inspection part, and the method further includes:
preprocessing the image of the inspection part by a preprocessing unit, wherein the preprocessing comprises color chip color correction, image denoising or image alignment;
correspondingly, the step of inputting the image of the inspection part into a preset convolution expert network model through a positioning and segmentation unit to perform landmark positioning and image segmentation and outputting the image of the inspection part comprises the following steps:
and inputting the preprocessed inspection part image into a preset convolution expert network model through a positioning and segmentation unit to perform landmark point positioning and image segmentation, and outputting an inspection part segmentation image.
A third aspect of the present application provides a device for automated inspection of chinese medicine, the device comprising a processor and a memory;
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is used for executing the traditional Chinese medicine automatic inspection method according to any one of the second aspect according to the instructions in the program codes.
A fourth aspect of the present application provides a computer-readable storage medium for storing program code for executing the automated inspection method for chinese medical science according to any one of the second aspects.
According to the technical scheme, the method has the following advantages:
the application provides an automatic inspection system of traditional chinese medical science, includes: the system comprises an acquisition unit, a display unit and a control unit, wherein the acquisition unit is used for acquiring a inspection part image which comprises a face image, a tongue image, an ear image, an eye image and a human body shape image; the positioning and segmentation unit is used for inputting the inspection part image into a preset convolution expert network model for landmark point positioning and image segmentation and outputting the inspection part segmentation image; and the classification unit is used for inputting the inspection part segmentation image into a preset depth convolution neural network model for classification, or extracting the directional gradient histogram characteristics of the inspection part segmentation image, inputting the directional gradient histogram characteristics into a preset classifier for classification, and outputting the diagnosis result of the inspection part segmentation image.
In the application, by acquiring the inspection part image comprising the face image, the tongue image, the ear image, the eye image and the human body shape image, classification is performed based on a plurality of part images, and the accuracy of the diagnosis result is higher compared with the diagnosis result obtained by classifying based on a single face image or a single tongue image; in addition, the method also carries out landmark point positioning and image segmentation on the inspection part image through a preset convolution expert network model, inputs the segmented inspection part segmented image into a preset depth convolution neural network model for classification, or extracts the directional gradient histogram characteristics of the segmented inspection part image and inputs the directional gradient histogram characteristics into a preset classifier for classification, thereby reducing the influence of redundant characteristics on the classification result, further improving the accuracy of the diagnosis result, and solving the technical problem that the accuracy of the diagnosis result is not high because the traditional Chinese medical inspection method directly carries out characteristic extraction and analysis on the acquired single facial image or single tongue image to obtain the final diagnosis result.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic structural diagram of an automated inspection system for traditional Chinese medicine according to an embodiment of the present disclosure;
fig. 2 is a schematic flow chart of the automated inspection system for traditional Chinese medicine provided in the embodiment of the present application;
fig. 3 is another schematic structural diagram of an automated inspection system for traditional Chinese medicine according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of a convolutional expert network according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a hybrid expert network according to an embodiment of the present application;
fig. 6 is a schematic diagram of a landmark positioning result of an unobstructed facial image according to an embodiment of the present application;
fig. 7 is a schematic diagram of a landmark positioning result of an occlusion face image according to an embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of a preset deep convolutional neural network model according to an embodiment of the present application;
FIG. 9 is a schematic diagram of a inspection three-dimensional calibration system according to an embodiment of the present disclosure;
FIG. 10 is a diagram of histogram of oriented gradients features provided in an embodiment of the present application;
fig. 11 is a diagram illustrating a multi-classification support vector machine according to an embodiment of the present disclosure.
Detailed Description
The application provides a traditional Chinese medicine automatic inspection system, a traditional Chinese medicine automatic inspection method, a traditional Chinese medicine automatic inspection device and a traditional Chinese medicine automatic inspection medium, which are used for solving the technical problem that the accuracy of a diagnosis result is low because the traditional Chinese medicine inspection method directly extracts and analyzes the characteristics of an acquired single facial image or a single tongue image to obtain a final diagnosis result.
In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
For easy understanding, referring to fig. 1, an embodiment of an automated inspection system for traditional chinese medicine provided by the present application includes:
an acquisition unit 101, configured to acquire a inspection site image, where the inspection site image includes a face image, a tongue image, an ear image, an eye image, and a human body shape image;
the positioning and segmentation unit 102 is used for inputting the inspection part image into a preset convolution expert network model for landmark positioning and image segmentation, and outputting the inspection part segmentation image;
and the classification unit 103 is used for inputting the inspection part segmentation image into a preset depth convolution neural network model for classification, or extracting the directional gradient histogram characteristics of the inspection part segmentation image, inputting the directional gradient histogram characteristics into a preset classifier for classification, and outputting the diagnosis result of the inspection part segmentation image.
The tongue diagnosis is an extremely important content in the inspection of traditional Chinese medicine, and is an effective method for assisting diagnosis by observing the tongue quality and tongue coating of a tongue image model. The tongue is the viscera which can be exposed outside the human body and is also the important organ for reporting diseases. Each organ of the human body can represent the health information of the human body, so that the pathological changes of the organs of the human body can be reflected on the tongue body really. Generally speaking, the tongue diagnosis is more reliable than the pulse diagnosis because the pulse diagnosis is influenced by the emotional changes of the patient and the inaccuracy of the measuring instrument, and the tongue image is obtained by artificial observation, is not easily influenced by other factors, and is a relatively real external mirror. Therefore, tongue diagnosis is a diagnosis basis which can reflect the health information of human body more effectively.
The facial diagnosis is an important content of inspection in traditional Chinese medicine, and the physiological functions and pathological changes of human body can be understood by observing the spirit and complexion. The facial diagnosis is mainly composed of inspection of spirit and inspection of facial color (abbreviated as inspection of spirit color). Inspection of the spirit: can distinguish the abundance or insufficiency of the spirit and the severity of the disease. Spirit refers to the sum of mental state, expression and other conditions of the patient. The key points of inspection are the eye spirit and the expression. Inspection of the complexion: the color is the color of the zang-fu organs, which reflects the appearance of qi and blood and is also the manifestation of disease. The human body health condition is diagnosed by judging the spirit of the essence and the spirit of a person through the diagnosis of the spirit of the eyes and the complexion, and the method is an important and reliable means for the inspection diagnosis of the traditional Chinese medicine.
The eye diagnosis is a key ring of traditional Chinese medicine inspection, and is based on the theory of traditional Chinese medicine: liver governs eyes; all the pulses of the human body belong to the eyes; the essential qi of five zang-organs in the human body is totally injected upwards into the eyes; the eye is the emissary of the heart, and the heart is the residence of the eye; since eyes are so closely related to human body, meridians and zang-fu organs are necessarily related to eyes. Wherein, the inspection of the eyes is one of the methods for eye diagnosis. The disease of the eyes and the health condition of the internal organs of the human body can be diagnosed by observing the spirit, the color and the shape of the eyes, and the purpose of inspection in the traditional Chinese medicine can be effectively achieved.
Ear diagnosis is a ring of inspection in traditional Chinese medicine, and diagnosis and treatment of diseases are not few in China through ear characteristics. Ear diagnosis is mainly performed by observing the color, shape, acupoints, etc. of the ear and its surroundings. Because the abundance, insufficiency, deficiency, excess, cold and heat of the ears, viscera, meridians, qi, blood and body fluids and the mild and serious adverse reactions of diseases can all appear on the ears, the internal conditions of the human body can be observed by acquiring the biological information of the ears. The detection of diseases, such as treatment with auricular points, has gone through two fundamental stages. The first stage is to observe the color and luster of the auricular points; the second stage is to utilize the bioelectrical information of the auricular points to realize detection. The information collected in the two stages is recorded to form the final diagnosis and treatment result, which reflects the health condition of the internal organs of the human body to a certain extent.
The morphological diagnosis is also an essential content of inspection in TCM. The observation of the morphology is mainly achieved by observing the body and posture of the patient. Firstly, observing the body mainly includes observing the strength, the thinness and the fatness of the body and limb shapes; then, the patient can observe the posture, i.e. the dynamic and static postures and the behavior of the patient. These all reflect the human health information to some extent.
In summary, in the embodiment of the present application, the acquisition unit 101 acquires the inspection part image including the face image, the tongue image, the ear image, the eye image, and the body shape image to perform diagnosis, and the accuracy of the diagnosis result is ensured by integrating a plurality of part information to perform diagnosis.
The image of the inspection site is collected by an image acquisition device, which can be placed in the relevant cooperating unit of TCM for collecting data. The acquisition unit acquires a relevant inspection part image from the image acquisition device. The image acquisition device can adopt an Intel RealSense D435 infrared structure optical camera, can be programmed through an API of D435, can manufacture software for acquiring color image information and depth information, can align a depth map with a color map so as to generate three-dimensional mapping on pixel coordinates of the color map, and can finally store the pixel coordinates and the camera three-dimensional coordinates in a text. In order to accelerate the image acquisition speed, a graphical user interface can be manufactured for the software, and the software is packaged into software running on an independent desktop, so that a plurality of electric energies can simultaneously carry out image acquisition operation, the regional space limitation is broken through, and three-dimensional calibration software is also manufactured when images are manually calibrated. After a large amount of data is collected, simple manual calibration work can be carried out, the software can be carried out by multiple persons at the same time, the portability of multiple platforms is achieved, the limitation of time and space can be broken through, and the calibration work can be carried out by multiple persons, so that the image calibration efficiency is improved.
The positioning and segmentation unit 102 inputs the inspection part image into a preset convolution expert network model for landmark point positioning, and performs image segmentation based on the landmark points obtained by positioning so as to segment the region surrounded by the landmark points and the background region, thereby obtaining the inspection part segmentation image.
The classification unit 103 inputs the segmentation image of the inspection part into a preset depth convolution neural network model for classification, or extracts the histogram feature of the directional gradient of the segmentation image of the inspection part, inputs the histogram feature of the directional gradient into a preset classifier for classification, and outputs the diagnosis result of the segmentation image of the inspection part, that is, the classification unit 103 can perform classification diagnosis by adopting two ways, one way is to classify the segmentation image of the inspection part directly through the preset depth convolution neural network model and output the diagnosis result of the segmentation image of the inspection part. The preset depth convolution neural network is a mapping model of the inspection part segmentation image and the diagnosis result, and the inspection part segmentation image is input into the preset depth convolution neural network model for classification and the diagnosis result is output. The method is an end-to-end classification method, and the classification speed is higher. And the other way is to further extract the directional gradient histogram characteristics of the inspection part segmentation image, classify the directional gradient histogram characteristics through a preset classifier and output the diagnosis result of the inspection part segmentation image. The directional gradient histogram feature of the inspection part segmentation image is extracted, and the surface phase feature can be extracted: pale complexion, red complexion and the like; tongue characteristics: pale red and small tongue, red and tender tongue, red tongue tip, etc. And after the relevant features are extracted, performing multi-classification by using a preset classifier.
Further, the specific steps of extracting the directional gradient histogram features of the inspection part segmentation image, classifying the directional gradient histogram features through a preset classifier, and outputting the diagnosis result of the inspection part segmentation image are as follows:
s1, calculating the gradient of the inspection part segmentation image in the horizontal direction X and the vertical direction Y. First by gradient operator [ -1,0,1 [ -1 [ ]]And [ -1,0,1 [ -1]TRespectively carrying out convolution operation on the segmentation images of the inspection part to obtain a gradient component g in the direction of X, Yx、gy(ii) a Then, calculating the gradient size g and the direction angle theta of the pixel point (x, y), wherein the specific calculation formula is as follows:
Figure BDA0002910064190000081
Figure BDA0002910064190000082
s2, constructing a histogram of directional gradients of each cell unit, wherein 8 × 8 cell (cell) units are used, and then calculating the gradient size and direction of each pixel point in the cell unit. Dividing 0-180 degrees by taking 20 degrees as an interval, wherein 9 amplitudes correspond to 0 degree, 20 degrees, … degrees and 160 degrees respectively; amplitude values are selected according to gradient directions of pixels, and all pixel points in the cell unit are mapped to (0 to 8)9 amplitude values through linear interpolation, so that a directional gradient histogram of the cell unit is constructed, as shown in fig. 10.
And S3, normalization processing. Because the single cell is sensitive to the light of the whole picture, the local histogram is normalized in a larger area, the light can be adjusted to a certain extent after normalization, and the influence of the reduction of the picture classification accuracy caused by uneven illumination is weakened. In the embodiment of the application, 16-by-16 block normalization is adopted, and finally, the whole picture is traversed by taking 8 as step length.
And S4, generating directional gradient histogram features. And calculating directional gradient histogram feature vectors of all blocks of the image, and finally combining the directional gradient histogram feature vectors into a directional gradient histogram feature.
And S5, classifying the histogram features of the directional gradients through a preset classifier, and outputting the diagnosis result of the segmentation image of the inspection part.
The embodiment of the application constructs a multi-classifier by combining a plurality of two-class Support Vector Machines (SVM). During training, samples of a certain class are classified into one class, and other remaining samples are classified into another class, so that k SVM are constructed by the samples of k classes, and the k SVM constitutes a preset classifier. The classification classifies the unknown sample as the class having the largest classification function value. Suppose there are three classes to be classified, A, B, C respectively. When the training set is extracted, the following extraction method is adopted:
(1) the characteristics corresponding to A are used as a positive set, and the characteristics corresponding to B and C are used as a negative set;
(2) the vector corresponding to B is used as a positive set, and the vectors corresponding to A and C are used as a negative set;
(3) the vector corresponding to C is used as a positive set, and the vectors corresponding to A and B are used as a negative set;
the 3 training sets are used to train 3 SVMs, respectively. In actual classification, the extracted directional gradient histogram features are respectively input into the trained 3 SVM, 3 classification results are correspondingly obtained, each classification result is the probability of the corresponding class, and finally the class with the maximum probability value is selected as the classification result.
Further, the process of realizing classification by a single SVM is as follows: firstly, defining a hyperplane: y ═ ωTX + b, parameters omega and b are normal vector and intercept of hyperplane, X ═ X1,x2,…,xn]For the training set, y ═ y1,y2,…,yn]And y ∈ { -1,1} is a training set label. For any training sample, the following conditions are satisfied:
yi(ωTxi+b)≥1，
Figure BDA0002910064190000091
on the premise of satisfying the above formula, the following objective functions need to be satisfied:
Figure BDA0002910064190000092
to solve for optimal omega*，b*The method adopts a Lagrange multiplier method for solving, and firstly adds a Lagrange multiplier lambdai> 0, a Lagrangian function is obtained, i.e.:
Figure BDA0002910064190000101
secondly, according to the strong dual relation and the KKT condition, the dual problem of the original problem can be generated, and the original objective function is converted into the dual problem
Figure BDA0002910064190000102
Thereby finding the optimum ω*,b*The classification decision function obtained is:
f(x)＝sign(ω*·x+b*)。
the above is a concrete implementation principle expression for realizing a single two-classifier by using a support vector machine. By adopting the same training mode, a plurality of two classifiers can be obtained, so that the multi-classification problem of the multi-feature sample to be diagnosed is solved, as shown in fig. 11.
The skilled person can select a specific classification method according to actual needs. The image processing of the automatic inspection system of traditional Chinese medicine in the embodiment of the application can be performed in a background server, and the acquired image data can also be stored in the background server.
In the embodiment of the application, the diagnosis part images including the face image, the tongue image, the ear image, the eye image and the human body shape image are obtained, and the classification is carried out based on a plurality of part images, so that the accuracy rate of the diagnosis result is higher compared with the diagnosis result obtained by classifying based on a single face image or a single tongue image; in addition, the method also carries out landmark point positioning and image segmentation on the inspection part image through a preset convolution expert network model, inputs the segmented inspection part segmented image into a preset depth convolution neural network model for classification, or extracts the directional gradient histogram characteristics of the segmented inspection part image and inputs the directional gradient histogram characteristics into a preset classifier for classification, thereby reducing the influence of redundant characteristics on the classification result, further improving the accuracy of the diagnosis result, and solving the technical problem that the accuracy of the diagnosis result is not high because the traditional Chinese medical inspection method directly carries out characteristic extraction and analysis on the acquired single facial image or single tongue image to obtain the final diagnosis result.
The above is an embodiment of the automated inspection system for traditional Chinese medicine provided by the present application, and the following is another embodiment of the automated inspection system for traditional Chinese medicine provided by the present application.
Referring to fig. 3, the automatic inspection system for traditional Chinese medicine in the embodiment of the present application includes:
an acquisition unit 101, configured to acquire a inspection site image, where the inspection site image includes a face image, a tongue image, an ear image, an eye image, and a human body shape image;
the positioning and segmentation unit 102 is used for inputting the inspection part image into a preset convolution expert network model for landmark positioning and image segmentation, and outputting the inspection part segmentation image;
and the classification unit 103 is used for inputting the inspection part segmentation image into a preset depth convolution neural network model for classification, or extracting the directional gradient histogram characteristics of the inspection part segmentation image, inputting the directional gradient histogram characteristics into a preset classifier for classification, and outputting the diagnosis result of the inspection part segmentation image.
As a further improvement, the method further comprises the following steps:
the preprocessing unit 104 is used for preprocessing the image of the observation part, wherein the preprocessing comprises color correction of a color chart, image denoising or image alignment;
correspondingly, the positioning and segmentation unit 102 is specifically configured to:
inputting the preprocessed inspection part image into a preset convolution expert network model to perform landmark point positioning and image segmentation, and outputting an inspection part segmentation image.
The automatic inspection system of traditional chinese medical science in this application embodiment mainly diagnoses after accurately segmenting through the inspection position image of gathering, can shoot the inspection position image of patient through the image acquisition device of clinic end, and the inspection position image of gathering can transmit to the clinic end or store in the server. The images may be pre-processed prior to segmentation of the image of the viewing site. Specifically, wavelet transformation can be performed on the image of the observation part to perform image denoising, and color card color correction and standardization processing can be performed on the image of the observation part to reduce the influence of a light source; because the image acquisition device can shoot a depth map by adopting a D435 camera, the depth map and the color map can be aligned.
As a further improvement, the automatic inspection system for traditional Chinese medicine in the embodiment of the present application further includes: a configuration unit 105;
the configuration unit 105 specifically includes:
an acquisition subunit 1051, configured to acquire a training image of a region to be examined;
a training subunit 1052, configured to input the inspection part training image into the flexible object three-dimensional point distribution model for training, so as to obtain a flexible object three-dimensional point distribution model parameter corresponding to the inspection part training image;
an input subunit 1053, configured to input the inspection part training image and the flexible object three-dimensional point distribution model parameters corresponding to the inspection part training image into a convolutional expert network, so as to obtain a landmark alignment probability of each pixel position in the inspection part training image;
and the optimization subunit 1054 is configured to perform network parameter optimization by combining the landmark alignment probability and the flexible object three-dimensional point distribution model parameter, so as to obtain a preset convolution expert network model.
The specific process of the automated inspection system for traditional Chinese medicine in the embodiment of the present application can refer to fig. 2. In the initial stage of constructing the automatic inspection system of traditional Chinese medicine, the image acquisition devices distributed in each traditional Chinese medicine cooperation unit can acquire the image of the inspection part of each patient, and the image acquisition devices can print labels according to the cases of each patient to obtain the training image of the inspection part, wherein the training image of the inspection part comprises a face training image, a tongue training image, an ear training image, an eye training image and a human body shape training image. And the diagnosis information related to the patient is stored in the inspection case database as a label together with the inspection part image of the patient after the patient is diagnosed by the system and confirmed so as to optimize the diagnosis model. The obtaining subunit 1051 obtains the training image of the inspection part through the image collecting device, and the training subunit 1052 inputs the training image of the inspection part into the three-dimensional point distribution model of the flexible object for training, so as to obtain the parameters of the three-dimensional point distribution model of the flexible object corresponding to the training image of the inspection part. The training image of the inspection part can be preprocessed before being input into the flexible object three-dimensional point distribution model, and the specific processing process is consistent with the preprocessing process of the image of the inspection part, and is not repeated herein.
Further, the flexible object three-dimensional point distribution model is as follows:
Figure BDA0002910064190000121
wherein the content of the first and second substances,
Figure BDA0002910064190000122
is as followsMean coordinate of i landmark points, ΦiA principal component matrix of the ith landmark point, q is an m-dimensional vector for controlling non-rigid parameters, the rigid parameters comprise a scaling quantity s, and a translation quantity t is [ t [ ]x,ty]TAnd a three-dimensional rotation matrix R ═ Rx,ry,rz]T，R2DThe first two rows of parameters of the rotation matrix R.
The shape of the region of interest of the training image of the inspection part is represented by a flexible object three-dimensional point distribution model for capturing the shape change of the landmark, and the statistical information of the shape change is obtained through the training of the training image of the inspection part; and estimates the change of the shape cloud using principal component analysis, which provides effective parameters of the shape model through dimension reduction. Taking the tongue training image in the inspection part training image as an example, assume that k three-dimensional landmark points are used to represent the shape of the tongue, i.e., X ═ X1,y1,z1,..,xk,yk,zk) If the landmark point shape of the tongue training image set is a gaussian distribution around the average landmark point, the landmark points can be constructed by principal component analysis as follows:
Figure BDA0002910064190000123
where X' is the possible landmark point location,
Figure BDA0002910064190000124
is the average position of landmark points, P is a 3k x d matrix, which is the first d eigenvectors of the principal component analysis, and b is a scale vector, each of which corresponds to a principal component. Thus, the tongue and its possible deformation of the flexible object can be described using the average landmark points and deformation parameters, and the detailed description can refer to the flexible object three-dimensional point distribution model described above. The flexible object three-dimensional point distribution model is trained through the tongue training image, and a flexible object three-dimensional point distribution model parameter p corresponding to the tongue training image can be obtained, namely the whole shape of the tongue can be represented by the parameter p [ s, t, R, q ]]And (4) showing. Correspondingly, the flexible object is trained in three dimensions through the training images of the inspection part such as the face training image or the ear training imageThe point distribution model can obtain the flexible object three-dimensional point distribution model parameter p corresponding to the training image of the inspection part such as the face training image or the ear training image, and the specific process is consistent with the processing process of the tongue training image, and is not repeated herein.
The input subunit 1053 inputs the inspection part training image and the flexible object three-dimensional point distribution model parameters corresponding to the inspection part training image into the convolutional expert network to obtain the landmark alignment probability of each pixel position in the inspection part training image. The main role of the convolutional expert network is to estimate the alignment position of a single landmark independently of other coordinate positions, and the structure of the convolutional expert network can be referred to fig. 4. After inputting the inspection part training image to the convolution expert network, the convolution expert network processes the n x n pixel region of interest (ROI) around the current landmark position estimation value, and outputs a response graph for estimating the landmark alignment probability of each pixel position. Taking a tongue training image as an example, the convolution expert network extracts a surrounding region of interest (around the tongue) with the size of n × n pixels according to the current landmark estimation value, and then performs contrast normalization convolution (preferably 500 × 11), wherein local contrast normalization is adopted, and edge information is more prominent, so that the method is a method for image preprocessing, meets a depth learning framework, performs Z-score normalization before calculating input and kernel correlation, and finally outputs a 500 × n response map. The output response graph is used as the input of the next convolutional layer, and then the response graph of 200 × n is output through the processing of the activation function ReLU, and then the convolution processing is performed with the hybrid expert network (preferably, 100 × 1) to output the result of 100 × n expert voting probability, and the structure of the hybrid expert network can refer to fig. 5, and the result of expert voting probability is combined with the non-negative weight to obtain the final alignment probability, that is:
Figure BDA0002910064190000131
wherein liIndicators aligned for landmark I, I being input image at landmark position xiThe region of interest of (a) is,
Figure BDA0002910064190000132
is the output landmark position xiA response map of alignment probabilities at (a).
The optimization subunit 1054 performs network parameter optimization by combining the landmark alignment probability and the flexible object three-dimensional point distribution model parameters to obtain a preset convolution expert network model. Specifically, the optimization subunit 1054 obtains a convolution expert constraint landmark model by combining the landmark alignment probability and the three-dimensional point distribution model parameter p of the flexible object, where the convolution expert constraint landmark model is:
Figure BDA0002910064190000133
wherein, P*For optimal parameter set for controlling landmark position, p ═ s, t, R, q]Three-dimensional point distribution model parameters of the flexible object, I training image of the inspection part, DiIth landmark position x in inspection part training image I output by convolution expert networkiN is the number of landmark positions,
Figure BDA0002910064190000141
the method is a penalty function of a flexible object three-dimensional point distribution model, so that most of flexible objects without deformation can be accurately described by model points, namely landmark points.
The optimization subunit 1054 performs optimization solution on the convolution expert constraint landmark model based on the non-uniform regularization mean shift until the convolution expert constraint landmark model converges to obtain a preset convolution expert network model. The convolution expert constraint landmark model is a local method, depends on the estimation of initialization parameters and is based on the initialization parameters p0And then updating the parameters by delta p to solve the optimal solution, so that the convolution expert constraint landmark model can be further expressed as:
Figure BDA0002910064190000142
in order to realize good real-time performance, eliminate the influence of noise and optimize network parameters, in the embodiment of the application, a convolution expert constraint landmark model is solved by adopting non-uniform regularization mean shift, that is:
Figure BDA0002910064190000143
wherein J is a Jacobian matrix, Λ, of landmark positions of a flexible object three-dimensional point distribution model parameter p-1To describe the prior matrix of the parameter p, v ═ v1,v2,…,vn]TIs a landmark mean shift vector. And adopting Gaussian kernel density estimation calculation:
Figure BDA0002910064190000144
calculation of mean-shift vector dependent on features
Figure BDA0002910064190000145
And empirically determined current estimates of the rho parameter, N being a full covariance Gaussian distribution function, xi、yi、ziCoordinate values of the x axis, the y axis and the z axis of the ith landmark point respectively,
Figure BDA0002910064190000146
setting the coefficient according to the actual condition; ΨiIs a set of coordinate values; w ═ W · diag (c)1；...ci；...；cn；c1；...；cn) For diagonal weight matrix, the w parameter is flexibly set according to actual conditions, ciThe reliability matrix W of the patch expert is calculated under each scale and view respectively for the correlation coefficient of the ith patch expert and the confidence of the i patch experts represented by the ith and the (i + n) th elements on the diagonal.
The following update rule is obtained based on the nonlinear least square method of Tikhonov regularization:
Δp＝-(JTWJ+r∧-1)(r∧-1p-JTWv)；
and calculating average displacement and iterative update parameters according to the regularization term r until a convergence condition is met to obtain a trained convolutional expert network model, and taking the trained convolutional expert network model as a preset convolutional expert network model.
The method and the device adopt a flexible object three-dimensional point distribution model to carry out three-dimensional modeling on the region of interest, carry out dimension reduction processing on modeling influence factors and eliminate correlation among components through principal component analysis, and provide effective parameters of a shape model; the flexible object three-dimensional point distribution model clearly restores the region of interest, and the outline, color and texture of the region of interest are well presented; and under the conditions of different attitude angles and larger part deformation, the device has good performance.
Furthermore, the method adopts a convolution expert constraint landmark model to effectively combine a convolution expert network and a flexible object three-dimensional point distribution model, and realizes high-precision landmark point positioning, so that the image segmentation precision is improved, and the accuracy of subsequent diagnosis results is improved; in order to obtain good real-time performance, the embodiment of the application adopts a non-uniform regularized mean shift optimization solution model, has very good detection performance for human body part shielding and large posture angle rotation of human body parts, and particularly can refer to schematic diagrams of landmark detection results of facial images provided by fig. 6 and 7, has very good detection performance for shielded and non-shielded facial images, and is beneficial to improving the segmentation accuracy of the inspection part image.
Further, the training of the preset deep convolutional neural network model can adopt unsupervised and supervised learning modes; (1) unsupervised learning: during training, firstly adopting unlabeled data to train each layer in a layered mode, firstly using the unlabeled data to train the first layer, learning to obtain the nth-1 layer, and then training the nth layer by taking the output of the n-1 layer as the input of the nth layer, so that the parameters of each layer are obtained; (2) top-down supervised learning: and training each layer through the data with the labels, and transmitting the error from top to bottom to finely adjust the deep convolutional neural network so as to obtain a preset deep convolutional neural network model. The preset depth convolutional neural network model is structured as shown in fig. 8, and the input training images of the inspection site are 3-channel images, wherein C1, C2, C3 and C4 are convolutional layers, preferably, convolutional layer C1 is 96 convolutional kernels of 11 × 11, convolutional layer C2 is 256 convolution kernels of 5 × 5, convolutional layer C3 is 384 convolution kernels of 3 × 3, and convolutional layer C4 is 256 convolution kernels of 3 × 3. The preset deep convolution neural network model has 4 layers of max-posing pooling layers, and the kernel of each layer of max-posing layers is 2 x 2; and the output of the max-pooling layer of the fourth layer is used as the input of the full connection layer, the full connection layer connects the output of the max-pooling layer of the fourth layer into a one-dimensional vector, and the output of the full connection layer is classified through the softmax layer.
The automatic inspection system of traditional chinese medical science in this application embodiment can adopt two kinds of mode: 1. is a training mode: when the system is not trained sufficiently and patients cannot be diagnosed independently, batch data training is carried out on the network models through a large number of labeled data sets, so that a high-precision automatic traditional Chinese medicine system is obtained; 2. and in a working and incremental learning mode, the system further trains each network model according to the diagnosis case, and the diagnosis accuracy is incrementally improved.
The above is another embodiment of the automated inspection system for traditional Chinese medicine provided by the present application, and the following is an embodiment of the automated inspection method for traditional Chinese medicine provided by the present application.
The embodiment of the present application provides an automatic inspection method for traditional Chinese medicine, which is applied to the automatic inspection system for traditional Chinese medicine in the aforementioned embodiments, and comprises:
step 201, acquiring a inspection part image through an acquisition unit, wherein the inspection part image comprises a face image, a tongue image, an ear image, an eye image and a human body shape image;
202, inputting the inspection part image into a preset convolution expert network model for landmark point positioning and image segmentation through a positioning and segmentation unit, and outputting an inspection part segmentation image;
and 203, inputting the inspection part segmentation image into a preset depth convolution neural network model for classification through a classification unit, or extracting the directional gradient histogram characteristics of the inspection part segmentation image, inputting the directional gradient histogram characteristics into a preset classifier for classification, and outputting the diagnosis result of the inspection part segmentation image.
Further, still include: the method comprises the following steps of configuring a preset convolution expert network model through a configuration unit, specifically:
acquiring a training image of the inspection part through an acquisition subunit; inputting the training image of the inspection part into a flexible object three-dimensional point distribution model for training through a training subunit to obtain flexible object three-dimensional point distribution model parameters corresponding to the training image of the inspection part; inputting the training image of the inspection part and the parameters of the flexible object three-dimensional point distribution model corresponding to the training image of the inspection part into a convolution expert network through an input subunit to obtain the landmark alignment probability of each pixel position in the training image of the inspection part; and carrying out network parameter optimization by combining the landmark alignment probability and the flexible object three-dimensional point distribution model parameters through an optimization subunit to obtain a preset convolution expert network model.
Further, the method comprises the following steps of inputting the image of the inspection part into a preset convolution expert network model through a positioning and segmentation unit to carry out landmark positioning and image segmentation, and outputting the segmented image of the inspection part, wherein the method also comprises the following steps:
preprocessing the image of the inspection part by a preprocessing unit, wherein the preprocessing comprises color correction of a color chart, image denoising or image alignment;
correspondingly, the positioning and segmentation unit inputs the image of the inspection part into a preset convolution expert network model for landmark positioning and image segmentation, and outputs the segmentation image of the inspection part, which comprises the following steps:
and inputting the preprocessed inspection part image into a preset convolution expert network model through a positioning and segmentation unit to perform landmark point positioning and image segmentation, and outputting an inspection part segmentation image.
In the embodiment of the application, the diagnosis part images including the face image, the tongue image, the ear image, the eye image and the human body shape image are obtained, and the classification is carried out based on a plurality of part images, so that the accuracy rate of the diagnosis result is higher compared with the diagnosis result obtained by classifying based on a single face image or a single tongue image; in addition, the method also carries out landmark point positioning and image segmentation on the inspection part image through a preset convolution expert network model, inputs the segmented inspection part segmented image into a preset depth convolution neural network model for classification, or extracts the directional gradient histogram characteristics of the segmented inspection part image and inputs the directional gradient histogram characteristics into a preset classifier for classification, thereby reducing the influence of redundant characteristics on the classification result, further improving the accuracy of the diagnosis result, and solving the technical problem that the accuracy of the diagnosis result is not high because the traditional Chinese medical inspection method directly carries out characteristic extraction and analysis on the acquired single facial image or single tongue image to obtain the final diagnosis result.
The embodiment of the application also provides a traditional Chinese medicine automatic inspection device, which comprises a processor and a memory;
the memory is used for storing the program codes and transmitting the program codes to the processor;
the processor is used for executing the traditional Chinese medicine automatic inspection method in the embodiment according to the instructions in the program codes.
The embodiment of the application also provides a computer readable storage medium, which is used for storing program codes, and the program codes are used for executing the traditional Chinese medicine automatic inspection method in the embodiment.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for executing all or part of the steps of the method described in the embodiments of the present application through a computer device (which may be a personal computer, a server, or a network device). And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims (10)

1. An automatic inspection system for traditional Chinese medicine, comprising:
the system comprises an acquisition unit, a display unit and a control unit, wherein the acquisition unit is used for acquiring a visual inspection part image which comprises a face image, a tongue image, an ear image, an eye image and a human body shape image;
the positioning and segmentation unit is used for inputting the inspection part image into a preset convolution expert network model for landmark point positioning and image segmentation and outputting an inspection part segmentation image;
and the classification unit is used for inputting the inspection part segmentation image into a preset depth convolution neural network model for classification, or extracting the directional gradient histogram characteristics of the inspection part segmentation image, inputting the directional gradient histogram characteristics into a preset classifier for classification, and outputting the diagnosis result of the inspection part segmentation image.
2. The automated inspection system of traditional Chinese medicine according to claim 1, further comprising: a configuration unit;
the configuration unit specifically includes:
the acquisition subunit is used for acquiring a training image of the inspection part;
the training subunit is used for inputting the inspection part training image into a flexible object three-dimensional point distribution model for training to obtain flexible object three-dimensional point distribution model parameters corresponding to the inspection part training image;
the input subunit is used for inputting the inspection part training image and the flexible object three-dimensional point distribution model parameters corresponding to the inspection part training image into a convolution expert network to obtain the landmark alignment probability of each pixel position in the inspection part training image;
and the optimization subunit is used for carrying out network parameter optimization by combining the landmark alignment probability and the flexible object three-dimensional point distribution model parameters to obtain the preset convolution expert network model.
3. The automated inspection system according to claim 2, wherein the three-dimensional point distribution model of the flexible object is:
Figure FDA0002910064180000011
wherein the content of the first and second substances,
Figure FDA0002910064180000012
is the mean coordinate of the ith landmark point, phiiA principal component matrix of the ith landmark point, q is an m-dimensional vector for controlling non-rigid parameters, the rigid parameters comprise a scaling quantity s, and a translation quantity t is [ t [ ]x,ty]TAnd a three-dimensional rotation matrix R ═ Rx,ry,rz]T，R2DThe first two rows of parameters of the rotation matrix R.
4. The automated inspection system of traditional Chinese medicine according to claim 3, wherein the optimized subunit is specifically configured to:
combining the landmark alignment probability and the flexible object three-dimensional point distribution model parameters to obtain a convolution expert constraint landmark model, wherein the convolution expert constraint landmark model is as follows:
Figure FDA0002910064180000021
wherein, P*For optimal parameter set for controlling landmark position, p ═ s, t, R, q]Three-dimensional point distribution model parameters of the flexible object, I training image of the inspection part, DiIth landmark position x in inspection part training image I output by convolution expert networkiN is the number of landmark positions,
Figure FDA0002910064180000022
distributing penalty function of model for flexible object three-dimensional point, so that model pointThe flexible object without deformation can be described accurately;
and carrying out optimization solution on the convolution expert constraint landmark model based on non-uniform regularization mean shift until the convolution expert constraint landmark model converges to obtain the preset convolution expert network model.
5. The automated inspection system of traditional Chinese medicine according to claim 1, further comprising:
the preprocessing unit is used for preprocessing the image of the inspection part, and the preprocessing comprises color card color correction, image denoising or image alignment;
correspondingly, the positioning and segmentation unit is specifically configured to:
inputting the preprocessed inspection part image into a preset convolution expert network model for landmark point positioning and image segmentation, and outputting an inspection part segmentation image.
6. An automated inspection method for traditional Chinese medicine, which is applied to the automated inspection system for traditional Chinese medicine according to any one of claims 1 to 5, comprising:
acquiring a inspection part image through an acquisition unit, wherein the inspection part image comprises a face image, a tongue image, an ear image, an eye image and a human body shape image;
inputting the inspection part image into a preset convolution expert network model for landmark point positioning and image segmentation through a positioning and segmentation unit, and outputting an inspection part segmentation image;
and inputting the inspection part segmentation image into a preset depth convolution neural network model for classification through a classification unit, or extracting the directional gradient histogram characteristics of the inspection part segmentation image, inputting the directional gradient histogram characteristics into a preset classifier for classification, and outputting the diagnosis result of the inspection part segmentation image.
7. The automated inspection method of traditional Chinese medicine according to claim 6, further comprising:
the preset convolution expert network model is configured through a configuration unit, and the configuration unit specifically comprises the following steps:
acquiring a training image of the inspection part through an acquisition subunit;
inputting the inspection part training image into a flexible object three-dimensional point distribution model for training through a training subunit to obtain flexible object three-dimensional point distribution model parameters corresponding to the inspection part training image;
inputting the inspection part training image and the flexible object three-dimensional point distribution model parameters corresponding to the inspection part training image into a convolution expert network through an input subunit to obtain the landmark alignment probability of each pixel position in the inspection part training image;
and carrying out network parameter optimization by combining the landmark alignment probability and the flexible object three-dimensional point distribution model parameter through an optimization subunit to obtain the preset convolution expert network model.
8. The automatic inspection method of traditional Chinese medicine according to claim 6, wherein the positioning and segmentation unit inputs the image of the inspection site into a preset convolutional expert network model for landmark positioning and image segmentation, and outputs the image of the inspection site segmentation, and the method further comprises:
preprocessing the image of the inspection part by a preprocessing unit, wherein the preprocessing comprises color chip color correction, image denoising or image alignment;
correspondingly, the step of inputting the image of the inspection part into a preset convolution expert network model through a positioning and segmentation unit to perform landmark positioning and image segmentation and outputting the image of the inspection part comprises the following steps:
and inputting the preprocessed inspection part image into a preset convolution expert network model through a positioning and segmentation unit to perform landmark point positioning and image segmentation, and outputting an inspection part segmentation image.
9. An automatic inspection device for traditional Chinese medicine, which is characterized by comprising a processor and a memory;
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is used for executing the traditional Chinese medicine automatic inspection method according to any one of claims 6 to 8 according to instructions in the program codes.
10. A computer-readable storage medium for storing a program code for executing the automated inspection method for chinese medical science according to any one of claims 6 to 8.
CN202110083762.2A 2021-01-21 2021-01-21 Automatic inspection system, method, equipment and medium for traditional Chinese medicine Pending CN112750531A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110083762.2A CN112750531A (en) 2021-01-21 2021-01-21 Automatic inspection system, method, equipment and medium for traditional Chinese medicine

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110083762.2A CN112750531A (en) 2021-01-21 2021-01-21 Automatic inspection system, method, equipment and medium for traditional Chinese medicine

Publications (1)

Publication Number Publication Date
CN112750531A true CN112750531A (en) 2021-05-04

Family

ID=75652862

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110083762.2A Pending CN112750531A (en) 2021-01-21 2021-01-21 Automatic inspection system, method, equipment and medium for traditional Chinese medicine

Country Status (1)

Country Link
CN (1) CN112750531A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113240606A (en) * 2021-05-24 2021-08-10 博奥生物集团有限公司 Traditional Chinese medicine inspection image color correction method and system
CN113610847A (en) * 2021-10-08 2021-11-05 武汉楚精灵医疗科技有限公司 Method and system for evaluating stomach markers in white light mode
CN113658702A (en) * 2021-08-26 2021-11-16 山西慧虎健康科技有限公司 Cerebral apoplexy characteristic extraction and intelligent risk prediction method and system based on traditional Chinese medicine inspection diagnosis
CN115661142A (en) * 2022-12-14 2023-01-31 广东工业大学 Tongue diagnosis image processing method, device and medium based on key point detection
CN117116432A (en) * 2023-10-23 2023-11-24 博奥生物集团有限公司 Disease characteristic processing method, device and equipment
CN117315357A (en) * 2023-09-27 2023-12-29 广东省新黄埔中医药联合创新研究院 Image recognition method and related device based on traditional Chinese medicine deficiency-excess syndrome differentiation classification

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2672425A1 (en) * 2012-06-08 2013-12-11 Realeyes OÜ Method and apparatus with deformable model fitting using high-precision approximation
CN104573723A (en) * 2015-01-08 2015-04-29 上海中医药大学 Feature extraction and classification method and system based on spirit of traditional Chinese medicine observation diagnosis
CN105193418A (en) * 2015-08-27 2015-12-30 李迎新 Body silhouette and extremity inspection system
CN106295139A (en) * 2016-07-29 2017-01-04 姹ゅ钩 A kind of tongue body autodiagnosis health cloud service system based on degree of depth convolutional neural networks
CN106991388A (en) * 2017-03-27 2017-07-28 中国科学院自动化研究所 Crucial independent positioning method
CN107242857A (en) * 2017-06-12 2017-10-13 南开大学 The intelligent traditional Chinese medical science based on deep learning integrates diagnosis and therapy system
CN107451407A (en) * 2017-07-31 2017-12-08 广东工业大学 A kind of traditional Chinese medical science intelligent diagnosing method, system and traditional medical system
CN109637660A (en) * 2018-12-19 2019-04-16 新绎健康科技有限公司 A kind of lingual diagnosis analysis method and system based on depth convolutional neural networks
CN111066060A (en) * 2017-07-13 2020-04-24 资生堂美洲公司 Virtual face makeup removal and simulation, fast face detection, and landmark tracking
CN111126180A (en) * 2019-12-06 2020-05-08 四川大学 Facial paralysis severity automatic detection system based on computer vision
CN111340932A (en) * 2018-12-18 2020-06-26 富士通株式会社 Image processing method and information processing apparatus
CN112107292A (en) * 2020-08-04 2020-12-22 刘佳友 Chinese medicine diagnosis method and instrument

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2672425A1 (en) * 2012-06-08 2013-12-11 Realeyes OÜ Method and apparatus with deformable model fitting using high-precision approximation
CN104573723A (en) * 2015-01-08 2015-04-29 上海中医药大学 Feature extraction and classification method and system based on spirit of traditional Chinese medicine observation diagnosis
CN105193418A (en) * 2015-08-27 2015-12-30 李迎新 Body silhouette and extremity inspection system
CN106295139A (en) * 2016-07-29 2017-01-04 姹ゅ钩 A kind of tongue body autodiagnosis health cloud service system based on degree of depth convolutional neural networks
CN106991388A (en) * 2017-03-27 2017-07-28 中国科学院自动化研究所 Crucial independent positioning method
CN107242857A (en) * 2017-06-12 2017-10-13 南开大学 The intelligent traditional Chinese medical science based on deep learning integrates diagnosis and therapy system
CN111066060A (en) * 2017-07-13 2020-04-24 资生堂美洲公司 Virtual face makeup removal and simulation, fast face detection, and landmark tracking
CN107451407A (en) * 2017-07-31 2017-12-08 广东工业大学 A kind of traditional Chinese medical science intelligent diagnosing method, system and traditional medical system
CN111340932A (en) * 2018-12-18 2020-06-26 富士通株式会社 Image processing method and information processing apparatus
CN109637660A (en) * 2018-12-19 2019-04-16 新绎健康科技有限公司 A kind of lingual diagnosis analysis method and system based on depth convolutional neural networks
CN111126180A (en) * 2019-12-06 2020-05-08 四川大学 Facial paralysis severity automatic detection system based on computer vision
CN112107292A (en) * 2020-08-04 2020-12-22 刘佳友 Chinese medicine diagnosis method and instrument

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
ZADEH A 等: "Convolutional Experts Constrained Local Model for 3D Facial Landmark Detection", 《2017 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION WORKSHOPS (ICCVW)》, pages 2520 *
袁善敏 等: "中医舌、面诊颜色校正方法研究进展", 《中华中医药杂志》, vol. 34, no. 9, 1 September 2019 (2019-09-01), pages 4183 - 4185 *
谢中华: "《MATLAB与数学建模》", vol. 2019, 30 September 2019, 北京：北京航空航天大学出版社, pages: 332 *
阮琦: "颅脑三维重建与空间配准算法研究", 《中国优秀硕士学位论文全文数据库》, no. 01, 15 January 2021 (2021-01-15), pages 1 - 76 *
黄诗茜 等: "卷积神经网络在中医舌诊中的应用综述", 《电脑知识与技术》, vol. 16, no. 26, pages 21 *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113240606A (en) * 2021-05-24 2021-08-10 博奥生物集团有限公司 Traditional Chinese medicine inspection image color correction method and system
CN113658702A (en) * 2021-08-26 2021-11-16 山西慧虎健康科技有限公司 Cerebral apoplexy characteristic extraction and intelligent risk prediction method and system based on traditional Chinese medicine inspection diagnosis
CN113658702B (en) * 2021-08-26 2023-09-15 山西慧虎健康科技有限公司 Cerebral apoplexy feature extraction and intelligent risk prediction method and system based on traditional Chinese medicine inspection
CN113610847A (en) * 2021-10-08 2021-11-05 武汉楚精灵医疗科技有限公司 Method and system for evaluating stomach markers in white light mode
CN113610847B (en) * 2021-10-08 2022-01-04 武汉楚精灵医疗科技有限公司 Method and system for evaluating stomach markers in white light mode
CN115661142A (en) * 2022-12-14 2023-01-31 广东工业大学 Tongue diagnosis image processing method, device and medium based on key point detection
CN117315357A (en) * 2023-09-27 2023-12-29 广东省新黄埔中医药联合创新研究院 Image recognition method and related device based on traditional Chinese medicine deficiency-excess syndrome differentiation classification
CN117315357B (en) * 2023-09-27 2024-04-30 广东省新黄埔中医药联合创新研究院 Image recognition method and related device based on traditional Chinese medicine deficiency-excess syndrome differentiation classification
CN117116432A (en) * 2023-10-23 2023-11-24 博奥生物集团有限公司 Disease characteristic processing method, device and equipment
CN117116432B (en) * 2023-10-23 2023-12-15 博奥生物集团有限公司 Disease characteristic processing device and equipment

Similar Documents

Publication Publication Date Title
AU2017292642B2 (en) System and method for automatic detection, localization, and semantic segmentation of anatomical objects
CN112750531A (en) Automatic inspection system, method, equipment and medium for traditional Chinese medicine
KR20210048523A (en) Image processing method, apparatus, electronic device and computer-readable storage medium
WO2020133636A1 (en) Method and system for intelligent envelope detection and warning in prostate surgery
Sridar et al. Decision fusion-based fetal ultrasound image plane classification using convolutional neural networks
Nurmaini et al. Accurate detection of septal defects with fetal ultrasonography images using deep learning-based multiclass instance segmentation
SE528068C2 (en) Three dimensional object recognizing method for e.g. aircraft, involves detecting image features in obtained two dimensional representation, and comparing recovered three dimensional shape with reference representation of object
CN107437252B (en) Method and device for constructing classification model for macular lesion region segmentation
CN113012173A (en) Heart segmentation model and pathology classification model training, heart segmentation and pathology classification method and device based on cardiac MRI
Zhang et al. A supervised texton based approach for automatic segmentation and measurement of the fetal head and femur in 2D ultrasound images
US11935246B2 (en) Systems and methods for image segmentation
Feng et al. Supervoxel based weakly-supervised multi-level 3D CNNs for lung nodule detection and segmentation
CN113781488A (en) Tongue picture image segmentation method, apparatus and medium
CA3207781A1 (en) System and method of using right and left eardrum otoscopy images for automated otoscopy image analysis to diagnose ear pathology
Song et al. The classification and segmentation of fetal anatomies ultrasound image: A survey
Xu et al. Application of artificial intelligence technology in medical imaging
Hsu A hybrid approach for brain image registration with local constraints
Pavithra et al. An Overview of Convolutional Neural Network Architecture and Its Variants in Medical Diagnostics of Cancer and Covid-19
Wang et al. Optic disc detection based on fully convolutional neural network and structured matrix decomposition
WO2020140380A1 (en) Method and device for quickly dividing optical coherence tomography image
Zhou et al. Automatic segmentation of liver from CT scans with CCP–TSPM algorithm
CN112102327A (en) Image processing method and device and computer readable storage medium
Liu et al. Vessel segmentation using principal component based threshold algorithm
Ye et al. Ultrasonic image segmentation method based on improved fully convolution network
CN117809122B (en) Processing method, system, electronic equipment and medium for intracranial large blood vessel image

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination