CN109431523B

CN109431523B - Autism primary screening device based on non-social voice stimulation behavior paradigm

Info

Publication number: CN109431523B
Application number: CN201811220432.8A
Authority: CN
Inventors: 李明; 邹小兵
Original assignee: Duke Kunshan University; Third Affiliated Hospital Sun Yat Sen University
Current assignee: Duke Kunshan University; Third Affiliated Hospital Sun Yat Sen University
Priority date: 2018-10-19
Filing date: 2018-10-19
Publication date: 2022-08-30
Anticipated expiration: 2038-10-19
Also published as: CN109431523A

Abstract

The invention discloses an autism primary screening device based on a non-social voice stimulation behavior paradigm, which comprises an acquisition module: the remote controller and the camera are used for controlling the sounding object to sound so as to collect audio and video data participating in the behavioral response of the experimental child to the non-social sound stimulation and the doctor instruction; a pretreatment module: processing audio data by adopting voice activity detection, detecting the starting sounding time of a sounding object, and taking the starting time as a starting endpoint to intercept the contents of a video for tens of seconds as video analysis data; a feature extraction module: analyzing and processing the video data frame by frame to obtain facial, eye and gesture characteristics; training a classification module: training a classifier by using the obtained features to obtain a classifier model for predicting the autism; a prediction module: and evaluating and predicting the autism of the tester by adopting the facial, eye and gesture features calculated by the feature extraction module. The invention is suitable for early screening and auxiliary diagnosis of autism behavioral appearance.

Description

Autism primary screening device based on non-social voice stimulation behavior paradigm

Technical Field

The invention relates to the field of multimedia audio and video processing, in particular to an autism primary screening device based on a non-social sound stimulation behavior paradigm.

Background

Autism Spectrum Disorder (ASD), is a neurologically developmental disease. The performance of the patients includes social communication disorder, repeated stereotyped behaviors, narrow interests and the like, and medical staff needs to judge each suspected patient from the aspect of behavioral phenotype. Although the existing autism assessment method is very effective, the existing autism assessment method is time-consuming and labor-consuming, has higher requirements on medical literacy and clinical experience, and has certain subjectivity in assessment results. The measurement methods most widely used at present include the Autism Diagnostic Observation Scale (ADOS), and the like.

The general judgment indexes based on the autism can be used for carrying out preliminary diagnosis on suspected patients in a programmed mode, and the programming enables the computer technology to be applied to autism diagnosis. When judging whether the suspected patient suffers from autism, the name calling reaction, the finger reaction, the following phenomenon and the like of the tested person are comprehensively considered, and finally, the final result is given according to the treatment experience of a doctor. On the computer side, the tested person can be scored in various items, and then the risk coefficient of the tested person suffering from the autism can be predicted by combining the scores in a data science manner. In the process, a proper and efficient algorithm is selected, so that the experience of doctors can be replaced, and the automatic early screening of the childhood autism can be completed by a computer.

The voice processing and computer vision technology is used for assisting doctors and parents in carrying out automatic early screening on suspected autism children, the performance of the link of finger-object reaction of the children in autism diagnosis is comprehensively considered from three aspects of face, eye and hand actions mainly through a camera and a recognition algorithm, and further computer technical support is provided for final confirmation of the children. However, the distance between the child and the camera is large, so that the eyes, gestures, and even the occupied area of the whole tested child in the video image is small, and it is a challenging problem how to extract more accurate information from such data and evaluate the risk coefficient of autism.

Disclosure of Invention

In view of the above problems in the prior art, the present invention is directed to a primary screening device for autism based on a non-social voice stimulation behavioral paradigm. The method estimates the behavioral response of the children to the non-social sound stimulation and the instructions of the doctors to predict the risk coefficient of the children with the autism spectrum disorder, and can be used for early screening of the autism.

In order to realize the purpose, the invention is realized according to the following technical scheme:

an autism primary screening device based on a non-social voice-stimulated behavioral paradigm, comprising:

an acquisition module: the system comprises a controllable sounding object for attracting the attention of children, a remote controller for controlling the sounding object to sound and a camera, wherein the remote controller and the camera are used for collecting audio and video data participating in the behavioral response of experimental children to non-social sound stimulation and doctor instructions;

a preprocessing module: processing audio data by adopting voice activity detection, detecting the time of a sound object starting to sound, taking the time as a starting endpoint, and intercepting the content of a video for tens of seconds as video analysis data;

a feature extraction module: analyzing and processing the video data frame by frame to obtain the characteristics of the face, the eye and the gesture;

training a classification module: training a classifier by using the obtained features to obtain a classifier model for predicting the autism;

a prediction module: and evaluating and predicting the autism of the tester by adopting the facial, eye and gesture features calculated by the feature extraction module.

In the above technical solution, the feature extraction module includes a front face detection unit, a gaze tracking unit, and a gesture detection unit; the front face detection unit detects the front face of the child for the extracted video data to obtain facial features, and if the child looks at the sound-producing object and the front face is detected by a camera beside the sound-producing object, a small window only containing the face is generated for a face area to serve as subsequent image analysis data; the gaze tracking unit further judges whether gaze is concentrated on an object or not to the extracted front face image data to obtain eye features; the gesture detection unit analyzes the video data frame by frame, positions experimenters, eliminates background colors, positions hands and cuts hand picture data, trains extracted data to obtain a gesture classifier model, and detects gesture direction on the basis of the gesture classifier model to obtain gesture characteristics.

In the above technical solution, the gesture detection unit positions the hand of the child in each frame of the video by the following steps, including:

step S1: extracting the positions of the doctor and the tested child in the image by using a trained image-based target detection and recognition algorithm model to obtain the positions of the doctor and the tested child;

step S2: according to the skin color range of people, reserving the part of the frame with the color chroma in the range, and setting the area outside the range to be black, thereby filtering the frame according to the color value and finishing background color removal;

step S3: and finally positioning by using a cascade classifier based on Haar features, wherein the cascade classifier is formed by cascading a plurality of weak classifiers, when the weak classifiers are constructed, detection windows slide from left to right and from top to bottom in a picture, each detection window judges one feature and calculates a result value, the obtained result is compared with a threshold value to judge whether the detection window accords with the feature and classify, and when an image in the detection window successfully passes the inspection of the classifier, the image in the detection window is considered to be an expected target.

In the above technical solution, the training classification module uses a supervised learning classifier to train and classify the labeled features.

In the technical scheme, the prediction module predicts the risk coefficient of the autism by adopting a supervised learning classifier method and a weighted addition method respectively.

Compared with the prior art, the invention has the following advantages:

the invention provides a machine learning-based framework, and provides a software and hardware integrated device for predicting autism by analyzing facial, eye and gesture characteristics through collecting audio and video data of experimenters. Compared with the traditional ADOS evaluation method, the method provided by the invention has remarkable objectivity and does not need the participation of an experienced doctor. Although the device provided by the invention can not completely replace the traditional autism diagnosis method, the device can be taken as an auxiliary device for evaluating the autism risk coefficient, so that the early autism screening is more accurate and convenient.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a general framework block diagram of the machine learning based primary screening device for autism of the present invention;

FIG. 2 is a schematic flow diagram of a reactant reaction;

fig. 3 is a schematic layout of a collection site.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention.

Aiming at the problems of high labor cost, long diagnosis period and high subjectivity of the traditional autism diagnosis method, the invention extracts the facial, eye and gesture characteristics of a test person by quantifying all indexes in the finger reaction and trains data according to the characteristics so as to evaluate the risk coefficient of autism spectrum disorder, thereby being capable of carrying out early screening on autism, reducing the labor cost and saving a large amount of time.

As shown in fig. 1, the primary screening device for autism based on the non-social voice stimulation behavioral paradigm of the present invention includes:

specifically, the experimenter is located the experiment place center with medical personnel, places a camera in the position that the non-of experimenter is positive but the afterglow is visible (oblique the place ahead), hangs an aircraft appearance toy that can the remote control sound production directly over the camera, and medical personnel can the steerable toy sound production. The camera records the whole experiment process as original data.

In this embodiment, the experimenter of the autism behavior analysis data is 115 children, including 58 children diagnosed with autism and 57 normal children, and these children are aged in 24-48 months and have sufficient behavior reaction capability. For each child participating in the experiment, a complete finger reaction test procedure was performed, and the flow chart is shown in fig. 2. Fig. 3 depicts the layout format of the acquisition site.

in this embodiment, a Voice Activity Detection (VAD) algorithm based on energy is used to detect the sounding time of the toy, and record the sounding time as an experiment starting stage, and intercept the video content of tens of seconds of video as audio and video data to be subsequently processed.

the feature extraction module comprises a face detection unit, a gaze tracking unit and a gesture detection unit; the front face detection unit detects the front face of the child for the extracted video data to obtain facial features, and if the child looks at the sound-producing object and the front face is detected by a camera beside the sound-producing object, a small window only containing the face is generated for a face area to serve as subsequent image analysis data; the gaze tracking unit further judges whether gaze is concentrated on an object or not to the extracted front face image data to obtain eye features; the gesture detection unit analyzes the video data frame by frame, positions experimenters, eliminates background colors, positions hands and cuts hand picture data, trains extracted data to obtain a gesture classifier model, and can accurately detect gesture pointing on the basis of the gesture classifier model to obtain gesture characteristics.

Specifically, the front face detection unit of the present invention uses a DLib library (http:// DLib. net /) method to achieve front face detection of a face. Under the condition that the DLib human face is not optimized, the situation that the side faces in many images cannot be detected occurs, and the situation that the side faces of the children face the camera can be screened out by using the characteristic. Then, for the recognized front face, a smaller window containing only the face is reproduced for the face region thereof as subsequent image analysis data.

Specifically, the gaze tracking unit of the invention trains by using an end-to-end convolutional neural network and can obtain a relatively accurate effect on a mobile phone, but because an experiment adopts a common independent camera to acquire data, the condition for judging whether the tested child looks at the camera needs to be modified. The ordinate of the gaze is not considered, and the selection of the abscissa is combined with the actual situation, that is, the child to be tested is located on the left side of the screen, so that the judgment condition of the gaze orthophoria is shifted to the right. In the experiment, the interval of [ -9,1] is selected, namely, the child to be tested can be considered to be directly watching the camera, namely the target object, only when the distance is-9 < = x < = 1.

The gesture detection unit of the invention finishes the positioning of the hand of the child in each frame of picture of the video by the following steps, and comprises the following steps:

step S1: extracting the positions of the doctor and the tested child in the image by using a trained YOLO (you Only Look one) algorithm model to obtain the positions of the doctor and the tested child; equivalently, two sub-images are respectively obtained, and other information such as background in the original picture can be filtered. In addition, the invention can also adopt a Fast Region-based conditional Network method (Fast R-CNN) algorithm.

Step S2: according to the skin color range of a person, reserving the part of the frame with the color chroma in the range, and setting the area outside the range to be black, so as to filter the frame according to the color value and finish background color removal;

furthermore, although finding the position of a person in each frame of a picture can remove much background information, since the body of the person and the background close to the body still exist, the hand cannot be positioned in real time. In order to be able to remove the above mentioned interferences again, further filtering operations can be done with skin tones.

In the separation of human skin color, HSV (Hue, Saturation) space in a tapered shape may be used instead of RGB space. According to the statistical result of the HSV space, the skin color range of human is 9< = h < =15,50< = s < =255 and 50< = v < =255, according to the defined range, the part of the frame, the color chroma of which is in the range, is reserved, and the area which does not belong to the range is blackened, so that the frame is filtered according to the color value.

Step S3: and finally positioning by using a cascade classifier based on Haar features, wherein the cascade classifier is formed by cascading a plurality of weak classifiers, when the weak classifiers are constructed, detection windows slide from left to right and from top to bottom in a picture, each detection window judges one feature and calculates a result value, the obtained result is compared with a threshold value to judge whether the detection window accords with the feature and classify, and one weak classifier is enough to be constructed to carry out weak classification. Combining a plurality of weak classifiers to form a tree-shaped cascade structure, thereby building a strong classifier. When the image in the window passes the strong classifier test successfully, i.e. a positive result is obtained, the program considers the window to be the target we want. The classifier is capable of classifying gestures to select a gesture directed to a target object.

After the hand picture data is located and cut out, the gesture detection unit adopts a microsoft open source deep learning Toolkit (CNTK) tool to perform migration learning, and uses the ResNet _18 as a basic depth model. In practice, the last layer of features in the model may be cut out and replaced with convolutional layers trained using the collected data. Modification is made on the basis of a mature model, so that the requirement for huge data can be avoided. Only hundreds of image data are used as a training set, and the transfer learning classifier with ideal performance can be trained.

The cascade classifier based on the Haar features can find objects similar to data with positive labels in an image area by a classification method, and the method is a common image object positioning algorithm. We build a classifier based on the data used for this project that can effectively cull out human faces and other regions in the image. Two important features, the Haar feature and the cascade classifier, are used in the algorithm.

The Haar characteristic is a characteristic that reflects the gray level change of an image, and a pixel division module calculates a difference value, and the Haar characteristic is divided into the following steps: edge features, linear features, center features, and diagonal features are widely used in computer vision technology. The Haar feature contains three operators: the method comprises the following steps of edge operators, line operators and rectangle operators, wherein two symmetric regions are contained in the operators, and the value of the whole operator can be obtained by subtracting the sum of pixels in one region from the sum of pixels in the other region. The Haar characteristic is used, so that the unit pixel can be prevented from being directly operated by a program, and the operation rate of the whole program is improved. In addition, in order to accelerate the calculation process, an integral graph calculation method can be used, and when the image matrix is traversed, the pixel sum of each point to the submatrix formed by the origin is recorded, so that the time complexity required for calculating the pixel sum of any submatrix in the subsequent steps is greatly reduced.

However, in practical applications, in addition to extracting face, eye and gesture information, two continuous variables that record time, namely the reaction time from the beginning of the link to the attention of the child to the target object and the duration from the attention of the child to the transition from the target object to the attention of the child, can be added. And 5 features in total are used as data for training the model.

Training a classification module: training a classifier by using the obtained features to obtain a classifier model for predicting the autism; the training classification module of the invention uses a support vector machine to train and classify the marked features.

The training classification module of the invention uses a supervised learning classifier to train and classify the marked features, and the supervised learning classifier comprises a support vector machine, a logistic regression, K neighbor, a neural network and the like.

In the present embodiment, a Support Vector Machine (SVM) is used to classify the labeled features. The linear SVM learns a boundary from the training data, so that the feature discrimination of two different classes is as large as possible. However, the features obtained in this embodiment are not linearly separable, so the kernel SVM is used in this embodiment to distinguish the features by introducing a non-linear boundary.

A prediction module: and evaluating and predicting the autism of the tester by adopting the facial, eye and gesture characteristics calculated by the characteristic extraction module.

The prediction module of the invention predicts the risk coefficient of autism by respectively adopting an SVM method and a weighted addition method.

The prediction module predicts risk coefficients of autism by respectively adopting a supervised learning classifier method and a weighted addition method. Likewise, supervised learning classifiers include support vector machines, logistic regression, K-nearest neighbors, neural networks, and the like.

When the SVM method is used, the model for predicting the autism acquired by the training classification module is adopted to test a tester, and the risk coefficient of suffering the autism is predicted.

When the weighted addition method is used, the score is within the standardized range, namely a score of 0-2 is used for judging the performance of the tested child. Because the characteristics are all in the same order of magnitude or even the same range, the scores of all the characteristics can be directly weighted and added, the weight coefficient is adjusted according to the effect on the development set, and the obtained result is compared with a preset threshold value, so that the risk coefficient of the autism can be obtained.

The experimental setup of the invention was:

1: test data and training data are obtained.

Specifically, a cross-comparison strategy is adopted, namely data of each individual is sequentially selected as test data according to the sequence, and data of the rest participants are used as training data.

2: a prediction method.

Except for SVM, the scores of all the characteristics are directly weighted and added, and the obtained result is compared with a preset threshold value, so that the risk coefficient of the children suffering from the autism can be predicted.

3: evaluation benchmark

Face correction detection accuracy: the ratio of the number of people turning round to the total number of people is accurately detected.

Catch of eye detection accuracy: the ratio of the number of people who annotate the target object with the eye to the total number of people is accurately detected.

Gesture detection accuracy: the ratio of the number of people who are not pointing to the target object to the total number of people is accurately detected.

The evaluation accuracy of the risk coefficient of the autism is as follows: the ratio of the number of people with higher risk of suffering from the autism to the number of children actually diagnosed with the autism is accurately predicted.

The experimental results of this experiment are shown in table 1:

the invention better verifies that a better prediction result of 72 percent can be achieved through a series of characteristic extraction according to the finger reaction.

Although the invention has been described in detail above with reference to a general description and specific examples, it will be apparent to one skilled in the art that modifications or improvements may be made thereto based on the invention. Accordingly, such modifications and improvements are intended to be within the scope of the invention as claimed.

Claims

1. An autism primary screening device based on a non-social voice-stimulated behavioral paradigm, comprising:

a pretreatment module: processing audio data by adopting voice activity detection, detecting the time of a sound object starting to sound, taking the time as a starting endpoint, and intercepting the content of a video for tens of seconds as video analysis data;

a prediction module: evaluating and predicting the autism of the tester by adopting the facial, eye and gesture characteristics calculated by the characteristic extraction module;

the feature extraction module comprises a front face detection unit, a gaze tracking unit and a gesture detection unit; the front face detection unit detects the front face of the child for the extracted video data to obtain facial features, and if the child looks at the sound-producing object and the front face is detected by a camera beside the sound-producing object, a small window only containing the face is generated for a face area to serve as subsequent image analysis data; the gaze tracking unit further judges whether gaze is concentrated on an object or not to the extracted front face image data to obtain eye features; the gesture detection unit analyzes the video data frame by frame, positions experimenters, eliminates background colors, positions hands and segments hand picture data, trains extracted data to obtain a gesture classifier model, and detects gesture direction on the basis of the gesture classifier model to obtain gesture features;

the gesture detection unit finishes the positioning of the hand of the child in each frame of the video picture by the following steps of:

step S3: and finally positioning by using a cascade classifier based on Haar features, wherein the cascade classifier is formed by cascading a plurality of weak classifiers, when the weak classifiers are constructed, detection windows slide from left to right and from top to bottom in a picture, each detection window judges one feature and calculates a result value, the result is obtained and compared with a threshold value to judge whether the detection window accords with the feature and classify, and when the image in the detection window successfully passes through the inspection of the classifier, the image in the detection window is considered as an expected target.

2. The non-social voice-stimulated behavioral paradigm-based primary screening device for autism according to claim 1, wherein the training classification module trains and classifies the labeled features using a supervised learning classifier.

3. The device for primary screening of autism based on the non-social voice stimulation behavioral paradigm as claimed in claim 1, wherein the prediction module predicts the risk coefficient of autism by respectively adopting a supervised learning classifier method and a weighted addition method.