CN110349674A - Autism-spectrum obstacle based on improper activity observation and analysis assesses apparatus and system - Google Patents
Autism-spectrum obstacle based on improper activity observation and analysis assesses apparatus and system Download PDFInfo
- Publication number
- CN110349674A CN110349674A CN201910605989.1A CN201910605989A CN110349674A CN 110349674 A CN110349674 A CN 110349674A CN 201910605989 A CN201910605989 A CN 201910605989A CN 110349674 A CN110349674 A CN 110349674A
- Authority
- CN
- China
- Prior art keywords
- information
- autism
- subject
- acquiring unit
- expression
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment
Abstract
The invention proposes a kind of, and the autism-spectrum obstacle based on improper activity observation and analysis assesses apparatus and system, and described device includes: data acquisition module, for obtaining audio data and video data during testing in experimental situation；Feature obtains module, for according to the correlated characteristic information of subject and caretaker during audio data and video data acquisition test；Model training module, for successively using the multiple Weak Classifiers of machine learning algorithm training, and the Weak Classifier being cascaded into autism-spectrum obstacle assessment models according to acquired feature；Forecast assessment module carries out forecast assessment by autism-spectrum obstacle assessment models, and obtain assessment result for the correlated characteristic information according to subject.The present invention can help doctor truely and completely to obtain every capability state of children, can more objectively assess whether children suffer from autism system obstacle.
Description
Technical field
The present invention relates to audio-video processing and machine learning fields, more particularly to based on the lonely of improper activity observation and analysis
Disease pedigree obstacle assesses apparatus and system, passes through acquisition test participant activity trajectory, movement, sight, head direction, appearance
The characteristic informations such as gesture, facial expression, position coordinates, voice content and language mood are to carry out autism-spectrum obstacle assessment.
Background technique
Autism-spectrum obstacle (ASD, Autism Spectrum Disorder, abbreviation autism) is a kind of brain hair
Educate the disease of obstacle.Children with autism-spectrum obstacle can point of interest, locomitivity, in terms of show
With the different performance of common children.Children with autism can especially show itself action it is inappropriate, to object
Product use inappropriate, thought inappropriate and feeling inappropriate.In order to evaluate children to be estimated with the presence or absence of improperly showing,
The usual health care professional activity that can get along with children for a period of time or watches the video recording of its home videos, to analyze children true
The performance grown directly from seeds in living.However the active performance of one rapid lapse of time of children is watched, need appraiser at least to put into same duration
Energy, the efficiency assessed in this way is very low.Children's performance is analyzed by the activity of a period of time, to appreciable levels
Requirement it is very high, it is enough empirical and professional that this needs appraiser to have.And the assessment to a period of time children's activity,
Even if also still remaining the subjectivity artificially given a mark, this makes assessment result vary with each individual, and there are errors based on agreeing scale.
Common Infinite scene is move freely, especially the children's activity video recording of home scenarios, and viewing angle is single, and environmental condition
Limited, the index that each children can be made to be analyzed has differences, can not comprehensive full assessment children every ability, obtain
Accurately autism-spectrum obstacle risk coefficient is estimated out.
The patent document of Patent No. CN201810184072.4 discloses commenting for a kind of autism of children or hypoevolutism
Estimate interfering system and method, children are assessed by way of questionnaire survey, is then intervened according to assessment result.By
It in the mode of questionnaire survey is carried out by evaluator, there is subjectivity, and to tie only by the mode of questionnaire survey
Fruit is more unilateral, causes assessment result inaccurate.
The patent document of Patent No. CN201811473064.8 discloses a kind of autism high-risk infants sieve based on APP
System is looked into, by allowing children to carry out corresponding normal form test in face of mobile device APP, the phase of children during being tested according to normal form
It answers behavior reaction and eye movement to check and examine and carries out autism assessment.But the screening system needs children's ability in face of mobile device always
Correlation test is carried out, the freedom of movement of children is limited, is easy that children is allowed to generate oppressive psychology, causes the inaccurate of assessment result
Really.
Therefore, be badly in need of a kind of autism-spectrum obstacle assessment system or device, by the arrangement to environment, it is real there are currently no
In the case where guide, semi-structured test is carried out, and completely adopt using the audio, video data that acquisition equipment completes acquisition test
Collect full angle data, and children gone out using unified dimensional analysis using computer and whether there is inappropriate behavior and correlated frequency,
With more complete, more acurrate, more objectively assessment children every capability states.Summary of the invention
In view of the deficiencies of the prior art, the present invention proposes a kind of autism-spectrum obstacle based on improper activity observation and analysis
Assess apparatus and system, by completely acquiring subject and caretaker in standardized assay environment, the row being move freely
For data, and evaluated using different capacity indexes of the machine learning algorithm to subject, thus objectivity to subject
Whether assessed with autism-spectrum obstacle, is made reference for doctor.
The technical scheme of the present invention is realized as follows:
A kind of autism-spectrum obstacle assessment device based on improper activity observation and analysis, including
Data acquisition module, for obtaining audio data and video data during testing in experimental situation；
Feature obtains module, for obtaining subject and caretaker during test according to audio data and video data
Correlated characteristic information, the feature include expression in the eyes directional information, head orientation information, gesture pose information, finger directional information,
One of facial expression information, location coordinate information, language emotional information and voice content information are a variety of；
Model training module, for successively training multiple weak typings using machine learning algorithm according to acquired feature
Device, and the multiple Weak Classifier is cascaded into autism-spectrum obstacle assessment models；Wherein, the multiple Weak Classifier includes
The inappropriate Weak Classifier of activity trajectory, the inappropriate Weak Classifier of posture gesture, the inappropriate Weak Classifier of eye motion, language expression
Inappropriate Weak Classifier, article are tended to not using inappropriate Weak Classifier, society referring to the inappropriate Weak Classifier of normal form and interest
Any combination in appropriate Weak Classifier；
Forecast assessment module is assessed for the correlated characteristic information according to subject by autism-spectrum obstacle
Model carries out forecast assessment, and obtains assessment result.
Further, further include
Preprocessing module, for based on timestamp by video data and audio data on a timeline synchronize be aligned.
Further, the autism-spectrum obstacle assessment models Ft(x) expression formula are as follows:
Wherein, ftIt (x) is that obtained Weak Classifier, α are trained according to machine learning algorithmtIt is corresponding inappropriate weak
The weight of classifier, etIt is Weak Classifier to the error rate of corresponding training sample set.
Further, forecast assessment module is also used to the correlated characteristic information according to subject during test, to subject
Person carries out risk of autism spectrum disorders coefficient and/or capability analysis assessment；Wherein, the content assessed includes the movement of self, to object
Use, thought and the feeling of product, it is described self movement, to the use of article, thought and feel to can be used for indicating autism solution
The different dimensions released, to analyze autism.
Further, it includes Application on Voiceprint Recognition unit, expression in the eyes directional information acquiring unit, head court that the feature, which obtains module,
It is obtained to information acquisition unit, facial expression information acquiring unit, gesture pose information acquiring unit, language emotional information single
Member, voice content information acquisition unit and location coordinate information acquiring unit, wherein
Application on Voiceprint Recognition unit, for carrying out identification to speaker in audio data by Application on Voiceprint Recognition, to determine quilt
Examination person and caretaker's identity, and then pass through expression in the eyes directional information acquiring unit, head orientation information acquiring unit, facial expression again
Information acquisition unit, gesture pose information acquiring unit, language emotional information acquiring unit, voice content information acquisition unit
And/or location coordinate information acquiring unit obtains the individual features information of subject and caretaker；
Expression in the eyes directional information acquiring unit, for the pixel coordinate and depth number according to eyes characteristic point in video frame
According to determining the space 3D coordinate of eyes characteristic point；And neural network is tracked using the space 3D coordinate of eyes characteristic point as sight
The input of model, to obtain expression in the eyes directional information；
Head orientation information acquiring unit for obtaining facial feature points by face detection, and determines the 3D in its space
Characteristic point cloud；The normal line vector of face's plane is determined by the 3D characteristic point cloud of facial feature points, to obtain head towards letter
Breath；
Facial expression information acquiring unit obtains face rectangle for carrying out recognition of face to the subject in video frame
Input after frame and alignment as Expression Recognition model, to obtain the expression classification of subject；
Gesture pose information acquiring unit, the gesture detector for being trained by object detection neural network, obtains
Gesture and finger directional information in video frame；
Language emotional information acquiring unit, for obtaining the embeding layer information comprising emotion information by deep neural network
The classification of emotion is carried out, to obtain voice mood information；
Voice content information acquisition unit, for the voice of speaker to be carried out text conversion by speech recognition, to obtain
Take voice content information；
Location coordinate information acquiring unit, for according to the abscissa of subject corresponding in video frame and/or caretaker, vertical
Depth data in coordinate and corresponding depth map, and the CCD camera assembly optical parameter obtained in advance is combined, determine subject
And/or the 3D coordinate of caretaker's relative camera coordinate system.
Further, the feature acquisition module further includes
Scene stage property location coordinate information acquiring unit, for obtaining the location coordinate information of scene stage property.
A kind of autism-spectrum obstacle assessment system based on improper activity observation and analysis, including based on not described in any one
When multiple depth images acquisition dress that the autism-spectrum obstacle of measuring behavior analysis assesses device, is installed on experimental site surrounding
The voice acquisition device set, being set in experimental site and the scene stage property being placed in experimental site；Wherein
Depth image acquisition device, for acquiring the video data during testing in experimental situation, the video data
In include image depth information；
Voice acquisition device, for acquiring the audio data during testing；
Scene stage property, for attracting subject to serve as inducer；
The depth image acquisition device and voice acquisition device are all connect with autism-spectrum obstacle assessment device.
The scene stage property includes real food, emulates food, is tacked with rope to limit the toy of scope of activities, wherein
Real food, emulation food and toy are placed in side opposite with caretaker present position in experimental site.
The depth image acquisition device be RGB-D camera, the voice acquisition device include array microphone and/or
Wearable microphone.
Compared with prior art, the invention has the following advantages that
(1) allow to test in the environment of not guide moveing freely, the data of acquisition can restore real-life.
(2) by the arrangement to environment, invisible semi-structured test can be induced, compared so that test data has more
Property.
(3) experimental design can completely cover the improper index of several major class, map the brain development health shape of subject comprehensively
Condition.
(4) test scene, complete documentation test situation can be completely restored to using the data that multi-angle multichannel acquires.
(5) data are analyzed using the method for machine learning using computer, scale is unified, and it is more objective to analyze, more
It is advantageous to promote.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention without any creative labor, may be used also for those of ordinary skill in the art
To obtain other drawings based on these drawings.
Fig. 1 is the structural schematic diagram of a standardized assay environment during present invention test；
Fig. 2 is that the present invention is based on the structural block diagrams that the autism-spectrum obstacle of improper activity observation and analysis assesses device；
Fig. 3 is that the present invention is based on the structural block diagrams of the autism-spectrum obstacle assessment system of improper activity observation and analysis.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
Referring to Fig.2, a kind of autism-spectrum obstacle based on improper activity observation and analysis disclosed in embodiment of the present invention
Device, including data acquisition module 10, preprocessing module 20, model training module 30 and forecast assessment module 40 are assessed, wherein
Data acquisition module 10, for obtaining audio data and video data during testing in experimental situation；
Specifically, embodiment of the present invention has built a standardized assay ring for the ease of assessing subject
Border, including for attracting the real food 3 and emulation food 4 of subject's attention, such as any of several broadleaf plants of smelling good, false banana；It further include using
It further includes hiding the camera and microphone that are arranged in experimental situation surrounding to limit the toy 5 of scope of activities that rope, which is tied down,.
During test, subject move freely in standardized assay environment.Wherein, camera was tested for acquiring
Video data in journey, microphone are used to acquire the task sound and ambient sound during test.
Feature obtains module 20, for according to subject and caretaker during audio data and video data acquisition test
Correlated characteristic information, the feature include expression in the eyes directional information, head orientation information, gesture pose information, finger be directed toward letter
One of breath, facial expression information, location coordinate information, language emotional information and voice content information are a variety of；
In embodiment of the present invention, when subject, which is in standardized assay environment, to be move freely, obtains subject and exist
Whether Behavioral change and speech variation in experimental situation, analyze subject according to the Behavioral change of subject and speech variation
With autism-spectrum obstacle.
Model training module 30, for successively using support vector machines or other machines according to acquired feature
The multiple Weak Classifiers of algorithm training are practised, and the Weak Classifier is cascaded into autism-spectrum obstacle assessment models；Wherein, described
Multiple Weak Classifiers include that the inappropriate Weak Classifier of activity trajectory, the inappropriate Weak Classifier of posture gesture, eye motion are inappropriate
Weak Classifier, language express inappropriate Weak Classifier, article using inappropriate Weak Classifier, society referring to inappropriate weak point of normal form
Class device and interest tend to any combination in inappropriate Weak Classifier；
Wherein, autism-spectrum obstacle assessment models Ft(x) expression formula are as follows:
In above-mentioned expression formula, ftIt (x) is that obtained Weak Classifier, α are trained according to machine learning algorithmtIt is corresponding
The weight of inappropriate Weak Classifier, etIt is Weak Classifier to the error rate of corresponding training sample set.
Forecast assessment module 40 is commented for the correlated characteristic information according to subject by autism-spectrum obstacle
Estimate model and carry out forecast assessment, and obtains assessment result.
Embodiment of the present invention is by using machine learning algorithm, first to known subject's autism-spectrum obstacle situation
Corresponding data is trained classifier, and obtains autism-spectrum obstacle assessment models Ft(x) after, then pass through autism-spectrum barrier
Hinder assessment models Ft(x) subject of unknown autism-spectrum obstacle situation is assessed, objectively to provide assessment result,
Reference is provided for doctor.Embodiment of the present invention mainly carries out the early screening of autism-spectrum obstacle, therefore this hair to children
Subject in bright embodiment can be children, but be not limited to children.
Further, forecast assessment module is also used to the correlated characteristic information according to subject during test, to subject
Person carries out risk of autism spectrum disorders coefficient and/or capability analysis assessment；Wherein, the content assessed includes the movement of self, to object
Use, thought and the feeling of product, it is described self movement, to the use of article, thought and feel to can be used for indicating autism solution
The different dimensions released, to analyze autism.
In embodiment of the present invention, four major class are broadly divided into according to the capability analysis of subject: the movement of self, to object
The use of product, thought, feeling, this four capability analysis are equally the performances of autism different dimensions, according to subject self
Movement, the use to article, thought, feeling preferably can assess autism and be intervened.Wherein, this four abilities point
Analysis can by the activity trajectory of subject whether appropriate, posture gesture whether appropriate, eye motion whether appropriate, language is expressed
Whether appropriate, article using whether appropriate, society referring to normal form whether appropriate and interest trend whether appropriately this 7 indexs are commented
Estimate, it therefore, can be by the individual features information of subject during test to the movement of self of subject, make to article
With, thought, feel to be analyzed.Specifically
Activity trajectory is inappropriate, refer to autism-spectrum impaired patients be more likely to make arbitrarily, repeat, do not have it is functional
Activity trajectory, such as: loop turn is walked up and down.
Posture gesture is inappropriate, refer to autism-spectrum impaired patients be more likely to make arbitrarily, repeat, do not have it is functional
The inappropriate body stereotypy of itself posture gesture, such as rub hands, clap hands.
Eye motion is inappropriate, refer to autism-spectrum impaired patients be more likely to unconditionally to make arbitrarily, repeat, do not have
The inappropriate eye motion of function, such as: sight sideling see to head towards inconsistent direction.
Language expression is inappropriate, refer to autism-spectrum impaired patients be more likely to say arbitrarily, do not have functional whole sentence
It repeats or the duplicate inappropriate mechanical language of subordinate sentence, shape is such as repeated the words of others like a parrot.
Article refers to that autism-spectrum impaired patients are more likely to mistake using article, such as: throwing out without care, eat using inappropriate
Common toy 5 etc..
Society is inappropriate referring to normal form, refers to when external environment finds variation and has uncertain, autism-spectrum
Impaired patients are more not inclined to relative to ordinary people searches expression information from adult face, and takes and take action or do accordingly
Corresponding reaction out.
Interest tends to inappropriate, refers to that autism-spectrum impaired patients are more not inclined to generate people relative to ordinary people and surpasses
Cross the interest to article.
The ability of subject is analyzed by above this seven inappropriate classification, assesses the dynamic of self of subject
Work, thought, feels this several Xiang Nengli at the use to article, to judge whether subject suffers from autism-spectrum obstacle.
Then the present invention acquires subject and caretaker in standardized assay environment by arrangement standardized assay environment
The behavior characteristic information being move freely, according to the behavior characteristic information of subject successively use support vector machines or its
He is machine learning algorithm training classifier ft(x), and by the classifier stage it is unified into autism-spectrum obstacle assessment models Ft(x)；
Finally by autism-spectrum obstacle assessment models Ft(x) subject is assessed.The present invention can help doctor true, complete
Every capability state that is whole, comprehensively obtaining children, can more objectively evaluate whether children suffer from autism system obstacle.
Further, the autism-spectrum obstacle assessment device based on improper activity observation and analysis further includes pretreatment mould
Block, for based on timestamp by video data and audio data on a timeline synchronize be aligned.
In order to accurately acquire the data of subject and caretaker, camera is set to standard by embodiment of the present invention
Change the surrounding of environment, so as to the Behavioral change of subject and caretaker during energy multi-angle multi-angle of view record test, therefore,
Before the correlated characteristic information for obtaining subject and caretaker, the video data for first being recorded multiple cameras is based on timestamp
Synchronize alignment, then by video data and audio data on a timeline synchronize be aligned after, then obtain subject and caretaker
Correlated characteristic information.
Specifically, it includes Application on Voiceprint Recognition unit 201, expression in the eyes directional information acquiring unit 202, head court that feature, which obtains module,
To information acquisition unit 203, facial expression information acquiring unit 204, gesture pose information acquiring unit 205, language mood letter
Acquiring unit 206, voice content information acquisition unit 207 and location coordinate information acquiring unit 208 are ceased, wherein
Application on Voiceprint Recognition unit 201, for carrying out identification to speaker in audio data by Application on Voiceprint Recognition, with determination
Subject and caretaker's identity, so again by expression in the eyes directional information acquiring unit 202, head orientation information acquiring unit 203,
Facial expression information acquiring unit 204, gesture pose information acquiring unit 205, language emotional information acquiring unit 206, voice
Content information acquiring unit 207 and/or location coordinate information acquiring unit 208 obtain the individual features letter of subject and caretaker
Breath；
Expression in the eyes directional information acquiring unit 202, for the pixel coordinate and depth according to eyes characteristic point in video frame
Data determine the space 3D coordinate of eyes characteristic point；And nerve net is tracked using the space 3D coordinate of eyes characteristic point as sight
The input of network model, to obtain expression in the eyes directional information；
Head orientation information acquiring unit 203 for obtaining facial feature points by face detection, and determines its space
3D characteristic point cloud；The normal line vector of face's plane is determined by the 3D characteristic point cloud of facial feature points, to obtain head direction
Information；
Facial expression information acquiring unit 204 obtains face square for carrying out recognition of face to the subject in video frame
Input after shape frame and alignment as Expression Recognition model, to obtain the expression classification of subject；
Gesture pose information acquiring unit 205, the gesture detector for being trained by object detection neural network, is obtained
Take the gesture and finger directional information in video frame；
Language emotional information acquiring unit 206, for obtaining the embeding layer comprising emotion information by deep neural network
Information carries out the classification of emotion, to obtain voice mood information；
Voice content information acquisition unit 207, for the voice of speaker to be carried out text conversion by speech recognition, with
Obtain voice content information；
Location coordinate information acquiring unit 208, for the horizontal seat according to subject corresponding in video frame and/or caretaker
Depth data in mark, ordinate and corresponding depth map, and the CCD camera assembly optical parameter obtained in advance is combined, determine quilt
The 3D coordinate of examination person and/or caretaker's relative camera coordinate system.
In embodiment of the present invention, when video data and audio data on a timeline synchronize be aligned after, then obtain frame by frame
Expression in the eyes directional information, head orientation information in video frame, gesture pose information, finger directional information, facial expression information, position
Set coordinate information, language emotional information and voice content information.Wherein, specifically
Expression in the eyes directional information acquiring unit 202 is the people obtained in rgb video frame by the positive face detection using Dlib
Facial feature points combine depth data to calculate eyes characteristic point using space coordinate transformation after therefrom determining the characteristic point of eyes
Space 3D coordinate, using the space 3D coordinate of eyes as sight track neural network model input, obtain watching side
To so that it is determined that the expression in the eyes direction of subject.
Head orientation information acquiring unit 203 is by utilizing the positive face detection of Dlib obtaining the people in rgb video frame just
68 signature points of face, and obtain this 68 characteristic points for depth data, according to space coordinate transformation formula meter
The space 3D coordinate for calculating each characteristic point forms the characteristic point cloud coordinate of face；Face's plane is calculated according to characteristic point cloud coordinate
Normal line vector, and convert normal line vector to the Yaw of head pose, the Eulerian angles of Pitch, Roll indicate, so that it is determined that by
The head direction of examination person.
Facial expression information acquiring unit 204, first using ResNet20 network in faces such as general FER2013, CK+
Training Expression Recognition model on expression data collection, the Expression Recognition model trained can be identified including liking 8 kinds of anger sorrow etc. not
Same expression；Then after carrying out recognition of face by the subject to frame video frame every in rgb video, its face rectangle is obtained
Input after frame is aligned again, as Expression Recognition model, so that it may obtain the expression classification of subject.
Gesture pose information acquiring unit 205 is to train one by using object detection neural network such as Yolov3
The detector of a gesture, the detector can detect the hand in picture, identify gesture, and provide its rectangle frame coordinate and affiliated
Finger direction.
Language emotional information acquiring unit 206 is to be extracted by using the deep neural network based on ResNet comprising feelings
The embeding layer information for feeling information, then carries out the classification of emotion, to obtain the language mood of subject again.
Voice content information acquisition unit 207 is obtained by speech recognition, and the voice of speaker is carried out text and is turned
It changes, to obtain the voice content of subject.
Specifically, location coordinate information acquiring unit 208, is according to subject in video frame and/or the corresponding picture of caretaker
Abscissa u, the ordinate v and the depth data d in correspondence depth map of vegetarian refreshments, and combine the CCD camera assembly light obtained in advance
Learn parameter (cx, cy, fx, fy), determine the 3D coordinate (x, y, z) of the pixel relative camera coordinate system, in which:
Z=d
According to location coordinate information acquiring unit 208, it may be determined that the activity trajectory of subject.
Further, the feature in the autism-spectrum obstacle assessment device based on improper activity observation and analysis obtains module
It further include scene stage property location coordinate information acquiring unit, for obtaining the location coordinate information of scene stage property.
By obtaining the location coordinate information of corresponding scene stage property, the change in location of corresponding scene stage property is determined, thus really
It is whether appropriate for article use to determine subject.
According to the above acquired feature, i.e. the expression in the eyes directional information of subject and caretaker, head orientation information, gesture
Pose information, finger directional information, facial expression information, location coordinate information, language emotional information and voice content information, also
There is the location coordinate information of scene stage property, the inappropriate classifier f of activity trajectory can be trained1(x), the inappropriate classification of posture gesture
Device f2(x), the inappropriate classifier f of eye motion3(x), language expresses inappropriate classifier f4(x), article uses inappropriate classification
Device f5(x), society is referring to the inappropriate classifier f of normal form6(x) and interest tends to inappropriate classifier f7(x).Specifically, for this
Seven inappropriate classifiers to consider method as follows:
Activity trajectory is inappropriate, determines whether the activity trajectory of subject during testing exception occurs.Used spy
Sign is that the location coordinate information of subject is then marked when the stereotypies such as occurring loop turn in activity trajectory, walking up and down
It is inappropriate for activity trajectory.
Posture gesture is inappropriate, determines whether itself posture gesture of subject during testing exception occurs.It is used
Feature be subject gesture pose information.When repeat to rub hands, clap hands etc. in gesture stereotypies when, be labeled as appearance
Gesture gesture is inappropriate.
Eye motion is inappropriate, determines whether the expression in the eyes of subject during testing exception occurs.Used feature is
The sight orientation information and head orientation information of subject.When the angle of sight orientation information and head orientation information obviously occurs
It is inappropriate labeled as eye motion when inconsistent.
Language expression is inappropriate, determines whether the language of subject during testing exception occurs.Used feature is
Vocal print feature information, voice content information, the voice mood information of subject.It is duplicated as subject when mood is flat
When voice content, expressed labeled as language inappropriate.
Article uses inappropriate, when subject is using toy 5 during determining test usage mode.Used feature
The position coordinates of location coordinate information, gesture pose information for subject, real food 3 and emulation food 4 (such as banana) make
Use the time.When the gesture of subject is presented the movement thrown away, toy position coordinates and subject's position coordinates separate rapidly or toy
When coordinate is close to subject's oral area, label is using inappropriate.
Society is inappropriate referring to normal form, and subject is anti-when sound equipment plays life-stylize anxiety sound during determining test
It answers.Used feature be the facial emotions information of subject, sight orientation information, head orientation information, location coordinate information,
Speech recognition content information, voice mood information, the location coordinate information of caretaker, voice content information.When subject hears
It such as barks or when sound that child crys, does not see to caretaker, when not running to caretaker and asking for help, be labeled as social reference
Normal form is inappropriate.
Interest tends to inappropriate, and subject compares the interest-degree of toy 5 and caretaker during determining test.It is used
Feature be subject location coordinate information, the location coordinate information of toy 5 and the location coordinate information of caretaker.If subject
Time of the person close to toy 5, hence it is evident that more than the time close to caretaker, tend to inappropriate labeled as interest.
For inappropriate classification among these above, successively using support vector machines or the training of other machines learning algorithm
Each inappropriate classifier ft(x), t=1 ..., 7, and calculate its error rate e to training sample sett；Then not by this 7
Appropriate classifier stage is unified into autism-spectrum obstacle assessment models Ft(x).Therefore, autism-spectrum obstacle assessment models F can be obtainedt
(x) expression formula is
Wherein,αtFor the weight of corresponding inappropriate classifier.
In addition, refering to fig. 1 and Fig. 3, embodiment of the present invention additionally provide a kind of orphan based on improper activity observation and analysis
Only disease pedigree obstacle assessment system specifically includes foregoing invention implementation for carrying out the assessment of autism-spectrum obstacle to subject
Multiple depths that autism-spectrum obstacle in mode based on improper activity observation and analysis assesses device, is installed on experimental site surrounding
Spend image collecting device 1, the voice acquisition device 2 being set in experimental site and the scene stage property being placed in experimental site；
Wherein
Depth image acquisition device 1, for acquiring the video data during testing in experimental situation, the video data
In include image depth information；
Voice acquisition device 2, for acquiring the audio data during testing；
Scene stage property, for attracting subject to serve as inducer；
The depth image acquisition device 1 and voice acquisition device 2 all connect with autism-spectrum obstacle assessment device
It connects.
Specifically, the depth image acquisition device 1 in embodiment of the present invention can be but not limited to RGB-D camera,
Convenient for obtaining the depth information of RGB image and image；Voice acquisition device 2 includes array microphone and/or wearable microphone,
For the task sound and ambient sound during multichannel acquisition test.
Wherein, refering to fig. 1, scene stage property includes real food 3, emulates food 4, is tacked with rope to limit scope of activities
Toy 5, wherein real food 3, emulation food 4 and toy 5 are placed in opposite with caretaker present position one in experimental site
Side.
Subject is induced using real food 3, emulation food 4 and toy 5, tests the reaction of subject, and obtain subject
Person face these scene stage properties when correlated characteristic information, as expression in the eyes directional information, head orientation information, gesture pose information,
Finger directional information, facial expression information, location coordinate information, language emotional information and voice content information, and then live
Dynamic rail mark is inappropriate, posture gesture is inappropriate, eye motion is inappropriate, the language inappropriate, article of expression use it is inappropriate, social
Reference normal form is inappropriate and interest tends to inappropriate classification, to assess the ability of subject, and then assesses whether it suffers from solitarily
Disease pedigree obstacle.
Wherein, real food 3, emulation food 4 and toy 5 are placed in opposite with caretaker present position one in experimental site
Side, convenient for judging that subject compares the interest-degree of toy 5 and caretaker.
The autism-spectrum obstacle assessment system based on improper activity observation and analysis of embodiment through the invention, allow by
Examination person is move freely in standardized assay environment, the correlated characteristic data during being tested according to subject, is carried out lonely
Only disease system obstacle assessment, is conducive to the every capability state for helping doctor truely and completely to obtain children, avoids Traditional measurements
The subjectivity of evaluator in method.
The present invention invisible can induce semi-structured test, subject is enable not have by the arrangement to experimental site
It is tested in the environment of guide moveing freely, the data of acquisition can restore the real-life of subject；And due to
It is used uniformly machine learning algorithm to analyze data, scale is unified, objective, accurately can carry out autism spectrum to subject
It is obstacle assessment.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Within mind and principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (9)
1. a kind of autism-spectrum obstacle based on improper activity observation and analysis assesses device, which is characterized in that including
Data acquisition module, for obtaining audio data and video data during testing in experimental situation；
Feature obtains module, for according to the correlation of subject and caretaker during audio data and video data acquisition test
Characteristic information, the feature include expression in the eyes directional information, head orientation information, gesture pose information, finger directional information, face
One of expression information, location coordinate information, language emotional information and voice content information are a variety of；
Model training module, for successively training multiple Weak Classifiers using machine learning algorithm according to acquired feature, and
The multiple Weak Classifier is cascaded into autism-spectrum obstacle assessment models；Wherein, the multiple Weak Classifier includes activity
The inappropriate Weak Classifier in track, the inappropriate Weak Classifier of posture gesture, the inappropriate Weak Classifier of eye motion, language expression be not proper
When Weak Classifier, article tend to inappropriate referring to the inappropriate Weak Classifier of normal form and interest using inappropriate Weak Classifier, society
Any combination in Weak Classifier；
Forecast assessment module passes through autism-spectrum obstacle assessment models for the correlated characteristic information according to subject
Forecast assessment is carried out, and obtains assessment result.
2. the autism-spectrum obstacle based on improper activity observation and analysis assesses device as described in claim 1, which is characterized in that
Further include
Preprocessing module, for based on timestamp by video data and audio data on a timeline synchronize be aligned.
3. the autism-spectrum obstacle based on improper activity observation and analysis assesses device as described in claim 1, which is characterized in that
Forecast assessment module is also used to the correlated characteristic information according to subject during test, carries out autism to subject
Risk factor and/or capability analysis assessment；Wherein, the content assessed includes the movement of self, to the use of article, thought
And feeling, it is described self movement, the use to article, thought and feel to can be used for indicating the different dimensions that autism is explained,
To analyze autism.
4. the autism-spectrum obstacle based on improper activity observation and analysis assesses device as described in claim 1, which is characterized in that
The autism-spectrum obstacle assessment models Ft(x) expression formula are as follows:
Wherein, ftIt (x) is that obtained Weak Classifier, α are trained according to machine learning algorithmtFor corresponding inappropriate weak typing
The weight of device, etIt is Weak Classifier to the error rate of corresponding training sample set.
5. the autism-spectrum obstacle based on improper activity observation and analysis assesses device as described in claim 1, which is characterized in that
It includes Application on Voiceprint Recognition unit, expression in the eyes directional information acquiring unit, head orientation information acquiring unit, face that the feature, which obtains module,
Portion's expression information acquiring unit, gesture pose information acquiring unit, language emotional information acquiring unit, voice content acquisition of information
Unit and location coordinate information acquiring unit, wherein
Application on Voiceprint Recognition unit, for carrying out identification to speaker in audio data by Application on Voiceprint Recognition, to determine subject
And caretaker's identity, and then pass through expression in the eyes directional information acquiring unit, head orientation information acquiring unit, facial expression information again
Acquiring unit, gesture pose information acquiring unit, language emotional information acquiring unit, voice content information acquisition unit and/or
Location coordinate information acquiring unit obtains the individual features information of subject and caretaker；
Expression in the eyes directional information acquiring unit, for the pixel coordinate and depth data according to eyes characteristic point in video frame, really
Determine the space 3D coordinate of eyes characteristic point；And neural network model is tracked using the space 3D coordinate of eyes characteristic point as sight
Input, to obtain expression in the eyes directional information；
Head orientation information acquiring unit for obtaining facial feature points by face detection, and determines the 3D feature in its space
Point cloud；The normal line vector of face's plane is determined by the 3D characteristic point cloud of facial feature points, to obtain head orientation information；
Facial expression information acquiring unit obtains face rectangle frame simultaneously for carrying out recognition of face to the subject in video frame
Input after alignment as Expression Recognition model, to obtain the expression classification of subject；
Gesture pose information acquiring unit, the gesture detector for being trained by object detection neural network, obtains video
Gesture and finger directional information in frame；
Language emotional information acquiring unit is carried out for obtaining the embeding layer information comprising emotion information by deep neural network
The classification of emotion, to obtain voice mood information；
Voice content information acquisition unit, for the voice of speaker to be carried out text conversion by speech recognition, to obtain language
Sound content information；
Location coordinate information acquiring unit, for abscissa, the ordinate according to subject corresponding in video frame and/or caretaker
And the depth data in corresponding depth map, and combine the CCD camera assembly optical parameter obtained in advance, determine subject and/or
The 3D coordinate of caretaker's relative camera coordinate system.
6. the autism-spectrum obstacle based on improper activity observation and analysis assesses device as claimed in claim 5, which is characterized in that
The feature obtains module
Scene stage property location coordinate information acquiring unit, for obtaining the location coordinate information of scene stage property.
7. a kind of autism-spectrum obstacle assessment system based on improper activity observation and analysis, which is characterized in that including such as right
It is required that the autism-spectrum obstacle described in any one of 1-6 based on improper activity observation and analysis assesses device, is installed on experimental site
It multiple depth image acquisition devices of surrounding, the voice acquisition device being set in experimental site and is placed in experimental site
Scene stage property；Wherein
Depth image acquisition device wraps in the video data for acquiring the video data during testing in experimental situation
Include the depth information of image；
Voice acquisition device, for acquiring the audio data during testing；
Scene stage property, for attracting subject to serve as inducer；
The depth image acquisition device and voice acquisition device are all connect with autism-spectrum obstacle assessment device.
8. the autism-spectrum obstacle assessment system based on improper activity observation and analysis as claimed in claim 7, which is characterized in that
The scene stage property includes real food, emulates food, is tacked with rope to limit the toy of scope of activities, wherein true food
Object, emulation food and toy are placed in side opposite with caretaker present position in experimental site.
9. the autism-spectrum obstacle assessment system based on improper activity observation and analysis as claimed in claim 7, which is characterized in that
The depth image acquisition device is RGB-D camera, and the voice acquisition device includes array microphone and/or wearable wheat
Gram wind.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910605989.1A CN110349674A (en) | 2019-07-05 | 2019-07-05 | Autism-spectrum obstacle based on improper activity observation and analysis assesses apparatus and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910605989.1A CN110349674A (en) | 2019-07-05 | 2019-07-05 | Autism-spectrum obstacle based on improper activity observation and analysis assesses apparatus and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110349674A true CN110349674A (en) | 2019-10-18 |
Family
ID=68177829
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910605989.1A Pending CN110349674A (en) | 2019-07-05 | 2019-07-05 | Autism-spectrum obstacle based on improper activity observation and analysis assesses apparatus and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110349674A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110705523A (en) * | 2019-10-23 | 2020-01-17 | 上海大学 | Entrepreneur performance evaluation method and system based on neural network |
| CN110837784A (en) * | 2019-10-23 | 2020-02-25 | 中山大学 | Examination room peeping cheating detection system based on human head characteristics |
| CN111048200A (en) * | 2019-11-25 | 2020-04-21 | 上海交通大学 | System, method and terminal for assessing stereotypy behavior of autistic patient |
| CN111128368A (en) * | 2019-10-31 | 2020-05-08 | 东南大学 | Automatic autism spectrum disorder detection method and device based on video expression behavior analysis |
| CN111370000A (en) * | 2020-02-10 | 2020-07-03 | 厦门快商通科技股份有限公司 | Voiceprint recognition algorithm evaluation method, system, mobile terminal and storage medium |
| CN112233515A (en) * | 2020-10-12 | 2021-01-15 | 萱闱(北京)生物科技有限公司 | Unmanned examination and intelligent scoring method applied to physician CPR examination |
| CN112908476A (en) * | 2019-12-04 | 2021-06-04 | 苏州中科先进技术研究院有限公司 | Application of stress disorder test training method and test training equipment thereof |
| CN113269079A (en) * | 2021-05-20 | 2021-08-17 | 北京阿叟阿巴科技有限公司 | Children autism early evaluation device and system based on video layering comparison analysis and grade behavior paradigm |
| CN113378733A (en) * | 2021-06-17 | 2021-09-10 | 杭州海亮优教教育科技有限公司 | System and device for constructing emotion diary and daily activity recognition |
| CN114420294A (en) * | 2022-03-24 | 2022-04-29 | 北京无疆脑智科技有限公司 | Psychological development level assessment method, device, equipment, storage medium and system |
| CN114469091A (en) * | 2022-01-27 | 2022-05-13 | 杭州行熠科技有限公司 | Auxiliary diagnosis method, system, equipment and medium for autism |
| CN114842957A (en) * | 2022-03-28 | 2022-08-02 | 暨南大学 | Senile dementia auxiliary diagnosis system and method based on emotion recognition |
| CN116665310A (en) * | 2023-07-28 | 2023-08-29 | 中日友好医院(中日友好临床医学研究所) | Method and system for identifying and classifying tic disorder based on weak supervision learning |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180070810A1 (en) * | 2015-07-13 | 2018-03-15 | Megan Lorane Richardson | Method and system for detection and treatment of autism spectrum disorders and other disorders |
| CN109431523A (en) * | 2018-10-19 | 2019-03-08 | 昆山杜克大学 | Autism primary screening apparatus based on asocial's sonic stimulation behavior normal form |
| CN109830281A (en) * | 2019-01-08 | 2019-05-31 | 上海交通大学 | A kind of " name reaction " assessment system and method based on machine vision |
-
2019
- 2019-07-05 CN CN201910605989.1A patent/CN110349674A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180070810A1 (en) * | 2015-07-13 | 2018-03-15 | Megan Lorane Richardson | Method and system for detection and treatment of autism spectrum disorders and other disorders |
| CN109431523A (en) * | 2018-10-19 | 2019-03-08 | 昆山杜克大学 | Autism primary screening apparatus based on asocial's sonic stimulation behavior normal form |
| CN109830281A (en) * | 2019-01-08 | 2019-05-31 | 上海交通大学 | A kind of " name reaction " assessment system and method based on machine vision |
Non-Patent Citations (1)
| Title |
|---|
| 谷学静: "基于Unity3D 的ASD 儿童认知教育系统设计与实现", 《系统仿真学报》 * |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110837784A (en) * | 2019-10-23 | 2020-02-25 | 中山大学 | Examination room peeping cheating detection system based on human head characteristics |
| CN110705523A (en) * | 2019-10-23 | 2020-01-17 | 上海大学 | Entrepreneur performance evaluation method and system based on neural network |
| CN110837784B (en) * | 2019-10-23 | 2023-06-20 | 中山大学 | Examination room peeping and cheating detection system based on human head characteristics |
| CN110705523B (en) * | 2019-10-23 | 2021-09-24 | 上海大学 | Entrepreneur performance evaluation method and system based on neural network |
| CN111128368B (en) * | 2019-10-31 | 2023-04-07 | 东南大学 | Automatic autism spectrum disorder detection method and device based on video expression behavior analysis |
| CN111128368A (en) * | 2019-10-31 | 2020-05-08 | 东南大学 | Automatic autism spectrum disorder detection method and device based on video expression behavior analysis |
| CN111048200A (en) * | 2019-11-25 | 2020-04-21 | 上海交通大学 | System, method and terminal for assessing stereotypy behavior of autistic patient |
| CN112908476A (en) * | 2019-12-04 | 2021-06-04 | 苏州中科先进技术研究院有限公司 | Application of stress disorder test training method and test training equipment thereof |
| CN112908476B (en) * | 2019-12-04 | 2023-11-14 | 苏州中科先进技术研究院有限公司 | Application of stress disorder test training method and test training equipment thereof |
| CN111370000A (en) * | 2020-02-10 | 2020-07-03 | 厦门快商通科技股份有限公司 | Voiceprint recognition algorithm evaluation method, system, mobile terminal and storage medium |
| CN112233515A (en) * | 2020-10-12 | 2021-01-15 | 萱闱(北京)生物科技有限公司 | Unmanned examination and intelligent scoring method applied to physician CPR examination |
| CN113269079A (en) * | 2021-05-20 | 2021-08-17 | 北京阿叟阿巴科技有限公司 | Children autism early evaluation device and system based on video layering comparison analysis and grade behavior paradigm |
| CN113269079B (en) * | 2021-05-20 | 2024-01-05 | 北京阿叟阿巴科技有限公司 | Child autism early evaluation device and system based on video layering comparison analysis and hierarchical behavior paradigm |
| CN113378733A (en) * | 2021-06-17 | 2021-09-10 | 杭州海亮优教教育科技有限公司 | System and device for constructing emotion diary and daily activity recognition |
| CN114469091A (en) * | 2022-01-27 | 2022-05-13 | 杭州行熠科技有限公司 | Auxiliary diagnosis method, system, equipment and medium for autism |
| CN114420294A (en) * | 2022-03-24 | 2022-04-29 | 北京无疆脑智科技有限公司 | Psychological development level assessment method, device, equipment, storage medium and system |
| CN114842957A (en) * | 2022-03-28 | 2022-08-02 | 暨南大学 | Senile dementia auxiliary diagnosis system and method based on emotion recognition |
| CN114842957B (en) * | 2022-03-28 | 2023-04-07 | 暨南大学 | Senile dementia auxiliary diagnosis system and method based on emotion recognition |
| CN116665310A (en) * | 2023-07-28 | 2023-08-29 | 中日友好医院(中日友好临床医学研究所) | Method and system for identifying and classifying tic disorder based on weak supervision learning |
| CN116665310B (en) * | 2023-07-28 | 2023-11-03 | 中日友好医院(中日友好临床医学研究所) | Method and system for identifying and classifying tic disorder based on weak supervision learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110349674A (en) | Autism-spectrum obstacle based on improper activity observation and analysis assesses apparatus and system | |
| Kaur et al. | Prediction and localization of student engagement in the wild | |
| CN110313923B (en) | Autism early-stage screening system based on joint attention ability test and audio-video behavior analysis | |
| CN110349667B (en) | Autism assessment system combining questionnaire and multi-modal model behavior data analysis | |
| CN111046823A (en) | Student classroom participation degree analysis system based on classroom video | |
| US20040152060A1 (en) | Learning condition judging program and user condition judging system | |
| Ochoa et al. | Multimodal learning analytics-Rationale, process, examples, and direction | |
| De Carolis et al. | “Engaged Faces”: Measuring and Monitoring Student Engagement from Face and Gaze Behavior | |
| CN110363129A (en) | Autism early screening system based on smile normal form and audio-video behavioural analysis | |
| Zaletelj | Estimation of students' attention in the classroom from kinect features | |
| CN109381200B (en) | Primary screening device for autism based on separation and reunion normal form and computer vision | |
| Choi et al. | Robot-assisted ADHD screening in diagnostic process | |
| Kadar et al. | Affective computing to enhance emotional sustainability of students in dropout prevention | |
| Ray et al. | Design and implementation of technology enabled affective learning using fusion of bio-physical and facial expression | |
| CN110364260A (en) | Autism earlier evaluations apparatus and system based on indicative language paradigm | |
| Samad et al. | A pilot study to identify autism related traits in spontaneous facial actions using computer vision | |
| Ashwin et al. | Unobtrusive students' engagement analysis in computer science laboratory using deep learning techniques | |
| Güngör | The Relationship between Attitudes towards Learning and Success Orientation in Undergraduate Students. | |
| CN109447863A (en) | A kind of 4MAT real-time analysis method and system | |
| CN114385013B (en) | Remote online education system based on VR technology | |
| Holden et al. | Skills classification in cardiac ultrasound with temporal convolution and domain knowledge using a low-cost probe tracker | |
| CN110353703B (en) | Autism assessment device and system based on parrot tongue learning language model behavior analysis | |
| CN108777171A (en) | A kind of method and device of evaluation and test state of feeling | |
| Masmoudi et al. | Meltdowncrisis: Dataset of autistic children during meltdown crisis | |
| Beacon et al. | Assessing the suitability of Kinect for measuring the impact of a week-long Feldenkrais method workshop on pianists’ posture and movement |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Li Ming Inventor after: Zou Xiaobing Inventor after: Pan Yueran Inventor after: Cai Kunjing Inventor before: Zou Xiaobing Inventor before: Pan Yueran Inventor before: Cai Kunjing |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191018 |