CN110931111A - Autism auxiliary intervention system and method based on virtual reality and multi-mode information - Google Patents

Autism auxiliary intervention system and method based on virtual reality and multi-mode information Download PDF

Info

Publication number
CN110931111A
CN110931111A CN201911183662.6A CN201911183662A CN110931111A CN 110931111 A CN110931111 A CN 110931111A CN 201911183662 A CN201911183662 A CN 201911183662A CN 110931111 A CN110931111 A CN 110931111A
Authority
CN
China
Prior art keywords
user
image
data
module
intervention
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911183662.6A
Other languages
Chinese (zh)
Inventor
李明
潘悦然
蔡昆京
程铭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Duke University Of Kunshan
Original Assignee
Duke University Of Kunshan
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Duke University Of Kunshan filed Critical Duke University Of Kunshan
Priority to CN201911183662.6A priority Critical patent/CN110931111A/en
Publication of CN110931111A publication Critical patent/CN110931111A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
    • G16H20/70—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mental therapies, e.g. psychological therapy or autogenous training
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
    • G16H10/60—ICT specially adapted for the handling or processing of patient-related medical or healthcare data for patient-specific data, e.g. for electronic patient records

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Epidemiology (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Primary Health Care (AREA)
  • Public Health (AREA)
  • Psychiatry (AREA)
  • Computer Hardware Design (AREA)
  • Hospice & Palliative Care (AREA)
  • Child & Adolescent Psychology (AREA)
  • Psychology (AREA)
  • Social Psychology (AREA)
  • Computer Graphics (AREA)
  • Developmental Disabilities (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an autism auxiliary intervention system and method based on virtual reality and multi-mode information, wherein the system comprises: the system comprises a scene customization module, a real image acquisition module, a real image fusion module, a behavior data acquisition module, a behavior characteristic extraction module, a training module, a prediction module, an interaction module and a course evaluation module. The invention provides the autism auxiliary intervention teaching by using the virtual reality equipment, so that the intervention learning is popularized in different occasions; the customized scene and the figure image are used for teaching, the intervention course learning scene is transferred to the real life, and the intervention effect is enhanced; the image fusion technology is used for fusing the image images of real people and objects into the customized scene, so that the course is more intimate to the user; the sound image of the real person and object is fused into the customized scene by using a sound fusion technology, so that the person and object image is more three-dimensional; various user behavior data are collected for analysis, user behavior characteristics are comprehensively and timely obtained, and teaching feedback is favorably obtained.

Description

Autism auxiliary intervention system and method based on virtual reality and multi-mode information
Technical Field
The invention relates to the field of autism treatment, in particular to an autism auxiliary intervention system and scheme based on virtual reality and multi-mode information.
Background
Autism, also known as Autism Spectrum Disorder (ASD), is a representative disease of Pervasive Developmental Disorder (PDD). People with autism may have abnormal manifestations in comprehension, communication, social, and interest. More than 200 ten thousand children in China now suffer from autism. Studies have shown that the gold period for autism intervention in children is 1-6 years of age. How to intervene and treat the autism patient in time is a social problem which needs to be paid extensive attention. Currently, autism therapy includes: methods of treatment of gambling, language associations, use of enhancers, occupational therapy, and the like. Professors in pawns propose a structured social behavioral intervention model (BSR model) that enables effective autism intervention.
However, the professional treatment is difficult to popularize due to the high requirements of the professional and standard of the therapist for the intervention treatment. With the development of the fields of computer science and artificial intelligence, the popularization of the autism intervention method by using portable equipment is a research target of many scientific researchers. Among them, the treatment scheme using virtual reality devices to assist intervention has already made a certain research progress, and can provide intervention schemes for users in different places such as hospitals, schools and families. At present, a scheme is provided for analyzing the perception degree of a user by combining a virtual reality technology with analysis of biological signals such as heart rate data and arm activity data, and evaluating the training effect of the user by selecting options through a handle. However, the common non-customized general virtual reality scene still easily causes the autism patient to generate a virtual feeling, and is not favorable for the migration from the learning scene to the real life scene. Moreover, the measurement of biological signals such as heart rate requires children to wear more instruments, which is inconvenient to use and the direct readability and interpretability of biological signals is not strong. The evaluation judgment of the answer is carried out through the handle, only the analysis level of the condition judgment is remained, other multi-mode behaviors and interaction information of the user are omitted, and the interactivity is poor.
If a virtual reality system can be developed, real scenes and real characters can be customized in a personalized manner, the acceptance of autism patients is increased, action signals of the activities of the autism patients are explained directly, and enhanced teaching with different strengths is provided in time according to the analysis and training effect degree of the multi-mode action signals of the users, so that the interactivity of the intervention system can be enhanced, and the effect of auxiliary intervention can be enhanced.
Disclosure of Invention
Aiming at the technical problems, the invention aims to provide an autism auxiliary intervention system and method based on virtual reality and multi-mode information, an intervention scene is fused into image images and sound images of real people and objects, places where participants use wearable equipment are not limited, multi-mode behavior signals of users with explanatory performance are collected and analyzed in real time, strong interactive teaching training is realized through multi-mode multi-intensity levels of sound, images and optical signals, and teaching suggestions are provided in combination with user performance, so that the mobility and the ductility of auxiliary intervention teaching are increased, more personalized teaching is realized, and the curative effect of auxiliary intervention is better enhanced.
In order to realize the purpose, the invention is realized according to the following technical scheme:
the invention provides an autism auxiliary intervention system based on virtual reality and multi-mode information, which comprises:
the course customizing module is used for making virtual reality scenes and teaching contents for outputting sound information and image information;
the file module is used for initializing and establishing a user personal file and a training plan according to medical records or evaluation results, recording user training data, and enabling the training plan to be generated by a manual modification system;
the real image acquisition module is used for acquiring sound and image data of the fused person or object;
the real image fusion module is used for learning the sound and image images of the fused person or object collected by the real image collection module, extracting the sound and image characteristics of the fused person or object, fusing the extracted sound and image characteristics with the virtual reality scene manufactured by the tutorial customization module to obtain the virtual reality fused person or object image with the sound and image images of the fused person or object, wherein the virtual reality fused person image can speak by using the sound color of the original fused person and make different appearances, expressions and actions by using the image of the original fused person, and the virtual reality fused object image emits the sound of the original fused object and shows an equal-ratio stereo image of the original fused object;
the behavior data acquisition module is used for acquiring multi-modal behavior data of the user in the intervention test process;
the behavior feature extraction module is used for extracting the behavior features of the face orientation, the eye spirit, the gesture, the position, the emotion and the language of the user in the experimental multi-modal behavior data collected by the behavior data collection module;
the training module is used for training the data of the behavior characteristics extracted by the behavior characteristic extraction module through a machine learning algorithm to obtain an evaluation model which accords with a scene;
the prediction module is used for analyzing the behavior characteristics of the user through the trained evaluation model obtained by the training module so as to obtain the user performance description;
the interactive module is used for carrying out information communication according to the user performance description obtained by the prediction module, and directly prompting to suggest or prompt the user to carry out correct behavior response by outputting sound information, image information and optical information in a scene;
and the course evaluation module is used for performing overall evaluation on the user learning according to the user course performance and the course learning progress in the whole course of the intervention test and providing related suggestions.
Preferably, the course customization module performs course customization by reference to structured social behavioral intervention patterns, behavioral therapy, key response training, verbal behavior, relationship development intervention, early Danver pattern, structured education, floor time.
Preferably, the real image collecting module includes:
the voice acquisition unit is used for acquiring the speaking voice data of a person and the voice of an animal or other articles in a course scene;
the image acquisition unit is used for acquiring multi-angle plane or depth image data of a user so as to obtain a two-dimensional or three-dimensional model image of the user.
Preferably, the real image blending module blends the image of the person or object in the real life into the course of the customized scene, so as to realize the effect of blending the head, the face and the sound of the real person and blending the image and the sound of the real object in equal proportion, so that the blended person and object image are more three-dimensional, and the method specifically comprises the following steps:
the specific person voice synthesis unit is used for extracting the sound information of tone according to the provided audio data and synthesizing the character voice fused into the virtual reality scene by utilizing a multi-speaker voice synthesis technology;
an image synthesis unit for extracting image information of a head and a motion from the supplied character data and synthesizing a character moving image merged into the virtual reality scene; and extracting image information of the shape and the size according to the provided article data, and synthesizing the movable rotating article image blended into the virtual reality scene.
Preferably, the behavior data acquisition module acquires data by using virtual reality equipment with an internal eye tracker, a sensor, a gyroscope, a microphone, an intra-cavity camera, an extra-cavity camera, a handle and gloves,
the method comprises the steps of obtaining orientation and movement data of a head and a body by using a sensor and a gyroscope, obtaining eye movement data by using a built-in eye tracker, obtaining movement data of muscles and skin around eyes by using an intracavity camera, obtaining voice data by using a microphone, obtaining muscle movement data of a chin and two cheeks by using an extracavity camera to shoot a lower half of a face, and obtaining hand movement data and course learning operation data by using a handle.
Preferably, the behavior feature extraction module includes:
a head direction extraction unit which calculates the movement of the head according to the moving freedom degree of the x, y and z rectangular coordinate axes of the object and the rotation freedom degree around the three coordinate axes by using a sensor or a gyroscope;
the facial expression extraction unit is used for combining the data of the left and right eyes of the user and the data of the mouth acquired by the VR helmet intracavity camera and the VR helmet extraluminal camera respectively and calculating a combined expression feature classification vector;
an eye focus extraction unit which uses the data of the eye movement tracking of the eye tracker to calculate the direction and focus of the eye movement;
the hand motion extraction unit is used for acquiring options made by a user for a scene question by using data acquired by the handle or the gloves, the sensor and the button, and calculating the motion mode and amplitude of the user in cooperation with the scene;
the voice feature extraction unit is used for obtaining the speaking data of the user by using the microphone, obtaining the speaking voice content of the user by voice recognition, extracting and analyzing keywords and comprehension paraphrases of the phrases, and obtaining the emotion scores of the interactive languages of the user by emotion recognition;
and the relative position extraction unit is used for obtaining the individual movement acceleration of the user by using the VR helmet sensor and calculating the movement track of the user.
Preferably, the behavior interaction module realizes a strong interaction effect between the user and the equipment, takes characteristic data of user behavior as input, uses machine learning analysis in real time, carries out grading feedback output according to the behavior data of the user, and circularly inputs and outputs to realize strong interaction; wherein the feedback output comprises a reinforced teaching output and a raised encouragement output;
preferably, the enhanced teaching output includes different grade cues or explicit reminders, the enhanced teaching output being implemented by sound information, image information, or optical information.
Preferably, the course evaluation module performs integrated calculation by combining the data recorded in the user profile module and the performance of the current training of the user, and provides suggestions for the training plan of the user, wherein the calculation method of the integrated result includes a method not limited to using condition judgment or machine learning.
The invention also provides an autism auxiliary intervention method based on virtual reality and multi-mode information, which is realized according to the autism auxiliary intervention system and comprises the following steps:
s1: preparing informed information and establishing a file: a guardian or a nurse of a user reads an informed consent, and the awareness system acquires and analyzes behavior data and medical record related data of the user and an assisted intervention fused person, knows that the system does not leak user data to protect the privacy of the user, and establishes a file after confirming the consent;
s1: establishing a file: filling basic data of a user, including sex, age, development condition, family members and medical advice, by the user, the guardian or the caretaker according to the prompt and the description of the virtual reality equipment or the computer;
s2: establishing a teaching plan: generating a teaching plan according to the user file, and manually adjusting the user or a guardian or a caretaker of the user according to the actual situation to determine the teaching plan;
s3: image collection and integration: a guardian or an intervention auxiliary teacher of a user or a person living together with the guardian or the intervention auxiliary teacher for more than two weeks serves as a person to be integrated, and image data acquired by an RGBD camera or multi-angle RGB image data is provided; recording with a specified length or longer; integrating the voice and the image of the person to be integrated into the scene of the designed intervention course in the user teaching plan to obtain a customized course with the image and the voice image of the person to be integrated;
s4, auxiliary intervention test: the user performs strong interactive intervention test training according to a teaching training scene, and a system acquires multi-modal data of the user in an experiment;
s5: and (3) training and summarizing: and analyzing by the system to obtain the training score of the user according to all the training on the current day, judging whether the training on the current day is qualified, and recommending a subsequent training plan.
Compared with the prior art, the invention has the following advantages:
the invention provides the teaching of the autism auxiliary intervention by using the virtual reality equipment, so that the intervention learning is easier to popularize in different occasions.
The invention uses customized scene courses and character images for teaching, is more favorable for transferring the learning scene of the intervention course to the real life and enhances the intervention effect.
The invention uses the image fusion technology to fuse the image images of real people and objects into the customized scene, so that the course is more intimate to the user.
The invention uses the sound integration technology to integrate the sound images of real people and objects into the customized scene, so that the images of people and objects in the course are more three-dimensional and are easier to be accepted by users, and the intervention effect is better.
The invention collects and analyzes various user behavior data, more comprehensively and timely acquires the user behavior characteristics, and is more favorable for acquiring teaching feedback.
The invention uses various effects of sound, image and optics to guide the teaching of the user, and can cultivate the user more naturally and invisibly.
The invention can timely provide reinforced teaching with different strengths according to the multi-modal behavior signal analysis training effect degree of the user, and is beneficial to providing strong interactive personalized teaching.
The invention provides teaching suggestions in time by combining with the teaching effect, is more favorable for updating and adjusting the teaching strategy and assists in achieving a better intervention effect.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic block diagram of a system of the present invention;
FIG. 2 is a general flowchart of the teaching of the present invention;
FIG. 3 is a flow chart of a single course teaching step of the present invention;
FIG. 4a is a schematic diagram of voiceprint recognition of a specific person;
FIG. 4b is a flow chart of person-specific speech synthesis;
fig. 5 is a schematic diagram of a head-mounted VR device acquiring a facial image.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention.
As shown in fig. 1, the embodiment of the present invention discloses an autism assistance intervention system based on virtual reality and multi-modal information, comprising:
the course customizing module is used for making virtual reality scenes and teaching contents for outputting sound information and image information;
the course customization module refers to a structured social behavior intervention mode (BSR mode), a behavioral therapy (ABA), a key response training (PRT), a speech behavior training method (VB), a relationship development intervention therapy (RDI), an early Danver mode (ESDM), a structured teaching method (TEACCH) and a Floor time therapy (Floor time) to customize the course.
In this embodiment, the course customization module performs course customization according to a structured social behavior intervention mode (BSR mode);
the file module is used for initializing and establishing a user personal file and a training plan according to medical records or evaluation results, recording user training data, and enabling the training plan to be generated by a manual modification system;
in this embodiment, the training program is created using a physician scoring or other autism assessment method. And (3) scoring according to the dimensions of simulation ability, sensory behavior ability, stereotypy behavior ability, language understanding ability, game behavior ability, social behavior ability, language expression ability and the like of the user evaluated by doctor evaluation or other manners, selecting courses with different dimensions and corresponding difficulties according to the gender, age and multi-dimensional scores of the user, and integrating the courses to be used as the training plan of the user.
The real image acquisition module is used for acquiring sound and image data of the fused person or object;
the real image fusion module is used for learning the sound and image images of the fused person or object collected by the real image collection module, extracting the sound and image characteristics of the fused person or object, fusing the extracted sound and image characteristics with the virtual reality scene manufactured by the tutorial customization module to obtain the virtual reality fused person or object image with the sound and image images of the fused person or object, wherein the virtual reality fused person image can speak by using the sound color of the original fused person and make different appearances, expressions and actions by using the image of the original fused person, and the virtual reality fused object image emits the sound of the original fused object and shows an equal-ratio stereo image of the original fused object;
the behavior data acquisition module is used for acquiring multi-modal behavior data of the user in the intervention test process;
the behavior feature extraction module is used for extracting the behavior features of the face orientation, the eye spirit, the gesture, the position, the emotion and the language of the user in the experimental multi-modal behavior data collected by the behavior data collection module;
the training module is used for training the data of the behavior characteristics extracted by the behavior characteristic extraction module through a machine learning algorithm to obtain an evaluation model which accords with a scene;
the prediction module is used for analyzing the behavior characteristics of the user through the trained evaluation model obtained by the training module so as to obtain the user performance description;
the interactive module is used for carrying out information communication according to the user performance description obtained by the prediction module, and directly prompting to suggest or prompt the user to carry out correct behavior response by outputting sound information, image information and optical information in a scene;
and the course evaluation module is used for performing overall evaluation on the user learning according to the user course performance and the course learning progress in the whole course of the intervention test and providing related suggestions.
The real image acquisition module of the invention comprises:
the voice acquisition unit is used for acquiring the speaking voice data of a person and the voice of an animal or other articles in a course scene;
the image acquisition unit is used for acquiring multi-angle plane or depth image data of a user so as to obtain a two-dimensional or three-dimensional model image of the user.
In an embodiment of the invention, the image acquisition uses a 3D scan reconstruction technique. Collecting color and depth image data of the head and the periphery of the body, performing point cloud calculation and registration fusion, performing meshing and post-processing on the point cloud by using Meshlab, and reconstructing a 3D model. And calculating the point cloud of each point in the picture by combining the collected RGB picture and the depth data, and performing rough registration matching on the point cloud of each angle according to the external parameter matrix of the images of the plurality of angles. On the basis of rough registration point cloud, ICP (iterative Closest point) iterative Closest point algorithm is used for further fine registration and fusion, and then point cloud of several frames of continuous picture data in a short time is subjected to average smoothing in one time. And finally, performing post-processing on the point cloud by using the Meshlab, performing mesh triangulation and spatial smooth filtering after down-sampling of the point cloud data, and coloring according to the RGB data corresponding to the point cloud.
The real image integration module of the invention comprises:
the specific person voice synthesis unit is used for extracting the sound information of tone according to the provided audio data and synthesizing the character voice fused into the virtual reality scene by utilizing a multi-speaker voice synthesis technology;
an image synthesis unit for extracting image information of a head and a motion from the supplied character data and synthesizing a character moving image merged into the virtual reality scene; and extracting image information of the shape and the size according to the provided article data, and synthesizing the movable rotating article image blended into the virtual reality scene.
Further, as shown in fig. 4a and 4b, the voice synthesis unit combines end-to-end deep learning-based voiceprint recognition with voice synthesis using a person-specific voice synthesis technique, and merges the current high-fidelity voice synthesis system and the high-accuracy voiceprint system, and the two systems are merged to form a countermeasure system. The whole architecture system can be divided into three components: acoustic models, vocoders, and voiceprint models. In the training process, the information flow direction is as follows: 1. training voice, extracting acoustic features, and acquiring speaker features through a voiceprint system; 2. the speaker characteristics and the text simulate acoustic parameters through an acoustic model; 3. the acoustic parameters restore the sound through a vocoder; 4. and carrying out classification decision on the acoustic parameters obtained by simulation through a voiceprint system, and carrying out similarity evaluation on the synthesized speaker characteristics output by the mapping layer. The three modules work cooperatively, wherein the voiceprint systems used in the steps 1 and 4 are shared, so that the consistency of the whole framework can be guaranteed, namely, the voiceprint systems really play a role in distinguishing the synthesized voice from the real voice in the whole system, and can provide valuable speaker feature vectors for the whole system and can be used as the input of a decision layer.
And an image synthesis unit for generating a face-changed video using a face model of a specified object by using a technique related to an auto-encoder and generation countermeasure network in deep learning. The technology comprises the steps that firstly, a shared depth encoder and two independent depth decoders learn the hidden features (mainly comprising information such as positions of five sense organs and expressions) of two faces A and B respectively, then the positions of the decoders are exchanged, the hidden features of the face A are restored by the decoder of the face B, and the face A in an image can be changed into the face B.
The behavior data acquisition module comprises:
the virtual reality device is internally provided with an eye tracker, a sensor, a gyroscope, a microphone, an intracavity camera, an extraluminal camera, a handle and gloves, orientation and movement data of the head and the body are obtained by using the sensor and the gyroscope, eye movement is obtained by using the eye tracker, movement data of the skin and the eyes of the eyes are obtained by using the intracavity camera, speaking data are obtained by using the microphone, the movement data of the chin and the two cheeks are obtained by using the extraluminal camera to shoot the next half face, and hand movement data and course learning operation data are obtained by using the handle.
The behavior feature extraction module comprises:
the head orientation extraction unit calculates the motion of the head according to the moving freedom degree of the x, y and z rectangular coordinate axes of the object and the rotation freedom degree around the three coordinate axes by using an acceleration sensor or a gyroscope which is arranged on the head orientation extraction unit;
the facial expression extraction unit is used for combining the data of the left and right eyes of the user and the data of the mouth acquired by the VR helmet intracavity camera and the VR helmet extraluminal camera respectively and calculating a combined expression feature classification vector;
an eye focus extraction unit which uses the data of the eye movement tracking of the eye tracker to calculate the direction and focus of the eye movement;
the hand motion extraction unit is used for acquiring options made by a user for a scene question by using data acquired by the handle, the gloves, the sensor and the button, and calculating the motion mode and amplitude of the user in cooperation with the scene;
the voice feature extraction unit is used for obtaining the speaking data of the user by using the microphone, obtaining the speaking voice content of the user by voice recognition, extracting and analyzing keywords and comprehension paraphrases of the phrases, and obtaining the emotion scores of the interactive languages of the user by emotion recognition;
and the relative position extraction unit is used for obtaining the individual movement acceleration of the user by using the VR helmet sensor and calculating the movement track of the user.
Furthermore, the facial expression extraction unit is used for respectively shooting images of the left eye, the right eye and the periphery of the eyes through three cameras on the head-mounted VR equipment, wherein two cameras are positioned in the glasses; one is mounted outside the device and is used for shooting high-definition images of the face at a short distance. The method comprises the steps of performing feature extraction on images collected by three cameras by channels based on a deep neural network, and finally fusing physical signs extracted by different channels to output expression recognition results. The three channels are pre-trained by using the existing large-scale data set and then transplanted to a local framework, and self-adaptation is carried out on the real user facial expression data set collected by the VR equipment.
In this embodiment, a schematic diagram of the head mounted VR device acquiring a face image is shown in fig. 5.
Further, the voice feature extraction unit uses an end-to-end deep learning technology when extracting the voice emotion feature. The specific technology comprises model structure, loss function design, data enhancement and transfer learning. In terms of model structure, the coding layer scheme is used for construction. In the aspect of loss function design, scale information in the emotion two-dimensional representation is applied to optimization. In terms of data enhancement, more efficient data enhancement methods are employed to scale up the training data. On the premise of not greatly influencing quality, training data is selectively expanded by adding a small amount of noise, reverberation, equalizer effect, custom filtering, voice change of specific people, change of fundamental frequency, change of speed of speech, transcription of different channels and the like. Generally, a highly uniform end-to-end deep Learning framework is adopted, and existing multiple databases are better utilized to cooperatively research the migration Learning technology and the Multitask Learning technology by utilizing Multitask Learning (multi task Learning) and joint Learning (joint Learning) methods of multiple databases.
Further, the training module is used for training the user behavior characteristic data through a machine learning algorithm, learning the scores of reactions such as learning simulation and answers of the user in the corresponding intervention training scene according to the characteristic data of the language, the expression and the action of the user, and acquiring an evaluation model of scene learning through the user characteristic data.
In this embodiment, a model of various feature data and scene training scores is constructed using a support vector machine algorithm. When the binary classification is needed, a support vector machine is used for calculation; when multiple classifications are needed to evaluate different levels of user performance, a plurality of two-classification support vector machines are used for combined classification. Specifically, the support vector machine of the binary problem takes the evaluation score as a learning target Y of the support vector machine, and multi-modal characteristic data form a multi-characteristic sample Xi＝[x1，x2，-xn]As input data, according to the following equation ω2X + b is 0, and the optimum ω can be obtained by training2The corresponding Y can be calculated. When multiple classifiers are used for realizing multi-classification, samples of a certain class are classified into one class and other samples are left in the class during trainingThe rest samples are classified into another class, so that a plurality of support vector machines are constructed by a plurality of classes of samples. The classification classifies the unknown sample as the class having the largest classification function value.
Furthermore, the interaction module of the invention uses the trained model to grade and judge the user characteristic data and evaluate whether the user behavior performance meets the learning effect standard required by the intervention course. And analyzing characteristic data, including answer options selected by the user through handle operation and the collected and extracted multi-modal behavior characteristic data of the user. When the learning effect or the answering score of the user reaches the course standard, recording the user score and entering a subsequent learning link; when the learning effect or the answering score of the user does not reach the course standard, recording the score of the user, performing intervention teaching again, and adding sound information, image information and optical information to strengthen the teaching user to make a correct learning response; when the user still cannot correctly feed back after the weak-level teaching, recording the user score and performing intervention teaching again, and directly prompting the user to make a correct learning response by using sound information, image information and optical information; and when the user throws the teaching material after the strong-level prompt and cannot correctly feed back, recording the user performance.
In the embodiment, the voice information is a voice prompt of the integrated real character image; the image information is the action demonstration and the character teaching content of the integrated real figure; the optical information guides the user to focus the important points needing attention by blurring and dimming the surrounding environment, focusing clearly and brightening the objects needing attention. The reinforced teaching of the weak classes includes: the optical information is used for blurring the surrounding environment, focusing key pictures clearly, prompting key words by sound, and playing and responding to related articles and partial actions by using image pictures. The strong level of reinforcement teaching includes: the optical information is used for directly dimming the surrounding environment and brightening key pictures, the required response operation is directly indicated by using sound, and the required response operation is directly played by using image pictures.
Further, the course evaluation module obtains the result of one or more courses of the user on the same day according to the learning result recorded by the interaction module, obtains the total score of the learning on the same day, and puts the score into the user file. And according to the scored ability level and the user file, a machine learning model or condition judgment calculation is used for suggesting a subsequent course plan.
In this embodiment, when the training score reaches the hit line for one week or reaches the high-split line for 2 days, a course with higher difficulty level is recommended to be replaced; when the achievement can not reach the goal line stably in one continuous week, the system updates the suggestion to change different courses at the same level.
In this embodiment, the activity program is taught by speaking voice and action pictures through the merged family avatar of the caretaker of the user (such as mom of the user, dad of the user, hereinafter abbreviated as mom and dad) according to the structured social behavior intervention mode (BSR mode provides a series of training scenes for the user of children suffering from autism).
The first embodiment is as follows: morning course
Getting up to train: in a family bedroom scene, the children are guided to learn to use the handle to carry out quilt lifting and getting off the bed through the alarm clock sound and the voice of mother calling to get up. When the child does not operate the specified action successfully, the information is enhanced by using the sound and picture information, and the teaching is repeated.
Toilet training: in the scene at the door of the toilet, a child stands in front of a closed toilet door, the user is guided to knock the door through sound and picture information, a child dad is told to go to the toilet inside by the voice of dad to tell the child dad that 'dad is inside and please wait for a moment', the child is taught to queue up for a moment of the toilet, and the phenomenon that the child mom teaches that the child uses an inquiry sentence to negotiate 'dad' with dad in the toilet, please feel a bit sooner, and the child is urgent for urination. When the child does not operate the specified action successfully, the information is enhanced by using the sound and picture information, and the teaching is repeated.
Example two: movable course
And (3) stability ability training: in the studio scene, dad says "begin to draw a painting lesson now", while children stabilize the painting and do not move, mom is rewarded dad and children; in the picture, when father stands up and leaves the seat, mom beats the hand of father and educates children to' baby, you see, dad does not listen to the phone, and beats it.
Jigsaw training: in the scenario of a toy room, mom and children play a jigsaw puzzle, first know in language that the child remembers the jigsaw puzzle picture "baby," you remember the picture bar, "and then instruct the child in language to select the correct puzzle" where is the piece? "baby, you are very young, spelled in! "finally, train child to pick up the toy played with the handle" time out, we put the puzzle back in the box.
Example three: cognitive course
Animal cognitive training: in the park scene, a mother and a child learn about pets, the mother asks about what animal is, the child answers to the animal by voice, and the mother corrects and evaluates the answer. The mother asks the question where the kitten is, the child points the object through the handle, and the mother evaluates and feeds back according to the answer.
Fruit cognitive training: in a fruit store, dad educates children to know what is different fruits? ", the child answered the fruit by voice, dad asked" what color was apple? ", the child answered by voice, dad indicated positive and rated the child's answer.
Example four: teaching way for going out
And (3) road passing training: in the road scene, mom educates children to see the red light and can not cross the road, and mom indicates that children can cross the road by seeing the red light and the green light when turning green light. To the next intersection, mom asks baby "what lamp is now? Can we not cross the road? "
Vegetable buying training: in the scene before going out, mom educates children "baby" and we go to buy the dish. In a vegetable field scene, a mother asks a question "telling mother where carrots are? ", the child answers by voice and handle. Mom evaluates and then asks "baby, how do we buy several at a time? "and educate children to put the dish in the basket with the handle. Aunt does a settlement teaching? "finally, mother tells child" who laughs aunt who won you, and won you see again ".
In this embodiment, as shown in fig. 2, the user will perform training in different courses every day, obtain a summary of teaching performance in one day and teaching advice provided by the system, and perform subsequent learning with reference to the teaching advice. Specifically, firstly, a user obtains a training plan containing various courses on the current day according to a file; then, the user respectively learns different courses according to the training plan, and the system records the training results of the user; then, the system marks and passes the file according to the training result of the day, arrange the subsequent training plan; so going to the next day, the user and the system repeat the aforementioned steps.
As shown in fig. 3, a specific teaching and training process is as follows: in a specific teaching scene, firstly, outputting sound and image signals through a teaching system, and learning a certain autism auxiliary intervention system through a video teaching user integrated with a real character, wherein the system is characterized in that the system is recognized; then, asking the user to answer questions related to teaching through a teaching system, or asking the user to simulate certain behaviors; then, the user answers the teaching by using the handle or by self-behavior; next, the system evaluates the response score of the user through a prediction module, and judges whether the response is qualified or not and whether the teaching needs to be strengthened or not; when the user behavior is judged to need to be subjected to enhanced teaching, the system performs enhanced teaching according to the corresponding grade through multi-mode signals according to the grade of user response and the turn of enhanced teaching; when the user is judged to need to enhance teaching again, the system enhances the level according to the increase of the turns, and performs the enhanced teaching on the user; when the response performance of the user is evaluated to be qualified, the system carries out promotion on the user through sound and image signals; when the response performance of the user is evaluated to be qualified or the turn of strengthening the teaching reaches the upper limit, the teaching process of the training is finished, and the system records the training evaluation of the user in the whole process.
In this embodiment, the invention is used for assisting in the intervention of autism for a user, and the use flow of the user is as follows:
s1: informed preparation and archive establishment
A guardian or a nurse of a user (a person subjected to auxiliary intervention) reads an informed consent, knows that the system can collect and analyze behavior data and medical record related data of the user and the person subjected to auxiliary intervention, knows that the system does not leak the user data to protect the privacy of the user, and can establish a file after confirming the consent;
s1: file creation
And filling basic data of the user, including sex, age, development condition, family members and medical orders, by the user, the guardian or the caretaker according to the prompt and the description of the virtual reality equipment or the computer.
S2: teaching plan establishment
The system generates a teaching plan according to the user file, and the user or a guardian or a caretaker thereof manually adjusts according to the actual situation to determine the teaching plan.
In this embodiment, the teaching plan of the user is: the vegetable buying training in the above-described tutorial embodiment.
Vegetable buying training: in the scene before going out, mom educates children "baby" and we go to buy the dish. In a vegetable field scene, a mother asks a question "telling mother where carrots are? ", the child answers by voice and handle. Mom evaluates and then asks "baby, how do we buy several at a time? "and educate children to put the dish in the basket with the handle. Aunt does a settlement teaching? "finally, mother tells child" who laughs aunt who won you, and won you see again ".
S3: image collection and integration
Parents of users or intervening auxiliary teachers or people living together with the auxiliary teachers for more than two weeks are taken as persons to be integrated, and image data collected by the RGBD camera or multi-angle RGB image data are provided; and recording of not less than a specified length is performed. The system blends the voice and the image of the person to be blended into the scene of the designed intervention course in the user teaching plan to obtain the customized course with the image and the voice image of the person to be blended.
In the embodiment, the fusing person is the mother of the user, the RGBD image is collected from the mother of the user by using a 3D scanning restoration technology, and after the collected RGBD image is fused by using a self-encoder in deep learning and a countermeasure network generation related technology, the virtual reality mother role image in the vegetable buying course is presented as the head portrait of the user mother; the voice of the user's mother is recorded, the voice of the user's mother is integrated into the system through the specific human voice synthesis unit, and the virtual reality mother role in the course vividly speaks the tone of the voice content designated by the system as the voice of the user's mother.
S4 auxiliary intervention test
The user carries out strong interactive intervention test training according to a teaching training scene, and the system collects multi-mode data of the user in an experiment.
In this embodiment, after wearing the virtual reality helmet and gloves, the user enters into a vegetable buying training. The virtual reality presents a starting scene, and the virtual mother role integrated into the image of the user mother firstly calls the voice and the head portrait root of the user mother so as to enable the user to be familiar with the virtual reality environment. The virtual mom role uses the face of the head portrait of the mom of the user to make smiling expressions and speaking actions, and the virtual mom role speaks to the user: "baby, we go to buy the dish bundle". The virtual reality shows the scene of the vegetable garden, the virtual mother role uses the sound of the user mother to provide a first question to the user, "tell the user where carrots? "the user indicates the location of the carrot in the virtual reality scene using the glove. According to the collection of the user actions by the sensors on the gloves and the analysis of the system, the user is judged by the system to be successfully pointed to the carrots. The virtual mom role is clapped and carries out voice expression on the user, namely baby, true club! This is a carrot, you are too clever! "the user laughs. The system collects facial expressions of the user through the intracavity camera and the extraluminal camera on the virtual reality helmet, collects laughter of the user through the microphone on the virtual reality helmet, analyzes facial data and sound to judge that the user laughs, and enters into learning on the next step. The virtual mom role asks the user the second question, "baby, does we buy several things at a time? "then, the virtual mom role demonstrates, meaning the carrot," baby, you see, a few in total? "the user fails to answer, the no-reaction time exceeds the reaction time set by the system. The system prompts the user in a cued mode, other pictures except the carrots are blurred, the carrot pictures are emphasized, and the virtual mom role asks questions again: "baby, you see again, a few carrots? Is the utterance tells mom how good? "the user fails to answer, the no-reaction time exceeds the reaction time set by the system. The system clearly reminds the user, the virtual character mother makes a motion of counting radishes, and accurately indicates that each radish is a baby, and what is a total of 1, 2, 3, a? "the user answers" 3 "using the microphone. The system detects that the user has answered correctly by analyzing the sound data collected by the microphone and enters the next question. Is the virtual mother role ask "baby, how well we put carrot in the vegetable basket? "the user uses the handle to place the dish in the basket and the system detects that the answer is successful based on the glove sensor data. Virtual mom horn carries out line settlement teaching' do you give money to aunt? The user uses the glove to cover money of the role of the virtual mother in the virtual reality scene, the money is handed to the aunt virtual salesman, and the system detects that the answer is successful according to the data of the glove sensor. Finally, the virtual role mother makes the action of waving the forever, and teaches the child to laugh the aunt who refuses to be remitted and come to the aunt again. The user waves and smiles and the aunt role of the virtual salesman says goodbye. The system detects that the user waves the hand according to the data of the glove sensor, recognizes the smiling expression of the user according to the data image expressions of the intracavity camera and the extraluminal camera, recognizes that the user says ' aunt ' and then sees ' according to the data voice collected by the microphone, and analyzes the emotion of the user according to the data voice emotion collected by the microphone to be happy, so that the problem is completed. And ending the course.
S5: training summary
And analyzing by the system to obtain the training score of the user according to all the training on the current day, judging whether the training on the current day is qualified, and recommending a subsequent training plan.
In this embodiment, the questions or instructions in the training are scored, for a total of 4 questions and 2 instructions. Because the user answers a question only through obvious prompt and cannot reach high branching, the system judges that the user continues training with the same difficulty and type courses tomorrow.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.

Claims (10)

1. An autism assistance intervention system based on virtual reality and multimodal information, comprising:
the course customizing module is used for making virtual reality scenes and teaching contents for outputting sound information and image information;
the file module is used for initializing and establishing a user personal file and a training plan according to medical records or evaluation results, recording user training data, and enabling the training plan to be generated by a manual modification system;
the real image acquisition module is used for acquiring sound and image data of the fused person or object;
the real image fusion module is used for learning the sound and image images of the fused person or object collected by the real image collection module, extracting the sound and image characteristics of the fused person or object, fusing the extracted sound and image characteristics with the virtual reality scene manufactured by the tutorial customization module to obtain the virtual reality fused person or object image with the sound and image images of the fused person or object, wherein the virtual reality fused person image can speak by using the sound color of the original fused person and make different appearances, expressions and actions by using the image of the original fused person, and the virtual reality fused object image emits the sound of the original fused object and shows an equal-ratio stereo image of the original fused object;
the behavior data acquisition module is used for acquiring multi-modal behavior data of the user in the intervention test process;
the behavior feature extraction module is used for extracting the behavior features of the face orientation, the eye spirit, the gesture, the position, the emotion and the language of the user in the experimental multi-modal behavior data collected by the behavior data collection module;
the training module is used for training the data of the behavior characteristics extracted by the behavior characteristic extraction module through a machine learning algorithm to obtain an evaluation model which accords with a scene;
the prediction module is used for analyzing the behavior characteristics of the user through the trained evaluation model obtained by the training module so as to obtain the user performance description;
the interactive module is used for carrying out information communication according to the user performance description obtained by the prediction module, and directly prompting to suggest or prompt the user to carry out correct behavior response by outputting sound information, image information and optical information in a scene;
and the course evaluation module is used for performing overall evaluation on the user learning according to the user course performance and the course learning progress in the whole course of the intervention test and providing related suggestions.
2. The autism assistance intervention system of claim 1, wherein the course customization module customizes the course by reference to a structured social behavioral intervention modality, behavioral therapy, key response training, verbal behavioral training, relationship development intervention therapy, early Danver modality, structured teaching, floor time therapy.
3. The autism assistance system according to claim 1, wherein the avatar acquisition module comprises:
the voice acquisition unit is used for acquiring the speaking voice data of a person and the voice of an animal or other articles in a course scene;
the image acquisition unit is used for acquiring multi-angle plane or depth image data of a user so as to obtain a two-dimensional or three-dimensional model image of the user.
4. The autism assistance intervention system of claim 1, wherein the real image blending module blends the image of the real life person or object into the course of the customized scene, so as to achieve the effect of blending the real person into the head, face and voice, and blending the real object into the image and voice in equal proportion, so that the blended person and object image is more stereoscopic, and specifically comprises:
the specific person voice synthesis unit is used for extracting the sound information of tone according to the provided audio data and synthesizing the character voice fused into the virtual reality scene by utilizing a multi-speaker voice synthesis technology;
an image synthesis unit for extracting image information of a head and a motion from the supplied character data and synthesizing a character moving image merged into the virtual reality scene; and extracting image information of the shape and the size according to the provided article data, and synthesizing the movable rotating article image blended into the virtual reality scene.
5. The autism assistance intervention system of claim 1, wherein the behavioral data collection module collects data using a virtual reality device with an internal eye tracker, a sensor, a gyroscope, a microphone, an intra-cavity camera, an extra-cavity camera, a handle, and gloves,
the method comprises the steps of obtaining orientation and movement data of a head and a body by using a sensor and a gyroscope, obtaining eye movement data by using a built-in eye tracker, obtaining movement data of muscles and skin around eyes by using an intracavity camera, obtaining voice data by using a microphone, obtaining muscle movement data of a chin and two cheeks by using an extracavity camera to shoot a lower half of a face, and obtaining hand movement data and course learning operation data by using a handle.
6. The autism assistance intervention system of claim 1, wherein the behavioral feature extraction module comprises:
a head direction extraction unit which calculates the movement of the head according to the moving freedom degree of the x, y and z rectangular coordinate axes of the object and the rotation freedom degree around the three coordinate axes by using a sensor or a gyroscope;
the facial expression extraction unit is used for combining the data of the left and right eyes of the user and the data of the mouth acquired by the VR helmet intracavity camera and the VR helmet extraluminal camera respectively and calculating a combined expression feature classification vector;
an eye focus extraction unit which uses the data of the eye movement tracking of the eye tracker to calculate the direction and focus of the eye movement;
the hand motion extraction unit is used for acquiring options made by a user for a scene question by using data acquired by the handle or the gloves, the sensor and the button, and calculating the motion mode and amplitude of the user in cooperation with the scene;
the voice feature extraction unit is used for obtaining the speaking data of the user by using the microphone, obtaining the speaking voice content of the user by voice recognition, extracting and analyzing keywords and comprehension paraphrases of the phrases, and obtaining the emotion scores of the interactive languages of the user by emotion recognition;
and the relative position extraction unit is used for obtaining the individual movement acceleration of the user by using the VR helmet sensor and calculating the movement track of the user.
7. The autism assistance intervention system of claim 1, wherein the behavior interaction module implements a strong interaction effect between the user and the device, takes the characteristic data of the user behavior as input, uses machine learning analysis in real time and performs hierarchical feedback output according to the behavior data of the user, and implements strong interaction by cycling input and output; wherein the feedback output includes a reinforced instructional output and a stylish encouragement output.
8. The autism assistance system according to claim 7, wherein the augmented instructional output comprises different levels of cueing or explicit reminders, the augmented instructional output being implemented by audio, visual or optical information.
9. The system of claim 8, wherein the course evaluation module performs an integrated calculation combining the data recorded in the user profile module and the performance of the current training of the user to provide suggestions for the training plan of the user, and the calculation method of the integrated result includes a method not limited to using condition judgment or machine learning.
10. An autism assistance intervention method based on virtual reality and multi-modal information, which is implemented by the autism assistance intervention system according to any one of claims 1 to 9, comprising the steps of:
s1: preparing informed information and establishing a file: a guardian or a nurse of a user reads an informed consent, and the awareness system acquires and analyzes behavior data and medical record related data of the user and an assisted intervention fused person, knows that the system does not leak user data to protect the privacy of the user, and establishes a file after confirming the consent;
s1: establishing a file: filling basic data of a user, including sex, age, development condition, family members and medical advice, by the user, the guardian or the caretaker according to the prompt and the description of the virtual reality equipment or the computer;
s2: establishing a teaching plan: generating a teaching plan according to the user file, and manually adjusting the user or a guardian or a caretaker of the user according to the actual situation to determine the teaching plan;
s3: image collection and integration: a guardian or an intervention auxiliary teacher of a user or a person living together with the guardian or the intervention auxiliary teacher for more than two weeks serves as a person to be integrated, and image data acquired by an RGBD camera or multi-angle RGB image data is provided; recording with a specified length or longer; integrating the voice and the image of the person to be integrated into the scene of the designed intervention course in the user teaching plan to obtain a customized course with the image and the voice image of the person to be integrated;
s4, auxiliary intervention test: the user performs strong interactive intervention test training according to a teaching training scene, and a system acquires multi-modal data of the user in an experiment;
s5: and (3) training and summarizing: and analyzing by the system to obtain the training score of the user according to all the training on the current day, judging whether the training on the current day is qualified, and recommending a subsequent training plan.
CN201911183662.6A 2019-11-27 2019-11-27 Autism auxiliary intervention system and method based on virtual reality and multi-mode information Pending CN110931111A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911183662.6A CN110931111A (en) 2019-11-27 2019-11-27 Autism auxiliary intervention system and method based on virtual reality and multi-mode information

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911183662.6A CN110931111A (en) 2019-11-27 2019-11-27 Autism auxiliary intervention system and method based on virtual reality and multi-mode information

Publications (1)

Publication Number Publication Date
CN110931111A true CN110931111A (en) 2020-03-27

Family

ID=69847430

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911183662.6A Pending CN110931111A (en) 2019-11-27 2019-11-27 Autism auxiliary intervention system and method based on virtual reality and multi-mode information

Country Status (1)

Country Link
CN (1) CN110931111A (en)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111461937A (en) * 2020-03-31 2020-07-28 北京复米教育科技有限公司 Automatic tracking method and system for autism intervention teaching course
CN111625098A (en) * 2020-06-01 2020-09-04 广州市大湾区虚拟现实研究院 Intelligent virtual avatar interaction method and device based on multi-channel information fusion
CN111640479A (en) * 2020-05-29 2020-09-08 京东方科技集团股份有限公司 Health management method for post-traumatic stress disorder patient and related device
CN111739612A (en) * 2020-06-28 2020-10-02 华中师范大学 Autism self-adaptive intervention system based on key reaction training mode
CN111986781A (en) * 2020-08-24 2020-11-24 龙马智芯（珠海横琴）科技有限公司 Psychological treatment method and device based on man-machine interaction and user terminal
CN112120716A (en) * 2020-09-02 2020-12-25 中国人民解放军军事科学院国防科技创新研究院 Wearable multi-mode emotional state monitoring device
CN112259218A (en) * 2020-09-29 2021-01-22 垒途智能教科技术研究院江苏有限公司 Training method for auditory stimulation of infantile autism based on VR interaction technology
CN113096805A (en) * 2021-04-12 2021-07-09 华中师范大学 Autism emotion cognition and intervention system
CN113270187A (en) * 2021-07-21 2021-08-17 奥罗科技(天津)有限公司 Intelligent remote autism rehabilitation training system
CN113434714A (en) * 2021-07-16 2021-09-24 李东霖 Auxiliary learning device and method
CN113593013A (en) * 2021-07-21 2021-11-02 吴浩诚 Interaction method, system, terminal and VR (virtual reality) equipment based on VR dead person simulation
CN114360691A (en) * 2021-12-08 2022-04-15 深圳大学 Training method of psychological toughness, terminal equipment and computer readable storage medium
CN114721520A (en) * 2022-04-07 2022-07-08 江苏中科小达人智能科技有限公司 Interactive training system and method based on virtual reality
CN114974517A (en) * 2022-08-01 2022-08-30 北京科技大学 Social anxiety intervention method and system based on simulation scene and interaction task design
US20230063681A1 (en) * 2021-08-25 2023-03-02 Sony Interactive Entertainment Inc. Dynamic augmentation of stimuli based on profile of user

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102354349A (en) * 2011-10-26 2012-02-15 华中师范大学 Human-machine interaction multi-mode early intervention system for improving social interaction capacity of autistic children
CN106354251A (en) * 2016-08-17 2017-01-25 深圳前海小橙网科技有限公司 Model system and method for fusion of virtual scene and real scene
CN106792246A (en) * 2016-12-09 2017-05-31 福建星网视易信息系统有限公司 A kind of interactive method and system of fusion type virtual scene
CN109620185A (en) * 2019-01-31 2019-04-16 山东大学 Self-closing disease assistant diagnosis system, equipment and medium based on multi-modal information
US10311645B1 (en) * 2016-10-14 2019-06-04 Floreo, Inc. Methods and systems for treating autism
CN109919712A (en) * 2019-01-30 2019-06-21 上海市精神卫生中心（上海市心理咨询培训中心） Neurodevelopmental disorder shopping training system and its training method

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102354349A (en) * 2011-10-26 2012-02-15 华中师范大学 Human-machine interaction multi-mode early intervention system for improving social interaction capacity of autistic children
CN106354251A (en) * 2016-08-17 2017-01-25 深圳前海小橙网科技有限公司 Model system and method for fusion of virtual scene and real scene
US10311645B1 (en) * 2016-10-14 2019-06-04 Floreo, Inc. Methods and systems for treating autism
CN106792246A (en) * 2016-12-09 2017-05-31 福建星网视易信息系统有限公司 A kind of interactive method and system of fusion type virtual scene
CN109919712A (en) * 2019-01-30 2019-06-21 上海市精神卫生中心（上海市心理咨询培训中心） Neurodevelopmental disorder shopping training system and its training method
CN109620185A (en) * 2019-01-31 2019-04-16 山东大学 Self-closing disease assistant diagnosis system, equipment and medium based on multi-modal information

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
戚静瑜: "《基于WorldViz平台的儿童社会能力干预训练的虚拟现实技术研究》", 《中国优秀博硕士学位论文全文数据库(硕士)医药卫生科技辑》 *

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111461937A (en) * 2020-03-31 2020-07-28 北京复米教育科技有限公司 Automatic tracking method and system for autism intervention teaching course
CN111640479A (en) * 2020-05-29 2020-09-08 京东方科技集团股份有限公司 Health management method for post-traumatic stress disorder patient and related device
CN111625098A (en) * 2020-06-01 2020-09-04 广州市大湾区虚拟现实研究院 Intelligent virtual avatar interaction method and device based on multi-channel information fusion
CN111739612A (en) * 2020-06-28 2020-10-02 华中师范大学 Autism self-adaptive intervention system based on key reaction training mode
CN111986781B (en) * 2020-08-24 2021-08-06 龙马智芯（珠海横琴）科技有限公司 Psychological treatment device and user terminal based on human-computer interaction
CN111986781A (en) * 2020-08-24 2020-11-24 龙马智芯（珠海横琴）科技有限公司 Psychological treatment method and device based on man-machine interaction and user terminal
CN112120716A (en) * 2020-09-02 2020-12-25 中国人民解放军军事科学院国防科技创新研究院 Wearable multi-mode emotional state monitoring device
CN112259218A (en) * 2020-09-29 2021-01-22 垒途智能教科技术研究院江苏有限公司 Training method for auditory stimulation of infantile autism based on VR interaction technology
WO2022067871A1 (en) * 2020-09-29 2022-04-07 垒途智能教科技术研究院江苏有限公司 Vr-interaction-technology-based auditory stimulation training method for children with autism
CN113096805B (en) * 2021-04-12 2024-02-13 华中师范大学 Autism emotion cognition and intervention system
CN113096805A (en) * 2021-04-12 2021-07-09 华中师范大学 Autism emotion cognition and intervention system
CN113434714B (en) * 2021-07-16 2024-04-05 李东霖 Auxiliary learning device and method
CN113434714A (en) * 2021-07-16 2021-09-24 李东霖 Auxiliary learning device and method
CN113593013A (en) * 2021-07-21 2021-11-02 吴浩诚 Interaction method, system, terminal and VR (virtual reality) equipment based on VR dead person simulation
CN113270187A (en) * 2021-07-21 2021-08-17 奥罗科技(天津)有限公司 Intelligent remote autism rehabilitation training system
US20230063681A1 (en) * 2021-08-25 2023-03-02 Sony Interactive Entertainment Inc. Dynamic augmentation of stimuli based on profile of user
CN114360691A (en) * 2021-12-08 2022-04-15 深圳大学 Training method of psychological toughness, terminal equipment and computer readable storage medium
CN114721520A (en) * 2022-04-07 2022-07-08 江苏中科小达人智能科技有限公司 Interactive training system and method based on virtual reality
CN114974517A (en) * 2022-08-01 2022-08-30 北京科技大学 Social anxiety intervention method and system based on simulation scene and interaction task design

Similar Documents

Publication Publication Date Title
CN110931111A (en) Autism auxiliary intervention system and method based on virtual reality and multi-mode information
Tapus et al. The use of socially assistive robots in the design of intelligent cognitive therapies for people with dementia
Chevalier et al. Impact of sensory preferences of individuals with autism on the recognition of emotions expressed by two robots, an avatar, and a human
Picard A ective Computing
JP2020507835A (en) Robot, server and human-machine interaction method
CN111081371A (en) Virtual reality-based early autism screening and evaluating system and method
US8714982B2 (en) System and method for teaching social skills, social thinking, and social awareness
CN110890140A (en) Virtual reality-based autism rehabilitation training and capability assessment system and method
D'errico Powers of Divergence: an experimental approach to music performance
US20120021390A1 (en) Method and system for developing language and speech
CN108615439A (en) Method, apparatus, equipment and medium for formulating ability training scheme
Asada Modeling early vocal development through infant–caregiver interaction: A review
Wik The Virtual Language Teacher: Models and applications for language learning using embodied conversational agents
Liu et al. An interactive speech training system with virtual reality articulation for Mandarin-speaking hearing impaired children
CN110812803B (en) Interactive auxiliary intervention system for cognitive improvement
Rathinavelu et al. Three dimensional articulator model for speech acquisition by children with hearing loss
Hamidi Digital tangible games for speech intervention
Moustakas et al. Using modality replacement to facilitate communication between visually and hearing-impaired people
Al-Qbilat Accessibility requirements for human-robot interaction for socially assistive robots
Bondioli et al. A survey on technological tools and systems for diagnosis and therapy of autism spectrum disorder
Zhian Tracking Visible Features of Speech for Computer-Based Speech Therapy for Childhood Apraxia of Speech
TWI824883B (en) A virtual reality interactive system that uses virtual reality to simulate expressions and emotions for training
CN112634684B (en) Intelligent teaching method and device
CN111831102A (en) Expansion auxiliary system based on infant brain superiority assessment
Pérez-Espinosa et al. Analyzing children’s affective reactions and preferences towards social robots using paralinguistic and self-reported information

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20200327

RJ01 Rejection of invention patent application after publication