CN109101107A - A kind of system and method that VR virtual classroom trains virtual robot - Google Patents
A kind of system and method that VR virtual classroom trains virtual robot Download PDFInfo
- Publication number
- CN109101107A CN109101107A CN201810715294.4A CN201810715294A CN109101107A CN 109101107 A CN109101107 A CN 109101107A CN 201810715294 A CN201810715294 A CN 201810715294A CN 109101107 A CN109101107 A CN 109101107A
- Authority
- CN
- China
- Prior art keywords
- virtual
- voice
- action
- virtual robot
- trainer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims description 22
- 230000009471 action Effects 0.000 claims abstract description 109
- 238000012549 training Methods 0.000 claims abstract description 68
- 230000033001 locomotion Effects 0.000 claims abstract description 18
- 230000000875 corresponding effect Effects 0.000 claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 16
- 230000006399 behavior Effects 0.000 claims description 19
- 230000008569 process Effects 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 abstract 1
- 238000004321 preservation Methods 0.000 abstract 1
- 230000003993 interaction Effects 0.000 description 8
- 239000010410 layer Substances 0.000 description 6
- 238000010801 machine learning Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 239000002346 layers by function Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Electrically Operated Instructional Devices (AREA)
Abstract
The present invention provides a kind of system that VR virtual classroom trains virtual robot, including virtual application server, front end training device and virtual robot；Front end training device acquires trainer for voice and movement caused by each predeterminable event；Virtual application server obtain front end training device voice under collected each predeterminable event and movement and handled, and action command and output needed for the processing result of each predeterminable event is converted into virtual robot by preset algorithm model；Virtual robot receives the action command of virtual application server conversion output, and action command based on the received, forms corresponding action behavior and records preservation.Implement the present invention, by virtual classroom training come training virtual robot, makes the speech performance of virtual robot that there is independence.
Description
Technical Field
The invention relates to the technical field of virtual reality, in particular to a system and a method for training a virtual robot in a VR virtual classroom.
Background
With the maturity and popularization of the virtual reality technology, the technology is used in a plurality of fields at home and abroad, and is developed vigorously in recent years in aspects of scientific experiments, training and game entertainment.
However, these applications or systems are not flexible enough in the aspect of interaction core, and especially lack a more flexible and intelligent interaction mechanism cast through VR virtual classroom training, and cannot utilize the currently mature machine learning algorithm, so that the speech behavior of the virtual robot has autonomy, and the purpose of targeted training of users can be achieved.
Disclosure of Invention
The technical problem to be solved by the embodiments of the present invention is to provide a system and a method for training a virtual robot in a VR virtual classroom, which can train the virtual robot through virtual classroom training, so that the speech behavior of the virtual robot has autonomy, and the purpose of training a user in a targeted manner can be achieved.
In order to solve the above technical problems, an embodiment of the present invention provides a system for training a virtual robot in a VR virtual classroom, including a virtual application server, and a front end training device and a virtual robot both connected to the virtual application server; wherein,
the front-end training device is used for collecting the voice and the action of a trainer aiming at each preset event;
the virtual application server is used for acquiring and processing the voice and the action under each preset event acquired by the front-end training device, and converting the processing result of each preset event into an action instruction required by the virtual robot through a preset algorithm model and outputting the action instruction;
and the virtual robot is used for receiving the action instruction converted and output by the virtual application server, forming corresponding action behaviors according to the received action instruction, and recording and storing the corresponding action behaviors.
The front-end training device comprises a front-end collector, and a motion capture instrument and a voice array recognition instrument which are connected with the front-end collector; wherein,
the motion capture instrument is used for capturing various motions of the trainer; wherein the actions include body actions and gesture actions;
the voice array recognizer is used for recognizing the voice of a trainer;
the front-end collector is used for receiving various actions of the trainee captured by the action capture instrument and the voice of the trainee recognized by the voice array recognition instrument, establishing association between the received various actions and the received voice of the trainee and each preset event, and transmitting the association to the virtual application server.
The embodiment of the invention also provides a method for training the virtual robot in the VR virtual classroom, which comprises the following steps:
the front-end training device collects the voice and the action of a trainer aiming at each preset event;
the virtual application server acquires and processes the voice and the action under each preset event acquired by the front-end training device, and converts the processing result of each preset event into an action instruction required by the virtual robot through a preset algorithm model and outputs the action instruction;
and the virtual robot receives the action instruction converted and output by the virtual application server, forms corresponding action behaviors according to the received action instruction, and records and stores the corresponding action behaviors.
The specific steps of the front-end training device for collecting the voice and the action generated by a trainer aiming at each preset event comprise:
capturing various actions of a trainer by using the action capturing instrument and recognizing the voice of the trainer by using the voice array recognition instrument; wherein the actions include body actions and gesture actions;
the front-end collector receives various actions of the trainee captured by the action capture instrument and the voice of the trainee recognized by the voice array recognition instrument, and establishes association between the received various actions and voice of the trainee and each preset event and transmits the association to the virtual application server.
Wherein the method further comprises:
the front-end collector verifies the relevant information of the trainer; wherein, the information comprises the identity and the password of the trainer.
Wherein the method further comprises:
the virtual application server builds a scene by using a Unity engine and OpenGL in advance, and defines an algorithm model which is converted into action instructions required by the virtual robot in advance by using a machine algorithm.
The embodiment of the invention has the following beneficial effects:
the invention collects the voice and the action generated by a trainer aiming at each preset event through the front-end training device, converts the voice and the action into the action instruction required by the virtual robot through the virtual application server, leads the virtual robot to form corresponding action after receiving the action and record and store the action, and trains subsequent users through the virtual robot recording and storing the action, thereby training the virtual robot through virtual classroom training, leading the speech action of the virtual robot to have autonomy and achieving the aim of pertinently training the users.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is within the scope of the present invention for those skilled in the art to obtain other drawings based on the drawings without inventive exercise.
Fig. 1 is a schematic structural diagram of a system for training a virtual robot in a VR virtual classroom according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a system structure of the front-end collector in FIG. 1;
fig. 3 is a flowchart of a method for training a virtual robot in a VR virtual classroom according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings.
As shown in fig. 1 and fig. 2, in an embodiment of the present invention, a VR virtual classroom virtual robot training system includes a virtual application server 1, and a front end training device 2 and a virtual robot 3 both connected to the virtual application server 1; wherein,
the front-end training device 2 is used for collecting the voice and the action of a trainer aiming at each preset event;
the virtual application server 1 is used for acquiring and processing the voice and the action under each preset event acquired by the front-end training device 2, and converting the processing result of each preset event into an action instruction required by the virtual robot 3 through a preset algorithm model and outputting the action instruction;
the virtual robot 3 is used for receiving the action command converted and output by the virtual application server 1, and forming corresponding action behaviors according to the received action command, recording and storing the action behaviors, so that a subsequent user can train through the action behaviors recorded and stored in the virtual robot 3.
It should be noted that the virtual application server 1 builds a scene by using the Unity engine and OpenGL in advance, and defines an algorithm model that is converted into an action instruction required by the virtual robot 3 in advance by using a machine algorithm.
It should be noted that the virtual application server 1 has preset virtual classroom training software, and the virtual classroom training software includes:
the platform layer adopts a Windows operating system as an operating platform of the VR virtual classroom training system, and builds, programs and renders scenes by using a Unity engine and OpenGL;
and rendering the layer, performing visual processing on the built scene to enable eyes to feel more real, and processing the image by using the inverse distortion. Utilizing ATW to eliminate image jitter; the scene tends to be real by light field imaging, and the scene container dynamically loads the scene and the like;
the data layer is used for acquiring various data in the system, labeling the data and analyzing the data, and is mainly used for evaluation;
and the algorithm layer is divided into a semantic event analysis part and a machine learning part, wherein the semantic event analysis part is mainly used for carrying out semantic matching on the voice of a trainer and triggering an external definition event. The machine learning determines which events which are not good enough and are not good enough to be processed and processed by the trainer when the trainer trains next time according to the processing results of the trainer on the system events and the internal definition events, and performs intensive training on the trainer. Meanwhile, the virtual robot 3 can be trained, so that the virtual robot 3 has the capability of asking questions independently, the questions are related to and meaningful in classroom content, and the virtual robot 3 is more intelligent;
the perception layer is used for recognizing the voice of the trainer and processing the interaction of the virtual robot 3;
and the application support layer is divided into an interactive standard component and a system operation component. The interaction standard component is used for supporting general interaction of trainers and mainly consists of an HTC five suite. The system operation component is used for supporting the normal operation of the training system and mainly comprises a PC (personal computer);
the functional layer, the trainer can directly use the various interactions and other functions of the training system. These functions include global menus, voice interaction systems, rating systems, classroom settings, file serving, identification, motion capture and surveillance systems.
In the embodiment of the present invention, the front end training device 2 includes a front end collector 23, and a motion capture instrument 21 and a voice array recognition instrument 22 both connected to the front end collector 23; wherein,
a motion capture instrument 21 for capturing various motions of the trainer; wherein the motion comprises a body motion and a gesture motion;
a voice array recognizer 22 for recognizing the voice of the trainer;
the front-end collector 23 is configured to receive various types of actions of the trainee captured by the action capture instrument 21 and the voice of the trainee recognized by the voice array recognition instrument 22, associate the received various types of actions and voice of the trainee with each preset event, and transmit the association to the virtual application server 1.
Further description is made on an application scenario of the virtual robot training system in the VR virtual classroom in the embodiment of the present invention, which is specifically as follows:
training is started, and after the trainer is ready, the trainer selects to start training in the global menu. The motion capture device 21 in the front-end training apparatus 2 captures and stores the motion behavior information of the trainee. The voice array recognizer 22 in the front-end training device 2 captures the voice of a trainer, the trainer asks the virtual robot 3 through virtual classroom training software preset in the virtual application server 1, and when the interaction of the language is needed in various events, the action and the voice corresponding to each preset event are determined according to the front-end training device 2;
the virtual application server 1 records the method and the result of the trainer for processing various events in the preset virtual classroom training software and converts the method and the result into the corresponding action instruction of the virtual robot 3; the virtual robot 3 will make corresponding actions according to the instruction of the trainer. Of course, the virtual application server 1 will also control the virtual robot 3 to make relevant actions to wait for the handler to handle.
In the whole training process, by monitoring the synchronous playing of the virtual application server 1 on the large screen, teachers and other audiences can watch the training at the visual angle of the trainer.
As shown in fig. 3, in an embodiment of the present invention, a method for a VR virtual classroom training system for a virtual robot is provided, including the following steps:
step S1, the front end training device collects the voice and the action generated by the trainer aiming at each preset event;
the front-end training device comprises a front-end collector, a motion capture instrument and a voice array recognition instrument. Capturing various actions of a trainer by using an action capturing instrument and recognizing the voice of the trainer by using a voice array recognition instrument; wherein the actions include body actions and gesture actions;
and continuously receiving various actions of the trainer captured by the action capture instrument and the voice of the trainer recognized by the voice array recognition instrument through the front-end collector, establishing association between the received various actions and the received voice of the trainer and each preset event, and transmitting the association to the virtual application server.
Certainly, the front-end collector can also verify the relevant information of the trainer; wherein, the information comprises the identity, the password and the like of the trainer.
Step S2, the virtual application server obtains and processes the voice and the action under each preset event collected by the front-end training device, and converts the processing result of each preset event into an action instruction required by the virtual robot through a preset algorithm model and outputs the action instruction;
the specific process is that the virtual application server builds a scene by using a Unity engine and OpenGL in advance, and defines an algorithm model which is converted into action instructions required by the virtual robot in advance by using a machine algorithm.
And once the virtual application server acquires the voice and the action under each preset event collected by the front-end training device, the voice and the action are processed in time, and the processing result of each preset event is converted into an action instruction required by the virtual robot through a preset algorithm model and is output.
And step S3, the virtual robot receives the action command converted and output by the virtual application server, and forms corresponding action behavior according to the received action command, and records and stores the action behavior.
The virtual robot forms corresponding action behaviors according to the action instructions converted by the virtual application server, records and stores the action behaviors, so that a subsequent user can train through the action behaviors recorded and stored in the virtual robot.
The embodiment of the invention has the following beneficial effects:
the invention collects the voice and the action generated by a trainer aiming at each preset event through the front-end training device, converts the voice and the action into the action instruction required by the virtual robot through the virtual application server, leads the virtual robot to form corresponding action after receiving the action and record and store the action, and trains subsequent users through the virtual robot recording and storing the action, thereby training the virtual robot through virtual classroom training, leading the speech action of the virtual robot to have autonomy and achieving the aim of pertinently training the users.
While the invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not to be limited to the disclosed embodiment, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (6)
1. A system for training a virtual robot in a VR virtual classroom is characterized by comprising a virtual application server, a front-end training device and a virtual robot, wherein the front-end training device and the virtual robot are connected with the virtual application server; wherein,
the front-end training device is used for collecting the voice and the action of a trainer aiming at each preset event;
the virtual application server is used for acquiring and processing the voice and the action under each preset event acquired by the front-end training device, and converting the processing result of each preset event into an action instruction required by the virtual robot through a preset algorithm model and outputting the action instruction;
and the virtual robot is used for receiving the action instruction converted and output by the virtual application server, forming corresponding action behaviors according to the received action instruction, and recording and storing the corresponding action behaviors.
2. The VR virtual classroom virtual robot training system of claim 1, wherein the front end training device includes a front end collector and a motion capture instrument and a voice array recognizer coupled to the front end collector; wherein,
the motion capture instrument is used for capturing various motions of the trainer; wherein the actions include body actions and gesture actions;
the voice array recognizer is used for recognizing the voice of a trainer;
the front-end collector is used for receiving various actions of the trainee captured by the action capture instrument and the voice of the trainee recognized by the voice array recognition instrument, establishing association between the received various actions and the received voice of the trainee and each preset event, and transmitting the association to the virtual application server.
3. A method for training a virtual robot in a VR virtual classroom is characterized by comprising the following steps:
the front-end training device collects the voice and the action of a trainer aiming at each preset event;
the virtual application server acquires and processes the voice and the action under each preset event acquired by the front-end training device, and converts the processing result of each preset event into an action instruction required by the virtual robot through a preset algorithm model and outputs the action instruction;
and the virtual robot receives the action instruction converted and output by the virtual application server, forms corresponding action behaviors according to the received action instruction, and records and stores the corresponding action behaviors.
4. The method for VR virtual classroom training a virtual robot as claimed in claim 3, wherein the step of the front end training device collecting the trainer's voice and actions for each predetermined event includes:
capturing various actions of a trainer by using the action capturing instrument and recognizing the voice of the trainer by using the voice array recognition instrument; wherein the actions include body actions and gesture actions;
the front-end collector receives various actions of the trainee captured by the action capture instrument and the voice of the trainee recognized by the voice array recognition instrument, and establishes association between the received various actions and voice of the trainee and each preset event and transmits the association to the virtual application server.
5. The method of VR virtual classroom training a virtual robot as claimed in claim 4, the method further comprising:
the front-end collector verifies the relevant information of the trainer; wherein, the information comprises the identity and the password of the trainer.
6. The method of VR virtual classroom training a virtual robot as claimed in claim 3, the method further comprising:
the virtual application server builds a scene by using a Unity engine and OpenGL in advance, and defines an algorithm model which is converted into action instructions required by the virtual robot in advance by using a machine algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810715294.4A CN109101107A (en) | 2018-06-29 | 2018-06-29 | A kind of system and method that VR virtual classroom trains virtual robot |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810715294.4A CN109101107A (en) | 2018-06-29 | 2018-06-29 | A kind of system and method that VR virtual classroom trains virtual robot |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109101107A true CN109101107A (en) | 2018-12-28 |
Family
ID=64845449
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810715294.4A Pending CN109101107A (en) | 2018-06-29 | 2018-06-29 | A kind of system and method that VR virtual classroom trains virtual robot |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109101107A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103488291A (en) * | 2013-09-09 | 2014-01-01 | 北京诺亦腾科技有限公司 | Immersion virtual reality system based on motion capture |
| CN105355200A (en) * | 2015-11-20 | 2016-02-24 | 深圳狗尾草智能科技有限公司 | System and method for training and modifying interactive content of robot directly |
| CN106313113A (en) * | 2015-06-30 | 2017-01-11 | 芋头科技（杭州）有限公司 | System and method for training robot |
| CN106826833A (en) * | 2017-03-01 | 2017-06-13 | 西南科技大学 | Independent navigation robot system based on 3D solid cognition technologies |
| CN107248342A (en) * | 2017-07-07 | 2017-10-13 | 四川云图瑞科技有限公司 | Three-dimensional interactive tutoring system based on virtual reality technology |
| CN107765859A (en) * | 2017-11-09 | 2018-03-06 | 温州大学 | A kind of training system and method based on VR virtual classrooms |
| CN107958433A (en) * | 2017-12-11 | 2018-04-24 | 吉林大学 | A kind of online education man-machine interaction method and system based on artificial intelligence |
-
2018
- 2018-06-29 CN CN201810715294.4A patent/CN109101107A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103488291A (en) * | 2013-09-09 | 2014-01-01 | 北京诺亦腾科技有限公司 | Immersion virtual reality system based on motion capture |
| CN106313113A (en) * | 2015-06-30 | 2017-01-11 | 芋头科技（杭州）有限公司 | System and method for training robot |
| CN105355200A (en) * | 2015-11-20 | 2016-02-24 | 深圳狗尾草智能科技有限公司 | System and method for training and modifying interactive content of robot directly |
| CN106826833A (en) * | 2017-03-01 | 2017-06-13 | 西南科技大学 | Independent navigation robot system based on 3D solid cognition technologies |
| CN107248342A (en) * | 2017-07-07 | 2017-10-13 | 四川云图瑞科技有限公司 | Three-dimensional interactive tutoring system based on virtual reality technology |
| CN107765859A (en) * | 2017-11-09 | 2018-03-06 | 温州大学 | A kind of training system and method based on VR virtual classrooms |
| CN107958433A (en) * | 2017-12-11 | 2018-04-24 | 吉林大学 | A kind of online education man-machine interaction method and system based on artificial intelligence |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230362457A1 (en) | Intelligent commentary generation and playing methods, apparatuses, and devices, and computer storage medium | |
| CN107203953A (en) | It is a kind of based on internet, Expression Recognition and the tutoring system of speech recognition and its implementation | |
| CN109036374B (en) | Data processing method and device | |
| CN109101879B (en) | Posture interaction system for VR virtual classroom teaching and implementation method | |
| CN112614399A (en) | Dance teaching equipment based on virtual reality and teaching method thereof | |
| CN102947774A (en) | Natural user input for driving interactive stories | |
| CN107066081B (en) | Interactive control method and device of virtual reality system and virtual reality equipment | |
| CN109754653B (en) | Method and system for personalized teaching | |
| CN110490173B (en) | Intelligent action scoring system based on 3D somatosensory model | |
| CN103240746B (en) | A kind of finger-guessing game robot and finger-guessing game gesture identification method with image identification system | |
| CA3048542A1 (en) | System for peer-to-peer, self-directed or consensus human motion capture, motion characterization, and software-augmented motion evaluation | |
| CN104598027B (en) | A kind of motion sensing control multi-media Training System based on user behavior analysis | |
| CN110767005A (en) | Data processing method and system based on intelligent equipment special for children | |
| CN107452381B (en) | Multimedia voice recognition device and method | |
| CN113556599A (en) | Video teaching method and device, television and storage medium | |
| CN108877371A (en) | A kind of VR teaching in VR classroom training system and implementation method | |
| CN116880699A (en) | Multi-person immersive teaching interaction system | |
| CN109101107A (en) | A kind of system and method that VR virtual classroom trains virtual robot | |
| CN111435268A (en) | Human-computer interaction method based on image recognition and reconstruction and system and device using same | |
| CN113362806A (en) | Intelligent sound evaluation method, system, storage medium and computer equipment thereof | |
| CN110808075A (en) | Intelligent recording and broadcasting method | |
| CN110221694A (en) | Control method and device of interactive platform, storage medium and interactive platform | |
| CN114220160A (en) | Anti-cheating remote examination equipment based on VR technology and use method thereof | |
| CN110413130B (en) | Virtual reality sign language learning, testing and evaluating method based on motion capture | |
| CN112086091A (en) | Intelligent endowment service system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181228 |
|
| RJ01 | Rejection of invention patent application after publication |