JP2020003624A - System for optimizing and virtualizing video and voice for virtual reality lesson room for remote instruction - Google Patents

Abstract

To solve problems for example, in videos and voices transmitted and received between remote sites in the ICT instruction system, if speech productions are performed simultaneously, speech production of a participant personal cannot be selected and listened, and in exchange between a learner and an instructor, it is impossible to identify an individual on the basis of a position of a speech production person which a listener can identify in a normal state, and delay of transmission occurs always, especially, communication between an instructor and a learner is not smoothly performed.SOLUTION: The invention is configured so that, it is assumed that, by transmission of voices and videos by ICT equipment and the Internet, it is impossible to completely dissolve delay, and even there is delay of transmission of voices and videos, a function for recording each learner is achieved as a corresponding function, a delay state of videos and voices between the instructor and learner can be visualized, and a virtual lesson room which is approximated to an image in which learners who present at respective local areas are arranged in seats in one class room, can be achieved by control of voice delay.SELECTED DRAWING: Figure 1

Description

  The present invention makes it possible to save and reproduce individual recordings and recordings of teachers and learners in ICT distance learning, and to confirm them individually. In addition, the three-dimensional positional relationship between the lecturer and the learner in the school is virtually realized, so that both the teacher and the learner can be drawn into the realism of the school. Further, with regard to the inevitable communication delay, it is possible to identify whether the communication delay is caused by the response delay of the person or the communication delay by the visualization.

  The video and audio of ICT education is used through remote areas and Internet lines, and is widely used, especially when it comes to providing language education to learners in areas where there is a shortage of lecturers or overseas learners. However, the learner's voice via the Internet line is a mixture of many voices, and it was not possible to selectively hear whether the pronunciation of a specific speaker was proper or not. In addition, since the learner does not hear from the position where the learner sits as in a normal school, the instructor relies on the learner's position and direction in addition to the sound quality and speech habits of the learner. I cannot identify the Lord. In fact, instructors tended to be incomplete and inadequate in practical instruction, as they misidentified or lost judgment of the main voice. In addition, video and audio are delayed due to technical problems with the Internet line and ICT equipment and camera system, and it is determined whether the learner is reacting to the instructor's pronunciation without hesitation or whether there is no hesitation or anxiety Because of the difficulty in doing so, there were significant constraints on educational effectiveness.

  Selectively listen to the utterance of a specific individual from the crowded voice of many speakers, realize a virtual reality classroom that does not exist in existing systems with 3D sound, improve the line speed of the Internet, ICT equipment Although the performance improvement of ICT equipment is progressing, it is difficult for all learners to prepare the highest performance ICT equipment even if the performance improvement of ICT equipment of the highest performance is progressing, and it is difficult to provide Internet environment and ICT It was necessary to solve problems that did not rely only on improving the delay caused by equipment.

In a class using ICT equipment, which is currently widely used, when many learners (including instructors) speak at once, the voice is mixed, and the voice of a specific individual can be selectively heard after the fact. There were no flaws. In addition, when multiple learners speak simultaneously, the instructor cannot instantaneously identify the main voice based on the direction of the learner, so the instructor must rely on the voice quality of the learner and the habit of answering. However, the instructor was not able to respond accurately and quickly as in a normal school, which hindered the instruction by the instructor.
In addition, the lesson style using ICT equipment, which is currently widely used, is such that one instructor can shoot video with ICT equipment and deliver audio as monaural or stereo sound source, and multiple learners can buy It is common to answer with smartphones, tablets, computer camcorders and monaural microphones. The line is a general Internet line, and many overseas have low quality lines. The learner's ICT equipment is a general commercial product. Both video and audio generated a delay of 0.2 seconds to a maximum of 1 second or more, which hindered communication between the instructor and the learner.

  Therefore, as shown in FIG. 1, a method was devised and invented in which a video and a sound of a learner in a class are separately acquired and recorded and simultaneously recorded and simultaneously processed and appropriately reproduced in a communication destination. As a result, it is possible to specify individual learners of up to about 30 students, and it is possible to individually confirm the learner's response state to the instructor. Next, FIG. 2 invents a function capable of realizing a three-dimensional sound effect as if the positional relationship between the instructor and the learner's utterance was virtually arranged in a classroom. This acoustic effect allows the instructor to imagine the positional relationship with respect to the utterance from the learner, and can approach the state of giving a lecture in a classroom. Furthermore, a technique for relatively visualizing the delay between video and audio during a lesson as shown in FIG. 3 was invented. The instructor speaks, the individual learners react, and the estimated time to reach the instructor is visualized, so that it is possible to grasp whether the learner is responding appropriately to the instructor.

JP 2015-080255 A JP 2012-239184 A JP-T-2009-518996

  The problems that this system is trying to solve are that in video conferencing used in ICT education, the utterances of the participating lecturers and learners cannot be separated as mixed voices, and the mutual positional relationship such as the classroom is lost. And the problem of communication delays. However, the delay itself does not become zero, and a delay of about several tens of milliseconds to one second occurs.

  The present invention obtains the video and audio of the instructor and the learner individually, and records and records each of them, and adds processing such as delay and reverberation to the sound to determine the positional relationship between the instructor and the learner. By approaching the image and visualizing the delay of the voice reaching the instructor and the learner, the mechanism that solves the problems inherent in ICT distance education and the communication between the instructor and the learner becomes closer And

  The virtual reality classroom for ICT distance education of the present invention acquires video and audio for each learner, realizes virtual reality stereoscopic sound effect, realizes a virtual state of lecture in the classroom, voice and video through the Internet By analyzing the delay state of the learner and visualizing the state, communication with the learner who has a clear timing for properly recognizing the utterance of the learner is tighter, and the video and audio can be reproduced as necessary And make the instructor's judgment appropriate.

FIG. 1 is an explanatory diagram for recording video and audio for each learner. (Example 1) FIG. 2 is an explanatory diagram for virtualizing the positional relationship between the instructor and the learner. (Example 2) FIG. 3 is an explanatory diagram for visualizing the delay between audio and video. (Example 3)

  The optimization and virtualization of the video and audio of the ICT education system was made possible by various programming technologies in the virtual environment of the Internet.

  FIG. 1 shows an embodiment of the present invention 1001 is a computer on the instructor side, 1002 is a WEB camera with a built-in microphone, 1003 is a function for recording video and audio in an Internet virtual environment for each learner, 1004 is a built-in camera and microphone Learners' smartphones, tablet PCs, and personal computers.

  The video and audio recording function of the Internet virtual environment of 1003 allows the administrator of the training side to record video and audio arbitrarily by specifying the student.

  Recording and recording can record the instructor and the learner individually, and it is also possible to record all the participants in the course at the same time. Reproduction can be performed individually and a plurality of recordings can be reproduced simultaneously.

FIG. 2 shows an invention for reproducing a relationship between a learner and a lecturer scattered in a remote area 2001 as a virtual classroom. 2003
In the virtual classroom, the learners are placed at the seats in the order in which the learners log in from remote locations.

  Instructors will be able to visually check on their own PC which seats each learner has been placed in.

  Next, the sound effect that 2003 learners placed in the seats are speaking at the position of each seat is produced.

This mechanism enables the delay of the sound reaching the lecturer to be controlled by the left and right speakers of the lecturer in 2004. From the above, a virtual classroom in which a learner is placed in a designated seat in the classroom is realized assuming a classroom of up to about 30 people.

  FIG. 3 is a screen image of the instructor who performs the ITC course, and the learner's voice reaches the learner, and the learner responds to it so that the learner's utterance can be visualized.

  The program analyzes the current delay status through the Internet, and at the same time analyzes the delay of the ICT equipment of the instructor and the delay of the learner, and shows the delay to the learner and the delay from the learner on the instructor screen by graph . This display is arbitrarily performed by the instructor as needed, and is used as a reference for judging the response of the learner.

  In FIG. 3, a sound wave is first transmitted from the instructor's 3002PC, and the transmission time is recorded up to the microsecond. The 3003 terminal on the student side receives the sound wave, reflects the 3004 received sound wave, and measures the time when the reflected sound wave reaches the instructor side to the microsecond in the same manner. At 3006, the difference time between two points on the instructor side and the student side is determined up to microsecond. The 3002 instructor's PC visualizes the 3007 difference time between the two points with video and displays it so that the user can feel the sensation delay time.

  In particular, realization of virtual classrooms is expected not only for ICT learning but also for WEB conferences frequently conducted by companies and organizations, if the image of the utterance position of the participants is imaged, the conference will be smooth and widely spread Is done.

1001 Instructor's computer 1002 Web camera 1003 Individual recording and recording in the Internet environment 1004 Student's terminal 2001 Local learner's terminal 2002 System on the Internet 2003 Virtualization classroom with learners and voice delay 2004 Instructor's computer 3001 Program start 3002 Instructor computer 3003 Learner terminal 3004 Sound wave reflection by learner terminal 3005 Measurement of sound wave arrival time 3006 Sound wave transmission / reception time comparison 3007 Visualization of sound wave delay time

Claims (3)

  A function to record and play back audio and video separately on the Internet server and each learner's terminal for each learner in the ICT learning system using the Internet.   In the Internet-based ICT learning system, the sound wave sent from the instructor arrives for each learner and the actual time the sound wave returns from the learner is measured, and the time is visualized so that the instructor can recognize the delay visually. .   In an ICT learning system using the Internet, learners scattered in distant places are assigned to seats in a virtual classroom at the time of logging in, and stereo voices are sent so that each voice reaches the lecturer from the assigned seat position. A function to adjust the image and realize a two-dimensional virtual classroom.

Images