JP2005055846A - Remote educational communication system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a remote educational communication system allowing students to have feelings of participation in education and learning. <P>SOLUTION: Inherent other self object image information 11 stored in the respective arithmetic operation means of a plurality of terminals 4, 5 set in an operating state is displayed on respective display means 6, etc., of the terminals 5, etc., set in the operating state. When each operator of the terminals 5, etc., set in the operating state inputs information from an external input means of the terminal, the inputted information is displayed on the inherent other self object image information 11, etc., displayed on the respective display means 6 of the terminals 5, etc., set in the operating state. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a remote communication education system that uses a combination of a communication network such as the Internet and a computer to connect a remote student and an instructor and perform distance education and distance learning in real time and in both directions. In particular, the present invention relates to a distance education communication system that allows students to feel that they are participating in education and learning.

Recently, distance learning using a video conference system has been known as a method for performing remote education and distance learning in real time and in both directions using a combination of a communication network such as the Internet and a computer. In such a lecture, the face and figure of the instructor and students are imaged by the camera, and the images of the other instructor and student are communicated by connecting the remote locations with two-way communication and transmitting the camera image. Teaching materials and materials are displayed as images, and the instructor explains by voice and receives questions from students while conducting remote education and learning.
Table 01/009867

  Hereinafter, an example of a conventional distance learning system will be described in more detail.

  FIG. 21 is a block diagram schematically showing an example of a conventional distance learning system.

  In FIG. 21, this distance education system 101 includes a camera 102 on the instructor side, a microphone 103 on the instructor side, a CODEC (terminal unit) 104 for transmitting and receiving video and audio on the instructor side, and a CODEC (terminal for transmitting and receiving audio and video on the student side). A terminal 105), a student-side image display means 106, a student-side speaker 107, the Internet or a LAN (communication network) 108, and a server (distribution server) 109.

  In the conventional distance learning system, the lecturer's lecture is photographed by the camera 102, voice is inputted by the microphone 103, and these signals are transmitted to the lecturer side by the voice / video CODEC (terminal unit) 104 of the lecturer side. The student side receives the same audio / video codec (terminal) 105, displays it on the image display means 106, and outputs it through the speaker 107. The audio / video signal transmitted by the instructor is transmitted to the server (distribution server) 109 via the Internet or LAN (communication network) 108, and the server (distribution server) 109 sends the same signal to all participating students. It was realized by the delivery method.

  Because of this method, video and audio on the instructor side are sent to the student side, but there is a problem that the state on the student side cannot be grasped.

  In addition, a method has been implemented in which a camera and microphone are installed on the student side, the student side is photographed with the camera, and distributed to all participants via the distribution server. Since the video has a fixed viewpoint and only a limited amount of information that is partially cut off can be transmitted, image quality deterioration due to time delay or noise due to image transmission is added, and it is only used as a reference for grasping the situation. It was only a role. The remote camera image cannot be used to determine where the student is looking, and the instructor can speak directly to the student to determine whether the student really knows or how the student feels. I had to ask and ask if I understood. For this reason, it is difficult to grasp the various reactions of students from the remote camera image, such as whether the student understands, pays attention to, or shows interest in the lecturer's story. Only a limited effect was achieved, better than no.

  In the case of one-way video communication from the instructor side to the student side where the camera video cannot be transmitted from the student side, it is further difficult to grasp the situation on the student side, and there is a problem that the instructor must be a one-sided lecture was there.

  In other words, in the case of real-time and interactive distance learning and distance learning by combining a communication network such as the Internet and a computer, in the conventional video conference system, the situation of the other instructor or student is grasped. As a method, a method has been realized in which the faces and appearances of instructors and students are captured by a camera and connected by one-way or two-way communication. In this case, the instructor grasps the state of the student by looking at the camera video showing the student's situation taken and transmitted by the remote camera. It is difficult to grasp the various reactions of students, such as whether they understand, pay attention to, or are interested in the instructor's story from remote camera images, rather than without camera images Only a limited amount of effect was achieved.

  On the other hand, in the case of one-way video communication from the instructor side to the student side where the camera video cannot be transmitted from the student side, it is further difficult to grasp the situation on the student side, and the lecturer must be a one-sided lecture There was a problem.

  Compared to actually speaking face-to-face at the same physical location, the remote camera can take a fixed point of view and conveys only limited information that is partially clipped. . With the addition of time delay due to image transmission and degradation of image quality due to noise, remote camera images have only served as a reference for understanding the situation. It ’s not possible to know where the student is looking from the camera image, and the instructor talks directly to the student with voice to see if the student really knows, how the student feels, I had to ask if I understood that and understood it.

  Voice questions and responses to them have technical problems such as grasping the speaker, nomination, and microphone input control. In addition, speaking in a lecture in an environment where you do not know what kind of participants are present and cannot understand the situation is a considerable mental pressure for the students of the participants, and the active questioning is realized. There is a big problem as a human interface. In addition, there is only one person who speaks by voice, and it is very difficult to grasp the overall situation by voice as an educational environment where many people participate at the same time.

  In remote lectures, communication using text chat is also used, but text chat takes time to input text, and smooth communication is difficult until you get used to the operation considerably. It is difficult to convey the mood and atmosphere that cannot be expressed in letters. In addition, since the chat message of each participant is displayed as a sequence in the character chat, there is a problem that when a large number of participants input all at once, the character string increases, making it difficult to see and grasping the situation.

  In lectures and lectures that are physically gathered, the instructor can experience the student's understanding based on information other than language based on the students' facial expressions and gazes. It was difficult to understand the linguistic situation, and in fact, the instructor often kept talking at his / her own pace without understanding the student's understanding. The student side cannot use silent feedback via facial expressions, gaze, or gestures in physical space as a way to communicate their situation to the instructor. It could not be overcome, and the sense of unity with the teacher disappeared, resulting in a decrease in motivation and motivation for learning.

  Even if the instructor speaks a joke in an attempt to boost the atmosphere of the lecture participants, it is difficult to continue such a joke unless the student can know whether or not he / she laughed. It's hard to talk about a joke without a response from the participant right away. In such a case, is the instructor funny? It is not appropriate to ask. Lectures with no jokes and no reaction are inconvenient and difficult for both the lecturer and the listener.

  A similar sense of disconnection occurred not only between the teacher and students, but also between students. In physically gathered training and lectures, students can grasp each other's interests and understanding status from each other's facial expressions and actions. If you are physically present, you can understand in real time how your classmates understood the lecturer's story through non-verbal visual information such as gaze and gestures. However, it is very difficult to grasp the situation between students in the case of teleconferencing and distance learning using a video conference and a TV lecture system. In many TV conferences and TV lecture systems, lecturers and speakers are displayed as large images, but images regarding students who do not speak are not displayed in detail. Therefore, in conventional distance learning, it is difficult for students to exchange their learning enthusiasm and understanding status nonverbal as classmates, and the impact of the collective learning effect caused by lecture participation is reduced. This was one of the major causes of student motivation and motivation.

  The object of the present invention is to solve the above-mentioned problems of distance learning and distance learning, and the instructor, students, and students can mutually communicate and exchange their mutual understanding, interest, and emotion in real time. It is to realize a system and a system that can advance distance learning while exchanging information smoothly.

  The distance learning communication system according to the present invention includes a server and a plurality of terminals connected to the server via a communication network. Each of the plurality of terminals includes an arithmetic processing unit, an external input unit, and a display. Means, imaging means, voice input means and voice output means, each of the plurality of terminals is assigned a unique number, and each arithmetic processing means of the plurality of terminals is assigned a unique number. Corresponding alternation object image information can be stored, and when a plurality of terminals are put into an operation state, a unique alternation stored in each arithmetic processing means of the plurality of terminals put into an operation state The object image information is displayed on the display means of each of the plurality of terminals that are in the operating state via the server, and each operator of the plurality of terminals that are in the operating state receives the terminal When information is input from the external input means, the input information is displayed on the unique alternation object image information displayed on the display means of each of the plurality of terminals in the operation state. ing.

    Furthermore, the present invention realizes a communication method using a self-contained screen expression for distance education using voice communication and image communication. As means for realizing this, it is realized by combining the following means.

(A) A means to connect a remote lecturer and multiple students via a network, and to deliver and display the lecturer's voice, lecturer's face image, or document image to the student (i) The names and names of all students taking the remote lecture Distributing facial photos or camera images to all participants over the network and displaying them on the screen as a collection of “alternate objects” for each participant (e) Emotions and intentions during the course, opinions to be expressed, Various message lists regarding thoughts, gestures, actions, etc., thoughts, psychology, and actions are displayed on the participant's screen in a list, and each one matches their sensibility with one touch or one click with a pointing device such as a mouse. Means to select “Message” and enter it into the system (E) Each participant selects from the message list in (U) above The “message” is distributed to all participants via the network and displayed as a “balloon object” from the “personal object” for each individual. A means for delivering the “message” selected by the participant to all participants via the network and outputting synthesized speech and sound effects for the selected “message”. , A means to distribute the “message” selected by each participant to all participants via the network, and to express the animation on the screen as individual actions of the “separate object” for each individual. Other participants, such as “Balloon Object” from “Personal Object” for each individual, and individual actions of “Curtain Object” above For objects sent and displayed from each individual, the participant agrees and agrees with his / her sensibility with one touch or one click with a pointing device such as a mouse and inputs it as a “consent signal” to the system・ Distribution means (c) In response to the “agreement signal” from the participant delivered via the network in (a) above, the original “separate object” or “speech object” and the supporter ’s Means to animate “agreement object” on the screen, which is a modified version of “another object”. (K) Number of information transmissions such as “message” selection and “agreement signal” input from participants during the course If there is little information dissemination, the participants can either automatically from the system or manually by the instructor. Means for displaying and outputting alarms (input prompt messages, alarm sounds) (ko) Participants add new “messages” themselves to the message list in (c) above, and the corresponding “speech balloon” A means to process or add animation representations of "objects", synthesized speech, and "another object" itself

  In the present invention, an object corresponding to all the participants (another object image information) is displayed on the screen of all the participants, and the sensibility felt by the participants is delivered as a message with a simple one-click operation. In response to the message, the behavior and response of the participants in the lecture can be quickly and easily understood by dynamically adding an action as an image and a balloon-type decoration display to each of the other objects (another object image information). introduce. This facilitates the understanding of the sensuous situation by means other than direct audio dialogue between the instructor and students, and at the same time, realizes the effect of proactively expressing opinions and receiving responses from students. Can do.

  In addition, the seats of the alternate objects (alternate object image information) are displayed as a virtual classroom arrangement on the screen, and display control is performed as a seating image on a predetermined seat or a seat that can be freely selected by each person. Thus, a sense of entry into the virtual classroom of the participant is generated. As a result, the sense of distance between the student and the instructor in the virtual classroom, the sense of distance between students, the sense of neighbors, etc., through the seats and positions of each participant's alternate object (another object image information) A sensuous environment similar to physical space is realized. In addition, when allocating seats, the function of dividing the group of participants is combined with the image representation of the partition image, and the seat is divided by the partition image on the screen, thereby blocking the group audio and regulating the interaction. The group management control can be executed with a simple operation. This makes it possible to easily switch between environments, such as dividing a large number of people into groups of two at a single touch, or switching between lectures and group discussions, which cannot be performed in a physical classroom.

  In addition, the balloon-type decoration display on the alternation object (the alternation object image information) is applied to the voice speaker switching and microphone switching control in the distance learning environment, and the person who is speaking is clearly indicated and he / she wants to speak Realize a system that links the participant's clarification and voice control. With the control system that clearly indicates the speaker and the participant who wants to speak, it is difficult to control the speaking right among a large number of participants in the conventional system, which is difficult without artificial control by a moderator. It solves the problem of mobility control and realizes a method that facilitates smooth transition of speaking rights by autonomous judgment between participants.

  FIG. 1 is a diagram schematically illustrating the concept of a distance learning communication system according to the present invention. 1A is a conceptual diagram of a conventional distance learning system, and FIG. 1B is a concept of a distance education communication system according to the present invention.

  As shown in FIG. 1 (a), the conventional distance learning system is a concept in which a lecturer lectures in a distance education place while showing contents such as teaching materials to a student. It was something that was captured as a group.

  On the other hand, in the distance learning communication system according to the present invention, the concept of distance learning is a place where a lecturer and a plurality of students interact with each other through the lecturer's lecture as shown in FIG. And realize a system for that exchange. As a means to achieve this, a function that expresses the students participating in the distance learning field clearly on the screen as a separate representation of each student's existence, rather than grouping them together into a set of students To do.

  FIG. 2 is a block diagram schematically showing an example of a distance learning communication system according to the present invention.

  The distance learning communication system 1 includes a lecturer-side camera 2, a lecturer-side microphone 3, a lecturer-side video and audio CODEC 4, and a student-side voice and video CODEC 5. , A student-side image display means 6, a student-side speaker 7, the Internet or LAN (communication network) 8, and a server (distribution server) 9.

  The above-described configuration of this distance education communication system 1 is the same as that of the conventional distance education system 101 shown in FIG. 21, but this distance education communication system 1 further represents the participants' students to be displayed on the screen. The duplicate object image information 11 can be stored in the CODECs (terminal units) 4 and 5.

  It is to be noted that each of the alternation object image information 11 is made into unique image information (for example, a photograph of the user) so as to correspond to the unique number (ID) of the CODEC (terminal) 4 and 5.

  Each of the CODECs (terminals) 4 and 5 is a client object process for executing display control and communication control of the message list 12, the alternate object image information 11 and the message list 12 displayed on the image display means 6. A system 13 is provided.

  The server (distribution server) 9 has functions of a video / audio distribution server 9a and a message distribution server 9b.

  In this distance learning communication system 1, when a plurality of terminals 4 and 5 are put into an operating state, a camera 2, a microphone, and a conventional distance learning system instructor and student participants are connected as a communication function to connect the instructor and the student. 3. Audio / image CODECs 4 and 5 are used not only for the camera video and voice communication connecting the instructor and the student, but also on the participant's terminal screen on the participant's alternate object image information 11, message list 12 and message By displaying the list 12 and combining them, communication between participants can be realized.

  In this distance education communication system 1, instructors and student terminals are based on information distributed by a video / audio distribution server 9a and a message distribution server 9b connected via the Internet or a LAN (communication network) 8. The client object processing system 13 of the machine 5 displays the alternation object image information 11 and the message list 12 on the image display means 6. In the image display means 6 on the student side, in addition to transmission / reception and display of audio and video by the audio / image CODEC (terminal) 5 with the instructor side, the alternation object image information 11 and the message list 12 are displayed, and the mouse is displayed. The pointing device 14 is operated.

  In the message list 12, various messages such as “interesting”, “don't know”, and “surprising” that are assumed during the course are presented as a list in advance. The student selects a message that matches what he / she feels at any time with the pointing device 14 with one touch. The selected message is sent by the client object processing system 13 to the message delivery server 9b and delivered to all participants. The message distributed from the message distribution server 9b is processed by the client object processing system 13 of each participant, and is displayed as an operation of the alternate object image information 11 on the screen, a balloon type message, or the like.

  FIG. 3 is a diagram illustrating an example of message information distribution processing in the distance learning communication system 1. In FIG. 3, 4 is a teacher terminal, 5 is a student terminal, 9b is a message distribution server, and 8 is the Internet or a LAN (communication network).

  In this distance education communication system 1, a message input by a student participating in distance education is distributed to the instructor and all other students. As a method for realizing such mutual delivery of messages, there is a method in which each other delivers to each participant's terminal in a peer-to-peer manner. In this case, the total number of terminals is m. Then, when each terminal delivers the same message to each other all at once, m × (m−1) times of delivery processing excluding delivery to itself occurs. This is because traffic increases in proportion to the square when the number of participating terminals m increases, and there is a problem in practicality except in the case of a small number of participants.

  In order to prevent this problem, there is a method in which a message delivery server 9b as shown in FIG. 3 is prepared, all of the messages are delivered to this server 9b, and the messages of others are acquired from the server 9b. . In this case, if the total number of terminals is m, even if each student's terminals 5... Distribute the same message all at once, the distribution processing is performed by transmitting to the message distribution server 9b and from the server 9b. The reception of 2 times of 2m is sufficient.

  The message distribution in the distance learning communication system 1 can be realized by either a method of preparing the server 9b or a method of mutual distribution by peer-to-peer.

  FIG. 4 is a diagram showing an example of a configuration of a clone object on the screen realized in the distance education communication system 1.

  In this distance education communication system 1, the state and reaction of each remote participant are expressed by actions and balloons of the alternate object image information 23 that expresses each participant as a symbol.

  The basic form of the alternation object image information is the participant's face photo information or camera video information 21, the participant's name or nickname 22, the edge 23 indicating the shape of the alternation object image information, and the balloon displayed in addition to the alternation object image information 23. An object 24 is provided.

  Thereby, the participant can grasp | ascertain explicitly the presence of another participant by the participant's alternation object image information displayed on the screen of the image display means 6 ....

  In the conventional distance learning system, a list of participants' names was displayed, but face photographs and name tags as a whole were not handled in a divided manner. Also, in conventional video conferences and lectures, it was possible to shoot the video of the other party's face and upper body and situation video with the camera and display it on the other party's side, but the representation is only displayed for each video. It could only be displayed on the screen as a window.

  Even when many windows are displayed on the screen and the camera images of participants from different locations are displayed on each window, each window is simply expressed as a screen split. It was not possible to realize the image processing that was captured as a substitute.

  However, in this distance education communication system 1, in order to obtain a psychological recognition effect when the object image information is regarded as a lump as a part of the participant and can be handled, The edge 23 plays an important role.

  Here, as the shape of the edge 23 of the alternation object image information, a rectangular shape corresponding to the shape of the face photo or name tag is easy to realize as the image display processing, but the psychological effect as a human alternation is obtained. In order to increase, the effect of a softer oval or rounded shape rather than a rectangle is great.

  In addition, with regard to the representation of the alternate on the screen of the image display means 6, in order to increase the effect of human expression of a person, a shape close to the shape of an actual human being is expressed using three-dimensional computer graphics or the like. However, in such a case, the reality becomes too high, and a psychological problem occurs that it becomes difficult to actively perform various gestures and other actions.

  In this regard, in the distance education communication system 1, the constituent elements of the alternate object image information are limited to only two-dimensional materials such as a face photograph or a camera image of a face and a name tag, and the alternation is symbolically represented by the edge shape of the alternate object. This makes it possible to realize the effect that the action and modification of the alternation object do not have a realistic atmosphere, and are merely expressed as a symbol.

  This kind of symbolic expression is easy and positive for various emotional expressions such as apologize, surprise, and appreciation, as seen in the use of face marks that combine character symbols in character chat. It has the effect of making it transmit. On the receiving side, a symbolic overaction expression amplifies a humorous impression and produces an effect of smooth communication.

  Compared to face marks made up of character symbols, the body objects realized in the present invention have a face photo or camera image as an element, and thus have a presence as a real person, and are light as symbols Achieving both a sense of alternation and a psychological effect.

  Compared with a simple face mark, or a portrait display using a person expression by a pseudo or cartoon animation, or a portrait expression using an animal mark, etc., the face photo, camera image and name tag in the present invention The expression of the body by the combination of and uses the actual person's figure and name, so there is a high sense of reality as a person who is actually participating in the place.

  In communication under the real name such as education, learning, discussion, discussion, etc., by using the expression of the present invention, the presence of mutual trust among participants and the communication with the actual person is performed. The effect of nurturing will be expanded and willingness to participate.

  As described above, from the viewpoint of the balance between the presence effect and the symbol effect, the clone object in the present invention includes a realistic clone image by 3D CG, an abstract symbol such as a face mark, and the like. This is an intermediate configuration that extracts both necessary functions.

  FIG. 5 is a diagram illustrating an example of a shape changed from a rectangle in order to increase the psychological effect as a human divide as the shape of a divide object.

  FIG. 6 is a diagram showing an example of the screen configuration of the image display means on the terminal 5 side of the distance education communication system 1.

  In FIG. 6, 31 is a video image of the lecturer's lecture received and displayed, 32 a, 32 b, and 32 c are images of the participant's students, 33 is a message list, and 14 is a message list. A pointing device such as a mouse.

  In the distance education communication system 1, a student participating in distance education displays on the screen a camera image 31 that captures the state of the lecture sent from the lecturer, and listens to the lecturer's lecture sound while watching it. Divided objects 32a, 32b, and 32c consisting of participants' face photos, videos and name tags are displayed on the screen, and the images of students participating in the lecture are clearly shown. The student selects what he / she felt about the instructor's story during the lecture from the message list 33 displayed on the screen, and clicks on the pointing device 14 at any time to feel the feeling he / she felt at that time. Enter. The input message is transmitted to the terminals 5 of all the participants via the server 9. In response to the input message, the word representing the message itself is output as a balloon-type display from the branch object (such as a balloon display of a message “interesting”), or as an animation of the branch object 32c. It is displayed as an action (animated display of a one-turn action of a clone object as an expression of a “surprise” message). Instructors and other participants can clearly understand the psychological state of each student by looking at messages and actions displayed on the alternation object. Such explicit expression of the participant's psychological state and sensation can be realized with a simple operation (one click with a mouse), so that the participant can easily and positively express his / her senses. Compared to direct utterances and physical body movements, the animations of the balloon objects and the body objects that are output are deformed, exaggerated, and a humorous atmosphere is added, enabling smooth communication between participants. Great effect on brewing. In traditional physically attended interviews, the interviewer's senses and psychological state expressed by nonverbal means such as facial expressions and gestures can be expressed more actively as self-expression. it can.

  FIG. 7 is a flowchart showing an example of the processing flow of the distance learning communication system 1.

  The processing of the remote education communication system 1 is realized by dividing into processing on the client side (object processing) on the terminal 5 side and processing (message delivery) on the server 9 side.

  On the client side, there are a message object 41, a body object 42, a sound object 43, a balloon object 44, and an object communication process 45 relating to these objects. On the server 9 side, there is a message table 46 that defines operations corresponding to messages.

  In order to operate this, in the message object 41, first, a message list is displayed on the screen of the image display means 6 of the client terminal 5. The distance learning communication system 1 monitors a user input to the message list and detects a selection such as a click by a pointing device.

  Next, when it is detected which message the user has selected, the detection result is transmitted to the object communication control of the system.

  In the object communication control, a message selected by the user is transmitted to the server 9 side.

  The server 9 receives the message selected by the user and reads the target object corresponding to the message and the instruction operation for the object by referring to the message table registered in advance.

  Next, the system on the server 9 side distributes the object corresponding to the received message and the operation instruction for the object to all the clients participating in the distance learning.

Next, the object communication control on the client side receives an operation instruction for the object from the server 9 and transmits it to the target object on the client side.

  If the action instruction is for a clone object, the clone object displays the designated action or balloon object for the target clone object.

  If the action instruction is for a voice object, the voice object outputs an instructed voice message, or outputs an instructed sound effect or synthesized voice from the client.

  If the action instruction is an expression of approval for another participant, a balloon object connected to the target alternation object is displayed, and the approval expression is executed.

  FIG. 7 shows a method in which a message table that defines the types of objects and operation instructions corresponding to messages on the client side is registered in advance on the server side, and the operation instructions for messages on the server side are referred to. However, as another method, the message table is distributed to each client in advance, and the server simply executes distribution of the received message to all clients, and refers to the operation instruction for the message. There is also a method that each client can execute by referring to the message table. Whether such a message table is arranged on the server side or on the client side can be appropriately selected according to other system conditions such as the scale of the server and the frequency of modification to the message table.

  FIG. 8 is an example of a message prepared on the client-side screen for the remote education participant in the distance education communication system 1. Psychological assumptions made during attendance and meetings so that participants can easily enter their feelings and thoughts with one click or one touch while taking remote lectures or participating in teleconferences. Create a message list that expresses events as words and displays them as a list. Psychological events recalled by participants are roughly classified into “understanding / agreement”, “interest / interest”, “impression / emotion”, and “technology / hope” according to the type of intention. The In order to make it easier for participants to select words and vocabulary that correspond to their own feelings, the message list as a list display that is selected and entered by participants covers all four categories. It is necessary to prepare the kind of words and vocabulary.

  When displaying a list of vocabulary in the message list, it is easy to select the vocabulary that best matches the feeling that participants felt by classifying them according to the type of intention. The effect of becoming can be realized. In the above four classifications, positive vocabulary and negative vocabulary are made to correspond to each other and arranged and displayed in a list, thereby making it easier to select a vocabulary corresponding to each sense.

  The vocabulary to be displayed in the message list is not only psychological words but also “applause”, “nod”, “waving his head”, “yawning”, “laughing”, “shaking his head”, “eyes” There is also a method of preparing a vocabulary that directly corresponds to a specific non-verbal communication operation such as “round up”, “waving hands”, “bowing”. By selecting a non-verbal communication action that the participant wants to do from the message list and displaying it as a character on a balloon object from the alternation object or expressing it as a simulated animation action of the alternation object, the intention or intention Can be communicated to other participants.

  As a method for displaying the message list, a method in which the entire vocabulary is always displayed in a tabular form is a method that is most easily selected by the user. The vocabulary that is always displayed on the screen is always easily in the user's field of view, facilitating positive self-expression and response reactions. If it is not displayed in a list, the user is less likely to click on the reaction because of the troublesome operation.

  On the other hand, if the message list cannot be displayed at all times due to the convenience of the display screen, there are two types of call-type display methods using a pull-down menu format or icon-type buttons. This is an effective method for a user who is proficient in self-expression and operation.

  FIG. 9 is an example of an action expression of a clone object of the distance learning communication system 1 according to the present invention.

  Reference numeral 51 denotes an example of the structure of a clone object, which shows a clone object composed of a face photo and a name tag with a name.

  52 shows an example of a balloon object from a clone object. The balloon object is used to display the input message as a text message. A message box that shows the message content and the caller makes it easy to see who is sending what message.

  53 shows an example of animation modification in which the color and thickness of the edge of the clone object are changed. By dynamically changing the color and thickness of the edge of the clone object, the emotion and psychological state of the user corresponding to the clone object are expressed. For example, by changing the edge color to red for a positive message and changing to blue for a negative message, the psychology expressed in the message is amplified and expressed more clearly. Can do.

  Reference numeral 54 denotes an example of animation modification in which a clone object is rotated. In conventional video conferences and lectures, the facial images and camera images of a person that were displayed as a fixed display window rotate with the name tag and edge, so that the participants' psychology Can be expressed strongly and impressively. It can be realized as an effective and easy-to-understand expression to emphasize surprised emotions and the strength of emotions received. In the actual place, it does not perform intense movements that rotate the body, but in such a place as a symbol, by rotating the alternation object, as a non-verbal action of your own psychological expression Can be used.

55 is an example of a vertical motion or a left-right motion of a clone object. By expressing the movement of the alternation object up and down or left and right, it is possible to express the impression of non-verbal gestures such as “nodding” and “no”.

  Reference numeral 56 denotes an example of animation modification in which the brightness of a clone object is blinked. By blinking the alternation object, it can be used as an expression corresponding to a non-verbal motion of joy that makes the eyes crackle and the heart pounds in an actual place.

  The animation modifications to these alternation objects can be applied independently or in combination. What kind of animation modification is displayed for the message selected and input by the user is registered in advance as a message table, and the system performs the corresponding operation.

  FIG. 10 is an example of emotional expression of approval between the two separate objects.

  An example of a screen representing the environment of distance learning or teleconferencing that expresses non-verbal movements in which multiple objects arranged in a sequence on the screen shake each other and touch each other's hands It is.

  1) Name: AA participants select the message “OK!”, And display a balloon object corresponding to the message on their alternation object.

  2) A participant of name: FF who sees this balloon object and agrees or agrees with it selects the balloon object of name: AA and clicks or touches it. This sensation can generate an impression corresponding to a non-verbal movement in which the user shakes or touches the opponent's hand.

  3) Name: A balloon object linked to or combined with a balloon object of name: AA as a modification to express a touch action of a participant of FF with respect to a message of a name: AA participant. Displayed in expressions. As a result, all the participants on the spot are explicitly informed that the participants of the name: FF agree with the participants of the name: AA.

  In conventional distance learning and video conferencing systems, participants cannot shake hands or touch each other, and although they are visible as camera images, they sit in the same place and are non-verbal There was a frustration that they could not interact with each other. Constraint that physical interaction such as touching opponents, hitting shoulders, shaking hands, and small pokes is not possible, and everyone is distributed in divided windows and cannot reach each other was there. In the present invention, by realizing the interaction between the two different objects through the symbol of the second object, each other can share the same place seamlessly and interact with each other with one another. A feeling of being can be generated.

  Such a combined representation of the self-portrait objects between participants can be applied as an animation modification of non-verbal movements of mutual interaction such as psoriasis, handshake, clapping shoulders, embrace, etc. as well as approval and approval. Interaction through a self-portrait object has the effect of leading to a more amplified interaction than actually sitting in a real place. With a simple one-click, you can clearly show your favor to the other party by displaying the approval object and modifying the alternation object as if you were hitting the opponent's shoulder. Even those who are said to be poor at self-expression can use it as a gesture of over action with a simple object through the alternate object.

  FIG. 11 shows an example of the correspondence between the animation modification of the clone object and the message according to the present invention.

  In response to the message “I see (nodding)” 61a, the repetitive motion 61b of the alternation object is repeated. For 63a, the action 63b of rotating the body object to the left and right is shaken, and for "No (shake the neck sideways)" 64a, the action of repeating the body object left and right 64b, "Absurd (shrug yourself) ) “65a”, an operation 65b for reducing the entire size of the surrender object, and “sorry (head down)” 66a, an operation 66b for reducing the vertical size of the survivor object and reducing the height, For "surprised (surprise action)" 67a, an action 67b for rotating the alternation object, For (joy Operation) "68a, respectively made to correspond to operation 68b for changing the blinking or color alter ego object, and holds it as a message table 46.

  Such a message table 46 is prepared, the operation corresponding to the message issued from the participant is referred to, the animation operation of the alternation object is executed, and the vocabulary of the message is displayed on the balloon object.

  As for the animation modification operation of the body object, as in this embodiment, other than selecting the body object motion that has an impression corresponding to the motion of the actual person and is easy to symbolize as a common motion to all participants. It is also possible for the user to freely select or process the behavior of the alternation object and register it as a message table in advance as the behavior function of the participant's own alternation object.

  FIG. 12 shows an example of realizing correspondence with various messages by preparing a plurality of face photographs of participants as a body object of the present invention and switching them to display them in the body object. .

  In order to operate this, as a face picture of the participant, a face picture 71b with a normal expression, a face picture 72b with an unknown expression, a face picture 73b with a laughing face when he thought it was interesting, and so on A face photograph 74b is prepared, and a surrogate object using them is created.

  The correspondence between each message and the self-portrait object is as follows: a face photo 72b with an unknown expression for the message 72a "I don't know"; A message table 46 corresponding to a face photograph 74b when the message “I see” 74a is satisfied, and a face picture 71b with a normal expression is set for other messages 71a.

  The registered message table 46 is distributed to the client in advance.

  In response to a message input from a participant, the display is switched to a display of a duplicate object consisting of a face photo of the corresponding expression and is displayed on the screen. As a result, the face photograph of the alternation object is switched according to each inputted message, and the change in the facial expression of the participant can be expressed with emphasis even if there is no human image photographed by the camera. When preparing a face photo in advance, it is possible to freely select and register facial expressions that one wants to intend, and to easily realize the self-expressions that each person likes.

  FIG. 13 is an implementation example of the seat arrangement of the alternate object of the present invention.

  A surrogate object consisting of a face photo and a name tag is a symbol that clearly indicates the presence of a participant on the screen, and its position on the screen is controlled as a seat in a distance education classroom or a video conference room. FIG. 13 shows an example in which an instructor's video or an instructor's alternation object is arranged in the upper stage, and a student's alternation object is arranged in a matrix form below. Participants' students have seats in the classrooms that are lined up toward the teacher's platform, and each has a seated expression.

  FIG. 14 is an example of another seat location.

  It is controlled as a seat arrangement that surrounds the video of the instructor or the alternate object of the instructor, and the alternate object of the student surrounds it. In this case, the participating students are expressed as having the impression that everyone is sitting on the round table surrounding the instructor.

  The instructor's video can be displayed as a simple camera video as it is, or it can be displayed and controlled as a separate object using the instructor's own face photo as a material. If the instructor himself is expressed as a separate object consisting of a face photo, remote education and conference interaction can be realized only by voice communication and the operation of the separate object without the instructor's camera image.

  FIG. 15 is a diagram for explaining an example of a method for controlling the seat arrangement of a substitute object according to the present invention.

  As for the seat arrangement of the participant's alternate object, it can be placed at a predetermined seat position on the screen when the participant logs in and connects to the system. Alternatively, it is also possible to specify a seat in a predetermined order according to the order of entering the classroom (order of connection), and to make a seat.

  In FIG. 15, first, 1) the client connects to the server, logs in, and joins the classroom.

  Next, 2) The server side distributes to the participating clients a seat table indicating the correspondence relationship between the participation order of the participants, the names or ID numbers of the participants, and the seat positions on the screen.

  Next, 3) Based on the delivered seat table information, the client determines the arrangement on the screen of each alternate object that participated in the classroom.

  Next, 4) the client displays the alternation object at the determined position.

  If a participant's seating arrangement is not fixed in advance, such as first-come-first-served basis, information is added to the seat table each time a participant is added, and the client receives the added seat table and Determine the seating arrangement for participants.

  FIG. 16 is a diagram showing an implementation example of the seat and partition function in the present invention.

  In lectures, exercises, and discussions in distance education and video conferencing, there are cases where it is inconvenient if all the participants listen to the same story at the same place and discuss together. For example, depending on the purpose, such as a mode in which discussions are divided into several small groups, a mode in which two people are paired and practiced, a mode in which all members are divided into separate rooms and work on exercises and tests on their own. It may be necessary to realize the room configuration. Such a separation method between large room mode and individual room mode and realization method of a set requires the installation of movable walls and partitions, etc., which is very troublesome when trying to realize in physical space There is.

  Conventional distance learning systems and video conferencing systems aim to realize only the function of all participants gathering in the same place, and such a function of freely dividing the space of classrooms and conference rooms by partitions has not been realized.

  A server that distributes information, video, and audio to a remote client can control what information and audio signals are distributed to which participants, and which information and audio signals are not distributed. The method of designation was an issue. It is very cumbersome to specify or designate who and who of a large number of participants are in groups and what information to block. The problem was how to transmit grouping information to the server.

  Also, in classrooms and conference rooms for educational services, instructors and presenters gather all together or group them individually, and freely circulate and patrol the divided partitions. However, it is necessary to provide guidance and grasp the situation, but no simple and easy-to-use method has been realized for the operation method related to separation and movement of such a space.

  FIG. 17 shows an example of realization of a method of quickly switching a group mode of a classroom by an easy-to-understand and simple operation by interaction with an image of an object on the screen by using an object on the screen, a seat object, and a partition object. It is.

  In order to operate this, in the system constituted by the alternation object of the present invention, 1) the design (shape, color, etc.) of the seat object and the partition object is registered in the server. The seat object is displayed as a figure when the chair is viewed from above, and the partition object is seen from the ceiling of the partition wall that surrounds the seat when the screen is viewed as a classroom space as shown in the figure. Expressed as a two-dimensional line drawing layout. Express the partition wall as a line. An area surrounded by a line represents a place surrounded by a partition, is closed to the outside, and is a target of blocking voice exchange with the outside and blocking interaction.

  2) Register partition object functions (such as voice blocking and interaction blocking) in the server. Define what kind of shielding is performed by the partition object to be used, and register the function. For example, specify audio blocking or interaction restrictions with the outside of the partition.

  3) The arrangement is determined by the combination of the seat object and the partition object, and is registered in the server. The arrangement and size of partitions and seats are determined in the manner of creating a two-dimensional floor plan, how many seats are arranged inside the partition, and how they are grouped.

  4) Distribute seat and partition information (shape and function) from the server to the client. Displays a clone object, seat object, and partition object on the client's screen. These individual objects are expressed as display objects on the screen that can be freely moved on the screen.

  The alternate object is seated on the seat object on the client screen. In the seating instruction operation, a clone object displayed on the screen is selected by an image operation such as drag and drop or selection click, and moved to a seat in the partition. This instruction operation is executed individually by manual operation on the screen, or as an automatic movement to a seat predetermined by the system. By moving the display object on the screen, the grouping within the whole participant and the designation of the restriction of the communication function between the participants are transmitted to the server side. With respect to the self-portrait object placed on the seat surrounded by the line drawing representing the partition wall, the specification of restrictions that restrict communication and interaction with the self-portrait object outside the partition is transmitted to the server. For communication between the different objects placed inside the partition, the function is specified so that voice and interaction can be freely exchanged.

  5) Perform communication control according to the function specification of the partition object (blocking voice with the outside partition, blocking interaction, etc.) The server side executes information distribution restriction corresponding to the designated restriction function. The server side restricts or switches the delivery of voice and interaction based on the designation information of the group configuration of the participants.

  In such a two-dimensional floor plan type partition layout diagram on the screen, constraint control is performed on the split object placed on the seat surrounded by the line drawing-like partition wall by a simple operation of moving the split object. By carrying out, it is possible to implement group division and set functions in a virtual classroom or conference room. The function of the instructor or the moderator to circulate between the divided partitions can be instructed and realized by a simple operation by moving his / her own object to the target partition.

  FIG. 18 shows an example of seat arrangement control using a body object and a partition object.

  In advance, the seat arrangement, who and who of the participants are divided into groups, and what kind of partition arrangement is made are determined as a seat table. Information necessary for group division, such as the participant's name or ID, functional conditions in the group configuration such as the seat position, voice blocking, and the group number for grouping, is registered in the server as a seat table. Keep it.

  1) The designated seat table information is distributed from the server to the client. This distribution instruction is input to the system by a button or command input by an operator who instructs group division such as a lecturer or a moderator.

  2) Based on the delivered seat table information, the client-side system determines the placement of each of the surrogate objects that participated in the classroom.

  3) A clone object, a seat object, and a partition object are displayed at a designated position. Based on the seat table, the display of the partition and the object of the seat is created, and the alternation object is arranged at the same time.

4) The server and the client control interaction between objects and interactive communication according to the designated partition information.
In this embodiment, the information of the seat table relating to the group division of the participants is delivered to each client at the same time according to the delivery instruction to the server by the instructor such as an instructor, and the group division is performed based on the information. Easily and quickly divide participants into partitions in an instant. It is easy to change classroom conditions for lectures with full participation, individual exercises, role play for two people, and small group discussions.

  FIG. 19 shows an embodiment in which the right to speak of a speaker is controlled by voice using a substitute object in the present invention.

  In a distance education system or a video conference system in which a large number of people participate, in principle, only one person can speak by voice at the same time. When a plurality of people speaks simultaneously, a plurality of voices are output to one speaker on the client side, making it very difficult to hear. Therefore, in lectures and conferences, a method for controlling the right to speak is an issue.

  In remote lectures and video conferences, voice signal transmission delay occurs due to voice signal encoding processing and communication processing, and a large delay occurs between the time when the voice is actually spoken and the time when the voice is transmitted to the other party. Many delay times such as the operation time for the participant to obtain the right to speak by the microphone, the operation time from when the right to speak is spoken to when speaking, the delay time until the voice is actually spoken and transmitted to the other party, etc. Will overlap. For this reason, in conventional distance learning systems and video conferencing systems, who is speaking to the participant, who is seeking the right to speak, who is the right to speak next, etc. There was a problem that it was difficult to speak because the situation was not understood. Since it was difficult to understand the transition status of the right to speak, there were many problems such as the fact that many people collided at once and who did not know who should be granted the right to speak in order.

  In FIG. 19, the transition of the speaking right between the participants is realized by visualizing the transition state of the speaking right by using the participant display by the alternation object.

  (1) All participants are displayed on the screen as alternate objects, and a speech button is displayed on the screen. The comment button is displayed as “speak button (accepting)” as an initial state (step S1).

  (2) A participant who wants to speak asks for a speech by clicking or pressing the speech button (step S2). By this input, the system detects a person who wants to speak, and displays “Speaking Accepted” on the speaking button of the speaking button input person. At the same time, a balloon object “I want to speak” is displayed on the alternation object of the comment input person (step S3).

  (3) It is often the case that a plurality of speaking applicants press the speaking button almost simultaneously. In this case as well, a balloon object “I want to speak” is displayed on the alternation object of the person who wants to speak. At this time, the system detects the order of the input time of the speech requester within the reception time range of a certain time, and displays the order in accordance with the order. By setting a reception time range for a certain period of time, confusion caused by a speaker's collision due to a voice delay or a delay in the processing of the right to speak is avoided (steps S4 and S5).

  This reception time process detects a new “speaking acceptance”, starts the process, determines a new first speaker, the speaker speaks, and a voice of a predetermined time T is distributed. During the time until the participant hears a new comment, all other participants who have pressed the speak button within the previous time are not designated to move the microphone to other speakers. Implement in a way that disclaims the right to speak. As a result, the participant is always guided to listen to somebody else's remarks and then seek new remarks (step S6).

  (4) The system determines a participant who gives the right to speak by the microphone and a participant who rejects the right to speak based on the order of the voice requesters. For the participant who gives the floor, a message such as “microphone connection ON” is displayed, and the voice signal of the floor is delivered to the participant. At the same time, a “speaking” balloon object is displayed on the alternation object of the speaker (step S7).

  (5) To the participant who rejects the right to speak, the message “other is first” is displayed on the comment button to notify that the right to speak is rejected (steps S8 and S9).

  (6) When the speaker presses the speaking button and turns off the microphone (step S10) or when the speaking time exceeds a predetermined time T, the initial state returns to the “speaking button (accepting)”. .

  In this method, since the speaker and the desired situation of the speech are displayed as a separate object, it is easy to clearly understand who is requesting the speech, and the transfer of the speech right between the participants becomes smooth. In addition to the determination based on a uniform time order, there is a method for determining the right to speak, as well as a method in which the lecturer is individually nominated and instructed manually by the positions and priorities of the participating members. By linking the display of the clone object and the determination process, these determination processes can be easily incorporated into this method.

  FIG. 20 is a diagram showing another embodiment of the system of the present invention.

  In FIG. 20, 2 is a camera on the instructor side, 3 is a microphone on the instructor side, 4 is a CODEC (terminal) for transmitting and receiving video and audio on the instructor side, and 11 is a participant student to be displayed on the image display means 6. A surrogate object to be represented, 12 is a message list to be displayed on the screen, 13 is a client object processing system for executing display control and communication control of the survivor object 11 and the message list 12, and 6 is an image display of a lecturer video on the student side. Means, 7 is a voice input / output of a speaker and a microphone on the student side, 5 is a CODEC (terminal) that transmits and receives voice and video on the student side, and 13 is a display control and a communication control of a student's side object and message list. Is a client object processing system, 12 is a student message list, and 14 is a client terminal mouse Computing device, 81 remark button lecturer, 82 remark button student side, 8 Internet or LAN ((communication network) 8,9 is a server (distribution server).

  The server (distribution server) 9 includes a video / audio distribution server 9a and a message distribution server 9b.

  In the present invention, as a communication function for connecting a lecturer and a student, a lecturer and a student are connected by a camera 2, a microphone 3, and a voice / image CODEC (terminal device) 4 that connect a lecturer and a student participant of a conventional distance learning system. In addition to the camera video and voice communication that connect to each other, the participant's alternation object 11, the message list 12, and the speech buttons 81 and 82 are displayed on the screen of the participant's terminal 5, and these are combined. Realize communication between participants.

  Based on the information distributed by the video / audio distribution server 9a and the message distribution server 9b connected via the Internet or LAN (communication network) 8, the client object processing system 13 of the teacher and student terminals In the student's terminal, in addition to the transmission / reception and display of audio / video by the audio / image CODEC (terminal) 5 with the instructor, the student object 11 and the message list are displayed on the screen. A list 12 and a speech button 82 are displayed and operated by the pointing device 14 such as a mouse.

  In the message list 12, various messages assumed during the course are presented as a list. The student selects a message that matches what he / she feels at any time with the pointing device 14 with one touch. The selected message is sent by the client object processing system 13 to the message delivery server 9b and delivered to all participants. The message distributed from the message distribution server 9b is processed by the client / object processing system 13 of each participant, and is displayed as an action of a duplicate object on the screen, a balloon-type message, or the like.

  The voice signal on the lecturer side or student side is requested to speak by pressing the speech buttons 81 and 82 on the screen. The right to speak by pressing the right to speak button is transmitted to the message distribution server 9b via the Internet or LAN ((communication network) 8), and the right control process and the display of the speaker by the balloon object are executed. At the same time, in the video / audio distribution server 9a, a speaker switching process is performed as communication control of a speaker's audio signal.

  When it is necessary to divide the group of the participants, the seat table information of the participants is distributed from the message distribution server 9b to each client according to the instruction of the lecturer. Splitting is performed. At the same time, distribution of control information such as audio blocking and interaction restriction is distributed from the video / audio server 9a according to the partition function instructed by the seat table.

  As described above, according to the present invention, a number of terminals, clients, and servers connected via a network are connected, and messages can be freely sent with a simple operation from the message list displayed on the screen with one click. By selecting and distributing to participants and outputting and expressing the behavior of the body object corresponding to the message and the balloon object display, various functions for non-verbal communication from participants can be realized, It is possible to improve the communication expression of participants such as distance learning and video conferencing, and support to increase learners' willingness to participate.

  FIG. 1 is a diagram schematically illustrating a concept of a distance education communication system according to the present invention, FIG. 1A is a conceptual diagram of a conventional distance education system, and FIG. This is the concept of the distance education communication system.   It is a block diagram which shows roughly an example of the distance education communication system which concerns on this invention.   It is a figure which shows the example of the delivery process of the message information in the distance education communication system which concerns on this invention.   It is a figure which shows the example of the structure of the clone object on the screen implement | achieved in the distance education communication system which concerns on this invention.   It is a figure which shows the example of the shape deform | transformed from the rectangle, in order to increase the psychological effect as a human part as a shape of a clone object.   It is a figure which shows the example of a structure of the screen of the image display means by the side of the terminal of the distance education communication system which concerns on this invention.   It is a flowchart which shows the example of the flow of a process of the distance education communication system which concerns on this invention.   In the distance learning communication system according to the present invention, it is an example of a message prepared on a client-side screen for a participant in distance learning.   It is an example of the action expression of the alternation object of the distance education communication system concerning the present invention.   This is an example of an emotional expression of approval between the other objects.   It is an Example of the correspondence between the animation modification of the surrender object and the message of the present invention.   This is an example of realizing correspondence with various messages by preparing a plurality of facial photographs of participants as an alternate object of the present invention and switching them to display them in the alternate object.   It is the implementation example of the seat arrangement | positioning of the alternation object of this invention.   It is an Example of another seat arrangement | positioning of this invention.   It is a figure explaining the example of the method of controlling seat arrangement | positioning of the surrender object of this invention.   It is a figure which shows the implementation example of the seat and partition function in this invention.   This is an example of realization of a method for quickly switching a group mode of a classroom by an easy-to-understand and simple operation by interaction with an object image on the screen by using an on-screen object, seat object, and partition object.   It is an Example of seat arrangement control using a separate object and a partition object.   The present invention is an embodiment in which a speaker's speaking right control is performed by voice using a clone object.   It is a figure which shows another Example of the system of this invention.   It is a block diagram which shows an example of the conventional distance education system roughly.

Explanation of symbols

1 Distance Education Communication System 2 Lecturer's Camera 3 Lecturer's Microphone 4 CODEC (Terminal) that Transmits and Receives Lecturer's Video and Audio
5 CODEC (terminal) that sends and receives audio and video on the student side
6 Image display means 7 Speaker 8 Internet or LAN (communication network)
9 Server 11 Alternate object image information 12 Message list 13 Client object processing system 14 Pointing device

Claims (1)

Server,
A plurality of terminals connected to the server via a communication network;
Each of the plurality of terminals is
Computation processing means, external input means, display means, imaging means, voice input means and voice output means,
A unique number is assigned to each of the plurality of terminals, and
The arithmetic processing means of each of the plurality of terminals is adapted to be able to store a separate object image information corresponding to a unique number,
When the plurality of terminals are in an operating state,
The unique part object image information stored in the arithmetic processing means of each of the plurality of terminals in the operating state is sent to each of the plurality of terminals in the operating state via the server. Displayed on the display means,
When an operator of each of the plurality of terminals in the operating state inputs information from an external input unit of the terminal, the input information is stored in each of the plurality of terminals in the operating state. A distance education communication system adapted to be displayed in unique alternation object image information displayed on a display means.

Images