JP4651981B2 - Education information management server - Google Patents

Description

  The present invention improves the educational effect by customizing the learning environment of a user who is a student, particularly the interface between the user and the system, in an educational field such as distance learning (e-learning). The present invention relates to an information processing apparatus equipped with the above technology.

  Recently, the e-learning business has started in Japan. There are two types of e-learning: asynchronous e-learning and synchronous e-learning.

  A popular method for asynchronous e-learning is a web-based training method using ASP or the like, in which a user can use teaching materials at a place where he / she wants to learn. However, with regard to content, it is a multi-pass method that leads to different paths according to the answer results of students, but general-purpose information is not content that is adapted to each student in one page of the web page. It is presented as

  On the other hand, in synchronous e-learning, a video chat method using ASP and a two-way video communication method using an ISDN line connect teachers and students who are remote from each other, and conduct classes in the same classroom. Can learn in situations like

  The following proposals have been made for these e-learning. For example, Patent Document 1 discloses a training support method in which a training server individually provides training guidance and support to a user using a virtual space. Further, in Patent Document 2, in the question presentation method of a host computer that accepts a question from a user, an optimum question sentence is presented to the user, and the user himself / herself asks a question from a plurality of question sentences presented. A method of selecting a question sentence close to the content to be attempted is disclosed.

JP 2004-16738 A JP 200405322 A

  However, in synchronous e-learning, it is necessary for teachers to correspond to the number of students, and there are many cases where teachers are not able to respond to the time required by students, and the tuition fees for attendance are high accordingly. This is the current situation.

  An object of the present invention is to generate a pseudo-environment in which a student student is interacting with an actual teacher, thereby providing an education support apparatus and an education information management server having contents adapted to individual students Is to provide.

  Another object of the present invention is to provide an education support apparatus that can improve learning efficiency by projecting a conversation situation that a student who is a user normally conducts to an animation character and having friendliness to a teacher, and To provide an educational information management server.

  In order to solve the above-described problems, the outline of typical ones of the inventions disclosed in the present application will be briefly described as follows.

Education for providing a connection unit connected to the student side terminal and the teacher side terminal, and for assisting the student in interactive learning with the teacher or a digital instructor corresponding to the teacher using the student side terminal An information management server that generates dialogue pattern data in dialogue learning between the student and the teacher from the storage unit, the student information and the teacher information acquired from the student side terminal and the teacher side terminal dialogue pattern generating means for the data of the interactive pattern, when a registration means for registering in the storage unit in the sequence, from among the interactive pattern registered in the storage unit, extracts characteristic interaction pattern dialogue pattern extraction function to be registered in the storage unit, in the characteristic interaction patterns registered in the storage unit, certain interactions patterns of the teacher for specific interaction pattern of the student The, and a unique digital instructor generation function to generate a digital instructor corresponding to該教nurses by animation characters and synthesized speech.

  According to the present invention, for example, a student who is a user can perform learning while maintaining a conversational environment with a real teacher based on the motion information and audio information of the output animation character. Improvement can be expected.

  Hereinafter, an embodiment of an education support apparatus according to the present invention will be described with reference to the drawings. Implementation

The system shown as an example of the form is a system that mainly handles educational information, and is a system in which a student who is a user learns using learning content.

  First, an embodiment of the present invention will be described with reference to FIG. FIG. 1A is a diagram showing the overall hardware configuration of the education support apparatus of the present invention, and FIG. 1B shows the education support apparatus of the present invention from the viewpoint of processing functions. Reference numeral 101 denotes an education information management server for accumulating learning related information and analyzing the accumulated information. 102 is a PC used as a teacher's terminal for a teacher to give a lecture to a plurality of students in real time, and interactive classes with students. 103 is for a student who performs learning PC as a terminal. Reference numeral 104 denotes a video information storage server. Each server and PC have a communication control function and are configured to be connected to each other via a communication network 105.

  The teacher PC 102 is equipped with a speaker 10201 for communicating with the student, a camera 10202 for imaging the teacher, a microphone 10203 for the teacher to input voice, and a keyboard and mouse for information input. Yes. The student PC 103 is equipped with a microphone 10301 for the student to communicate with the teacher, a camera 10302, a speaker 10303 for acquiring voice from the teacher, and a keyboard and mouse for information input. In order to simplify the explanation, an example is shown in which one teacher PC is connected to one education information management server. However, in practice, a plurality of teacher PCs can be connected. And In this example, one student PC is connected to the server. However, it goes without saying that a plurality of student PCs can be connected to the teacher PC. Each server and PC is composed of a computer, and includes a CPU, a memory, and an input / output device, and holds a plurality of programs that realize various functions by being loaded into the memory and executed. Yes.

  As shown in FIG. 1B, the education information management server 101 includes, as main functions, a dialogue communication function 101A between a teacher and a student, a text display editing function 101B, a dialogue pattern extraction function 101C, a unique digital instructor generation function 101D, a teacher ( A learning support function 101E using a real instructor, a learning support function 101F using a digital instructor, and a learning schedule management function 101G. In other words, each of these functions is executed and processed in the control unit configured by the CPU, memory, and program of the education information management server 101.

  The dialogue communication function 101A between the teacher and the student has a function of registering dialogue pattern data in the learning means in the time series in the dialogue learning between the student and the teacher. The dialogue pattern extraction function 101C has a function of extracting a characteristic dialogue pattern from the registered dialogue patterns. More specifically, a function for performing motion pattern extraction and utterance pattern extraction by analyzing moving images and sounds of teachers and students at the time of learning, and at least two sets of extracted motion patterns and utterance patterns, respectively A dialogue pattern is extracted in association with the voice and moving image separately acquired from the teacher and the student, and a function for determining the similarity between the extracted motion pattern and utterance pattern, and the similarity is a predetermined value or more. And a function of registering as a dialogue pattern.

  The unique digital instructor generating function 101D has a function of generating a teacher's specific dialog pattern for a student's specific dialog pattern as a digital instructor corresponding to this teacher using animation characters and synthesized speech.

  Further, the teacher PC 102 has, as main functions, a dialogue communication function 102A between the teacher and the student, a text display editing function 102B, a unique digital instructor generation function 102D, a learning support function 102E by a real instructor, and a learning support function 102F by a digital instructor. And a learning schedule management function 102G.

  Furthermore, the student PC 103 has, as main functions, an interactive communication function 103A between the teacher and the student, a text display editing function 103B, a learning support function 103E using a real instructor, and a learning support function 103F using a digital instructor. The video information storage server 104 has a video / audio storage function 104H.

  As described above, the education support apparatus according to the present invention has, as an overall configuration, an input unit that acquires a moving image obtained by capturing a set of interacting users and the set of uses interacting at the terminal. An input unit that obtains the voice of the person, an input time detection unit that records the input time of the moving image and sound obtained from the input unit, an image output unit that displays the animation character, and a voice that is synchronized with the animation character. An audio output unit for outputting and a control unit are included. The control unit analyzes the moving image to extract the motion pattern of the set of users, analyzes the voice to extract the utterance pattern of the user, and extracts the extracted motion pattern and utterance. Dialog patterns are extracted in association with time series for at least two sets of patterns, and the degree of similarity between the action pattern and the utterance pattern extracted from the same user as the voice or moving image separately obtained from the user is determined. If the degree of similarity is high, the other user's action pattern for the extracted one user's dialog pattern is output as a moving image by the animation character, and the other person's utterance pattern of the extracted dialog pattern is simultaneously spoken. Output.

  According to the present invention, for example, a student who is a user can perform learning while maintaining a conversational environment with a real teacher based on the motion information and audio information of the output animation character. Improvement can be expected. In addition, the animation character can be controlled based on the action pattern, and the action described as the pattern can be performed. Furthermore, the synthesized voice can be output as the voice of the character based on the utterance pattern. become. Also, by displaying the motion pattern in the dialogue pattern data set on the screen as the motion of the target animation character, and further displaying the voice pattern in the dialogue pattern data set as voice of the target animation character by voice display, Based on the utterance pattern, a synthesized voice can be output as the voice of the character.

  Next, the configuration of the education information management server 101 will be described with reference to FIG. Reference numeral 1011 denotes a CPU that performs processing in accordance with the activated program, 1012 denotes a memory that stores the activated program, and 1013 denotes a hard disk that stores memory data and the like. Data to be accessed is read into the memory 1012 as necessary, and data processing for educational support based on the CPU 1011 is performed. The hard disk 1013 stores operation pattern data 101301 for extracting operation information from the stored video data, a voice recognition dictionary 101302 for recognizing language information from the stored audio data, and an operation pattern from the stored video data. Video pattern recognition data 101303 for extraction, student text data 101304 used by students in the class, and problem correct answer data 101305 which is correct answer data for the problem described in 101304 are stored.

  In addition, the memory 1012 includes a system program 101201 for controlling the entire system when the server is started, a data transmission / reception program 101202, a voice recognition program 101203, a video recognition program 101204, a communication pattern extraction program 101205, a character generation A program 101206, a speech synthesis program 101207, and a character-user interaction control program 101208 are stored. The memory 1012 further stores a user data table 700 corresponding to each teacher and student. For example, as shown in FIG. 6, the user data table 700 is stored in time series (702) for each event data, audio data, and video data as the data section 701 and for each user. Note that the user data table 700 is used for temporary data storage, and when it is necessary to hold these data for a long time for the convenience of processing, these data are temporarily stored in the hard disk. Stored in a predetermined area.

  Next, details of the teacher PC 102 will be described with reference to FIG. The memory 1021 of the teacher PC 102 includes a system program 102101 and a data transmission / reception program 102102 for controlling the PC. Also, user input information collection / storage program 102103 for collecting and storing information input from the keyboard and mouse by the teacher, video input / storage program 102104 for storing the video acquired by the camera, and acquiring and storing the sound by the microphone A voice input / storage program 102105 and a communication program 102107 for performing a dialogue with the student s are read into the memory. The information read into the memory is used for a dialog communication function between the teacher and the student, and also used for a dialog pattern extraction function and a unique digital instructor generation function.

  On the display of the teacher PC 102, the video of the student is displayed together with the text content by the interactive communication function 102A between the teacher and the student, and further, the learning history of each student is displayed by the learning schedule management function 102G. That is, as shown in FIG. 8, the teacher screen 500 includes a student video display location 501 and a learning history display location 502 on the left side of the screen, a text display location 503 on the right side of the screen, an ON button 504, and an OFF button 505. There is a display area. The student's voice is also output. The teacher performs interactive learning guidance with the student by the learning support function 102E by the real instructor. Furthermore, the learning schedule management function 102G can know the learning history of each student or make a learning plan.

  As shown in FIG. 4, the student PC 103 includes a system program 103101 and a data transmission / reception program 103103 on a memory 1031. Also, a user input information collection / storage program 103102 that collects and stores information input by the student from the keyboard and mouse, a video input / storage program 103104 that stores the video acquired by the camera, and obtains audio by a microphone and further stores it. The voice input / storage program 103105, the attendance text display / editing program 103106 for the student to take the class, and the communication program 103107 for the conversation of the student are read into the memory. An example of the structure of the communication program 103107 is shown in FIG. The communication program includes a real instructor student communication program 1031071 and a digital instructor student communication program 1031072.

  On the display of the student's PC 103, the video of the teacher or digital instructor is displayed together with the text content, and further, the voice of the teacher and the data input by the teacher for explaining to the student are also displayed. That is, as shown in FIG. 9, the student screen 600 has a teacher video display location 601 and a teacher input data display location 602 on the left side of the screen, a text display location 603 on the right side of the screen, an ON button 604, and an OFF button 605. There is a display area.

  Further, as shown in FIG. 5, the system information 104201 and the video / audio storage program 104202 are read into the memory 1042 in the video information storage server 104. Also, the video data 104301 and audio / sound data 104302 that have been shot are stored in the hard disk 1043.

  Information such as the voice of the teacher and students and the data input by the teacher for explaining to the students is used for the dialogue pattern extraction function 101C and the unique digital instructor generation function 101D of the education information management server 101.

  Note that the functions of the servers and PCs, for example, the interactive communication functions 101A, 102A, and 103A, are all the same functions, and are given different symbols for the sake of convenience on the servers and PCs. Yes. Hereinafter, when there is no need to distinguish between them, the function corresponding to the education information management server 101 will be described.

  The student PC 103 supports learning support by the teacher and learning support by the digital instructor. As will be described later, when the teacher learns, the dialogue between the teacher and the student is displayed on the display of the student PC 103. The communication function 101A displays the teacher's video together with the text content, and the text display editing function 101B outputs the teacher's voice. On the other hand, at the time of learning support by the digital instructor, the image of the digital instructor is displayed together with the text content on the display of the student's PC 103, and the synthesized speech of the teacher is output. The initial digital instructor prepared in advance (hereinafter referred to as the original digital instructor) is not different among the students, but becomes a unique instructor unique to the student as the lesson progresses. Therefore, the digital instructor generated in this way is hereinafter referred to as a unique digital instructor.

  Next, the learning support processing in this system will be described. First, the learning support function processing by the real instructor will be described. In the learning support by the real instructor, the teacher and the learner advance the learning along a predetermined text while performing a dialogue using each PC between remote places.

  The student PC 103 supports learning support by a teacher and learning support by a digital instructor. At the time of learning support by the real instructor, in parallel with this learning support, a characteristic dialog pattern is extracted by the dialog pattern extraction function 101C of the education information management server 101, and a unique digital instructor is generated by the unique digital instructor generation function 101D. Generate. Using this unique digital instructor, a learning support function 101E by the digital instructor is generated.

  First, on the teacher PC 102, the communication program 102107 is started after the PC is started, and for example, a teacher screen 500 as shown in FIG. 8 is displayed. In the student PC 103, the communication program 103107 is activated after the PC is activated, and a student screen 600 as shown in FIG. 9 is displayed, for example.

  In the learning support by the real instructor, when the ON button 504 of the teacher PC screen 500 is pressed, the communication mode can be entered with the student who has requested communication by the data transmission / reception program. Here, when the ON button 604 of the screen 600 is pressed on the student PC 103, the data transmission / reception program enters a communication mode in which the teacher can interact with the teacher PC 102, and the student is displayed on the teacher video display location 601. Learning with the teacher is performed while browsing the text data for the students displayed in the image and text display location 603 and the data displayed in the teacher input data display location 602.

  Next, a process for recording and holding various data (event data, audio data, video data) at the time of learning support by the teacher will be described with reference to FIG.

  Here, a method for accumulating dialogue data between a teacher and a student will be described. First, the educational information management server 101, teacher PC 102, and student PC 103 are activated (S1001, S1002). At this point, each program installed in the teacher PC is activated. Education information is transmitted from the education information management server 101 to each PC (S1003). Further, the communication program of each PC is activated (S1003), and the education information is displayed as text on the screen of each PC (S1004). Here, the video data of the teacher is captured by the camera, and the data is transmitted to the education information management server by the user input information collection / storage program (S1007). At the same time, when the teacher's voice data is input by the microphone connected to the teacher PC, the data is transmitted to the education information management server 101 by the user input information collection / storage program.

  The voice data is A / D converted by a voice input / storage program and transmitted to the education information management server. The video data is A / D converted when captured by the camera and transmitted to the educational information management server by the data transmission / reception program. Data input from the keyboard or mouse by the teacher or students is also transmitted to the server as event code information. Further, the data of the input time and input end time of each data is also transmitted to the server. Each data transmitted to the server is stored in the user data table 700 (S1008 to S1014).

  For example, as shown in FIG. 6, these event data are stored in the user data table 700 in time series for each event such as a key event and a mouse input event, and the voice data is the voice rise time when the voice is input. Until the voice falling time when voice input is completed is stored in the user data table as a voice data input time zone. Since the video data is always in an input state, the video data is stored in the user data table 700 as an input time zone from when the camera is activated until the end of shooting. Even when the teacher ends the communication mode, the OFF button 505 on the right side of the screen is pressed.

  When in the communication mode, the student PC 103 records the voice data of the student from the connected microphone, and at the same time records the video data of the student by the camera by the user input information collection / storage program (S1007). At the same time, the student text display / editing program is started, and when the student inputs information to the displayed text using an input device such as a keyboard or a mouse, the text is displayed according to the input information. Edited. As with the teacher's voice data and video data, the voice data of the students is A / D converted by the voice input program when the voice data is sent to the server, and the video data is input to the camera. A / D converted and transmitted to the server by the data transmission / reception program (S1009, S1013). In addition, data input from the keyboard or mouse by the teacher or students is transmitted to the server as code information, and data of the input time and input end time of each data is also transmitted to the server.

  Each transmitted data is stored in time series in the user data table 700 which is a memory area of the server 101 for each event data, audio data, and video data as shown in FIG. S1010, S1011).

  When the student ends the communication mode, the user presses the OFF button 605 on the right side of the screen.

  When the OFF button is pressed on the teacher PC 102 and the student PC 103, the education information management server 101 assumes that the learning support by the real instructor has been completed and updates the student learning history (S1017). The process ends.

  On the other hand, during the learning support by the real instructor, the video data stored in the user data table 700 is stored in the frame data storage area as video frame data to which the imaging time is added. Furthermore, the voice / sound data is also stored in the voice / sound data storage area as data to which the voice rise time and fall time are added.

  Next, with reference to FIGS. 11 to 14, a description will be given of a process of extracting a characteristic dialogue pattern at the time of learning support by the teacher by the dialogue pattern extraction function 101C from the various data acquired at the time of learning support by the teacher. To do.

  The data recognition process will be described with reference to FIG. In order to perform this recognition processing, first, necessary data is read from the hard disk to the user data table 700 (S1101). At the same time, the voice recognition program 101203 and the video recognition program 101205 on the education information management server 101 are activated (S1102, 1103).

  Next, the stored voice / acoustic data is converted into text data by the voice recognition program 101203 and the voice recognition dictionary 101302 (S1104).

  Next, the rise time and fall time of the voice part corresponding to the converted text data are added to the text data, and for example, Speech [m] [n] (m = number of voice patterns) , N = 0: voice rise time, n = 1: voice fall time, n = 2: recognized text data).

  Next, the still time zone is detected from all the captured video data by the video recognition program and the video pattern recognition data. Next, the video portion sandwiched between the still time periods is cut out as an operation pattern, and stored as operation pattern data, for example, in the memory area as array data of Move [i] [j] (i = number of operation patterns, j = number of frames) . Next, for example, Move_t [i] [p] [t] (i = number of motion patterns, p = 0: video pattern start time, p = 1: video pattern end time) ) As array data.

  Furthermore, a pattern similar between the extracted motion patterns is detected. When a similar pattern is detected, for example, it is stored in an array format such as Move_a [v] [w] (v = number of similar patterns, w = action pattern number (i) with similar patterns) ( S1105).

  Further, regarding the voice pattern data and video pattern data of the teacher, when the voice pattern appears in the same time zone as the time when the video pattern appeared, as the related information between user media, for example, M_S_A [s] [ q] (s = number of related information, q = 1: audio pattern number, q = 2: video pattern number) (S1106 to S1108).

  For example, the student's motion pattern data (1) in the time T1 to T2 sandwiched between the stationary time zones in FIG. 14, the teacher's motion pattern data (1) and the time T5 in the time T3 to T4 sandwiched between the stationary time zones. When the motion pattern data (2) of the teacher in .about.T6 has previously appeared as similar motion patterns, they are extracted and stored as motion patterns that are similar patterns. In addition, when audio patterns similar to the time T1 to T2, the time T3 to T4, and the time T5 to T6 that appear, respectively, are recorded as the related information between user media.

  Next, the pattern extraction process will be described with reference to FIG. In order to perform this process, the communication pattern extraction program 101205 is activated (S1201). Here, the dialogue pattern between the teacher and the student is extracted using the voice recognition program, the voice data and video data recognized by the video recognition program, and the user event data stored in the user data table 700. .

  First, communication patterns between voice data are extracted. For the speech sequence data obtained from the teacher and the speech sequence data obtained from the students, the utterance time zones of both are mapped in time series. For example, if the Speech sequence data obtained from the teacher is T_Speech [m_t] [n_t] and the Speech sequence data obtained from the student is S_Speech [m_s] [n_s], T_Speech [m_t] [1] and S_Speech [m_s] The time of [1] is compared, and the pattern data with the earlier appearance time is mapped in time series. For example, Speech_t_a [s_1] [s_2] [s_3] (s_1 = total number of voice patterns, s_2 = 0 (teacher voice), s_2 = 1 (student voice), s_3 = 0 (start time), s_3 = 1 (End time)).

  Furthermore, for example, the silent time length between the voice pattern data of the student and the voice pattern data of the adjacent teacher is calculated, and the calculated silent time length is calculated in time series, for example, Blank_t [a] (a = blank Number)).

  For example, a period between times T2 and T3 between the two voice pattern data in FIG. 14 is a silent period, that is, a voice blank period until the teacher responds to the next speech.

  Next, a communication pattern between video data is extracted (S1202).

  For video information, as in the case of audio information, a teacher's motion pattern and a student's motion pattern are mapped in time series. For example, Move_t_a [a_1] [a_2] [a_3] (a_1 = total number of motion patterns, a_2 = 0 (operation by teacher), a_2 = 2 (operation by student), a_3 = 0 (start time), a_3 = 1 (End time)) is stored in an array format (S1203).

  Next, the communication status in the media is extracted based on the pattern extraction result performed above. Regarding Speech_t_a, when the values of s_2 of adjacent voice patterns are different and the same voice pattern is detected, for example, three times or more, the same pattern is regarded as a voice communication pattern, and C_S_P [c_1] [c_2] (c_1 = number of patterns, c_2 = 0 (teacher voice pattern), c_2 = 1 (student voice pattern)). As for the silent time length, only blank_t [a] is extracted only when the value of s_2 is different for the voice patterns corresponding to both ends of Blank_t. When s_2 is 0 to 1, s_2 is 1 to 0 In each case, the average value, the maximum value, and the minimum value of the blank are calculated. For example, B_0_1 [b_1] [b_2] [b_3] (b_1 = 0 (when s_2 is 0 to 1), b_1 = 1 (when s_2 is 1 to 0), b_2 = 0 (average value), b_2 = 1 (maximum value), b_2 = 2 (minimum value)) (S1204).

  Next, with respect to Move_t_a, when the values of a_2 of adjacent operation patterns are different and the same operation pattern is detected, for example, three or more times, the same pattern is regarded as an operation communication pattern. And stored in an array format such as C_A_P [c_1] [c_2] (c_1 = number of patterns, c_2 = 0 (teacher action pattern), c_2 = 1 (student action pattern)) (S1205).

  Next, the communication status between media is extracted based on the pattern extraction result performed above (S1206). In the case of between media, as in the case of the media, when the student's motion pattern and the teacher's voice pattern appear adjacent to each other, for example, when there are three or more times, this is communicated between the media. The pattern is stored in an array format such as In_M_P [p_1] [p_2] (P_1 = number of patterns, P_2 = number of media, P_3 = 0 (voice pattern), P_3 = 1 (operation pattern)) (S1207).

  Next, character generation for the unique digital instructor will be described with reference to FIG. First, the character generation character generation program 101206 is activated (S1301). For example, motion generation by a character is performed using a sign language animation generation system program or the like. Here, the motion pattern of the teacher extracted above is read (S1302), and is stored as motion basic motion data as motion feature quantities in the system (S1303). For example, the animation basic motion data generated for the motion pattern Move_t [0] [1] [1] is an array format as Anim [0] [1] [1] [a_4] (a_4: number of motion points). Described and stored.

  Next, the animation basic motion data is edited to be associated with the original digital instructor or the unique digital instructor created so far, and is generated and registered as a new unique digital instructor (S1304).

  For example, after the similar pattern of the voice and action pattern data (1) of the student at time T1 to T2 in FIG. 14, the voice data and action pattern data (1) of the teacher at the timing of time T3 to T4. When the similar pattern of (2) appears and is stored at a timing of approximately T5 to T6, the voice data of the student and the pattern data (1) of the movement are extracted, and then the movement pattern data of the teacher ( An animation similar to 1) and (2) is generated, and a voice corresponding to the voice of the teacher is synthesized.

  As described above, the motion image is analyzed to extract the motion pattern of the teacher or the student (user), the voice is analyzed to extract the utterance pattern of the user, and at least the extracted motion pattern and utterance pattern are extracted. Dialog patterns are extracted in association with each of the two sets in time series, and the similarity between the action pattern and the utterance pattern extracted from the same user as the voice or moving image acquired separately from the user is determined, and the similarity If it is high, the movement pattern of the other user of the extracted dialogue pattern is output as a moving image by the animation character, and the voice pattern of the other user of the extracted conversation pattern is output at the same time.

  The unique digital instructor can also be generated on the teacher's PC.

  When text, an image, or the like is input to the character generation character generation program 101206, the text is converted into speech corresponding to the selected dialog pattern for the already selected animation character, and the dialog pattern corresponding to the text is being displayed. The motion pattern is displayed by an animated character. In addition, by displaying images at the specified position, time, or time zone, for example, when a teacher creates his / her lecture content with a character that replicates himself / herself, the teacher does not need to record his / her voice, and once created It becomes possible to make corrections to the content that has been made.

  As an example, the student's motion pattern data and voice pattern data (1) asks "I don't know this meaning" while pointing the text display location on the screen 600 with a mouse. The pattern data and voice pattern data (1) and (2) of the teacher's action responding to the above are expressed as follows. And repeat. In this case, as the animation corresponding to the motion pattern of the teacher, for example, in addition to the animation of the original digital instructor as shown in FIG. 19A, an animation corresponding to the characteristic motion pattern as shown in FIG. 19B is generated. A synthesized voice “e --- e-to” linked to this animation is recorded as the related information between user media.

  The digital instructor generated in this way is generated by the relationship between the teacher and each student, so even for the same teacher, it is generated as a different digital instructor for each student and continues to grow while the lesson continues. . The digital instructor is updated each time the lesson is repeated. Therefore, the original digital instructor prepared in advance has no difference between the students, but becomes a unique digital instructor unique to the student as the lesson progresses. Such a unique digital instructor is generated by reflecting the individuality of the student, so that the interest of the student can be attracted and the effect of the lesson can be enhanced.

Next, referring to FIG. 15, a flow when the student learns using the digital instructor generated by the character generation program after the above operation and analysis are completed will be described. First, the student starts the student's PC (S1401). Next, when the student starts the communication program (S1402), for example, a Yes / No button is displayed on the student's PC screen with a text display such as “A digital instructor will be activated at the moment. It is displayed (S1403). When the student presses the Yes button, the user-digital instructor communication program and the character-user interaction control program, which are subprograms of the communication program, are activated (S1404), and the student screen 600 is displayed. On the student screen, a character is displayed at a teacher video display location 610 of the screen shown in FIG. 19A. The place where the user is learning while inputting information in the student text is captured by the camera of the student PC (S1405). The imaged data is transmitted to the server, and operation patterns are extracted in the same manner as in the above method. The extracted action pattern is searched in C_A_P [c_1] [c_2] (S1406). If the same action pattern is detected, a teacher action pattern associated with the same action pattern is extracted (S1407). . Further, the Anim data associated with the same action pattern is searched, and the character displayed on the screen is moved using the searched Anim data. At the same time, perform a search even In_M_P, when the same pattern is detected, generates a speech synthesis data by a corresponding voice pattern text using sound voice synthesis program, output from the speaker of the student PC (S1408~ S1410).

  Further, when the student utters a voice, the voice pattern is detected and the detected voice pattern is searched in C_S_P, as in the above voice processing. If text having the same voice pattern is detected, the teacher voice pattern associated with this pattern is extracted, and voice information is generated from the text information that is the same voice pattern by a voice synthesis program. And output from the speaker of the student PC. At the same time, a search is also performed on In_M_P. If the same pattern is detected, Anim data is searched using the corresponding operation pattern data and displayed on the student PC. The audio output timing is set according to the time length of the corresponding B_0_1 [b_1] [b_2] [b_3].

  FIG. 16 shows details of the process (S1409) for generating the speech synthesis data of FIG.

  First, a search is also performed for In_M_P, and if the same pattern is detected, speech synthesis data is generated by a speech synthesis program using the text of the corresponding speech pattern and output from the speaker of the student PC (S1501). . Next, the detected voice pattern is searched in C_S_P (S1502). Further, when text having the same voice pattern is detected, a teacher voice pattern associated with this pattern is extracted, and voice information is generated from the text information having the same voice pattern by a voice synthesis program. (S1503). Then, the speech synthesis data is output from the speaker of the student PC (S1504).

  Details of the process of generating speech synthesis data (S1410) are shown in FIG. First, a search is also performed on In_M_P. If the same pattern is detected, Anim data is searched using the corresponding operation pattern data and displayed on the student's PC (S1506). The audio output timing is set according to the time length of the corresponding B_0_1 [b_1] [b_2] [b_3] (S1507).

  For example, in FIG. 18, when voice information and video information input from a student are analyzed and a student's action and voice pattern data (1) at time T11 to T12 are extracted, a dialogue pattern corresponding to this is extracted. From the registered dialogue pattern data, the operation of the teacher at time T13 to T14, the voice pattern data (1), the voice pattern data of teacher at time T15 to T16 (2), and the user media between times T2 and T3 Inter-related information is extracted. Then, the motion pattern in the selected dialogue pattern data set is displayed on the character display portion 610 of the student's screen 600 as the motion of the target animation character, and the voice pattern in the dialogue pattern data set is further selected as the target. By displaying the voice of the animated character as a voice from the student's terminal, the synthesized voice can be output as the voice of the character based on the utterance pattern.

  As a result, the student can take an interest in the lesson in an atmosphere similar to a lesson by a real instructor. In addition, by projecting unconsciously repeated movements by teachers and students during lessons to animated characters, the conversation situation that is actually happening during regular lessons is reflected in the digital instructor, making the teacher more familiar, The student's willingness to learn can be expected to be maintained over a long period of time.

  In addition, by comparing the temporal dialogue patterns between the student and the animated character, the dialogue pattern actually desired by the student can be extracted and reflected in the lesson of the real instructor.

  In addition, after the pattern data (2) of the teacher's operation, a pattern (3A) in which the teacher “explains while entering an answer in the teacher data display location 602 of the screen 602” of the student, If it is assumed that there is a pattern (3B) that responds when the user answers, the pattern (3A) may be subjected to response processing by extracting a similar pattern by the method described above. On the other hand, with regard to the latter, for example, a “Q & A” function is added to a server or PC, for example, and the contents of the above-mentioned student's questions are sent to the teacher's PC. May be created and sent to the student's PC by e-mail separately, or may be answered directly during the lesson as the next real instructor.

  FIG. 20 is a diagram illustrating an example of a learning history of a plurality of students A, B, and-that a teacher has, obtained by the learning schedule management function. Immediately after the first real instructor lesson, each student learns using the same original digital instructor. As the lesson progresses, a unique digital instructor corresponding to each student is generated and learning is performed.

  As described above, according to the present invention, motion patterns are analyzed to extract a motion pattern of a set of users (a student and a teacher), the voice is analyzed to extract a user's utterance pattern, and extracted. The dialogue pattern is extracted in association with each of the time series of at least two sets of the motion pattern and the utterance pattern, and the motion pattern and the utterance pattern extracted from the same user as the voice or moving image separately acquired from the user, If the similarity is high, the action pattern of the other user (teacher) with respect to the extracted user (student) dialog pattern is output as a moving image by the animation character, and the extracted dialog Outputs the utterance pattern of the pattern partner user (teacher). As a result, for example, a student who is a user can learn while maintaining a conversational environment with a real teacher by using the motion information and audio information of the animation characters that are output, thereby improving the learning effect of the student. I can expect.

  According to the present invention, the temporal dialogue pattern between the student and the teacher can be extracted, and further, by comparing the temporal dialogue pattern between the student and the animation character, In fact, it will be possible to extract the desired dialogue pattern.

  In addition, by registering the motion pattern and the utterance pattern as synchronized data, that is, by registering these as related information between user media, the animation character is controlled based on the motion pattern, and the motion described as the pattern Furthermore, based on the utterance pattern, a synthesized voice can be output as the voice of the character.

  Also, it analyzes the audio information and video information input from the user, selects the dialogue pattern corresponding to the analyzed result from the registered dialogue pattern data, and selects the operation pattern in the selected dialogue pattern data set. The screen is displayed as the action of the target animation character, and the voice pattern in the dialogue pattern data set is voice-displayed as the voice of the target animation character, so that the voice of the character is synthesized based on the utterance pattern. Audio can be output.

  In addition, it is registered every time a dialogue is performed between users, the dialogue pattern is re-extracted at the time of registration, and the similarity and frequency of the dialogue pattern are further extracted. It is possible to extract dialogue patterns in both of them recently and to reproduce past dialogue patterns, and to change the dialogue pattern to be used according to the current learning situation. become.

The figure which shows an example of a structure of the apparatus in one Example of this invention. The figure which shows the functional block of the apparatus structure of FIG. 1A. The figure which shows an example of the structure in the education information management server in one Example of this invention. The figure which shows an example of a structure of PC for teachers in one Example of this invention. The figure which shows an example of a structure of PC for students in one Example of this invention. The figure which shows an example of a structure of the image | video information storage server in one Example of this invention. The figure which shows an example of the structure of the user data table which stores the event data input from a user, user audio | voice data, and user video data. The figure which shows an example of the structure of a communication program. The figure which shows an example of PC screen for teachers. The figure which shows an example of the student's PC screen. The figure which shows an example of the flow of the learning assistance process by the teacher in one Example of this invention. The figure which shows an example of the flow of the recognition process of the data in one Example of this invention. The figure which shows an example of the flow of the process of the pattern extraction in one Example of this invention. The figure which shows an example of the character production | generation processing flow for the unique digital instructor in one Example of this invention. Explanatory drawing of an operation | movement pattern data and audio | voice pattern data of a student and a teacher. The figure which shows an example of the flow in case a student learns using a digital instructor. The figure which shows the detail of the process (S1409) which produces | generates and outputs the speech synthesis data of FIG. The figure which shows the detail of the process (S1410) which produces | generates and outputs the speech synthesis data of FIG. The figure which shows an example of the relationship between a student's audio | voice pattern data and a teacher's synthetic | combination audio | voice when a student learns using a digital instructor. The figure which shows an example of the animation of an original digital instructor. The figure which shows an example of the animation corresponding to a teacher's characteristic motion pattern. It is a figure which shows the example of the learning log | history of the several student which one teacher takes charge obtained by the learning schedule management function.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Educational information management server, 102 ... Teacher PC, 103 ... Student PC, 104 ... Video information storage server, 105 ... Communication network, 10201 ... Speaker, 10202 ... Camera, 10203 ... Microphone, 101A ... Teacher and student 101B ... Text display editing function, 101C ... Dialog pattern extraction function, 101D ... Unique digital instructor generation function, 101E ... Learning support function by teacher (real instructor), 101F ... Learning support function by digital instructor, 101G Learning schedule management function, 1011 ... CPU, 1012 ... memory, 1013 ... hard disk, 101301 ... operation pattern data, 101302 ... speech recognition dictionary, 101303 ... video pattern recognition data, 1 1304 ... Student text data, 101,305 ... problem correct data, 103,107 ... communication program 500 ... Teacher screen, 600 ... Student screen, 610 ... character display location, 700 ... user data table.

Claims (5)

Education for providing a connection unit connected to the student side terminal and the teacher side terminal, and for assisting the student in interactive learning with the teacher or a digital instructor corresponding to the teacher using the student side terminal An information management server,
A storage unit;
Dialog pattern generation means for generating dialog pattern data in dialog learning between the student and the teacher from the student information and teacher information acquired from the student side terminal and the teacher side terminal,
Registration means for registering the dialogue pattern data in the storage unit in time series;
A dialogue pattern extraction function for extracting a characteristic dialogue pattern from the dialogue patterns registered in the storage unit and registering it in the storage unit;
Of the characteristic dialogue patterns registered in the storage unit, a specific dialogue pattern of the teacher for the specific dialogue pattern of the student is generated as a digital instructor corresponding to the teacher by an animation character and synthesized speech Educational information management server characterized by having a unique digital instructor generation function for The dialog pattern extraction function according to claim 1,
A function of performing motion pattern extraction and utterance pattern extraction by analyzing moving images and voices of the teacher and the student at the time of learning acquired from the teacher-side terminal and the student-side terminal, and the extracted operation A dialogue pattern is extracted in association with each of a time series for at least two sets of a pattern and a utterance pattern, and voices and moving images separately acquired from the teacher-side terminal and the student-side terminal, and the extracted action pattern, A dialogue pattern extraction function having a function of determining the similarity to the utterance pattern and a function of registering the characteristic dialogue pattern in the storage unit when the similarity is equal to or greater than a predetermined value. Educational information management server characterized by In claim 1,
The student information and the teacher information include utterance content, utterance input time and utterance input end time,
The dialog pattern extraction function is:
When the student information and the teacher information registered in the storage unit are referred to, and it is determined that the teacher's utterance has been performed after the student's utterance, the utterance content of the teacher is used as the utterance content of the student. And as a dialogue sub-pattern, and record the silent time length from the student's utterance to the teacher's utterance in the storage unit as the sub-pattern dialogue time data of the dialogue sub-pattern,
In the unique digital instructor generation function,
An education information management server, wherein the animation character is made to respond to the student by the teacher based on the inter-pattern time data. The dialog pattern extraction function according to claim 1,
The degree of synchronization between the data extracted as the action pattern in the dialog pattern and the data extracted as the utterance pattern is determined, and when it is determined that both data are synchronized, the action pattern and the utterance An educational information management server, wherein patterns are registered in the storage unit as synchronized data. 2. The educational information management server according to claim 1, wherein the storage unit is a user data table for recording event data, audio data, and video data in time series.

Images