JP2007328806A - Display control device and display control processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a voice display output control device for synchronously outputting data such as a voice, a text or an image wherein voice output, text display and pronunciation image display are easily synchronously output when synchronously outputting the text data, the voice data and the image data. <P>SOLUTION: When synchronous reproduction output of the text, the voice, and the image constituted by a plurality of conversation parts is performed, a mouth image corresponding to the read character is displayed on a mouth position of a plurality of displayed human images M1, M2 and M3 in accordance with text display for each of parts such as conversations as a reproduction target, and a user can surely know which human image reads and which character part is read, and can effectively practice the conversation. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a voice display output control device, an editing device, a voice text playback device, a voice display output control processing program, an editing processing program, and a voice text playback processing program for synchronously outputting data such as voice, text, and images. .

  Conventionally, as a technique for reproducing music, text, images, and the like simultaneously in parallel, for example, for each frame of an audio file information-compressed by MPEG-3, audio is added to an additional data area provided in each frame. By embedding synchronization information of a text file or an image file to be synchronized and reproduced in a file, for example, in the case of karaoke, there is one that synchronizes and reproduces a karaoke voice and its image image and lyrics text.

  In addition, on the assumption that temporal correspondence information of characters with respect to speech is prepared in advance, a device that extracts the feature amount of the speech signal and displays it in association with the corresponding character is also considered. (For example, refer to Patent Document 1.)

Japanese Patent Publication No. 06-025905

  However, in the conventional synchronized reproduction technology of a plurality of types of files using the additional data area of the MPEG file as described above, the synchronization data is mainly embedded in the additional data area for each frame of the MP3 audio file. Therefore, unless the MP3 audio file is reproduced, the synchronization information cannot be extracted, and other types of files can be synchronized and reproduced only with the reproduction of the MP3 file as an axis.

  For this reason, for example, when the synchronization information of a text file and an image file is embedded in an MP3 audio file, the audio reproduction process is not continuously performed as silent data even during a period in which the audio file is not reproduced. There is a problem that it is not possible to reproduce the synchronization target text and image.

  For example, when you practice listening or reading with English conversation text, it is better to display an image to clearly indicate what pronunciation should be done, along with the display of the text and the output of the text to be read out. desirable. However, when creating an English conversation text using such an MP3 audio file, it is necessary not only to embed corresponding text in English text and image synchronization information for pronunciation display for each frame of the MP3 audio file. Even if a silent period is set for reading practice, as described above, text and images cannot be displayed during the silent period unless the voice reproduction process is continued.

  Further, even in the conventional device described in Patent Document 1, since the characters corresponding to the audio signal are associated in advance and only the voice and characters are reproduced synchronously, the audio output and the characters are synchronously displayed. In addition, the corresponding image cannot be displayed synchronously.

  On the other hand, some karaoke monitors display character strings and background images in synchronization with audio output. However, as in the MP3 file described above, the sound, character strings, and images are separately combined in advance. A file for synchronized playback must be created.

  The present invention has been made in view of the above-described problems. When synchronous output of text data, audio data, and image data, audio output, text display, and pronunciation image display are easily output in synchronization. An object is to provide a voice display output control device, an editing device, a voice text reproduction device, a voice display output control processing program, an editing processing program, and a voice text reproduction processing program.

  That is, in the voice display output control device according to the first aspect of the present invention, the text data is displayed by the text synchronization display control means so as to be synchronized with the voice data output by the voice data output means, and is synchronized with the voice data. Thus, an image expressing mouth movement is displayed by the image display control means corresponding to the synchronous display portion of the displayed text.

  According to this, not only the text display corresponding to the voice output is synchronized, but also the expression image of the mouth movement corresponding to the voice and the text can be displayed.

  A voice display output control device according to a second aspect of the present invention is the voice display output control device according to the first aspect, wherein the image display control means has different mouths depending on each part of the text data. Image storage means for storing a plurality of images expressing the movement of the image, and an image corresponding to a synchronous display portion of the text displayed so as to be synchronized with audio data by the text synchronous display control means is provided by the image storage means. It is read out from a plurality of stored images and displayed.

  According to this, it is possible to display an expression image of mouth movement most corresponding to voice output and text display from a plurality of images expressing mouth movements that differ depending on each part of the text data.

  An audio display output control apparatus according to claim 3 of the present invention is the audio display output control apparatus according to claim 1, wherein the image display control means includes an image display means for displaying an image, Mouth image storage means for storing a plurality of mouth images expressing different mouth movements according to pronunciation, and pronunciation of a synchronous display portion of text displayed to be synchronized with voice data by a text synchronization display control means The mouth image corresponding to is read from the plurality of mouth images stored by the mouth image storage means by the mouth image display control means and displayed at the mouth position of the image displayed by the image display means.

  According to this, the expression image of the mouth movement most corresponding to the sound output and the pronunciation of the text display portion is displayed from among a plurality of images expressing the mouth movements different according to the pronunciation of each part of the text data. Can be displayed at the mouth position of the image.

  The voice display output control device according to claim 4 of the present invention is the voice display output control device according to claim 3, wherein the mouth image storage means has different mouth movements according to various pronunciations. And an internal image of the mouth is stored in association with each of the plurality of mouth images. Corresponding to the mouth image displayed by the mouth image display control means, the inside image of the mouth is read and enlarged and displayed by the mouth enlargement display means.

  According to this, the expression image of the mouth movement most corresponding to the sound output and the pronunciation of the text display portion is displayed from among a plurality of images expressing the mouth movements different according to the pronunciation of each part of the text data. In addition to being displayed at the position of the mouth of the image, the internal image of the mouth corresponding to the pronunciation can be clearly displayed.

  A voice display output control apparatus according to claim 5 of the present invention is the voice display output control apparatus according to any one of claims 1 to 4, wherein the text synchronous display control means includes: Instruction code storage means for storing an instruction code for causing the text data to be displayed in synchronization with the voice data output by the voice data output means in accordance with a preset elapsed time is provided. In accordance with the stored instruction code, the text data is displayed so as to be synchronized with the voice data. The image display control means displays an image corresponding to voice and text according to the instruction code stored by the instruction code storage means.

  According to this, the text data and the image corresponding to the text can be displayed in synchronization with the audio data in accordance with the instruction according to the set elapsed time by the instruction code stored by the instruction code storage means.

  In the audio display output control processing program according to claim 6 of the present invention, the program is installed in the computer of the electronic device so that the computer of the electronic device synchronizes with the audio data output by the audio data output means. Thus, text data is displayed by the text synchronous display control means, and an image expressing mouth movement is displayed by the image display control means corresponding to the synchronous display portion of the text displayed so as to be synchronized with the voice data. The

  As a result, in the electronic device, not only the text display corresponding to the voice output is synchronized, but also the expression image of the mouth movement corresponding to the voice and the text can be displayed.

  An editing apparatus according to claim 7 of the present invention is an editing apparatus that edits correspondence data for outputting a text and an image in synchronization with voice, and is associated with various audio data, and is different from each other. Mouth image storage means for storing the mouth image is stored, and the corresponding data editing means reads the mouth image stored in the mouth image storage means in association with the voice data corresponding to the text to be output. And stored in association with the position of the text or audio data to be output.

  According to this, as the corresponding data for outputting the text and the image in synchronization with the voice, the mouth image corresponding to the voice data corresponding to the output target text is placed at the position of the output target text or voice data. It can be stored and edited in association with each other.

  An audio text reproduction apparatus according to claim 8 of the present invention is an audio text reproduction apparatus provided with the editing apparatus according to claim 7, wherein the text is synchronized with the audio and output by the synchronous reproduction means. At the same time, based on the correspondence data stored by the correspondence data editing means, the mouth image associated with the text to be output is read and displayed.

  According to this, it is possible to easily display the mouth image corresponding to the synchronized output of the voice and the text based on the edited correspondence data.

  The text reproduction apparatus according to claim 9 of the present invention further includes a dictionary storage unit that stores a plurality of headwords and image data of the mouth in association with the headwords. When the headword stored in the dictionary storage means is searched, the mouth image data stored in the dictionary storage means in association with the headword is displayed together with the searched headword. The characters of the headword and the image data of the mouth are sequentially displayed in synchronization with each other.

  According to this, it is possible to synchronously display mouth image data such as a pronunciation mouth type corresponding to the headword together with the headword searched with the dictionary search.

  The speech text reproduction apparatus according to claim 10 of the present invention further includes dictionary storage means for storing voice data and mouth image data in association with a plurality of headwords and the headwords. When the word search means searches for the headword stored in the dictionary storage means, the voice data stored in the dictionary storage means in association with the searched headword is output by the voice output means. Further, the mouth image data associated with the headword and stored by the dictionary storage means is read by the display control means, and the headword and mouth image data are sequentially sequentially synchronized with the voice output. Displayed synchronously.

  According to this, it is possible to display the headword searched in accordance with the dictionary search, and the voice synchronized output corresponding to the headword and the mouth image synchronous display such as the pronunciation mouth type.

  In the editing processing program according to the eleventh aspect of the present invention, by installing the program in the computer of the electronic device, the computer of the electronic device has a plurality of different mouth images associated with various audio data. Mouth image storage means is stored, and the corresponding data editing means reads the mouth image stored in the mouth image storage means in association with the voice data corresponding to the text to be output, and the output It is stored in association with the position of the target text or audio data.

  Accordingly, in the electronic device, as the corresponding data for outputting the text and the image in synchronization with the voice, the mouth image corresponding to the voice data corresponding to the text to be output is converted to the position of the text or the voice data to be output. It is possible to store and edit in association with each other.

  A speech text reproduction processing program according to claim 12 of the present invention is a speech text reproduction processing program including the editing processing program according to claim 11, and the program is installed in a computer of an electronic device. In the computer of the electronic device, the synchronized reproduction means outputs the text in synchronization with the voice, and the mouth image associated with the output target text based on the correspondence data stored by the correspondence data editing means. Is read and displayed.

  Thus, the electronic device can easily display the mouth image corresponding to the synchronized output of the voice and the text based on the edited correspondence data.

  In the voice display output control device according to the thirteenth aspect of the present invention, the first storage means that stores a plurality of sets of text phonetic symbols and mouth-type images in association with each other, and the characters and voices are associated with each other. A plurality of sets of second storage means, and the first control means sequentially outputs the sound stored in the second storage means to the outside and outputs the sound to the output sound. Synchronously, the characters stored in association with the second storage means are sequentially identified and displayed. Then, in synchronization with the characters sequentially identified and displayed by the control of the first control means, the second control means is associated with the first storage means in correspondence with the phonetic symbols of the characters to be identified and displayed. The stored mouth shape image is read out, and the read mouth shape images are sequentially switched and displayed.

  According to this, it is possible to sequentially identify and display the characters stored correspondingly in synchronism with the sequential output of the voice, and further, to synchronize with the sequentially identified and displayed characters, the phonetic symbols of the identification display characters It is possible to sequentially switch and display the mouth type images corresponding to.

  As described above, according to the voice display output control device of the first aspect of the present invention, the text data is displayed by the text synchronous display control means so as to be synchronized with the voice data output by the voice data output means. An image representing the movement of the mouth is displayed by the image display control means corresponding to the synchronous display portion of the text displayed so as to be synchronized with the voice data, so that the text display corresponding to the voice output is synchronized. In addition to this, it is possible to display an expression image of mouth movement corresponding to the voice and text.

  According to the voice display output control device according to claim 2 of the present invention, in the voice display output control device according to claim 1, the image display control means is provided according to each part of the text data. Image storage means for storing a plurality of images representing different mouth movements is provided, and an image corresponding to a synchronous display portion of text displayed to be synchronized with audio data by the text synchronous display control means is stored in the image storage Since it is read out and displayed from multiple images stored by the means, it is most compatible with audio output and text display from among multiple images that express different mouth movements according to each part of the text data An expression image of mouth movement can be displayed.

  According to the voice display output control device according to claim 3 of the present invention, the voice display output control device according to claim 1, wherein the image display control means is an image display means for displaying an image; Mouth image storage means for storing a plurality of mouth images expressing different mouth movements according to various pronunciations, and a synchronous display portion of text displayed to be synchronized with voice data by a text synchronization display control means The mouth image corresponding to the pronunciation of the mouth is read from the plurality of mouth images stored by the mouth image storage means by the mouth image display control means and displayed at the mouth position of the image displayed by the image display means. Therefore, from the multiple images that represent different mouth movements according to the pronunciation of each part of the text data, the mouth movement expression image that best corresponds to the sound output and text display part pronunciation , It becomes possible to display the position of the mouth of the image displayed.

  According to the voice display output control device according to claim 4 of the present invention, in the voice display output control device according to claim 3, the mouth image storage means has different mouths according to various pronunciations. In addition to a plurality of mouth images representing the movement of the mouth, an internal image of the mouth is stored in association with each of the plurality of mouth images. Then, in response to the mouth image displayed by the mouth image display control means, the inner image of the mouth is read and enlarged and displayed by the mouth enlargement display means, so that it varies depending on the pronunciation of each part of the text data. Out of multiple images that represent mouth movements, not only can you display the mouth movement expression image that best corresponds to the sound output and text display pronunciation at the mouth position of the displayed image, but also its pronunciation The internal image of the mouth corresponding to can be clearly displayed.

  According to a voice display output control device according to claim 5 of the present invention, in the voice display output control device according to any one of claims 1 to 4, in the text synchronous display control means, Comprises an instruction code storage means for storing an instruction code for displaying the text data in synchronization with the voice data output by the voice data output means in accordance with a preset elapsed time. The text data is displayed in synchronization with the voice data in accordance with the instruction code stored by the means. The image display control means displays an image corresponding to the voice and text in accordance with the instruction code stored in the instruction code storage means, so that the setting process by the instruction code stored in the instruction code storage means is performed. According to the instruction according to time, the text data and the image corresponding to the text can be displayed in synchronization with the audio data.

  According to the voice display output control processing program of claim 6 of the present invention, the voice data output by the voice data output means in the computer of the electronic device by installing the program in the computer of the electronic device. Text data is displayed by the text synchronous display control means so as to be synchronized with the image, and an image expressing mouth movements by the image display control means corresponding to the synchronous display portion of the text displayed so as to be synchronized with the audio data is displayed. Therefore, in this electronic device, not only the text display corresponding to the voice output is synchronized, but also the expression image of the mouth movement corresponding to the voice and the text can be displayed.

  The editing device according to claim 7 of the present invention is an editing device that edits correspondence data for outputting a text and an image in synchronism with speech, and is different in correspondence with various speech data. Mouth image storage means for storing a plurality of mouth images, and the corresponding data editing means reads the mouth image stored in the mouth image storage means in association with voice data corresponding to the text to be output. Since it is stored in association with the position of the text or audio data to be output, it corresponds to the audio data corresponding to the text to be output as the corresponding data for outputting the text and the image in synchronization with the audio The mouth image to be stored can be stored and edited in association with the position of the text or audio data to be output.

  According to an eighth aspect of the present invention, there is provided an audio text reproduction device comprising the editing device according to the seventh aspect, wherein the text is synchronized with the audio and output by the synchronous reproduction means. At the same time, based on the correspondence data stored by the correspondence data editing means, the mouth image associated with the text to be output is read and displayed. The mouth image corresponding to the synchronous output of voice and text can be easily displayed.

  The text reproduction apparatus according to claim 9 of the present invention further includes dictionary storage means for storing mouth image data in association with a plurality of headwords and the headwords. When the headword stored in the dictionary storage means is retrieved by the means, the mouth image data stored in the dictionary storage means in association with the headword is displayed together with the retrieved headword. Since each character of the headword and the image data of the mouth are sequentially and synchronously read out by the control means, the pronunciation mouth type corresponding to the headword and the headword searched along with the dictionary search Mouth image data can be displayed synchronously.

  In addition, according to the speech text reproduction apparatus according to claim 10 of the present invention, a dictionary storage means for storing a plurality of headwords and voice data and mouth image data in association with each headword is provided, When the headword stored in the dictionary storage means is searched by the headword search means, the voice data stored in the dictionary storage means in association with the searched headword is output by the voice output means. The mouth image data output and stored in the dictionary storage means in association with the headword is read out by the display control means, and the headword and mouth image data are synchronized with the voice output. Are sequentially displayed in synchronization with each other, so that it is possible to display the headword searched in accordance with the dictionary search, and the voice synchronized output corresponding to the headword and the mouth image synchronous display such as the pronunciation mouth type.

  According to the editing processing program of the eleventh aspect of the present invention, by installing the program in the computer of the electronic device, the computer of the electronic device has a plurality of different mouths associated with various audio data. The mouth image storage means for storing the image of the mouth, and the corresponding data editing means reads the mouth image stored in the mouth image storage means in association with the voice data corresponding to the text to be output, Since it is stored in association with the position of the text or audio data to be output, in this electronic device, the audio data corresponding to the text to be output is used as the corresponding data for outputting the text and the image in synchronization with the audio. The mouth image corresponding to is stored in correspondence with the position of the text or audio data to be output and edited. It becomes so that.

  A speech text reproduction processing program according to claim 12 of the present invention is a speech text reproduction processing program including the editing processing program according to claim 11, and the program is installed in a computer of an electronic device. In the computer of the electronic device, the synchronized reproduction means outputs the text in synchronization with the voice, and the mouth image associated with the output target text based on the correspondence data stored by the correspondence data editing means. Is read and displayed. As a result, the electronic device can easily display the mouth image corresponding to the synchronized output of the voice and the text based on the edited correspondence data.

  According to the voice display output control apparatus of the thirteenth aspect of the present invention, the first storage means for storing a plurality of sets of text phonetic symbols and mouth-shaped images in association with each other, and the characters and voices. And a second storage means for storing a plurality of sets in association with each other, and the sound stored in the second storage means is sequentially outputted to the outside by the first control means and outputted. The characters stored in association with the second storage means are sequentially identified and displayed in synchronization with the voice. Then, in synchronization with the characters sequentially identified and displayed by the control of the first control means, the second control means is associated with the first storage means in correspondence with the phonetic symbols of the characters to be identified and displayed. The stored mouth-shaped images are read out, and the read mouth-shaped images are sequentially switched and displayed, so that the characters stored correspondingly are sequentially identified and displayed in synchronization with the sequential output of voice. Furthermore, in synchronization with the characters that are sequentially identified and displayed, the mouth-shaped images corresponding to the phonetic symbols of the identified and displayed characters can be sequentially switched and displayed.

  Therefore, according to the present invention, a voice display output control device that can easily synchronize and output voice output, text display, and image display for pronunciation at the time of synchronous output of text data, voice data, and image data. An editing device, a voice text reproduction device, a voice display output control processing program, an editing processing program, and a voice text reproduction processing program can be provided.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of an electronic circuit of a portable device 10 according to an embodiment of a voice display output control device (editing device) (voice text reproduction device) of the present invention.

  The portable device (PDA: personal digital assistants) 10 is configured by a computer that reads a program recorded on various recording media or a program transmitted by communication and whose operation is controlled by the read program. The electronic circuit includes a CPU (central processing unit) 11.

  The CPU 11 is a PDA control program stored in advance in the ROM 12A in the memory 12, or a PDA control program read into the memory 12 from the external recording medium 13 such as a ROM card via the recording medium reading unit 14, or the Internet. The operation of each part of the circuit is controlled according to the PDA control program read into the memory 12 from the other computer terminal (30) on the communication network N via the power transmission control unit 15, and is stored in the memory 12. The PDA control program is an input signal in response to a user operation from the input unit 17a composed of switches and keys and the coordinate input device 17b composed of a mouse and a tablet, or another communication network N received by the transmission control unit 15. There is a communication signal from the computer terminal (30). The external communication device is received via the communication unit 16 by the short-range wireless connection or a wired connection using Bluetooth (R): is activated in response to the communication signal from the (PC personal computer) 20.

  The CPU 11 is connected to the memory 12, the recording medium reading unit 14, the power transmission control unit 15, the communication unit 16, the input unit 17a, and the coordinate input device 17b. Are connected to a sound input unit 19a for inputting a sound, a stereo sound output unit 19b for outputting sound with left and right channel speakers L and R, and the like.

  The CPU 11 has a built-in timer for processing time counting.

  The memory 12 of the portable device 10 includes a ROM 12A, a FLASH memory (EEP-ROM) 12B, and a RAM 12C.

  The ROM 12A includes a system program that controls the overall operation of the mobile device 10 and a network communication program for data communication with each computer terminal (30) on the communication network N via the transmission control unit 15 and a communication unit 16. In addition to storing an external device communication program for data communication with an external communication device (PC) 20, a schedule management program, an address management program, and various files such as voice, text, and images are synchronously reproduced. Various PDA control programs such as the reproduction processing program 12a1 are stored.

  Further, the ROM 12A further stores dictionary data 12a2, mouthpiece data 12a3 (see FIG. 2), and headword voice data 12a4. As the dictionary data 12a2s, data of various dictionaries such as an English-Japanese dictionary, a Japanese-English dictionary, and a national language dictionary are stored.

  FIG. 2 is a diagram showing the contents of the mouthpiece data 12a3 stored in the ROM 12A of the portable device 10.

  As the mouth shape data 12a3, for example, the mouth shape image viewed from the front direction in association with each English phonetic symbol and the mouth shape number, and the mouth from the lateral direction inside the mouth expanded. A mold section image and a small explanation (comment) data thereof are stored.

  The headword audio data 12a4 will be described in a second embodiment described later.

  The FLASH memory (EEP-ROM) 12B stores an encrypted reproduction file (CAS file) 12b to be subjected to reproduction processing based on the reproduction processing program 12a1, as well as the schedule management program and address management. A user's schedule managed based on the program and addresses of friends / acquaintances are stored.

  Here, the encrypted playback file 12b stored in the FLASH memory (EEP-ROM) 12B is a file for performing, for example, practice of English and singing by synchronized playback of text, sound, and images. It is compressed and encrypted by an algorithm.

  The encrypted reproduction file 12b is recorded and distributed on a CD-ROM, for example, or distributed and distributed from a file distribution server 30 on a communication network (Internet) N. 30) is read into the communication device (PC) 20 as the user's home PC, for example, and then the FLASH memory (EEP-) is transmitted via the communication unit 16 of the portable device (PDA) 10. ROM) 12B.

  The RAM 12C stores a decrypted playback file (CAS file) 12c obtained by decompressing and decrypting the encrypted playback file 12b, and an image file in the decrypted playback file 12c is expanded. A stored image expansion buffer 12k is provided. The decrypted CAS file 12c is composed of header information (12c1) for storing a processing unit time (12c1a) of a reproduction command, a file sequence table (12c2), a time code file (12c3), and content content data (12c4) described later. Composed. The RAM 12C stores an image developed flag 12j indicating the developed image number of each image file developed and stored in the image development buffer 12k.

  The RAM 12C also stores the designated text number 12d in the selected content of the reproduction file 12b (12c), the designated part number 12e designated by the user for the English conversation text corresponding to the designated text number, and the designated text number. Text-corresponding pronunciation type data 12f (see FIG. 7) in which all pronunciation symbols and mouth type numbers of each English word of the corresponding English conversation text are retrieved from the dictionary data 12a2 and mouth type data 12a3 and stored, and the reproduction Text highlight that is detected and stored in real time as the current highlight identification character when each character of the English conversation text is synchronized with the reading voice and the reading animation image and is highlighted and displayed based on the file 12b (12c) The designated character 12g and the playback file 1 2b (12c) includes an image (on / off) flag 12h for setting whether to perform synchronous playback of an image file included as a synchronized playback file, and is included as a synchronized playback file in the playback file 12b (12c). A sound (on / off) flag 12i for setting whether or not the sound file is synchronously reproduced is stored.

  Further, the RAM 12C is provided with a work area for temporarily storing various data input / output to / from the CPU 11 according to various other processes.

  FIG. 3 is a diagram showing a time code file 12c3 constituting the reproduction file 12b (12c) stored in the memory 12 of the portable device 10. As shown in FIG.

  FIG. 4 is a view showing a file sequence table 12c2 constituting the reproduction file 12b (12c) stored in the memory 12 of the portable device 10. As shown in FIG.

  FIG. 5 is a diagram showing content content data 12c4 constituting the reproduction file 12b (12c) stored in the memory 12 of the portable device 10. As shown in FIG.

  The reproduction file 12b (12c), which is the reproduction target file of the portable device 10, is composed of a combination of a time code file 12c3, a file sequence table 12c2, and content content data 12c4, as shown in FIGS. .

  The time code file 12c3 shown in FIG. 3 includes a time code for performing various file synchronous playback command processing at an original (reference) processing unit time (for example, 25 ms) at a predetermined time interval preset for each file. Each time code is a reference number of the file sequence table 12c2 (FIG. 4) for associating the command code indicating the instruction with the file contents related to the command (see FIG. 5). And parameter data consisting of specified numerical values.

  Note that the original (reference) processing unit time 12c1a at a fixed time interval for sequentially executing command processing according to the time code is described and set in the header information 12c1 of the time code file 12c3.

  For example, the file playback time of the playback file 12b (12c) consisting of the time code file 12c3 shown in FIG. 3 is a playback consisting of a time code of 2400 steps when the original (reference) processing unit time set in advance is 25 ms. After processing, it becomes 60 seconds.

  The file sequence table 12c2 shown in FIG. 4 includes parameter data and actual data of each command described in the time code file 12c3 (see FIG. 3) for each type of a plurality of types of files (HTML / image / text / sound). Is a table in which the file content storage destination (ID) numbers are associated with each other.

  In the content content data 12c4 shown in FIG. 5, the file data such as actual voice, image, and text associated with each command code by the file sequence table 12c2 (see FIG. 4) corresponds to the respective ID numbers. It is memorized.

  The content content data 12c4 linked in the file sequence table 12c2 of this embodiment will be briefly described. For example, the data content corresponding to ID = 5 has a basic image G0 of English conversation text by three conversation parts (FIG. 15). (Refer to FIG. 12 (A) and (D)) is prepared for the data content corresponding to ID = 6, and the part 1 conversation image G1 (see FIGS. 12A and 12D) based on the basic image G0 is prepared. Corresponding data contents include a conversation image G2 of part 2 based on the basic image G0 (see FIG. 12B), and the data contents corresponding to ID = 8 are based on the basic image G0. Then, a conversation image G3 (see FIG. 12C) of Part 3 is prepared.

  Note that the English conversation text basic image G0 (see FIG. 15) corresponding to ID = 5 of the content content data 12c4 has the mouth area coordinate data x1y1, x2y2 (see M1 to M3 in FIG. 15) for each part person image. ) Are stored in association with each other.

  For example, the data content corresponding to ID = 21 includes English conversation text data (2) (see FIGS. 12 and 15) to which the conversation images G1 to G3 of the English conversation text by the three conversation parts are synchronized. Further, for example, in the data content corresponding to ID = 32, the English conversation voice data (2) to which the conversational images G1 to G3 of the English conversation text by the three conversation parts and the English conversation text data (2) are synchronized. (See 19b in FIGS. 12 and 15).

  FIG. 6 is a diagram showing the command codes of various commands described in the time code file 12c3 (see FIG. 3) of the mobile device 10 in association with the command data to be analyzed based on the parameter data and the reproduction processing program 12a1. It is.

  Commands used for the time code file 12c3 include standard commands and extended commands. The standard commands include LT (i-th text load). VD (i-th text phrase display). BL (Character counter reset / i-th phrase block designation). HN (no highlight, character counter count up). HL (up to i-th character, character count). LS (1 line scrolling / character counter count up). DH (i-th HTML file display). DI (i-th image file display). PS (i-th sound file play). CS (Clear All File). PP (pause for basic time i seconds). FN (end of processing). There are NP (invalid) commands, and extended commands include PT (i-th part designation). There are various commands such as PI (image file display for each part).

  FIG. 7 is a diagram showing the contents of the text corresponding mouthpiece type data 12f stored in the RAM 12C of the portable device 10.

  That is, the reproduction processing program 12a1 stored in the ROM 12A of the portable device (PDA) 10 is activated, and the decryption / reproduction file 12c decrypted from the FLASH memory 12B and stored in the RAM 12C is shown in FIGS. 3 to 5, for example. If the third command code “DI” and parameter data “00” are read during command processing for each set processing unit time, the command “DI” is the i-th image file. Since this is a display command, the English conversation text basic image G0 of the content content data 12c4 (see FIG. 5) is determined according to the ID number = 5 of the image file linked to the file sequence table 12c2 (see FIG. 4) from the parameter data i = 00. It is read and displayed.

  When the sixth command code “VD” and parameter data “00” are read in accordance with the command processing for each set processing unit time, this command “VD” is the i-th text phrase display command. According to the parameter data i = 00, the 0th clause of the text is displayed.

  When the eighth command code “PI” and parameter data “00” are read in accordance with the command processing for each set processing unit time, this command “PI” is an image file display command for each part. The image display of the designated part is instructed according to the subsequent command code PT (i-th part designation).

  When the ninth command code “PT” and parameter data “01” are read in accordance with the command processing for each set processing unit time, the command “PT” is the i-th part designation command, so the parameter According to the data i = 01, the first part in the reproduction target file is designated.

  Further, when the eleventh command code “NP” and parameter data “00” are read in accordance with the command processing for each set processing unit time, the command “NP” is an invalid instruction, and therefore the current file output State is maintained.

  The detailed reproduction operation for the reproduction file 12b (12c) having the file contents shown in FIGS. 3 to 5 will be described later.

  Next, the file reproduction function by the portable device 10 having the above-described configuration will be described.

  FIG. 8 is a flowchart showing a reproduction process according to the reproduction process program 12a1 of the portable device 10.

  FIG. 9 is a flowchart showing the sound output type data creation process accompanying the reproduction process of the portable device 10.

  FIG. 10 is a flowchart showing image display processing A for each corresponding part accompanying the playback processing of the portable device 10.

  FIG. 11 is a diagram showing a learning content selection operation / display state associated with the reproduction processing of the mobile device 10, where FIG. 11A shows a learning content selection screen G, and FIGS. It is a figure which shows the selection operation key which makes the said learning content selection screen G object.

  For example, an English teaching material playback file in which English study can be performed with text, images, and voices is imported from a server 30 on a CD-ROM or communication network (Internet) N to a communication device (PC) 20 that is a home PC, and a portable device (PDA) ) In a state where the reproduction file (CAS file) 12b is stored in the FLASH memory 12B or the decoded reproduction file 12c in the RAM 12C via the communication unit 16, the input unit 17a or the coordinate input device (mouse) When the reproduction of the reproduction file 12b (12c) is instructed by the operation of the / tablet 17b, first, as shown in FIG. 11A, a learning content selection screen G for allowing the user to select learning content is displayed. It is displayed on the part 18 (step S1).

  In this learning content selection screen G, as shown in FIGS. 11 (B) and 11 (C), listening to the entire English conversation by operating the cursor key 17a1 and the “decision” key 17a2 of the input unit 17a, or practice by part When any one of the conversation parts (1: 2: 3) is selected, initialization processing such as clearing of each work area in the RAM 12C is performed, and the part number of the selected English conversation part is the designated part number. 12e is stored in the RAM 12C (steps S1 and S2).

  Then, the reproduction file (CAS file) 12b stored in the FLASH memory 12B is read (step S3), and it is determined whether or not the reproduction file (CAS file) 12b is an encrypted file (step S4). ).

  If it is determined that the file is an encrypted reproduction file (CAS file) 12b, the CAS file 12b is decrypted and decrypted (step S4 → S5), transferred to the RAM 12C, and stored (step S4). Step S6).

  Here, the sound output type data creation processing in FIG. 9 is executed (step SA).

  In the pronunciation type data creation processing, first, the English conversation text data stored as the content content data 12c4 is read as text corresponding pronunciation type data 12f in the RAM 12C as shown in FIG. 7, for example (step A1). .

  Then, all the words of the English conversation text data read into the text-corresponding pronunciation type data 12f are sequentially looked up based on the dictionary data 12a2 stored in the ROM 12A, and the respective phonetic symbols are read out (step). A2).

  Then, the mouth shape number data corresponding to each of the phonetic symbols read for all the words of the English conversation text data is read from the mouth shape data 12a3 (see FIG. 2) stored in the ROM 12A. The text words, phonetic symbols, and mouth type numbers for each part are associated and edited and stored in the text corresponding phonetic mouth type data 12f in the RAM 12C.

  In the pronunciation type data creation process, when the English conversation text data stored as the content content data 12c4 is read as the text corresponding pronunciation type data 12f in the RAM 12C, the same English conversation text data has already been previously recorded. When the editing of the association with the pronunciation type data is completed with the reproduction process, the pronunciation type data creation process for the same English conversation text data is omitted and not performed.

  In this way, when the correspondence editing between the English conversation text and the pronunciation type data is completed by the pronunciation type data creation processing, the processing unit time 12c1a (for example, 25 ms) of the reproduction file (CAS file) 12c by the CPU 11 is the time code file 12c3. Header information 12c1 is set (step S7).

  Then, a read pointer is set at the head of the decrypted reproduction file (CAS file) 12c stored in the RAM 12C (step S8), and a timer for timing the reproduction processing timing of the reproduction file 12c is started ( Step S9).

  Here, the prefetch process is started in parallel with the reproduction process (step S10).

  In this pre-reading process, if there is a “DI” command for displaying an image file after the command processing at the position of the current read pointer according to the time code file 12c3 (see FIG. 3) of the reproduction file 12c, the “ By pre-reading the image file designated by the parameter data of the “DI” command and developing it in the image development buffer 12k, the processing can be performed when the read pointer has actually moved to the position of the subsequent “DI” command. The specified image file can be output and displayed immediately without delay.

  When the processing timer is started in step S9, the read pointer set in step S8 is set every processing unit time (25 ms) corresponding to the current reproduction target file 12c set in step S7. The command code and its parameter data of the time code file 12c3 (see FIG. 3) constituting the reproduction file 12c at the position are read (step S11).

  Then, it is determined whether or not the command code read from the time code file 12c3 (see FIG. 3) in the reproduction file 12c is “FN” (step S12), and if “FN” is determined, At that time, the stop process of the file reproduction process is instructed and executed (steps S12 → S13).

  On the other hand, if it is determined that the command code read from the time code file 12c3 (see FIG. 3) in the reproduction file 12c is not “FN”, whether or not the command code is “PT”. Determination is made (steps S12 → S14).

  When it is determined that the command code is “PT”, the corresponding part-by-part image display processing A in FIG. 10 is executed (step SB).

  In the corresponding part-by-part image display process A, first, the part number p of the English conversation text designated by the command code “PT” and its parameter data is detected (step Ba1), and this part number p is used as the time code file 12c3. A value added to the command number “DI” (see FIG. 3) and the image number of the English conversation basic image G0 specified by the parameter data is set as the image number i of the reproduction target part (step Ba2). For example, the basic image number specified by the parameter data of the first command code “DI” in the time code file 12c3 (see FIG. 3) is “00”, and the part specified by the parameter data of the command code “PT”. When the number p = “01”, the reproduction target part image number i = 01 (00 + 01).

  Then, a command (i-th image display command) corresponding to the command code “DI” is executed. For example, the ID number associated with the image number i = 01 in the file sequence table 12c2 and stored in the content content data 12c4. = 6 during conversation G1 of the part 1 is displayed (step Ba3).

  Thus, when the part-by-part image designated by the command code “PT” is displayed, the part number p of the English conversation part designated by the parameter data of the command code “PT” is selected from the learning content selection process (step In S1), it is determined whether or not it matches the designated part number 12e when performing the part-specific practice selected and stored by the user (step S15).

  Here, when the part number p of the English conversation part designated by the parameter data of the command code “PT” does not match the designated part number 12e selected by the user, that is, the conversation part is different from the conversation practice part desired by the user. If it is determined, the voice-on flag 12i is set in the RAM 12C in order to perform voice output in the conversation part portion (steps S15 to S16), and the text character string of the relevant conversation part according to the command code “HL”. The highlight type for identifying and displaying is changed and set to the identification display form by underlining (step S17).

  On the other hand, when the part number p of the English conversation part designated by the parameter data of the command code “PT” matches the designated part number 12e selected by the user, that is, when it is determined that the conversation practice part is desired by the user. The voice off flag 12i is set in the RAM 12C in order to stop the voice output in the conversation part portion (steps S15 → S18), and the text character string of the conversation part is identified and displayed according to the command code “HL”. The type of highlight for this is changed and set to the identification display form by inversion (step S19).

  Then, based on the timing operation by the processing timer again, it is determined whether or not the time measured by the timer has reached the next processing unit time 12c1a (step S20).

  On the other hand, if it is determined in step S14 that the command code read from the time code file 12c3 (see FIG. 3) in the reproduction file 12c is not “PT”, the process proceeds to another command process. Then, processing corresponding to each command content (see FIG. 6) is executed (step SC).

  If it is determined in step S20 that the time measured by the timer has reached the next processing unit time 12c1a, the read pointer for the decrypted reproduction file (CAS file) 12c stored in the RAM 12C is the next. The position is updated and set (step S20 → S21), and the process from reading the command code and its parameter data in the time code file 12c3 (see FIG. 3) at the position of the read pointer in step S11 is repeated (step S21 → S11). ~ S19 (SC)).

  That is, the CPU 11 of the portable device 10 performs a time code file 12c3 (see FIG. 5) for each command processing unit time preset in the reproduction file 12b (12c) according to the synchronized content reproduction processing program 12a2 stored in the ROM 12A. 3), the command code and its parameter data are read out, and the text, sound, and image are synchronously played back according to each command described in the time code file 12c3 simply by instructing the processing corresponding to the command. Output processing is executed.

  When the instruction for synchronous playback output of voice, text, and image is issued in accordance with the reading of the command code from the time code file 12c3 (see FIG. 3) in the playback file 12c, the corresponding part-by-part image In accordance with the display process A, the in-conversation image Gn corresponding to the next playback part instructed by the command code “PT” is displayed, so that an image representing the movement of the mouth for each conversation part can be easily displayed. The user can practice the conversation more effectively.

  When the next playback part designated by the command code “PT” is the part to be practiced set by the user, the sound output of the part part is stopped, and the synchronous playback output of only the text and part-specific image Gn is performed. By entering the state, the user can practice the conversation by reading the text himself / herself while viewing the text and the part-by-part image Gn displayed and output in the practice part set by the user himself / herself.

  When the next playback part designated by the command code “PT” is not the part to be practiced set by the user, the sound output of the part part is not stopped, and the synchronous playback output state of text, sound, and image Thus, the user can practice the conversation by listening to the voice output of the text while viewing the text and images displayed and output for the other parts.

  Here, the synchronized playback output operation of audio / text / image files by the synchronized content playback processing program 12a1 (see FIGS. 8 to 10) based on the English teaching material playback file 12c shown in FIG. 3 will be described in detail.

  FIG. 12 is a diagram showing the synchronized playback state (part 1) of the English conversation text / sound / image file based on the English teaching material playback file 12c in FIGS.

  This English teaching material file (12c) is one in which command processing is executed every (c) processing unit time (for example, 25 ms) 12c1a preset in its header. First, a time code file 12c3 (see FIG. 3). When the first command code “CS” (clear all file) and its parameter data “00” are read out, an instruction to clear the output of all the files is given, and the output of the text / audio / image file is cleared ( Step SC).

  When the second command code “DH” (i-th HTML file display) and its parameter data “01” are read, the file sequence table 12c2 according to the parameter data (i = 1) read together with the command code DH. The ID number = 2 of HTML number 1 is read from (see FIG. 4).

  Then, in accordance with the English conversation text / image frame data of the HTML data read from the content content data 12c4 (see FIG. 5) in association with the ID number = 2, as shown in FIG. A text display frame X and an image display frame Y for are set (step SC).

  When the third command code “DI” (i-th image file display) and its parameter data “00” are read, the file sequence table 12c2 is determined according to the parameter data (i = 0) read together with the command code DI. The ID number = 5 of the image number 1 is read from (see FIG. 4).

  Then, the image data (English conversation text basic image G0) read from the content content data 12c4 (see FIG. 5) and expanded in the image expansion buffer 12k in association with the ID number = 5 is set in the HTML file. Is displayed in the displayed image display frame Y (step SC).

  When the fourth command code “PS” (i-th sound file play) and its parameter data “02” are read, the file sequence table 12c2 is determined according to the parameter data (i = 2) read together with the command code PS. The ID number = 32 of the voice number 2 is read from (see FIG. 4).

  Then, the English voice data (2) read from the content content data 12c4 (see FIG. 5) in association with the ID number = 32 is output from the stereo voice output unit 19b (step SC).

  When the fifth command code “LT” (i-th text load) and its parameter data “02” are read, the file sequence table 12c2 (i = 2) is read according to the parameter data (i = 2) read together with the command code LT. ID number = 21 of text number 2 is read from (see FIG. 4).

The English conversation text data (2) read from the content content data 12c4 (see FIG. 5) in association with the ID number = 21.
Is loaded into the work area of the RAM 12C (step SC).

  When the sixth command code “VD” (i-th text phrase display) and its parameter data “00” are read, the file sequence table 12c2 is determined according to the parameter data (i = 0) read together with the command code VD. The ID number = 19 of the text number 0 is read from (see FIG. 4), and the phrase of the English conversation title character specified in the content content data 12c4 (see FIG. 5) is loaded into the RAM 12C. It is called out from the English conversation text data (2), and is displayed in the text display frame X on the display screen (step SC).

  When the seventh command code “BL” (character counter reset / i-th phrase block designation) and its parameter data “00” are read, the character counter of the English conversation phrase being displayed in the text display frame X is reset, and the 0th Are designated (step SC).

  When the eighth command code “PI” (part-by-part image file display) and its parameter data “00” are read, an instruction to display the image of the designated part is issued (step SC).

  When the ninth command code “PT” (i-th part designation) and its parameter data “01” are read, based on the English conversation basic image G0, the English conversation voice data (2) and the English conversation text to be reproduced from now on The conversation part 1 (Mr. A) in the data (2) is designated (step S14).

  Then, according to the corresponding part-by-part image display processing A in FIG. 9, the in-conversation image G1 representing the movement of the mouth of the part 1 person corresponding to the image number i = 1 corresponding to the specified part number p = 1. Is read and displayed as shown in FIG. 12A (step SB).

  If the designated part number 12e preset by the user is the conversation part 2 (Mr. B), it does not coincide with the conversation part 1 (Mr. A) designated by the ninth command code “PT” (step A). S15), the audio output of the corresponding part 1 is turned on (step S16), and the highlight processing of the processing content corresponding to the command code “HL” is changed to underline processing (step S17).

  Then, when the 10th command code “HL” (highlight / character count up to the i-th character) and its parameter data “07” are read, according to the parameter data (i = 7) read together with the command code HL. Then, as shown in FIG. 12A, the seventh character “A: What” (including space) of the text data is underlined (underlined), and the character counter is counted up to the seventh character. (Step SC).

  At this time, during the underline display of the conversation character string which is the part 1 (Mr. A) portion of the text data, since the voice output is set to ON, the stereo voice according to the fourth command code “PS” is set. As for the English conversation voice data (2) output from the output unit 19b, the voice of reading the underline display part of the same conversation part 1 (Mr. A) is output.

  When the eleventh command code “NP” is read, the current image and the English conversation text data synchronous display screen and the English conversation voice data synchronous output state are maintained.

  Thereafter, in accordance with the 13th command code “HL”, the 35th command code “HL”, and the 58th command code “HL”, the text data of the conversation part 1 (Mr. A) portion in the display state of the part 1 conversation-in-progress image G1. Are sequentially displayed as underlined (underlined) as the 12th character “high”, the 19th character “school”, and the 22nd character “do”, and the fourth command code “ As for the English conversation voice data (2) output from the stereo voice output unit 19b in response to “PS”, the voice that reads out the underline display part of the conversation part 1 (Mr. A) is sequentially output (steps S11 to S11). S14 → SC, S20, S21 → S11).

  That is, when the next English conversation playback part designated by the command code “PT” is not the part to be practiced set by the user (for example, Part 2 (Mr. B)), the audio output of the other part part is not stopped. , The English conversation text, its reading voice, and the synchronized playback output state of the corresponding part-by-part image Gn, the user can practice the conversation by listening to the voice output of the text while viewing the text and images displayed for other parts. It can be performed.

  When the 119 command code “PT” (i-th part designation) and its parameter data “02” are read out, the English conversation voice data (2), the English conversation text data (2) and the part to be reproduced next are synchronized. A conversation part 2 (Mr. B) in another image Gn is designated (step S14).

  Then, in accordance with the corresponding part-by-part image display processing A in FIG. 9, the in-conversation image G2 representing the movement of the mouth of the part 2 person corresponding to the image number i = 2 corresponding to the designated part number p = 2. Is read and displayed as shown in FIG. 12B (step SB).

  Here, when the designated part number 12e set in advance by the user is the conversation part 2 (Mr. B), it matches the conversation part 2 (Mr. B) designated by the 119th command code “PT” ( In step S15), the audio output of the corresponding part 2 is turned off (step S18), and the highlight processing of the processing content corresponding to the command code “HL” is changed to the inversion processing (step S19).

  When the 120th command code “HL” (highlight / character count up to i-th character) and its parameter data “37” are read, according to the parameter data (i = 37) read together with the command code HL. Then, as shown in FIG. 12 (B), the 37th character of the text data, that is, the character “B: I” (including the space) of the conversation part 2 is now displayed by inversion, and the character counter is the 37th character. Counts up to characters (step SC).

  At this time, during the reverse display of the conversation character string which is the part 2 (Mr. B) portion of the text data, since the voice output is set to OFF, the stereo voice output is performed according to the fourth command code “PS”. The English conversation voice data (2) sequentially output from the unit 19b is stopped.

  Thereafter, in accordance with the 132nd command code “HL” and the 140th command code “HL”, the text data of the conversation part 2 (Mr. B) portion remains without sound output in the display state of the part 2 conversation-in-progress image G2. The 40th character “go” and the 43rd character “to” are sequentially displayed in reverse video (steps S11 to S14 → SC, S20, S21 → S11).

  That is, when the next English conversation playback part instructed by the command code “PT” is the part to be practiced set by the user (for example, Part 2 (Mr. B)), the sound output of the part part is stopped and the English conversation is stopped. By entering the synchronized playback output state of only the text part 1 and the corresponding part-specific image G2, the user reads the text himself / herself while viewing the text / image displayed and output in the practice part set by the user himself / herself. Can practice.

  Further, when the 159th command code “PT” (i-th part designation) and its parameter data “03” are read, the English conversation voice data (2), the English conversation text data (2), and the part to be synchronously reproduced next are read. A conversation part 3 (Mr. C) in another image Gn is designated (step S14).

  Then, in accordance with the corresponding part-by-part image display processing A in FIG. 9, the in-conversation image G3 representing the movement of the mouth of the part 3 person corresponding to the image number i = 3 corresponding to the designated part number p = 3. Is read and displayed as shown in FIG. 12C (step SB).

  If the designated part number 12e preset by the user is the conversation part 2 (Mr. B), it does not match the conversation part 3 (Mr. C) designated by the 159th command code “PT” ( In step S15), the audio output of the corresponding part 3 is turned on (step S16), and the highlight processing of the processing content corresponding to the command code “HL” is changed to underline processing (step S17).

  Then, when the 160th command code “HL” (highlight / character count up to i-th character) and its parameter data “70” are read out, according to the parameter data (i = 70) read out together with the command code HL. Then, as shown in FIG. 12C, the text data is underlined (underlined) up to the 70th character “C: Me,” (including spaces), and the character counter counts up to the 70th character. (Step SC).

  At this time, during the underline display of the conversation character string which is the part 3 (Mr. C) portion of the text data, since the voice output is set to ON, the stereo voice according to the fourth command code “PS” is set. As for the English conversation voice data (2) output from the output unit 19b, the voice of reading the underline display part of the conversation part 3 (Mr. C) is output.

  Thereafter, in accordance with the 172nd command code “HL”, the text data of the conversation part 3 (Mr. C) part is underlined (underlined) until the 75th character “too.” In the display state of the part 3 in-conversation image G3. ) And the English conversation voice data (2) output from the stereo voice output unit 19b in response to the fourth command code “PS” is also displayed in the underline display part of the same conversation part 3 (Mr. C). The voice to be read out is continuously output (steps S11 to S14 → SC, S20, S21 → S11).

  That is, when the next English conversation playback part instructed by the command code “PT” is no longer the part to be practiced set by the user (for example, part 2 (Mr. B)), the audio output of the other part part is performed. Is resumed, and the English conversation text, its reading voice, and the corresponding part image Gn are in a synchronized playback output state, so that the user again listens to the text output displayed for the other parts and the voice output of the text while viewing the image. Can practice conversation.

  Thus, the playback file is set in advance according to the time code file 12c3 (see FIG. 3), the file sequence table 12c2 (see FIG. 4), and the content content data 12c4 (see FIG. 5) in the English conversation teaching material playback file (12c). As shown in FIGS. 12 (A) to 12 (D), by executing command processing every (standard) processing unit time (for example, 25 ms) 12c1a, the English text data is displayed in the text display frame X on the display screen. Is displayed while the part portion is being identified and displayed, and the part-by-part image Gn showing the movement of the part person's mouth is synchronously displayed in the image display frame Y, and is further being identified and displayed from the stereo sound output unit 19b. English conversation voice data that reads out the text of the English conversation part of Reading clause is to be sequential synchronization highlighted (highlighted) display for each character.

  At this time, according to the corresponding part-by-part image display processing A, the conversational image Gn indicating the movement of the part person's mouth corresponding to the next reproduction part instructed by the command code “PT” is displayed. Image display representing mouth movement for each conversation part can be performed, and the user can practice conversation more effectively.

  Also, during the playback output period of the conversation part specified by the user, the synchronized output of the audio data is stopped, so that the user views the text and the part-specific image Gn displayed and output in the practice part set by the user himself / herself. However, it is possible to practice the conversation by reading the text by itself, and in other conversation parts, it is possible to practice by listening to the speech output while viewing the text and the image by part Gn.

  Therefore, according to the file playback function of the mobile device 10 having the above-described configuration, the command processing reference unit time (25 ms) preset in the playback file 12b (12c) according to the playback processing program 12a1 stored in advance in the ROM 12A. / 50 ms), the command code and its parameter data arranged in the time code file 12c3 are read out, and the text corresponding to each command described in the time code file 12c3 can be obtained simply by instructing the processing corresponding to the command. -Synchronous playback processing for images and audio files is executed.

  For example, when synchronous playback output of text, sound, and images consisting of a plurality of conversation parts is performed using the English conversation teaching material playback file 12b (12c), the command code “PT” and Since the in-conversation image Gn indicating the movement of the mouth of the person corresponding to the part to be reproduced next designated by the parameter data is displayed, the movement of the mouth for each conversation part is easily expressed. Images can be displayed, and the user can practice conversation more effectively.

  Further, when the next output target conversation part specified in accordance with the command code and its parameter data matches the practice target conversation part set by the user, the set conversation part stops the voice output. Identification of part-by-part text / Synchronized display output of corresponding part-by-part image Gn is performed, and if it does not match the conversation part to be practiced, voice output is resumed and identification of part-by-part text and corresponding part-by-part image Gn Since synchronized playback is performed, the user can practice the pronunciation by simply turning off the text-to-speech output of only the conversation part desired by the user, and the practice for each part can be performed easily and effectively.

  Further, according to the file playback function by the mobile device 10 having the above-described configuration, the audio data output instruction “PS” or the audio data is matched according to the command code for each reference processing unit time described in the time code file 12c3. When the text data display instruction “VD” “HL” and the part-by-part image Gn display instruction “DI” “PI” are given, the start of each conversation part is designated by “PT”, and the corresponding part-by-part is designated. Since the image Gn is displayed or the voice output of the part is turned off when it matches the practice part set by the user, the practice for each part can be performed very easily.

  Further, according to the file playback function of the mobile device 10 having the above-described configuration, the part number (type) specified by the command code “PT” of the time code file 12c3 is a parameter described in a pair with the command code “PT”. Since it is set by data, it is possible to easily create a time code file 12c3 in which each playback part is designated.

  In the corresponding part-by-part image display process A (see FIG. 10) associated with the reproduction process of the first embodiment, each conversation part is previously set according to the part number p specified by the command code PT of the time code file 12c3. The conversation image Gn for each part prepared by expressing the movement of the mouth of the part person is switched and displayed, and the conversation state of each part is also expressed synchronously on the image. As described in accordance with the corresponding part-specific image display processing B and the text corresponding mouth display processing shown in FIGS. 13 to 15, the command code PT is based on the English conversation basic image G0 having no mouth movement in any of the parts. For each designated part person's mouth area M1 to M3, a text corresponding pronunciation mouth corresponding to the current identification display character of each part English conversation text By the sound outlet type image of the relevant text characters to composite display to be read from the data 12f, it may be configured cities representing synchronously more realistic conversational state of each part on the image.

  FIG. 13 is a flowchart showing the corresponding part-specific image display process B accompanying the reproduction process of the portable device 10.

  FIG. 14 is a flowchart showing the text corresponding mouth display process executed by interruption in accordance with the highlight (highlight) display process of the text by part according to the command code HL in the reproduction process of the portable device 10.

  FIG. 15 is a diagram showing the synchronized playback state (part 2) of the English conversation text / sound / image file based on the English teaching material playback file 12c in FIGS.

  That is, when the command code PT is read from the time code file 12c3 in step S14 of the reproduction process described with reference to FIG. 8, the process proceeds to the corresponding part-specific image display process B in FIG. First, the English conversation text basic image G0 of the content content data 12c4 is read and displayed in the image display frame Y on the display screen as shown in FIG. 15 (step Bb1).

  Then, the next reproduction target part number p designated by the parameter data of the command code PT is detected (step Bb2), and the corresponding part person image on the English conversation text basic image G0 corresponding to the designated part number p is detected. The rectangular area data (position coordinates) Mn of the mouth position is read from the content content data 12c4 in which the English conversation text basic image G0 is stored (step Bb3).

  Then, in step SC of the reproduction process, according to the command code HL read from the time code file 12c3, up to the current read-out character of the English conversation text being displayed is identified and displayed by the highlight process. When the text corresponding mouth display process is activated by interruption, the character at the current text highlight processing position is detected based on the parameter data of the command code HL (step D1).

  Then, the mouth shape image data of the pronunciation corresponding to the character at the text highlight position is in accordance with the mouth number of the text corresponding mouthpiece data 12f (see FIG. 7) created by the pronunciation mouth data creation processing in FIG. It is read from the mouth shape data 12a3 (see FIG. 2) in the ROM 12A (step D2).

  Then, the mouth shape image data of the pronunciation corresponding to the character at the current text highlight position is the rectangular area data (the mouth position of the designated part person image read out in step Bb3 of the corresponding part image display processing B) ( According to the position coordinates (Mn), as shown in FIG. 15, it is synthesized and displayed on the displayed English conversation basic image G0 (step D3).

  As a result, for each part of the English conversation text currently being played, the mouth shape corresponding to the pronunciation can be synthesized and displayed at the mouth position of the corresponding part person image, and the conversation status of each part can be displayed on the image. It is possible to express more realistically and to practice more effective conversation.

  When a key (button) operation for instructing “in-mouth enlarged display” is performed by the input unit 17a or the coordinate input device 17b (step D4), for example, as shown in FIG. Corresponding to the highlight display, an enlarged image G4 of the mouth shape image being synthesized and displayed on the English conversation basic image G0 is read from the mouth shape data 12a3 (see FIG. 2) in the ROM 12A (step D5), and A window is displayed in a display area different from the part-by-part mouth area M1 on the English conversation basic image G0 (step D6).

  In this case, in FIG. 15D, it corresponds to “th” of “both” synchronized with the text highlight display “Oh, both” corresponding to Part 1 (Mr. A) and the reading voice output (19b). The combined display of the mouth-shaped image in the part-one mouth area M1 and the window display state of the pronunciation-mouthed enlarged image G4 are shown.

  Thereby, the user can know the mouth shape of pronunciation during the conversation practice in real time and in more detail, and can practice English conversation most effectively.

  In the embodiment of the mouth type composition display according to the text highlight position, the mouth type image corresponding to the actual pronunciation is read from the mouth type data 12a3 and is synthesized and displayed in the mouth area Mn of the individual image for each part. However, it may be configured such that the opened mouth image is simply combined and displayed for each part with respect to the English conversation text basic image G0 in which the person images of all the parts are closed.

(Second Embodiment)
Next, the portable device 10 according to the second embodiment of the voice display output control device (editing device) of the present invention will be described.

  In the mobile device 10 according to the second embodiment, in accordance with the search for the headword based on the dictionary data 12a2, according to the time code file 12c3 similar to the first embodiment prepared for each headword, the headword. It is configured to display the mouth-shaped image of the pronunciation synchronized with the display of the text and the output of the reading voice.

  FIG. 16 is a diagram showing data contents of the dictionary data 12a2 stored in the ROM 12A of the portable device 10 according to the second embodiment of the present invention.

  FIG. 17 is a diagram showing data contents of the headword audio data 12a4 stored in the ROM 12A of the mobile device 10 according to the second embodiment of the present invention.

  In the dictionary data 12a2, for example, a headword of an English word and its meaning content indicate a phonetic symbol and the type of mouth shape in the mouth shape data 12a3 (see FIG. 2) for uttering according to the phonetic symbol. The mouth type number and the headword voice data indicating the headword voice data in the headword voice data 12a4 (see FIG. 17) are stored in association with each other.

  In the headword voice data 12a4, the read-out voice data of each headword of all English words stored as the dictionary data 12a2 is stored as data consisting of a PCM sound source.

  FIG. 18 is a flowchart showing dictionary search / display processing by the mobile device 10 of the second embodiment.

  FIG. 19 is a diagram showing a synchronized playback state of search headword display / voice output / mouth-shaped image display associated with dictionary search / display processing by the portable device 10 of the second embodiment.

  For example, when searching for the English word “able”, as shown in FIG. 19A, when the “English-Japanese” key 17a3 of the input unit 17a is operated, an English-Japanese dictionary entry word input screen G5 is displayed. The In this headword input screen G5, when the search target headword “able” is input and displayed and a “determine” key (not shown) is operated (step E1), the search is performed based on the dictionary data 12a2 (see FIG. 16). A search for the search target headword “able” is executed (step E2), and it is determined whether or not a plurality of corresponding headwords have been searched (step E3).

  Here, when a plurality of headwords corresponding to the search target headword “able” are searched from the dictionary data 12 a 2, the searched headwords are displayed in a list on the display unit 18. (Step E3 → E4).

  One headword corresponding to the search target headword “able” is selected from a plurality of headwords displayed in the headword list display by a key input (step E5), or If it is determined in step E3 that only one relevant headword has been searched, the phonetic symbol stored as the dictionary data 12a2 together with the one heading word “able” is read out. It is displayed on the display unit 18 (step E6).

  Then, in order to practice the pronunciation of the headline word “able” displayed by the search, as shown in FIG. 19B, when the “pronunciation” key 17a4 in the input unit 17a is operated (step E7), In synchronization with the text display and voice output of the heading word “able”, the process proceeds to a voice / text / mouth image synchronization process for displaying a mouth image of the pronunciation (step ES).

  The voice / text / mouth image synchronization processing associated with the dictionary search / display processing is performed in step S3 of the reproduction processing (see FIG. 8) according to the time code file 12c3 prepared for each headword as in the first embodiment. To S21 (excluding SA).

  For example, by using the command code “PS” of the time code file 12c3 and its parameter data, the voice data of the current headword “able” is read from the headword voice data 12a4 (see FIG. 17), and the voice output is obtained. The command code “VD” and its parameter data indicate the text display of the heading word “able”, and the command code “HL” and its parameter data indicate the text for the heading word “able”. A highlight display for each character in accordance with the output of the voice data is instructed, and by using the command code “PT” and its parameter data, the heading word “able” is highlighted and the pronunciation type adapted to the output of the voice data An instruction to display an image file is displayed.

  Accordingly, as shown in FIG. 19B, the heading word “Able” of the heading word “able” searched for the dictionary is output from the stereo voice output unit 19b, and the heading word “ "Able" is sequentially highlighted for each character, and the mouth-shaped images of the pronunciation are sequentially switched and displayed.

  Then, when the readout switching display of the pronunciation-portion image corresponding to the searched headword is completed (step E8), a series of dictionary search / display processing corresponding to the search target headword “able” is ended.

  Therefore, according to the file reproduction function by the mobile device 10 of the second embodiment having the above-described configuration, in the dictionary search mode, it is stored in the entry word data 12a4 in association with each entry word in the dictionary data 12a2. Using the mouth image data for each phonetic symbol stored in the speech data and the mouth shape data 12, the pronunciation mouth synchronized with the display of the headword text and the output of the reading speech very easily The mold image can be displayed.

  Note that each processing method by the mobile device 10 described in the embodiment, that is, the file reproduction process shown in the flowchart of FIG. 8, the pronunciation type data creation process associated with the file reproduction process shown in the flowchart of FIG. Corresponding part-by-part image display process A accompanying the file reproduction process shown in the flowchart of FIG. 13, Corresponding part-by-part image display process B accompanying the file reproduction process shown in the flowchart of FIG. 13, and commands in the file reproduction process shown in the flowchart of FIG. A program that can cause a computer to execute any of the text-corresponding mouth display processing that accompanies the highlight (highlight) display processing of part-by-part text according to the code HL and the dictionary search / display processing shown in the flowchart of FIG. Memory card (ROM card, R M card), a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.) can be distributed and stored in the external recording medium 13 such as a semiconductor memory. Various computer terminals having a communication function with the communication network (Internet) N read the program stored in the external recording medium 13 into the memory 12 by the recording medium reading unit 14, and the operation is performed by the read program. By being controlled, the synchronized playback function of text, sound, and image described in the first embodiment and the second embodiment can be realized, and the same processing by the above-described method can be executed.

  The program data for realizing each of the above methods can be transmitted on a communication network (Internet) N in the form of a program code, and the above-mentioned data can be transmitted from a computer terminal connected to the communication network (Internet) N. The program data can be fetched to realize the synchronized reproduction function of the text, sound, and image of each of the above-described embodiments.

  Note that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention at the stage of implementation. Further, each of the embodiments includes inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in each embodiment or some constituent features are combined, the problems described in the column of the problem to be solved by the invention can be solved. When the effects described in the column of the effect of the invention can be obtained, a configuration in which these constituent elements are deleted or combined can be extracted as an invention.

The block diagram which shows the structure of the electronic circuit of the portable apparatus 10 which concerns on embodiment of the audio | voice display output control apparatus (editing apparatus) (voice text reproduction apparatus) of this invention. The figure which shows the content of the mouth type data 12a3 memorize | stored in ROM12A of the said portable apparatus 10. FIG. The figure which shows the time code file 12c3 which comprises the file 12b (12c) for reproduction | regeneration stored in the memory 12 of the said portable device 10. FIG. The figure which shows file sequence table 12c2 which comprises the file 12b (12c) for reproduction | regeneration stored in the memory 12 of the said portable device 10. FIG. The figure which shows the content content data 12c4 which comprises the file 12b (12c) for reproduction | regeneration stored in the memory 12 of the said portable apparatus 10. FIG. The figure which matches and shows the command code of the various commands described in the time code file 12c3 (refer FIG. 3) of the said portable apparatus 10, the parameter data, and the command content analyzed based on the reproduction | regeneration processing program 12a1. The figure which shows the content of the text corresponding sound-mouth type | mold data 12f memorize | stored in RAM12C of the said portable apparatus 10. FIG. 7 is a flowchart showing a playback process according to the playback process program 12a1 of the portable device 10. 6 is a flowchart showing sound generation type data creation processing that accompanies the reproduction processing of the mobile device 10. 6 is a flowchart showing image display processing A for each corresponding part that accompanies playback processing of the mobile device 10. It is a figure which shows the selection operation and display state of the learning content accompanying the reproduction | regeneration processing of the said mobile device 10, The figure (A) is a figure which shows the learning content selection screen G, The figure (B) (C) is the said learning content. The figure which shows the selection operation key which makes selection screen G object. The figure which shows the synchronous reproduction | regeneration state (the 1) of an English conversation text, an audio | voice, and an image file based on the English teaching material reproduction | regeneration file 12c in the said FIG. 3 thru | or FIG. 6 is a flowchart showing a corresponding part-by-part image display process B accompanying a reproduction process of the mobile device 10. 6 is a flowchart showing a text corresponding mouth display process executed by interruption in accordance with a highlight display process of part-by-part text according to a command code HL in the reproduction process of the mobile device 10. The figure which shows the synchronous reproduction | regeneration state (the 2) of an English conversation text, an audio | voice, and an image file based on the English teaching material reproduction | regeneration file 12c in the said FIG. 3 thru | or FIG. The figure which shows the data content of the dictionary data 12a2 memorize | stored in ROM12A of the portable apparatus 10 of 2nd Embodiment of this invention. The figure which shows the data content of the headword audio | voice data 12a4 memorize | stored in ROM12A of the portable apparatus 10 of 2nd Embodiment of this invention. The flowchart which shows the dictionary search and display process by the portable apparatus 10 of the said 2nd Embodiment. The figure which shows the synchronous reproduction | regeneration state of the search headword display, audio | voice output, and mouth type | mold image display accompanying the dictionary search and display process by the portable apparatus 10 of the said 2nd Embodiment.

Explanation of symbols

10 ... Mobile device 11 ... CPU
12 ... Memory 12A ... ROM
12a1... File reproduction processing program 12a2... Dictionary data 12a3... Mouth data 12a4 .. entry speech data 12B... FLASH memory 12b... Encrypted reproduction file (CAS file)
12C ... RAM
12c: Decoded playback file (CAS file)
12c1 ... Header information 12c1a ... Processing unit time 12c2 ... File sequence table 12c3 ... Time code file 12c4 ... Content content data 12d ... Designated text number 12e ... Designated part number 12f ... Text-corresponding pronunciation type data 12g ... Text highlight designated character 12h ... Image (on / off) flag 12i ... Audio (on / off) flag 12j ... Image developed flag 12k ... Image developed buffer 13 ... External recording medium 14 ... Recording medium reading unit 15 ... Electric transmission control unit 16 ... Communication unit 17a ... Input unit 17b ... Coordinate input device 18 ... Display unit 19a ... Audio input unit 19b ... Stereo audio output unit 20 ... Communication equipment (home PC)
30 ... Web server N ... Communication network (Internet)
X ... Text display frame Y ... Image display frame G0 ... English conversation text basic images G1 to G3 ... Part 1 to Part 3 conversation-in-progress image G4 ... Sound-entry type enlarged image G5 ... Headword input screen M1 to M3 ... Part 1 to part 3 mouth area

Claims (14)

Audio data output means for outputting audio data;
Text synchronization display control means for displaying text data so as to be synchronized with the voice data output by the voice data output means;
Image display control means for displaying an image expressing the movement of the mouth corresponding to the synchronous display portion of the text displayed so as to be synchronized with the audio data by the text synchronous display control means;
An audio display output control device comprising: The image display control means includes
Image storage means for storing a plurality of images expressing different mouth movements according to each part of the text data;
The image corresponding to the synchronous display portion of the text displayed so as to be synchronized with the audio data by the text synchronous display control means is read out from the plurality of images stored by the image storage means and displayed.
The voice display output control device according to claim 1. The image display control means includes
Image display means for displaying an image;
Mouth image storage means for storing a plurality of mouth images expressing different mouth movements according to various pronunciations;
The mouth image corresponding to the pronunciation of the synchronous display portion of the text displayed so as to be synchronized with the voice data by the text synchronization display control means is read out from the plurality of mouth images stored by the mouth image storage means, and the image Mouth image display control means for displaying the mouth position of the image displayed by the display means;
The voice display output control device according to claim 1, comprising: The mouth image storage means stores a plurality of mouth images expressing different mouth movements according to various pronunciations, and stores an internal image of the mouth in association with each of the plurality of mouth images,
4. The voice display according to claim 3, further comprising an intra-mouth enlarged display unit that reads and enlarges and displays an internal image of the mouth corresponding to the mouth image displayed by the mouth image display control unit. Output control device. The text synchronous display control means includes:
There is an instruction code storage means for storing an instruction code for displaying the text data in synchronization with the voice data output by the voice data output means in accordance with a preset elapsed time, and the instruction code storage means In accordance with the stored instruction code, the text data is displayed in synchronization with the voice data,
The image display control means causes an image to be displayed according to the instruction code stored by the instruction code storage means.
The voice display output control apparatus according to any one of claims 1 to 4, wherein the voice display output control apparatus is configured as described above. An audio display output control processing program for controlling a computer of an electronic device to synchronously reproduce audio data, text data, and image data,
The computer,
Audio data output means for outputting audio data;
Text synchronization display control means for displaying text data so as to be synchronized with voice data output by the voice data output means;
Image display control means for displaying an image expressing mouth movement corresponding to a synchronous display portion of text displayed so as to be synchronized with the audio data by the text synchronous display control means;
A computer-readable audio display output control processing program designed to function as a computer. An editing device that edits corresponding data for outputting text and images in synchronization with audio,
Mouth image storage means for storing a plurality of different mouth images in association with various audio data;
Corresponding data editing means for reading out the mouth image stored in the mouth image storage means in association with the voice data corresponding to the output target text and storing it in correspondence with the position of the output target text or voice data; ,
An editing apparatus comprising:   Synchronous playback means for outputting text in synchronization with the voice and reading and displaying an image of the mouth associated with the text to be output based on the correspondence data stored by the correspondence data editing means. A speech text reproducing apparatus comprising the editing apparatus according to claim 7. Dictionary storage means for storing mouth image data in association with a plurality of headwords and each headword;
An entry word search means for searching for an entry word stored by the dictionary storage means;
The mouth image data stored in the dictionary storage means in association with the headword searched by the headword search means is read out, and each character of the headword and the mouth image data are read out. Synchronous display control means for sequentially and synchronously displaying,
A speech text reproducing apparatus comprising: Dictionary storage means for storing voice data and mouth image data in association with a plurality of headwords and each headword;
An entry word search means for searching for an entry word stored by the dictionary storage means;
Voice output means for outputting voice data stored in the dictionary storage means in association with the headword searched by the headword search means;
Each character of the headword searched by the headword search means is sequentially displayed in synchronization with the voice data output by the voice output means, and stored in association with the headword by the dictionary storage means. Synchronous display control means for reading out the image data of the mouth and sequentially displaying it in synchronism with the display of each character of the headword;
A speech text reproducing apparatus comprising: An edit processing program for editing correspondence data for controlling a computer to output text and images in synchronization with voice,
The computer,
Mouth image storage means for storing a plurality of different mouth images in association with various audio data;
Corresponding data editing means for reading out the mouth image stored in the mouth image storage means in association with the voice data corresponding to the output target text, and storing it in correspondence with the position of the output target text or voice data,
A computer-readable editing processing program designed to function as a computer. The computer further
Synchronous reproduction means for reading out and displaying an image of the mouth associated with the text to be output based on the correspondence data stored by the correspondence data editing means, while outputting the text in synchronization with the voice;
A speech text reproduction processing program comprising the editing processing program according to claim 11, which is configured to function as First storage means for storing a plurality of sets of text phonetic symbols and mouth-shaped images in association with each other;
Second storage means for storing a plurality of sets of characters and voices in association with each other;
The voice stored in the second storage means is sequentially output to the outside, and the characters stored in association with the second storage means are sequentially identified and displayed in synchronization with the output voice. 1 control means;
Mouth-shaped images stored in association with the first storage means in correspondence with the phonetic symbols of the characters to be identified and displayed in synchronization with the characters that are sequentially identified and displayed by the control of the first control means Second control means for sequentially switching and displaying the read mouth-shaped images,
An audio display output control device comprising: An audio display output control processing program for controlling a computer of an electronic device to output audio, text, and an image in synchronization with each other,
The computer,
First storage means for storing a plurality of sets of text phonetic symbols and mouth-shaped images in association with each other;
Second storage means for storing a plurality of sets of characters and voices in association with each other;
First, the voice stored by the second storage means is sequentially output to the outside, and the characters stored in association with the second storage means are sequentially identified and displayed in synchronization with the output voice. Control means,
In synchronization with the characters that are sequentially identified and displayed under the control of the first control means, the mouth image stored in association with the first storage means in correspondence with the phonetic symbols of the characters that are identified and displayed. A second control means for sequentially reading and displaying the read mouth-shaped images;
A computer-readable audio display output control processing program designed to function as a computer.

Images