JPH1166345A - Image acoustic processor and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image acoustic processor and a recording medium which have expression performance that naturally simulates the way a person actually pronounces and produces illusions that a model really sings and talks in user's mind. SOLUTION: This processor is provided with a sound generating means 12 which generates a sound, image displaying means 10 and 11 which show an image the represents the movement of a mouth at the time of pronouncing a syllable that is pronounced by the sound generating means and storing means 102 and 121 which store as many image data that express a mouth which pronounces a vowel as the number of vowel kinds. The means 10 and 11 identify vowels that are included in syllables which are pronounced by the means 12, read image data to represent a mouth at the time of pronouncing the identified vowels from the means 102 and 121 and show an image based on the read image data.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image sound processing apparatus for expressing an image of a human or the like uttering voice along with its voice, and more particularly, to an image sound processing apparatus capable of expressing a mouth movement when a human actually sounds. Technology improvement.

[0002]

2. Description of the Related Art With the development of computer graphics technology, a model simulating a real person can be constituted by polygons or the like. Among them, there is an image processing apparatus that moves the mouth of the model in accordance with the voice and expresses a state in which the model is singing and talking.

[0003] In fact, the movement of the mouth when a human pronounces a language is complicated. For this reason, the conventional image processing apparatus holds only two types of image data, that is, the mouth is opened or closed, and the image that moves the mouth of the model alternately in accordance with the voice based on these two types of image data. Display was done.

However, in a conventional image processing apparatus,
There was an inconvenience that the model could not express a skillful mouth movement until it could be said that the model actually pronounced words. In other words, since only two types of expression, opening and closing the mouth, cannot actually mimic the various mouth movements when a human pronounces, even if the mouth is moved completely in synchronization with the sound, Only unnatural mouth movements could be reproduced.

On the other hand, storing image data capable of expressing all mouth movements requires a huge amount of data, which is not suitable for industrial applications.

[0006]

SUMMARY OF THE INVENTION In order to solve the inconvenience in the conventional image processing apparatus, the present invention naturally simulates how a human actually sounds, as if the model really sang or talked. It is an object of the present invention to provide an image-acoustic processing device having an expressive power capable of giving an illusion to the user and a recording medium in which program data for realizing the same is recorded.

[0007] That is, a first object of the present invention is to provide an image acoustic processing technique capable of simulating a manner in which a person actually sounds a syllable with a fixed amount of image data.

A second object of the present invention is to provide an image sound processing technique which can naturally simulate a human uttering even a syllable that moves the mouth specially.

A third object of the present invention is to provide an image-acoustic processing technique capable of naturally simulating a human sounding even if syllables having the same vowel are consecutive.

[0010] A fourth object of the present invention is to provide an image processing technique capable of more naturally simulating the manner in which a human pronounces even when vowels included in consecutive syllables are different.

A fifth object of the present invention is to provide an image sound processing technique capable of simulating a human being naturally producing sound even when syllables including sound repelling are continuous.

[0012]

According to the first aspect of the present invention, there is provided an image sound processing apparatus for generating a sound in accordance with an image representing a mouth movement. Means, image display means for displaying an image representing the movement of the mouth when the syllable is pronounced, and storage means for storing image data for expressing the mouth when the vowel is pronounced only for the type of vowel And.

The image display means identifies a vowel included in a syllable to be generated by the sound generation means, reads image data for expressing a mouth when the identified vowel is generated from the storage means, and reads the read image data. Display an image based on the data.

Note that a syllable means a unit of sound that gives a sense of one sound. In Japanese, vowels mean the five sounds of "A", "I", "U", "E", and "O". In English, "a", "i", "u", "e" and Says 'o'. Vowels included in syllables mainly refer to vowels included in the ending of syllables.

In the invention for solving the second problem, the storage means stores image data for expressing a half-open mouth in addition to image data for expressing a mouth when a vowel is pronounced. When the syllable to be generated by the sound generating means is a predetermined specific syllable, the image display means includes image data for expressing the half-open mouth stored in the storage means and vowels included in the syllable. Based on the image data for expressing the mouth when pronouncing the vowel, an image representing the mouth when pronouncing the vowel is displayed subsequent to the image for expressing the half-open mouth.

The half-open mouth is, for example, "A
(a) ”and“ e (e) ”.

In the invention for solving the third problem, the storage means stores image data for expressing a half-open mouth in addition to image data for expressing a mouth when a vowel is pronounced. If the image display means identifies that consecutive syllables to be generated by the sound generation means include the same vowel, based on the image data for expressing the half-open mouth stored in the storage means, An image expressing a half-open mouth is displayed between images for expressing a mouth that pronounces each successive syllable.

[0018] The invention for solving the above fourth problem is characterized in that, in the storage means, in addition to image data for expressing a mouth when a vowel is pronounced, one or more combinations of different vowels are associated with each combination. Image data for expressing an intermediate mouth opening method between two vowels is stored, and the image display unit stores the vowels in the storage unit when causing the sound generation unit to generate continuous syllables including different vowels. On the basis of the obtained image data, an image for expressing the way of opening the mouth between these two types of vowels is displayed between the images for expressing the mouths that emit different vowels.

In the invention for solving the fifth problem, the storage means stores image data for expressing a closed mouth in addition to image data for expressing a mouth when a vowel is pronounced. The image display means displays an image based on the image data for expressing the closed mouth stored in the storage means, when the syllable to be sounded by the sound generation means includes a sound repellent.

In the present invention, the storage means corresponds to a storage device. The image display means corresponds to a processing circuit and an image display circuit. The sound generation circuit generates sound of the syllable based on waveform data provided together with image data expressing mouth movement.

In the invention for solving the second problem,
Specific syllables include 'sa', 'se', 'ta', 'ma',
'Mi', 'mu', 'me', 'mo', 'wa', 'o',
'Ba', 'bi', 'bu', 'be', 'bo', 'ぱ',
Includes some or all of 'ぴ', 'ぷ', 'ぺ' and 'ぽ'.

In the invention for solving the first problem, syllables are identified based on character data.

The syllable according to the present invention can include any sound such as a repellent sound in addition to the vowel.

The recording medium in the present invention includes, for example, a floppy disk, a hard disk, a magnetic tape, a magneto-optical disk, a CD-ROM, a DVD, a ROM cartridge, a RAM memory cartridge with a battery backup, a flash memory cartridge, and a nonvolatile RAM.
Including cartridges.

The communication medium also includes a communication medium such as a wired communication medium such as a telephone line and a wireless communication medium such as a microwave line. The Internet is also included in the communication medium mentioned here.

A recording medium is a medium on which information (mainly digital data and programs) is recorded by some physical means, and which causes a processing device such as a computer or a dedicated processor to perform a predetermined function. Can be done. In short, any method may be used as long as the program is downloaded to the computer by some means and a predetermined function is executed.

[0027]

DESCRIPTION OF THE PREFERRED EMBODIMENTS In a preferred embodiment of the present invention, the image sound processing apparatus of the present invention is applied to a game machine. Hereinafter, description will be made with reference to the drawings.

(Apparatus Configuration) FIG. 1 shows an external view of a game apparatus according to the present embodiment. As shown in FIG. 1, the game device of the present embodiment includes a game device main body 1, a pad 2b, and a monitor device 5.

The game apparatus body 1 houses therein the image and sound processing apparatus of the present invention. On its exterior,
Interface for connecting ROM cartridge (I /
F) A section 1a and a CD-ROM drive 1b for reading a CD-ROM are provided. The pad 2b includes a cross cursor and a plurality of operation buttons, and is configured to be able to generate an operation signal based on a player's operation. The operation signal can be supplied to the connector 2a via the cable 2c. The pad 2b is detachably provided to the game apparatus main body 1 by the connector 2a, and is configured such that two pads can be connected so that two players can operate at the same time. Note that a mouse, a remote controller, a keyboard, or the like may be connected instead of the pad. The monitor device 5
It is connected to a video output terminal and an audio output terminal (not shown) of the game apparatus main body 1 via a video cable 4a and an audio cable 4b.

(Block Configuration) FIG. 2 shows a block diagram of a game apparatus to which the image and sound processing apparatus of the present invention is applied.
As shown in FIG. 2, the game device has a CPU block 1
0, a video block 11, a sound block 12, and a subsystem 13.

(CPU Block) CPU Block 10
Is a part and a processing circuit of the image display means of the present invention, which advances the game processing according to the program and controls the image sound processing of the present invention.
CU (System Control Unit) 100, main CPU 1
01, RAM 102, ROM 103, sub CPU 104
And a CPU bus 105 and the like.

The main CPU 101 has a DSP inside.
(Digital Signal Processor), CD-ROM1
It is configured to be able to execute a process based on the program data transferred from the PC at high speed. RAM 102 is a CD-R
The program data of the application software read from the OM1, the sound control program data, the waveform data, and the image data for expressing the shape of the mouth that sounds each syllable are stored. Also, the RAM 102 stores the MP
It is configured to be usable as a work area when decoding an EG image and as an error correction data cache when decoding a CD-ROM. The ROM 103 is configured to be able to store initial program data used for initialization processing of the device. The SCU 100 includes buses 105 and 106
And 107 are configured so as to be able to control the transfer of data performed through them. The SCU 100 has a DM inside.
An A controller is provided so that image data required during the execution of a game stored in the RAM 102 or the like can be transferred to the VRAM in the video block 11 and sound control program data and waveform data can be transferred to the sound block 12. ing.

The sub CPU 104 is an SMPC (System Man
ager & Peripheral Control), the main CPU
An operation signal from the pad 2b can be collected in response to the request of 101.

(Configuration of Video Block) The video block 11 is a part of the image display means and the image display circuit of the present invention.
0, a VDP 130, a VRAM 121, frame buffers 122 and 123, a VRAM 131, and a memory 132.

The VRAM 121 is configured to be able to store drawing commands transferred by the main CPU 101 via the SCU 100.

The VDP 120 generates bitmap image data, deforms graphics, performs color calculations such as shadows and shading, etc., based on the drawing commands stored in the VRAM 121, and transfers the generated image data to the frame buffer 122 and the frame buffer 122. 123 is configured to be writable.

The frame buffers 122 and 123 are
The image data generated by the VDP 120 can be stored.

The VRAM 131 is configured to store image data of a background image, a data table necessary for realizing the functions of the VDP 130, and the like.

The VDP 130 is based on image data stored in the VRAM 131 and the frame buffers 122 and 123, and performs window processing for setting a selection frame, shadowing processing, enlargement / reduction, rotation, mosaic processing, moving processing, clipping, Image processing such as hidden surface processing such as display priority processing is performed, and display image data can be stored in the memory 132.

The memory 132 stores image data for drawing by the VDP 130 and can output the image data to the encoder 160.

The encoder 160 converts the drawing image data stored in the memory 132 into a video signal format, D / A converts the data, and supplies it to the monitor device 5.

The monitor device 5 is configured to display an image based on the supplied video signal.

(Sound Block) Sound Block 1
Reference numeral 2 denotes a sound generation unit and a sound generation circuit of the present invention, which are constituted by the DSP 140 and the CPU 141.

The CPU 141 can transfer the sound control program data and the waveform data transferred from the main CPU 101 to the DSP 140. DSP1
Reference numeral 40 is configured to include an acoustic memory. Then, under the control of the CPU 141, the PC
Performs waveform generation, delayed data generation, and voice synthesis using an M sound source or an FM sound source, and generates the D / D
It is configured to be able to output to the A converter 170. DS
The P140 has functions such as frequency control, volume control, FM calculation, modulation, speech synthesis, reverb, etc. by these actions. The D / A converter 170 is configured to convert the waveform data generated by the DSP 140 into a two-channel signal and supply the signal to the speakers 5a and 5b.

The waveform data includes a human voice. The sound control program data includes a character code indicating this human syllable. This code is, for example, lyrics data of a song sung by a model to be displayed on an image.

(Subsystem) The subsystem 13 has a C
D-ROM drive 1b, CD interface (I /
F) 180, CPU 181, MPEG audio circuit 1
82 and an MPEG video circuit 183.

The CD-ROM drive 1b has a CD-RO
M. The program data, image data, sound control program data and waveform data of the application software are read from M.
It is configured to be able to be supplied to the PU block 10. Also,
The CPU 181 is configured to control the MPEG audio circuit 182 and the MPEG video circuit 183 so as to be able to decode the image data and the audio data that have been highly efficient code-compressed according to the MPEG standard.

(Explanation of Principle) Next, the principle of the present invention will be described. In the game of the present embodiment, a model that simulates an actual idle is constituted by so-called polygons and displayed. For example, as shown in FIG. 8, the face of the model is composed of a large number of minute triangles or quadrangular polygons. When displaying a certain image, prior to the next display timing (for example, vertical synchronization timing), the main CPU 101 sets coordinate data for specifying the spatial position of each polygon in the world coordinate system and polygons for specifying each vertex coordinate and the like. The data is transferred to the VRAM 121 of the video block 11. In video block 11,
The VDP 120 generates a segment such as a model face as shown in FIG. 8 with reference to these data, and applies (pastes) texture data as a pattern to the surface of each polygon in a bitmap format. In addition, VDP120
Performs a shading process for smoothing the surface of the segment to complete an image of the segment that resembles an actual idle as shown in FIG. This image is converted by the encoder 160 into a video signal that matches the display timing, and the model face is displayed on the monitor device 5.

By performing the above processing while updating the spatial coordinates of the polygon every frame period (period for displaying one screen), moving the mouth, closing the eyelids, Display the image of the model that moves the.

(Expression of mouth shape by vowel) When a human actually pronounces one syllable, the period during which a vowel is sounding is relatively long during the sounding period. In the present invention, based on this principle, it is determined what the vowel of the syllable to be sounded by the sound block 11 is, and an image for expressing the mouth when the vowel is pronounced is displayed simultaneously with the pronunciation of this syllable. . That is, the image display corresponding to each of the vowels “A (a)”, “I (i)”, “U (u)”, “E (e)”, and “O (o)” is shown in FIG. Through FIG.
It becomes as shown in.

[0051] taken as an example the word "Hello". If the length of each syllable are equal between pronounced: Based sound block to the waveform data "Hello", the displayed image would have moved to say "On'iia" as shown in FIG. 4 .

According to such processing, the vowel corresponding to each syllable is displayed during the sounding period of the syllable, thereby allowing the player to approximately recognize that the model is sounding the syllable. Can be.

[0053] (representation of the shape of the mouth in the case where the same vowel syllables are consecutive) By the way, referring to FIG. 4, successive of the word "Hello" and "to" and "blood" is,
It contains the same vowel "i (i)". For this reason, the mouth displayed as an image does not move in the same shape over these two syllables, which may be unnatural.

In order to cope with this, in the present invention, when the vowel of the syllable to be sounded previously and the vowel of the syllable to be sounded next time are the same, an image representing the mouth in a half-open state is further inserted. For example, as shown in FIG. 5, in the first half of the second syllable of syllables related to the same vowel, a state of an intermediate mouth between “A” and “E” as shown in FIG. 15 is expressed. Then, insert the image of the mouth in the half-open state.

According to such processing, even if the same vowel is continuous in two consecutive syllables, a different mouth shape is displayed between both vowels, so that a more natural mouth Can express movement.

The place where the half-opening is inserted is not limited to the beginning of the second syllable in two consecutive syllables, but may be after the first syllable. Also, both the end of the first syllable and the beginning of the second syllable may be applied. Further, the length of inserting the image representing the half-open mouth is not limited to half the length of the syllable, but may be shorter or longer. Further, the shape of the half-open mouth only needs to be different from the shape of the mouth when producing any vowel.
In addition to the mouth in the middle state between "A" and "E" as in 5,
Other mouth shapes such as "A" and "O" may be used.

(Expression of mouth shape in specific syllable) Further, when examining the mouth shape when each syllable shown in FIG. 5 is actually pronounced, the syllable of "wa (wa)" sounds the consonant at the beginning of the syllable. The shape of the mouth when performing and the shape of the mouth when pronouncing its vowel "A" are significantly different. As described above, some syllables in which the shape of the mouth of a consonant and the shape of the mouth of a vowel greatly change even in one syllable are listed in Table 1.

[0058]

[Table 1]

If such a syllable is expressed in the form of a mouth when the vowel is pronounced, it becomes unnatural. Therefore, in the present invention, when the syllables to be pronounced are the syllables listed in Table 1,
A half-open mouth is displayed at the beginning of the syllable. For example,
As shown in FIG. 6, an image expressing a half-open mouth as shown in FIG. 15 is displayed in the first half of the syllable of "wa".

According to such processing, even in a syllable in which the shape of the mouth greatly changes between the consonant and the vowel of the syllable, the half-open mouth shape is displayed at the beginning of the syllable. Can be expressed.

The shape of the half-open mouth can be variously changed, as in the case where consecutive syllables contain the same vowel.

(More Natural Expression of Mouth Shape Between Consecutive Syllables Containing Different Vowels) On the other hand, even when consecutive syllables contain different vowels, the vowel in the first syllable is pronounced. The shape of the mouth and the shape of the mouth when the vowel in the next syllable is pronounced may be significantly different.
In addition, consonants are included between the two syllables, and the shape of the mouth of a human who actually pronounces these two syllables is different from that in which different vowels are simply produced continuously. Therefore, when simulating a mouth that pronounces a continuous syllable containing different vowels, not only an image for expressing the mouth that pronounces these two different vowels, but also these two types of images By inserting an image that gives an intermediate expression of the mouth that produces these two types of vowels, a more natural human mouth movement can be simulated.

Therefore, according to the present invention, when producing continuous syllables including different syllables, as shown in FIG. 7, for example, as shown in FIG. Displays an image that represents the way the mouth opens between two vowels. For example, in the figure,
This shows the case of simulating the pronunciation of "Makura". Since the first syllable “ma” corresponds to the special syllable, a half-open mouth as shown in FIG. 15 is displayed. In the remaining syllables, the middle mouth of the vowel included in the preceding and following syllables is displayed. In terms of the combination of vowels included in each syllable, "ma" and "ku" and "ku" and "ra" are the same combination of "a" and "u" because they are the same combination. Display an image that represents the middle mouth of.

Note that image data for expressing an intermediate way of opening the mouth is prepared according to a combination of preceding and following syllables. For example, basically, the image data prepared is
The sum of the mouth that expresses the five vowels “A”, “I”, “U”, “E”, and “O”, the mouth that expresses the sound-repelling that represents “N”, and the mouth that expresses the special half-open state When expressing seven types of mouth shapes, image data is prepared by the number of combinations for selecting two from these seven types.
That is, ₇ C ₂ = 21 patterns are prepared.

However, this intermediate mouth image may be inserted only in a combination that looks unnatural when successively displaying mouth images that produce different vowels. For example, the shape of the mouth does not change much between the vowels "A" and "E", so if syllables containing these two different vowels are continuous, it is not necessary to insert an intermediate mouth image .

Further, instead of the above-described image for expressing an intermediate mouth shape of different vowels, an image for expressing another intermediate mouth may be used. For example, an image for expressing another intermediate mouth that can simulate the shape of the mouth when the consonant included in the later syllable is generated may be used.

The period during which the image of the intermediate mouth is displayed is not limited to the first half of the second syllable of two consecutive syllables.
It may be the second half of the first syllable. Also, both the rear part of the first syllable and the front part of the second syllable may be applied. Further, the length of inserting an image representing an intermediate mouth is not limited to half the length of a syllable, but may be shorter or longer.

(Expression of mouth shape when syllable is plucked) In the syllable, a syllable called a plucked sound, that is, a syllable of "n (n)" does not include any vowel. Therefore, in the present invention, when the syllable is sound-repellent, FIG.
As shown in FIG. 6, an image with the mouth closed is displayed.

According to such processing, an image in which the mouth is closed is displayed even when the syllable is sound-repellent, so that a natural movement of the mouth can be expressed.

Table 2 summarizes the above processes in relation to syllables and images displayed in correspondence with the syllables.

[0071]

[Table 2]

(Explanation of Operation) Next, processing based on the above principle will be described with reference to the flowchart of FIG.

Step S1 (Initial Setting): At the start of the game, the program data for the main CPU 101, the image data, the sound control program data for the CPU 141, and the waveform data are read from the cartridge interface circuit 1a or the CD-ROM, and stored in the RAM.
Transferred to 102. Hereinafter, the main CPU 101
The main CPU 101 executes the main CPU program stored in the M 102, and the CPU 141 executes the acoustic control program transferred by the main CPU 101 and stored in the DSP 140.

Step S2: The main CPU 101
The lyrics data of the sound control program data transferred to the sound block 11 is read at each image update timing. By successively referring to the characters indicated by the lyrics data, it is possible to determine what syllable is to be sounded next from the sound block 11.

For this reason, the main CPU 101 reads the syllable character to be pronounced next from the lyrics data, and refers to the table data defining the correspondence as shown in Table 2 included in the program data. Then, the following image data for expressing the mouth that emits the vowel specified from this table, that is, a set of coordinate data for specifying the spatial position of the polygon stored in the RAM 102 is transferred to the video block 11, and is generated. To express the mouth of the model corresponding to the syllable.

Step S3: First, it is determined whether or not the syllable generated by the waveform data transferred to the sound block 11 is a specific syllable listed in Table 1. If the syllable is a specific syllable (S3; YES), an image representing the half-open mouth is displayed to express the half-open mouth (S4), and the process proceeds to step S8. That is,
The coordinate data defining the spatial position of the polygon for expressing the half-open mouth is transferred to the video block 11, and
The mouth is displayed as shown in. The time length for displaying the half-open mouth may be set to a different time length for each specific syllable, or may be set to a fixed time length. Alternatively, it may be set by a ratio of a length of displaying a half-open mouth to a length of a syllable.

Step S5: If the syllable is not a specific syllable (S3; NO), it is discriminated whether or not the syllable to be generated last time and the syllable to be newly generated this time include the same vowel.
If the same vowel is included, the image data for displaying the half-open mouth is converted to the video block 1 as in step S3.
1 and the image representing the half-open mouth is displayed in step S
4 is displayed (S6), and the process proceeds to step S8. The time length for displaying the half-open mouth is the same as that in step 4 above.

Step S7: If the syllable that was generated last time and the syllable that is newly generated this time do not include the same vowel (S5; NO), it is considered that the vowels included in the preceding and following syllables are different. Therefore, the main CPU 101 converts image data that can represent an intermediate mouth between the sound (vowel) included in the previous syllable and the sound of the syllable to be generated this time (either a vowel, a repellent sound, a half-open sound, or a consonant) into video. Transfer to block 11. Further, the main CPU 101 transfers the waveform data for generating the syllable to the sound block 12. Note that the length of time for displaying an intermediate mouth is the same as in step 4 above.

Step S8: After displaying an image representing an intermediate mouth (S4, S6 or S7), it is determined whether or not the next syllable to be sounded is sound-repelling. If the sound is sound-repellent (S8; YES), the coordinate data defining the spatial position of the polygon constituting the mouth for expressing the closed mouth is transferred to the video block 11, and the mouth as shown in FIG. 16 is displayed. (S9).

If it is not a sound repellent (S8; NO), the next syllable is considered to include a vowel, so the shape of the mouth for producing the vowel is specified, and the spatial position of the specified polygon is defined. The coordinate data is transferred to the video block 11, and a mouth as shown in any of FIGS. 10 to 14 is displayed (S10).

Step S10: If the lyrics data still continues (S11; NO), the next lyrics data is referred to again (S2). When the lyrics data is over (S11; Y
ES), end the processing of the present invention.

(Effect) As described above, according to the present embodiment, the expression of the mouth for generating the vowel included in the syllable to be pronounced next is used instead of the expression of the mouth in the syllable. With a small amount of image data, it is possible to express a model that moves the mouth naturally according to the pronunciation of syllables.

In particular, even when the same vowel is continuous in two consecutive syllables, a half-open mouth shape different from any vowel is displayed between the two syllables, so that a more natural movement of the mouth can be expressed. .

Further, even when two consecutive syllables have different vowels, an image representing an intermediate mouth shape is displayed between the images representing the mouths producing the vowels. Therefore, a more natural movement of the mouth can be expressed.

Further, since the image in which the mouth is closed is displayed even when the syllable is sound-repellent, a more natural movement of the mouth can be expressed.

(Other Modifications) The present invention can be variously modified irrespective of the above embodiments. For example, in the above embodiment, the lyrics data is referred to assuming a case where the idol sings. However, the words may be spoken instead of the lyrics. At this time, in step S2, the shape of the mouth is determined based on data representing the contents of the dialogue in syllabic characters such as Hiragana or Katakana or Roman characters instead of the lyrics data.

In the above-described embodiment, the model is a human face. However, the model is not limited to humans. Various applications are possible.

In the above embodiment, the model is displayed by polygons. However, a plurality of bitmap image data may be stored according to the shape of the mouth, and these image data may be alternately read and displayed. In this case, the present invention can be applied to two-dimensional display.

[0089]

According to the present invention, there is provided an image sound processing apparatus having an expressive power capable of naturally simulating the manner in which a human actually pronounces and giving the illusion that the model is really singing or speaking. A recording medium can be provided.

That is, according to the present invention, the image of the mouth when the vowel corresponding to each syllable is generated is displayed during the sounding period of the syllable, so that the model sounds approximately. Images can be displayed. Since the image data is only vowels, only a certain amount of image data is required.

According to the present invention, even when syllables having the same vowel are consecutive, it is possible to naturally simulate the manner in which a human pronounces sound.

According to the present invention, even if syllables containing different vowels are consecutive, an image representing an intermediate mouth shape is inserted between images representing mouths that emit each of two kinds of vowels. Therefore, it is possible to more naturally simulate how a human pronounces.

According to the present invention, even when the syllable is sound-repelling, an image with the mouth closed is displayed, so that a natural movement of the mouth when the sound-repelling is pronounced can be expressed.

[Brief description of the drawings]

FIG. 1 is an external view of a game device to which an image sound processing device of the present invention is applied.

FIG. 2 is a block diagram of a game device provided with the image sound processing device of the present invention.

FIG. 3 is a flowchart illustrating the operation of the present embodiment.

FIG. 4 is a display order of the case to represent the "Hello" only in vowels.

FIG. 5 shows a display order when syllables including the same vowel are continuous.

FIG. 6 shows a display order in the case of a specific syllable.

FIG. 7 is a display order in a case where preceding and succeeding syllables include different sounds.

FIG. 8 is a configuration diagram of a model composed of only polygons.

FIG. 9 is a display image diagram of a model to which texture data is applied.

FIG. 10 is an example of a display image for producing “a”.

FIG. 11 is an example of a display image for generating “i (i)”.

FIG. 12 is an example of a display image for generating “u (u)”.

FIG. 13 is an example of a display image for producing “e (e)”.

FIG. 14 is an example of a display image for generating “O (o)”.

FIG. 15 is an example of a display image representing a half-open mouth.

FIG. 16 is an example of a display image expressing a closed mouth (sound repellency).

[Description of Signs] 10: CPU block (part of image display means, processing circuit) 11: Video block (part of image display means, image display circuit) 12: Sound block (sound generation means, sound generation circuit) 102 ... RAM (part of storage means, part of storage circuit) 121 ... VRAM (part of storage means, part of storage circuit)

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁶ Identification code FI G10L 3/00 G06F 15/62 350A

Claims (17)

[Claims] 1. An image sound processing apparatus for generating sound in accordance with an image representing mouth movement, a sound generating means for generating sound, and expressing mouth movement when a syllable is pronounced. Image display means for displaying an image, and storage means for storing image data for expressing a mouth when a vowel is pronounced only for the type of vowel, wherein the image display means comprises the sound generation means Identifying vowels included in syllables to be pronounced, reading image data for expressing a mouth when the identified vowel is pronounced from the storage means, and displaying an image based on the read image data. Image sound processing device. 2. The storage means stores image data for expressing a half-open mouth in addition to image data for expressing a mouth when the vowel is pronounced. When the syllable to be generated by the sound generating means is a predetermined specific syllable, the image data for expressing the half-open mouth stored in the storage means and the vowel included in the syllable are generated. 2. The image sound processing apparatus according to claim 1, wherein an image representing the mouth when the vowel is pronounced is displayed subsequent to the image representing the half-open mouth based on the image data for representing the mouth at the time. . 3. The storage means stores image data for expressing a half-open mouth in addition to image data for expressing a mouth when the vowel is pronounced. If it is determined that the continuous syllables produced by the sound generating means contain the same vowel, the continuous syllables are identified based on the image data for expressing the half-open mouth stored in the storage means. The image acoustic processing apparatus according to claim 1, wherein an image representing the half-open mouth is displayed between images for representing a mouth that pronounces each syllable. 4. The storage means includes, in addition to image data for expressing a mouth when the vowel is pronounced, an intermediate of two types of vowels related to each combination in one or more combinations of different vowels. Image data for expressing a natural mouth is stored, When the image display means causes the sound generation means to generate continuous syllables including different vowels, based on the image data stored in the storage means, The image acoustic processing according to claim 1, wherein an image for expressing a way of opening an intermediate mouth between these two types of vowels is displayed between images for expressing a mouth that pronounces each of the different vowels. apparatus. 5. The storage means stores image data for expressing a closed mouth in addition to image data for expressing a mouth when the vowel is pronounced, wherein the image display means comprises: 2. The image acoustic processing apparatus according to claim 1, wherein when a syllable to be sounded by the sound generating means includes a sound repelling, an image is displayed based on the image data for expressing the closed mouth stored in the storage means. 6. An image sound processing apparatus for generating a sound in accordance with an image expressing a movement of a mouth, wherein image data for expressing a mouth when a vowel is pronounced, waveform data for generating a sound, and a program A storage device in which data is stored; a sound generation circuit that generates a voice based on the waveform data read from the storage device; and an image expressing mouth movement based on the image data read from the storage device. An image display circuit for displaying, and a processing circuit for controlling the sound generation circuit and the image display circuit based on program data read from the storage device, wherein the processing circuit is sounded by the waveform data For identifying a vowel included in the syllable, generating a sound based on the waveform data, and expressing a mouth when the vowel is pronounced; The serial image data transferred to the image display circuit, image sound processing apparatus characterized by displaying an image based on the image data. 7. The storage device stores image data for expressing a half-open mouth in addition to image data for expressing a mouth when the vowel is pronounced. When the syllable pronounced by the waveform data is a predetermined specific syllable, image data for expressing the half-open mouth stored in the storage device while generating a sound based on the waveform data. And transferring image data for expressing the mouth when the vowel included in the syllable is to be generated to the image display circuit, and expressing the mouth when the vowel is generated following the image expressing the half-open mouth. 7. An image to be displayed is displayed.
3. The image acoustic processing device according to claim 1. 8. The storage device stores image data for expressing a half-open mouth, in addition to image data for expressing the mouth when the vowel is pronounced. If it is determined that consecutive syllables produced by the waveform data contain the same vowel, a sound based on the waveform data is generated,
The image data for expressing the mouth when the vowel included in the syllable included in the syllable stored in the storage device and the image data for expressing the half-open mouth are transferred to the image display circuit, The image sound processing apparatus according to claim 6, wherein an image representing the half-open mouth is displayed between images for representing a mouth that pronounces each syllable. 9. The storage device may include, in addition to image data for expressing a mouth when the vowel is pronounced, an intermediate of two types of vowels related to each combination in one or more combinations of different vowels. Image data for expressing a vowel is stored.When the processing circuit causes continuous syllables including different vowels to sound, the processing circuit converts each of the different vowels based on the image data stored in the storage device. 7. The image sound processing apparatus according to claim 6, wherein an image for expressing a way of opening an intermediate mouth between these two types of vowels is displayed between images for expressing a mouth to be pronounced. 10. The storage device stores image data for expressing a closed mouth in addition to image data for expressing a mouth when the vowel is pronounced. When a syllable pronounced by the waveform data includes a sound repellent, a sound based on the waveform data is generated, and image data for expressing the closed mouth stored in the storage device is transferred to the image display circuit. The image sound processing apparatus according to claim 6, wherein an image representing the closed mouth is displayed. 11. The specific syllable includes 'sa',
'Set', 'ta', 'ma', 'mi', 'mu', 'me',
'Well', 'Wa', 'O', 'B', 'Bi', 'Bu',
The image according to claim 2, wherein the image includes part or all of “be”, “bo”, “ぱ”, “ぴ”, “ぷ”, “ぺ”, and “ぽ”. Sound processing device. 12. The image acoustic processing apparatus according to claim 1, wherein the syllable is identified based on character data. 13. A computer for identifying a vowel included in a syllable pronounced by the waveform data, generating a sound based on the waveform data, and an image for expressing a mouth when the vowel is pronounced. Transferring data to an image display circuit; and displaying an image based on the image data;
A machine-readable recording medium in which program data for executing the program is stored. 14. A computer for identifying whether or not a syllable pronounced by waveform data is a predetermined specific syllable; and when the specific syllable is identified, the computer outputs Generating sound based on the image data, and, when the specific syllable is identified, image data for expressing a half-open mouth and image data for expressing a mouth when a vowel included in the syllable is to be pronounced. Transferring the image to the image display circuit, and displaying an image representing the mouth when the vowel is pronounced, following the image representing the half-open mouth,
A machine-readable recording medium in which program data for executing the program is stored. 15. A computer, comprising: determining whether or not continuous syllables produced by waveform data include the same vowel; and determining if the continuous syllable includes the same vowel. Generating sound based on the waveform data; and, when identifying that the continuous syllables include the same vowel, image data for expressing a mouth when the vowel included in the syllable is to be pronounced. And transferring image data for expressing the half-open mouth to an image display circuit; and an image for expressing the half-open mouth between images for expressing the mouth that pronounces each of the continuous syllables. And a machine-readable recording medium storing program data for executing the steps. 16. A computer for identifying whether or not continuous syllables produced by the waveform data include different vowels. If the computer determines that the continuous syllables include different vowels, Generating sound based on the image data, and, when the consecutive syllables are identified as including different vowels, image data for expressing a mouth when each vowel is pronounced and an opening of an intermediate mouth between the different vowels Transferring image data for expressing one of the two vowels to an image display circuit, and opening an intermediate mouth between these two types of vowels between the images for expressing the mouths producing each of the different vowels Displaying an image representing the following, and a machine-readable recording medium storing program data for executing the image. 17. A computer, comprising: determining whether a syllable pronounced by waveform data includes a sound-repelling sound; and generating a sound based on the waveform data when the syllable is determined to include a sound-repelling sound. Transferring the image data for expressing the closed mouth to an image display circuit when the syllable includes a sound-repelling sound; anddisplaying an image representing the closed mouth,
A machine-readable recording medium in which program data for executing the program is stored.