JP4459415B2 - Image processing apparatus, image processing method, and computer-readable information storage medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus, an image processing method, and a computer-readable information storage medium that generate an image of an object whose mouth movement or the like changes in response to sound.
[0002]
[Prior art]
In fields such as voice-over of animation and live-action video, it is necessary to draw mouth movements according to the voice of the characters in order to realistically express the facial expressions of the characters. For example, in the case of an animation image, as a technique for expressing the movement of the mouth, a technique in which the creator of the original picture manually determines the shape of the mouth and its surroundings may be used. While listening to the voice of the character in a scene or reading the dialogue prepared for this scene, the creator imagines the movement of the character's mouth and creates a mouth shape. A certain amount of experience and intuition are required to reproduce the shape.
[0003]
In addition, when voice of a live-action video is dubbed, for example, when voice in English is dubbed into Japanese, Japanese Patent No. 2795084 assumes that the periphery of a person's mouth in a live-action video is deformed in accordance with Japanese voice. A mouth shape image synthesizing device disclosed in the publication is known. In this device, the contents of speech after dubbing are analyzed to extract information on phonemes and durations, and the shape of the mouth and its surroundings are determined based on these information.
[0004]
[Problems to be solved by the invention]
By the way, in the method of determining the shape of the character's mouth and its surroundings by the above-mentioned manual work, because it depends on the skill level and ability of each worker such as the worker's experience and intuition, work efficiency is poor, Moreover, since different results are obtained for each worker, there is a problem that it is difficult to generate a natural expression with low reproducibility.
[0005]
In addition, in the method of determining the shape of the mouth and its surroundings based on phoneme and duration information, it is necessary to extract phoneme and duration information based on the contents of speech as preprocessing, and differs for each phoneme. Since it is necessary to register the mouth shape pattern in advance, there is a problem that the processing load is heavy and the data capacity necessary for the processing increases. For example, the phoneme included in the voice can be extracted by the voice recognition process. To perform this voice recognition process, a voice pattern corresponding to each phoneme is registered in advance and the actual voice is analyzed. The pattern obtained in this way is compared with each registered pattern, and the phoneme corresponding to the registered pattern with the highest similarity is extracted. Therefore, a registration pattern is required for each phoneme to be extracted, and a processing load necessary for searching for a registration pattern having a high degree of similarity becomes heavy.
[0006]
In addition, when generating an image of the entire head, if an attempt is made to obtain a natural expression, it is necessary to open and close not only the mouth shape described above but also the wrinkle at an appropriate timing, and the processing load is reduced. There is a demand for a method that can produce a light and natural expression.
[0007]
The present invention has been created in view of the above points, and its purpose is to provide an image processing apparatus capable of obtaining a natural expression, reducing the processing load, and reducing the required storage capacity. An image processing method and a computer-readable information storage medium are provided.
[0008]
[Means for Solving the Problems]
  In order to solve the above-described problems, the image processing apparatus of the present invention includes frequency analysis means, mouth shape analysis means, and coordinate correction means. The frequency analysis means performs frequency analysis on the input voice and detects a sound pressure level for each of a plurality of frequency components. The mouth shape setting means sets the mouth shape based on the sound pressure level for each of the plurality of frequency components detected by the frequency analysis means. The coordinate correcting means corrects the coordinates of a plurality of polygons constituting the three-dimensional object of the head so that the mouth shape set by the mouth shape setting means is obtained.Further, the mouth shape setting means uses the sound pressure level value for each of a plurality of frequency components to calculate the respective values of the upper lip position, the lower lip position, and the mouth corner position as mouth shape parameters. Set the shape.Since the voice to be uttered and the mouth shape are closely related, a more natural expression can be obtained by detecting the sound pressure level for each frequency component of the input voice and reflecting it in the mouth shape. Moreover, since the same mouth shape is always obtained for the same input voice, high reproducibility can be realized. In addition, since the mouth shape can be set only by performing frequency analysis on the input speech, the processing load is reduced as compared with the conventional method in which phonemes have to be extracted by speech recognition processing. Furthermore, since a dictionary necessary for the speech recognition process is unnecessary and it is not necessary to register a mouth shape pattern corresponding to each phoneme, the storage capacity of a memory or the like can be reduced.
[0009]
In addition, it is desirable to give texture information in which a pattern is set in advance to each polygon constituting the three-dimensional object by the texture mapping means. Since the mouth shape corresponding to the input sound is set and the vertex coordinates of each polygon are corrected, texture mapping is performed on these polygons, so that the shape of the mouth and its surroundings is changed. Images can be easily obtained.
[0010]
  Further, the mouth shape setting means described above calculates the mouth shape parameter value necessary for specifying the mouth and the shape of the periphery of the mouth, using the sound pressure level values for each of the plurality of frequency components. Set the shapeing. By defining a relational expression between the mouth shape parameter and each sound pressure level in advance, the mouth shape can be easily set simply by calculating the value of the mouth shape parameter using this relational expression. The burden can be greatly reduced.
[0011]
  In particular, the mouth shape parameters described above are parameters corresponding to the positions of the upper lip, the lower lip, and the mouth corner, respectively.Is included,By setting each of these positions, the mouth shape can be specified.
  In addition, the above-described mouth shape parameter is set to an upper limit value or a lower limit value for each of the upper lip position, the lower lip position, and the mouth corner position, and the result calculated using the value of the sound pressure level is the upper limit value. It is desirable to set the parameter value to the upper limit value or the lower limit value when the value exceeds the lower limit value. Considering the actual movement of the mouth and the structure around the mouth, it is considered that there are restrictions on the movement of the upper lip, the lower lip, or the mouth corner. Therefore, it is possible to prevent the mouth shape from becoming unnatural by moving the upper lip, the lower lip, or the mouth corner within a predetermined range. For example, when a sound having an extremely large sound pressure level of each frequency component is input, it is possible to prevent the mouth shape from being impossible in practice.
[0012]
Further, the wrinkle parameter setting means sets a wrinkle parameter indicating the timing to move the wrinkle based on the sound pressure level detected by the frequency analysis means, and the coordinate correcting means described above corrects the coordinates of the polygon related to the mouth shape. At the same time, it is desirable to correct the coordinates of the polygon related to the opening / closing of the eyelid based on the eyelid parameter. Considering the actual human head, the mouth closes and opens with the movement of the mouth when uttering some sound. , You can get a more natural look.
[0013]
Alternatively, the correction of the polygon coordinates in accordance with the opening / closing timing of the eyelid using the eyelid parameters described above may be performed separately from the processing related to the mouth shape. In this case, the image processing apparatus of the present invention includes specific frequency component analysis means, wrinkle parameter setting means, and coordinate correction means. The specific frequency component analyzing means detects the sound pressure level of the specific frequency component included in the input voice. The wrinkle parameter setting means sets a wrinkle parameter indicating the timing for moving the wrinkle based on the sound pressure level of the specific frequency component detected by the specific frequency component analyzing means. The coordinate correcting means corrects the coordinates of a plurality of polygons constituting the three-dimensional object of the head so as to open and close the eyelid based on the eyelid parameter set by the eyelid parameter setting means. Since the opening / closing timing of the bag is set in conjunction with the sound pressure level of the specific frequency component included in the input voice, a natural expression close to the actual expression of the person can be generated.
[0014]
In particular, the frequency component of the sound pressure level used for setting the wrinkle parameter described above is desirably a frequency component that is not linked to the movement of the mouth. It can be prevented that the movement of the heel and the movement of the mouth are always linked and unnatural. In addition, since the 瞼 parameter is set based on the sound pressure level of a specific frequency component and the timing for moving the 瞼 is determined, other calculations such as moving the 瞼 at random timing are unnecessary, and processing Simplification is possible.
[0015]
  Further, the image processing method of the present invention includes:An image processing method in an image processing apparatus comprising a frequency analysis means, a mouth shape setting means, and a coordinate correction means, wherein the frequency analysis meansA first step of performing frequency analysis on the input speech to detect a sound pressure level for each of a plurality of frequency components;In the first step by frequency analysis meansMouth shape based on the sound pressure level for each frequency component detectedBy mouth shape setting meansA second step to set,In the second step, the mouth shape setting meansThe coordinates of the polygons that make up the 3D object of the head so that the mouth shape is set.Correction by coordinate correction meansAnd a third step toThe mouth shape setting means calculates the mouth shape by calculating the values of the upper lip position, the lower lip position, and the mouth corner position as mouth shape parameters using the sound pressure level values for each of a plurality of frequency components. Set uping. Also,The image processing apparatus further includes a habit parameter setting means,In the first stepBy frequency analysis meansBased on the detected sound pressure level, the 瞼 parameter indicating when to move 瞼に よ っ て By parameter setting meansA fourth step of setting, and in the third step:Coordinate correction meansWhile correcting the coordinates of the polygon related to the mouth shape, the coordinates of the polygon related to the opening / closing of the eyelid may be corrected based on the eyelid parameter set in the fourth step.
[0016]
  The information storage medium of the present invention isBased on the sound pressure level for each of the plurality of frequency components detected by the frequency analyzing means for performing frequency analysis on the input sound and detecting the sound pressure level for each of the plurality of frequency components. A mouth shape setting means for setting the mouth shape, and a coordinate correcting means for correcting the coordinates of a plurality of polygons constituting the three-dimensional object of the head so as to be the mouth shape set by the mouth shape setting means. The mouth shape setting means uses a sound pressure level value for each of a plurality of frequency components, and each value of the position of the upper lip, the position of the lower lip, and the position of the mouth corner The mouth shape is set by calculating as a mouth shape parameter.
[0017]
By implementing the image processing method of the present invention or by executing a program stored in the information storage medium of the present invention, a natural mouth shape reflecting the contents of the input voice can be formed. In particular, since the same mouth shape is always obtained for the same input voice, it is possible to realize an image having high reproducibility and having a natural expression. In addition, since the mouth shape can be set only by performing frequency analysis on the input speech, the processing load is reduced as compared with the conventional method in which phonemes have to be extracted by speech recognition processing. Furthermore, since a dictionary necessary for the speech recognition process is unnecessary and it is not necessary to register a mouth shape pattern corresponding to each phoneme, the storage capacity of a memory or the like can be reduced.
[0018]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment to which the present invention is applied will be described in detail with reference to the drawings.
FIG. 1 is a diagram showing a configuration of an image processing apparatus according to an embodiment to which the present invention is applied. An image processing apparatus 100 illustrated in FIG. 1 includes an analog-digital (A / D) conversion unit 10, an object control unit 20, and an image generation unit 30.
[0019]
The A / D converter 10 receives an analog audio signal, samples the audio signal at a predetermined sampling frequency, and outputs digital audio data at predetermined time intervals.
The object control unit 20 performs processing for calculating the shape of a three-dimensional object that simulates the human head. For this purpose, the object control unit 20 includes a frequency analysis unit 22, a mouth shape parameter conversion unit 24, a heel parameter conversion unit 26, and a coordinate correction unit 28.
[0020]
The frequency analysis unit 22 performs frequency analysis of input voice based on voice data input at predetermined time intervals. For example, the frequency analysis unit 22 divides the audible frequency band into 64, and detects the sound pressure level of the input sound for each divided region.
The mouth shape parameter conversion unit 24 sets the value of the mouth shape parameter necessary for determining the shape of the mouth and its surroundings included in the head based on the frequency analysis result by the frequency analysis unit 22. For example, in the present embodiment, the following five mouth shape parameters Pa to Pe are used.
[0021]
1) Parameter Pa indicating the amount of change in the upper lip direction
2) Parameter Pb indicating the amount of change in the front lip direction
3) Parameter Pc indicating the amount of change in the lower lip direction
4) Parameter Pd indicating the amount of change in the forward direction of the lower lip
5) Parameter Pe indicating the amount of change in the lateral direction of the mouth corner
The heel parameter conversion unit 26 sets the value of the heel parameter indicating the movement of the heel included in the head, that is, the timing to close the heel, based on the frequency analysis result by the frequency analysis unit 22. Since the facial expression becomes unnatural when the movement of the eyelid is linked to the movement of the mouth, in this embodiment, the value of the eyelid parameter is set based on the sound pressure level of the frequency component that does not seem to be directly related to the movement of the mouth. Set and determine when to close the bag.
[0022]
In order to reflect the mouth shape determined by the mouth shape parameter conversion unit 24 and the movement of the eyelid determined by the eyelid parameter conversion unit 26, the coordinate correction unit 28 reflects the polygons constituting the three-dimensional object simulating the head. A process of changing the vertex coordinates as necessary is performed. For example, a state in which no voice is input and the bag is opened is “normal”. For a predetermined range around the mouth and its surroundings, the amount of change in the vertex coordinates of each polygon based on the normal state is calculated based on the values of the five mouth parameters Pa to Pe described above. Further, when the timing at which the eyelid moves is instructed by the eyelid parameter conversion unit 26, the amount of change in the vertex coordinates of each polygon necessary for reproducing a series of motions of the eyelid from the normal state until the eyelid is closed and reopened is obtained. Calculated.
[0023]
In addition, the image generation unit 30 performs processing necessary for actually displaying the image of the three-dimensional object on which the vertex coordinates of each polygon are calculated by the object control unit 20 on the display device 110. For this purpose, the image generation unit 30 includes a perspective projection conversion unit 32, a texture mapping unit 34, and a frame memory 36.
[0024]
The perspective projection conversion unit 32 performs perspective projection conversion based on a predetermined viewpoint position set in advance. Thereby, a two-dimensional image (pseudo three-dimensional image) obtained by viewing a three-dimensional object arranged in a virtual three-dimensional space from a predetermined viewpoint position is obtained. The texture mapping unit 34 performs a texture mapping process and a shading process for associating a texture image with each polygon included in the two-dimensional image obtained by the perspective projection conversion unit 32. A specific example of the configuration of the texture mapping unit 34 will be described later. After the texture mapping process and the shading process are performed, the two-dimensional image is stored in the frame memory 36, read out in a predetermined order, and displayed on the screen of the display device 110.
[0025]
FIG. 2 is a diagram illustrating a detailed configuration of the texture mapping unit 34. As shown in FIG. 2, the texture mapping unit 34 includes a mapping coordinate calculation unit 50, a shape memory 52, a texture adding unit 54, a texture memory 56, and a shading correction unit 58.
[0026]
The mapping coordinate calculation unit 50 calculates the correspondence between each polygon and texture constituting the three-dimensional object. The shape memory 52 stores information regarding each polygon. The texture imparting unit 54 imparts a texture to each polygon. The texture memory 54 stores information regarding each texture. The shading correction unit 58 corrects the brightness of the texture image assigned to each polygon from the position of the light source and the environmental conditions surrounding the three-dimensional object. For example, when a shadow is generated by unevenness such as a nose or lips, the specific shadow is reproduced by this shading correction.
[0027]
The frequency analysis unit 22 is the frequency analysis unit, the specific frequency component analysis unit, the mouth shape parameter conversion unit 24 is the mouth shape setting unit, the heel parameter conversion unit 26 is the heel parameter setting unit, and the coordinate correction unit 28 is the coordinate correction unit. The texture mapping unit 34 corresponds to the texture mapping means.
[0028]
The image processing apparatus 100 of the present embodiment has such a configuration, and the operation thereof will be described next.
3 to 5 are diagrams showing frequency analysis results of human voice. 3 shows a frequency analysis result when the vowel “a” is uttered, FIG. 4 shows a frequency analysis result when the vowel “a” is uttered, and FIG. 5 shows a frequency analysis result when the vowel “c” is uttered. Each is shown. In these drawings, the horizontal axis corresponds to the frequency, and the audio level for each frequency band when the audible band is divided into 64 is shown.
[0029]
As shown in FIGS. 3 to 5, when looking at the frequency distribution when uttering three different vowels “a”, “b”, and “c”, there are three peaks that show a characteristic change. I understand that. The frequency bands in which these three peaks exist are defined as regions A, B, and C in order from the low frequency side.
[0030]
For example, the frequency distribution of the vowel “a” shown in FIG. 3 shows that the sound level corresponding to the region A and the region B is large and the sound level corresponding to the region C is small. Further, when the frequency distribution of the vowel “I” shown in FIG. 4 is seen, it can be seen that the sound level corresponding to the region A and the region C is large and the sound level corresponding to the region B is small. Furthermore, when the frequency distribution of the vowel “c” shown in FIG. 5 is seen, it can be seen that the sound level corresponding to the region A is very large and the sound levels corresponding to the regions B and C are small.
[0031]
As described above, when each of the three vowels “a”, “b”, and “c” is observed, the sound levels in the three regions A to C have different tendencies. Since the sound to be uttered and the mouth shape are closely related, if the difference in tendency of these voice levels can be reflected in the mouth and its surrounding shape, the mouth and its surrounding shape according to the frequency distribution of the input speech Can be reproduced.
[0032]
6 and 7 are diagrams showing the shape of the mouth and its surroundings when the vowel “a” is uttered. FIG. 6 shows a state where a person's face is viewed from the front, and FIG. 7 shows a state where the person's face is viewed from the right side.
Similarly, FIG. 8 and FIG. 9 are diagrams showing the shape of the mouth and its surroundings when the vowel “I” is uttered. FIG. 8 shows a state when a human face is viewed from the front, and FIG. 9 shows a state when the human face is viewed from the right side.
[0033]
10 and 11 are diagrams showing the shape of the mouth and its surroundings when the vowel “U” is uttered. FIG. 10 shows a state when a human face is viewed from the front, and FIG. 11 shows a state when the human face is viewed from the right side.
12 and 13 are diagrams showing the shape of the mouth and its surroundings in a normal state where nothing is uttered. FIG. 12 shows a state where a human face is viewed from the front, and FIG. 13 shows a state where the human face is viewed from the right side.
[0034]
As shown in FIG. 6 to FIG. 11, when the vowels to be uttered are different when observing the shape of the mouth and its surroundings for each of the three vowels “A”, “I”, “U”, 1) the upper lip It can be seen that the position, 2) the position of the lower lip, and 3) the lateral position of the mouth corner changes. Moreover, according to the frequency analysis results shown in FIGS. 3 to 5, each vowel has a tendency that the sound pressure levels of the three frequency bands A, B, and C are different from each other. Can be used to define the calculation formulas for calculating the position of the upper lip, the position of the lower lip, and the lateral position of the corner of the mouth, the upper lip can be calculated based on the sound pressure level value of each frequency band corresponding to an arbitrary sound. , The position of the lower lip, and the lateral position of the corner of the mouth are calculated, whereby the mouth shape can be determined.
[0035]
Specifically, in this embodiment, when the sound pressure levels of the three frequency bands shown in FIGS. 3 to 5 are set to L1, L2, and L3 in order from the low frequency side, the mouth shape parameter conversion unit 24 The values of the five parameters Pa to Pe described above are calculated using the following relational expressions. The sound pressure levels L1, L2, and L3 can be obtained by accumulating the sound pressure level values for each of the 64 divided regions included in each frequency band, but an average value is used instead of the accumulated value. You may make it use.
[0036]
1) Parameter Pa indicating the amount of change in the upper lip direction:
A parameter Pa indicating the upward change amount of the upper lip position p1 shown in FIG.
Pa = A × L2 (1)
It is possible to calculate using the following relational expression. Here, A is an appropriate coefficient, for example, A = 0.5.
[0037]
2) Parameter Pb indicating the amount of change in the front lip direction:
The parameter Pb indicating the forward change amount of the upper lip position p1 shown in FIG.
Pb = B * L1-C * L2-D * L3 (2)
It is possible to calculate using the following relational expression. Here, B, C, and D are appropriate coefficients, for example, B = 1.0, C = 1.0, and D = 1.0.
[0038]
3) Parameter Pc indicating the amount of change in the lower lip direction:
The parameter Pc indicating the downward change amount of the lower lip position p2 shown in FIG.
Pc = E × L2 (3)
It is possible to calculate using the following relational expression. Here, E is an appropriate coefficient, and is set to E = 3.0, for example.
[0039]
4) Parameter Pd indicating the amount of change in the lower lip forward direction:
The parameter Pd indicating the forward change amount of the lower lip position p3 shown in FIG.
Pd = F * L1-G * L2-H * L3 (4)
It is possible to calculate using the following relational expression. Here, F, G, and H are appropriate coefficients, for example, B = 1.0, C = 1.0, and D = 1.0.
[0040]
5) Parameter Pe indicating the amount of change in the lateral direction of the mouth corner:
The parameter Pe indicating the amount of change in the lateral direction of the mouth corner position p4 shown in FIG.
Pe = I * L3-J * L1 (5)
It is possible to calculate using the following relational expression. Here, I and J are appropriate coefficients, for example, set to I = 3.0 and J = 2.0.
[0041]
Note that each of the positions p1 to p4 described above is set for convenience, and the vertical position and the like may be slightly changed around the mouth. In addition, the actual movement of the mouth is considered to be within the common sense range, but when calculated using the above-mentioned formulas (1) to (5), it is out of this common sense range. Can also happen. For example, when the sound pressure level of the input voice is very high, there is a possibility that the value of the mouth shape parameter corresponding to the mouth size which cannot be actually obtained is obtained as a calculation result. Therefore, it is desirable to set an upper limit value for each mouth shape parameter Pa to Pe so that the mouth shape does not become unnatural. When a value exceeding the upper limit value is obtained as a calculation result, the value of the mouth shape parameter is forcibly set to the upper limit value. Moreover, since the value of each mouth shape parameter is set on the basis of the normal state with the mouth closed, it is not preferable that each value be negative. Therefore, it is desirable to provide a lower limit value so that each mouth shape parameter Pa to Pe does not become negative. If a negative value is obtained as a calculation result, the value of the mouth shape parameter is forcibly set to the lower limit “0”.
[0042]
By the way, in this embodiment, the mouth shape is changed according to the content of the input voice, and the eyelid is opened and closed so as not to be interlocked with the change of the mouth shape, and the eyelid parameter indicating the opening and closing timing is set to a predetermined frequency. It is set based on the sound pressure level of the band.
[0043]
14 and 15 are diagrams showing frequency bands used for setting the wrinkle parameter. As shown in FIG. 14, the heel parameter conversion unit 26 sets a heel parameter indicating the timing for closing the heel when the sound pressure level in the frequency domain D exceeds a predetermined threshold. For example, the value of the wrinkle parameter M is set to 1 at this time. Further, as shown in FIG. 15, when the sound pressure level in the frequency domain D is smaller than a predetermined threshold, the heel parameter conversion unit 26 sets a heel parameter indicating that the heel is in an open state. . For example, the value of the wrinkle parameter M is set to 0 at this time.
[0044]
FIG. 16 is a diagram illustrating an operation procedure for generating a head image in the image processing apparatus 100 according to the present embodiment. For example, an operation procedure for generating a head image for one screen is shown.
When an analog audio signal is input, the A / D conversion unit 10 converts it to digital audio data at a predetermined sampling frequency and inputs the digital audio data to the frequency analysis unit 22. The frequency analysis unit 22 performs a predetermined frequency analysis process by performing an FFT (Fast Fourier Transform) operation based on the input voice data (step 100). Specifically, the frequency analysis unit 22 obtains the sound pressure level for each divided area obtained by dividing the audible frequency band into 64 parts, and then, for each of the three frequency bands A, B, and C shown in FIG. The sound pressure level of each divided area is accumulated, and three sound pressure levels L1, L2, and L3 used for calculating mouth shape parameters are obtained and input to the mouth shape parameter converting unit 24. Further, the frequency analysis unit 22 extracts only the sound pressure level of the specific frequency band D shown in FIGS. 14 and 15 from the sound pressure level for each divided region divided into 64, and the soot parameter conversion unit 26 To enter.
[0045]
Next, the mouth shape parameter converting unit 24 uses the five parameters Pa used to determine the shape of the mouth and its surroundings based on the three sound pressure levels L1, L2, and L3 input from the frequency analyzing unit 22. , Pb, Pc, Pd, and Pe are set (step 101). The specific calculation method is as described above, and each parameter value is obtained by a simple calculation using the equations (1) to (5).
[0046]
In parallel with or before or after the processing by the mouth shape parameter conversion unit 24, the soot parameter conversion unit 26 sets the value of the soot parameter based on the sound pressure level of the frequency band D input from the frequency analysis unit 22. (Step 102). As described above, the setting of the value of the wrinkle parameter can be performed by a simple process of examining the magnitude relationship between the sound pressure level of the frequency band D and the predetermined threshold value.
[0047]
Next, the coordinate correction unit 28 simulates the head based on the values of the mouth shape parameters Pa to Pe set by the mouth shape parameter conversion unit 24 and the value of the eyelid parameter M set by the eyelid parameter conversion unit 26. Processing for correcting the vertex coordinates of each polygon constituting the three-dimensional object is performed (step 103).
[0048]
Actually, the change amounts of the specific positions p1 to p4 are indicated by the mouth shape parameters Pa to Pe, but the correction of the vertex coordinates of each polygon is a polygon included in a predetermined range centered on these specific positions. It is performed on the subject. Focusing on the upper lip position p1 shown in FIG. 12, the range from the upper lip to the base of the nose in the vertical direction is extracted as a predetermined range including the upper lip position p1 in the horizontal direction, and a range similar to the width of the lips is extracted. The vertex coordinates of the polygon included in the predetermined range are corrected based on the parameter Pa calculated by the equation (1). The degree of correction is set so that the amount of correction decreases as the distance from the upper lip position p1 does not change the vertex coordinates of each polygon included in the predetermined range uniformly.
[0049]
Further, for the lower lip position p2 shown in FIG. 13, a process of correcting the lower lip position p2 with the predetermined position p5 as the rotation center is performed in consideration of the rotation of the entire lower jaw. Accordingly, the vertex coordinates of the polygon are corrected so that the amount of correction in the downward direction gradually decreases from the lower lip position p2 toward the rotation center position p5.
[0050]
17 to 20 are diagrams showing how the vertex coordinates of the polygon are corrected. FIG. 17 is a diagram illustrating a normal state three-dimensional object viewed from the front, and FIG. 18 is a normal state three-dimensional object viewed from the right side. FIG. 19 is a diagram showing a front view of a three-dimensional object in a state where the vowel “a” is uttered, and FIG. 20 shows a three-dimensional object in a state where the vowel “a” is uttered. It is a figure which shows a mode that was seen from the right side surface. As shown in these drawings, by correcting the positions p1 to p4, the vertex coordinates of the polygons arranged around the mouth are corrected, and each of the mouth shapes corresponding to the mouth shape in which the vowel “a” is uttered is corrected. It becomes the vertex coordinate position of the polygon.
[0051]
Next, the perspective projection conversion unit 32 performs pseudo-projection conversion on the basis of a predetermined viewpoint position for each polygon whose vertex coordinates are corrected as necessary to generate pseudo three-dimensional image data (step 104). ).
Next, the texture mapping unit 34 performs texture mapping processing (step 105) and shading processing (step 106) for associating the texture, and stores them in the frame memory 36. Thereafter, the pseudo three-dimensional image data stored in the frame memory 36 is read for each display line and displayed on the display device 110 (step 107).
[0052]
21 to 28 are diagrams illustrating specific examples of the pseudo three-dimensional image displayed on the display device 110. FIG. 21 shows an image in a normal state where nothing is uttered, and the viewpoint position is set in front of the head. FIG. 22 shows a normal state image in which the viewpoint position is set on the right side of the head. FIG. 23 shows an image in a state in which the vowel “a” is uttered, and the viewpoint position is set in front of the head. FIG. 24 shows an image in a state where the vowel “a” is uttered, and the viewpoint position is set on the right side of the head. FIG. 25 is an image in a state where the vowel “I” is uttered, and shows the case where the viewpoint position is set in front of the head. FIG. 26 shows an image in a state where the vowel “I” is uttered, and the viewpoint position is set on the right side of the head. FIG. 27 is an image in a state where the vowel “U” is uttered, and shows a case where the viewpoint position is set in front of the head. FIG. 28 is an image in a state where the vowel “U” is uttered, and shows a case where the viewpoint position is set on the right side surface of the head. As shown in these drawings, the shape of the mouth and its surroundings according to the content of the input voice can be reproduced, and an image of a three-dimensional object having a natural expression can be generated.
[0053]
As described above, the image processing apparatus 100 according to the present embodiment forms a natural mouth shape corresponding to the input sound by detecting the sound pressure level for each frequency component of the input sound and reflecting it in the mouth shape. Can do. Moreover, since the same mouth shape is always obtained for the same input voice, high reproducibility can be realized.
[0054]
In addition, since the mouth shape can be set only by performing frequency analysis on the input speech, the processing load is reduced as compared with the conventional method in which phonemes have to be extracted by speech recognition processing. Furthermore, since a dictionary necessary for the speech recognition process is unnecessary and it is not necessary to register a mouth shape pattern corresponding to each phoneme, the storage capacity of a memory or the like can be reduced.
[0055]
In particular, as shown in the equations (1) to (5) described above, this relationship can be obtained by defining in advance the relational expressions between the mouth shape parameters Pa to Pe and the sound pressure levels L1, L2, and L3. The mouth shape can be easily set simply by calculating the value of each mouth shape parameter using the equation, and the processing burden can be greatly reduced.
[0056]
In addition, the opening / closing operation of the eyelid is reproduced in parallel with the setting of the mouth shape, and a realistic three-dimensional object image having a natural expression can be generated. In particular, since the movement of the eyelid is set so as not to be interlocked with the movement of the mouth, it is possible to avoid the unnaturalness of the facial expression that occurs when these are interlocked. In addition, since the opening / closing timing of the kite is set according to the level of the sound pressure level of the specific frequency component included in the input voice, other calculations such as moving the kite at random timing are unnecessary. Also, the processing can be simplified.
[0057]
In addition, this invention is not limited to the said embodiment, A various deformation | transformation implementation is possible within the range of the summary of this invention. In the above-described embodiment, the case where the pseudo three-dimensional image corresponding to the three-dimensional object is simply displayed has been described. However, various specific applications are conceivable. For example, the present invention can be used for adding motion when creating a CG (computer graphics) animation of a face. Further, the present invention can be applied to a case where a mouth movement is displayed from voice without having facial motion data for a character appearing in a game or the like. Alternatively, the present invention can be applied to moving a person's mouth on the screen based on the voice of the other party in a videophone that displays a person's facial expression using CG.
[0058]
In the above-described embodiment, the values of the mouth shape parameters Pa to Pe are calculated using the equations (1) to (5), but the equation corresponding to each mouth shape parameter is (1). You may make it employ | adopt other than Formula-(5) Formula. For example, in the above-described embodiment, when three vowels “a”, “b”, and “c” are uttered, the mouth shape is observed to obtain a formula for calculating the mouth shape parameter. The mouth shape when uttering “o” or other consonants is observed, and a formula for calculating mouth shape parameters may be obtained. Further, the mouth shape parameters are calculated using the sound pressure levels L1, L2, and L3 of the three frequency bands A, B, and C shown in FIG. 4, but the sound pressure levels of two or more frequency bands are calculated. The mouth shape parameter may be calculated using
[0059]
In addition, the calculation formulas shown in the formulas (1) to (5) use a method of determining the contents while comparing the mouth shape when the three vowels are actually spoken with the frequency analysis result at this time. However, the relationship between the mouth shape parameters Pa to Pe and the sound pressure levels L1 to L3 may be obtained by solving simultaneous equations. For example, if each parameter is calculated by linear combination of three sound pressure levels L1 to L3, three unknown coefficients may be determined for each parameter, and the value of 15 coefficients in total may be determined.
[0060]
In the above-described embodiment, five mouth shape parameters are used to determine the mouth shape. However, a number of mouth shape parameters other than five may be used. For example, in order to determine the detailed shape around the mouth, a case where more than five mouth shape parameters are used can be considered.
[0061]
In addition, the image processing apparatus 100 according to the above-described embodiment can realize the functions corresponding to the respective components by hardware. However, in order to easily change a part of the specification, a memory or the like can be used. It may be realized by reading a program stored in the information storage medium and executing it by the CPU. In this case, each component except the A / D converter 10 and the frame memory 36 may be replaced with a computer configuration including a CPU and a memory. As an information storage medium for providing the program, in addition to a semiconductor memory such as a ROM and a RAM, an optical disk such as a CD and a DVD, a hard disk device, and the like can be used.
[0062]
Further, in the above-described embodiment, five types of parameters Pa, Pb, Pc, Pd, and Pe are defined based on the frequency analysis result for the input voice, and using these parameters, the amount of change in the upper lip direction, The mouth shape was determined by directly calculating the amount of change in the front lip direction, the amount of change in the lower lip direction, the amount of change in the lower lip direction, and the change amount in the lateral direction of the mouth corner. You may make it employ | adopt a method other than the method mentioned above about how the frequency analysis result with respect to an audio | voice is reflected in a mouth shape. Specifically, the mouth shape corresponding to the input voice may be set using the following method.
[0063]
Since the mouth shape parameter Pa is a parameter indicating the amount of change in the upper lip upward direction, a mouth shape with the mouth greatly opened up and down (for example, the mouth shape shown in FIGS. 6 and 7) corresponds to a case where the value of this parameter is large. Let For example, when the parameter Pa ′ obtained by normalizing the mouth shape parameter Pa is considered and the value is 1, the mouth shape shown in FIGS. 6 and 7 is set.
[0064]
Similarly, since the mouth shape parameter Pb is a parameter indicating the amount of change in the upper lip front direction, the mouth shape (for example, the mouth shape shown in FIGS. 10 and 11) that protrudes forward from the mouth has a large value for this parameter. Correspond to the case. For example, when the parameter Pb ′ obtained by normalizing the mouth shape parameter Pb is considered and the value is 1, the mouth shape shown in FIGS. 10 and 11 is set.
[0065]
Further, since the mouth shape parameter Pe is a parameter indicating the amount of change in the lateral direction of the mouth corner, a mouth shape (for example, the mouth shape shown in FIGS. 8 and 9) with the mouth wide open to the side is large. To correspond to. For example, when the parameter Pe ′ obtained by normalizing the mouth shape parameter Pe is considered and the value is 1, the mouth shape shown in FIGS. 8 and 9 is set.
[0066]
In this way, when the representative mouth shape is associated with each of the three parameters Pa ′, Pb ′, and Pe ′, the values of these three parameters are actually calculated corresponding to the input voice. In addition, the mouth shape is set by synthesizing images in consideration of the values of the respective parameters.
[0067]
For example, let Q be the vertex of an arbitrary polygon around the mouth. A vector having a vertex Q corresponding to the closed state shown in FIGS. 12 and 13 as a start point and a vertex Q corresponding to a state where the mouth shown in FIGS. It is defined as Further, a vector having the vertex Q corresponding to the state where the mouth is closed as the start point and the vertex Q corresponding to the state where the mouth shown in FIGS. Similarly, a vector having a vertex Q corresponding to a state where the mouth is closed as a starting point and a vertex Q corresponding to a state where the mouth shown in FIGS.
[0068]
Using the above-described three vectors Xa, Xb, and Xe, the coordinates of the vertex Q after changing the mouth shape corresponding to the input speech are calculated using the formula Pa ′ · Xa + Pb ′ · Xb + Pe ′ · Xe. can do. By performing such a calculation for all the vertices of the polygons necessary for changing the mouth shape, the mouth shape corresponding to the input voice can be set.
[0069]
【The invention's effect】
As described above, according to the present invention, a more natural expression can be obtained by detecting the sound pressure level for each frequency component of the input voice and reflecting it in the mouth shape. Moreover, since the same mouth shape is always obtained for the same input voice, high reproducibility can be realized. Moreover, since the mouth shape can be set only by performing frequency analysis on the input voice, the processing load can be reduced. Furthermore, since a dictionary necessary for the speech recognition process is unnecessary and it is not necessary to register a mouth shape pattern corresponding to each phoneme, the storage capacity of a memory or the like can be reduced.
[0070]
In addition, since the opening / closing timing of the bag is set in conjunction with the sound pressure level of the specific frequency component included in the input voice, a natural facial expression close to an actual facial expression of a person can be generated.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a configuration of an image processing apparatus according to an embodiment.
FIG. 2 is a diagram illustrating a detailed configuration of a texture mapping unit.
FIG. 3 is a diagram illustrating a frequency analysis result of a human voice.
FIG. 4 is a diagram illustrating a frequency analysis result of a human voice.
FIG. 5 is a diagram illustrating a frequency analysis result of a human voice.
FIG. 6 is a diagram showing the shape of the mouth and its surroundings when the vowel “a” is uttered.
FIG. 7 is a diagram showing the shape of the mouth and its surroundings when the vowel “a” is uttered.
FIG. 8 is a diagram showing the shape of the mouth and its surroundings when the vowel “I” is uttered.
FIG. 9 is a diagram showing the shape of the mouth and its surroundings when the vowel “I” is uttered.
FIG. 10 is a diagram showing the shape of the mouth and its surroundings when the vowel “U” is uttered.
FIG. 11 is a diagram showing the shape of the mouth and its surroundings when the vowel “U” is uttered.
FIG. 12 is a diagram showing the shape of the mouth and its surroundings in a normal state.
FIG. 13 is a diagram showing the shape of the mouth and its surroundings in the normal state.
FIG. 14 is a diagram showing a frequency band used for setting a soot parameter.
FIG. 15 is a diagram illustrating a frequency band used for setting a habit parameter.
FIG. 16 is a diagram illustrating an operation procedure for generating a head image in the image processing apparatus according to the present embodiment;
FIG. 17 is a diagram showing how the vertex coordinates of a polygon are corrected.
FIG. 18 is a diagram showing how the vertex coordinates of a polygon are corrected.
FIG. 19 is a diagram showing how the vertex coordinates of a polygon are corrected.
FIG. 20 is a diagram illustrating how the vertex coordinates of a polygon are corrected.
FIG. 21 is a diagram illustrating a specific example of displaying a pseudo three-dimensional image.
FIG. 22 is a diagram illustrating a specific example of displaying a pseudo three-dimensional image.
FIG. 23 is a diagram illustrating a specific example of displaying a pseudo three-dimensional image.
FIG. 24 is a diagram illustrating a specific example of displaying a pseudo three-dimensional image.
FIG. 25 is a diagram illustrating a specific example of displaying a pseudo three-dimensional image.
FIG. 26 is a diagram illustrating a specific example of displaying a pseudo three-dimensional image.
FIG. 27 is a diagram illustrating a specific example of displaying a pseudo three-dimensional image.
FIG. 28 is a diagram illustrating a specific example of displaying a pseudo three-dimensional image.
[Explanation of symbols]
10 Analog-digital (A / D) converter
20 Object control unit
22 Frequency analyzer
24 Mouth shape parameter converter
26 瞼 Parameter converter
28 Coordinate correction part
30 Image generator
32 perspective projection converter
34 Texture mapping section
36 frame memory
100 Image processing apparatus
110 Display device

Claims (9)

A frequency analysis means for performing a frequency analysis on the input sound and detecting a sound pressure level for each of a plurality of frequency components;
A mouth shape setting means for setting a mouth shape based on a sound pressure level for each of the plurality of frequency components detected by the frequency analysis means;
Coordinate correcting means for correcting the coordinates of a plurality of polygons constituting the three-dimensional object of the head so as to be the mouth shape set by the mouth shape setting means;
The mouth shape setting means calculates the values of the upper lip position, the lower lip position, and the mouth corner position as mouth shape parameters using the sound pressure level values for each of the plurality of frequency components. A mouth shape is set by the image processing apparatus. In claim 1,
An image processing apparatus, further comprising a texture mapping unit that gives texture information in which a pattern is set in advance to each polygon constituting the three-dimensional object. In claim 1 or 2,
In the mouth shape parameter, an upper limit value is set for each of the position of the upper lip, the position of the lower lip, and the position of the mouth corner, and the result calculated using the value of the sound pressure level indicates the upper limit value. An image processing apparatus characterized in that a parameter value is set to the upper limit value when exceeding. In claim 1 or 2,
The mouth shape parameter has a lower limit set for each of the position of the upper lip, the position of the lower lip, and the position of the mouth corner, and the result calculated using the value of the sound pressure level is based on the lower limit value. An image processing apparatus, wherein a parameter value is set to the lower limit value when the value becomes smaller. In any one of Claims 1-4,
Based on the sound pressure level detected by the frequency analysis means, further comprises a heel parameter setting means for setting a heel parameter indicating the timing for moving the heel,
The image processing apparatus, wherein the coordinate correcting means corrects the coordinates of the polygon related to the mouth shape and corrects the coordinates of the polygon related to opening / closing of the eyelid based on the eyelid parameter. Specific frequency component analysis means for detecting the sound pressure level of the specific frequency component included in the input speech;
瞼 parameter setting means for setting a 瞼 parameter indicating a timing for closing the heel when the sound pressure level of the specific frequency component detected by the specific frequency component analysis means exceeds a predetermined threshold ;
Coordinate correcting means for correcting the coordinates of a plurality of polygons constituting the three-dimensional object of the head so as to open and close the heel based on the heel parameter set by the heel parameter setting means;
An image processing apparatus comprising: An image processing method in an image processing apparatus comprising frequency analysis means, mouth shape setting means, coordinate correction means,
A first step of performing a frequency analysis on the input speech by the frequency analysis means to detect a sound pressure level for each of a plurality of frequency components;
A second step of setting a mouth shape by the mouth shape setting means based on a sound pressure level for each frequency component detected by the frequency analysis means in the first step;
A third step of correcting the coordinates of a plurality of polygons constituting the three-dimensional object of the head by the coordinate correcting means so that the mouth shape set by the mouth shape setting means in the second step;
The mouth shape setting means calculates the values of the upper lip position, the lower lip position, and the mouth corner position as mouth shape parameters using the sound pressure level values for each of the plurality of frequency components. An image processing method characterized by setting a mouth shape . In claim 7,
The image processing apparatus further includes a wrinkle parameter setting unit,
Based on the sound pressure level detected by the frequency analysis means in the first step, a fourth step is set by the heel parameter setting means indicating a heel parameter indicating the timing for moving the heel,
In the third step, the coordinate correcting means corrects the coordinates of the polygon related to the mouth shape, and coordinates of the polygon related to the opening / closing of the eyelid based on the eyelid parameter set in the fourth step. The image processing method characterized by performing correction of this. Computer
A frequency analysis means for performing a frequency analysis on the input sound and detecting a sound pressure level for each of a plurality of frequency components;
A mouth shape setting means for setting a mouth shape based on a sound pressure level for each of the plurality of frequency components detected by the frequency analysis means;
Coordinate correcting means for correcting the coordinates of a plurality of polygons constituting the three-dimensional object of the head so as to be the mouth shape set by the mouth shape setting means;
An information storage medium storing a program for causing the program to function,
The mouth shape setting means uses the sound pressure level value for each of the plurality of frequency components to calculate the mouth lip position, the lower lip position, and the mouth corner position as mouth shape parameters. A computer-readable information storage medium for setting a shape .

Images