JP2007079624A - Utterance detection device, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately detect a speaker's utterances, without being affected by sudden factors, such as noise. <P>SOLUTION: A lip pattern is segmented from a speaker's image (Step 108), and correlation values are calculated between lip pattern f(t) and a lip-inclusive pattern F(t-i) (i = 1, 2, ..., N) (Step 109). A maximum correlation value s_max(t, t-i) is calculated between f(t) and F(t-i), and such correlation values s_max with i from 1 to N are summed as a lip variation E(t) (Step 110). A lip variation E(t) that is not less than the threshold (Step 111: YES) identifies utterance duration (Step 112), and a lip variation E(t) less than the threshold (Step 111: NO) identifies non-utterance duration (Step 113). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an utterance detection device, method, and program, and more particularly, to an utterance detection device, method, and program for detecting an utterance from an image of a speaker's lips.

  A speech recognition system in a general environment is affected by ambient noise and the like, and performs speech recognition based on the noise even when the speaker is not speaking, resulting in erroneous recognition. As an effective method for reducing misrecognition, a method of detecting a speaker's utterance section and performing speech recognition only in the meantime can be considered.

Therefore, in order to improve the speech recognition rate, it has been studied to detect an utterance interval from the movement of the speaker's lips. According to Patent Document 1, opening / closing of a mouth is detected from a difference between a vertical distance of a lip contour and a reference value, or a curvature value of a lip contour, and a speaker is specified from a plurality of subjects. Are listed. Non-Patent Document 1 describes that the utterance state is determined from the difference between the current lip pattern and the lip pattern N frames before.
JP 2000-338987 A Murai, Nakamura, “Detecting Conversation Consistent with Noise from Mouth Images”, Spoken Language Information Processing 37-10, 2001

  In general, when a speaker has a conversation, the position of the speaker's head varies within a certain range, and as a result, the appearance of the speaker on the image also varies. Therefore, it is difficult to robustly detect the utterance section from the lip image.

  In the case of Patent Document 1, since the outline of the lips changes during utterance, the reference value itself fluctuates, and there is a problem that the utterance state cannot be detected with high accuracy. In the case of Patent Document 2, since the utterance state is determined by paying attention only to the difference between the current lip pattern and the lip pattern N frames before, there is a problem that it is susceptible to sudden fluctuation factors such as noise.

  The present invention has been proposed to solve the above-described problem, and an utterance detection apparatus, method, and program for detecting a speaker's utterance with high accuracy without being affected by sudden factors such as noise. The purpose is to provide.

  An utterance detection apparatus according to the present invention specifies at least an imaging unit that images a lip of a speaker and a lip feature region that can specify a shape of the lip in an image frame continuously captured by the imaging unit. Lip feature area specifying means; and lip inclusion area specifying means for specifying a lip inclusion area including a lip whose shape is specified by the lip feature area in image frames continuously captured by the imaging means; Among the image frames continuously captured by the imaging means, the lip shape specified by the lip feature region in a specific image frame, and one or a plurality of continuous image frames captured immediately before the specific image frame The correlation value calculating means for comparing the lip shape included in the lip inclusion region of the lip and calculating the correlation value thereof, and the correlation value calculating means A fluctuation amount calculating means for calculating the amount of fluctuation of the lips based on the calculation result, and an utterance interval detecting means for detecting whether or not the utterance interval is based on the fluctuation amount calculated by the fluctuation amount calculating means; It has.

  The speaker detection method according to the present invention captures at least a speaker's lips, specifies a lip feature region in which the shape of the lips can be specified in the continuously captured image frames, and continuously In the captured image frame, the lip inclusion region including the lip whose shape is specified by the lip feature region is specified, and the lip feature region in the specific image frame in the continuously captured image frames And the lip shape included in the lip inclusion region in one or a plurality of continuous image frames captured immediately before the specific image frame, and calculating a correlation value thereof, A variation amount of the lips is calculated based on the correlation value calculation result, and it is detected whether or not it is an utterance section based on the calculated variation amount.

  The speaker detection program according to the present invention causes a computer to image at least a speaker's lips, and to specify a lip feature region capable of specifying the shape of the lips in the continuously captured image frames, In a continuously captured image frame, a lip inclusion region including a lip whose shape is specified by the lip feature region is specified, and the specific image frame in the continuously captured image frame The lip shape specified by the lip feature region is compared with the lip shape included in the lip inclusion region in one or a plurality of consecutive image frames captured immediately before the specific image frame, and the correlation value thereof is calculated. Whether or not the lip variation amount is calculated based on the correlation value calculation result, and whether or not the utterance section is based on the calculated variation amount To detect.

  The imaging means is installed so that the lips of the speaker can be imaged. The lip feature region specifying unit specifies a lip feature region that can specify the shape of the lips in continuously captured image frames. The lip inclusion region specifying unit specifies a lip inclusion region including the lip whose shape is specified by the lip feature region in continuously captured image frames.

  The correlation value calculation means includes a lip shape specified by the lip feature region in a specific image frame in continuously captured image frames, and one or a plurality of continuous images captured immediately before the specific image frame. The lip shape included in the lip inclusion region in the image frame is compared, and the correlation value thereof is calculated. Note that “one or a plurality of consecutive image frames captured immediately before a specific image frame” does not have to be all image frames continuous from the specific image frame, and may be an arbitrary image frame. .

  The fluctuation amount calculation means calculates the fluctuation amount of the lips based on the calculation result of the correlation value calculation means. When the speaker is not speaking, the lip shape specified by the lip feature region in a specific image frame, and the lip inclusion region in one or a plurality of continuous image frames captured immediately before the specific image frame There is almost no amount of variation between the lip shape and the amount of variation when the speaker is speaking. Therefore, the amount of lip variation can be obtained from the amount of variation.

  Therefore, the speech segment detection means can detect whether or not the speech segment is based on the calculated fluctuation amount.

  Note that the invention further includes a cutting-out unit that cuts out a density pattern that represents the characteristics of the speaker's lips and tracks and identifies the shape of the lips from the image frames continuously captured by the imaging unit. You may prepare. At this time, if the lip feature region specifying unit specifies a rectangular region including the density pattern cut out by the cutting unit, which is an image frame continuously picked up by the imaging unit, as the lip feature region. Good. At this time, the density pattern may be at least a gray value of one of the mouth corner, upper lip, and lower lip of the speaker.

  The utterance detection device, method, and program according to the present invention capture a lip shape specified by the lip feature region in a specific image frame in an image frame continuously captured, and immediately before the specific image frame. The lip shape included in the lip inclusion region in one or a plurality of consecutive image frames is compared, the correlation value thereof is calculated, and whether or not it is an utterance section is detected based on the calculated variation amount By doing so, it is possible to detect an utterance section with high accuracy without being affected by sudden factors such as noise.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram showing a configuration of a speech recognition system according to an embodiment of the present invention. The speech recognition system includes a CCD image sensor 1 that captures a speaker at 30 frames per second, an A / D converter 2 that performs analog / digital conversion on an image captured by the CCD image sensor 1, and an A / D converter 2. An image processing apparatus 10 that detects an utterance of a speaker by performing image processing based on image data, a microphone 21 to which the voice of the speaker is input, and an A / D converter that performs analog / digital conversion of the voice from the microphone 21 22 and a speech recognition device 30 that performs speech recognition based on speech detected by the image processing device 10 and speech data.

  The CCD image sensor 1 is installed so that an image of the speaker's face or the periphery of the speaker's mouth can be taken. The microphone 4 is installed at a position where a speaker's voice is input.

  The image processing apparatus 10 includes a CPU 11 that performs image processing, a RAM 12 that is a work area, and a ROM 13 that stores a control program for the CPU 11. The ROM 13 stores an utterance section detection routine program and other programs.

  FIG. 2 is a flowchart showing a speech segment detection routine of the image processing apparatus 10. The image processing apparatus 10 executes the following process of step 101 in order to detect a speaker's speech section.

  In step 101, when a speaker image is input from the CCD image sensor 2, the CPU 11 of the image processing apparatus 10 proceeds to step 102.

  In step 102, the CPU 11 determines whether or not the speaker's lip feature pattern is registered (stored in the RAM 12).

  3A and 3B are diagrams illustrating an example of the lip feature pattern. The lip feature pattern is a density pattern for tracking and specifying the shape of the lips in the image. A density pattern is a pattern of luminance levels for each pixel of an image. The lip feature pattern may be a density pattern at two positions (mouth corners) of the mouth as shown in FIG. 3A, or a density pattern at two positions of the upper lip and lower lip as shown in FIG. 3B. The lip feature pattern is not limited to two places as described above, but may be a combination of FIGS. 3A and 3B, or may be a density pattern of three or more places.

  If the lip feature pattern has already been registered, the process proceeds to step 106 in order to track the lip feature pattern for the currently captured image. On the other hand, when the lip feature pattern is not registered, the process proceeds to step 103 to register the lip feature pattern. The lip feature pattern is not limited to the RAM 12 and may be stored in a storage medium such as a non-volatile RAM or a magnetic disk (not shown).

  In step 103, the CPU 11 detects a characteristic pattern of the lips (lip characteristic pattern) based on the image supplied from the A / D converter 2, and proceeds to step 104. In the detection of the lip feature pattern, various image processing methods can be used.

  For example, by preparing a neural network that responds to the lip feature pattern of the horn corner shown in FIG. 3A, and applying the neural network to the input image, the lip feature pattern of FIG. Can be easily detected. Alternatively, the mouth corner may be specified from the edge histogram distribution of the image, and the lip feature pattern may be registered.

  In step 104, the CPU 11 determines whether or not the lip feature pattern has been detected. If it has been detected, the process proceeds to step 105. If not, the process returns to step 101.

  In step 105, the CPU 11 registers the lip feature pattern in the RAM 12 and proceeds to step 106.

  In step 106, the CPU 11 tracks the lip feature pattern from the current frame image based on the position of the lip feature pattern in the previous frame image, and proceeds to step 107.

  FIG. 4 is a diagram illustrating a search range of the lip feature pattern. First, the CPU 11 sets a search range indicated by a rectangular dotted line based on the position of the lip feature pattern of the previous frame image. Then, an area having a pattern most similar to the lip feature pattern registered in advance within the search range is detected. Here, for example, a normalized correlation processing method which is one of image processing methods can be used.

  In step 107, the CPU 11 determines whether or not the tracking of the lip feature pattern is successful. If successful, the CPU 11 proceeds to step 108, and if not successful, returns to step 101 again. Note that tracking of the lip feature pattern may fail due to the speaker moving his head violently. Whether or not it has failed can be determined by examining the correlation value obtained by the normalized correlation processing.

  In step 108, the CPU 11 cuts out the lip pattern from the image that has been successfully tracked, and proceeds to step 109. The lip pattern is a rectangular pattern that surrounds the lips, unlike the lip feature pattern that is an image pattern with gray values. That is, the lip pattern is a rectangular region of lip features that can specify the shape of the lips.

  FIG. 5 is a diagram for explaining lip pattern extraction. Here, the center coordinates of the two lip feature patterns being tracked (in this embodiment, the density patterns of the two mouth corners) are (x1, y1) and (x2, y2) (where x2> x1). At this time, the CPU 11 converts the rectangular area having the width (x2−x1) · r1, the height (x2−x1) · r2, and the center coordinates ((x1 + x2) / 2, (y1 + y2) / 2) into the lip pattern f (t). As long as you ask. r1 and r2 are coefficients set in advance for cutting out the lip pattern f (t).

  When tracking the lip feature pattern shown in FIG. 3B, for example, if the center coordinates of the lip feature pattern are (x1, y1) and (x2, y2), the width W, the height H, and the center coordinates ( The rectangular area of (x1 + x2) / 2, (y1 + y2) / 2) may be used as the lip pattern f (t). That is, f (t) may be set according to the lip feature pattern to be tracked.

  Next, the CPU 11 cuts out the lip inclusion pattern F (t) from the image so as to include the lip pattern f (t). That is, the lip inclusion pattern F (t) is a rectangular region that includes the lip whose shape is specified by the lip pattern f (t).

  FIG. 6 is a diagram for explaining the clipping of the lip inclusion pattern F (t). For example, the center position of the lip pattern f (t) is (x0, y0), and the width and height of the lip pattern are W0 and H0, respectively. At this time, the CPU 11 may obtain the lip inclusion pattern F (t) having the center position (x0, y0), the width W0 · r0, and the height H0 · r0. r0 is a coefficient (> 1) for cutting out the lip inclusion pattern F (t).

  In step 109, the CPU 11 compares the cut lip pattern f (t) and the lip inclusion pattern F (ti) (i = 1, 2,..., N) for the previous N frames from the present. A correlation value is calculated. Thereby, the lip shape specified by the lip pattern f (t) is compared with the lip shape specified by the lip inclusion pattern F (ti) (i = 1, 2,..., N). And a correlation value is calculated.

  Specifically, the CPU 11 performs image correlation processing using the lip pattern f (t) as a reference (template) image and the lip inclusion pattern F (ti) as a search image, and performs a correlation value image s (f (t)). , F (t−1)). Here, if the image size W0 · H0 of the lip pattern f (t) and the image size of the lip inclusion pattern F (t) are W1 · H1 (W1> W0, H1> H0), the correlation value image s ( The image size of f (t), F (t−i) is (W1−W0) · (H1−H0). Note that a known technique such as a normalized correlation processing method may be used as the image correlation processing method. However, the correlation value s takes a value from 0 to 1, and is normalized so that the higher the correlation (the more similar), the larger the value.

  Hereinafter, the correlation value at the coordinates (x, y) of the correlation value image s (f (t), F (t−i)) is represented as si (x, y). However, x = 0, 1,... (W1-W0-1) and y = 0, 1,..., (H1-H0-1).

Next, the CPU 11 calculates the maximum value s_max (t, i) of si (x, y) and the coordinates (sx (t, i), sy (t, i)) at that time. At this time,
s_max (t, ti) = si (sx (t, i), sy (t, i))
It is.

  7A is a diagram illustrating an example of a reference image and a search image, FIG. 7B is a diagram illustrating a state in which the reference image is scanned in the search image, and FIG. 7C is a diagram illustrating the most correlation between the reference image and the search image. The figure showing a high position and (d) are figures showing an example of a correlation value image. Here, a quadrangular pyramid image is used instead of the speaker's lip image.

  The CPU 11 calculates the correlation value between the reference image and the search image and searches for the position where the correlation value is the highest, whereby the reference image is scanned in the search image as shown in FIG. 7B, and FIG. As shown in FIG. 5, the matching position between the reference image and the search image is found. At this time, as shown in FIG. 7D, the correlation value image at the matching position is obtained as s_max. Note that the search image (lip inclusion pattern F (t−i)) used in the image correlation process preferably has N frames as follows.

  FIG. 8 is a diagram for explaining image correlation processing between the lip pattern f (t) and the lip inclusion pattern F (t−i) (i = 1,..., N). First, image correlation processing is performed with f (t) and F (t−1), and the highest correlation value s_max (t, 1) is obtained. Next, image correlation processing is performed using f (t) and F (t−2), and the highest correlation value s_max (t, 2) is obtained. Similarly, image correlation processing is performed using f (t) and F (t−3), and the highest correlation value s_max (t, 3) is obtained. The N correlation values s_max obtained in this way are used in the next step S110.

If the lip inclusion patterns F (t) and F (t−1) are the same image,
s_max (t, ti) = 1
sx (t, i) = (W1-W0) / 2
sy (t, i) = (H1-H0) / 2
become.

  In step 110, the CPU 11 obtains {s_max (t, ti), sx (t, i), sy (t, i); i = 1, 2,. .., N}, the lip variation E (t) is calculated from the equation (1).

  FIG. 9 is a diagram showing the lip feature pattern, the lip pattern f, and the lip inclusion pattern F obtained at (a) time t, (b) time t + 1, (c) time t + 2, and (d) time t + 3, respectively. In FIG. 9, two frames are traced back from the current frame, but the number of frames that are traced back is not particularly limited. Each step mentioned above is demonstrated using FIG.

  First, the lip inclusion pattern F (t) at time t shown in FIG. 9A is obtained as follows. The lip feature pattern for detecting the lip pattern f (t) is registered in advance as a characteristic pattern of the lip feature pattern, for example, a rectangular pattern with a left and right mouth corner (steps 102 to S105). Then, a lip pattern f (t) represented by a rectangular dotted line is detected from the position of the lip feature pattern (steps 107 and S108). Next, a lip inclusion pattern F (t) including the lip pattern f (t) is set. Here, the images of F (t−2) and F (t−1) indicate the lip inclusion patterns obtained at the times t−2 and t−1, respectively. Then, for each of F (t−2) and F (t−1), an image region having a high similarity to the lip pattern f (t) at time t is searched by correlation processing.

  FIG. 10 is a diagram for explaining a search for a pattern having the highest similarity with the lip pattern f (t).

  As shown in the figure, correlation processing is performed between the lip pattern f (t) and the lip inclusion pattern F (t-1), and the pattern represented by the solid line rectangle has the highest correlation. . The correlation value at this time (correlation is higher as it is closer to 0 to 1: 1) is expressed as s (t, t−1). The amount of deviation of the position with respect to the lip pattern f (t−1) is represented as (Δx (t, t−1), Δy (t, t−1)). The shift amounts are | xp−xq | and | yp−yq |, where the upper left coordinate values of the rectangle in the drawing are (xp, yp) and (xq, yq).

If the speaker does not move the lips at all, ideally F (t) and F (t-1) are exactly the same image pattern,
s (t, t-1) = 1.0,
(Δx (t, t−1), Δy (t, t−1)) = (0, 0)
It becomes.

  Lip variation amount E (t) is defined by equation (1) described above. From this definition, when it is considered that the mouth is not moving (not speaking), the lip variation E (t) is a small value. When it is considered that the mouth is moving greatly (speaking) from time t-N to t, the lip variation E (t) becomes a large value. Therefore, based on the lip variation E (t), it is determined whether or not it is an utterance section as follows.

  In step 111, the CPU 11 compares the lip fluctuation amount E (t) calculated by the equation (1) with a preset threshold Eth, and if E (t)> Eth, the CPU 11 proceeds to step 112. If E (t) ≦ Eth, the routine proceeds to step 113.

  In step 112, the CPU 11 determines that the current frame is an utterance section, and proceeds to step 114.

  In step 113, the CPU 11 determines that the current frame is not an utterance section, and proceeds to step 114.

  In step 114, CPU11 transmits the determination result of an utterance area to the speech recognition apparatus 30, and complete | finishes a process. Then, the process of step 101 is executed again. As a result, the voice recognition device 30 can perform voice recognition on the voice data supplied from the A / D converter 22 while considering the determination result of the image processing device 10, that is, whether or not it is a speech segment. , The recognition rate can be improved.

  As described above, the speech recognition system according to the embodiment of the present invention includes the lip pattern f (t) obtained from the registered lip feature pattern and the lip inclusion pattern F (t−i) traced back N frames. ) And the lip fluctuation amount calculated from the correlation value. As described above, since the amount of change up to N frames before is used for detecting the utterance period, the voice recognition system is not affected even when the speaker's lips change minutely. An utterance section can be detected reliably.

  Also, the speech recognition system does not simply calculate the correlation value between patterns, but determines the position with the highest correlation by the correlation process between the current lip pattern f (t) and the past lip inclusion pattern F (t). Therefore, it can be reduced that the lip feature pattern fluctuates and the lip pattern cut-out error is affected.

  Furthermore, the voice recognition system does not use a complicated image processing method for obtaining lip color information, contour information, end point information, and the like, and uses a rectangular pattern lip pattern f (t) and lip from the density pattern and its position information. Since the inclusion pattern F (t) is calculated, it is possible to detect an utterance section at high speed while reducing the calculation load.

  And since the speech recognition system performs speech recognition using the speech data of the speaker while taking into consideration the determination result of whether or not it is the speech section, the recognition rate is improved and speech recognition is performed with high accuracy. be able to.

  Note that the present invention is not limited to the above-described embodiment, and it is needless to say that the present invention can also be applied to a design modified within the scope of the claims.

  For example, the CPU 11 of the image processing apparatus 10 may use any of the following expressions (2), (3), and (4) instead of the expression (1).

In the equations (2) and (3), the lip variation E (t) is calculated without using the distance information Δd. In equations (3) and (4), a weighting coefficient α (i) corresponding to the frame is used. Further, the following formula (5) may be used.

  FIG. 11 is a diagram for explaining another image correlation process of the lip pattern f (t) and the lip inclusion pattern F (ti). As shown in the figure and formula (5), f (t) and F (t-1), f (t-1) and F (t-2), f (t-2) and F (t-3) The image correlation process may be performed between That is, the highest correlation value is calculated between the lip pattern f (t) and the lip inclusion pattern F (t-1) of the immediately preceding frame, and the lip variation E (t) is calculated from the sum of the highest correlation values. May be calculated.

  Further, the CPU 11 may thin out the frames by using i = 1, 3, 5,..., N as the frames used for calculating the lip variation E (t). Thereby, processing time can be shortened and an utterance area can be detected.

It is a figure which shows the structure of the speech recognition system which concerns on embodiment of this invention. 3 is a flowchart showing a speech section detection routine of the image processing apparatus 10; It is a figure which shows an example of a lip feature pattern. It is a figure which shows the search range of a lip feature pattern. It is a figure explaining extraction of a lip pattern. It is a figure explaining extraction of the lip inclusion pattern F (t). (A) is a diagram showing an example of a reference image and a search image, (b) is a diagram showing a state in which the reference image is scanning the search image, and (c) is a position having the highest correlation between the reference image and the search image. (D) is a figure showing an example of a correlation value image. It is a figure explaining the image correlation process of the lip pattern f (t) and the lip inclusion pattern F (t-i). It is a figure which shows the lip feature pattern, the lip pattern f, and the lip inclusion pattern F obtained at (a) time t, (b) time t + 1, (c) time t + 2, and (d) time t + 3, respectively. It is a figure for demonstrating searching for the pattern with the highest similarity with the lip pattern f (t). It is a figure explaining the other correlation processing of the lip pattern f (t) and the lip inclusion pattern F (t-i).

Explanation of symbols

1 CCD image sensor 2, 22 A / D converter 10 Image processing device 11 CPU
12 RAM
13 ROM
21 Microphone 30 Voice recognition device

Claims (9)

Imaging means for imaging at least the speaker's lips;
Lip feature region specifying means for specifying a lip feature region capable of specifying the shape of the lips in the image frames continuously captured by the imaging unit;
Lip inclusion area specifying means for specifying a lip inclusion area including a lip whose shape is specified by the lip feature area in image frames continuously captured by the imaging means;
The lip shape specified by the lip feature region in a specific image frame in the image frames continuously captured by the imaging means, and one or a plurality of continuous image frames captured immediately before the specific image frame A correlation value calculating means for comparing the lip shape included in the lip inclusion region in the inside and calculating these correlation values;
A fluctuation amount calculating means for calculating a fluctuation amount of the lip based on a calculation result of the correlation value calculating means;
Utterance interval detection means for detecting whether or not the utterance interval is based on the fluctuation amount calculated by the fluctuation amount calculation means;
An utterance detection device comprising: The image processing apparatus further includes a cutting-out unit that cuts out a density pattern for representing the characteristics of the lip of the speaker and tracking and specifying the shape of the lip from image frames continuously captured by the imaging unit
The lip feature region specifying unit specifies a rectangular region including the density pattern cut out by the cutout unit, which is an image frame continuously picked up by the image pickup unit, as the lip feature region. The utterance detection device according to claim 1. The correlation value calculating means includes the lip shape specified by the lip feature region in the specific image frame, and the arbitrary image frame of a plurality of consecutive image frames captured immediately before the specific image frame. The utterance detection device according to claim 1, wherein the highest correlation value with each lip shape included in the lip inclusion region is calculated. The utterance detection device according to claim 2, wherein the density pattern is a gray value of at least one of a mouth angle of the speaker, an upper lip, and a lower lip. Image at least the speaker's lips,
In the continuously captured image frames, specify a lip feature region that can specify the shape of the lips,
In the continuously captured image frames, specify a lip inclusion region that includes a lip whose shape is specified by the lip feature region;
The lip shape specified by the lip feature region in a specific image frame in the continuously captured image frames, and the one or more consecutive image frames captured immediately before the specific image frame Compare the lip shape contained in the lip inclusion area, calculate these correlation values,
Calculate the amount of lip variation based on the correlation value calculation result,
An utterance detection method comprising: detecting whether or not an utterance section is based on the calculated fluctuation amount. Further, from the continuously captured image frames, a density pattern for representing the characteristics of the speaker's lips and tracking and specifying the shape of the lips is cut out,
6. The lip feature region is specified by specifying, as the lip feature region, a rectangular region that is the continuously captured image frame and includes the extracted density pattern. Utterance detection method. In the correlation value calculation, the lip shape specified by the lip feature region in the specific image frame and the lip of an arbitrary image frame among a plurality of continuous image frames captured immediately before the specific image frame. The utterance detection method according to claim 5, wherein the highest correlation value with each lip shape included in the inclusion region is calculated. The utterance detection method according to claim 6, wherein the density pattern is a gray value of at least one of a speaker's mouth corner, upper lip, and lower lip. On the computer,
At least image the lips of the speaker,
In the continuously captured image frames, the lip feature region that can specify the shape of the lips is specified,
In the continuously captured image frames, a lip inclusion region including a lip whose shape is specified by the lip feature region is specified,
The lip shape specified by the lip feature region in a specific image frame in the continuously captured image frames, and the one or more consecutive image frames captured immediately before the specific image frame Compare the lip shape included in the lip inclusion area, calculate these correlation values,
Based on the correlation value calculation result, to calculate the amount of lip variation,
An utterance detection program for detecting whether or not an utterance section is based on the calculated fluctuation amount.