JP2011014985A - Imaging apparatus, imaging method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To pickup images according to a fact that an object utters a prescribed keyword.SOLUTION: A digital still camera 60 includes an imaging part 61, an image processing part 62, a recording part 63, an U/I part 64, an imaging control part 65, and an automatic shutter control part 66. The imaging part 61 obtains the optical image of an object, converts it to an electric signal and outputs an image signal obtained as a result to the next stage. The imaging control part 65 controls the imaging part 61 to perform imaging according to a shutter operation signal from the U/I part 64 or an automatic shutter signal from the automatic shutter control part 66. When detecting the utterance of a shutter keyword by a person to be the object on the basis of a finder moving image input from the imaging part 61, the automatic shutter control part 66 outputs the automatic shutter signal to the imaging control part 65 corresponding to that. The invention is applied to the digital camera and the digital video camera.

Description

  The present invention relates to an imaging apparatus, an imaging method, and a program, and more particularly to an imaging apparatus, an imaging method, and a program that perform imaging in response to a subject speaking a predetermined keyword.

  Conventionally, there are cameras having an auto shutter function.

  In the old days, there are a timer type and a remote controller. In recent years, what has a so-called smile shutter function (for example, refer to Patent Document 1) that captures an image in response to a smiling face of a subject, or has a so-called wink shutter function that performs image capturing in response to a wink of a subject (for example, (See Patent Document 2).

  Japanese Patent Application Laid-Open No. 2004-228561 describes that imaging is performed according to the movement of the subject's mouth.

JP 2005-56387 A JP 2008-72183 A

  In the case of the smile shutter function described above, the degree of smile of the subject is detected by the camera, and imaging is performed when the degree exceeds a predetermined threshold. Therefore, even when the subject consciously creates a smiling expression, it is difficult to arbitrarily adjust the imaging timing.

  Further, in the case of the above-described wink shutter function, imaging is performed when a wink (opening / closing of one eye) of a subject is detected by the camera. In this case, if the subject can intentionally and reliably execute winking, it is possible to perform imaging at the timing when the subject intends. However, when the subject blinks, the camera may erroneously detect this as a wink. In addition, when the wink itself is a movement that is not normally performed for the subject, it may not be possible to wink due to embarrassment or the like.

  In addition, when taking a group photo with a plurality of people, it is necessary to inform other people who are photographed at the same time of the imaging timing by uttering, for example, “high cheese” in addition to winking.

  In addition, by providing a voice recognition function in the camera, for example, it may be possible to take an image in response to a keyword such as “Hi-Cheese”. Since it cannot be used under conditions where there is a lot of environmental noise, it is not practical.

  Furthermore, in the method of performing imaging according to the utterance of the subject (mouth movement) described in Patent Document 2 or the like, instead of detecting the utterance of a predetermined keyword, the movement of the mouth is simply detected. Therefore, nothing can be said until the desired imaging timing.

  The present invention has been made in view of such a situation, and identifies the utterance content of a subject based on a finder image, and performs imaging in response to the utterance of a predetermined keyword.

  An image pickup apparatus according to one aspect of the present invention includes an image pickup unit that outputs a finder image at the time of composition determination and outputs a recorded image at the time of image pickup, and the input of the lip image. From the finder image, a multi-class classifier that outputs a multi-dimensional score vector indicating how similar, a registration database in which modeled registration time series feature amounts are registered in association with keywords, and The lip image including the lip region of the subject is generated and input to the multi-class classifier, and the multi-dimensional score vectors corresponding to the lip image based on the finder image obtained as a result are arranged in time series. Based on the comparison result between the generating means for generating the recognition time-series feature quantity, the generated recognition time-series feature quantity, and the model registered in the registration database, And controls the image pickup means and a automatic shutter control means for executing an imaging process.

  The multi-class classifier can be generated by AdaBoostECOC learning using an image feature amount of a lip image to which a class label indicating a viseme is added.

  The image feature amount may be a pixel difference feature.

  An imaging apparatus according to one aspect of the present invention generates a lip image for registration from a finder image for registration with a subject who speaks any of the keywords as a subject, and uses the lip image for registration as the multi-class classifier. The multi-dimensional score vector corresponding to the registration lip image obtained as a result is arranged in time series to generate the registration time series feature quantity, and the registration is performed in association with the arbitrary keyword. It may further include a registering means for modeling the use time series feature quantity and registering it in the registration database.

  The registration unit may model the registration time-series feature amount using an HMM.

  An imaging method according to an aspect of the present invention includes an imaging unit that outputs a finder image at the time of composition determination and outputs a recorded image at the time of imaging, and the input of the lip image. An imaging apparatus comprising: a multi-class discriminator that outputs a multi-dimensional score vector indicating how much similarity is present; and a registration database in which modeled registration time-series feature amounts are registered in association with keywords In the imaging method, the imaging device generates the lip image including a lip region of a subject from the finder image, inputs the lip image to the multi-class classifier, and corresponds to the lip image based on the finder image obtained as a result Generating a time series feature quantity for recognition by arranging the multidimensional score vectors in time series, the generated time series feature quantity for recognition, and the registration Based on the result of comparison between the model registered in the database, and a automatic shutter control step of executing an imaging process by controlling the imaging means.

  A program according to one aspect of the present invention includes an imaging unit that outputs a finder image at the time of composition determination and outputs a recorded image at the time of imaging, and a lip image input. A computer of an imaging apparatus, comprising: a multi-class discriminator that outputs a multi-dimensional score vector indicating a degree of similarity; and a registration database in which registration time-series feature quantities modeled in association with keywords are registered Then, the lip image including the lip region of the subject is generated from the finder image and input to the multi-class classifier, and the multidimensional score vector corresponding to the lip image based on the finder image obtained as a result is obtained. Generating means for generating a time series feature for recognition by arranging in time series, the generated time series feature for recognition, and registered in the registration database; That on the basis of a comparison result of a model, and controls the imaging unit to function as an auto shutter control means for executing an imaging process.

  In one aspect of the present invention, a lip image including a lip region of a subject is generated from a finder image and input to a multi-class classifier, and a multidimensional score vector corresponding to the lip image based on the obtained finder image is obtained. Time-series feature quantities for recognition are generated by being arranged in time series. Based on the comparison result between the generated time-series feature quantities for recognition and models registered in the registration database, the imaging means is controlled to capture images. Processing is executed.

  According to an aspect of the present invention, it is possible to identify the utterance content of a subject based on a finder image and perform imaging in response to a predetermined keyword being uttered.

It is a block diagram which shows the structural example of the speech recognition apparatus to which this invention is applied. It is a figure which shows the example of a face image, a lip area | region, and a lip image. It is a figure which shows an example of the conversion table which converts a phoneme label into a viseme label. It is a figure which shows the example of a learning sample. It is a figure which shows an example of a time series feature-value. It is a flowchart explaining an utterance recognition process. It is a flowchart explaining a learning process. It is a flowchart explaining the process of the learning speech moving image. It is a flowchart explaining the process of the speech voice for learning. It is a flowchart explaining an AdaBoostECOC learning process. It is a flowchart explaining the learning process of a binary discrimination weak discriminator. It is a flowchart explaining a registration process. It is a flowchart explaining a K-dimensional score vector calculation process. It is a flowchart explaining a recognition process. It is a figure which shows the example of the utterance word for registration. It is a figure which shows recognition performance. It is a block diagram which shows the structural example of the digital still camera to which this invention is applied. It is a block diagram which shows the structural example of an auto shutter control part. It is a flowchart explaining an auto shutter registration process. It is a flowchart explaining an auto shutter execution process. Computer configuration example

Hereinafter, the best mode for carrying out the invention (hereinafter referred to as an embodiment) will be described in detail with reference to the drawings. The description will be given in the following order.
1. 1. First embodiment Second embodiment

<1. First Embodiment>
[Configuration example of speech recognition device]
FIG. 1 shows a configuration example of an utterance recognition device 10 according to the first embodiment. This utterance recognition device 10 identifies the utterance content of a subject based on a moving image obtained by video-taking a speaker as a subject.

  The utterance recognition apparatus 10 includes a learning system 11 that performs learning processing, a registration system 12 that performs registration processing, and a recognition system 13 that performs recognition processing.

  The learning system 11 includes an image sound separation unit 21, a face region detection unit 22, a lip region detection unit 23, a lip image generation unit 24, a phoneme label assignment unit 25, a phoneme dictionary 26, a viseme label conversion unit 27, and a viseme label. The addition unit 28, the learning sample holding unit 29, the viseme classifier learning unit 30, and the viseme classifier 31 belong.

  The registration system 12 includes a viseme classifier 31, a face area detection unit 41, a lip area detection unit 42, a lip image generation unit 43, an utterance period detection unit 44, a time series feature quantity generation unit 45, and a time series feature quantity learning unit. 46 and an utterance recognizer 47 belong.

  The recognition form 13 includes a viseme discriminator 31, a face region detection unit 41, a lip region detection unit 42, a lip image generation unit 43, an utterance period detection unit 44, a time series feature quantity generation unit 45, and an utterance recognition unit 47. .

  That is, the viseme classifier 31 overlaps with the learning system 11, the registration system 12, and the recognition form 13, and the recognition system 13 is obtained by deleting the time-series feature amount learning unit 46 from the registration system 12.

  The image / sound separation unit 21 receives a moving image with sound (hereinafter referred to as a learning moving image with sound) obtained by video-taking a speaker who speaks an arbitrary word, and uses this as a learning speech video for learning. Separation into images and speech for learning. The separated learning utterance moving image is input to the face area detection unit 22, and the separated learning utterance voice is input to the phoneme label adding unit 25.

  Note that the learning speech-added speech moving image may be prepared by taking a video for this learning, or may use content such as a television program.

  The face area detection unit 22 divides the learning utterance moving image into each frame, detects a face area including a human face for each frame, as shown in FIG. 2A, and together with the learning utterance moving image, The position information of the face area is output to the lip area detection unit 23.

  As shown in FIG. 2B, the lip area detection unit 23 detects a lip area including the end of the mouth corner of the lip from the face area of each frame of the learning utterance moving image, and the lip of each frame together with the learning utterance moving image. The position information of the region is output to the lip image generation unit 24.

  As a method for detecting the face region and the lip region, any existing method such as the method disclosed in Japanese Patent Application Laid-Open Nos. 2005-284348 and 2009-49489 can be applied.

  The lip image generation unit 24 appropriately performs rotation correction on each frame of the learning speech moving image so that the line connecting the end points of the mouth corners of the lips becomes horizontal. Further, the lip image generation unit 24 extracts the lip region from each frame after the rotation correction, and resizes the extracted lip region to a predetermined image size (for example, 32 × 32 pixels) as shown in FIG. 2C. By doing so, a lip image is generated. The lip image corresponding to each frame generated in this way is supplied to the viseme label adding unit 28.

  The phoneme label assigning unit 25 refers to the phoneme dictionary 26, assigns a phoneme label indicating the phoneme to the utterance speech for learning, and outputs it to the viseme label conversion unit 27. As a method for giving a phoneme label, for example, a method called automatic phoneme labeling in the research field of speech recognition can be applied.

  The viseme label conversion unit 27 converts the phoneme label given to the learning speech voice into a viseme label indicating the shape of the lips at the time of utterance, and outputs the viseme label addition unit 28. For this conversion, a conversion table prepared in advance is used.

  FIG. 3 shows an example of a conversion table for converting phoneme labels into viseme labels. When the conversion table of FIG. 5 is used, the phoneme labels classified into 40 types are converted into viseme labels classified into 19 types. For example, phoneme labels [a] and [a:] are converted into viseme labels [a]. Also, for example, phoneme labels [by], [my], and [py] are converted into viseme labels [py]. Note that the conversion table is not limited to that shown in FIG. 3, and other conversion tables may be used.

  The viseme label adding unit 28 gives the learning speech input from the viseme label conversion unit 27 to the lip image corresponding to each frame of the learning speech moving image input from the lip image generating unit 24. The viseme label is diverted and added, and the lip image to which the viseme label is added is output to the learning sample holding unit 29.

  The learning sample holding unit 29 holds a plurality of lip images to which viseme labels are added (hereinafter referred to as lip images with viseme labels) as learning samples.

More specifically, as shown in FIG. 4, class labels y _k (k = 1, 1) corresponding to viseme labels are added to M lip images x _i (i = 1, 2,..., M). 2,..., K) and M learning samples (x _i , y _k ) are held. In this case, the number K of class label types is 19.

  The viseme classifier learning unit 30 obtains image feature amounts from viseme-labeled lip images as a plurality of learning samples held in the learning sample holding unit 29, learns a plurality of weak classifiers using AdaBoostECOC, A viseme classifier 31 composed of a plurality of weak classifiers is generated.

  As the image feature amount of the lip image, for example, PixDif Feature (pixel difference feature) proposed by the present inventors can be used.

  As for PixDif Feature (pixel difference feature), “Sabe, Nitadai,“ Learning real-time arbitrary posture face detector using pixel difference feature ”, Proceedings of 10th Image Sensing Symposium, pp.547- 552, 2004. ”, JP-A-2005-157679, and the like.

The pixel difference feature is obtained by calculating a difference (I ₁ −I ₂ ) between _two pixel values (luminance values) I ₁ and I ₂ on an image (in this case, a lip image). In the binary discriminant weak discriminator h (x) corresponding to each combination of two pixels, as shown in the following equation (1), true (+1) by the pixel difference feature I ₁ -I ₂ and the threshold Th, or False (-1) is determined.
h (x) = − 1 if I ₁ −I ₂ ≦ Th
h (x) = + 1 if I ₁ −I ₂ > Th
... (1)

  For example, when the size of the lip image is 32 × 32 pixels, 1024 × 1023 pixel difference features are obtained. These two combinations of two pixels and the threshold value Th thereof become the parameters of each binary discrimination weak discriminator, and the optimum one of these is selected by boosting learning.

  The viseme classifier 31 calculates a K-dimensional score vector corresponding to the lip image input from the lip image generation unit 43 in the utterance period notified from the utterance period detection unit 44, and outputs it to the time-series feature amount generation unit 45. Output.

  Here, the K-dimensional score vector is an index indicating which of the K (in this case, K = 19) types of visemes the input lip image corresponds to. It consists of a K-dimensional score indicating the possibility of corresponding to each viseme.

  The face area detection unit 41, the lip area detection unit 42, and the lip image generation unit 43 of the registration system 12 and the recognition system 13 are the face area detection unit 22, the lip area detection unit 23, and the lip image that belong to the learning system 11 described above. This is the same as the generation unit 24.

  The registration system 12 combines a predetermined utterance content (registration utterance word) and a moving image (hereinafter referred to as a registration utterance moving image) obtained by taking a video of a speaker speaking the utterance. A plurality of registration data is input.

  The recognition system 13 is input with a moving image (hereinafter, referred to as a recognition utterance moving image) obtained by taking a video of a speaker who speaks the utterance content to be recognized.

  That is, during the registration process, the face area detection unit 41 divides the registration utterance moving image into each frame, detects the face area for each frame, and the position information of the face area of each frame together with the registration utterance moving image. Is output to the lip region detection unit 42.

  The lip region detection unit 42 detects the lip region from the face region of each frame of the registration moving image, and outputs the position information of the lip region of each frame to the lip image generation unit 43 together with the registration utterance moving image.

  The lip image generation unit 43 appropriately rotates and corrects each frame of the utterance moving image for registration, extracts a lip region from each frame, generates a lip image by resizing, and detects the viseme discriminator 31 and the utterance period detection. To the unit 44.

  In the recognition process, the face area detecting unit 41 divides the recognition utterance moving image (moving image in which the speaker's utterance content is unknown) into each frame, detects the face area for each frame, and recognizes it. The position information of the face area of each frame is output to the lip area detection unit 42 together with the utterance moving image.

  The lip area detection unit 42 detects the lip area from the face area of each frame of the recognition utterance moving image, and outputs the position information of the lip area of each frame to the lip image generation unit 43 together with the recognition utterance moving image.

  The lip image generation unit 43 appropriately rotates and corrects each frame of the recognition utterance moving image, extracts a lip region from each frame, resizes and generates a lip image, and detects the viseme discriminator 31 and the utterance period detection. To the unit 44.

  The utterance period detection unit 44 is based on the lip image of each frame of the registration utterance moving image or the recognition utterance moving image input from the lip image generation unit 43 (hereinafter, the utterance period). And the viseme discriminator 31 and the time-series feature value generation unit 45 are notified whether or not the lip image of each frame corresponds to the utterance period.

  The time series feature quantity generation unit 45 generates a time series feature quantity by arranging the K-dimensional score vector input from the viseme discriminator 31 in the time series in the utterance period notified from the utterance period detection unit 44. To do.

  FIG. 5 shows time-series feature amounts corresponding to the utterance period when the speaker speaks “interesting”. That is, assuming that this utterance period is 1 second and the frame rate is 60 frames / second, a time-series feature amount having a score of 60K is generated. The generated time-series feature amount is output to the time-series feature amount learning unit 46 during the registration process, and is output to the utterance recognition unit 47 during the recognition process.

  The time-series feature quantity learning unit 46 is input from the time-series feature quantity generation unit 45 in association with the registration utterance word (the utterance content of the speaker in the registration utterance moving image) input during the registration process. Time series features are modeled using HMM (Hidden Markov Model). Note that the modeling method is not limited to the HMM, and any modeling method can be used as long as it can model time-series feature amounts. The modeled time-series feature amount is held in a learning database 48 built in the utterance recognizer 47.

  The utterance recognizer 47 identifies the model most similar among the models held in the learning database 48 with respect to the time series feature quantity input from the time series feature quantity generation unit 45 during the recognition process. To do. Further, the utterance recognizer 47 outputs the registered utterance word associated with the identified model as an utterance recognition result corresponding to the recognition utterance moving image.

[Description of operation]
FIG. 6 is a flowchart for explaining the operation of the speech recognition apparatus 10.

  In step S1, the learning system 11 of the utterance recognition device 10 generates a viseme discriminator 31 by executing a learning process.

  In step S2, the registration system 12 of the utterance recognition device 10 generates a time-series feature amount corresponding to the utterance moving image for registration by executing a registration process, models it using the HMM, and registers the utterance for registration. The words are associated and registered in the learning database 48.

  In step S <b> 3, the recognition system 13 of the utterance recognition device 10 recognizes the utterance content of the speaker in the recognition utterance moving image by executing recognition processing.

  Hereinafter, the details of the processing in steps S1 to S3 described above will be described.

[Details of learning process]
FIG. 7 is a flowchart for explaining the learning process in step S1 in detail.

  In step S <b> 11, the learning speech-added speech moving image is input to the image sound separation unit 21. The image sound separation unit 21 separates the learning speech-added speech moving image into the learning speech moving image and the learning speech speech, the learning speech moving image to the face area detection unit 22, and the learning speech speech to the phoneme label. It outputs to the grant part 25.

  In step S12, the learning utterance moving image is processed. In step S13, the learning speech is processed. Note that step S12 and step S13 are actually executed simultaneously in parallel. Then, the learning utterance moving image processing output (lip image) and the corresponding learning utterance speech processing output (learning speech with viseme label) are simultaneously supplied to the viseme label adding unit 28. It will be.

  FIG. 7 is a flowchart for explaining in detail the processing of the learning speech moving image in step S12.

  In step S <b> 21, the face area detection unit 22 divides the learning utterance moving image into frames, and sets each frame as a processing target. In step S22, the face area detection unit 22 detects a face area from the frame to be processed, and determines in step S23 whether the face area has been detected. If it is determined that the face area has been detected, the process proceeds to step S24. On the other hand, if it is determined that the face area cannot be detected, the process proceeds to step S26.

  In step S <b> 24, the face area detection unit 22 outputs position information of the face area to the lip area detection unit 23 together with the learning utterance moving image for one frame to be processed. The lip area detection unit 23 detects a lip area from the face area of the processing target frame, and determines whether the lip area has been detected in step S25. If it is determined that the lip region has been detected, the process proceeds to step S27. On the other hand, if it is determined that the lip region has not been detected, the process proceeds to step S26.

  When the process proceeds from step S23 or step S25 to step S26, the position information of at least one of the face area or the lip area one frame before the frame to be processed is used.

  In step S <b> 27, the lip region detection unit 23 outputs the position information of the lip region to the lip image generation unit 24 together with the learning utterance moving image for one frame to be processed. The lip image generation unit 24 appropriately performs rotation correction on one frame of the learning speech moving image to be processed so that the line connecting the end points of the mouth corners of the lips becomes horizontal. Further, the lip image generation unit 24 extracts a lip region from each frame after rotation correction, generates a lip image by resizing the extracted lip region to a predetermined image size, and a viseme label addition unit 28. Output to.

  Thereafter, the process returns to step S21, and the processes of steps S21 to S27 are repeated until the input of the learning utterance moving image signal is completed.

  Next, FIG. 9 is a flowchart for explaining in detail the processing of the learning speech voice in step S13.

  In step S <b> 31, the phoneme label assigning unit 25 refers to the phoneme dictionary 26, assigns a phoneme label indicating the phoneme to the learning speech, and outputs the phoneme label conversion unit 27.

  In step S32, the viseme label conversion unit 27 converts the phoneme label given to the learning speech to a viseme label indicating the shape of the lips at the time of utterance by using a conversion table held in advance. The data is output to the label adding unit 28.

  Thereafter, the process returns to step S31, and the processes of steps S31 and S32 are repeated until the input of the learning speech voice is completed.

  Returning to FIG. In step S <b> 14, the viseme label adding unit 28 applies the learning utterance input from the viseme label conversion unit 27 to the lip image corresponding to each frame of the learning utterance moving image input from the lip image generation unit 24. The viseme label attached to the voice is diverted and added, and the lip image to which the viseme label is added is output to the learning sample holding unit 29. The learning sample holding unit 29 holds a viseme-labeled lip image as a learning sample. After a predetermined number M of learning samples are held in the learning sample holding unit 29, the processes after step S15 are performed.

In step S15, the viseme discriminator learning unit 30 obtains image feature amounts of lip images as a plurality of learning samples held in the learning sample holding unit 29, learns a plurality of weak discriminators using AdaBoostECOC, and A viseme classifier 31 including a plurality of weak classifiers is generated.

  FIG. 10 is a flowchart for explaining in detail the process of step S15 (AdaBoostECOC learning process).

In step S41, the viseme classifier learning unit 30 acquires M learning samples (x _i , y _k ) from the learning sample holding unit 29 as illustrated in FIG.

In step S42, the viseme discriminator learning unit 30 initializes the sample weight P _t (i, k) represented by M rows and K columns according to the following equation (2). Specifically, the initial values P ₁ (i, k) of the sample weights P _t (i, k) are set to 0 for the actual learning samples (x _i , y _k ), otherwise A uniform value is set so that the sum is 1.
P ₁ (i, k) = 1 / M (K−1) for y _k ≠ k
... (2)

  The processes in steps S43 to S48 described below are repeated an arbitrary number T. The arbitrary number of repetitions T can be the maximum number of pixel difference features obtained on the lip image, and as many weak discriminators as the number of repetitions T are generated.

In step S43, the viseme discriminator learning unit 30 generates an ECOC table with 1 row and K columns. The value μ _t (k) in the k column of the ECOC table is −1 or +1, and is randomly allocated so that the number of −1 and +1 is the same.
μ _t (k) = {− 1, + 1}
... (3)

・・・（４） In step S44, the viseme discriminator learning unit 30 calculates a binary discriminating weight D _t (i) represented by M rows and 1 column according to the following equation (4). In Expression (4), the values in [] are logical expressions, and are 1 if true and 0 if false.

... (4)

・・・（５） In step S45, the viseme discriminator learning unit 30 minimizes the weighted error rate ε _t shown in the following equation (5) under the binary discrimination weight D _t (i) obtained in step S44. The value discriminating weak discriminator _ht is learned.

... (5)

  FIG. 11 is a flowchart for explaining the process of step S45 in detail.

In step S61, the viseme discriminator learning unit 30 randomly selects two pixels from all the pixels of the lip image. For example, when the lip image is 32 × 32 pixels, selection of 2 pixels selects one of 1024 × 1023 patterns. Here, the pixel positions of the selected two pixels are S ₁ and S ₂ , and the pixel values (luminance values) are I ₁ and I ₂ .

In step S62, the viseme discriminator learning unit 30 calculates pixel difference features (I ₁ -I ₂ ) using the pixel values I ₁ and I ₂ of the two pixels selected in step S61 for all learning samples. The frequency distribution is obtained.

In step S63, the viseme discriminator learning unit 30 obtains a threshold value Th _min that makes the weighted error rate ε _t shown in Equation (5) the minimum ε _min based on the frequency distribution of the pixel difference features.

In step S <b> 64, the viseme discriminator learning unit 30 obtains a threshold Th _max that sets the weighted error rate ε _t shown in Equation (5) to the maximum ε _max based on the frequency distribution of pixel difference features. Furthermore, the viseme classifier learning unit 30 inverts the threshold value Th _{max and the} like according to the following equation (6).
ε ′ _max = 1−ε _max
S ′ ₁ = S ₂
S ′ ₂ = S ₁
Th ' _max = -Th _max
... (6)

In step S65, the viseme discriminator learning unit 30 determines the position of two pixels that are parameters of the binary discriminant weak discriminator based on the magnitude relationship between the minimum ε _min and the maximum ε _max of the weighted error rate ε _t described above. S ₁ and S ₂ and a threshold value Th are determined.

That is, when ε _min <ε ′ _max , the positions S ₁ and S _{2 of} two pixels and the threshold Th _min are adopted as parameters. When ε _min ≧ ε ′ _max , the positions S ′ ₁ and S ′ _{2 of} two pixels and the threshold Th ′ _max are employed as parameters.

  In step S66, the viseme discriminator learning unit 30 determines whether or not the processing in steps S61 to S65 described above has been repeated a predetermined number of times, returns to step S61 until it is determined that the predetermined number of times has been repeated, and the subsequent steps are repeated. . If it is determined that the processes in steps S61 to S65 have been repeated a predetermined number of times, the process proceeds to step S67.

In step S67, the viseme discriminator learning unit 30 has a weighted error rate ε _t among the binary discriminant weak discriminators (parameters) determined in the process of step S65 repeated a predetermined number of times as described above. The smallest one is finally adopted as one binary discriminant weak discriminator h _t (parameter thereof).

As described above, after one binary discrimination weak discriminator _ht is determined, the process returns to step S46 in FIG.

・・・（７） In step S46, the viseme discriminator learning unit 30 determines the reliability α _t according to the following equation (7) based on the weighted error rate ε _t corresponding to the binary discriminant weak discriminator h _t determined in the process of step S45. Calculate

... (7)

In step S47, the viseme classifier learning unit 30, as shown in the following equation (8), and binary classification weak classifier h _t determined in the processing in step S45, the reliability α calculated in the processing of step S46 By multiplying _t , a binary discriminant weak discriminator f _t (x _i ) with reliability is obtained.
f _t (x _i ) = α _t h _t
... (8)

・・・（９） In step S48, the viseme classifier learning unit 30 updates the sample weight P _t (i, k) represented by M rows and K columns according to the following equation (9).

... (9)

・・・（１０） However, Z _i in the equation (9) is as shown in the following equation (10).

(10)

  In step S49, the viseme discriminator learning unit 30 determines whether or not the processes in steps S43 to S48 described above have been repeated a predetermined number of times T, and returns to step S43 until it is determined that the predetermined number of times T has been repeated. Repeat after that. If it is determined that the processes in steps S43 to S48 have been repeated a predetermined number of times T, the process proceeds to step S50.

・・・（１１） In step S50, the viseme discriminator learning unit 30 performs the following based on the binary discriminant weak discriminator f _t (x) with reliability obtained by the same number as the predetermined number T and the ECOC table corresponding to each. The final discriminator H _k (x), that is, the viseme discriminator 31 is obtained according to the equation (11).

(11)

The obtained viseme discriminator 31 has the number of classes (number of visemes) K and the number T of weak discriminators as parameters. Each weak discriminator has, as parameters, positions S ₁ and S ₂ of two pixels on the lip image, a threshold Th for discriminating pixel difference features, a reliability α, and an ECOC table μ.

As described above, the final discriminator H _k (x), that is, the viseme discriminator 31, is obtained, and the AdaBoostECOC learning process is ended.

  According to the viseme discriminator 31 generated as described above, the image feature amount of the input lip image can be expressed by a K-dimensional score vector. That is, it is possible to numerically express how much the lip image generated from each frame of the registration utterance moving image is similar to each of K (19 in this case) types of visemes. Similarly, a lip image generated from each frame of the recognition utterance moving image can be expressed numerically as to how much it resembles each of the K types of visemes.

[Details of registration process]
FIG. 12 is a flowchart illustrating in detail the registration process in step S2.

  In step S <b> 71, the registration system 12 executes the same processing as the learning speech moving image processing by the learning system 11 described above with reference to FIG. 7, so that the lips corresponding to each frame of the registration speech moving image. Generate an image. The generated lip image is input to the viseme determination unit 31 and the utterance period detection unit 44.

  In step S72, the utterance period detection unit 44 specifies the utterance period based on the lip image of each frame of the registration utterance moving image, and determines whether the lip image of each frame corresponds to the utterance period. Notify the discriminator 31 and the time-series feature value generation unit 45. The viseme determination unit 31 calculates a K-dimensional score vector corresponding to the lip image sequentially input corresponding to the speech period.

  FIG. 13 is a flowchart for explaining in detail the K-dimensional score vector calculation process by the viseme determiner 31.

In step S81, the viseme determiner 31 initializes a parameter k (k = 1, 2,..., K) indicating a class to 1. In step S82, the viseme determiner 31 initializes the score H _{k of} each class to 0.

  In step S83, the viseme determiner 31 initializes a parameter t (t = 1, 2,..., T) for specifying a weak classifier to 1.

In step S84, the viseme determiner 31, a parameter of the binary classification weak classifier h _t, that is, the position of the two pixels on the lip image x S _1, S _2, the threshold Th for determination of pixel difference feature, reliable Set degree α and ECOC table μ.

In step S85, the viseme determination unit 31 reads out the pixel values I ₁ and I ₂ from the two pixel positions S ₁ and S ₂ on the lip image x, and calculates a pixel difference feature (I ₁ −I ₂ ). by comparing with the threshold value Th, obtained determination value of the binary classification weak classifier h _t a (-1 or +1).

In step S86, the viseme determiner 31, the reliability alpha _t by multiplying the determined value of the binary classification weak classifier h _t obtained in step S85, the value of the ECOC table of further first row and K columns mu _t (k ) To obtain a class score H _k of 1 row and K columns corresponding to the parameter t.

In step S87, the viseme determiner 31, obtained in step S86, the class scores H _k of 1 row and K columns corresponding to the parameter t, the previous (i.e., t-1) to the cumulative value of the class scores H _k of by adding to, to update the class scores H _k of 1 row and K columns.

  In step S88, the viseme determiner 31 determines whether or not the parameter t = T, and if not, the process proceeds to step S89 to increment the parameter t by 1. Then, the process returns to step S84, and the subsequent processes are repeated. Thereafter, when it is determined in step S88 that the parameter t = T, the process proceeds to step S90.

  In step S90, the viseme determination unit 31 determines whether or not the parameter k = K. If it is determined that the parameter k = K is not satisfied, the process proceeds to step S91 and the parameter k is incremented by one. Then, the process returns to step S83, and the subsequent processes are repeated. Thereafter, when it is determined in step S90 that the parameter k = K, the process proceeds to step S92.

In step S92, the viseme determiner 31, the output of the viseme determiner 31 a class score H _k of 1 row and K columns are obtained at that time, i.e., when a K-dimensional score vector subsequent (now, when It outputs to the series feature-value production | generation part 45). This completes the K-dimensional score vector calculation process.

  Returning to FIG. In step S73, the time-series feature value generation unit 45 registers the K-dimensional score vectors sequentially input from the viseme discriminator 31 in time series in the utterance period notified from the utterance period detection unit 44. A time-series feature amount corresponding to the utterance period of the utterance video is generated.

  In step S74, the time-series feature amount learning unit 46 associates the registration-speech moving image with the registration utterance word (the utterance content of the speaker in the registration utterance video image) supplied from the outside in association with the time-series feature amount. The time series feature value input from the generation unit 45 is modeled by the HMM. The modeled time series feature amount is held in a learning database 48 built in the utterance recognizer 47. This completes the registration process.

[Details of recognition processing]
FIG. 14 is a flowchart for explaining the recognition process in detail.

  The recognition system 13 performs the same processing as steps S71 to S73 of the registration processing by the registration system 12 described above with reference to FIG. 12 as the processing of steps S101 to S103 on the input speech moving image for recognition. As a result, a time-series feature amount corresponding to the utterance period of the recognition utterance moving image is generated. A time-series feature amount corresponding to the utterance period of the generated utterance moving image for recognition is input to the utterance recognizer 47.

  In step S <b> 104, the utterance recognizer 47 identifies the most similar model stored in the learning database 48 with respect to the time series feature quantity input from the time series feature quantity generation unit 45. . Further, the utterance recognizer 47 outputs the registered utterance word associated with the identified model as an utterance recognition result corresponding to the recognition utterance moving image. This completes the recognition process.

[Results of recognition experiment]
Next, the result of the recognition experiment by the speech recognition apparatus 10 will be described.

  In this recognition experiment, 73 learning subjects (speakers) who uttered 216 words were used in the learning process, and utterance moving images with voice for learning were used. In the registration process, 20 words shown in FIG. 15 out of 216 words in the learning process are selected as registered utterance words, and the learning utterance moving images corresponding to the 20 words are used as registration utterance moving images. In the modeling using the HMM, the transition probability is restricted to left-to-right, and a 40-state transition model is used.

  In the recognition process, a close evaluation using the same subject utterance moving image as the learning process and the registration process, and an open evaluation using the subject utterance moving image different from the learning process and the registration process, The recognition rate shown in FIG. 16 could be obtained.

  FIG. 16 shows how similar the time-series feature amount corresponding to the recognition utterance moving image uttering a certain utterance word W for registration is to each HMM corresponding to each of the 20 types of utterance words for registration. The probability (vertical axis) that the correct answer (HMM corresponding to the utterance word W for registration) is included up to the Mth (horizontal axis) is shown.

  According to the figure, in the case of close evaluation, an identification rate of 96% could be obtained. In the case of open evaluation, an identification rate of 80% was obtained.

  In the above-described recognition experiment, the learning process and the registration process subject (speaker) are shared, and the learning utterance moving image is diverted to the registration utterance moving image, but the learning process and registration process subject (speaker). May be another person, and the subject (speaker) of the recognition process may be another person.

  According to the utterance recognition device 10 according to the first embodiment described above, the discriminator for calculating the feature amount of the input image (in this case, the lip image) is generated by learning. There is no need to design a new discriminator each time the target is desired. Therefore, by changing the type of the label, the present invention can be easily applied to a recognition device that identifies, for example, a gesture or a handwritten character from a moving image.

  In addition, by the learning process, it is possible to extract a versatile feature amount for an image of a part having a large individual difference.

  Furthermore, since the pixel difference having a relatively small amount of calculation is used as the image feature amount, real-time recognition processing can be performed.

<2. Second Embodiment>
[Configuration example of digital still camera]
Next, FIG. 17 shows a configuration example of a digital still camera 60 according to the second embodiment. The digital still camera 60 has an auto shutter function that applies lip reading technology. Specifically, when it is detected that a person who is a subject speaks a predetermined keyword (hereinafter referred to as a shutter keyword) such as “high, cheese”, the shutter is released (a still image is taken). ).

  The digital still camera 60 includes an imaging unit 61, an image processing unit 62, a recording unit 63, a U / I unit 64, an imaging control unit 65, and an auto shutter control unit 66.

  The imaging unit 61 includes a lens group and an imaging element (none of which is shown) such as a CMOS (Complementary Metal-Oxide Semiconductor), acquires an optical image of a subject, converts it into an electrical signal, and an image obtained as a result The signal is output to the subsequent stage.

  That is, the imaging unit 61 outputs an image signal to the imaging control unit 65 and the auto shutter control unit 66 in a stage before imaging in accordance with control from the imaging control unit 65. The imaging unit 61 performs imaging in accordance with control from the imaging control unit 65 and outputs an image signal obtained as a result to the image processing unit 62.

  Hereinafter, a moving image that is output to the imaging control unit 65 for determining a composition before imaging and displayed on a display (not shown) included in the U / I unit 64 is referred to as a finder image. The viewfinder image is also output to the auto shutter control unit 66. An image signal output from the imaging unit 61 to the image processing unit 62 as a result of imaging is referred to as a recorded image.

  The image processing unit 62 performs predetermined image processing (for example, camera shake correction, white balance correction, pixel interpolation, etc.) on the recorded image input from the imaging unit 61, and then encodes the result according to a predetermined encoding method. The obtained encoded image data is output to the recording unit 63. Further, the image processing unit 62 decodes the encoded image data input from the recording unit 63 and outputs an image signal (hereinafter referred to as a reproduced image) obtained as a result to the imaging control unit 65.

  The recording unit 63 records the encoded image data input from the image processing unit 62 on a recording medium (not shown). Further, the recording unit 63 reads out the encoded image data recorded on the recording medium and outputs it to the image processing unit 62.

  The imaging control unit 65 controls the entire digital still camera 60. In particular, the imaging processing unit 65 controls the imaging unit 61 to execute imaging in accordance with a shutter operation signal from the U / I unit 64 or an auto shutter signal from the auto shutter control unit 66.

  The U / I (user interface) unit 64 includes various input devices typified by a shutter button that receives a shutter operation by a user, and a display that displays a finder image, a reproduced image, and the like. In particular, the U / I unit 64 outputs a shutter operation signal to the imaging control unit 65 in response to a shutter operation from the user.

  When the auto shutter control unit 66 detects the utterance of the shutter keyword by the person who is the subject based on the viewfinder image input from the imaging unit 61, the auto shutter control unit 66 outputs an auto shutter signal to the imaging control unit 65 accordingly.

  Next, FIG. 18 shows a detailed configuration example of the auto shutter control unit 66.

  As is clear from comparison between FIG. 1 and FIG. 1, the auto shutter control unit 66 has an auto shutter signal output unit 71 in addition to the same configuration as the registration system 12 and the recognition system 13 of the utterance recognition device 10 of FIG. 1. Is added and configured. Since the same number is attached | subjected to the same component as the speech recognition apparatus 10 of FIG. 1 of the auto shutter control part 66, the description is abbreviate | omitted.

  However, the viseme discriminator 31 in the auto shutter control unit 66 has already been learned.

  When the utterance recognition result from the utterance recognizer 47 indicates that the shutter keyword is registered in advance, the auto shutter signal output unit 71 generates an auto shutter signal and outputs it to the imaging control unit 65.

[Description of operation]
Next, the operation of the digital still camera 60 will be described. The operation of the digital still camera 60 is provided with a normal shooting mode, a normal playback mode, an auto shutter registration mode, an auto shutter execution mode, and the like.

  In the normal shooting mode, shooting is performed according to the shutter operation by the user. In the normal playback mode, captured images are played back and displayed in response to playback operations by the user.

  In the shutter keyword registration mode, a time-series feature amount HMM indicating the movement of the lips of a subject (such as a user) who speaks an arbitrary word as a shutter keyword is registered. It should be noted that, at the stage of shipping the digital still camera 60 as a product, a shutter keyword and an HMM of a time-series feature amount indicating the corresponding lip movement may be registered in advance.

  In the auto shutter execution mode, a time series feature amount indicating the movement of the lips of the person who is the subject is detected based on the finder image, and it is recognized that the shutter keyword is uttered based on the detected time series feature amount. Shooting is done in case.

[Details of shutter keyword registration process]
Next, FIG. 19 is a flowchart for explaining shutter keyword registration processing.

  The shutter keyword registration process is started when the shutter keyword registration mode is set according to a predetermined operation from the user, and is ended according to a predetermined operation from the user.

  In addition, after instructing the start of the shutter keyword registration process, the user causes the face of the speaker who speaks the word to be registered as the shutter keyword to appear in the finder image. For this speaker, it is desirable to use a person who becomes a subject during the auto-shutter execution process, but other users, for example, may be the speaker. Then, after the utterance of the shutter keyword is finished, the end of the auto shutter learning process is instructed.

  In step S121, the imaging control unit 65 determines whether or not the end of the auto shutter registration process has been instructed. If not, the process proceeds to step S122.

  In step S122, the face area detection unit 41 of the registration system 12 divides the finder image into frames, and sets each frame as a processing target. A face area is detected from a frame to be processed. In step S123, the face area detection unit 41 determines whether or not only one face area is detected from the processing target frame, and when a plurality of face areas are detected or no face area is detected. If so, the process proceeds to step S124.

  In step S124, the U / I unit 64 urges the user to pay attention so that only one speaker who speaks a word to be registered as a shutter keyword appears in the viewfinder image. Thereafter, the process returns to step S121, and the subsequent steps are repeated.

  If only one face area is detected from the processing target frame in step S123, the process proceeds to step S125.

  In step S <b> 125, the face area detection unit 41 outputs the position information of the face area to the lip area detection unit 42 together with the finder image for one frame to be processed. The lip region detection unit 42 detects the lip region from the face region of the frame to be processed, and outputs the position information of the lip region to the lip image generation unit 43 together with the finder image for one frame to be processed.

  The lip image generation unit 43 appropriately performs rotation correction on one frame of the finder image to be processed so that the line connecting the end points of the mouth corners of the lips is horizontal. Further, the lip image generation unit 43 extracts a lip region from each frame after the rotation correction, and generates a lip image by resizing the extracted lip region to a predetermined image size. The generated lip image is input to the viseme determination unit 31 and the utterance period detection unit 44.

  In step S126, the utterance period detection unit 44 determines whether the frame is the utterance period based on the lip image of the frame to be processed, and uses the determination result as the viseme discriminator 31 and the time-series feature value generation. Notify the unit 45. And when it determines with it being an utterance period, a process is advanced to step S127. On the other hand, when it is determined that it is not the speech period, step S127 is skipped.

  In step S127, the viseme determiner 31 calculates a K-dimensional score vector corresponding to the lip image sequentially input corresponding to the speech period, and outputs the K-dimensional score vector to the time-series feature value generation unit 45. Thereafter, the process returns to step S121, and the processes of steps SS121 to SS127 are repeated until the end of the auto shutter registration process is instructed.

  If it is determined in step S121 that the end of the auto shutter registration process has been instructed, the process proceeds to step S128.

  In step S128, the time-series feature value generation unit 45 registers the K-dimensional score vectors sequentially input from the viseme discriminator 31 in the utterance period notified from the utterance period detection unit 44 in time series. A time-series feature amount corresponding to the desired shutter keyword is generated.

  In step S129, the time-series feature amount learning unit 46 associates the time-series feature amount input from the time-series feature amount generation unit 45 with the HMM in association with the shutter keyword text data input from the U / I unit 64. To model. The modeled time series feature amount is held in a learning database 48 built in the utterance recognizer 47. This completes the shutter keyword registration process.

[Details of auto shutter execution processing]
Next, FIG. 20 is a flowchart for explaining the auto shutter execution process.

  The auto shutter execution process is started when the auto shutter execution mode is set according to a predetermined operation from the user, and is ended according to a predetermined operation from the user.

  In step S <b> 141, the face area detection unit 41 of the recognition system 12 divides the finder image into frames and sets the frames as processing targets. A face area is detected from a frame to be processed.

  In step S142, the face area detection unit 41 determines whether or not the face area has been detected from the processing target frame, and the process returns to step S141 until the face area can be detected. If the face area can be detected from the processing target frame, the process proceeds to step S143.

  Here, unlike the shutter keyword registration process, a plurality of face regions may be detected from one frame. When a plurality of face areas are detected from one frame, the subsequent processes are executed in parallel with each detected face area.

  In step S143, the face area detection unit 41 outputs the position information of the face area to the lip area detection unit 42 together with the finder image for one frame to be processed. The lip region detection unit 42 detects the lip region from the face region of the frame to be processed, and outputs the position information of the lip region to the lip image generation unit 43 together with the finder image for one frame to be processed.

  The lip image generation unit 43 appropriately performs rotation correction on one frame of the finder image to be processed so that the line connecting the end points of the mouth corners of the lips is horizontal. Further, the lip image generation unit 43 extracts a lip region from each frame after the rotation correction, and generates a lip image by resizing the extracted lip region to a predetermined image size. The generated lip image is input to the viseme determination unit 31 and the utterance period detection unit 44.

  In step S144, the speech period detection unit 44 determines the speech period based on the lip image of the frame to be processed. That is, when it is determined that the frame to be processed is the start point of the utterance period or is in the utterance period, the process proceeds to step S145.

  In step S <b> 145, the viseme determiner 31 calculates a K-dimensional score vector corresponding to the lip image sequentially input corresponding to the utterance period, and outputs the K-dimensional score vector to the time-series feature value generation unit 45. Thereafter, the process returns to step S141, and the subsequent steps are repeated.

  If it is determined in step S144 that the frame to be processed is the end point of the speech period, the process proceeds to step S146.

  In step S146, the time-series feature value generation unit 45 arranges the K-dimensional score vectors sequentially input from the viseme discriminator 31 in time series in the utterance period notified from the utterance period detection unit 44. A time-series feature amount corresponding to the movement of the lips is generated.

  In step S 147, the time-series feature value generation unit 45 inputs the generated time-series feature value to the utterance recognizer 47. In step S148, the utterance recognizer 47 compares the time series feature quantity input from the time series feature quantity generation unit 45 with the HMM corresponding to the shutter keyword held in the learning database 48, and compares the lip of the subject. It is determined whether or not the movement corresponds to the shutter keyword. If it is determined that the movement of the lips of the subject corresponds to the shutter keyword, the process proceeds to step S149. If it is determined NO, the process returns to step S141 and the subsequent steps are repeated.

  In step S149, the utterance recognizer 47 notifies the auto shutter signal output unit 71 that the movement of the lips of the subject corresponds to the shutter keyword. In response to this notification, the auto shutter signal output unit 71 generates an auto shutter signal and outputs it to the imaging control unit 65. In accordance with the auto shutter signal, the imaging control unit 65 controls the imaging unit 61 and the like to perform imaging. The imaging timing can be arbitrarily set by the user, for example, after a predetermined time (for example, 1 second) after the utterance of the shutter keyword. Thereafter, the process returns to step S141, and the subsequent steps are repeated.

  In the above description, when a plurality of face regions (subjects) are detected from the finder image, anyone of the plurality of subjects may speak the shutter keyword.

  However, such a specification may be changed, and for example, imaging may be performed when a majority of subjects speak a shutter keyword. With such a specification, it is possible to give users playability when taking a group photo. In addition, since a plurality of face recognitions are performed, the recognition result is robust, and an effect of suppressing erroneous detection of the shutter keyword can be expected.

  Furthermore, a shutter keyword may be detected by focusing on only a specific person among a plurality of subjects by combining personal identification techniques for identifying a person's face. This specific person may be plural. If this specific person is a subject (subject) and the shutter keyword registration process described above is performed, more robust and accurate speech recognition can be performed.

  As described above, according to the digital still camera 60 according to the second embodiment, a subject at a distant position only utters a shutter keyword even in a noise environment without using a remote controller or the like. The imaging timing can be instructed. This shutter keyword can be set arbitrarily.

  Note that the present invention can be applied not only to a digital still camera but also to a digital video camera.

By the way, the above-described series of processing can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 21 is a block diagram illustrating a configuration example of hardware of a computer that executes the above-described series of processing by a program.

  In the computer 200, a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, and a RAM (Random Access Memory) 203 are connected to each other via a bus 204.

  An input / output interface 205 is further connected to the bus 204. The input / output interface 205 includes an input unit 206 composed of a keyboard, mouse, microphone, etc., an output unit 207 composed of a display, a speaker, etc., a storage unit 208 composed of a hard disk or nonvolatile memory, and a communication unit 209 composed of a network interface. A drive 210 for driving a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 201 loads, for example, the program stored in the storage unit 208 to the RAM 203 via the input / output interface 205 and the bus 204 and executes the program. Is performed.

  The program executed by the computer (CPU 201) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. The program is recorded on a removable medium 211 that is a package medium composed of a memory or the like, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the storage unit 208 via the input / output interface 205 by attaching the removable medium 211 to the drive 210. The program can be received by the communication unit 209 via a wired or wireless transmission medium and installed in the storage unit 208. In addition, the program can be installed in the ROM 202 or the storage unit 208 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The program may be processed by a single computer, or may be distributedly processed by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 10 Speech recognition apparatus, 21 Image | video sound separation part, 22 Face area detection part, 23 Lip area detection part, 24 Lip image generation part, 25 Phoneme label provision part, 26 Phoneme dictionary, 27 Viseme label conversion part, 28 Viseme label Addition unit, 29 learning sample holding unit, 30 viseme classifier learning unit, 31 viseme classifier, 41 face region detection unit, 42 lip region detection unit, 43 lip image generation unit, 44 utterance period detection unit, 45 time series Feature amount generation unit, 46 Time series feature amount learning unit, 47 Speech recognizer, 48 Learning database, 60 Digital still camera, 61 Imaging unit, 62 Image processing unit, 63 Recording media, 64 U / I unit, 65 Imaging control unit , 66 Auto shutter control unit, 71 Auto shutter signal output unit, 200 computer, 201 CPU

Claims (7)

Imaging means for outputting a finder image at the time of composition determination and outputting a recorded image at the time of imaging;
A multi-class classifier that outputs a multi-dimensional score vector corresponding to the input of the lip image and indicating how similar the lip image is to each of a plurality of types of visemes;
A registration database in which modeled registration time series feature amounts are registered in association with keywords,
The lip image including the lip region of the subject is generated from the finder image and input to the multi-class classifier, and the multi-dimensional score vector corresponding to the lip image based on the finder image obtained as a result is time-series. Generating means for generating a recognition time-series feature amount,
An image pickup apparatus comprising: an auto shutter control unit configured to control the image pickup unit to execute an image pickup process based on a comparison result between the generated time series feature quantity for recognition and a model registered in the registration database. . The imaging apparatus according to claim 1, wherein the multi-class classifier is generated by AdaBoostECOC (Error Correct Output Coding) learning using an image feature amount of a lip image to which a class label indicating a viseme is added. The imaging apparatus according to claim 2, wherein the image feature amount is a pixel difference feature. A lip image for registration is generated from a finder image for registration whose subject is a subject who speaks the keyword, the lip image for registration is input to the multi-class classifier, and the registration image obtained as a result The registration time-series feature quantity is generated by arranging the multi-dimensional score vectors corresponding to the lip images in time series, the registration time-series feature quantity is modeled in association with the arbitrary keyword, and the registration is performed. The imaging apparatus according to claim 2, further comprising registration means for registering in the database. The imaging apparatus according to claim 4, wherein the registration unit models the registration time-series feature amount using an HMM (Hidden Markov Model). Imaging means for outputting a finder image at the time of composition determination and outputting a recorded image at the time of imaging;
A multi-class classifier that outputs a multi-dimensional score vector corresponding to the input of the lip image and indicating how similar the lip image is to each of a plurality of types of visemes;
In an imaging method of an imaging apparatus including a registration database in which registration time-series feature amounts modeled in association with keywords are registered,
According to the imaging device,
The lip image including the lip region of the subject is generated from the finder image and input to the multi-class classifier, and the multi-dimensional score vector corresponding to the lip image based on the finder image obtained as a result is time-series. Generating step for generating a recognition time-series feature value by arranging in
An image capturing method comprising: an auto shutter control step for controlling the image capturing unit to execute an image capturing process based on a comparison result between the generated time series feature amount for recognition and a model registered in the registration database. . Imaging means for outputting a finder image at the time of composition determination and outputting a recorded image at the time of imaging;
A multi-class classifier that outputs a multi-dimensional score vector corresponding to the input of the lip image and indicating how similar the lip image is to each of a plurality of types of visemes;
In a computer of an imaging device provided with a registration database in which modeled registration time-series feature amounts are registered in association with keywords,
The lip image including the lip region of the subject is generated from the finder image and input to the multi-class discriminator, and the multidimensional score vector corresponding to the lip image based on the finder image obtained as a result is time-series. Generating means for generating a recognition time-series feature amount,
Based on the comparison result between the generated time series feature quantity for recognition and the model registered in the registration database, the imaging means is controlled to function as an auto shutter control means for executing an imaging process. program.

Images