JP2012231327A - Imaging apparatus, imaging method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent images of similar facial expressions and scenes from being taken excessively and to simply take images of various facial expressions of a subject.SOLUTION: A face detection part 104 detects the face of a subject imaged by an imaging system 102, and a facial expression detection part 105 calculates an evaluation value relating to the facial expression of the detected face. Memory 108 contains past evaluation values. If the calculated evaluation value is larger than a threshold and a difference between an evaluation value most approximate to a present evaluation value in the past evaluation values stored in the memory 108 and the present evaluation value is a predetermined value or more, an automatic imaging control part 106 gives an instruction for imaging.

Description

  The present invention relates to an imaging apparatus, an imaging method, and a program, and more particularly to a technique suitable for use in an auto shutter imaging technique.

  2. Description of the Related Art Conventionally, there is known a technique for automatically taking a picture by determining the facial expression of a subject in a digital camera having a face detection function. Among them, as a representative technique, an auto shutter photography technique using a smile detection function is known.

  In the automatic shutter photographing technique using the smile detection function, first, the face of the subject is detected from the image displayed on the display device by the face detection function. Next, a smile evaluation value is calculated for the detected face by a smile discrimination function. Known methods for determining a smile include a method of correlating with a representative smile image by pattern matching, and a smile determination algorithm based on vector analysis of edges included in a face. Next, the evaluation value of the smile calculated in time series is analyzed, and an optimal timing for photographing the best smile of the subject is instructed to the imaging unit. Then, the imaging unit captures an image and records the captured image in the memory.

  As a technique for instructing shooting as described above, for example, Patent Document 1 discloses a method of comparing a smile evaluation value with a predetermined smile threshold and instructing shooting when the threshold is exceeded. . Japanese Patent Application Laid-Open No. 2004-228561 discloses a method for estimating the optimum timing for photographing from the transition of the smile evaluation value as a method for determining a more suitable smile photographing timing. Furthermore, as another technique, a method of preventing failure by continuously shooting a plurality of still images at the time of shooting is known.

JP 2008-42319 A Japanese Patent No. 4240108

"Estimation method of face direction from grayscale image using eigenspace method" IEICE-40 IE 111 ~ “Robust Front Face Recognition by Gray Image Matching” IEICE Transactions '93 /7Vol.J76-D-2 No.7 P1363〜 "Research Trends of Face Recognition by Computer" IEICE Journal '93 Vol80 No.3 P257-

  However, in the prior art described in Patent Document 1 described above, shooting is always performed when the evaluation value of a subject's smile exceeds a predetermined threshold. Further, in the conventional technique described in Patent Document 2, photographing is performed every time an optimum smile photographing timing is detected. For this reason, if the subject continues to laugh with a similar expression, there is a problem that a similar smile image is excessively shot.

  SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and it is an object of the present invention to prevent excessive photographing of similar facial expressions and scene images and to easily photograph various facial expressions of a subject.

  An imaging apparatus according to the present invention calculates an evaluation value relating to an imaging means for imaging a subject, a face detection means for detecting a face of the subject imaged by the imaging means, and a facial expression detected by the face detection means. Evaluation value calculating means, imaging control means for instructing photographing based on the evaluation value calculated by the evaluation value calculating means, and storage means for storing past evaluation values calculated by the evaluation value calculating means The imaging control means includes a difference between a current evaluation value closest to a current evaluation value calculated by the evaluation value calculation means and a current evaluation value among past evaluation values stored in the storage means When the value is equal to or greater than a predetermined value, the photographing is instructed.

  According to the present invention, it is possible to prevent a facial expression or scene image similar to an already captured image from being excessively captured, and to capture a facial expression that the user wants to acquire without excess or deficiency.

It is a block diagram which shows the structural example of the digital camera which concerns on embodiment. 6 is a flowchart illustrating an example of a processing procedure of a shooting instruction in the first embodiment. It is a figure which shows an example of the facial expression evaluation value and facial expression evaluation threshold value which can be image | photographed. It is a figure which shows an example of the facial expression evaluation value which can be image | photographed for every person. 14 is a flowchart illustrating an example of a processing procedure of a shooting instruction in the third embodiment. It is a figure which shows an example of the evaluation value which comprises similarity.

(First embodiment)
Hereinafter, a first preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a block diagram illustrating a configuration example of a digital camera 100 according to the present embodiment. The digital camera 100 according to the present embodiment is equipped with an automatic photographing function based on facial expression discrimination.

  In FIG. 1, an imaging system 102 converts light imaged by the optical system 101 into a digital electrical signal. The image processing circuit 103 converts the converted digital electric signal into image data. The imaging control unit 107 performs imaging sequence control by the imaging system 102, such as operation control of the imaging element, mechanical control such as a shutter and a diaphragm related to AE (Auto Exposure), and lens position control related to AF (Auto Focus). .

  The face detection unit 104 performs face detection on the image data converted by the image processing circuit 103. The expression detection unit 105 functions as an evaluation value calculation unit, and calculates an expression evaluation value F for each face image detected by the face detection unit 104. In the present embodiment, as an example of facial expression detection, a moment when a subject laughs is detected. As a method of detecting a smile, a general method such as a method of matching with a typical smile image or a method of performing vector analysis of face edge components is used.

  The automatic photographing control unit 106 includes a facial expression determination threshold value F_Th held in the memory 108, a facial expression evaluation value P (i) (i = 1, 2,..., N) of a previously photographed image, and time stamp information. Based on, the shooting conditions are updated adaptively. Further, the automatic photographing control unit 106 instructs the photographing control unit 107 to perform photographing when the facial expression evaluation value F calculated by the facial expression detecting unit 105 satisfies a predetermined condition. Then, the facial expression evaluation value F at that time is stored in the memory 108 as the latest photographed facial expression evaluation value P (n).

  Here, the past shooting information that can be stored in the memory 108 is a maximum of N times (N is arbitrary). When new information is always stored in the memory 108 and N times of shooting information is stored, the oldest information is stored. It shall be deleted from the information. The past shooting information may be information stored in a period from when the digital camera 100 shifts to the automatic shooting mode based on facial expression detection until the digital camera 100 exits the same mode, and the digital camera 100 is turned on. It may be a period from immediately after the power is turned off.

Next, the shooting determination algorithm of the automatic shooting control unit 106 will be described in detail with reference to FIG. FIG. 2 is a flowchart illustrating an example of a processing procedure of a shooting instruction by the automatic shooting control unit 106 in the present embodiment.
First, when the facial expression evaluation value F of the input image calculated by the facial expression detection unit 105 is input to the automatic photographing control unit 106, the process starts. In step S201, a facial expression evaluation value P (i) obtained in the past from the memory 108, a past photographing timestamp tm (i), and a predetermined facial expression determination threshold F_Th are acquired.

  Next, in S202, the facial expression determination threshold F_Th is compared with the facial expression evaluation value F of the input image. As a result of the comparison, if the facial expression evaluation value F is equal to or smaller than the facial expression determination threshold F_Th, the photographing is not instructed and the process is terminated. On the other hand, when the input facial expression evaluation value F exceeds the facial expression determination threshold F_Th as a result of the comparison in S202, the process proceeds to the next S203.

  Next, in S203, the latest time stamp tm (n) of the image captured in the past is referred to and it is determined whether or not a sufficient time has elapsed since the previous shooting. If a sufficient time has passed as a result of this determination, it is determined that there is a high possibility that the scene has changed, and in step S205, the imaging control unit 107 is instructed to perform imaging. At this time, the latest facial expression evaluation value F is stored in the memory 108. On the other hand, if a sufficient time has not elapsed as a result of the determination in S203, there is a possibility that a plurality of similar images may be taken, and the process proceeds to S204.

  Next, in S204, shooting determination is performed based on the past facial expression evaluation value P (i). In the shooting determination, as shown in FIG. 3A, the peripheral section [P (i) −α, P (i) + α] around the facial expression evaluation value P (i) is set as a non-shootable region. If the facial expression evaluation value F of the input image is present in this region as a result of the photographing determination, it is determined that the image has a similar facial expression, and the processing is terminated without giving a photographing instruction.

  On the other hand, if the facial expression evaluation value F does not exist in the section shown above as a result of the photographing determination in S204, it is determined that the facial expression is to be photographed. In S205, the automatic photographing control unit 106 determines to the imaging control unit 107. In addition to instructing photographing, the latest facial expression evaluation value F is stored in the memory 108.

  In the determination of S204, when the facial expression evaluation value F is larger than the maximum value P_max = MAX [P (1), P (2), P (3),. F> P_max) may be taken only. Further, in the determination of S202, as shown in FIG. 3B, the facial expression determination threshold F_Th is gradually increased according to the number of previous shootings, and the frequency of shooting a similar level of facial expression is decreased. Good.

  As described above, according to the present embodiment, it is possible to narrow down the shooting target based on facial expressions photographed in the past, and it is possible to prevent continuous photographing and recording of images with similar facial expressions. it can.

(Second Embodiment)
Hereinafter, the second embodiment of the present invention will be described with reference to FIGS. The configuration of the digital camera according to this embodiment is the same as that shown in FIG.

Next, the shooting determination algorithm according to the present embodiment will be described in detail with reference to FIGS. Note that the processing procedure of the present embodiment is basically the same as that shown in FIG.
First, the face detection unit 104 classifies face images included in an image using a person identification algorithm typified by personal authentication technology. For example, as shown in FIG. 4A, the persons in the image are classified and assigned IDs. Here, in the personal authentication technology, the similarity between the input face image and a registered face image or a face image taken in the past is calculated. Note that these registered face images or face images taken in the past are held in the memory 108. As a method for calculating the similarity, after normalizing the size, angle, brightness, etc. of the input face image, a method of calculating using the KL expansion method of the Fourier spectrum is used. Alternatively, a pattern matching method, a method of normalizing the extracted face image into blocks, and obtaining a distance between the luminances may be used. Details regarding these methods are disclosed in Non-Patent Documents 1 to 3, for example.

As described above, the facial expression evaluation value F _x is calculated by the facial expression detection unit 105 for all the faces, and when the calculated facial expression evaluation value F _x is input to the automatic photographing control unit 106, it is shown in FIG. The shooting determination process is started. Note that x indicates an ID. Then, in S201, the automatic photographing control unit 106, from the memory 108, the personal authentication by classified taken in the past for each individual facial expression evaluation value P (i _x), past imaging time stamp tm (i _x), and A predetermined facial expression determination threshold value F_Th is acquired.

Next, in S202, the facial expression determination threshold F_Th is compared with the facial expression evaluation values F _x of all the faces included in the input image. As a result of the comparison, if the facial expression evaluation values F _x of all the faces included in the input image are equal to or smaller than the facial expression determination threshold F_Th, the shooting is not instructed and the process is terminated.

On the other hand, the result of the comparison of S202, if at least one of facial expression evaluation value F _x is greater than the facial expression determination threshold value F_th, in S203, with reference latest timestamp tm of shot in the past image (n _x), It is determined whether or not a sufficient time has passed since the last shooting. As a result of this determination, if a sufficient time has elapsed, it is determined that there is a high possibility that the scene has changed, and the process proceeds to S205 to instruct shooting. At this time, the latest facial expression evaluation value F is stored in the memory 108.

For example, when the facial expression evaluation value F ₁ of the person with ID 1 exceeds the facial expression determination threshold F_Th, the latest time stamp tm (n ₁ ) of the image captured using the person with ID 1 in the past as a trigger is compared with the current time. If enough time has passed, instruct the camera to shoot. On the other hand, if a sufficient time has not elapsed as a result of the determination in S203, there is a high possibility that a plurality of similar images will be taken, and the process proceeds to S204. Note that the determination target is only a face whose facial expression evaluation value F _x exceeds the facial expression determination threshold F_Th, and there is a face for which a sufficient time has passed with the facial expression evaluation value F _x equal to or less than the facial expression determination threshold F_Th. Even so, it is not considered in the determination.

Next, in S204, shooting determination is performed based on the past facial expression evaluation value P (i _x ). In the shooting determination, as in the first embodiment, it is determined whether or not the facial expression evaluation value F _x exists in the non-shootable area as shown in FIG. If the facial expression evaluation value F _x exists in this region as a result of the photographing determination, the photographing is not instructed and the process is terminated. On the other hand, as a result of the determination in S204, if the facial expression evaluation value F _x does not exist in the above-described section, in S205, the automatic shooting control unit 106 instructs the shooting control unit 107 to shoot and also stores in the memory 108. The latest facial expression evaluation value F _x is stored. Note that, as in the first embodiment, photographing may be performed only when the facial expression evaluation value F is greater than the maximum value P_max of the past photographing evaluation values by a predetermined value (F> P_max).

The above shooting determination is performed for each ID. For example, as shown in FIG. 4B, it is assumed that the past facial expression evaluation values for the person with ID1 are P (1) = 180, P (2) = 220, n ₁ = 2 and α = 15. Also, it is assumed that there is no past facial expression evaluation value for the person with ID3, and n ₃ = 0. In this case, even if the facial expression evaluation value of the person ID1 is F ₁ = 210, it belongs to the non-photographable area shown by the oblique lines shown in FIG. However, when the facial expression evaluation values are the same for the person ID1 and the person ID3, the person ID3 has never been photographed in the past around F ₃ = 210, and thus becomes a photographable area. Accordingly, when at least one person is in the shootable area in this way, the process proceeds to S205.

  As described above, according to the present embodiment, it is possible to narrow down the shooting target based on the facial expression of each person photographed in the past, and to continuously capture and record images with similar facial expressions. Can be prevented.

(Third embodiment)
Hereinafter, the third embodiment of the present invention will be described with reference to FIGS. 1, 5, and 6. In the first and second embodiments, shooting control is performed based on a change in the facial expression of a person, but in this embodiment, shooting control is performed based on the similarity of the entire image. The configuration of the digital camera according to the present embodiment is almost the same as that shown in FIG.

  The automatic shooting control unit 106 determines whether the scene should be shot based on the facial expression determination threshold F_Th held in the memory 108 and the similarity between the image shot in the past and the input image. If the input image has a low similarity, the imaging control unit 107 is instructed to shoot. It should be noted that the memory 108 has information necessary for obtaining the similarity as a past photographed image, a reduced image thereof, or an AF photographing control value (distance information) at the past photographing, or an AE photographing control value ( At least one piece of information such as exposure information) is stored.

Next, the shooting determination algorithm of the automatic shooting control unit 106 will be described in detail with reference to FIG. FIG. 5 is a flowchart illustrating an example of a processing procedure of a shooting instruction by the automatic shooting control unit 106 and the face detection unit 104 in the present embodiment.
First, when the facial expression evaluation value F of the input image calculated by the facial expression detection unit 105 is input to the automatic photographing control unit 106, the process starts. In step S501, the automatic photographing control unit 106 reads the facial expression determination threshold value F_Th from the memory 108, and compares the facial expression determination threshold value F_Th with the facial expression evaluation value F of the input image. If the facial expression evaluation value F is equal to or smaller than the facial expression determination threshold value F_Th as a result of this comparison, the imaging is not instructed and the process is terminated. On the other hand, if the facial expression evaluation value F exceeds the facial expression determination threshold F_Th as a result of the comparison in S501, the process proceeds to the next S502.

  Next, in S502, the face detection unit 104 refers to the face included in the past photographed image held in the memory 108 and has been a trigger for the photographing in the past n photographing, The subject is collated by the personal authentication technique described above. As a result of this collation, when the face included in the input image is different from any face that has been a shooting trigger in the past, the process proceeds to S505.

On the other hand, if it is determined in step S502 that the face has been taken in the past, the process proceeds to step S503. In step S <b> 503, the automatic shooting control unit 106 functions as a similarity calculation unit, and calculates the similarity of the entire image using detailed information at the time of previous shooting held in the memory 108. Here, there are evaluation values shown in the following (1) to (6) as elements constituting the similarity.
(1) Distance difference calculated from AF information at the time of shooting (P1)
(2) Exposure difference calculated from AE information at the time of shooting (P2)
(3) Face size difference calculated from face information at the time of shooting (P3)
(4) Difference between the number of faces included in the past photographed image and the number of faces included in the input image (P4)
(5) Correlation value of the entire image calculated by correlating the past captured image with the input image (P5)
(6) Difference in histogram distribution of luminance or color included in the image (P6)

  The degree of similarity is calculated using at least one condition among (1) to (6), and is calculated by performing weighted addition on each of the evaluation values P1 to P6. FIG. 6 is a diagram illustrating an example of an outline for calculating the degree of similarity from the above-described conditions (1) to (6). As shown in FIG. 6, the difference between AF, AE, face information, etc. between the past photographed image A and the input image is set as the evaluation value P1 to P4, the image correlation value is set as the evaluation value P5, and the absolute difference between the histograms The sum of the values is set as an evaluation value P6. In this case, the similarity T can be obtained by the following equation, for example.

  Here, w1 to w6 are weights for the evaluation values P1 to P6, respectively. In next step S504, it is determined whether or not the calculated similarity T is greater than or equal to a predetermined value. If the similarity T is lower than the predetermined value as a result of this determination, it is determined that the subject is in a different scene. In S505, the automatic shooting control unit 106 instructs the shooting control unit 107 to shoot, and the memory 108 Save the latest shooting conditions and images. On the other hand, if the result of determination in S504 is that the degree of similarity T is greater than or equal to a predetermined value, it is determined that the input image is a scene similar to a scene captured in the past, and the processing is terminated without performing a shooting instruction. .

  As described above, according to the present embodiment, it is possible to narrow down the shooting target based on a scene change from an image shot in the past, and to continuously shoot and record images of similar scenes. Can be prevented.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 102 Imaging system 104 Face detection part 105 Expression detection part 106 Automatic imaging | photography control part 107 Imaging | photography control part 108 Memory

Claims (13)

Imaging means for imaging a subject;
Face detection means for detecting the face of the subject imaged by the imaging means;
Evaluation value calculating means for calculating an evaluation value related to facial expression detected by the face detecting means;
Based on the evaluation value calculated by the evaluation value calculation means, imaging control means for instructing imaging,
Storage means for storing past evaluation values calculated by the evaluation value calculation means,
The imaging control unit has a predetermined difference between the current evaluation value and the evaluation value closest to the current evaluation value calculated by the evaluation value calculation unit among the past evaluation values stored in the storage unit. An imaging apparatus characterized by instructing photographing when the value is greater than or equal to a value.   The imaging control unit has a predetermined difference between the current evaluation value and the evaluation value closest to the current evaluation value calculated by the evaluation value calculation unit among the past evaluation values stored in the storage unit. The imaging apparatus according to claim 1, wherein imaging is instructed when the current evaluation value exceeds a predetermined value that is equal to or greater than a value.   The imaging apparatus according to claim 2, wherein the predetermined value differs according to the number of past evaluation values stored in the storage unit. Further comprising specifying means for specifying the person of the face detected by the face detecting means;
The evaluation value calculating means calculates the evaluation value for each person specified by the specifying means;
The storage means stores a past evaluation value calculated by the evaluation value calculation means for each person,
The imaging control means, for at least one person, out of the past evaluation values stored in the storage means, the evaluation value closest to the current evaluation value calculated by the evaluation value calculation means, The imaging apparatus according to claim 1, wherein imaging is instructed when a difference from a current evaluation value is a predetermined value or more.   The storage means is an evaluation value calculated by the evaluation value calculation means between immediately after the power is turned on and until the present time, or after shifting to a mode for instructing shooting by the imaging control means. The imaging device according to claim 1, wherein the imaging device is stored.   The imaging apparatus according to claim 1, wherein the evaluation value calculation unit calculates an evaluation value related to a facial smile detected by the face detection unit. Imaging means for imaging a subject;
Face detection means for detecting the face of the subject imaged by the imaging means;
Specifying means for specifying the person of the face detected by the face detecting means;
Storage means for storing shooting information of images taken in the past;
When shooting information of an image including a person specified by the specifying unit is stored in the storage unit, a similarity with an image shot in the past based on shooting information of an image including the specified person is calculated. Similarity calculating means for calculating;
An image pickup apparatus comprising: an image pickup control unit that instructs shooting when the degree of similarity calculated by the degree of similarity calculation unit is smaller than a predetermined value. An evaluation value calculating means for calculating an evaluation value related to the facial expression detected by the face detecting means;
The imaging control means is when the similarity calculated by the similarity calculation means is smaller than a predetermined value, and when the evaluation value calculated by the evaluation value calculation means exceeds a predetermined value The imaging apparatus according to claim 7, wherein imaging is instructed.   The similarity calculation means uses the at least one information among the distance information, the exposure information, the face information of the face included in the image, the correlation value with the past image, and the difference of the luminance or color histogram. The imaging device according to claim 7, wherein the degree is calculated. A face detection step of detecting the face of the subject imaged by the imaging means;
An evaluation value calculating step of calculating an evaluation value related to the facial expression detected in the face detecting step;
An imaging control step for instructing imaging based on the evaluation value calculated in the evaluation value calculating step,
In the imaging control step, of the past evaluation values calculated in the evaluation value calculation step, an evaluation value closest to the current evaluation value calculated in the evaluation value calculation step, and the current evaluation value An imaging method characterized by instructing imaging when the difference is greater than or equal to a predetermined value. A face detection step of detecting the face of the subject imaged by the imaging means;
A specifying step of specifying a person of the face detected in the face detection step;
When shooting information of an image including a person specified in the specifying step is stored in a storage unit as shooting information of an image shot in the past, based on shooting information of an image including the specified person in the past A similarity calculation step for calculating the similarity with the captured image;
An imaging method comprising: an imaging control step for instructing imaging when the similarity calculated in the similarity calculation step is smaller than a predetermined value. A face detection step of detecting the face of the subject imaged by the imaging means;
An evaluation value calculating step of calculating an evaluation value related to the facial expression detected in the face detecting step;
Based on the evaluation value calculated in the evaluation value calculation step, causing the computer to execute an imaging control step for instructing shooting,
In the imaging control step, of the past evaluation values calculated in the evaluation value calculation step, an evaluation value closest to the current evaluation value calculated in the evaluation value calculation step, and the current evaluation value A program for instructing photographing when the difference is equal to or greater than a predetermined value. A face detection step of detecting the face of the subject imaged by the imaging means;
A specifying step of specifying a person of the face detected in the face detection step;
When shooting information of an image including a person specified in the specifying step is stored in a storage unit as shooting information of an image shot in the past, based on shooting information of an image including the specified person in the past A similarity calculation step for calculating the similarity with the captured image;
A program for causing a computer to execute an imaging control step for instructing imaging when the similarity calculated in the similarity calculation step is smaller than a predetermined value.

Images