JP2009194422A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To acquire an image in eyes-open state by controlling photographing timing. <P>SOLUTION: Reception of a first operation makes the shift to a preparation period, and eye opening/closing detection processing is carried out on each of frame images obtained one after another in the preparation period to measure an eye closing time (reference eye closing continuance T<SB>X</SB>) inherent in a subject. After the eye closing time (T<SB>X</SB>) is measured, it is monitored whether a state where the eyes are closed is longer than the eye closing time and when the state is observed, it is decided that the subject intends to close the eyes to instruct photographing, thereby performing exposure for an object image to be recorded in a recording medium. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital still camera.

  When a person is photographed using an imaging device, the blink timing may not match the shutter timing, and photographing may be performed with the eyes closed. If the subject tries to keep his eyes open to avoid this, the possibility of shooting with an unnatural expression increases. Therefore, there is a need for a technique that enables an eye-open image to be acquired without forcing the subject to open the eye.

JP 2006-53666 A

  SUMMARY An advantage of some aspects of the invention is that it provides an imaging device that contributes to acquiring an image in an open eye state.

  A first imaging device according to the present invention includes an imaging device that outputs a signal representing a captured image by photoelectrically converting an incident optical image, and opening and closing of a human eye as a subject based on an output signal of the imaging device. Based on the detection result of the open / close state detecting means, the eye is continuously closed based on the detection result of the open / close state detecting means for detecting the state, the photographing control means for controlling the photographing timing for obtaining the target photographed image from the image sensor. A time measuring unit that measures a length of time for which the eye is continuously open or a time length for which the eye is continuously open, and the imaging control unit captures the target captured image based on a measurement result of the time measuring unit. The imaging timing for obtaining is controlled.

  By measuring the actual eye-closing time and eye-opening time of the subject and controlling the shooting timing of the target shot image using the measurement result, the possibility of acquiring an eye-opened image increases.

  Specifically, for example, in the first imaging device, the imaging control unit may detect the target imaging on the imaging element after observing a state in which the eyes are continuously closed longer than a set specified time. Take a picture to get an image.

  As a result, it is possible to perform shooting timing control that distinguishes between the eye closed intentionally by the subject and the eye closed by unconscious blink, and the possibility of acquiring an eye-opened image that matches the intention of the subject increases.

  More specifically, for example, in the first imaging device, the imaging control unit may use the timing when the transition from the closed state to the open state of the eye is observed after the observation of the state as a reference. The imaging device is caused to execute imaging for obtaining the target captured image.

  Further, for example, when the state in which the eyes are continuously open for a longer time than the set specified time is observed, the imaging control unit switches from the closed state of the eyes to the open state. After waiting for the transition, the image sensor is caused to execute imaging for obtaining the target captured image.

  Since humans cannot keep their eyes open for a long time, the longer the eye-open state continues, the higher the probability that an image in the closed-eye state will be taken. Therefore, as described above, when it is determined that the eye-opening state has continued for a predetermined time or longer, control is performed so that the target captured image is captured after waiting for the next blink. As a result, the possibility of acquiring an image in an open eye state increases.

  A second imaging device according to the present invention includes an imaging device that outputs a signal representing a captured image by photoelectrically converting an incident optical image, and opening and closing of a human eye as a subject based on the output signal of the imaging device. An open / closed state detecting unit for detecting a state, a shooting control unit for executing continuous shooting using the image sensor, and the open / closed state detecting unit for a captured image group obtained from the image sensor before the continuous shooting is performed. And a time measuring means for measuring a reference closed eye duration indicating the length of time that the eye is continuously closed based on a detection result, and the imaging control means is continuously based on the reference closed eye duration. A shooting interval in shooting is set.

  As a result, it is possible to perform shooting timing control such that at least one eye-open image is included in a group of shot images obtained by continuous shooting.

  Further, for example, in the second imaging device, the time measuring unit may be a reference eye opening duration that represents a length of time that the eye is continuously open based on a detection result of the open / closed state detecting unit with respect to the captured image group. The time is also measured, and the photographing control means sets the photographing interval based on the reference eye closing duration and the reference eye opening duration.

  As a result, it is possible to further increase the probability that the image of the open eye state is included in the captured image group obtained by continuous shooting.

  In the first or second imaging device, when the subject includes a plurality of persons, the position in the captured image of each person, the size of the face in the captured image of each person, and the capturing of each person Based on at least one of the comparison result between the face area image in the image and the face image registered in advance and the size of the eye in the photographed image of each person, One person is selected as the specific person, and the open / close state detecting means detects the open / closed state of the eyes of the specific person.

  A third imaging device according to the present invention includes: an imaging device that outputs a signal representing a captured image by photoelectrically converting an incident optical image; and opening and closing of a human eye as a subject based on an output signal of the imaging device An open / closed state detecting unit for detecting a state, and a shooting control unit for controlling a shooting timing for obtaining a target shot image from the image sensor, wherein the shooting control unit is based on a detection result of the open / closed state detecting unit. Then, the photographing timing for obtaining the target photographed image is controlled.

  By controlling the shooting timing of the target captured image based on the open / closed state of the eyes detected from the actual captured image, the possibility that an image in the open state can be acquired increases.

  Specifically, for example, in the third imaging apparatus, the imaging control unit may cause the imaging device to capture the target imaging with reference to a timing at which a transition from the closed state to the open state is observed. Take a picture to get an image.

  Also, for example, in the third imaging apparatus, when the subject includes a plurality of persons, the photographing control unit may perform the shooting for the specific person among the plurality of persons based on the open / closed state for all persons. Based on the open / closed state, the photographing timing for obtaining the target photographed image is controlled.

  Further, for example, the first, second, or third imaging device may be further provided with eye region detection means for detecting a human eye region in the captured image. For example, the open / close state detection means includes feature point extraction means for extracting feature points in the eye region, and detects the open / close state of the eyes based on the number of extracted feature points.

  This makes it possible to accurately detect the open / close state.

  ADVANTAGE OF THE INVENTION According to this invention, the imaging device which contributes to the image acquisition of an open eye state can be provided.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to fourth embodiments will be described later. First, matters that are common to each embodiment or items that are referred to in each embodiment will be described.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging apparatus 1 in FIG. 1 is a digital still camera that can shoot and record still images, or a digital video camera that can shoot and record still images and moving images.

  The imaging device 1 includes an imaging unit 11, an image memory 12, a main control unit 13, a shutter control unit 14, a display unit 15, a recording medium 16, and an operation unit 17. The operation unit 17 is provided with a shutter button 17a.

  The imaging unit 11 includes an optical system and a diaphragm (not shown), an imaging element 11a including a CCD (Charge Coupled Devices) and a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and a driver (not shown) for controlling the optical system and the diaphragm. ) And. The driver controls the zoom magnification and focal length of the optical system and the aperture of the diaphragm based on the AF / AE control signal from the main control unit 13. The image sensor 11a photoelectrically converts an optical image representing a subject incident through an optical system and a diaphragm, and outputs an electrical signal obtained by the photoelectric conversion. More specifically, the image pickup device 11a includes a plurality of light receiving pixels that are two-dimensionally arranged in a matrix, and each light receiving pixel stores a signal charge having a charge amount corresponding to an exposure time in each photographing. An analog signal from each light receiving pixel having a magnitude proportional to the amount of the stored signal charge is sequentially output to a subsequent circuit (AFE described later) in accordance with a drive pulse generated in the imaging device 1. In the following description, “exposure” means exposure of the image sensor 11a.

  An AFE (Analog Front End) (not shown) amplifies an analog signal output from the image sensor 11a and converts it into a digital signal, and then the obtained digital signal is formed by an SDRAM (Synchronous Dynamic Random Access Memory) or the like. Stored in the image memory 12. One whole image obtained by photographing by the imaging unit 11 is hereinafter referred to as a “frame image”. The image memory 12 stores image data as a digital signal representing a frame image.

  The main control unit 13 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and includes an image processing unit 21, a face region detection unit 22, an eye region detection unit 23, and an open / closed state detection. The unit 24 is configured to be included. The image data stored in the image memory 12 is given to the image processing unit 21. The image processing unit 21 appropriately performs necessary processing on the image data of the frame image to generate a video signal corresponding to the frame image, and sends the video signal to the display unit 15 to display an image based on the frame image. Displayed on the unit 15. Further, the image processing unit 21 compresses the image data of the frame image as necessary, and stores the compressed image data in the recording medium 16.

  The shutter control unit 14 controls the timing of shooting by the imaging unit 11 in cooperation with the main control unit 13. The display unit 15 is a display device formed from a liquid crystal display panel or the like, and the recording medium 16 is a nonvolatile memory such as an SD (Secure Digital) memory card. The operation unit 17 receives an operation from the outside. The content of the operation on the operation unit 17 is transmitted to the main control unit 13. The shutter button 17a is a button for instructing to capture and record a still image.

  Image data of a frame image is given to the face region detection unit 22, the eye region detection unit 23, and the open / close state detection unit 24 through the image processing unit 21. At this time, the image processing unit 21 may perform necessary image processing and send the image data after the image processing to the face region detection unit 22, the eye region detection unit 23, and the open / close state detection unit 24. An image (that is, the frame image itself or the frame image after the above image processing) represented by the image data sent to the face area detection unit 22, the eye area detection unit 23, and the open / closed state detection unit 24 is hereinafter referred to as “input image”. "

  The face area detection unit 22 detects and extracts a face area, which is an area including a human face portion, from the input image, and generates face area information representing the position and size of the extracted face area in the input image. . The process performed by the face area detection unit 22 including the process of extracting the face area and generating the face area information is hereinafter referred to as “face detection process”. Note that the face area is a rectangular area. However, the outer shape of the face area may be other than a rectangle. The position of the face area is defined by the coordinate value of the center of the face area on the input image, for example. The size of the face area is defined by, for example, the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the face area on the input image.

  Various methods are known as a method for detecting a face included in an image, and the face region detection unit 22 can employ any method. For example, a face area may be detected by extracting a skin color area from an input image as in the method described in Japanese Patent Application Laid-Open No. 2000-105819, or Japanese Patent Application Laid-Open No. 2006-21111 or Japanese Patent Application Laid-Open No. 2006-72770. The face area may be detected using the method described in the publication.

  Typically, for example, the image of the region of interest set in the input image is compared with a reference face image having a predetermined image size to determine the similarity between both images, and the region of interest is determined based on the similarity. It is detected whether or not a face is included (whether or not the region of interest is a face region). In the input image, the region of interest is shifted one pixel at a time in the horizontal direction or the vertical direction. Then, the image in the region of interest after the shift is compared with the reference face image, the similarity between both images is determined again, and the same detection is performed. In this way, the region of interest is updated and set, for example, while being shifted pixel by pixel from the upper left to the lower right of the input image. In addition, the input image is reduced at a certain rate, and the same face detection process is performed on the reduced image. By repeating such processing, a face of any size can be detected from the input image.

  The eye area detection unit 23 recognizes the position and size of the face area based on the face area information, detects and extracts an eye area that includes an eye part from the face area, and extracts the extracted eye area. The eye region information representing the position and size in the input image is generated. The process performed by the eye area detection unit 23 including the process of extracting the eye area and generating the eye area information is hereinafter referred to as “eye detection process”. Although it is possible to consider an eye area as an area including two eyes existing in one face area, now, an area including only one of the two eyes is considered as an eye area. To do. Note that the eye area is a rectangular area. However, the outer shape of the eye region can be other than a rectangle. The position of the eye region is defined by, for example, the coordinate value of the center of the eye region on the input image. The size of the eye region is defined by, for example, the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the eye region on the input image.

  FIG. 2 shows an image 200 as an example of an input image and a face area 201 extracted from the image 200. A broken line inside area denoted by reference numeral 201 is a face area. FIG. 3A is an enlarged view of the face area 201. A broken line internal region denoted by reference numeral 202 is an eye region extracted from the face region 201 by the eye region detection unit 23. FIG. 3B is an enlarged view of the eye region 202.

  Various methods are known as a method for extracting an eye region from a face region, and the eye region detection unit 23 can employ any method. For example, the reference face image for detecting the face area may be replaced with the reference eye image, and the same process as that for detecting the face area may be performed. That is, for example, the image of the region of interest set in the input image is compared with the reference eye image having a predetermined image size to determine the similarity between both images, and the eye is included in the region of interest based on the similarity It is only necessary to detect whether or not the target region is the eye region. At this time, the region of interest is scanned within the face region.

  The open / close state detection unit 24 recognizes the position and size of the eye region based on the eye region information, and detects the open / closed state of the eye existing in the eye region. That is, it is determined whether the eye is in an open state (hereinafter referred to as an open eye state) or in a closed state (hereinafter referred to as an eye closed state). Processing for performing this determination by the open / close state detection unit 24 is hereinafter referred to as “open / close state detection processing”.

  For example, an image representing a standard pupil as shown in FIG. 4 is stored in the imaging apparatus 1 in advance as a template, and template matching between the template and an image in the eye area of the input image is performed, thereby It is detected whether or not a pupil is present in the region. Then, it is determined that the eye is open when the presence of the pupil is detected, while it is determined that the eye is closed when the presence of the pupil is not detected. When the eyes are closed, the template is placed at any position in the eye region and the template matching is performed, the similarity between the template and the image in the eye region is smaller than a predetermined reference similarity. In such a case, it is determined that there is no pupil in the eye area of the input image. Conversely, if a similarity greater than the reference similarity is observed, it is determined that a pupil is present in the eye area of the input image.

  Alternatively, for example, the open / closed state of the eye may be detected through a feature point extraction process. In this method, when the eyes are open, a large number of feature points are extracted from the inside and the periphery of the pupil. On the other hand, when the eyes are closed, the feature points are extracted only from the end portions of the eyes. Is used. When this method is used, the open / close state detection unit 24 first performs a process of extracting feature points existing in the eye area of the input image (see FIG. 5A). The feature point means a pixel having a large amount of change in shading in the horizontal and vertical directions. Such a feature point is obtained by using a feature point extractor (not shown) such as a Harris corner detector or a SUSAN corner detector. Can be automatically extracted. Further, feature points may be extracted using a feature point extraction method used in an image tracking method called Kanade & Lucas tracker.

  Images 210 and 220 in FIG. 5A represent an image of the eye region in the open eye state and an image of the eye region in the closed eye state, respectively. The cross-shaped marks shown in the eye region images 210 and 220 represent feature points extracted from the eye region images 210 and 220. In FIG. 5 (a) and FIGS. 5 (b) and 5 (c) described later, reference numeral 211 is given only to some of the feature points extracted from the eye region image 210, Reference numerals 221 are given only to some of the feature points extracted from the eye region image 220. The cross-shaped mark and the star-shaped mark described later are shown superimposed on the eye region image for convenience of explanation, and such a mark exists in the actual image. Not a translation.

  After the feature points are extracted, the open / close state detection unit 24 detects the positions of the two end points of the eyes existing in the eye region based on the feature point extraction results. The end point means the end of the eye in the left-right direction of the face. Specifically, a feature point located on the left side of the face and a feature point located on the right side of the face among the plurality of feature points extracted from the eye region are detected as both end points of the eye. Note that the eye area detection unit 23 uses as small an area as possible as the eye area, including the edge of the eyelid, so that a point far from the true end point of the eye (such as the temple or between eyebrows) is not erroneously detected as the end point of the eye. Try to extract.

  The left side of FIG. 5B shows a plurality of feature points 211 and detected end points superimposed on the eye region image 210, and the right side of FIG. 5B shows a plurality of feature points 221 and detected. An example in which both end points are superimposed on the eye region image 220 is shown. The star-shaped marks 212 and 213 shown in the eye area image 210 of FIG. 5B are the end points of the eye area for the eye area image 210, and in the eye area image 220 of FIG. 5B. The star-shaped marks 222 and 223 shown are the end points of the eye area for the eye area image 220.

  After that, the open / close state detection unit 24 sets a rectangle having two line segments passing through each end point of the eye as two sides (hereinafter referred to as an end point reference rectangle) in the eye region, and the feature points existing in the end point reference rectangle Count the number of However, it is assumed that the two sides are two opposite sides that form a rectangle. If the number is larger than a predetermined reference value, it is determined that the focused eye is open, and if not, it is determined that the focused eye is closed. Note that since the number of feature points increases as the size of the eye region increases, the reference value may be changed according to the size of the endpoint reference rectangle (for example, the size thereof). Normalization of the reference value according to

  The left side of FIG. 5C shows a plurality of feature points 211, both end points 212 and 213 and the end point reference rectangle 215 superimposed on the eye region image 210, and the right side of FIG. 5C shows a plurality of feature points. 221, both end points 222 and 223 and set end point reference rectangle 225 are superimposed on the eye region image 220.

  Focusing on the eye area image 210, a rectangular setting method will be supplementarily described. The length in the left-right direction of the end point reference rectangle 215 set in the eye region image 210 is determined by the positions of both end points 212 and 213 of the eye. The length in the vertical direction of the end point reference rectangle 215 is, for example, a length obtained by multiplying the distance between the both end points 212 and 213 of the eye by a predetermined coefficient (for example, 0.5). In addition, a bisector dividing the end point reference rectangle 215 vertically (that is, a bisector to extend in the left-right direction) passes through both end points 212 and 213. The end point reference rectangle 225 for the eye region image 220 is set in the same manner.

  Thus, by using feature point extraction, it is possible to accurately detect an open / closed state that is not easily affected by differences in pupil color or illumination.

  When an operation for instructing shooting and recording of one or a plurality of still images is performed, the main control unit 13 records one or a plurality of frame images obtained by the exposure after the operation on the recording medium 16. However, the imaging apparatus 1 can perform a characteristic photographing process using the open / close state detection unit 24 and the like.

When this characteristic shooting process is performed, a preparation period and an actual shooting period are set as shown in FIG. When a transition is made from another operation mode to an operation mode in which still image shooting is possible, first, a transition is made to the normal period. Thereafter, when the user performs a predetermined operation (hereinafter referred to as a first operation) on the operation unit 17, the preparation period starts. The start time of the preparation period is the time when the first operation is performed. The start time of the preparation period is represented by timing t1. After that, when the prescribed conditions are satisfied, the preparation period shifts to the actual shooting period. Beginning of the end and the real shooting period of the preparation period is consistent, represents the start time of the actual recording period at timing t _A. In the actual shooting period, a frame image to be acquired corresponding to the first operation is shot, and when the shooting is finished, the transition is made from the actual shooting period to the normal period. A frame image obtained by shooting in the actual shooting period is hereinafter referred to as “target image”.

  In the normal period and the preparation period, the imaging unit 11 sequentially captures images at a predetermined frame period under the control of the main control unit 13 and the shutter control unit 14, and periodically acquires frame images. As described above, the obtained image data of each frame image is stored in the image memory 12. The image memory 12 can store one or a plurality of frame images, and new frame images are sequentially overwritten and saved. Assume that the first, second, third,... Frame images are sequentially acquired in this order as time progresses.

  For each frame image obtained sequentially in the preparation period (strictly, for each input image based on each frame image), the face area detection unit 22, the eye area detection unit 23, and the open / close state detection unit 24 are: The face detection process, the eye detection process, and the open / closed state detection process described above are performed to detect the open / closed state of the eyes of a person as a subject in each frame image. Strictly speaking, the face area detection unit 22, the eye area detection unit 23, and the open / close state detection unit 24 perform the face detection process, the eye detection process, and the open / close state detection process on the input image. Since the input image is an image based on the frame image, the difference between the input image and the frame image is ignored in the following description. In the following description, it is assumed that each frame image includes a human face as a subject.

  As examples for explaining the operation of the above-described characteristic photographing processing, first to fourth examples will be described below. As long as there is no contradiction, the matters described in one embodiment also apply to other embodiments.

<< First Example >>
First, the first embodiment will be described. FIG. 7 is a flowchart showing the procedure of the photographing processing operation according to the first embodiment. FIG. 8 is a conceptual diagram showing the temporal flow of this operation. In the first embodiment, one target image is acquired.

  First, in step S11, the main control unit 13 confirms whether or not the shutter button 17a has been pressed. If the shutter button 17a has been pressed, the main control unit 13 proceeds to step S12. In the first embodiment, the operation of pressing the shutter button 17a corresponds to the first operation, and the pressing timing of the shutter button 17a is the timing t1. When the shutter button 17a is pressed, the process proceeds to the preparation period, and during the preparation period, the process of step S12 is repeatedly executed until the process proceeds to step S13.

In step S <b> 12, the open / close state detection unit 24 detects the open / close state of the human eye in the sequentially obtained frame images. Based on the detection result, the main control unit 13 constantly monitors whether or not the human eye state in the frame image has changed from the closed eye state to the open eye state. When the transition is confirmed, it is determined that the condition for shifting to the actual photographing period is satisfied, and the process proceeds to step S13. The timing when the transition is confirmed is the timing t _A in the first embodiment (see FIG. 6).

More specifically, for example (see FIG. 8), after the timing t1, the first, second,..., 20th frame images are sequentially obtained, and the eye states in the first to 16th frame images are determined. It is detected that the eye is in an open state, the eye state in the 17th to 19th frame images is detected to be a closed eye state, and the eye state in the 20th frame image is detected to be an open eye state. In this case, it is determined that the eye state has changed from the closed eye state to the open eye state between the 19th and 20th frame images. Time for this transition is confirmed a timing t _A.

In step S13, the main control unit 13 and the shutter control unit 14 take the target image at the earliest possible time after the timing t _A (that is, perform exposure to obtain the target image), and then image the target image. Get the data. Although it is most preferable to start the exposure of the target image from the timing t _A, the exposure of the target image actually starts slightly later than the timing t _A due to various restrictions. This delay is called time lag. The image data of the target image is once stored in the image memory 12, and then stored in the recording medium 16 through necessary processing such as compression processing (step S14).

  As described above, it is possible to acquire the target image with the eyes open by controlling the photographing timing.

  Note that if there is only one person within the shooting area of the imaging apparatus 1, it is only necessary to monitor the open / closed state of the eye of that person. When a plurality of face areas are detected from the above, one person is selected as the specific person from the plurality of persons. In step S12, when the eye opening / closing state of the specific person is monitored and the transition from the closed eye state to the open eye state of the specific person is confirmed, the transition condition to the actual photographing period is satisfied. The determination may be made to proceed to step S13. Processing for selecting a specific person is executed by the main control unit 13.

  Assuming that a plurality of persons are included in the shooting area, a method for selecting a specific person will be exemplified. First, attention is paid to a certain frame image (hereinafter referred to as a determination frame image) obtained after the timing t1 and during the preparation period. The determination frame image is typically a frame image obtained by photographing immediately after timing t1, for example. Note that a frame image obtained immediately before the timing t1 may be handled as a determination frame image. When a plurality of face regions are extracted from the determination frame image, a specific person is selected using any of the following first to fourth selection methods.

  In the first selection method, as shown in FIG. 9A, the person located at the center in the determination frame image is selected as the specific person. Specifically, the distance (the distance on the image) between the center position of the face area extracted from the determination frame image and the center position of the determination frame image is obtained for each face area, and the distance of the smallest face area is determined. A corresponding person may be selected as a specific person.

  In the second selection method, as shown in FIG. 9B, the person who has the largest face in the determination frame image is selected as the specific person. In other words, the person with the shortest subject distance is selected as the specific person. Specifically, the size of a plurality of face areas extracted from the determination frame image may be compared, and the person corresponding to the largest face area may be selected as the specific person.

  In the third selection method, as shown in FIG. 9C, a person registered in advance in the imaging apparatus 1 is selected as a specific person. When the third selection method is adopted, a face image representing the face of a person to be selected as a specific person is stored in advance in a memory (not shown) in the imaging apparatus 1 as a registered face image. The main control unit 13 calculates the degree of similarity between the image in each face area and the registered face image by comparing the image in each face area extracted from the determination frame image with the registered face image. Then, based on the calculated similarity, the face area for the registered person's face is specified, and the person corresponding to the specified face area is selected as the specified person.

  In the fourth selection method, considering that the detection accuracy of the open / closed state of the eye deteriorates when the size of the eye in the image is small, as shown in FIG. A person with a large size is selected as a specific person. That is, the size of the plurality of eye regions for the plurality of face regions extracted from the determination frame image is compared, and the person corresponding to the largest eye region is selected as the specific person. The size of the eye region used in this comparison is, for example, the length of the eye region in the vertical direction of the eye (that is, the direction connecting the upper eyelid and the lower eyelid) (the length on the determination frame image).

  It is also possible to select a specific person by combining two or more of the first to fourth selection methods described above. For example, the distance used in the selection of the first selection method, the size of the face region used in the selection of the second selection method, the similarity used in the selection of the third selection method, and the fourth selection method The specific person may be selected in consideration of the size of the eye region used in the selection.

  In addition, without selecting a specific person, the eye open / closed state of each person included in the imaging region is monitored in step S12, and the transition from the closed eye state to the open eye state of any person's eyes is confirmed. If it is determined that the condition for shifting to the actual shooting period is satisfied, the process may proceed to step S13.

  Alternatively, in step S12, the open / closed state of the eyes of each person included in the shooting area is monitored without selecting a specific person, and when it is determined that the eyes of all the persons are open, the actual shooting period It may be determined that the condition for shifting to is satisfied, and the process may proceed to step S13. For example, consider a case where three persons are included in the shooting area. In this case, after the timing t1, when a frame image is obtained in which two eyes are open and the remaining one eye is closed, after that, pay attention to the eyes of the person whose eyes are closed, When the transition of the focused eye from the closed state to the open state is confirmed, it is determined that the condition for shifting to the actual photographing period is satisfied and the process proceeds to step S13.

<< Second Example >>
Next, a second embodiment will be described. According to the first embodiment, the target image can be acquired with the eyes open. However, if the eyes are open, the subject's facial expression is not necessarily good, and taking the target image immediately after an unconscious blink does not always match the intention of the subject. Therefore, in the second embodiment, the intentional closing of the subject is detected, and the target image is shot at a timing according to the intention of the subject.

  It should be noted that “eye” in the following explanation including explanations of the second embodiment and other embodiments described later is one person included in the imaging region and is extracted from the frame image unless otherwise specified. If the number of face areas to be extracted is 1, it refers to the eye of the eye area extracted from the face area, and the number of face areas extracted from the frame image when there are a plurality of persons included in the shooting area Is a plurality of eyes, it indicates an eye for an eye area extracted from the face area of a specific person among the plurality of persons. As a method for selecting a specific person, the method described in the first embodiment can be employed.

  FIG. 10 is a conceptual diagram illustrating a temporal flow of the photographing processing operation according to the second embodiment. In the second embodiment, one target image is acquired.

First, the main control unit 13 detects whether or not the first operation has been performed on the operation unit 17, and shifts to a preparation period when the first operation is performed. As described above, the timing when the first operation is performed is the timing t1. Thereafter, the main control unit 13 measures the length of time that the eyes are continuously closed based on the result of the open / closed state detection process for each frame image obtained after the timing t1. The measured time length, referred to as a reference eye closure duration, represent it by the symbol T _X. The time point at which the reference closed eye duration T _X is obtained is defined as timing t2.

For example, after the timing t1, first, second,..., Fifteenth frame images are sequentially obtained, and it is detected that the eye state in the first to tenth frame images is an open state, and When it is detected that the eye state in the 11th to 14th frame images is the closed eye state and the eye state in the 15th frame image is detected to be the open eye state, from 14-11 + 1 = 4, The length of time obtained by multiplying the frame period during the preparation period by 4 is obtained as the reference closed eye duration T _X.

After obtaining the reference closed eye duration T _X , the main control unit 13 determines that the time is longer than the time represented by α · T _X based on the result of the open / closed state detection process for each frame image obtained after timing t2. It is monitored whether or not the condition that the eyes are closed for a long time (hereinafter referred to as the first condition) is satisfied. A closed eye that satisfies the first condition is determined to be an intentional closing of the subject, and a closed eye that does not satisfy the first condition is determined to be a closed eye due to an unintended natural blink of the subject. Then, the first eye opening timing after the first condition is satisfied is set as timing t _A. Here, α is a predetermined coefficient having a value larger than 1. For example, α = 2.

The main control unit 13 and the shutter control unit 14 capture the target image at the earliest possible time after the timing t _A and acquire the image data of the target image. As described in the first embodiment, the exposure of the subject image is started after a lapse of a predetermined time lag from the timing t _A in practice. The image data of the target image is once stored in the image memory 12 and then stored in the recording medium 16 through necessary processing such as compression processing.

After timing t2, i-th, (i + 1),... (I + 30) frame images are sequentially obtained, and α · T _X is a time length obtained by multiplying the frame period in the preparation period by 8. A numerical example is given assuming some cases (i is a natural number).

For example, it is detected that the eye state in the i th to (i + 20) th frame images is an open eye state, and the eye state in the (i + 21) th to (i + 29) th frame images is a closed eye state. When the eye state in the (i + 30) th frame image is detected to be an open eye state, the total time length of the imaging periods of the (i + 21) th to (i + 29) th frame images is 9 frames. Since it is the period, the first condition is satisfied. Then, in the (i + 30) th frame image, the first transition from the closed eye state to the open eye state after the first condition is satisfied is observed. Accordingly, in this case, it is determined that the eye state has changed from the closed eye state to the open eye state between the capturing of the (i + 29) and (i + 30) th frame images, and the timing at which this transition is confirmed is the timing t _A. Shoot the target image.

  On the other hand, for example, it is detected that the eye state in the i-th to (i + 22) th frame image is an open eye state, and the eye state in the (i + 23) -th (i + 29) th frame image is the closed eye state. And when the eye state in the (i + 30) th frame image is detected to be an open eye state, the total time length of the imaging periods of the (i + 23) th to (i + 29) th frame images Since the length is 7 frame periods, the first condition is not satisfied. Therefore, in this case, paying attention to each frame image after the (i + 31) th frame image, whether the first condition is satisfied or not is determined again, and the same processing is repeated until the first condition is satisfied.

  As described above, by controlling the shooting timing, it is possible to distinguish between the closed eye intentionally performed by the subject and the closed eye caused by the unconscious blink. Can be obtained.

In the above-described example, the reference closed eye duration T _X is actually measured based on the result of the open / closed state detection process for each frame image obtained after the timing t1, but it is set in advance without performing the measurement. This time may be used as the reference closed eye duration T _X. In this case, an average closed eye duration (for example, 0.2 seconds) in a human blink is set in advance as the reference closed eye duration T _X.

<< Third Example >>
Next, a third embodiment will be described. If the eye-open state continues when shooting the target image, it seems that the target image in the open state can be acquired at any time, but humans cannot keep their eyes open for a long time. It is considered that the longer the duration continues, the higher the probability that the target image in the closed eye state will be shot. Therefore, in the third embodiment, when it is determined that the eye-open state continues for a certain time T _W or longer during standby for capturing the target image, the target image is captured after waiting for the next blink. FIG. 11 is a conceptual diagram showing a temporal flow of the photographing processing operation according to the third embodiment. In the third embodiment, one target image is acquired.

First, the main control unit 13 detects whether or not the first operation has been performed on the operation unit 17, and when the first operation is performed, the main control unit 13 shifts to a preparation period that can be referred to as an imaging standby period. As described above, the timing when the first operation is performed is the timing t1. Thereafter, the main control unit 13, based on the result of the opening and closing state detection processing for each frame image obtained timing t1 after measuring the length of time that the eye is open continuously, time represented by "T _W It is monitored whether or not the condition that the eyes are continuously open for a longer period of time (hereinafter referred to as the second condition) is satisfied. The main control unit 13, when the second condition is determined to be satisfied in the preparation period, after fulfillment of the second condition, and the timing t _A the timing of opening of the first eye. Note that _TW has a preset positive value.

Now, after the timing t1, the timings t3, t4, and t5 are visited in this order during the preparation period, and the eyes are continuously opened between the timings t3 and t5 based on the result of the open / close state detection process. Assume that it is detected. It is assumed that the time length between timings t3 and t4 matches the above-described fixed time T _W. Further, based on the result of the open / close state detection process, it is assumed that the eye state is detected to have transitioned from the open eye state to the closed eye state at timing t5. Then, since the second condition is satisfied when the timing t4 is reached, the main control unit 13 detects the next open eye state during the period between the timing t4 and the timing t5 and after the timing t5. Shooting of the target image is prohibited during the period until it is performed (that is, exposure of the target image is not performed). Then, after the timing t5, the timing of opening of the first eye to the timing t _A.

After timing t1, first, second, ... 100th frame image is sequentially obtained of, and, on the assumption that T _W is 50 multiplied by the time length frame period during the preparation Give numerical examples. For example, it is detected that the eye state in the first to tenth frame images is a closed eye state, the eye state in the 11th to 90th frame images is detected to be an open eye state, and the 91st When the eye state in the 99th frame image is detected to be the closed eye state and the eye state in the 100th frame image is detected to be the open eye state, the 11th to 60th frame images Since the total time length of the shooting period is 50 frame periods, the second condition is satisfied at the time of acquisition of the 60th frame image. For this reason, after that, photographing of the target image is prohibited until the open eye state for the 100th frame image is detected. Then, in the 100th frame image, the first transition from the closed eye state to the open eye state after the second condition is satisfied is observed. Therefore, in this case, it is determined that the eye state has transitioned from the closed eye state to the open eye state between the shooting of the 99th and 100th frame images, and the timing at which this transition is confirmed is taken as the timing t _A. I do.

The main control unit 13 and the shutter control unit 14 capture the target image at the earliest possible time after the timing t _A and acquire the image data of the target image. As described in the first embodiment, the exposure of the subject image is started after a lapse of a predetermined time lag from the timing t _A in practice. The image data of the target image is once stored in the image memory 12 and then stored in the recording medium 16 through necessary processing such as compression processing.

A usage example of the above processing will be described. For example, the shutter button 17a in FIG. 1 is formed so that it can be pressed in two stages. When the photographer lightly presses the shutter button 17a, the shutter button 17a is half pressed, and when the shutter button 17a is further pressed from this state, the shutter button 17a is fully pressed. Then, the operation of half-pressing the shutter button 17a is regarded as the first operation, and the time point when the half-pressed state is detected is defined as timing t1. When the shutter button 17a is fully pressed by a photographer's operation, the main control unit 13 immediately shifts from the preparation period to the actual shooting period in principle to shoot the target image. If two conditions are satisfied, exception handling is performed. That is, taking as an example the timing relationship of FIG. 11, when a operation to full-press the shutter button 17a during the period leading up to and the timing t _A a timing t4 or later, the operation of pushing the entire Immediately after the target image is not captured, the target image is captured after the transition from the closed eye state to the open eye state is observed at timing t _A.

  As described above, it is possible to acquire the target image with the eyes open by controlling the photographing timing.

<< 4th Example >>
Next, a fourth embodiment will be described. In the fourth embodiment, it is assumed that a plurality of target images are acquired by continuous shooting. Conventionally, a technique has been proposed in which a plurality of images are continuously photographed and an eye-open image is extracted from the obtained image group (see, for example, Patent Document 1). This conventional method is based on the premise that an image with an open eye is included in an image group obtained by continuous shooting, but an image with an open eye is included in the image group. However, depending on the shooting timing, there is a possibility that the eyes of the subject in all the images are in a closed or half-open state. In order to cope with such a problem, in the fourth embodiment, the timing control for continuous shooting is performed in consideration of the eye-closing time unique to the subject.

FIG. 12 is a conceptual diagram illustrating a temporal flow of the photographing processing operation according to the fourth embodiment. First, the main control unit 13 detects whether or not the first operation has been performed on the operation unit 17, and shifts to a preparation period when the first operation is performed. As described above, the timing when the first operation is performed is the timing t1. After that, the main control unit 13 determines the length of time that the eyes are continuously closed and the length of time that the eyes are continuously opened based on the result of the open / close state detection processing for each frame image obtained after the timing t1. Measure the thickness. The former time length is the reference closed eye duration T _X described in the second embodiment. The latter time length is referred to as the reference eye opening duration and is represented by the symbol T _Y. Hereinafter, abbreviated as a reference eye closure duration of T _X time T _X, the reference open-eye duration T _Y abbreviated as time T _Y.

For example, after timing t1, first, second,..., 100th frame images are sequentially obtained, and it is detected that the eye state in the first to tenth frame images is an open eye state, and It is detected that the eye state in the 11th to 14th frame images is the closed eye state, and the eye state in the 15th to 99th frame images is detected to be the open eye state, and in the 100th frame image When it is detected that the eye state is the closed eye state, the time length obtained by multiplying the frame period during the preparation period by 4 is measured as time T _X from 14-11 + 1 = 4, and 99-15 + 1 = 85. more, the time length obtained by multiplying the 85 frame period during the preparation period is measured as the time T _Y. Note that the timing t6 shown in FIG. 12 represents the transition timing from the closed eye state to the open eye state between the fourteenth and fifteenth frame images. In the numerical example corresponding to FIG. 12, the time T _X is obtained before the time T _Y , but the order of the derivation timings of both can be reversed.

After the times T _X and T _Y are obtained, the timing t _A is reached and the actual shooting period starts. For example, the timing at which the first eye opens after the times T _X and T _Y are obtained is defined as timing t _A. The method for determining the eye opening timing is as described in the other embodiments.

The main control unit 13 and the shutter control unit 14 capture the first target image at the earliest possible time after the timing t _A and acquire the image data of the first target image. As described in the first embodiment, the exposure of the first sheet of the target image after the elapse of a predetermined time lag from the timing t _A is started in practice. Due to the presence of this time lag, the eyes may be closed at the time of exposure of the first target image. FIG. 12 shows a case where the subject blinks again immediately after timing t _A. Considering this, the shooting interval between the first and second target images is set based on the time T _X.

The shooting operation after timing t _A will be described in more detail. When the number of target images to be continuously shot is represented by n, after timing t _A , timings t _A1 , t _A2 , t _A3 ,..., T _An come in this order, and the j th target at timing t _Aj . An image is taken. Here, n is an integer greater than or equal to 2, and j is a natural number less than or equal to n. Incidentally, strictly speaking, the exposure of the j-th target image is started at the timing t _Aj, the exposure of the j-th target image at the time of the lapse of time of the exposure period from the timing t _Aj is to complete Since the exposure period is sufficiently shorter than the time T _X and the time T _Y , the existence of the exposure period is ignored (the time length of the exposure period is regarded as zero).

The length of time between timings t _Aj and t _{A (j + 1)} represents the shooting interval of the j-th and (j + 1) -th target images that are continuously shot (where j <n). . The main control unit 13 and the shutter control unit 14 control the imaging unit 11 so that the shooting interval in the continuous shooting becomes equal to the time length represented by β · T _X. Here, β is a coefficient having a value larger than 1. It is estimated that the closed eye time unique to the subject of interest is the time T _X after the timing t _A. Therefore, if the shooting interval is set in this manner, even if the subject blinks again immediately after timing t _{A as} shown in FIG. 12 and the eyes are closed when the first target image is exposed, the second image is taken. It is expected that the eyes are open when the target image is exposed. That is, if two target images are acquired, at least one target image in an open eye state can be acquired.

However, if the value of the coefficient β is too large, as shown in FIG. 13, the first and second blink timings after the timing t _A overlap with the shooting timings of the first and second target images. It can also be considered. Therefore, it may be determined based on the coefficient β to the time T _Y. Specifically, the value of the coefficient β may be set so that the time length represented by β · T _X is longer than the time T _X and shorter than the time T _Y. Since blinking is considered to be performed periodically with a substantially uniform blinking period, it is possible to acquire at least one target image within one blinking period by setting the value of the coefficient β in this way. Become.

Normally, the time T _Y (eg, 3 seconds) is sufficiently longer than the time T _X (eg, 0.2 seconds). Utilizing this characteristic, it is also possible to omit the measurement of the time T _Y during the preparation period. For example, in a normal blink, when it can be assumed that the length of time that the eyes are continuously open is at least five times the length of time that the eyes are continuously closed, the result of the open / close state detection process The value of the coefficient β can be set in advance within a range satisfying the inequality “1 <β <5”. Even if it is set in this way, substantially the same effect as when the coefficient β is set using the actually measured time T _Y can be obtained, and the time for measuring the time T _Y can be obtained. There is no longer any restrictions. When omitting the measurement of the time T _Y may be handled after it has been determined that the time T _X, the timing of opening of the first eye as a timing t _A, the timing t6 cover the timing t6 in FIG. 12 as a timing t _A It is also possible to shift from the preparation period to the actual shooting period at the boundary.

  Image data of n target images obtained by continuous shooting as described above are temporarily stored in the image memory 12 and then stored in the recording medium 16 through necessary processing such as compression processing. By controlling the shooting interval in continuous shooting as in this embodiment, it is possible to reliably acquire at least one target image with the eyes open.

In the above example, the first eye opening timing after the time T _X (or times T _X and T _Y ) is obtained is the timing t _A , but a predetermined operation is performed on the operation unit 17. It is possible to wait for the operation to be performed, and the timing at which the operation is performed may be a timing t _A. That is, after the time T _X (or the times T _X and T _Y ) is obtained, the main control unit 13 performs an operation for instructing the operation unit 17 to start continuous shooting (hereinafter referred to as a second operation). waiting for is made, the timing at which the second operation is performed may be handled as the timing t _a. Alternatively, the open / closed state detection process is performed after the second operation is performed, and the first eye opening timing after the second operation is performed as the timing t _A based on the result of the process. Good. Further alternatively, a timing at which the face in the face area in the frame image is determined to be a smile after the time T _X (or the times T _X and T _Y ) is obtained using a known smile recognition process. May be handled as the timing t _A.

In the second and third embodiments, it is assumed that only one target image is acquired. However, the continuous shooting method in the fourth embodiment can be applied to the second and third embodiments. . When the continuous photographing method of the fourth embodiment is applied to the second or third embodiment, the time T _X (or times T _X and T _Y ) is measured during the preparation period, and the second or third embodiment is performed. the timing t _a n pieces of the target image in the scanning interval beta · T _X from (t _a of FIG. 10 or FIG. 11) may be continuously captured in the example. Also in this case, actually, the first target image is taken after a predetermined time lag from the timing t _A.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 4 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
In the above-described example, the length of time that the eyes are continuously closed is measured only once during the preparation period, and the measurement time is set as the time T _X. Measurement may be performed a plurality of times during the preparation period, and an average of a plurality of measured time lengths may be used as the time T _X. Similarly, the eye is measured a plurality of times during the preparation time length open and duration, may be used an average of a plurality of time lengths measured as the time T _Y.

[Note 2]
In the above example, the eye area is an area including only one eye existing in one face area, and one eye area is extracted from one face area. In this case, each of the above-described imaging controls is performed based on the open / closed state of one eye of a person as a subject, but two eye areas are extracted from one face area, and the two eye areas are extracted. An open / close state detection process may be performed on each to monitor the open / close state of both eyes of the subject. In this case, for example, when it is detected that both eyes are open by the open / closed state detection process, it is determined that the eye of the person as the subject is in the open state, and it is detected that at least one eye is closed. In this case, it may be determined that the eyes of the person as the subject are in the closed state. Further, the imaging device 1 may be formed so that it is possible to freely change whether to monitor only the open / closed state of one eye or whether to monitor the open / closed state of both eyes.

[Note 3]
The imaging apparatus 1 in FIG. 1 can be realized by hardware or a combination of hardware and software. In particular, the function of the main control unit 13 can be realized by hardware, software, or a combination of hardware and software. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. Further, all or part of the functions realized by the main control unit 13 are described as a program, and the program is executed on a program execution device (for example, a computer) to realize all or part of the functions. You may make it do.

[Note 4]
For example, it can be considered as follows. The photographing control means for controlling the photographing timing of the target image is realized by the main control unit 13 and the shutter control unit 14. In the fourth embodiment, the photographing control unit acquires a plurality of target images by causing the image sensor 11a to perform continuous photographing. Further, the time measuring means for measuring the time T _X or T _Y based on the result of the open / close state detection process is realized by the main control unit 13 including the open / close state detection unit 24. In addition, when detecting the open / closed state of the eye through the feature point extraction process, the open / closed state detection unit 24 also has a function as a feature point extracting unit.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is an example of an input image to the face area | region detection part of FIG. 1, an eye area | region detection part, and an open / close state detection part. FIG. 3 is an enlarged view (a) of a face area extracted from the input image of FIG. 2 and an enlarged view (b) of an eye area extracted from the face area. It is a figure showing the pupil template which can be utilized for the process in the opening / closing state detection part of FIG. It is a figure for demonstrating an example of the processing content in the opening-and-closing state detection part of FIG. It is a figure which shows the time relationship of the normal period, preparation period, and real imaging | photography period defined regarding imaging | photography operation | movement of the imaging device of FIG. 3 is a flowchart illustrating a procedure of a photographing processing operation according to the first embodiment of the present invention. It is a conceptual diagram showing the time flow of the imaging | photography process operation which concerns on 1st Example of this invention. It is a figure for demonstrating the method of selecting one specific person from several persons. It is a conceptual diagram showing the time flow of the imaging | photography process operation which concerns on 2nd Example of this invention. It is a conceptual diagram showing the time flow of the imaging | photography processing operation which concerns on 3rd Example of this invention. It is a conceptual diagram showing the time flow of the imaging | photography process operation which concerns on 4th Example of this invention. It is another conceptual diagram showing the time flow of the imaging | photography process operation | movement which concerns on 4th Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 11a Imaging element 12 Image memory 13 Main control part 14 Shutter control part 17 Operation part 17a Shutter button 21 Image processing part 22 Face area detection part 23 Eye area detection part 24 Opening / closing state detection part

Claims (11)

An image sensor that outputs a signal representing a captured image by photoelectrically converting the incident optical image; and
An open / closed state detecting means for detecting an open / closed state of a human eye as a subject based on an output signal of the image sensor;
Photographing control means for controlling photographing timing for obtaining a target photographed image from the image sensor;
Based on the detection result of the open / closed state detecting means, comprising: a time measuring means for measuring a length of time for which the eye is continuously closed or a time length for which the eye is continuously opened;
The imaging apparatus, wherein the imaging control unit controls imaging timing for obtaining the target captured image based on a measurement result of the time measuring unit. The imaging control unit causes the imaging device to perform imaging for obtaining the target captured image after observing a state in which the eye is continuously closed for longer than a set specified time. The imaging device according to claim 1. The imaging control means performs imaging for obtaining the target captured image on the image sensor with reference to a timing at which a transition from the closed state of the eye to the open state is observed after the observation of the state. The imaging apparatus according to claim 2, wherein the imaging apparatus is executed. The imaging control means waits for the next transition from the closed state of the eye to the open state when the state where the eye is continuously open is observed for longer than a set specified time. The imaging apparatus according to claim 1, wherein the imaging device is caused to perform imaging for obtaining the target captured image. An image sensor that outputs a signal representing a captured image by photoelectrically converting the incident optical image; and
An open / closed state detecting means for detecting an open / closed state of a human eye as a subject based on an output signal of the image sensor;
Photographing control means for performing continuous photographing using the image sensor;
Based on the detection result of the open / close state detection means for the group of captured images obtained from the image sensor before the continuous imaging is performed, a reference eye-closed duration that represents the length of time that the eye is continuously closed is measured. A time measuring means for performing,
The imaging apparatus, wherein the imaging control unit sets an imaging interval in continuous imaging based on the reference closed eye duration. The time measuring means also measures a reference eye opening duration representing the length of time that the eye is continuously open based on the detection result of the open / close state detecting means for the captured image group,
The imaging apparatus according to claim 5, wherein the imaging control unit sets the imaging interval based on the reference eye closing duration and the reference eye opening duration. When the subject includes a plurality of persons,
The position of each person in the captured image,
The size of the face in each person's captured image,
A comparison result between the face area image in the photographed image of each person and the face image registered in advance, and
The size of the eye in each person's captured image,
Selecting one person among the plurality of persons as a specific person based on at least one of
The imaging apparatus according to claim 1, wherein the opening / closing state detecting unit detects an opening / closing state of the eyes of the specific person. An image sensor that outputs a signal representing a captured image by photoelectrically converting the incident optical image; and
An open / closed state detecting means for detecting an open / closed state of a human eye as a subject based on an output signal of the image sensor;
Photographing control means for controlling a photographing timing for obtaining a target photographed image from the image sensor,
The imaging apparatus, wherein the imaging control unit controls an imaging timing for obtaining the target captured image based on a detection result of the open / close state detection unit. The imaging control unit causes the imaging device to perform imaging for obtaining the target captured image with reference to a timing at which a transition from the closed state to the open state of the eye is observed. The imaging device according to claim 8. When the subject includes a plurality of persons,
The photographing control means controls the photographing timing for obtaining the target photographed image based on the open / closed state for all persons or based on the open / closed state for a specific person among the plurality of persons. The imaging apparatus according to claim 8 or 9, wherein the imaging apparatus is characterized. An eye area detecting means for detecting an eye area of a person in the captured image;
The open / closed state detecting unit includes a feature point extracting unit that extracts feature points in the eye region, and detects the open / closed state of the eye based on the number of extracted feature points. The imaging device according to claim 1.