JP2012034276A - Imaging apparatus and image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of recording efficiently an image of a face with the eyes closed like a tearful face or a sleeping face of a photographic subject from among continuously photographed images.SOLUTION: A face evaluation value on the basis of prescribed conditions from a plurality of acquired images is calculated, and it is decided whether a specific photographic subject is in a state where the eyes are closed or the eyes are opened. Then, in the case where it is decided that the specific photographic subject is in a state where the eyes are closed, change over is carried out into a eyes-closed priority state, and in the case where it is decided that the specific photographic subject is in a state where the eyes are opened, change over is carried out into a eyes-opened priority state. In addition, in the case where it is decided that the specific photographic subject is in a state where the eyes are closed when it is in the eyes-closed priority state, and in the case where it is decided that the specific photographic subject is in a state where the eyes are opened when it is in the eyes-opened priority state, an image evaluation value is calculated to be high from the face evaluation value in comparison to the otherwise case. Then, images to be recorded are selected in accordance with the calculated image evaluation value.

Description

  The present invention relates to an imaging apparatus, an image processing method, and an image processing program, and in particular, an imaging apparatus and an image processing method for preferentially selecting and recording an image desired by a user from a plurality of images that are continuously captured, And an image processing program.

  In recent years, there has been a demand for an image processing technique that automatically records an image of a face of a specific subject from continuously shot images. For example, there has been proposed an imaging apparatus that evaluates each of a plurality of images based on a predetermined condition and ranks the images based on the evaluation (see Patent Document 1). In this imaging apparatus, the photographer can select, for example, “blind” or “smile” as evaluation items. When “blind” is selected, the image evaluation unit increases the image evaluation for the image data with the eyes open, and decreases the image evaluation for the image data with the eyes closed. Then, the order determination unit 56 determines the priority order of the image data based on the image evaluation value of the image data, and the image selection unit selects the image data with the determined high priority order.

JP 2009-272740 A

  However, in the imaging apparatus described in Patent Document 1 described above, when “blind” is selected as the evaluation item, the image evaluation value of the image data in which the eyes are not meditated is high. If the photographer wants to shoot a face with a closed eye such as a crying face or sleeping face, the priority of those image data will be lower, and it is more likely not to be selected by the image selector, There is a possibility that an image desired by the photographer cannot be obtained.

  The present invention has been made in view of the above-described problems, and can efficiently record an image of a face with a closed eye such as a crying face or a sleeping face of a subject from continuously shot images. An object is to provide an imaging apparatus, an image processing method, and an image processing program.

  In order to achieve the above object, an imaging apparatus according to the present invention has an imaging unit capable of continuously imaging a subject, and a first evaluation value based on a predetermined condition from an image obtained from the imaging unit. A first evaluation value calculating means for calculating; a determining means for determining whether a specific subject is in a state where his eyes are meditated or in a state where his eyes are open from the image obtained from the imaging means; If it is determined from the plurality of images obtained from the imaging means that the specific subject is in a state of meditating eyes, priority is given to the eye-meditation image in which the specific subject is meditating eyes When the determination means determines that the specific subject is in an open state, the specific subject switches to an opening priority state that prioritizes the opening image in which the specific subject has open eyes. Switching means and front Second evaluation value calculation means for calculating a second evaluation value to be applied to the image from the first evaluation value, and selection means for selecting an image to be recorded according to the calculated second evaluation value And the second evaluation value calculating means is in the eye-meditation priority state, when the judgment means determines that the specific subject is in a state of meditating eyes, The second evaluation value calculated from the first evaluation value is set higher than the case where it is determined that the state is open. The second evaluation value calculated from the first evaluation value when it is determined that the subject is in an open state is more than in the case where it is determined that the subject is in a closed state. It is characterized by making it high.

  According to the present invention, it is possible to efficiently record an image of a face with a closed eye such as a crying face or a sleeping face of a specific subject, and it is also possible to efficiently record an image in which the subject's eyes are open.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. FIGS. 2A and 2B are diagrams for explaining switching between an eye-meditation priority state and an eye opening priority state in the imaging apparatus of FIG. 1, in which FIG. (C) is when the subject has eyes open at the beginning and the eyes are closed from the middle, and (d) is the subject has closed eyes from the beginning and opens the eyes once in the middle, but again Shown when eyes closed. FIG. 3 is a flowchart illustrating a flow of image selection processing executed by the imaging apparatus of FIG. 1 (No. 1). 6 is a flowchart (part 2) of a flow of image selection processing executed by the imaging apparatus of FIG. FIG. 6 is a flowchart (part 3) of a flow of image selection processing executed by the imaging apparatus of FIG. 1. It is a flowchart which shows the face evaluation value calculation process in step S202 of FIG. 3A. 5A and 5B are diagrams for explaining the evaluation of the face position in step S301 in FIG. 4, where FIG. 5A shows an example of the reference position of the face position in an image when the image pickup apparatus is held sideways, and FIG. An example of a reference position of a face position in an image when the image pickup apparatus is held vertically is shown. It is a flowchart which shows the flow of the image selection process performed with the imaging device which concerns on the 2nd Embodiment of this invention (the 1). It is a flowchart which shows the flow of the image selection process performed with the imaging device which concerns on the 2nd Embodiment of this invention (the 2). It is a flowchart which shows the flow of the image selection process performed with the imaging device which concerns on the 2nd Embodiment of this invention (the 3).

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a schematic configuration of an imaging apparatus according to the first embodiment of the present invention.

  The photographing lens 101 is a photographing lens including a zoom mechanism, a focus lens mechanism, and an aperture shutter mechanism. The imaging element 102 is an imaging element as a light receiving unit or a photoelectric conversion unit that converts reflected light from a subject into an electrical signal. The A / D conversion unit 103 is an A / D conversion unit including a CDS circuit that removes output noise of the image sensor 102 and a non-linear amplification circuit that is performed before A / D conversion. The image processing unit 104 performs image processing on the image data output from the A / D conversion unit 103. The image processing unit 104 can also extract a signal component in a specific frequency band related to luminance and detect the focus state of the subject.

  The format conversion unit 105 is an image format conversion unit that converts the image data output from the image processing unit 104 into a JPEG format or the like. The DRAM 106 is configured by a DRAM or the like capable of high-speed data access. The DRAM 106 is used as a temporary image storage means or a working memory for image compression / decompression. The image recording unit 107 is an image recording unit including a recording medium such as a memory card and its interface.

  A system control unit (hereinafter referred to as “CPU”) 108 is a system control unit that controls a system such as an imaging sequence. The operation unit 109 is an operation unit for operating the imaging apparatus from the outside. The operation unit 109 includes the following, for example. Menu switch for performing various settings such as shooting functions and image playback settings of the imaging device, zoom lever for instructing the zoom operation of the shooting lens, changeover switch for switching between shooting mode and playback mode, etc. It is.

  The shooting mode switch 110 is a shooting mode switch for setting the face detection mode to be turned on or off. The main switch 111 is a main switch for turning on / off the power of the imaging apparatus. SW1_112 is a switch for performing a shooting standby operation such as AF or AE. SW2_113 is a photographing switch for performing photographing after the operation of SW1_112.

  The face detection unit 114 detects a face area using the image data processed by the image processing unit 104, and extracts face information (for example, face position, size, reliability, etc.) from the detected face area. To the CPU. In addition, the face detection unit 114 performs a known face detection process on the image data from the image processing unit 104 to detect a human face area included in the image. As a known face detection process, for example, there is a method of extracting a skin color region from the gradation color of each pixel represented by image data and detecting the face with a matching degree with a face contour plate prepared in advance. . Also disclosed is a method for detecting a face by extracting facial feature quantities (for example, facial organs such as eyes, nose, mouth) using a well-known pattern recognition technique.

  The smile detection unit 115 calculates the smile level of the subject's face from the face area detected by the face detection unit 114. Specifically, the smile detection unit 115 extracts features necessary for calculating the smile level, such as the contours of the eyes, nose, mouth, and the like constituting the face, the positions of the eyes, the corners of the eyes, the nose, the corners of the mouth, and the lips. . As a method of extracting feature amounts, there is a method of extracting a part having a physical shape such as an eye or a nose from an image region by template matching. There is also a method of extracting feature amounts by machine learning learning using a plurality of sample images of the constituent parts of the face. Then, the smile detection unit 115 calculates a smile level indicating the degree of smile based on the extracted feature amount, and determines whether or not the face is a smile. As a method for calculating a smile level, for example, there is a method for calculating a smile level according to a difference between a position and a shape of a feature amount acquired from a full smile and a face not laughing.

  The eye area detection unit 116 detects an eye area where an eye exists from the face area detected by the face detection unit 114.

  The eye meditation detection unit 117 detects the area of the luminance region corresponding to the black eye from the histogram in the eye region detected by the eye region detection unit 116. Then, when the area of the luminance region corresponding to the black eye falls below a set threshold according to the size of the eye region, it is determined that the eye is closed.

  The line-of-sight detection unit 118 determines whether both eyes are detected from the eye region detected by the eye region detection unit 116 and whether the distance between the eyes is short. When the distance between both eyes is short, it is determined that the face is a front face, and the position of the pupil in the eye area detected by the eye area detection unit 116 is detected. And when the position of a pupil is located in the center of an eye area | region, it determines with the gaze direction facing the imaging device.

  The methods of face detection, smile level calculation, eye meditation detection, and line-of-sight detection described above are not limited to those described above, and various known methods can be used. Moreover, the face detection unit 114 to the line-of-sight detection unit 118 described above may be configured by hardware or software.

  The angle sensor unit 119 is configured by, for example, a gyroscope or the like, and has a function of detecting the amount of inclination of the imaging apparatus itself and sending it to the CPU 108.

  Next, the determination of the eye-meditation state / open-eye state in the imaging apparatus of FIG.

  FIG. 2 is a diagram for explaining switching between the eye-meditation priority state and the eye opening priority state in the imaging apparatus of FIG. FIG. 2A shows a state in which the subject has opened eyes from the beginning, and FIG. 2B shows a state in which the subject has closed eyes from the beginning. FIG. 2 (c) shows that the subject initially opened his eyes and closed his eyes from the middle. FIG. 2 (d) shows that the subject closed his eyes from the beginning and opened his eyes once. Indicates closed case.

  The eye-meditation priority state and the eye opening priority state are states in which each state is prioritized. That is, the eye-meditation priority state means a state in which the photographer determines that the subject wants to capture an image in which the subject is eye-closure, and the imaging device preferentially selects the eye-meditation image. Examples of images in which the subject is meditating include a crying face, a laughing face, and a sleeping face. On the other hand, the opening priority state means a state in which the photographer determines that the subject wants to take an image with an opening of the subject and the imaging device preferentially selects the opening image.

  2A to 2D, the number of shots 601 is the number of shots when the imaging apparatus performs continuous shooting. The subject's eye state 602 is the state of the subject's eye shown in each of a plurality of images obtained by continuous shooting. In the imaging apparatus, when it is determined that the eyes are meditated, “eye meditation” is performed, and when it is determined that the eyes are opened, the eyes are opened. When a plurality of subjects are shown in one image, the eye state of the subject specified by the user may be used, or the eye state of the most subject may be used. A priority state 603 indicates a priority state of the imaging apparatus.

  2 (a) and 2 (b) show the case where the maximum number of shots (15 shots in this case) is taken, and FIGS. 2 (c) and 2 (d) show that the photographer instructs the imaging apparatus to stop shooting. This is a case where shooting is finished before the maximum number of shots is reached.

  In the example shown in FIG. 2A, it can be seen that the subject is in the state of opening his eyes from the beginning. The reason why the subject is meditating in some places is because the subject blinks. In this case, in the image pickup apparatus, the photographer determines that it is a situation where the subject wants to take a picture with his eyes open.

  In the example shown in FIG. 2B, the subject is in a state where the eyes are closed from the beginning, but the initial setting of the priority state 603 is the opening priority state. Since the eye state 602 of the subject is in the eye-meditation state continuously, the eye opening priority state is switched to the eye-meditation priority state. In such a case, in the imaging apparatus, the photographer determines that it is a situation where the subject wants to take a picture with his eyes closed. In the present embodiment, the priority state 603 is switched when the subject's eye state 602 is the same two consecutive times. However, the present invention is not limited to this. For example, the number of sheets required for switching the priority state may be variable according to the frame speed.

  In the example shown in FIG. 2C, it can be seen that the subject opens his eyes at first, but closes his eyes from the middle. In this case, the number of shots is larger in the closed-eye state than in the open-open state, so when one of the continuously shot images is selected and recorded, the photographer It is determined that the subject wants to take a picture with his eyes closed.

  In the example shown in FIG. 2D, it can be seen that the subject closes his eyes from the beginning and opens his eyes once in the middle, but closes his eyes again. In this case, since the number of shots is larger in the open state than in the closed state, when the setting is made to select and record one of the continuously shot images, the photographer It is determined that the subject wants to take a picture with his eyes open.

  Next, based on the above-described operation, the flow of the eye closed / open eye state determination process in the imaging apparatus of FIG. 1 will be described with reference to FIGS. 3A, 3B, and 3C.

  3A to 3C are flowcharts showing a flow of image selection processing executed by the imaging apparatus according to the first embodiment of the present invention. Note that this processing is performed by the CPU 108 reading out and executing a control program from a memory (not shown) to control each functional unit shown in FIG.

  In FIG. 3A, in step S201, the CPU 108 controls the imaging element 102 and the like to execute imaging processing. In the imaging process, the charge signal read from the imaging element 102 is output to the DRAM 106 via the A / D conversion unit 103, the image processing unit 104, and the format conversion unit 105, and is written as image data.

  Next, in step S202, the CPU 108 calculates a face evaluation value (first evaluation value) from the image data written in the DRAM 106 (face evaluation value calculation processing). In step S202, the CPU 108 functions as a first evaluation value calculation unit. Details of this face evaluation value calculation processing will be described later.

  Next, in step S <b> 203, the CPU 108 controls the eye-meditation detection unit 117 to detect a subject that is eye-medicing from the captured image. Next, in step S204, the CPU 108 determines whether or not the main subject is in an eye-closed state based on the detection result in step S203. This main subject may exist near the center of the screen and may have the largest size or may be a subject registered in advance.

  As a result of the determination in step S204, if the CPU 108 determines that the main subject is in a state where the eyes are closed, that is, an eye-closed state, the process proceeds to step S214 in FIG. 3C. On the other hand, if it is determined that the main subject is in an open state, that is, an open state, the process proceeds to step S205.

  In step S205, the CPU 108 determines whether or not the priority state is the opening priority state. The priority state is determined by the CPU 108 reading information stored in a specific area in the DRAM 106. Information stored in a specific area in the DRAM 106 is updated as needed by processing to be described later. As an initial setting of this information, an eye opening state may be set, or an eye meditation state may be set.

  As a result of the determination in step S205, if the CPU 108 determines that the eye opening priority state is not set, that is, the eye closing priority state, the process proceeds to step S209. On the other hand, if it is determined that the eye opening priority state is set, the process proceeds to step S206, and the CPU 108 clears the eye closing counter. The eye-meditation counter is a variable for measuring how long the eye-meditation state has continued in the opening priority state.

  Next, in step S207, the CPU 108 increments the opening priority counter by one. The opening priority counter is a variable for measuring how long the opening priority state is continuous. The value of the opening priority counter is stored in the DRAM 106.

  Next, in step S208, the CPU 108 records the value of the opening priority counter in the DRAM 106, and proceeds to step S223 in FIG. 3B. The reason for recording the value of the opening priority counter is to determine how many priority states have continued. In the case where the setting for selecting and recording one image from the continuously shot images is made, the one that has been in the priority state for a longer time is preferentially left. The reason why the longer-lasting state is preferentially left is because the photographer is considered to continue the continuous shooting with such an intention.

  In step S209, the CPU 108 increments the opening counter by one. The eye-opening counter is a variable for measuring how long the time of the eye-opening state is in the eye-meditation priority state.

  Next, in step S210, the CPU 108 calculates an open face evaluation value (third evaluation value) in the eye-meditation priority state. Here, the calculation is performed so as to lower the face evaluation value obtained in step S202. For example, a predetermined numerical value (such as −100) is subtracted from the face evaluation value, or the face evaluation value is multiplied by 0.5. In step S210, the CPU 108 functions as a third evaluation value calculation unit.

  Next, in step S211, the CPU 108 determines whether the value of the opening counter is equal to or greater than a predetermined value. If the value of the eye-opening counter is greater than or equal to the predetermined value, that is, if it is determined that the eye is open more than a predetermined number of times, the photographer intends to shoot a subject with open eyes. Judge that there is. Then, the CPU 108 switches to the opening priority state in step S212. Subsequently, in step S213, the CPU 108 clears the opening priority counter and proceeds to step S223 in FIG. 3B.

  On the other hand, as a result of the determination in step S211, when it is determined that the value of the opening counter is less than the predetermined value, the process proceeds to step S223 in FIG. 3B.

  In FIG. 3C, in step S214, the CPU 108 determines whether or not the priority state is a closed eye priority state. As a result of this determination, if the CPU 108 determines that it is not in the priority state for eye closing, that is, in the priority state for opening eyes, the process proceeds to step S218. On the other hand, if it is determined that the eye-meditation priority state is set, the process proceeds to step S215, and the CPU 108 clears the opening counter.

  Next, in step S216, the CPU 108 increments the eyes closed priority counter. The eye-meditation priority counter is a variable for measuring how long the eye-meditation priority state continues. The value of the eyes closed priority counter is stored in the DRAM 106.

  Next, in step S217, the CPU 108 records the value of the eyes closed priority counter in the DRAM 106, and proceeds to step S223 in FIG. 3B. The reason for recording the value of the eye-meditation priority counter is to determine how many priority states have continued. In the case where the setting for selecting and recording one image from the continuously shot images is made, the one that has been in the priority state for a longer time is preferentially left. The reason why the longer-lasting state is preferentially left is because the photographer is considered to continue the continuous shooting with such an intention.

  In step S218, the CPU 108 increments the eyes closed counter by one. Next, in step S219, the CPU 108 calculates an eye-meditation face evaluation value (fourth evaluation value) in the opening priority state. Here, the calculation is performed so as to lower the face evaluation value obtained in step S202. For example, a predetermined numerical value (such as −100) is subtracted from the face evaluation value, or the face evaluation value is multiplied by 0.5. In step S218, the CPU 108 functions as a fourth evaluation value calculation unit.

  Next, in step S220, the CPU 108 determines whether or not the value of the eye closing counter is equal to or greater than a predetermined value. If the value of the eye meditation counter is greater than or equal to the predetermined value, that is, if it is determined that the eyes are closed for a predetermined number of times, the photographer willing to shoot the subject with the eyes closed Judge that there is. In step S221, the CPU 108 switches to the eye meditation priority state. Subsequently, in step S222, the CPU 108 clears the eyes closed priority counter and proceeds to step S223 in FIG. 3B.

In FIG. 3B, in step S223, the CPU 108 calculates an image evaluation value.
In this case, when it is determined that the eyes are closed in the eye-meditation priority state, or when it is determined that the eyes are opened in the opening priority state, the image evaluation value that is finally given to the image from the face evaluation value obtained in step S202. (Second evaluation value) is calculated. On the other hand, when it is determined that the eyes are open in the priority state of eye opening or when it is determined that the eyes are closed in the priority state of opening the eyes, from the eye opening face evaluation value calculated in step S210 or the eye closing face evaluation value calculated in step S219. Then, an image evaluation value (second evaluation value) finally given to the image is calculated. In step S223, the CPU 108 functions as a second evaluation value calculation unit.

  Next, in step S224, the CPU 108 determines whether the captured image to be processed is the first image. As a result of this determination, if it is determined that the image is the first image, the CPU 108 stores the first image in the DRAM 106 as a recording-scheduled image, and proceeds to step S227. On the other hand, if it is determined that the image is not the first image, that is, the image after the second image, the process proceeds to step S225.

  In step S225, the CPU 108 compares the image evaluation value of the captured image (current image evaluation value) with the evaluation value of the image stored in the DRAM 106 as a recording-scheduled image. The image evaluation value of the captured image to be compared is the image evaluation value calculated in step S223.

  In step S225, the CPU 108 compares the value of the eye priority priority counter with the value of the opening priority counter, and preferentially records the higher priority counter value by a predetermined value or higher regardless of the image evaluation value. An image. That is, by comparing the value of the eye-opening priority counter and the value of the opening priority counter, it is determined which priority state has continued continuously, and priority is given to the image having the priority state for a longer time. .

  In step S226, the CPU 108 determines whether or not the current image evaluation value is high. If the CPU 108 determines that the current image evaluation value is high, or if the current priority counter value is determined to be higher than a predetermined value, the CPU 108 updates the scheduled recording image to the current image in step S227. As a result, the image stored in the DRAM 106 is replaced with the current image. As a result of the determination in step S226, if the CPU 108 determines that the current image evaluation value is not high, the process proceeds to step S228.

  In step S228, the CPU 108 determines whether or not the current number of shots is the minimum number of shots. The minimum number of shots is the number of shots that must be taken when a shooting instruction is issued. In the image selection process, an image that meets the user's intention to shoot is selected from a plurality of captured images, and thus a plurality of images must be captured. If the CPU 108 determines in step S228 that the minimum number of shots has not been reached, the process returns to step S201 in FIG. 3A to continue shooting. If the CPU 108 determines that the minimum number of shots has been reached, the process proceeds to step S229.

  In step S229, the CPU 108 determines whether a shooting end instruction has been issued. Regarding the mode of the photographing end instruction, the SW2_113 pressed by the photographer may be released, or the user may directly operate the operation unit 109. If the CPU 108 determines that an instruction to end shooting has been issued, the CPU 108 records the image stored in the DRAM 106 in the image recording unit 107 in step S231, and ends this process.

  On the other hand, as a result of the determination in step S229, if it is determined that the shooting end instruction has not been issued, the CPU 108 determines whether or not the current shooting number is the maximum shooting number shooting. The maximum number of shots is set to be the same scene, because if the continuous shooting time exceeds a predetermined time, the scene to be shot changes, making it difficult to compare evaluation values. It is the number of sheets. If it is determined that the maximum number of shots has not been taken, the process returns to step S201 in FIG. 3A to perform the shooting process. On the other hand, if it is determined in step S230 that the maximum number of shots has been taken, the image written in the DRAM 106 is recorded in the image recording unit 107 in step S231, and this process ends. .

  In this way, in this process, when it is in the eye-meditation priority state, it is determined that the photographer wants to photograph an image that is meditating, such as a crying face, a laughing face, or a sleeping face. Even for an image, the image evaluation value is not lowered. On the other hand, when the eye opening priority state is set, the photographer determines that it is a situation in which an image having an opening is desired to be photographed, and lowers the evaluation value of the eye closed image. Switching to the priority state for opening the eyes or the priority state for closing the eyes can be determined based on whether the state continues continuously, or which priority state was set at the start of photographing. In addition, the user may set in advance which state is prioritized. If the priority state is not fixed, the open state is set.

  Next, the face evaluation value calculation process in step S202 of FIG. 3A will be described with reference to FIG.

  FIG. 4 is a flowchart showing the face evaluation value calculation process in step S202 of FIG. 3A.

  First, in step S301, the CPU 108 evaluates the face position. Specifically, the CPU 108 uses the face position information detected by the face detection unit 114 and the information of the photographed image, and the face position is close to the intersection of the screen divided into six or near the center of the screen. An evaluation is made based on whether or not it is in position. The face position evaluation method in step S301 will be described with reference to FIGS. 5 (a) and 5 (b).

  Reference numeral 501 in FIG. 5A denotes a photographed image when the photographer takes an image while holding the imaging device sideways. Reference numeral 502 denotes an intersection when the screen of the photographed image 501 is divided into six, and reference numeral 503 denotes a point near the center of the screen. The point near the center of the screen is arranged slightly above the center of the screen.

  Reference numeral 505 in FIG. 5B denotes a photographed image when the photographer holds the image pickup apparatus vertically and picks up an image. Reference numeral 506 denotes an intersection when the screen of the captured image 505 is divided into six, and 507 is a point near the center of the screen. A point near the center of the screen is placed above the center of the screen.

  In step S301, the closer the face position detected by the face detection unit 114 is to these points, the higher the CPU 108 increases the evaluation value of the face position (first face position).

  Next, in step S302, the CPU 108 evaluates whether the face protrudes from the image. Specifically, the CPU 108 determines whether or not the face protrudes from the size and coordinates of the face detected by the face detection unit 114 and the coordinate information of the image. When it is determined that the face protrudes from the image, the CPU 108 lowers the evaluation value of the face position (second face position).

  Next, in step S303, the CPU 108 evaluates a smile. Specifically, when it is determined that the subject is smiling based on the smile level detected by the smile detection unit 115, the evaluation value of the smile is high. There is a method of determining whether a smile is made by setting a predetermined threshold and detecting a smile level with a numerical value higher than the threshold.

  Next, in step S304, the CPU 108 evaluates the line of sight. When the eye area detection unit 116 and the line-of-sight detection unit 118 determine that the line of sight is facing the imaging apparatus, the CPU 108 increases the evaluation value of the line of sight.

  Next, in step S305, the CPU 108 evaluates the luminance. The evaluation of luminance is performed based on whether the luminance near the face is more appropriate. If it is determined that the luminance near the face is significantly different from the appropriate value, the CPU 108 decreases the luminance evaluation value.

  Next, in step S306, the CPU 108 evaluates the focus state. The CPU 108 detects the focus state of the image from the signal component of the characteristic frequency band extracted by the image processing unit 104, and when it is determined that the focus is not more than a predetermined due to blur or the like, the CPU 108 lowers the evaluation value of the focus state. .

  Then, the CPU 108 calculates a face evaluation value based on each evaluation value obtained in steps S301 to S306 described above. For example, all the evaluation values may be added together, or each evaluation value may be weighted and calculated. Alternatively, the values other than the focus evaluation value may be added together, and the added value may be multiplied by the focus evaluation value.

  According to the first embodiment, a face evaluation value based on a predetermined condition is calculated from a plurality of obtained images, and a specific subject is in a state where his eyes are closed or his eyes are open. Determine if there is. If it is determined that the specific subject is in the state of meditating, the eyes are switched to the priority state. If it is determined that the specific subject is in the state of opening the eyes, priority is given to opening the eyes. Switch to state. Furthermore, when it is determined that the specific subject is in the state of meditating when the eyes are in the priority state, and when the opening priority is in the state, the specific subject is in the state where the eyes are open. If it is determined, the image evaluation value is calculated from the face evaluation value. Then, an image to be recorded is selected according to the calculated image evaluation value. Thereby, it is possible to efficiently record an image of a face with a closed eye such as a crying face or sleeping face of a specific subject intended by the photographer, and also efficiently record an image with the subject's eyes open.

  In the above-described embodiment, the case where only one scheduled recording image is recorded has been described. However, a plurality of images may be recorded. For example, only an image to which an image evaluation value exceeding a preset reference value is assigned may be recorded. Alternatively, the number of sheets to be recorded may be set in advance, and the image may be recorded when the number is reached.

[Second Embodiment]
The imaging apparatus according to the second embodiment of the present invention has the same configuration as the imaging apparatus according to the first embodiment described with reference to FIG. 1, and the same components are denoted by the same reference numerals. The description will be omitted.

  Image selection processing in the imaging apparatus according to the second embodiment of the present invention will be described with reference to FIGS. 6A, 6B, and 6C.

  6A to 6C are flowcharts showing a flow of image selection processing executed by the imaging apparatus according to the second embodiment of the present invention. The same steps as those described with reference to FIGS. 3A to 3C are denoted by the same step numbers, and detailed description thereof is omitted.

  In FIG. 6A, when the CPU 108 determines that the opening priority state is set as a result of the determination in step S205, the process proceeds to step S401, and the opening face evaluation value in the opening priority state is calculated. Here, the calculation is performed so as to increase the face evaluation value obtained in step S202. For example, a predetermined numerical value is added to the face evaluation value, or a predetermined coefficient larger than 1 is multiplied.

  Then, the process proceeds to step S206, and after the CPU 108 clears the eyes closed counter, the process proceeds to step S405 in FIG. 6B.

  In FIG. 6A, after step S212, the process proceeds to step S405 in FIG. 6B without executing step S213 in FIG. 3A.

  In FIG. 6C, when the CPU 108 determines that the eye-meditation priority state is determined as a result of the determination in step S214, the process proceeds to step S402, and the eye-meditation face evaluation value in the eye-meditation priority state is calculated. Here, the calculation is performed so as to increase the face evaluation value obtained in step S202. For example, a predetermined numerical value is added to the face evaluation value, or a predetermined coefficient larger than 1 is multiplied.

  In step S215, the CPU 108 clears the opening counter, and then proceeds to step S405 in FIG. 6B.

  In FIG. 6C, after step S221, the process proceeds to step S405 in FIG. 6B without executing step S222 in FIG. 3C.

  In step S405 of FIG. 6B, the CPU 108 calculates an image evaluation value. Here, when it is determined that the eyes are closed in the eye-meditation priority state or when it is determined that the eyes are opened in the eye-open priority state, the face evaluation value obtained in step S401 or step S402 is finally given to the image. An image evaluation value is calculated. On the contrary, when it is determined that the eyes are open in the priority state of eye opening or when it is determined that the eyes are closed in the priority state of opening the eyes, the eye opening face evaluation value or the eye closing face evaluation value calculated in step S211 or step S220 is used. Then, an image evaluation value finally given to the image is calculated.

  Next, in step S224, the CPU 108 determines whether the captured image to be processed is the first image. In the present embodiment, image data is recorded in the image recording unit 107 at the time when the priority of eye opening and eye closing is switched. The first sheet after this recording is also determined here. If it is the first image after recording, the CPU 108 determines that it is the first image, and updates the scheduled recording image to the current image in step S227. As a result, the image stored in the DRAM 106 is replaced with the current image.

  In step S403 of FIG. 6B, the CPU 108 determines whether or not the priority state of eye opening and eye closing has been switched. When the priority state is switched, the CPU 108 records the image recorded in the DRAM 106 as the recording scheduled image in the image recording unit 107 in the evaluation so far (step S404), returns to step S201 in FIG. Process.

  As described above, by performing the image recording operation when the priority state is switched, it is possible to record a desired image of the photographed document in the opening priority state and the eye closing priority state.

  According to the second embodiment, the opening face evaluation value and the eye closing face evaluation value are calculated when the priority of eye opening and eye closing is switched, so that the processing speed and processing efficiency can be improved. It becomes possible.

  In the above-described embodiment, the case where the present invention is applied to the imaging apparatus has been described. However, the present invention is not limited to this and may be an image processing apparatus. Note that when applied to an image processing apparatus, since there is no imaging function and no shooting end instruction or the number of shots is counted, for example, steps S228 to S230 in FIGS. 3B and 6B may be deleted. Good.

Moreover, although the case where it applied to continuous shooting was demonstrated in the said embodiment, it is not limited to this, It can apply also when giving an image evaluation value with respect to each frame image at the time of video shooting. . In other words, it is possible to extract a frame image having a high image evaluation value from the moving image, or to use it as a guideline for a position where continuous frame images are separated using the image evaluation value.
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 101 Imaging lens 102 Image sensor 104 Image processing part 108 System control part (CPU)
114 face detection unit 115 smile detection unit 116 eye region detection unit 117 eye meditation detection unit 118 gaze detection unit

Claims (11)

Imaging means capable of continuously imaging a subject;
First evaluation value calculation means for calculating a first evaluation value based on a predetermined condition from an image obtained from the imaging means;
A judging means for judging whether a specific subject is in a state where his eyes are meditated or in a state where his eyes are open from the image obtained from the imaging means;
If it is determined from the plurality of images obtained from the imaging means that the specific subject is in a state of meditating eyes, priority is given to the eye-meditation image in which the specific subject is meditating eyes When the determination means determines that the specific subject is in an open state, the specific subject switches to an opening priority state that prioritizes the opening image in which the specific subject has open eyes. Switching means;
Second evaluation value calculating means for calculating a second evaluation value to be applied to the image from the first evaluation value;
Selecting means for selecting an image to be recorded in accordance with the calculated second evaluation value;
The second evaluation value calculation means is in a state where the eyes are open when the determination means determines that the specific subject is meditating in the eye-meditation priority state. The second evaluation value calculated from the first evaluation value is set higher than the case where it is determined that the specific subject is in the eye opening priority state by the determination means. When it is determined that the eye is open, the second evaluation value calculated from the first evaluation value is set higher than when it is determined that the eye is closed. An imaging apparatus characterized by the above. When the determination means determines that the specific subject is in an open state in the eye-meditation priority state, a third evaluation value is calculated from the first evaluation value. 3 evaluation value calculating means;
When the determination means determines that the specific subject is in a state where the eyes are meditated in the eye opening priority state, a fourth evaluation value is calculated from the first evaluation value. 4 evaluation value calculation means,
The imaging apparatus according to claim 1, wherein the second evaluation value calculation unit calculates the second evaluation value from the third evaluation value or the fourth evaluation value.   The first evaluation value calculation means calculates the first evaluation value from the position of the face of the specific subject, the feature amount of the specific subject, the focus state of the image, and the state of the image including luminance. The imaging device according to claim 1, wherein:   When the imaging unit continuously captures a plurality of images, the selection unit selects an image having the highest value among the second evaluation values assigned to each of the plurality of images. Item 4. The imaging device according to any one of Items 1 to 3. The third evaluation value calculating means calculates the third evaluation value so as to lower the first evaluation value;
The imaging apparatus according to claim 2, wherein the fourth evaluation value calculation unit calculates the fourth evaluation value so as to lower the first evaluation value. The determination means determines which priority state of the eye-meditation priority state or the opening priority state has continued continuously,
In the case where only one image is selected from a plurality of images continuously captured by the imaging unit, the selection unit selects the second image from the priority state image determined to be continuously continued by the determination unit. The image pickup apparatus according to claim 1, wherein an image having a high evaluation value is selected.   When the imaging unit continuously captures a plurality of images, the selection unit has a high second evaluation value in each priority state if both the eye-meditation priority state and the opening priority state exist. The image pickup apparatus according to claim 1, wherein an image is selected.   The imaging apparatus according to claim 1, wherein the switching unit sets the opening priority state when the priority state is not fixed. The second evaluation value calculating means calculates the second evaluation value so as to increase the first evaluation value;
The imaging apparatus according to claim 1, wherein a recording unit that records an image selected by the selection unit records an image when a priority state is switched by the switching unit. In an image processing method of an imaging device including an imaging unit capable of continuously imaging a subject,
A first evaluation value calculating step of calculating a first evaluation value based on a predetermined condition from an image obtained from the imaging means;
A determination step of determining whether a specific subject is in a state where his eyes are meditated or in a state where his eyes are open from the image obtained from the imaging means;
If it is determined from the plurality of images obtained from the imaging means that the specific subject is in a state of meditating eyes, priority is given to the eye-meditation image in which the specific subject is meditating eyes When it is determined in the determination step that the specific subject is in an open state, the specific subject is set to an opening priority state in which priority is given to an open image in which the eye is open. A switching process for switching;
A second evaluation value calculating step of calculating a second evaluation value to be applied to the image from the first evaluation value;
A selection step of selecting an image to be recorded according to the calculated second evaluation value,
In the second evaluation value calculation step, if the specific subject is determined to be in a state where the eyes are meditating in the determination step, the eyes are opened. The second evaluation value calculated from the first evaluation value is set higher than when it is determined to be in the state, and the specific subject is determined in the determination step in the opening priority state. When it is determined that the eyes are open, the second evaluation value calculated from the first evaluation values is higher than when it is determined that the eyes are closed. An image processing method. In a program for causing an imaging apparatus to execute an image processing method of an imaging apparatus including an imaging unit capable of continuously imaging a subject,
A first evaluation value calculating step of calculating a first evaluation value based on a predetermined condition from an image obtained from the imaging means;
A determination step of determining whether a specific subject is in a state where his eyes are meditated or in a state where his eyes are open from the image obtained from the imaging means;
If it is determined from the plurality of images obtained from the imaging means that the specific subject is in a state of meditating eyes, priority is given to the eye-meditation image in which the specific subject is meditating eyes When it is determined in the determining step that the specific subject is in an open state, the specific subject is set to an opening priority state that prioritizes the opening image in which the specific subject has an open eye. A switching step to switch,
A second evaluation value calculating step of calculating a second evaluation value to be given to the image from the first evaluation value;
A selection step of selecting an image to be recorded according to the calculated second evaluation value;
In the second evaluation value calculating step, the eyes are opened when it is determined in the determination step that the specific subject is meditating in the eyes in the eye-meditation priority state. The second evaluation value calculated from the first evaluation value is set higher than when it is determined to be in the state, and the specific subject is determined in the determination step in the opening priority state. When it is determined that the eyes are open, the second evaluation value calculated from the first evaluation values is higher than when it is determined that the eyes are closed. The program characterized by doing.

Images