JP5412894B2 - Electronic camera and program - Google Patents

Description

  The present invention relates to an electronic camera having a face detection function and a program.

  2. Description of the Related Art Conventionally, there is known an electronic camera that detects and warns when a subject's eyelid is closed or a state where the eyelid is half-closed (see, for example, Patent Document 1). In the electronic camera of Patent Document 1, it is determined whether the eyelid is in a closed state or a state in which the eyelid is half closed based on whether or not the eye area pattern in the captured image matches a predetermined eye pattern. To do.

JP 2006-344166 A

  However, in the electronic camera disclosed in Patent Document 1, there is a possibility that a person with originally thin eyes may be sensitively warned that the eyelid is half closed due to individual differences in subjects.

  Therefore, in view of the above circumstances, the present invention provides an electronic camera and a program for preventing an eyelid from being erroneously determined to be in a half-closed state even though the eyelid is not in a half-closed state due to individual differences in subjects. The purpose is to provide.

An electronic camera according to a first aspect includes an imaging unit, a face detection unit, a face recognition unit, an open / close state detection unit, a classification unit, a threshold setting unit, a determination unit, and a notification unit. The imaging unit captures a subject image and generates an image. The face detection unit detects a face area from the image. The face recognition unit extracts a feature amount for identifying a face from the face region. The open / close state detection unit detects the open / close state of the eyelid in the face region. The classification unit detects at least a first state image in which wrinkles are most open and a second state image in which wrinkles are half closed from the first state for a predetermined number of images that are temporally continuous, and based on the feature amount And classify by person and record in memory. The threshold setting unit sets a threshold indicating the second state for each person based on the first state image and the second state image . The determination unit checks an image that is a determination target of the open / close state of the bag with at least a first state image of images recorded in the memory for the same person, and based on the threshold value, It is determined whether or not the second state is open. The notification unit warns based on the determination result of the determination unit.

A program according to a second invention is a program for causing a computer to execute processing of a captured image, and includes face detection processing, face recognition processing, open / closed state detection processing, classification processing, determination processing, and notification processing. And have. In the face detection process, a face region is detected from an image. In the face recognition process, a feature amount for identifying a face is extracted from a face region. The open / close state detection process detects the open / close state of the eyelid in the face region. The classification process detects at least a first state image in which wrinkles are most open and a second state image in which wrinkles are half closed from the first state for a predetermined number of temporally continuous images, and based on the feature amount And classify by person and record in memory. In the threshold value setting process, a threshold value indicating the second state is set for each person based on the first state image and the second state image . In the determination process, the image to be determined for the open / close state of the bag is checked against at least a first state image among the images recorded in the memory for the same person , and the wrinkle in the determination target image is determined based on the threshold value. It is determined whether or not the second state is open. The notification process warns based on the determination result of the determination process.

  According to the electronic camera of the present invention, it is possible to prevent the eyelid from being erroneously determined to be in a half-closed state even though the eyelid is not half-closed due to individual differences in subjects.

The block diagram which shows the structure of the electronic camera 1 The flowchart which shows an example of the process which registers the image data for setting a threshold value The figure which illustrates notionally an example of registration of a new person (person A) The figure which illustrates notionally an example of registration of a new person (person B) The flowchart which shows an example of operation | movement of the electronic camera 1 in the open / closed state detection mode of a bag The figure which shows an example of the image imaged still image

(First embodiment)
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The “open / closed state detection mode” described in the first embodiment is a warning message after shooting when the eyelid of the person who is the subject is in a half-closed state (in other words, in a state where the eyelid is half open). This is a mode for informing the user (details will be described later).

  FIG. 1 is a block diagram showing the configuration of the electronic camera 1. As shown in FIG. 1, the electronic camera 1 includes a photographing lens 10, an imaging unit 11, an image processing unit 12, a RAM (Random Access Memory) 13, a ROM (Read Only Memory) 14, a display monitor 15, and a recording. An interface (recording I / F) 16, a recording medium 17, a release button 18, an operation unit 19, and a CPU (Central Processing Unit) 20 are provided.

  Among these, the image processing unit 12, the RAM 13, the ROM 14, the display monitor 15, the recording interface (recording I / F) 16, and the CPU 20 are connected to each other via a bus 21. The release button 19 and the operation unit 18 are connected to the CPU 20.

  The photographing lens 10 includes a plurality of lens groups including a zoom lens and a focus lens. For the sake of simplicity, FIG. 1 shows the photographic lens 10 as a single lens.

  The imaging unit 11 includes a lens driving mechanism, a diaphragm, a shutter mechanism, an imaging element, an AFE (analog front end), and the like. The imaging unit 11 generates image data of a subject image formed by the photographing lens 10. The imaging unit 11 generates image data of a still image for recording when a still image (main image) is captured. Further, the imaging unit 11 generates image data of a plurality of frames at a predetermined frame rate at the time of shooting for a registered person, which will be described later.

  Image data output by the imaging unit 11 is temporarily recorded in the RAM 13. This RAM 13 also functions as a frame memory. The ROM 14 is a non-volatile memory that stores a program for controlling the electronic camera 1 in advance.

  The image processing unit 12 reads the image data recorded in the RAM 13 and performs various image processing (gradation conversion processing, contour enhancement processing, white balance processing, etc.).

  The display monitor 15 displays a still image, a through image for composition confirmation, an operation menu of the electronic camera 1, and the like. As the display monitor 15, a liquid crystal monitor or the like can be appropriately selected and used.

  A connector for connecting the recording medium 17 is formed on the recording interface (recording I / F) 16. The recording interface (recording I / F) 16 accesses the recording medium 17 connected to the connector according to an instruction from the CPU 20 and performs image recording processing.

  The operation unit 18 includes, for example, a command dial for command selection, a power button, and the like. Then, the operation unit 18 receives an instruction input for operating the electronic camera 1.

  The release button 19 receives an instruction input for a half-press operation (an instruction input for starting an operation such as autofocus (AF) or automatic exposure (AE) before photographing) and a full-press operation (starting an imaging operation).

  The CPU 20 is a microprocessor that performs overall control of the electronic camera 1 and has a function as a computer. The CPU 20 also functions as a face detection unit 20a, a face recognition unit 20b, an open / close state detection unit 20c, a classification unit 20d, a threshold setting unit 20e, a determination unit 20f, and a notification unit 20g by executing the program of the present embodiment. To do.

  The face detection unit 20a detects a face area from the image. That is, the face detection unit 20a extracts a face area from the image data, and detects the position coordinates and size (range) of the face area. The face detection process can be realized by using a known technique.

  The face recognition unit 20b extracts a feature quantity for identifying a face from the face area. That is, the face recognition unit 20b performs a feature amount extraction process on the face area detected by the face detection unit 20a to extract a face feature amount. Specifically, the face recognizing unit 20b extracts shapes that express the features of a person's facial expression, such as the shape of the face outline, the shape of the eyes, and the shape of the mouth, and uses the information indicating these shapes as the feature amount. The face recognition unit 20b can identify a face for each person based on the feature amount. This feature amount extraction process can be realized by using a known technique.

  The open / close state detection unit 20c detects the open / close state of the eyelid and the degree of open / close from the face region. This open / closed state detection process can be realized by using a known technique.

  The classification unit 20d classifies the images in a state where the eyelids are switched from the opened state to the closing operation (hereinafter referred to as “first state”) for each person based on the feature amount, and stores the image data in the storage area of the RAM 13 as image data. (Details will be described later).

  The threshold setting unit 20e sets, for each person, a threshold indicating a state in which the eyelid is half closed from the first state (hereinafter referred to as “second state”). Specifically, the threshold setting unit 20e evaluates the degree of opening of the heel up to the first state by using a numerical value to evaluate the state of the heel completely closed. For example, for each person, the threshold setting unit 20e sets the value indicating the state where the heel is completely closed to zero, the value indicating the second state to 0.5, and the value indicating the first state to 1.0. Turn into. Thereby, the threshold setting unit 20e can set a value indicating the second state as a threshold for each person. Note that the state from the value indicating the state where the heel is completely closed to the value indicating the second state is equal to or less than the threshold value.

  The determination unit 20f collates the image that is the determination target of the open / close state of the bag with the image of the same person in the first state, and based on the threshold value, whether or not the bag in the determination target image is open more than the second state. Determine whether.

  Based on the determination result of the determination unit 20f, the notification unit 20g gives a warning via the display monitor 15 when the wrinkle in the determination target image is not opened in the second state.

  Next, an example of the operation of the electronic camera 1 in the first embodiment will be described.

  In the first embodiment, first, a process of registering image data for setting a threshold value in the open / closed state detection mode of the bag will be described. Next, an example of the operation of the electronic camera 1 in the bag open / closed state detection mode will be described. Note that the eyelid open / closed state detection mode of the present embodiment will be described on the assumption that a human face exists as a subject.

  FIG. 2 is a flowchart illustrating an example of processing for registering image data for setting a threshold for each face of a person. Here, after the electronic camera 1 is turned on, if the bag open / closed state detection mode is selected by a command dial or the like of the operation unit 18, the CPU 20 displays the menu screen of the bag open / closed state detection mode on the display monitor 15. Is displayed. In this state, when registration of a new person in the open / closed state detection mode of the bag is selected by the command dial of the operation unit 18, the CPU 20 starts the flowchart shown in FIG. 2.

  Step S101: The CPU 20 instructs the imaging unit 11 to capture a through image at a frame rate of 60 fps (frame / second). The through images are sequentially displayed on the display monitor 15 after being subjected to predetermined image processing. Here, it is known that blinking performed by a human (the time from when the eyelid is opened to the opening operation and again when the eyelid is closed) is generally about 100 to 150 msec. In the case of a through image of 60 fps (frame / second), blinking performed by a human corresponds to about 6 to 9 frames. Therefore, the open / closed state detection unit 20c, which will be described later, can detect a frame in the first state, a frame in the second state, a frame in a state where the eyelid is closed, and the like by analyzing 60 frames of image data.

  Step S102: The face detection unit 20a of the CPU 20 detects whether or not a human face exists in one frame image. Note that the face detection unit 20a may perform face detection processing for all frames, but here, for example, if the frame rate is 60 fps (frames / second) in consideration of the computation load of the CPU 20, The face detection unit 20a performs face detection processing on the image data of frames extracted every frame.

  Step S103: When the CPU 20 receives an instruction input for a full-press operation via the release button 19 (Step S103: Yes), the CPU 20 proceeds to Step S104. On the other hand, if the instruction input of the full press operation is not accepted (step S104: No), the process returns to step S101.

  Step S104: The CPU 20 temporarily records 60 frames of image data (60 frames) in the RAM 13 in order to capture a 60 fps (frame / second) through image (60 frames) output from the imaging unit 11 as a captured image. To do. The image processing unit 12 performs various types of image processing (gradation conversion processing, contour enhancement processing, white balance processing, etc.) on these image data.

  Step S105: The face recognition unit 20b of the CPU 20 performs a feature amount extraction process on the face area detected by the face detection unit 20a based on the image data recorded in the RAM 13. Here, when the face recognition unit 20b cannot extract the feature amount (step S105: No), the CPU 20 causes the display monitor 15 to display a message indicating that the feature amount cannot be extracted. (Step S109), the process returns to Step S101. On the other hand, when the face recognition unit 20b can extract the feature amount (step S105: Yes), the process proceeds to step S106.

  Step S106: The CPU 20 causes the display monitor 15 to display a menu screen as to whether or not to register an image of a person obtained by photographing this time. Here, when the CPU 20 receives an instruction input indicating that registration is not to be performed via the operation unit 18 (step S106: No), the CPU 20 displays a message indicating that photographing is performed again on the display monitor 15 ( Step S109) and return to Step S101. On the other hand, when receiving an instruction input indicating registration via the operation unit 18 (step S106: Yes), the CPU 20 proceeds to step S107.

  Step S107: The open / close state detection unit 20c of the CPU 20 analyzes the face area included in the image recorded in the RAM 13, and detects the open / close state of the eyelid and the degree of open / close of the eyelid.

  Specifically, the open / close state detection unit 20c sequentially analyzes the eyelid region including the eye shape extracted by the face recognition unit 20b from 1 frame to 60 frames. Here, in the first state, the pupil region of the eye appears most. Further, when the eyelid is closed, the pupil is hidden. For this reason, the open / close state detection unit 20c analyzes changes in the pupil region from the first frame to the 60th frame based on the luminance distribution of the image data, so that the open / close state and the The degree of opening / closing can be detected.

  FIG. 3 is a diagram conceptually illustrating an example of registration of a new person (person A). Here, the open / closed state detection unit 20c has, for example, a frame in the first state (FIG. 3 (a)) and a state between the first state and the second state (FIG. 3 (b)) out of 60 frames. The frame in the second state (FIG. 3C), the frame in the closed state (FIG. 3D), and the like can be detected.

  Subsequently, the threshold setting unit 20e of the CPU 20 sets a threshold indicating the second state (FIG. 3C) based on the first state (FIG. 3A). Next, the classification unit 20d of the CPU 20 identifies identification information (as an example) based on the feature amount extracted by the face recognition unit 20b with respect to the frame in the first state (FIG. 3A) recorded in the RAM 13. , Person A). Thereby, identification information is provided for each person. That is, the threshold setting unit 20e can associate a threshold for each person. Taking FIG. 3 as an example, the image of the frame in the first state (FIG. 3A) becomes a registered image for identifying the person A. Furthermore, the threshold setting unit 20e can set a threshold indicating the second state (FIG. 3B) based on the frame in the first state. This identification information may be given by user input via the operation unit 18.

  Step S108: The classification unit 20d records the image data (for example, FIGS. 3A to 3D) temporarily recorded in the RAM 13 in the storage area of the RAM 13. Then, this processing routine ends.

  In the above-described flowchart, the registration of the person A has been described based on the image data. Next, registration of another person B will be described.

  FIG. 4 is a diagram conceptually illustrating an example of registration of a new person (person B). Also for the person B, the classification unit 20d records the image data (for example, FIGS. 4A to 4D) temporarily recorded in the RAM 13 in the storage area of the RAM 13 in accordance with the procedure of the above-described flowchart ( Step S108). In this way, the CPU 20 can set a threshold for each person.

  In the above-described flowchart, the case where the threshold setting unit 20e sets a threshold by acquiring a through image has been described. For example, the threshold value setting unit 20e may set the threshold value by taking one frame of a still image in which the wrinkles are fully open or close to full open. In this case, in the electronic camera 1, for example, the threshold setting unit 20e estimates the size of the pupil with the eyelids half-opened from the size of the pupil when the eyelids are fully open or close to full open, so that the threshold setting unit 20e Set the threshold value shown.

  Next, an example of the operation of the electronic camera 1 in the bag open / closed state detection mode will be described. Here, when the eyelid open / closed state detection mode is selected by the command dial of the operation unit 18, the CPU 20 causes the display monitor 15 to display a menu screen of the eyelid open / closed state detection mode. In this state, when shooting in the open / closed state detection mode of the bag is selected with the command dial or the like of the operation unit 18, the CPU 20 starts the flowchart shown in FIG.

  FIG. 5 is a flowchart showing an example of the operation of the electronic camera 1 in the open / closed state detection mode of the bag.

  Step S201: The CPU 20 instructs the imaging unit 11 to output a through image at a frame rate of 60 fps (frame / second), for example.

  Step S202: The face detection unit 20a of the CPU 20 detects whether or not a human face exists in one frame image.

  Step S203: When the CPU 20 receives an instruction input for a full-press operation via the release button 19, the process proceeds to step S204. On the other hand, if the instruction input for the full-press operation is not accepted (step S203: No), the process returns to step S201.

  Step S204: The CPU 20 issues an instruction for shooting preparation processing such as AE and AF, and sets shooting conditions. Subsequently, the imaging unit 11 is instructed to shoot a still image. That is, the CPU 20 sends a drive signal to each of the image sensor and the A / D converter via the timing generator in the image capturing unit 11 to perform still image shooting. Image data output from the imaging unit 11 is temporarily recorded in the RAM 13. The image processing unit 12 performs various types of image processing (gradation conversion processing, contour enhancement processing, white balance processing, etc.) on the image data obtained by taking a still image. For convenience of explanation, the person A shown in FIG. 3 is taken.

  FIG. 6 is a diagram illustrating an example of a still image captured image. FIG. 6A shows a state in which a still image of the person A is displayed on the display monitor 15.

  Step S205: The face recognizing unit 20b performs face recognition processing based on the image data obtained by taking a still image, and a shape that expresses the facial expression features of the person A, such as the shape of the face contour, the shape of the eyes, the shape of the mouth, etc. And information indicating these shapes is used as a feature amount. When the face recognition process ends, the process proceeds to step S206.

  Step S206: The CPU 20 searches the RAM 13 for the first state image data of the same person (person A) based on the feature amount of the face taken with the still image. If there is image data of the same person in the first state (step S206: Yes), the process proceeds to step S207. On the other hand, when there is no image data of the same person in the first state (step S206: No), the process proceeds to step S211 described later.

  Step S207: The open / closed state detection unit 20c of the CPU 20 performs detection processing of the open / close state of the eyelids on the image data acquired by still image shooting recorded in the RAM 13, and detects the open / closed state of the eyelids. Subsequently, the determination unit 20f of the CPU 20 collates the still image captured image with the image of the same person in the first state, and based on the threshold value, whether or not the wrinkles in the still image are opened more than in the second state. Determine. Specifically, the determination unit 20f compares the opening / closing state of the person A's eyelid in the image taken with the still image with the first state shown in FIG. In this case, the determination unit 20f normalizes the size of the face of the person A in the still image captured image so as to match the size of the pattern matching template image. In addition, in order to raise the determination precision of the determination part 20f more, you may perform a determination process using the image of Fig.3 (a) to FIG.3 (d).

  Step S208: If the determination result of the determination unit 20f is that the wrinkle in the image taken with the still image is opened more than in the second state (step S208: Yes), the process proceeds to step S210. On the other hand, when the determination result of the determination unit 20f indicates that the wrinkles in the image taken with the still image are not opened more than in the second state (step S208: No), the process proceeds to step S209.

  Step S209: The notification unit 20g of the CPU 20 displays a message on the display monitor 15 indicating that the bag is half closed, and warns the user. Thereby, the user can determine whether or not to record a still image.

  Step S210: When the CPU 20 receives an instruction input for saving a still image photographed image via the operation unit 18, the CPU 20 compresses the image data and records it on the recording medium 17.

  In step S206, when there is no image data of the same person in the first state (step S206: No), the process proceeds to step S211 and the determination unit 20f opens and closes the bag with a predetermined (standard) threshold value set in advance. Determine the state. Then, after the determination unit 20f determines the open / closed state of the bag, the process proceeds to step S208. The subsequent processing is as described above. Thus, this processing routine is finished.

  In the example described above, one subject (person A) has been described. However, the determination unit 20f individually registers the first state for a plurality of subjects, so that the determination unit 20f can individually open and close the bag. Determine.

  FIG. 6B shows still images obtained by photographing the person A and the person B. In this case, for the person A, the determination unit 20f determines the open / close state of the heel of the person A illustrated in FIG. 6B based on the first state illustrated in FIG. On the other hand, for the person B, based on the first state shown in FIG. 4, the open / close state of the heel of the person B shown in FIG. 6B is determined. In this way, the determination unit 20f can accurately determine a state in which the eyelids are half closed. That is, since the person B is originally a person with thin eyes, the electronic camera of Patent Document 1 may erroneously determine that the eyelid is half closed. However, since the determination unit 20f according to the present embodiment individually determines the open / closed state of the heel for the registered person, it is not necessary to erroneously determine the person B as a state where the heel is half closed.

  As described above, the electronic camera 1 according to the present embodiment can prevent erroneous determination that the eyelid is half-closed due to individual differences in subjects, even though the eyelid is not half-closed.

  Note that, as an example of transition to step S211 in the flowchart illustrated in FIG. 5, a case where a plurality of persons are photographed and an unregistered person exists is assumed. For example, if a registered person A or person B and an unregistered person C (not shown) are photographed as subjects, the process of step S211 is performed for the unregistered person C. In this case, priority may be given to the determination for the person A and the person B, and the unregistered person C may not be warned even if the bag is half closed.

Further, in the present embodiment, the first state is a state where the operation is switched from the state where the heel is opened to the closing operation, and the second state is a state where the heel is half closed from the first state. In this case, taking person A as an example, as shown in FIG. 3, the state from the state where the heel is completely closed (FIG. 3 (d)) to the state where the heel is half closed (FIG. 3 (c)). Since it becomes below the threshold value which shows a 2nd state, you may warn about these states.
<Supplementary items of the embodiment>
(1) Although the program of this embodiment was applied to the electronic camera 1, it is not restricted to the electronic camera 1, For example, it can install in a personal computer etc. provided with an image reproduction function, and can perform the same process. In this case, the CPU of the personal computer reads image data for setting a threshold value and image data of a still image (main image) taken by the electronic camera 1. In the flowchart (FIG. 2) showing an example of the process of registering image data for setting a threshold, the CPU of the personal computer can set the threshold for each person by performing the processes after step S105. it can.

  Further, in the flowchart (FIG. 5) showing an example of the operation of the electronic camera 1 in the open / closed state detection mode of the bag, the CPU of the personal computer performs the processes in and after step S205, thereby taking a still image taken by the electronic camera 1. It is possible to determine the open / closed state of the bag for the person existing in

  (2) The notification unit 20g of the present embodiment gives a warning or the like to the user by displaying a message on the display monitor 15. For example, by providing a speaker in the electronic camera 1, the notification unit 20g may give a warning or the like by voice. Further, by providing a light emitting diode on the back surface of the electronic camera 1, the notification unit 20g may turn on the light emitting diode to give a warning or the like.

DESCRIPTION OF SYMBOLS 11 ... Imaging part, 20a ... Face detection part, 20b ... Face recognition part, 20c ... Opening / closing state detection part, 20d ... Classification part, 20e ... Threshold setting part, 20f ...・ Determining part, 20g ... notification part

Claims (2)

An imaging unit that captures a subject image and generates an image;
A face detection unit for detecting a face region from the image;
A face recognition unit that extracts a feature amount for identifying the face from the face region;
An open / closed state detector for detecting an open / closed state of the eyelid in the face region;
For a predetermined number of images continuous in time, at least a first state image in which the folds are most open and a second state image in which the folds are half closed from the first state are detected, and the feature amount is detected. A classification unit that classifies each person based on the result and records it in a memory;
A threshold setting unit that sets a threshold value indicating the second state for each person based on the image in the first state and the image in the second state;
The image that is the determination target of the open / close state of the eyelid is collated with at least the image in the first state among the images recorded in the memory for the same person, and the image in the image to be determined is determined based on the threshold value. A determination unit that determines whether or not is open more than the second state;
Based on the determination result of the determination unit, a notification unit for warning,
An electronic camera comprising: A program that causes a computer to process a captured image,
A face detection process for detecting a face region from the image;
A face recognition process for extracting a feature amount for identifying the face from the face region;
An open / closed state detection process for detecting an open / closed state of the eyelid in the face region;
For a predetermined number of images continuous in time, at least a first state image in which the folds are most open and a second state image in which the folds are half closed from the first state are detected, and the feature amount is detected. A classification process for classifying each person based on the result and recording it in a memory;
A threshold setting process for setting a threshold value indicating the second state for each person based on the first state image and the second state image ;
The image that is the determination target of the open / close state of the eyelid is collated with at least the image in the first state among the images recorded in the memory for the same person, and the image in the image to be determined is determined based on the threshold value. Determination processing for determining whether or not is open more than the second state;
A program that causes a computer to execute warning processing based on a determination result of the determination processing.

Images