JP2012252383A - Imaging apparatus, moving picture reproducing apparatus and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To effectively correct an opening/closing proportion of the eyes of a subject person when the opening/closing proportion of the eyes of the subject person in taken image data is inappropriate.SOLUTION: An imaging apparatus includes: a face detection part for detecting the face of a subject person in input image data; an eye opening/closing proportion determination part for determining an opening/closing proportion of the eyes of the subject person on the basis of a face image including the face detected by the face detection part in the image data; a storage part for storing eye opening/closing proportion information including the opening/closing proportion of the eyes of the subject person determined by the eye opening/closing proportion determination part; a comparison part for comparing the opening/closing proportion of the eyes of the subject person determined by the eye opening/closing proportion determination part with an eye opening/closing proportion obtained by multiplying the eye opening/closing proportion in the stored eye opening/closing proportion information by a specific correction allowable ratio; and a correction part for correcting the eye opening/closing proportion determined by the eye opening/closing proportion determination part according to a comparison result given by the comparison part.

Description

  The present invention relates to an imaging apparatus, a moving image reproduction apparatus, and an imaging method for imaging a subject such as a person.

  When taking a picture of one or more subject persons, the subject person may blink at the moment of imaging or may be in a half-eye state. In a photograph of a person, the closed state or half-eye state is not preferable in terms of quality, and the subject person has a strong tendency to not like the photograph. If it is possible to retake a picture, it is sufficient to take it again. However, if it is difficult to take a picture again, it is necessary to correct the photograph after taking a picture, and a complicated operation is required.

  For example, Patent Document 1 is known as an image processing apparatus that corrects a shooting failure by performing image processing for opening eyes on a person image with closed eyes.

  The image processing apparatus of Patent Document 1 corrects an image of a person whose input image data is in an eye-open state to an eye image in an open state. Specifically, the image processing apparatus prepares a plurality of open eye image samples, and uses the selected sample image of the prepared plurality of samples to generate an eye-open image. On the other hand, the images of the eyes of the same person are combined. According to such an image processing apparatus, it is possible to perform a process of opening eyes on a person image with closed eyes and output an image in which shooting failure is corrected.

JP 2001-197296 A

  However, the image processing apparatus of Patent Document 1 requires a memory for holding a plurality of open eye image samples in order to correct the eyes that are pinched and open eyes. The manufacturing cost of the image processing apparatus is increased.

  In addition, the image processing apparatus disclosed in Patent Document 1 synthesizes an eye image of the same person with an eye-open image. Combining an image with eyes closed with an image with open eyes may result in an unnatural image. For this reason, the subject that it cannot necessarily output the image according to the subject person's own preference exists.

  The present invention has been made in view of the above-described conventional circumstances, and effectively corrects the open / close ratio of the subject person's eyes when the open / close ratio of the subject person's eyes in the captured image data is inappropriate. It is an object to provide an imaging device, a moving image playback device, and an imaging method.

  The present invention is the above-described imaging device, wherein the subject is detected based on a face detection unit that detects the face of the subject person in the input image data, and the face detected by the face detection unit in the image data. An eye opening / closing rate determining unit that determines an eye opening / closing rate of a person's eyes; a storage unit that stores eye opening / closing rate information including an eye opening / closing rate of the subject person's eyes determined by the eye opening / closing rate determining unit; The eye opening / closing rate of the subject person determined by the eye opening / closing rate determining unit is compared with the eye opening / closing rate obtained by multiplying the eye opening / closing rate in the stored eye opening / closing rate information by a predetermined correction allowable rate. A comparison unit; and a correction unit that corrects the eye opening / closing rate determined by the eye opening / closing rate determination unit according to a comparison result of the comparison unit.

  In addition, the present invention is the above-described moving image reproducing device that can reproduce image data and audio data in synchronization with each other, and is input at the time of moving image reproduction or immediately after a pause operation in the moving image reproducing device. A face detection unit for detecting the face of the subject person in the image data, and an eye opening / closing rate for determining an eye opening / closing rate of the subject person based on the face detected by the face detection unit in the image data A determination unit; a storage unit that stores eye opening / closing rate information including an eye opening / closing rate of the subject person's eyes determined by the eye opening / closing rate determination unit; and a subject person determined by the eye opening / closing rate determination unit A comparison unit that compares an eye opening / closing rate with an eye opening / closing rate obtained by multiplying the eye opening / closing rate in the stored eye opening / closing rate information by a predetermined correction allowable rate, and according to a comparison result of the comparison unit , Eye opening / closing rate judgment A correction unit that corrects the eye opening / closing rate determined by the correction unit, wherein the correction unit determines that the determined eye opening / closing rate during the reproduction of the moving image is equal to the eye opening / closing rate determined immediately after the pause operation. If the eye opening / closing rate is smaller than the eye opening / closing rate multiplied by a predetermined correction allowance rate, the determined eye opening / closing rate is corrected immediately after the pause operation using a predetermined enlargement rate.

  According to another aspect of the present invention, there is provided an imaging method as described above, the step of detecting a face of a subject person in input image data, and the eye of the subject person based on the detected face in the image data. A step of determining an eye open / close rate; a step of storing eye open / close rate information including an eye open / close rate of the determined subject person in the storage; and an eye open / close of the determined subject person's eye Comparing the eye opening / closing rate obtained by multiplying the eye opening / closing rate in the eye opening / closing rate information stored in the storage unit by a predetermined correction allowable rate, and the determination is made according to the comparison result. And correcting the eye opening / closing rate.

  According to the configuration described above, the eye opening / closing rate of the subject person can be effectively corrected when the eye opening / closing rate of the subject person in the captured image data is inappropriate.

  According to the present invention, when the open / close ratio of the subject person's eyes in the captured image data is inappropriate, the open / close ratio of the subject person's eyes can be effectively corrected.

Explanatory drawing which shows simply the operation | movement outline | summary of the imaging device of 1st Embodiment. 1 is a block diagram showing a hardware configuration of an imaging apparatus according to a first embodiment 1 is a block diagram illustrating in detail an internal configuration of a control unit of an imaging apparatus according to a first embodiment. Explanatory drawing which shows a part of calculation example of eye opening / closing rate x Explanatory drawing which shows another one part of the example of calculation of the eye opening / closing rate x Flowchart for explaining the operation of the imaging apparatus at the time of preview according to the first embodiment The flowchart explaining operation | movement of the imaging device immediately after completion | finish of imaging of 1st Embodiment Explanatory drawing which shows simply the operation | movement outline | summary of the moving image reproduction apparatus of 2nd Embodiment. The block diagram which shows the hardware constitutions of the moving image reproducing device of 2nd Embodiment The flowchart explaining operation | movement of the moving image reproducing device at the time of the moving image reproduction of 2nd Embodiment The flowchart explaining operation | movement of the moving image reproducing apparatus immediately after accepting pause operation during the moving image reproduction of 2nd Embodiment. Explanatory drawing which shows simply the operation | movement outline | summary of the imaging device of 3rd Embodiment. A block diagram showing hardware constitutions of an imaging device of a 3rd embodiment The flowchart explaining operation | movement of the imaging device immediately after imaging of the image for right eyes of 3rd Embodiment. The flowchart explaining operation | movement of the imaging device immediately after imaging of the image for left eyes of 3rd Embodiment. 1 is a block diagram illustrating a hardware configuration of an imaging apparatus according to a first modification of the first embodiment. The block diagram which shows in detail the internal structure of the control part of the imaging device of the modification 1 of 1st Embodiment. The block diagram which shows the hardware constitutions of the imaging device of the modification 2 of 1st Embodiment. The block diagram which shows in detail the internal structure of the control part of the imaging device of the modification 2 of 1st Embodiment. The block diagram which shows in detail the internal structure of the control part of the imaging device of the modification 3 of 1st Embodiment.

  Hereinafter, embodiments of an imaging device, a moving image reproducing device, and an imaging method according to the present invention will be described with reference to the drawings. The imaging apparatus and the moving image playback apparatus according to the present invention are electronic devices equipped with a camera function capable of imaging a subject, and examples thereof include a mobile phone, a smartphone, a digital still camera, and the like.

  Note that the present invention can also be expressed as an apparatus such as an imaging apparatus or a moving picture reproduction apparatus, or a program for operating the imaging apparatus or the moving picture reproduction apparatus as a computer. Furthermore, the present invention can also be expressed as an imaging method or a moving image reproduction method including processing (steps) executed by the imaging device or the moving image reproduction device. That is, the present invention can be expressed in any category of the apparatus, method, and program.

  In each of the following embodiments, a parameter indicating the open / closed degree of both eyes of the face of the subject person is defined as “eye open / close rate” in the image data captured by the image capturing apparatus or the moving image reproducing apparatus according to the present invention. The contents of the eye opening / closing rate will be described in detail with reference to FIGS.

(First embodiment)
FIG. 1 is an explanatory diagram simply showing an outline of the operation of the imaging apparatus 1 of the first embodiment. In the first embodiment, as illustrated in FIG. 1, the imaging apparatus 1 is configured to display each subject person (A-san, B-san, and C-san) at the time of previewing the group photo of A-san, B-san, and C-san. Memorize eye open / close rate. Immediately after taking the group photo, the imaging apparatus 1 sets any or all of the eye opening / closing rates of each subject person (Mr. A, Mr. B, Mr. C) of the captured image data to each eye opening / closing rate at the time of preview. If it is considerably smaller than that, the eye opening / closing rate of the subject person in the captured image data is corrected.

  FIG. 1 shows that in the captured image data, Mr. B's eye opening / closing rate is considerably smaller than Mr. B's eye opening / closing rate in the image data at the time of preview. The imaging apparatus 1 uses the Mr. B's eye opening / closing ratio in the image data at the time of preview to correct the Mr. B's eye opening / closing ratio of the captured image data to be enlarged.

  FIG. 2 is a block diagram illustrating a hardware configuration of the imaging apparatus 1 according to the first embodiment. As shown in FIG. 2, the imaging device 1 includes a control unit 11, an operation unit 12, a storage unit 13, an imaging unit 14, a display control unit 15, a display unit 16, a RAM (Random Access Memory) 17, and a ROM (Read Only Memory). ) 18.

  The control unit 11 is configured using a CPU (Central Processing Unit) built in the imaging apparatus 1 and controls the operation of each unit of the imaging apparatus 1. FIG. 3 is a block diagram illustrating in detail the internal configuration of the control unit 11 of the imaging apparatus 1 according to the first embodiment. The operation of each unit in FIG. 3 is realized by a CPU built in the imaging apparatus 1.

  As shown in FIG. 3, the control unit 11 includes an imaging control unit 21, a face detection unit 22, an eye open / close rate determination unit 23, an eye open / close rate management unit 24, a comparison unit 25, and a correction unit 26.

  The imaging control unit 21 inputs a user operation from the operation unit 12 to be described later, for example, an operation signal indicating that a group photo of Mr. A, Mr. B, and Mr. C is captured (captured). The imaging control unit 21 outputs an imaging signal to the imaging unit 14 based on the input operation signal. The imaging control unit 21 acquires the image data captured by the imaging unit 14 and outputs the image data and a face detection instruction to the face detection unit 22.

  The face detection unit 22 acquires image data and a face detection instruction from the imaging control unit 21, and detects the face of the subject person in the image data based on the face detection instruction. A technique for detecting the face of a subject person in image data is already known. For example, the face detection unit 22 detects three faces of a subject person in the image data of FIG. 1 and assigns a face ID (Identification) for identifying each face to each detected face. The face detection unit 22 outputs the face detection result and the eye opening / closing rate determination instruction to the eye opening / closing rate determination unit 23. The face detection result includes the face ID given by the face detection unit 22 and the image data subjected to face detection by the face detection unit 22.

  The eye open / close rate determination unit 23 acquires a face detection result and an eye open / close rate determination instruction, and based on the image data and the eye open / close rate determination instruction acquired together with the face ID, the eye open / close of the subject person in the image data Determine the rate. The eye opening / closing rate determination unit 23 outputs the face ID and the determination result (eye opening / closing rate) for the image data acquired together with the face ID to the eye opening / closing rate management unit 24.

  Here, the eye opening / closing rate will be described with reference to FIGS. FIG. 4 is an explanatory diagram illustrating a part of a calculation example of the eye opening / closing rate x, and FIG. 5 is an explanatory diagram illustrating another part of the calculation example of the eye opening / closing rate x. In the following description, the eye opening / closing rate determined by the eye opening / closing rate determination unit 23 is represented by parameter x, and the eye opening / closing rate stored in the storage unit 13 described later is represented by parameter y.

  In the first example of FIG. 4, the eye opening / closing rate x is represented by the height He in the vertical direction of one eye (for example, the right eye) with respect to the height Hf in the vertical direction of the face detected by the face detection unit 22. The degree of opening and closing of both eyes. Therefore, in the first example of FIG. 4, the eye opening / closing rate x is expressed by Equation (1).

  In the second example of FIG. 4, the eye opening / closing rate x is the height He in the vertical direction of one eye (eg, right eye) with respect to the horizontal width Wf of the face in the face detected by the face detection unit 22. The open / closed degree of both eyes shown may be sufficient. Note that the left eye may be used as one eye described above, and so on. Therefore, in the second example of FIG. 4, the eye opening / closing rate x is expressed by Equation (2).

  In the third example of FIG. 5, the eye opening / closing rate x is the one eye (eg, right eye) with respect to the lateral width We of one eye (eg, right eye) in the horizontal direction of the face detected by the face detection unit 22. The degree of opening and closing both eyes represented by the height He in the vertical direction may be used. Therefore, in the third example of FIG. 5, the eye opening / closing rate x is expressed by Equation (3).

  In the fourth example of FIG. 5, the eye opening / closing rate x is the vertical direction of one eye (for example, the right eye) with respect to the distance Be between the eyes in the horizontal direction of the face detected by the face detection unit 22. The opening / closing degree of both eyes represented by the height He may be used. Therefore, in the fourth example of FIG. 4, the eye opening / closing rate x is expressed by Equation (4).

  In calculating the eye opening / closing rate x using any one of the formulas (1) to (4), the height He in the vertical direction of one eye (eg, right eye or left eye) of the subject person's face is the other eye. It may be different from the height of (eg left eye or right eye). The eye opening / closing rate determination unit 23 may average the heights in the vertical direction of both eyes when calculating the eye opening / closing rate x using any one of the mathematical formulas (1) to (4). Thereby, the eye opening / closing rate determination unit 23 can calculate the eye opening / closing rate x with high accuracy. Further, the eye opening / closing rate determination unit 23 selects a large value of each height He in the vertical direction of the left eye and the right eye in the calculation of the eye opening / closing rate x using any one of the mathematical formulas (1) to (4). May be.

  The eye opening / closing rate management unit 24 acquires the face ID and the eye opening / closing rate x output together with the face ID from the eye opening / closing rate determination unit 23. The eye open / close rate management unit 24 acquires the eye open / close rate y of the eye open / close rate information stored in the storage unit 13 in association with the same face ID, and acquires the face ID, the eye open / close rate x, the eye open / close rate y, and the eye open / close rate. A comparison instruction between the rate x and the eye opening / closing rate y is output to the comparison unit 25. The eye opening / closing rate management unit 24 stores the eyes stored in the storage unit 13 in association with the same face ID based on the comparison result output from the comparison unit 25, the face ID to be compared and the eye opening / closing rate x. The eye opening / closing rate y in the opening / closing rate information is updated to the eye opening / closing rate x. The comparison result from the comparison unit 25 will be described later.

  The comparison unit 25 acquires from the eye opening / closing rate management unit 24 a face ID, an eye opening / closing rate x, an eye opening / closing rate y, and a comparison instruction between the eye opening / closing rate x and the eye opening / closing rate y. The comparison unit 25 compares the eye opening / closing rate y of the eye opening / closing rate information stored in association with the face ID with the eye opening / closing rate x determined by the eye opening / closing rate determination unit 23. When the eye opening / closing rate x is larger than the eye opening / closing rate y, the comparison unit 25 obtains the comparison result that the eye opening / closing rate x is larger than the eye opening / closing rate y, the face ID to be compared, and the eye opening / closing rate x. The data is output to the eye opening / closing rate management unit 24.

  When the parameter μy obtained by multiplying the eye opening / closing rate y by a predetermined correction allowable rate μ is larger than the eye opening / closing rate x, the comparison unit 25 inputs the face ID to be compared and the imaging control unit 21. The image data, the eye open / close rate x, the eye open / close rate y, and the correction instruction are output to the correction unit 26. Further, when the parameter μy obtained by multiplying the eye opening / closing rate y by a predetermined correction allowable rate μ is equal to or smaller than the eye opening / closing rate x, the comparison unit 25 causes the correction unit 26 to store the image data input to the imaging control unit 21. A display instruction is output to the display control unit 15 so as to display on the display unit 16 without correction.

  Here, the correction allowable rate μ is, for example, a predetermined value provided in order to avoid correction that enlarges the face when the image data of a face whose subject person's eyes are slightly swollen is captured. For example, it is 0.8. However, it is assumed that the correction allowable rate μ is defined in advance in the operation of the comparison unit 25.

  The correction unit 26 acquires the face ID to be compared by the comparison unit 25, the image data input to the imaging control unit 21, the eye opening / closing rate x, the eye opening / closing rate y, and the correction instruction from the comparison unit 25. The correction unit 26 corrects the eye open / close rate of the subject person associated with the face ID of the image data using a predetermined magnification (see FIG. 1). Here, the predetermined enlargement ratio is, for example, eye opening / closing ratio y / eye opening / closing ratio x. The correction unit 26 outputs to the display control unit 15 image data in which the eye open / close ratio of the subject person associated with the face ID is corrected, and a display instruction for displaying the image data on the display unit 16.

  The operation unit 12 is a user interface for a user to input an operation on the imaging apparatus 1, and outputs an operation signal corresponding to the operation content of the user to the control unit 11. For example, the operation unit 12 outputs to the control unit 11 an operation signal indicating that the user's operation content, for example, a group photo of Mr. A, Mr. B, and Mr. C is taken (photographed).

  For example, when the imaging apparatus 1 is a mobile phone, the operation unit 12 can be configured with various keys such as a numeric keypad for inputting a telephone number, a telephone key for performing on-hook or off-hook, and a function key. The operation unit 12 may be a touch panel that is arranged on the display unit 16 and can accept an input operation with a user's finger or a stylus pen.

  The storage unit 13 is configured by using a hard disk or a flash memory built in the imaging apparatus 1, and includes image data captured by the imaging unit 14, and an eye opening / closing rate y associated with the face ID and the same face ID. Is stored.

  The imaging unit 14 is configured using a mechanism that includes a lens, a finder, and a CCD (Charged Couples Device) element built in the imaging device 1 and has a camera function. Based on the imaging signal output from the control unit 11, Take an image of a subject (eg, a person) projected through the viewfinder. The imaging unit 14 outputs image data generated by imaging to the control unit 11.

  The display control unit 15 is configured using a CPU built in the imaging apparatus 1, acquires a display instruction and image data output from the control unit 11, and displays image data on the display unit 16 based on the display instruction. .

  The display unit 16 is configured using an LCD (Liquid Crystal Display), and the image data captured by the imaging unit 14 in accordance with a user operation under the control of the display control unit 15 or the image data is stored in the control unit. The corrected image data corrected by 11 is displayed.

  The RAM 17 operates as a work memory in each operation of the control unit 11, the operation unit 12, the imaging unit 14, or the display control unit 15. In FIG. 2, an arrow from the operation unit 12, the imaging unit 14, or the display control unit 15 to the RAM 17 is omitted.

  The ROM 18 stores a program in which the operations of the control unit 11 and the display control unit 15 in the imaging apparatus 1 are defined in advance. The control unit 11 can be configured by hardware or software. In particular, when the control unit 11 is configured by software, the CPU incorporated in the imaging device 1 reads the program in which each operation of the control unit 11 is defined in advance from the ROM 18 so that the control unit 11 can operate. It becomes. In FIG. 2, the illustration of the arrow to the ROM 18 is omitted.

  Next, the operation of the imaging apparatus 1 during preview according to the first embodiment will be described in detail with reference to FIG. FIG. 6 is a flowchart for explaining the operation of the imaging apparatus 1 during the preview according to the first embodiment. In the description of FIG. 6, the operation outline of the imaging apparatus 1 of FIG. 1 is referred to as necessary.

  In FIG. 6, the imaging control unit 21 acquires image data captured by the imaging unit 14 (S11), and outputs the image data and a face detection instruction to the face detection unit 22. The face detection unit 22 acquires image data and a face detection instruction from the imaging control unit 21, and detects the face of the subject person in the image data (S12). It is assumed that there are three subject persons detected by the face detection in step S12 (see FIG. 1).

  The face detection unit 22 defines the parameter of the face ID representing the subject person detected by the face detection in step S12 as j, and the face detection result in the face ID of the first subject person (ie, j = 1, S13). The eye opening / closing rate determination instruction is output to the eye opening / closing rate determination unit 23. Note that the face detection result for the face ID of the first subject person includes the face ID and the image data subjected to face detection by the face detection unit 22. Further, it is assumed that the maximum value of the parameter j is known by the face detection unit 22 when the face is detected in step S12.

  The eye opening / closing rate determination unit 23 acquires a face detection result and an eye opening / closing rate determination instruction from the face detection unit 22, and based on the image data acquired together with the face ID, the eye of the first subject person in the image data The eye open / close rate x is determined (S14). The eye opening / closing rate determination unit 23 outputs the face ID and the determination result (eye opening / closing rate x) for the image data acquired together with the face ID to the eye opening / closing rate management unit 24.

  The eye opening / closing rate management unit 24 acquires the face ID and the eye opening / closing rate x output together with the face ID from the eye opening / closing rate determination unit 23. The eye open / close rate management unit 24 acquires the eye open / close rate y of the eye open / close rate information stored in the storage unit 13 in association with the same face ID (S15), and the face ID, the eye open / close rate x, and the eye open / close rate y. In addition, a comparison instruction between the eye opening / closing rate x and the eye opening / closing rate y is output to the comparison unit 25.

  The comparison unit 25 acquires from the eye opening / closing rate management unit 24 a face ID, an eye opening / closing rate x, an eye opening / closing rate y, and a comparison instruction between the eye opening / closing rate x and the eye opening / closing rate y. The comparison unit 25 compares the eye opening / closing rate y of the eye opening / closing rate information stored in association with the face ID with the eye opening / closing rate x determined by the eye opening / closing rate determination unit 23 (S16).

  When the eye opening / closing rate x is larger than the eye opening / closing rate y (S16, YES), the comparison unit 25 determines that the comparison result that the eye opening / closing rate x is larger than the eye opening / closing rate y, the face ID to be compared, and The eye opening / closing rate x is output to the eye opening / closing rate management unit 24. The eye opening / closing rate management unit 24 stores the eyes stored in the storage unit 13 in association with the same face ID based on the comparison result output from the comparison unit 25, the face ID to be compared and the eye opening / closing rate x. The eye opening / closing rate y in the opening / closing rate information is updated to the eye opening / closing rate x (S17). The eye opening / closing rate management unit 24 outputs to the comparison unit 25 that the eye opening / closing rate y of the eye opening / closing rate information stored in the storage unit 13 in association with the same face ID is updated to the eye opening / closing rate x.

  When the eye opening / closing rate x is less than the eye opening / closing rate y (S16, NO), or after step S17, the comparison unit 25 increments the parameter j (S18), and the next face ID in the same image data. Outputs an eye opening / closing rate determination instruction to the eye opening / closing rate determination unit 23. In FIG. 6, the operations in steps S14 to S18 are repeated until the parameter j incremented in step S18 exceeds the maximum value of the parameter j (S19, NO). When the parameter j exceeds the maximum value of the parameter j (S19, YES), the operation of the imaging device 1 in FIG.

  Next, the operation of the imaging apparatus 1 immediately after the end of imaging according to the first embodiment will be described with reference to FIG. FIG. 7 is a flowchart for explaining the operation of the imaging apparatus 1 immediately after the end of imaging according to the first embodiment. In the description of FIG. 7, the operation outline of the imaging device 1 of FIG. 1 is referred to as necessary.

  In FIG. 7, the imaging control unit 21 acquires image data captured by the imaging unit 14 (S 21), and outputs the image data and a face detection instruction to the face detection unit 22. The face detection unit 22 acquires image data and a face detection instruction from the imaging control unit 21, and detects the face of the subject person in the image data (S22). Assume that the number of subject persons detected by the face detection in step S22 is three as in the preview of FIG. 6 (see FIG. 1).

  The face detection unit 22 defines the parameter of the face ID representing the subject person detected by the face detection in step S12 as j, and the face detection result in the face ID of the first subject person (ie, j = 1, S23). The eye opening / closing rate determination instruction is output to the eye opening / closing rate determination unit 23.

  The eye opening / closing rate determination unit 23 acquires a face detection result and an eye opening / closing rate determination instruction from the face detection unit 22, and based on the image data acquired together with the face ID, the eye of the first subject person in the image data The eye open / close rate x is determined (S24). The eye opening / closing rate determination unit 23 outputs the face ID and the determination result (eye opening / closing rate x) for the image data acquired together with the face ID to the eye opening / closing rate management unit 24.

  The eye opening / closing rate management unit 24 acquires the face ID and the eye opening / closing rate x output together with the face ID from the eye opening / closing rate determination unit 23. The eye open / close rate management unit 24 acquires the eye open / close rate y of the eye open / close rate information stored in the storage unit 13 in association with the same face ID (S25), and the face ID, the eye open / close rate x, and the eye open / close rate y. In addition, a comparison instruction between the eye opening / closing rate x and the eye opening / closing rate y is output to the comparison unit 25.

  The comparison unit 25 acquires from the eye opening / closing rate management unit 24 a face ID, an eye opening / closing rate x, an eye opening / closing rate y, and a comparison instruction between the eye opening / closing rate x and the eye opening / closing rate y. The comparison unit 25 compares the parameter μy obtained by multiplying the eye opening / closing rate y by a predetermined correction allowable rate μ with the eye opening / closing rate x (S26).

  When the parameter μy is larger than the eye opening / closing rate x (S26, YES), the comparison unit 25 compares the face ID to be compared, the image data input to the imaging control unit 21, the eye opening / closing rate x, and the eye opening / closing. The rate y and the correction instruction are output to the correction unit 26. The correction unit 26 acquires the face ID to be compared by the comparison unit 25, the image data input to the imaging control unit 21, the eye opening / closing rate x, the eye opening / closing rate y, and the correction instruction from the comparison unit 25. The correction unit 26 corrects the eye open / close rate of the face of the subject person associated with the face ID (parameter j) of the image data using the predetermined magnification described above (S27).

  After step S27, the correction unit 27 increments the parameter j (S28), and outputs an eye opening / closing rate determination instruction for the next face ID to the eye opening / closing rate determination unit 23 in the same image data. In FIG. 3, an arrow input from the correction unit 27 to the eye opening / closing rate determination unit 23 is omitted. In FIG. 7, each operation of steps S24 to S28 is repeated until the parameter j incremented in step S28 exceeds the maximum value of the parameter j (S29, NO). When the parameter j exceeds the maximum value of the parameter j (S29, YES), the operation after the imaging of the imaging device 1 in FIG. 7 ends.

  On the other hand, when the parameter μy is equal to or smaller than the eye opening / closing rate x (S26, NO), the comparison unit 25 displays the image data input to the imaging control unit 21 on the display unit 16 without correcting the image data by the correction unit 26. In this manner, a display instruction is output to the display control unit 15. Thereafter, the comparison unit 25 increments the parameter j (S28), and outputs an eye open / close rate determination instruction for the next face ID to the eye open / close rate determination unit 23 in the same image data. In FIG. 3, an arrow input from the correction unit 27 to the eye opening / closing rate determination unit 23 is omitted. In FIG. 7, each operation of steps S24 to S28 is repeated until the parameter j incremented in step S28 exceeds the maximum value of the parameter j (S29, NO). When the parameter j exceeds the maximum value of the parameter j (S29, YES), the operation after the imaging of the imaging device 1 in FIG. 7 ends.

  As described above, the imaging apparatus 1 according to the first embodiment, for example, when capturing a group photo of a plurality of subject persons, stores the eye opening / closing ratio of each subject person determined at the time of previewing in advance in the storage unit. The eye open / close rate of the eye open / close rate information stored in association with the face ID of the subject person is compared. As a result of the comparison, when the eye open / close rate of each subject person determined at the time of preview is larger than the eye open / close rate stored in the storage unit, the imaging device 1 stores the eye open / close stored in the storage unit 13. Update rate.

  In other words, since the size of the eyes is different for each subject person, when the eye enlargement correction is uniformly performed on all subject persons, the eye size is corrected to an unintended size of the subject person himself / herself. In order to avoid such uniform enlargement correction, the imaging apparatus 1 compares the eye open / close rate determined for the face ID of each subject person with the eye open / close rate stored in the storage unit during preview. Then, the eye opening / closing rate having a large value is stored in the storage unit so as not to be a photograph of a state of eye-opening or a state of half-eye.

  Further, immediately after the group photo is taken, the imaging apparatus 1 similarly determines the eye open / close rate of each subject person's eyes, and the eye open / close rate stored in association with the same face ID in the storage unit. Compare the eye opening / closing rate of information. In accordance with the comparison result, the imaging apparatus 1 stores at the time of previewing even when the eye opening / closing rate of each subject person becomes an unintended eye opening / closing rate in the captured image data. The eye opening / closing ratio stored in the unit is used to correct the eye so that it can be enlarged.

  Therefore, according to the imaging device 1, when the opening / closing rate of the subject's human eye in the captured image data is inappropriate, the opening / closing rate of the subject's human eye can be effectively corrected.

(Second Embodiment)
FIG. 8 is an explanatory diagram simply showing an outline of the operation of the moving image playback apparatus of the second embodiment. In the second embodiment, as shown in FIG. 8, the moving image reproducing apparatus forms a moving image when reproducing the moving image data of Mr. A, Mr. B, and Mr. C stored in the storage unit. The eye open / close ratio of each subject person (Mr. A, Mr. B, Mr. C) in each image data is determined and stored. When the video playback device accepts a pause operation as a user operation, any or all of the eye open / close rates of each subject person (Mr. A, Mr. B, Mr. C) of the image data when the pause operation is accepted Is considerably smaller than each eye opening / closing rate stored during operation reproduction, the eye opening / closing rate of the subject person in the image data when the pause operation is accepted is corrected.

  FIG. 8 shows that in the image data when the pause operation is accepted, Mr. B's eye opening / closing rate is considerably smaller than Mr. B's eye opening / closing rate during operation reproduction. The moving image reproducing apparatus corrects the B eye opening / closing rate of the image data when the pause operation is received, using the B eye opening / closing rate in the image data at the time of moving image reproduction.

  FIG. 9 is a block diagram illustrating a hardware configuration of the moving image playback device 1b according to the second embodiment. As shown in FIG. 9, the moving image playback device 1 b includes a control unit 11, an operation unit 12 b, a storage unit 13, an imaging unit 14, an operation playback control unit 31, a display unit 16, an audio output unit 32, a RAM 17, and a ROM 18. It is. In FIG. 9, description of the same contents as those of the respective parts in FIG. 2 is omitted, and different contents are described. 9 is configured so as to include the imaging unit 14, it may be configured not to include the imaging unit 14. Furthermore, when the moving image reproducing device 1b is configured to include the imaging unit 14, it may be considered as an imaging device in terms of including the imaging unit 14.

  The storage unit 13 stores moving image data to be reproduced by the moving image reproducing device 1b of the second embodiment.

  The moving image playback control unit 31 is configured by using a CPU built in the moving image playback device 1b, and acquires a moving image playback operation signal selected by a user operation received by the operation unit 12b. The moving image reproduction control unit 31 reads out moving image data to be reproduced from the storage unit 13 based on a moving image reproduction operation signal from the operation unit 12b.

  Based on the read video data, the video playback control unit 31 synchronizes the video signal so as to synchronize the video signal (image signal) and audio signal of the video selected by the user operation received by the operation unit 12b. The same audio signal is output to the audio output unit 32 on the display unit 16. Furthermore, the moving image reproduction control unit 31 outputs a plurality of image data constituting the moving image data to the control unit 11 when reproducing the moving image data.

  The moving image reproduction control unit 31 acquires an operation signal of a moving image pause operation from the operation unit 12b when the operation unit 12b receives a pause operation of a moving image being reproduced as a user operation. The moving image reproduction control unit 31 outputs image data constituting the moving image at the time of suspension to the control unit 11 based on the operation signal of the moving image pause operation from the operation unit 12b.

  The audio output unit 32 is configured by a speaker or the like, and receives the audio signal of the moving image output from the moving image reproduction control unit 31 and outputs the audio signal.

  Next, the operation of the video playback device 1b during video playback according to the second embodiment will be described with reference to FIG. FIG. 10 is a flowchart for explaining the operation of the moving image playback apparatus 1b during the operation playback of the second embodiment. In the description of FIG. 10, the description of the same content as that of FIG. 6 is omitted. As described above, it is assumed that the moving image reproduction control unit 31 of the moving image reproduction device 1b is reproducing the moving image selected by the user operation received by the operation unit 12b as a premise of the description of FIG.

  In FIG. 10, the imaging control unit 21 acquires the image data output from the moving image reproduction control unit 31 (S31). The imaging control unit 21 outputs the image data and the face detection instruction output from the moving image reproduction control unit 31 to the face detection unit 22. In FIG. 10, the processes after step S31 are the same as steps S12 to S19 in FIG.

  Next, the operation of the video playback device 1b immediately after receiving a pause operation during video playback according to the second embodiment will be described with reference to FIG. FIG. 11 is a flowchart for explaining the operation of the video playback device immediately after receiving a pause operation during video playback according to the second embodiment. In the description of FIG. 11, the outline of the operation of the moving image playback device 1b of FIG. 8 is referred to as necessary, and the description of the contents overlapping with the description of FIG. As described above, as a premise of the description of FIG. 11, the video playback control unit 31 of the video playback device 1b accepts a user operation (video pause operation) from the operation unit 12b during playback of a video. Suppose it is time.

  In FIG. 11, when the operation unit 12b receives a pause operation for a moving image being played back as a user operation, the moving image reproduction control unit 31 acquires an operation signal for a moving image pause operation from the operation unit 12b. The moving image reproduction control unit 31 outputs image data constituting the moving image at the time of suspension to the control unit 11 based on the operation signal of the moving image pause operation from the operation unit 12b.

  The imaging control unit 21 acquires the image data at the time of the pause operation output from the moving image reproduction control unit 31 (S41). The imaging control unit 21 outputs the image data and the face detection instruction output from the moving image reproduction control unit 31 to the face detection unit 22. In FIG. 11, the processes after step S41 are the same as steps S22 to S29 in FIG.

  As described above, the moving image reproducing device 1b according to the second embodiment, for example, reproduces the moving image data stored in the storage unit 13 while reproducing each object person determined for each image data constituting the moving image data. The eye open / close rate is compared with the eye open / close rate of the eye open / close rate information stored in the storage unit in advance in association with the face ID of the subject person. As a result of the comparison, when the eye open / close rate of each subject person determined in each image data constituting the moving image data during reproduction is larger than the eye open / close rate stored in the storage unit, the imaging device 1b stores The eye opening / closing rate stored in the unit 13 is updated.

  Furthermore, immediately after receiving the pause operation from the user during the playback of the video data, the video playback device 1b also receives the eye of each subject person with respect to the image data constituting the video data when the pause operation is similarly received. The eye open / close rate is determined, and the eye open / close rate is compared with the eye open / close rate of the eye open / close rate information stored in the storage unit in association with the face ID. In accordance with the comparison result, the video playback device 1b has a case where the eye opening / closing rate of each subject person becomes an unintended eye opening / closing rate in the image data when the pause operation is received. However, when the moving image is reproduced, the eye opening / closing rate stored in the storage unit is used to correct the eye so that the eye can be enlarged.

  Therefore, according to the moving image reproducing device 1b, as with the imaging device 1 of the first embodiment, the opening / closing rate of the eye of the subject person in the image data constituting the moving image data when the pause operation is accepted is inappropriate. In such a case, it is possible to effectively correct the open / close ratio of the subject's eyes.

(Third embodiment)
FIG. 12 is an explanatory diagram simply showing an outline of the operation of the imaging apparatus 1c of the third embodiment. In the third embodiment, as shown in FIG. 12, the imaging device 1c, for example, first captures a right-eye image, for example, in order to capture a 3D group photo of Mr. A, Mr. B, and Mr. C. Take an image. The imaging device 1c determines the eye open / close ratio of each subject person (Mr. A, Mr. B, Mr. C) for the right-eye image data and the left-eye image data, as in the above-described embodiments, Compare eye opening and closing rates.

  As shown in FIG. 12, when the eye opening / closing rate of Mr. B in the left-eye image data is considerably smaller than the eye opening / closing rate of Mr. B in the right-eye image data, the imaging device 1c The eye open / close rate is corrected to be the same as the eye open / close rate of B in the right-eye image data. Similarly, when the eye opening / closing rate of Mr. B in the left-eye image data is considerably larger than the eye opening / closing rate of Mr. B in the right-eye image data, the imaging device 1c, conversely, The eye open / close rate is corrected to be the same as the eye open / close rate of Mr. B in the image data for the left eye. The imaging device 1c generates a 3D group photo of Mr. A, Mr. B, and Mr. C by synthesizing the right-eye image data and the corrected left-eye image data. Further, in the following third embodiment, it is described that the right-eye image and the left-eye image are captured in this order, but the left-eye image and the right-eye image may be captured in this order.

  FIG. 13 is a block diagram illustrating a hardware configuration of the imaging apparatus 1c according to the third embodiment. As illustrated in FIG. 13, the imaging device 1 c includes a control unit 11, an operation unit 12, a storage unit 13, an imaging unit 14 c, an image composition unit 41, a display control unit 15, a display unit 16, a RAM 17, and a RAM 18. . In FIG. 13, description of the same content as that of each part in FIG. 2 is omitted, and different content will be described.

  The imaging unit 14c is configured using a mechanism that includes a lens, a finder, and a CCD element built in the imaging device 1c and has a camera function, and is projected via the finder based on an imaging signal output from the control unit 11. The subject is imaged. Furthermore, the imaging unit 14c can capture a 3D image. For example, the imaging unit 14c captures a right-eye image and then a left-eye image. The imaging unit 14 outputs the left-eye image data and the right-eye image generated by imaging to the control unit 11, respectively.

  The image composition unit 41 acquires the left-eye image data and the right-eye image data output from the control unit 11, and generates 3D image data by synthesizing the left-eye image data and the right-eye image data. The image composition unit 41 outputs the generated 3D image data to the display control unit 15.

  Next, the operation of the imaging apparatus 1c immediately after imaging the right-eye image of the third embodiment will be described with reference to FIG. FIG. 14 is a flowchart for explaining the operation of the imaging apparatus 1c immediately after imaging the right-eye image according to the third embodiment. In the description of FIG. 14, the operation outline of the imaging device 1 c in FIG. 12 is referred to as necessary.

  In FIG. 14, the imaging control unit 21 acquires right-eye image data captured by the imaging unit 14 c (S <b> 51), and outputs the right-eye image data and a face detection instruction to the face detection unit 22. The face detection unit 22 acquires the image data for the right eye and the face detection instruction from the imaging control unit 21, and detects the face of the subject person in the image data for the right eye (S12). It is assumed that there are three subject persons detected by the face detection in step S12 (see FIG. 12).

  The face detection unit 22 defines the parameter of the face ID representing the subject person detected by the face detection in step S12 as j, and the face detection result in the face ID of the first subject person (ie, j = 1, S13). The eye opening / closing rate determination instruction is output to the eye opening / closing rate determination unit 23. Note that the face detection result in the face ID of the first subject person includes the face ID and the image data for the right eye that has been subject to face detection by the face detection unit 22. Further, it is assumed that the maximum value of the parameter j is known by the face detection unit 22 when the face is detected in step S12.

  The eye opening / closing rate determination unit 23 acquires the face detection result and the eye opening / closing rate determination instruction from the face detection unit 22, and based on the right eye image data acquired together with the face ID, the first eye data in the right eye image data is obtained. The eye opening / closing rate x1 of the subject person is determined (S54). The eye opening / closing rate determination unit 23 outputs the face ID and the determination result (eye opening / closing rate x1) for the image data for the right eye acquired together with the face ID to the eye opening / closing rate management unit 24.

  The eye opening / closing rate management unit 24 acquires the face ID and the eye opening / closing rate x1 output together with the face ID from the eye opening / closing rate determination unit 23. The eye opening / closing rate management unit 24 stores the face ID, the eye opening / closing rate x1, and the storage unit 13 (S55).

  Step S55 may be performed as follows.

  Specifically, when the eye opening / closing rate y of the eye opening / closing rate information is stored in advance in the storage unit 13 in association with the face ID, the eye opening / closing rate management unit 24 stores the eye opening / closing rate in association with the face ID. The eye opening / closing rate y of the current eye opening / closing rate information is acquired, and the face ID, the eye opening / closing rate x1, the eye opening / closing rate y, and the comparison instruction between the eye opening / closing rate x1 and the eye opening / closing rate y are output to the comparison unit 25. In this case, the comparison unit 25 compares the eye opening / closing rate y of the eye opening / closing rate information stored in association with the face ID and the eye opening / closing rate x1 determined by the eye opening / closing rate determination unit 23. Only when the eye opening / closing rate x1 is larger than the eye opening / closing rate y, the comparison unit 25 obtains the comparison result that the eye opening / closing rate x1 is larger than the eye opening / closing rate y, the face ID to be compared, and the eye opening / closing rate x1. The data is output to the eye opening / closing rate management unit 24. The eye opening / closing rate management unit 24 stores the eye stored in the storage unit 13 in association with the same face ID based on the comparison result output from the comparison unit 25, the face ID to be compared and the eye opening / closing rate x1. The eye opening / closing rate y in the opening / closing rate information is updated to the eye opening / closing rate x1 (S55).

  After step S55, the eye opening / closing rate management unit 24 increments the parameter j (S18), and outputs an eye opening / closing rate determination instruction for the next face ID to the eye opening / closing rate determination unit 23 in the same right-eye image data. In FIG. 14, the operations of steps S54, S55, and S18 are repeated until the parameter j incremented in step S18 exceeds the maximum value of the parameter j (S19, NO). When the parameter j exceeds the maximum value of the parameter j (S19, YES), the operation immediately after the imaging of the right-eye image of the imaging device 1c in FIG. 14 ends.

  Next, the operation of the imaging device 1c immediately after imaging the left-eye image of the third embodiment will be described with reference to FIG. FIG. 15 is a flowchart for explaining the operation of the imaging apparatus 1c immediately after imaging the left-eye image of the third embodiment. In the description of FIG. 15, the operation outline of the imaging device 1 c of FIG. 12 is referred to as necessary.

  In FIG. 15, the imaging control unit 21 acquires left-eye image data captured by the imaging unit 14 c (S 61), and outputs the left-eye image data and a face detection instruction to the face detection unit 22. The face detection unit 22 acquires the left-eye image data and the face detection instruction from the imaging control unit 21, and detects the face of the subject person in the left-eye image data (S22). It is assumed that there are three subject persons detected by the face detection in step S22 (see FIG. 12).

  The face detection unit 22 defines the parameter of the face ID representing the subject person detected by the face detection in step S22 as j, and the face detection result in the face ID of the first subject person (ie, j = 1, S23). The eye opening / closing rate determination instruction is output to the eye opening / closing rate determination unit 23. Note that the face detection result for the face ID of the first subject person includes the face ID and the image data for the left eye that has been subject to face detection by the face detection unit 22. Further, it is assumed that the maximum value of the parameter j is known by the face detection unit 22 at the time of face detection in step S22.

  The eye opening / closing rate determination unit 23 acquires the face detection result and the eye opening / closing rate determination instruction from the face detection unit 22, and based on the image data for the left eye acquired together with the face ID, the first eye data in the left eye image data The eye opening / closing rate x2 of the subject person is determined (S64). The eye opening / closing rate determination unit 23 outputs the face ID and the determination result (eye opening / closing rate x2) for the image data for the left eye acquired together with the face ID to the eye opening / closing rate management unit 24.

  The eye opening / closing rate management unit 24 acquires the face ID and the eye opening / closing rate x2 output together with the face ID from the eye opening / closing rate determination unit 23. The eye open / close rate management unit 24 acquires the eye open / close rate x1 (see FIG. 14) of the eye open / close rate information stored in association with the same face ID from the storage unit 13 (S65). The eye opening / closing rate management unit 24 outputs to the comparison unit 25 a face ID, an eye opening / closing rate x1, an eye opening / closing rate x2, and a comparison instruction between the eye opening / closing rate x1 and the eye opening / closing rate x2.

  The comparison unit 25 acquires from the eye opening / closing rate management unit 24 a face ID, an eye opening / closing rate x1, an eye opening / closing rate x2, and a comparison instruction between the eye opening / closing rate x1 and the eye opening / closing rate x2. The comparison unit 25 compares the eye opening / closing rate x1 with the eye opening / closing rate x2 (S66).

  When the eye opening / closing rate x2 is larger than the eye opening / closing rate x1 (S66, YES), the comparison unit 25 determines the face ID to be compared, the left-eye image data and the right-eye image input to the imaging control unit 21. The data, the eye opening / closing rate x1, the eye opening / closing rate x2, and the correction instruction are output to the correction unit 26. The correction unit 26 compares the face ID to be compared by the comparison unit 25, the left eye image data and the right eye image data input to the imaging control unit 21, the eye open / close rate x1, the eye open / close rate x2, and the correction instruction. Obtained from the unit 25.

  When the eye opening / closing rate x2 is larger than the eye opening / closing rate x1, the eye opening / closing rates of the right eye image data and the left eye image data do not match, and the eye opening / closing rate in the left eye image data is equal to the eye opening / closing rate in the right eye image data. It is necessary to correct the rate. Therefore, the correction unit 26 corrects the eye open / close rate of the face of the subject person associated with the face ID (parameter j) of the image data for the left eye using a predetermined magnification (S67). The predetermined enlargement ratio in step S67 is, for example, eye opening / closing ratio x2 / eye opening / closing ratio x1.

  On the other hand, when the eye opening / closing rate x1 is larger than the eye opening / closing rate x2 (S66, YES), the comparison unit 25 determines the comparison target face ID, the left-eye image data input to the imaging control unit 21, and the right eye. The image data, the eye opening / closing rate x1, the eye opening / closing rate x2, and the correction instruction are output to the correction unit 26. The correction unit 26 compares the face ID to be compared by the comparison unit 25, the left eye image data and the right eye image data input to the imaging control unit 21, the eye open / close rate x1, the eye open / close rate x2, and the correction instruction. Obtained from the unit 25.

  When the eye opening / closing rate x1 is larger than the eye opening / closing rate x2, the eye opening / closing rates of the right eye image data and the left eye image data do not match, and the eye opening / closing rate in the right eye image data is equal to the eye opening / closing rate in the left eye image data. It is necessary to correct the rate. Accordingly, the correction unit 26 corrects the eye opening / closing rate of the face of the subject person associated with the face ID (parameter j) of the right-eye image data using a predetermined magnification (S68). The predetermined enlargement ratio in step S68 is, for example, eye opening / closing ratio x1 / eye opening / closing ratio x2.

  After step S67 or step S68, the correction unit 27 increments the parameter j (S28) and outputs an eye opening / closing rate determination instruction for the next face ID to the eye opening / closing rate determination unit 23 in the same left-eye image data. In FIG. 15, the operations in steps S64 to S68 and step S28 are repeated until the parameter j incremented in step S28 exceeds the maximum value of the parameter j (S29, NO). When the parameter j exceeds the maximum value of the parameter j (S29, YES), the operation immediately after the imaging of the left-eye image of the imaging device 1c in FIG.

  As described above, the imaging apparatus 1c according to the third embodiment captures the subject person in the order of, for example, the right-eye image and the left-eye image in order to generate, for example, 3D image data. The imaging device 1c stores the eye opening / closing ratio x1 of each subject person determined for the right-eye image data in the storage unit 13. The imaging device 1c compares the eye opening / closing rate x2 of each subject person determined for the left-eye image data with the eye opening / closing rate x1 stored in the storage unit 13, and based on the eye opening / closing rate having a large value. In addition, the left-eye image data or the right-eye image data is corrected.

  Therefore, according to the imaging device 1c, immediately after the 3D image is captured, the eye opening / closing rate of the subject person's eyes is clogged due to an unintentional blink or the like, or the state of the half eye, etc. Even when it is inappropriate, the eye opening / closing ratio of each subject person in the left-eye image data and the right-eye image data can be corrected to the same level, and the operability of imaging can be improved.

(Modification 1 of the first embodiment)
FIG. 16 is a block diagram illustrating a hardware configuration of an imaging apparatus 1d according to Modification 1 of the first embodiment. In the first modification of the first embodiment, the imaging device 1d includes a control unit 51, a timer 52, an operation unit 12, a storage unit 13, an imaging unit 14, a display control unit 15, a display unit 16, a RAM 17, and a ROM 18. It is. Components in FIG. 16 that operate in the same manner as the corresponding components in FIG. 2 are given the same reference numerals. In the operation of each unit in FIG. 16, the description of the same operation as each corresponding unit in FIG. 2 will be omitted, and different contents will be described.

  In the first modification of the first embodiment, the control unit 51 can change the correction allowable rate μ at the time of comparison by the comparison unit 25 according to the browsing time of image data such as a photograph stored in the storage unit 13. A weighting coefficient for determining the weight is determined. The browsing time of image data such as a photograph is measured by the timer 52.

  FIG. 17 is a block diagram illustrating in detail the internal configuration of the control unit 51 of the imaging device 1d according to the first modification of the first embodiment. As shown in FIG. 17, the control unit 51 includes an imaging control unit 21, a face detection unit 22, an eye open / close rate determination unit 23, an eye open / close rate management unit 24, a comparison unit 25, a correction unit 26, and a weighting coefficient determination unit 27. It is the composition which includes. 17 that operate in the same manner as the corresponding units in FIG. 3 are denoted by the same reference numerals. In the operation of each part in FIG. 17, the description of the same operation as each corresponding part in FIG. 3 will be omitted, and different contents will be described.

  The weighting factor determination unit 27 acquires the browsing time of the image data output from the timer 52, and determines the weighting factor according to the browsing time. The viewing time may be the accumulated time. In this case, the weighting factor determination unit 27 reads the accumulated time information stored in the storage unit 13 for each image data, adds the viewing time output from the timer 52, and then weights. It is preferable to determine the coefficient.

  The weighting factor determination unit 27 uses a table (not shown) in which the measurement time from the timer 52 (cumulative time is acceptable, the same applies below) and the weighting factor are associated with each other, for example, the measurement time from the timer 52 is 30 seconds. In some cases, the weighting factor is determined to be 2.0. Furthermore, for example, the weighting factor determination unit 27 determines that the weighting factor is 2.5 when the measurement time from the timer 52 is 60 seconds. The weight coefficient determination unit 27 outputs the determined weight coefficient to the comparison unit 25.

  The comparison unit 25 acquires the weighting factor output from the weighting factor determination unit 27. In the first modification of the first embodiment, the comparison unit 25 is a coefficient for multiplying the eye opening / closing ratio y in step S26 (see FIG. 7) by a coefficient obtained by multiplying the predetermined correction allowable ratio described above by a weighting coefficient. Used as

  The weighting factor determination unit 27 may output the determined weighting factor to the correction unit 26 instead of the comparison unit 25. In this case, the correction unit 26 uses a coefficient obtained by multiplying the predetermined enlargement ratio described above by a weighting coefficient as a coefficient for further expanding the eye opening / closing ratio of the image data in step S27 (see FIG. 7). In FIG. 17, an arrow from the weight coefficient determination unit 27 to the correction unit 26 is not shown.

  As described above, according to the imaging device 1d of the first modification of the first embodiment, the correction allowable rate or the enlargement rate at the time of comparing or correcting the eye opening / closing rate is further increased according to the browsing time of image data such as a photograph. The eye opening / closing rate of a subject person that can be increased and has high visibility can be corrected more preferably and effectively.

(Modification 2 of the first embodiment)
FIG. 18 is a block diagram illustrating a hardware configuration of an imaging apparatus 1e according to the second modification of the first embodiment. In the second modification of the first embodiment, the imaging device 1e includes a control unit 61, a communication unit 62, an operation unit 12, a storage unit 13, an imaging unit 14, a display control unit 15, a display unit 16, a RAM 17, and a ROM 18. It is a configuration. 18 that operate in the same manner as the corresponding units in FIG. 2 are denoted by the same reference numerals. In the operation of each unit in FIG. 18, the description of the same operation as the corresponding unit in FIG. 2 will be omitted, and different contents will be described.

  In the second modification of the first embodiment, the control unit 61 sets the correction allowable rate μ at the time of comparison by the comparison unit 25 according to the total number of times of transmission of image data such as photographs stored in the storage unit 13. The weighting coefficient for making it variable is determined. The total number of transmissions of image data such as photographs is stored in the storage unit 13 for each image data.

  The communication unit 62 includes a wireless communication circuit and an antenna that can control communication of information or data with a base station device (not shown), and transmits image data output from the control unit 61, for example. When the image data is transmitted, the communication unit 62 outputs the image ID of the image data and the fact that the image data has been transmitted to the weight coefficient determination unit 27e (see later) of the control unit 61. The communication unit 62 may be configured by the above-described wireless communication circuit and antenna element.

  FIG. 19 is a block diagram illustrating in detail the internal configuration of the control unit 61 of the imaging apparatus 1e according to the second modification of the first embodiment. As shown in FIG. 19, the control unit 61 includes an imaging control unit 21, a face detection unit 22, an eye open / close rate determination unit 23, an eye open / close rate management unit 24, a comparison unit 25, a correction unit 26, and a weight coefficient determination unit 27e. It is the composition which includes. 19 that operate in the same manner as the corresponding units in FIG. 3 are denoted by the same reference numerals. In the operation of each part in FIG. 19, the description of the same operation as each corresponding part in FIG. 3 is omitted, and different contents will be described.

  The weighting factor determination unit 27e acquires that the image ID and the image data have been transmitted from the communication unit 62, and determines the number of transmissions of the image data stored in association with the image ID from the storage unit 13 based on the image ID. read out. The weighting coefficient determination unit 27e adds 1 to the read number of transmissions to calculate the total number of transmissions.

  The weighting factor determination unit 27e uses a table (not shown) in which the total number of transmissions and the weighting factor are associated with each other, for example, determines that the weighting factor is 2.0 when the total number of transmissions is 30. . Further, for example, the weighting factor determination unit 27e determines that the weighting factor is 3.0 when the total number of transmissions is 60. The weighting factor determination unit 27e outputs the determined weighting factor to the comparison unit 25.

  The comparison unit 25 acquires the weighting factor output from the weighting factor determination unit 27e. In Modification 2 of the first embodiment, the comparison unit 25 is a coefficient for multiplying the eye opening / closing ratio y in step S26 (see FIG. 7) by a coefficient obtained by multiplying the predetermined correction allowance ratio described above by a weighting coefficient. Used as

  Further, the weighting factor determination unit 27e may output the determined weighting factor to the correction unit 26 instead of the comparison unit 25. In this case, the correction unit 26 uses a coefficient obtained by multiplying the predetermined enlargement ratio described above by a weighting coefficient as a coefficient for further expanding the eye opening / closing ratio of the image data in step S27 (see FIG. 7). In FIG. 19, an arrow from the weight coefficient determination unit 27e to the correction unit 26 is not shown.

  As described above, according to the imaging device 1e of the second modification of the first embodiment, the correction allowable rate or the enlargement rate at the time of comparing or correcting the eye opening / closing rate is further increased according to the number of times image data such as a photograph is transmitted. The eye opening / closing rate of a subject person having high browsing ability that can be increased and that is also requested to be read by an acquaintance other than the user can be corrected more preferably and effectively.

(Modification 3 of the first embodiment)
FIG. 20 is a block diagram illustrating an internal configuration of the control unit 71 of the imaging apparatus according to the third modification of the first embodiment. In Modification 3 of the first embodiment, the hardware configuration of the imaging apparatus is the same as the hardware configuration of the imaging apparatus 1 of the first embodiment, and thus description thereof is omitted.

  In the third modification of the first embodiment, when the subject person of the image data has been edited previously by the user's operation, for example, the control unit 71 is not the image data input to the imaging control unit 21 at the time of preview. Using the previously edited image data, the eye open / close rate of each subject person in the edited image data is determined. It is assumed that the edited image data is stored in the storage unit 13.

  As shown in FIG. 20, the control unit 71 includes an imaging control unit 21, a face detection unit 22, an eye open / close rate determination unit 23, an eye open / close rate management unit 24, a comparison unit 25, a correction unit 26, and an image editing unit 28. It is a configuration. 20 that operate in the same manner as the corresponding units in FIG. 3 are denoted by the same reference numerals. In the operation of each unit in FIG. 20, the description of the same operation as each corresponding unit in FIG. 3 will be omitted, and different contents will be described.

  The image editing unit 28 acquires an operation signal output from the operation unit 12 in response to a user operation, for example, an editing operation of a subject person of image data, and edits image data to be edited. The image editing unit 28 stores the edited image data in the storage unit 13. For example, when the image editing unit 28 acquires from the operation unit 12 or the imaging control unit 21 that a group photo of A, B, and C has been captured, the group photo of A, B, and C is edited. The obtained image data is acquired from the storage unit 13. The image editing unit 28 outputs the edited image data and face detection instruction to the face detection unit 22.

  As described above, the imaging apparatus according to the third modification of the first embodiment uses the eye opening / closing rate for the edited image data in which the user's own preference is reflected in the comparison immediately after imaging (see step S26 in FIG. 7). The correction allowable rate can be made variable.

  While various embodiments have been described above with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It is obvious for those skilled in the art that variations and modifications of various embodiments, and combinations of various embodiments can be conceived within the scope of the claims. Of course, it is understood that it belongs to the technical scope of the present invention.

  Note that the image data stored in the storage unit in each of the above-described embodiments may further include user preference information for the image data as bibliographic information. When the user performs an operation such as increasing or decreasing the favorite degree of bibliographic information in the imaging apparatus, the imaging apparatus is stored in association with the favorite degree in accordance with the increase or decrease value of the favorite degree. Increase or decrease the eye open / close rate of each subject person in the image data. The relationship between the increase or decrease value of the favorite degree and the eye opening / closing rate of each subject person in the image data is preferably defined in advance in a table or the like.

  Note that when a user operation is performed to delete image data stored in the storage unit in each of the above-described embodiments, the imaging apparatus stores information indicating that the image data has been deleted and the content of the image data. . Furthermore, when image data having the same content as the deleted image data is input again, the imaging device, based on the fact that the image data having the same content stored in the storage unit is deleted, In the comparison of the eye open / close rate of each subject person in the image data, the correction allowable rate may be decreased.

  Further, in each of the embodiments described above, the eye opening / closing rate information in which the face ID and the eye opening / closing rate are associated is stored in the storage unit, but the imaging device deletes the eye opening / closing rate information after a predetermined time has elapsed. You may do it. In this case, for example, in the first embodiment, the imaging apparatus 1 directly compares the eye opening / closing rates determined at the time of preview and immediately after the end of imaging. For example, even when the eye opening / closing rate of the subject person is a half-eye state at the time of preview and immediately after the end of imaging, the imaging apparatus 1 can capture images in a manner in which the intention of the subject person is respected.

  The present invention provides an imaging apparatus, a moving image reproducing apparatus, and an imaging method that can effectively correct the opening / closing ratio of the subject person's eyes when the opening / closing ratio of the subject person's eyes is inappropriate in the captured image data. Useful.

1, 1c, 1d, 1e Imaging device 1b Movie playback device 11, 51, 61, 71 Control unit 12 Operation unit 13 Storage unit 14 Imaging unit 15 Display control unit 16 Display unit 17 RAM
18 ROM
DESCRIPTION OF SYMBOLS 21 Imaging control part 22 Face detection part 23 Eye opening / closing rate determination part 24 Eye opening / closing rate management part 25 Comparison part 26 Correction | amendment part 27, 27e Weight coefficient determination part 28 Image editing part 31 Movie reproduction control part 32 Audio | voice output part 41 Image composition part 52 Timer 62 Communication Unit

Claims (14)

A face detection unit for detecting the face of the subject person in the input image data;
An eye opening / closing rate determination unit that determines an eye opening / closing rate of the eye of the subject person based on the face detected by the face detection unit in the image data;
A storage unit that stores eye opening / closing rate information including an eye opening / closing rate of the eye of the subject person determined by the eye opening / closing rate determination unit;
The eye opening / closing rate of the subject person determined by the eye opening / closing rate determining unit is compared with the eye opening / closing rate obtained by multiplying the eye opening / closing rate in the stored eye opening / closing rate information by a predetermined correction allowable rate. A comparison unit to
An imaging apparatus comprising: a correction unit that corrects the eye opening / closing rate determined by the eye opening / closing rate determination unit according to a comparison result of the comparison unit. The imaging apparatus according to claim 1,
The comparison unit compares the eye open / close rate of the subject person determined by the eye open / close rate determination unit with the eye open / close rate in the eye open / close rate information stored in the storage unit,
When the determined eye opening / closing rate of the subject person's eye is larger than the eye opening / closing rate in the eye opening / closing rate information, the eye opening / closing rate in the eye opening / closing rate information is set to the eye opening / closing rate of the determined subject person's eye An imaging device further comprising: an eye opening / closing rate management unit that updates the rate. The imaging apparatus according to claim 1 or 2,
The image data is input at the time of previewing or after imaging of the subject person,
The correction unit uses a predetermined enlargement ratio when the determined eye open / close rate during the preview is smaller than the eye open / close rate obtained by multiplying the determined eye open / close rate after the imaging by the predetermined correction allowance rate. An imaging device that corrects the determined eye open / close rate after the imaging. A video playback device capable of synchronously playing back image data and audio data,
A face detection unit for detecting the face of the subject person in the image data input during the video playback or immediately after the pause operation in the video playback device;
An eye opening / closing rate determination unit that determines an eye opening / closing rate of the eye of the subject person based on the face detected by the face detection unit in the image data;
A storage unit that stores eye opening / closing rate information including an eye opening / closing rate of the eye of the subject person determined by the eye opening / closing rate determination unit;
The eye opening / closing rate of the subject person determined by the eye opening / closing rate determining unit is compared with the eye opening / closing rate obtained by multiplying the eye opening / closing rate in the stored eye opening / closing rate information by a predetermined correction allowable rate. A comparison unit to
A correction unit that corrects the eye open / close rate determined by the eye open / close rate determination unit according to the comparison result of the comparison unit, and
The correction unit has a predetermined eye opening rate when the determined eye opening rate is less than an eye opening rate obtained by multiplying the determined eye opening rate immediately after the pause operation by the predetermined correction allowable rate immediately after the pause operation. A moving image reproducing apparatus that corrects the determined eye opening / closing ratio immediately after the pause operation using an enlargement ratio. The imaging apparatus according to claim 1 or 2,
An image composition unit that generates 3D display image data based on the left-eye image data and the right-eye image data;
The image data is input after imaging the left-eye image data of the subject person or after imaging the right-eye image data of the subject person,
The correction unit determines the determination in the right-eye image data when the determined eye opening / closing rate after capturing the right-eye image data is smaller than the determined eye opening / closing rate after capturing the left-eye image data. An imaging device that corrects the eye opening / closing rate to be the same as the determined eye opening / closing rate in the left-eye image data. The imaging apparatus according to claim 2,
A timer for measuring the browsing time of the input image data;
A weighting factor determination unit that determines a weighting factor by which the correction allowable rate is multiplied based on the browsing time of the image data measured by the timer;
The comparison unit determines the eye open / close rate of the subject person determined by the eye open / close rate determination unit and the eye open / close rate in the stored eye open / close rate information with a predetermined correction allowable rate and the determination. An imaging device that compares an eye opening / closing rate multiplied by a weighting factor. The imaging apparatus according to claim 2,
A communication unit for transmitting the input image data;
A weighting factor determination unit that determines a weighting factor by which the correction allowable rate is multiplied based on the number of transmissions of the image data transmitted by the communication unit;
The comparison unit determines the eye open / close rate of the subject person determined by the eye open / close rate determination unit and the eye open / close rate in the stored eye open / close rate information with a predetermined correction allowable rate and the determination. An imaging device that compares an eye opening / closing rate multiplied by a weighting factor. The imaging apparatus according to claim 2,
An image editing unit that edits the eye open / close rate of the subject person in the image data stored in the storage unit,
The eye open / close rate management unit stores the eye open / close rate of the subject person of the image data edited by the image editing unit in the storage unit,
The imaging unit corrects the eye opening / closing rate determined by the eye opening / closing rate determination unit for the input image data to the stored eye opening / closing rate. The imaging apparatus according to claim 2,
An operation unit that receives a user operation on the imaging device;
The eye opening / closing rate management unit updates the eye opening / closing rate in the eye opening / closing rate information according to the user operation received by the operation unit. It is an imaging device as described in any one of Claims 1-3 and 5-9,
The eye opening / closing ratio is a value obtained by dividing the vertical height of the eyes of the subject person of the input image data by the height of the face of the subject person of the image data in the vertical direction. . It is an imaging device as described in any one of Claims 1-3 and 5-9,
The eye opening / closing ratio is an image pickup device that is a value obtained by dividing a vertical height of eyes of the subject person of the input image data by a horizontal width of the face of the subject person of the image data. It is an imaging device as described in any one of Claims 1-3 and 5-9,
The eye opening / closing ratio is a value obtained by dividing a vertical height of eyes of the subject person of the input image data by a horizontal width of the eyes of the subject person of the image data. It is an imaging device as described in any one of Claims 1-3 and 5-9,
The eye opening / closing ratio is an image pickup apparatus that is a value obtained by dividing a vertical height of eyes of the subject person of the input image data by a distance between both eyes of the subject person of the image data. Detecting the face of the subject person in the input image data;
Determining an eye open / close rate of the subject person based on the detected face of the image data;
Storing, in a storage unit, eye opening / closing rate information including an eye opening / closing rate of the eye of the determined subject person;
Comparing the determined eye opening / closing rate of the subject person with an eye opening / closing rate obtained by multiplying the eye opening / closing rate in the eye opening / closing rate information stored in the storage unit by a predetermined correction allowable rate; ,
Correcting the determined eye opening / closing rate in accordance with the comparison result.

Images