JP2015111777A - Imaging device and imaging device control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable display of a subject relevant to an audience by performing optimal image processing on the subject.SOLUTION: An imaging device comprises: display means for displaying an image on a display part; imaging means provided on the same side with the display part; personal authentication means for authenticating a subject from face image information; relationship recording means for recording information of another subject related to each subject; and image processing means for image processing an imaged image. When an image is displayed on the display part, the personal authentication means authenticates a first subject from first face image information imaged by the imaging means and authenticates a second subject from second face image information of an image displayed on the display part, and when it is determined that the second subject is the first subject or a subject related to the first subject extracted from the relationship recording means, the display means displays an image subjected to image processing suitable for the second subject, which is performed by the image processing means.

Description

  The present invention relates to an imaging apparatus and an imaging apparatus control method, and more particularly, to a technique suitable for use in an imaging apparatus using an authentication technique.

  In recent years, digital cameras and camera-equipped mobile phones have become widespread, and captured images are stored in a memory or the like as digital image data. Therefore, there is no need to worry about the remaining amount of film unlike a conventional film type camera, and a large number of images can be easily taken. Digital image data shot in this way is used not only for printing and appreciation, but also for enjoying in a slide show in which shot still images are played back sequentially (see Patent Document 1).

  Here, in Patent Document 2, since it takes time to view all of the large amount of image data stored in the memory of the slide show, a method for solving the problem is proposed. In other words, an image that a viewer wants to view is obtained by photographing the face of a person to be viewed by a camera or the like, and extracting image data including the same face information as the face information from image data stored in a memory. Only the data can be easily extracted and the slide show can be played.

  Further, an imaging apparatus having an authentication processing function such as face authentication or personal authentication has also been proposed (see Patent Document 3). In an imaging device having an authentication processing function such as face authentication or personal authentication, a subject image region including a specific subject region is detected from image data obtained by photographing with the imaging device. Feature information for identifying the subject area as the authentication target is extracted from the image data of the detected subject image area, and the extracted feature information is compared with the feature information of the registration target image registered in the database in advance. To do.

  Finally, the degree of similarity is determined, and based on the degree of similarity of the determination result, authentication is performed regarding which registration target is to be authenticated or whether there is no corresponding registration target. As an example of such a configuration, a face of a person is detected as a specific subject area from image data obtained by photographing, and feature information of the face is extracted to identify the person who is the face. There is a system to do.

JP 2000-293974 A JP 2006-318352 A JP 2010-27034 A JP 2000-030065 A JP 2004-038531 A

  However, in the method described in Patent Document 2, only an image in which the viewer himself / herself is captured is extracted. For this reason, for example, when viewing an image taken when traveling with a plurality of friends, the viewer does not want to see only the image of himself / herself, but also an image of a friend It is natural to want to see.

In addition, for example, in an image shot with a light field camera that can change the focus position after shooting, there is a problem that the subject that the viewer wants to see in the extracted image may not be in focus in advance. .
SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and an object of the present invention is to display images by performing optimum image processing on a viewer or a subject related to the viewer.

  The imaging apparatus according to the present invention includes a display unit that displays an image on a display unit, a photographing unit provided on the same side as the display unit, a personal authentication unit that authenticates a subject from face image information, and a relationship for each subject. And a relationship recording unit that records information on another subject having an image, and an image processing unit that performs image processing on the captured image, and when displaying the image on the display unit, the personal authentication unit includes: The first subject is authenticated from the first face image information photographed by the photographing means, the second subject is authenticated from the second face image information of the image displayed on the display unit, and the second subject is identified by the first face image information. When it is determined that the subject is related to the first subject extracted from the relationship recording unit, the display unit uses the processing unit to select an image suitable for the second subject. To display the processed image. And butterflies.

  According to the present invention, it is possible to display an image showing a viewer or a subject related to the viewer in a situation suitable for the subject.

It is a block diagram which shows the structural example of the imaging device in embodiment of this invention. It is a block diagram which shows schematic structure of the face detection and personal authentication part of embodiment. It is the figure which showed an example of the arrangement | positioning relationship of the imaging | photography part and display part of embodiment. It is a figure explaining a mode that the display part of embodiment is touched and a relationship is constructed | assembled. It is the figure which showed the relationship between the photographer and subject of embodiment. It is a flowchart explaining the process sequence of the relationship construction in embodiment. It is a block diagram which shows schematic structure of the relationship construction part of embodiment. It is a flowchart explaining the process sequence of the image reproduction in embodiment. It is a figure explaining the relationship between the browser of embodiment and the image displayed on a display part.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
(First embodiment)
In the first embodiment, the viewer who is browsing the display unit is personally authenticated by the imaging unit provided on the same side as the display unit that displays the image, and the subject in the image displayed on the display unit is identified. An example of an imaging device for personal authentication will be described. A subject related to the viewer is detected from the image using the recorded relationship between the subjects, and image processing corresponding to the detected person is performed.

FIG. 1 is a block diagram showing the overall configuration of the imaging apparatus 101 according to the first embodiment of the present invention.
The imaging device 101 is used to operate the first imaging unit 102, the second imaging unit 103, the control unit 116, the recording unit 117, the display unit 118, the relationship building unit 119, the image processing unit 120, and the imaging device 101. And an operation unit such as various operation switches (SW121).

  The control unit 116 controls the entire apparatus such as electrical and electronic operation control of the imaging apparatus 101 and control of various processes. The first imaging unit 102 photoelectrically converts the light that forms the subject image by the imaging lens 104 by the imaging element 105. Further, the analog signal output from the image sensor 105 is output from an analog signal processing unit 106 having a CDS circuit for removing output noise, a preprocessing circuit having an amplification circuit, and the like, and is digitally output by an A / D conversion unit 107. Convert to signal. Thereafter, the digital signal processing unit 108 performs signal processing such as sensor correction such as defective pixel correction and smear correction, chromatic aberration correction, and demosaic processing to generate image data.

  The display unit 118 is provided on the side opposite to the surface on which the imaging lens 104 is provided. In the shooting mode, the display unit 118 displays an image formed on the image sensor 105 in real time. Therefore, the photographer can confirm the state of the image to be photographed by observing the display image displayed on the display unit 118 of the imaging device 101 with the imaging lens 104 facing the subject. The display unit 118 can also be used as a viewfinder of the imaging apparatus 101.

  The imaging lens 110 of the second imaging unit 103 is provided on the surface of the imaging device 101 where the display unit 118 is provided. The optical axis direction of the imaging lens 110 is substantially opposite to the optical axis direction of the imaging lens 104. Therefore, when the imaging apparatus 101 is held as described above in order to capture a subject, an image of the photographer is formed on the imaging element 111 by the imaging lens 110 of the second imaging unit 103. Then, an analog signal is output from an analog signal processing unit 112 having a CDS circuit for removing output noise, a preprocessing circuit having an amplifier circuit, and the like, and is converted into a digital signal by the A / D conversion unit 113. Thereafter, the digital signal processing unit 114 performs image processing by performing signal processing such as sensor correction such as defective pixel correction and smear correction, chromatic aberration correction, and demosaic processing.

The optical axis direction and imaging field of view of the imaging lens 110 are set so that the photographer's face is photographed when the imaging apparatus 101 is held for photographing by the first photographing unit 102.
Since the image photographed by the second photographing unit 103 is not for viewing and it is sufficient that the person shown in the image can be identified, the resolution and optical performance of the second photographing unit 103 are the same as those of the first photographing unit 102. Not as good as resolution or optical performance. The image size may be small.

  Further, the image processing program of the present embodiment is a software implementation of each image processing function in the imaging apparatus 101. In this case, the software is composed of a program group for causing each computer to execute each image processing function. Each program may be used by being incorporated in advance in dedicated hardware, for example, or installed in a general-purpose personal computer or the like from a network or a recording medium.

  FIG. 3 is a diagram illustrating the positional relationship among the first imaging unit 102, the second imaging unit 103, and the display unit 118 in the imaging apparatus 101. The first imaging unit 102 is provided on the surface opposite to the display unit 118, and the second imaging unit 103 is provided on the same surface as the display unit 118.

  The face detection and personal authentication unit 109 performs face detection and personal authentication described later on the image data generated as described above, and outputs an authentication result from the personal authentication unit to the control unit 116. Although not shown, the control unit 116 again controls exposure control, white balance processing, distance measurement control, and the like using the authentication result.

  For example, the exposure control is started when the SW 121 is pressed halfway, and the brightness of the subject is detected from the image signal obtained by the image sensor 105 so that the desired subject can be imaged with higher quality. Adjust the brightness. When the SW 121 is fully pressed, the image data output from the digital signal processing unit 108 is recorded in the recording unit 117.

FIG. 2 is a block diagram showing a schematic configuration of the face detection and personal authentication unit 109 of FIG.
In FIG. 2, the face detection and personal authentication unit 109 includes a face detection unit 201, a personal authentication unit 202, and a registered face image data recording unit 203. The face detection and personal authentication unit 115 has the same configuration.

  For example, the face detection unit 201 scans the input image data with a template prepared for face detection, and calculates a correlation value at each position of the input image data. And the place where this correlation value showed the high value is determined as a face area. As another method, a face region extraction method using a subspace method or the like may be used.

  The personal authentication unit 202 calculates the similarity between the face image data detected from the input image data by the face detection unit 201 and the face image data registered in the registered face image data recording unit 203. As a method for calculating the similarity, for example, as disclosed in Patent Document 4, after normalizing the size, angle, brightness, etc. of the input face image data, a method using the KL expansion method of the Fourier spectrum is used. is there.

Alternatively, as disclosed in Patent Document 5, there is a technique in which detected face image data is normalized according to the distance to the face and matched. Note that the registered face image data defined in the present invention is face image data itself that is compressed and stored, or information indicating a feature amount related to a face such as eyes, nose, mouth, eyebrows, and the like.
The personal authentication unit 202 performs personal authentication for authenticating the subject from the face image information calculated as described above.

  The registered face image data recording unit 203 registers a plurality of registered face image data in advance at the time of shooting, EVF display, playback, or the like. Each registered face image data is associated with the personal name (registered name) of the person. Information indicating the feature amount related to the face is output to the control unit 116, and the control unit 116 outputs the information to the relationship construction unit 119 at a desired timing.

FIG. 7 is a block diagram showing a schematic configuration of the relationship construction unit 119 of FIG.
Information indicating the feature amount related to the face output from the face detection and personal authentication unit 109 (115) to the control unit 116 includes the personal authentication result temporary storage unit 701 by the first photographing unit of the relationship construction unit 119 and the second information. Each of them is temporarily stored in a personal authentication result temporary storage unit 702 by the imaging unit.

  The information indicating the feature amount related to the temporarily stored face is output to the personal authentication result associating unit 703 in response to an instruction from the display unit 118, and the first photographing is performed based on the information indicating the feature value related to each face. The relationship between the subject of the part and the subject of the second photographing part is constructed. The construction of the relationship will be described later with reference to FIG.

The constructed relationship is output to the relationship construction data recording unit 704, which is a relationship construction database, and a relationship recording process for recording information on another subject that is relevant for each subject is performed and stored. .
Further, since it is not necessary to save the relationship once constructed, it is determined whether the relationship has already been constructed, and an existing relationship flag for judging whether the relationship has been constructed is output to the control unit 116. To do. By looking at the existing relationship flag described above, the control unit 116 determines whether the display unit 118 displays / hides a registration button 401 shown in FIG. 4 to be described later, and if it is registered, it is not displayed or registered. If it is, the control unit 116 determines to display it.

FIG. 4 shows an image captured by the first photographing unit in the imaging device 101 on the display unit 118 for face detection / individual detection, and a second photographing unit for face detection / individual detection. FIG.
Here, as a result of the face detection, there are two subjects and a registration button 401 is displayed on the display unit 118.

In this embodiment, the registration button 401 is used. However, the photographer may touch an icon or an image different from the registration button 401 for a desired subject. Further, the photographer may be able to instruct a desired subject by operating the cross key.
In addition, as described above, the display unit 118 is a touch panel display device, and the subject (photographer) of the second imaging unit touches the registration button 401 to show that the mutual relationship is established. Yes.

  Further, when the display unit 118 receives a touch by the photographer and the touched target is the registration button 401, the display unit 118 sends a signal for performing the relationship construction to the control unit 116 in order to construct the relationship.

Here, as a result of individual detection, the subjects displayed on the display unit 118 are “Mr. A” and “Mr. B”, respectively, and the photographer photographed by the second photographing unit 103 is “Mr. C”. Here, when the photographer “Mr. C” presses the registration button 401 of “Mr. A” and “Mr. B”,
“Mr. C” ⇔ “Mr. A”
“Mr. C” ⇔ “Mr. B”
Record the relationship.

Here, “Mr. C” pressed the “Mr. A” registration button 401,
This shows that the relationship between “Mr. A” and “Mr. C” has been established. But,
“Mr. C” ⇒ “Mr. A”
“Mr. C” ⇒ “Mr. B”
In this way, when viewed from the subject (photographer) of the second photographing unit 103, the relationship from only one side may be recorded such that it is related to the subject of the first photographing unit 102.

FIG. 5 is a diagram for explaining the aforementioned relationship. Reference numeral 501 denotes information on a subject (photographer / browser) shown in the second photographing unit 103, and reference numeral 502 denotes information about a subject shown in the first photographing unit 102.
“Mr. C” Ａ “Mr. A”
When the relationship is established, the cell where “C” in 501 and “A” in 502 intersect is “1”, and the flag is on. By recording this flag in the relationship construction data recording unit 704, the relationship between the subject of the first photographing unit 102 and the subject (photographer / viewer) of the second photographing unit 103 is constructed.

FIG. 6 is a flowchart illustrating a processing procedure when the imaging apparatus 101 constructs a relationship between the subject of the first imaging unit 102 and the subject of the second imaging unit 103. This process is realized by developing a program recorded in the nonvolatile memory in the system memory and executing it by the control unit 116.
When the power of the imaging apparatus 101 is turned on, the EVF display state of S601 is entered. Although the EVF display state is set in S601, this is an example, and in order to realize the object of the present invention with the same flow, it may be at the time of reproduction or shooting.

  Next, in S602, it is determined whether or not the user has selected a mode called a relationship construction mode. When the user selects, a registration button 401 is displayed as shown in FIG. If the mode is not the relationship construction mode, the process proceeds to S609, where EVF display is performed in the same manner as the conventional imaging apparatus, and face detection and personal authentication in the second imaging unit 103 are not required.

When the relationship construction mode is selected, the process proceeds to S603 and the face of the subject photographed by the first photographing unit 102 is detected. Thereafter, in S604, personal authentication of the subject photographed by the first photographing unit 102 is performed.
Next, in step S605, the face of the subject photographed by the second photographing unit 103 is detected, and then in step S606, personal authentication of the subject photographed by the second photographing unit 103 is performed.

  Next, a subject registration button of the first photographing unit 102 in which the subject of the second photographing unit 103 (the photographer when EVF is displayed and the viewer when reproducing) is displayed on the touch panel type display unit 118 is displayed. It is determined whether or not 401 is touched (S607). If it is not touched here, the relationship is not established, and the process proceeds to S609 to start shooting when the shutter is pressed as in the conventional camera.

  If touched in the determination in S607, the process proceeds to S608, and the relationship between the subject of the second photographing unit 103 and the subject of the first photographing unit 102 is flagged as shown in FIG. Build and build. Since the registration button 401 is deleted from the display when the relationship is established, the subject of the second imaging unit 103 can know that the relationship has been established.

  As described above, the first imaging unit 102 and the second imaging unit 103 are provided, and an image captured by the first imaging unit 102 is displayed on the touch panel display unit 118. Then, the images photographed by the first photographing unit 102 and the second photographing unit 103 are subjected to face detection and personal authentication, respectively, and the subject (photographer, viewer) photographed by the second photographing unit is the first photographing. Touch the subject photographed in the section. In this way, the relationship between the subject (photographer or viewer) of the second imaging unit 103 and the subject of the first imaging unit 102 can be established.

  Here, the imaging device 105 of the first imaging unit 102 acquires light field data in which a plurality of pixels provided with a plurality of sub-pixels that receive light beams that pass through different pupil partial regions of the imaging optical system are arranged. Suppose that In this configuration, when the image shown in FIG. 9 (A) in which the focus position can be changed at the time of reproduction is photographed, Mr. D is in focus at the default focus position, and the subject is related to the viewer C. Mr. E may be out of focus.

Therefore, as described above, when the relationship between the subjects is established and the flag is recorded in the relationship construction data recording unit 704 as shown in FIG. An example of processing to be performed will be described.
In this case, the personal authentication unit 202 personally authenticates the subject of the second photographing unit 103 and the subject in the image displayed on the display unit 118, and the image processing unit 120 subjects related to the subject of the second photographing unit 103. Is displayed on the display unit 118.

The aforementioned processing procedure will be described with reference to the flowchart of FIG. This process is realized by developing a program recorded in the nonvolatile memory in the system memory and executing it by the control unit 116.
In step S801, an image playback display mode is selected.
In step S <b> 802, the face detection unit 201 detects the face of the subject on the image to be reproduced on the display unit 118. At that time, if the image to be reproduced is an image with a variable focal position as captured by the first imaging unit 102, if face detection is performed while shifting the focus position, the processing time becomes long. Face detection is performed with an omnifocal image that is in focus on this part.

In step S803, it is determined whether a face is detected in the image to be reproduced. If a face is detected, the process proceeds to S804. If no face is detected, the process proceeds to S810.
In step S <b> 804, the personal authentication unit 202 identifies the subject whose face has been detected in step S <b> 802.
In step S <b> 805, the face detection unit 201 performs face detection on an image obtained by shooting the subject of the second shooting unit 103.

In step S806, it is determined whether the face of the subject of the second photographing unit 103 has been detected. If a face is detected, the process proceeds to S807. If no face is detected, the process proceeds to S810.
In step S807, the personal authentication unit 202 identifies the viewer who has detected the face in step S805.

  In step S808, it is determined whether there is a relationship between the subject personally authenticated in step S804 and the viewer personally authenticated in step S807. Specifically, the subject that is personally authenticated in S804 is included by referring to the subject related to the subject of the second photographing unit 103 from the relationship between the subjects recorded in the relationship construction data recording unit 704. It is determined whether or not.

If it is determined that there is a relationship with the subject of the second photographing unit 103, the process proceeds to S809, and if it is determined that there is no relationship, the process proceeds to S810.
Note that here, the imaging apparatus 101 constructs the relationship between the subjects and refers to the relationship building data recording unit 704. However, the imaging for acquiring the relationship building data from the outside without having the relationship building unit 119. The processing of this flowchart can also be realized in the apparatus.

  In S809, the subject personally authenticated in S804 is related to the subject in the second photographing unit 103 and is considered to be a part that is particularly desired to be viewed in the reproduced image. Adjust the focus. As a result, as shown in FIG. 9B, an image focused on the subject E who is related to the viewer C is reconstructed.

If the face is not detected in the image to be reproduced in S810, or if it is determined that there is no relationship between the subject in the second photographing unit and the subject in the image to be reproduced, the focus position is determined by the image processing unit. At 120, the default position is set.
After the processing of S809 or S810, the reproduced image with the focus position adjusted is displayed on the display unit 118 in S811.

In this embodiment, the playback display mode is set in step S801. However, this is merely an example, and it is possible to realize the object of the present invention with the same flow not only for images to be played but also for EVF display during shooting. It is.
If the relationship “Mr. C” and “Mr. D” is newly established in the relationship construction mode, a plurality of subjects such as D and E are related to the viewer C in S808. It will be determined as. At that time, the image processing unit 120 controls the depth of field so that both Mr. D and Mr. E are in focus, and reconstructs an image as shown in FIG. The purpose can be realized.

  Furthermore, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist. The embodiment in the case of adjusting the focus position and the depth of field has been described as an example of the image processing. However, the present invention is not limited to the above-described method. Is the same.

  As described above, the relationship between the subjects is established, and the image captured by the first imaging unit and the image captured by the second imaging unit displayed on the display device are face-detected and personally authenticated, respectively, and the second A subject related to the subject photographed by the photographing unit is detected from an image displayed on the display device. Then, by performing optimal image processing for the subject and playing it back, you can not only play back the image that you see, but also the subject that is highly relevant to you. It is possible to reproduce an image that is optimal for viewing the subject.

(Other embodiments)
Although the present invention has been described in detail based on the preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms without departing from the gist of the present invention are also included in the present invention. included.

  In the above-described embodiment, the case where the present invention is applied to an imaging apparatus has been described as an example. However, this is not limited to this example, and an apparatus that displays an image recorded in a recording unit on a display unit. Applicable if available.

  For example, the present invention can be applied when image processing is performed on an image displayed on a display unit in a smartphone or a tablet PC. Therefore, the present invention is applicable to personal computers, PDAs, mobile phone terminals, and the like.

  The present invention can also be realized by executing the following processing. That is, software (computer program) that implements the functions of the above-described embodiments is supplied to a system or apparatus via a network or various computer-readable storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program.

DESCRIPTION OF SYMBOLS 101 Imaging device 102 1st imaging | photography part 103 2nd imaging | photography part 104,110 Imaging lens 105,111 Image pick-up element 106,112 Analog signal processing part 107,113 A / D conversion part 108,114 Digital signal processing part 109,115 Face detection and personal authentication unit 116 Control unit 117 Recording unit 118 Display unit 119 Relationship construction unit 120 Image processing unit 121 SW
201 face detection unit 202 personal authentication unit 203 registered face image data recording unit

Claims (8)

Display means for displaying an image on the display unit;
Photographing means provided on the same side as the display unit;
Personal authentication means for authenticating the subject from the face image information;
A relationship recording means for recording information on another subject that is relevant for each subject;
Image processing means for image processing the captured image, and when displaying the image on the display unit,
The personal authentication unit authenticates the first subject from the first face image information photographed by the photographing unit, and the second subject from the second face image information of the image displayed on the display unit,
When it is determined that the second subject is the first subject or a subject related to the first subject extracted from the relationship recording means, the display means is processed by the processing means. An image pickup apparatus that displays an image subjected to image processing suitable for the second subject. A second imaging unit provided on the opposite side of the display unit for capturing an image to be displayed on the display unit;
Instruction means for the first subject to point to the second subject;
A relationship building means for building a relationship between the second subject instructed by the instruction means and the first subject instructed;
The imaging apparatus according to claim 1, further comprising: The image processing means includes
When displaying an image whose depth of field can be changed on the display unit, the second subject is related to the first subject or the first subject extracted from the relationship recording means. The imaging apparatus according to claim 1, wherein when it is determined that the subject is a certain subject, the second subject is focused. The image processing means includes
When displaying an image whose depth of field can be changed on the display unit, the second subject is related to the first subject or the first subject extracted from the relationship recording means. The imaging apparatus according to claim 1, wherein when it is determined that the subject is a certain subject, the depth of field is changed so that the plurality of subjects are focused. The image processing means includes
When the image is displayed on the display unit, the second subject is determined to be the first subject or a subject related to the first subject extracted from the relationship recording unit. 3. The imaging apparatus according to claim 1, wherein the brightness is adjusted to a brightness or color suitable for the second subject. A display step of displaying an image on the display unit;
A photographing process provided on the same side as the display unit;
A personal authentication process for authenticating the subject from the face image information;
A relationship recording step of recording information on another subject that is relevant for each subject;
An image processing step for image processing of the captured image, and when displaying the image on the display unit,
The personal authentication step authenticates the first subject from the first face image information photographed in the photographing step, and the second subject from the second face image information of the image displayed on the display unit,
When it is determined that the second subject is the first subject or a subject related to the first subject extracted in the relationship recording step, the display step is performed in the processing step. An image pickup apparatus control method, comprising: displaying an image subjected to image processing suitable for the second subject. A display step of displaying an image on the display unit;
A photographing process provided on the same side as the display unit;
A personal authentication process for authenticating the subject from the face image information;
A relationship recording step of recording information on another subject that is relevant for each subject;
Causing the computer to execute an image processing step for image processing of the captured image;
When displaying an image on the display unit,
The personal authentication step authenticates the first subject from the first face image information photographed in the photographing step, and the second subject from the second face image information of the image displayed on the display unit,
When it is determined that the second subject is the first subject or a subject related to the first subject extracted in the relationship recording step, the display step is performed in the processing step. A program for controlling a computer to display an image subjected to image processing suitable for the second subject.   A computer-readable storage medium storing the program according to claim 7.

Images