JP4038817B2 - Imaging device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an imaging apparatus. Specifically, the subject is photographed at the same time as the photographer's eyes, the movement of the photographed pupil is observed, and the subject is focused. The present invention relates to an imaging apparatus.
[0002]
[Prior art]
In conventional imaging devices such as digital still cameras and video cameras, buttons and switches are always attached to the exterior when the user changes or implements settings for function items such as shooting and playback. , It has been operated as an input device.
However, in recent years, digital still cameras and video cameras have been miniaturized, and as a result, buttons and switches have been increasingly eliminated.
As a substitute for the lost button or switch, a method of assisting the operation by using a human gaze or voice has been adopted from the use of the present pressing type display device.
In particular, the method using the line of sight can be operated with almost no buttons or switches. In shooting using sound, the subject can be operated while following the subject with natural movement.
Moreover, in the present line-of-sight input, generally, a user or an electronic viewfinder is looked into, the LED is irradiated to the looked-in state, and the position of the pupil is detected from the reflection and the position is specified. It is.
[0003]
[Patent Document 1]
Japanese Patent Laid-Open No. 04-96580 (Page 2, Fig. 1)
[0004]
[Problems to be solved by the invention]
However, in the method of looking into the electronic viewfinder described in the prior art and detecting the position of the pupil by irradiating the LED to identify the position, the user always needs to look into the electronic viewfinder. There is a problem that a large-screen display device, which is often used in recent digital still cameras and video cameras, cannot be used, and an electronic viewfinder is required as a device, which is an obstacle to downsizing. is there.
Therefore, there is a problem that an imaging device to be miniaturized, for example, a digital still camera or a video camera, has to solve a technique for specifying an imaging function setting desired by a user by pupil movement.
[0005]
[Means for Solving the Problems]
In order to achieve the above object, an imaging apparatus according to the present invention is configured as follows.
[0006]
(1) An imaging apparatus captures a first imaging unit that captures an image of a subject, a display that displays an image captured by the first imaging unit, and an image of a photographer's eyes that are viewing the display. 2, and a gazing point detection unit that detects a gazing point of the photographer's eyes from the eye information captured by the second imaging unit, the gazing point detected by the gazing point detection unit The position information is information by calibration that causes a calibration character to be displayed at a specified position of the display, and aligns a gazing point calculated by looking at the calibration character with a display area of the display. The focus of the subject imaged by the first imaging means is caused to follow the display area watched by the display.
(2) The imaging apparatus according to (1), wherein the position information of the gazing point of the photographer is displayed on the display.
[0008]
In this way, when the subject is imaged, the focus of the subject can be automatically adjusted by the eye movement of the photographer by capturing the eye of the photographer and observing the gazing point.
[0009]
DETAILED DESCRIPTION OF THE INVENTION
Next, an embodiment of an imaging apparatus according to the present invention will be described with reference to the drawings.
[0010]
As shown in FIG. 1, the imaging apparatus according to the present invention has a first lens for forming an image on an imaging element via a first lens unit 11 for capturing a subject, a first shutter 12, and a first iris 13. A CCD unit 14, a first front end unit 15 that converts an image formed into an electrical signal and converts it into a digital signal, a signal processing unit 17 that includes a DSP that performs signal processing of the image signal, and a CPU A central control device 18, a RAM unit 19, a ROM unit 20, a display control unit 21 that controls OSD display, an LCD 22 that corresponds to a display that is a liquid crystal screen, and a memory interface unit that controls a memory card 23. 24, a timing generator 25 that is controlled by the central control unit 18 and generates a clock of an image captured by the first CCD unit 14, and a central control unit 8 and a vertical driver 26 that synchronizes an image captured by the first CCD unit 14 and a lens / iris driver that is controlled by the central control unit 18 and controls the first lens unit 11 and the first iris 13. 27, a power supply control unit 28, a temperature sensor 29, an external operation unit 30 comprising buttons and switches, a first network interface unit 31, and a second network interface unit 32. A second CCD unit 36 for forming an image on the image sensor via the second lens unit 33, the second shutter 34, and the second iris 35 for imaging the photographer's face, and the second CCD unit 36. And a second front end unit 37 for converting the image formed on the digital signal into a digital signal. The signal converted by the second front end unit 37 is a signal. It is sent to the processing unit 17, processing for detecting the gaze point of the eye by detecting the position of the pupil with respect to the subject is performed.
[0011]
Of these, the first lens unit 11, the first shutter 12, the first CCD unit 14 for forming an image on the image sensor via the first focus / iris 13, and the first front end unit 15 are the first ones. A second CCD unit for forming an image on the imaging element through the second lens unit 33, the second shutter 34, and the second iris 35 that form a single imaging unit 16. 36 and a second front end unit 37 that converts an image formed on the second CCD unit 36 into a digital signal forms a second imaging unit 38.
The central control device 18, RAM unit 19, ROM unit 20, display control unit 21, memory interface unit 24, signal processing unit 17, first network interface unit 31, and second network interface unit 32 are connected to the bus line 39. They are connected and controlled by the central controller 18.
[0012]
The signal processing unit 17 detects luminance from the video signal obtained by the first imaging unit 16 and detects the gaze point of the photographer from the eye information captured by the second imaging unit 38. Means for storing luminance information and gaze point information in the SDRAM memory 41 via the SDRAM controller 40.
[0013]
The central control device 18 controls the first focus / iris 13 to follow the focus from the gazing point information obtained by the signal processing unit 17 or the calibrated information, and so on. , Provided with operation mode display means for displaying various operation modes on the LCD 22 with predetermined characters, and gazing at any character based on position information of the gazing point detected by the gazing point detection means of the signal processing unit 17 When it is detected that the operation mode is detected, control is performed so that the corresponding operation mode is set.
[0014]
The operation of the imaging apparatus having such a configuration will be described below.
First, a method for actually detecting a photographer's gazing point will be described below with reference to a flowchart shown in FIG.
First, the user turns on this switch of the external operation unit 30 (step ST11). When the result is received by the central controller 18, the power control unit 28 is controlled with respect to the second CCD unit 36 and the second front end unit 37 to supply power and start imaging (step ST12). ).
[0015]
Image information converted into a digital signal by the first front end unit 15 is input to the signal processing unit 17 from the first CCD unit 14 facing the subject, and the luminance is calculated (step ST13).
[0016]
At the same time, the second CCD unit 36 facing the photographer forms an image of the photographer, particularly a face image, and a second front end unit 37 that directly connects the analog data to the second CCD unit 36. Thus, it is converted into a digital signal and the information is sent to the signal processor 17 (step ST14).
In the signal processing unit 17, the digital signal input from the second front end unit 37 is pasted and stored in the SDRAM memory 41 via the SDRAM controller 40 (step ST15).
[0017]
As shown in FIG. 2, the central control unit 18 performs sharpness and skin color area detection from face information stored in the SDRAM memory 41, specifies a face, and then detects an area with white information (step). ST15).
Then, as shown in FIG. 3, the vertical and horizontal ranges are determined from the detected white information area, and the region Z where the vertical and horizontal portions overlap on the straight line extending from the maximum and minimum points of the range is stored on the SDRAM memory 41. Extracted from the main image and stored in another memory area (steps ST16, ST17, ST18).
[0018]
There are two methods for detecting the actual pupil size from the stored area, ie, color difference and binarization. This time, the binarization method will be described.
The area of the human eye varies in shades of white, iris, and pupil. As shown in FIG. 4, the central control device 18 changes the binarization threshold value for each step in the stored area in order to accurately detect the shading (step ST19).
[0019]
After changing the threshold and binarizing, the sharpness is detected, the minimum circumference sharpness in the stored area is calculated, the whiteness of the width, the maximum circumference sharpness is calculated. The widths A21 and A22 of the circumference of the portions A11 and A12, the next iris portion, and the widths A31 and A32 of the outer periphery of the pupil are determined (step ST20).
If the information from A11 to A32 cannot be specified with the above threshold, the threshold is incremented or decremented again, and the same processing is performed again (step ST21).
[0020]
Further, from the coordinate information of the outer peripheral widths A31 and A32 of the pupil, the center point coordinate P1 that is the actual gazing point of the pupil is obtained by (A31 + A32) / 2 (step ST22).
[0021]
Next, the center point coordinate P1 is subjected to position correction using the result of calibration described later (step ST23).
[0022]
If the corrected center point coordinate P1 is not moved or exceeded the threshold value from the previous state, a position character display is generated from the result of the center point coordinate P1 and passed through the display control unit 21. It displays on LCD22 which is a display (step ST24, ST25).
[0023]
The calibration associates the relationship between the eye gazing point and the display area of the LCD 22, and at the time of calibration, as shown in FIGS. 6 and 7, the index (Po (127, 127)) and the LCD 22 are centered. The characters (Py (0, 255), Px (255, 0)) are displayed at the four corners of the screen, and the photographer pays attention so that the pupil center reference coordinates (Po (0, 0)) and the coordinates of the four corners are obtained. This calibration will be described later.
[0024]
Then, assuming that the center point is (0.0), each corner is calculated as a prescribed offset value therefrom. The result is held in the central controller 18. The recorded information can be stored in a memory such as an EEPROM.
[0025]
Next, when the position of the pupil of the photographer changes, as shown in FIG. 5, the human eye region has different shades of white, iris, and pupil, as in the process shown in FIG. The central control device 18 changes the binarization threshold value for each step in the stored area in order to accurately detect the shading. After changing the threshold and binarizing, the sharpness is detected, the minimum circumference sharpness in the stored area is calculated, the whiteness of the width, the maximum circumference sharpness is calculated. By determining the widths A21 and A22 of the circumference of the portions A11 and A12, the next iris portion, and the widths A31 and A32 of the outer periphery of the pupil, the center point coordinate P2 of the moved pupil is obtained. When the specified area extracted when the center point coordinate P2 is detected is extremely deviated, the previous extracted area is compared with Z (see FIG. 3), and after the position correction is performed, the final center point coordinates Let P2.
[0026]
As shown in FIG. 7, the center point determines the movement position in the screen based on the ratio to the calibrated center position. Here, if the information of the center point coordinate P2 exceeds the calibrated offset values of the four corners, the position character display on the LCD 22 remains at the previous position.
If it is within the range, as shown in FIG. 8, the central control unit generates a character display (P1 (10, -20)) at the position of the center point coordinate P2, and passes it through the display control unit 21 to the LCD 22. indicate.
If the coordinate position is within the character display area where the function is turned on, the central controller 18 starts counting. The central control unit 18 decrements the count, and if there is a gazing point in the area even when it becomes zero, the function in the area is turned on to change the setting or the like.
[0027]
It is obvious that it is possible to focus on a certain area using this method. That is, when the captured image is displayed on the LCD 22, the specific area determined by the gazing point is focused. In this way, for example, when the distant position of the subject is set as a gazing point, the distant area is focused.
This point will be described later.
If the user turns off the LCD 22, the central controller 18 determines this, and stops the power supply to the second CCD unit 36 and the second front end unit 37 for imaging the photographer, or the power saving mode. To migrate.
[0028]
When the central control unit 18 detects that the LCD 22 is turned on again by a user operation, the power supply of the second CCD unit 36 and the second front end unit 37 is restored, and the switch is turned on again. If there is, the above process is resumed.
[0029]
Next, calibration will be described with reference to the flowchart shown in FIG.
First, when the gazing point input switch is turned on, a calibration character is displayed on the LCD 22 (steps ST31 and ST32).
Next, the process of gazing point described in the flowchart of FIG. 10 is started, the counter is decremented, and the setting position is notified to the user by changing the character state (steps ST33 and ST34).
[0030]
When the counter is not zero, the counter is decremented at regular intervals until the counter reaches zero (steps ST35 and ST36).
[0031]
When the counter reaches zero, the gazing point processing result is registered in association with the designated position (step ST37).
[0032]
If there is a next calibration position, a counter is preset, and the gazing point processing results in steps ST34 to ST37 are registered in association with the designated position (steps ST38 and ST39).
[0033]
Next, when there is no calibration position, after completion display, the display returns to normal display (step ST40).
[0034]
The calibration is performed in this manner. For example, as shown in FIG. 9, the center position is (127, 127) on the LCD, whereas the user position is (30, 30). The position of the left top is (0, 255) on the LCD, whereas the position of the user is (20, 40), and the position of the right top is (255, 255) on the LCD, whereas the position of the user is The position is (40, 40), the position of the left bottom is (0, 0) on the LCD, whereas the position of the user is (20, 20), and the position of the right bottom is (0, 255) on the LCD. On the other hand, the user's position can be associated with (40, 20).
[0035]
Next, focusing on the captured image by obtaining the gazing point will be described with reference to a flowchart shown in FIG.
First, a gazing point input switch is turned on, the counter is reset, and gazing point processing starts (steps ST41 and ST42).
[0036]
Then, it is confirmed whether or not the gazing point is in the designated area (step ST43).
[0037]
When the gazing point exists in the designated area, the counter decrement is started (step ST44).
[0038]
When the counter is not zero, the counter is decremented at regular intervals (steps ST45 and ST46).
[0039]
When the counter is zero, it is checked whether the gazing point still exists in the designated area (step ST47).
[0040]
When the gazing point is present in the designated area, a process assigned to the designated area, that is, a process for focusing the designated area is executed (step ST48).
[0041]
As described above, the image pickup apparatus of the present invention has the second image pickup means 38 that can capture the face of the photographer, particularly the eyes, separately from the first image pickup means 16 for picking up the subject. From the received signal, the position of the pupil is detected from the photographer's face information, the eye gazing point is calculated from the result, and the gazing point is displayed on the LCD 22, so that no button is used for the user. It is possible to provide an input method that can be operated.
In addition, by performing so-called calibration that gives the relationship between the eye position and the display area of the LCD 22, versatility can be given to the movement range of the user's eyes.
Further, since the setting and the like can be changed by looking at the character on the LCD 22 by the above input method, buttons and switches can be reduced, and further miniaturization becomes possible.
Further, since there is no need to look into the conventional electronic viewfinder, a miniaturized and general-purpose line-of-sight input device can be provided.
[0042]
【The invention's effect】
As described above, the imaging apparatus according to the present invention includes the second imaging unit for imaging the face of the photographer, particularly the eyes, in addition to the first imaging unit for imaging the subject. By setting the imaging function of the captured image displayed on the LCD from the gazing point that is the position of the pupil of the imaged eye, various shooting conditions can be set and changed depending on the photographer's eyes Thus, operation members such as buttons and switches can be omitted, and the apparatus itself can be reduced in size and size.
[Brief description of the drawings]
FIG. 1 is a block diagram of an imaging apparatus according to the present invention.
FIG. 2 is an explanatory diagram in which a photographer's face portion imaged by the second imaging means is displayed on an imaging screen of a display unit.
FIG. 3 is an explanatory diagram showing a state in which a white part of an eye, an iris, and a pupil part are extracted.
FIG. 4 is an explanatory diagram showing a state in which the pupil of the eye has moved.
FIG. 5 is an explanatory diagram showing a state of calibration showing the relationship between the LCD area of the display unit and the movement of eyes;
FIG. 6 is an explanatory diagram showing the relationship between the display area of the LCD and the eyes.
FIG. 7 is an explanatory diagram showing the relationship between the display area of the LCD and the eyes.
FIG. 8 is a table showing the relationship between the calibrated LCD position and the eye position.
FIG. 9 is a table showing an image of calibration data.
FIG. 10 is a flowchart for inputting a gazing point.
FIG. 11 is a flowchart of calibration in gazing point input.
FIG. 12 is a flowchart of focus processing in gazing point input.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 11; 1st lens part, 12; 1st shutter, 13; 1st iris, 14; 1st CCD part, 15; 1st front end part, 16; 1st imaging means, 17; Processing unit 18; Central control unit 19; RAM unit 20; ROM unit 21; Display control unit 22; LCD 23; Memory card 24; Memory interface unit 25; Timing generator 26; Vertical driver 27; lens / iris driver, 28; power control unit, 29; temperature sensor, 30; external operation unit, 31; first network interface unit, 32; second network interface unit, 33; second lens unit, 34; second shutter, 35; second iris, 36; second CCD unit, 37; second front end unit, 38; Second imaging means, 39; bus line, 40; SDRAM controller, 41; SDRAM memory.

Claims (2)

A first image pickup means for picking up an image of a subject; a display on which an image picked up by the first image pickup means is displayed; a second image pickup means for picking up an eye of a photographer looking at the display; Gazing point detection means for detecting the gazing point of the photographer's eyes from the eye information imaged by the second imaging means,
The position information of the gazing point detected by the gazing point detection means displays a calibration character at a specified position of the display, and displays the gazing point calculated by looking at the calibration character in the display area of the display An image pickup apparatus characterized in that the focus of the subject imaged by the first image pickup means is made to follow the display area gazed at by the display device, as well as information by calibration for alignment with the display device   The imaging apparatus according to claim 1, wherein the position information of the gazing point of the photographer is displayed on the display.

Images