JP2004341112A - Imaging unit - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging unit equipped with a 1st imaging means for imaging an object and a 2nd imaging means for imaging photographer's eyes looking at a large screen, and constituted so that an imaging function such as a focusing function can be set by detecting the pupils of the eyes imaged by the 2nd imaging means, detecting a gazing point, and then, specifying the display area of the large screen indicated by the gazing point. <P>SOLUTION: The imaging unit is provided with a 1st imaging means for imaging the object, a 2nd imaging means for imaging the photographer's eyes viewing a display part where the image picked up by the 1st imaging means is displayed, and a gazing point detecting means for detecting the gazing point by the photographer on the basis of information on the photographer's eyes imaged by the 2nd imaging means, and then, the imaging function of the object is set on the basis of the information on the gazing point obtained by the gazing point detecting means. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an imaging device, and more specifically, simultaneously shoots a subject, shoots an eye portion of a photographer, observes the movement of the shot pupil, and focuses the subject. The present invention relates to an imaging device.
[0002]
[Prior art]
In an imaging device in the related art, for example, a digital still camera or a video camera, when a user changes a setting or implements a function item such as shooting or playback, a button or a switch is always attached to an exterior. , And operated it as an input device.
However, in recent years, digital still cameras and video cameras have been reduced in size, and as a result, buttons and switches have been removed as much as possible.
As a substitute for the missing buttons and switches, a method of assisting the operation by using a human line of sight or voice has been adopted from the use of a currently-displayed display device.
In particular, the method using the line of sight can be operated with little use of buttons and switches. Further, in the photographing using sound, the operation can be performed while following the subject with natural movement.
In addition, the current gaze input generally looks into a user or an electronic viewfinder, irradiates an LED to the looking state, and detects the position of the pupil from its reflection to specify the position. It is.
[0003]
[Patent Document 1]
Japanese Patent Application Laid-Open No. 04-96580 (Page 2 FIG. 1)
[0004]
[Problems to be solved by the invention]
However, in the technique of detecting the position of the pupil by illuminating the LED by looking into the electronic viewfinder described in the related art and specifying the position, the user always needs to act to look into the electronic viewfinder. There is a problem that a large-screen display device, which is often used in digital still cameras and video cameras in recent years, cannot be used, and an electronic viewfinder is required as a device, which is an obstacle to miniaturization. is there.
Therefore, there is a problem that an imaging apparatus to be miniaturized, for example, a digital still camera or a video camera, has to be solved by a method of specifying a setting of an imaging function desired by a user by movement of a pupil.
[0005]
[Means for Solving the Problems]
In order to achieve the above object, an imaging device according to the present invention has the following configuration.
[0006]
(1) An imaging apparatus includes: a first imaging unit that captures an image of a subject; a display that displays an image captured by the first imaging unit; and a second imaging unit that captures an eye of a photographer looking at the display. And a fixation point detecting means for detecting a fixation point of the photographer's eye from information on the eyes picked up by the second image pickup means, wherein the fixation point detected by the fixation point detection means is provided. That is, an imaging function of a subject is set based on position information.
(2) The position information of the gazing point is information by calibration that positions the gazing point calculated by looking at the designated position of the display on the display area of the display. An imaging device according to any one of the preceding claims.
(3) The imaging apparatus according to (1), wherein the position information of the point of gaze of the photographer is displayed on the display.
(4) The setting of the imaging function of the subject based on the position information of the gazing point detected by the gazing point detection means is to make the focus follow the display area gazed by the display. An imaging device according to any one of the preceding claims.
(5) The imaging device according to (1), wherein the setting of the imaging function is stopped when a photographer does not look at the display.
(6) The imaging apparatus according to (5), wherein stopping the setting of the imaging function is to stop power supply of the second imaging unit or to set the power to a power saving mode.
[0007]
(7) The image pickup apparatus has a first image pickup unit for picking up an image of a subject, a display for displaying an image picked up by the first image pickup unit, and various operation modes displayed on the display with a predetermined character. Operation mode display means, second imaging means for imaging the eyes of the photographer looking at the display, and detection of the gaze point of the photographer's eyes from information of the eyes imaged by the second imaging means A viewpoint detection unit, and based on the position information of the gazing point detected by the gazing point detection unit, when it is detected that one of the characters is gazing, a corresponding operation mode is set. That is what we did.
(8) The position information of the gazing point is information by calibration that positions the gazing point calculated by looking at the designated position of the display on the display area of the display. An imaging device according to any one of the preceding claims.
(9) The imaging device according to (7), wherein the position information of the point of gazing point of the photographer is displayed on the display.
(10) When the position information of the gazing point detected by the gazing point detecting means is the area of the image displayed on the display, the focus is made to follow the display area watched by the display. The imaging device according to (7), which is configured as described above.
(11) The imaging device according to (7), wherein the function of the second imaging unit is stopped when a photographer does not look at the display.
(12) The imaging device according to (11), in which stopping the function of the second imaging unit is to stop power supply of the second imaging unit or to set the second imaging unit in a power saving mode.
[0008]
In this way, by capturing the photographer's eyes and observing the gazing point when capturing the image of the subject, it is possible to automatically adjust the focus and the like of the subject according to the movement of the eyes of the photographer.
[0009]
BEST MODE FOR CARRYING OUT THE INVENTION
Next, an embodiment of an imaging device according to the present invention will be described with reference to the drawings.
[0010]
As shown in FIG. 1, the imaging apparatus according to the present invention includes a first lens unit 11 for capturing an object, a first shutter 12, and a first iris 13 for forming an image on an imaging device. A CCD section 14, a first front end section 15 for converting an image formed into an electric signal and converting it into a digital signal, a signal processing section 17 including a DSP for performing signal processing of an image signal, and a CPU. A central controller 18, a RAM unit 19, a ROM unit 20, a display control unit 21 for controlling an OSD display, an LCD 22 corresponding to a display as a liquid crystal screen, and a memory interface unit for controlling a memory card 23. 24, a timing generator 25 controlled by the central control unit 18 to generate a clock of an image captured by the first CCD unit 14, and a central control unit A vertical driver 26 controlled by the first CCD unit 14 to synchronize the image captured by the first CCD unit 14; a lens / iris driver controlled by the central controller 18 to control the first lens unit 11 and the first iris 13 27, a power control unit 28, a temperature sensor 29, an external operation unit 30 including buttons and switches, a first network interface unit 31, and a second network interface unit 32. A second CCD unit 36 for forming an image on an image sensor via a second lens unit 33 for capturing an image of the photographer's face, a second shutter 34, and a second iris 35; and a second CCD unit 36 And a second front end unit 37 for converting the image formed on the digital camera into a digital signal. The signal converted by the second front end unit 37 is a signal. It is sent to the processing unit 17, processing for detecting the gaze point of the eye by detecting the position of the pupil with respect to the subject is performed.
[0011]
Among them, the first lens unit 11, the first shutter 12, the first CCD unit 14 for forming an image on the imaging device via the first focus / iris 13, and the first front end unit 15 are the first A second CCD section for forming an image pickup means 16 and forming an image on an image pickup device via a second lens section 33, a second shutter 34, and a second iris 35 for taking an image of the face of the photographer; 36 and a second front end unit 37 that converts an image formed on the second CCD unit 36 into a digital signal form a second imaging unit 38.
The central controller 18, the RAM unit 19, the ROM unit 20, the display control unit 21, the memory interface unit 24, the signal processing unit 17, the first network interface unit 31, and the second network interface unit 32 are connected to the bus line 39. They are connected and controlled by the central control unit 18.
[0012]
The signal processing unit 17 detects the luminance from the video signal obtained by the first imaging unit 16 and detects the gazing point of the photographer's eyes from the information of the eyes captured by the second imaging unit 38. Means for storing luminance information and gazing point information in the SDRAM memory 41 via the SDRAM controller 40.
[0013]
The central control device 18 controls the first focus / iris 13 based on the information on the point of gaze obtained by the signal processing unit 17 or the calibrated information to control the first focus / iris 13 to follow the focus. An operation mode display unit for displaying various operation modes on the LCD 22 with a predetermined character, and gazing at any one of the characters based on the position information of the gazing point detected by the gazing point detection unit of the signal processing unit 17. When the operation mode is detected, control is performed so as to set the corresponding operation mode.
[0014]
The operation of the imaging apparatus having such a configuration will be described below.
First, a method of actually detecting the point of gaze of the photographer will be described below with reference to the flowchart shown in FIG.
First, the user turns on the main switch of the external operation unit 30 (step ST11). When the result is received by the central control unit 18, the power supply control unit 28 is controlled to the second CCD unit 36 and the second front end unit 37 to supply power and start imaging (step ST12). ).
[0015]
Image information converted into a digital signal by the first front end unit 15 is input to the signal processing unit 17 from the first CCD unit 14 facing the subject, and the luminance is calculated (step ST13).
[0016]
At the same time, the second CCD section 36 facing the photographer forms an image of the photographer, particularly an image of the face, and a second front end section 37 which directly connects the analog data to the second CCD section 36. Then, a digital signal is sent and the information is sent to the signal processing unit 17 (step ST14).
The signal processing section 17 pastes and stores the digital signal input from the second front end section 37 in the SDRAM memory 41 via the SDRAM controller 40 (step ST15).
[0017]
As shown in FIG. 2, the central control unit 18 detects the sharpness and the skin color area from the information on the face stored in the SDRAM memory 41, specifies the face, and then detects the area with white information (step). ST15).
Then, as shown in FIG. 3, the vertical and horizontal ranges of the white information are determined from the detected white information area, and the area Z where the vertical and horizontal overlaps on the straight line extending from the maximum and minimum points of the range is defined on the SDRAM memory 41. The image is extracted from the main image and stored in another memory area (steps ST16, ST17, ST18).
[0018]
Methods for detecting the actual pupil size from the stored area include color difference and binarization, but this time, the binarization method will be described.
In the human eye region, the shading differs in white, iris, and pupil. As shown in FIG. 4, the central controller 18 changes the binarization threshold for each step in the stored area in order to accurately detect the shading (step ST19).
[0019]
After changing the threshold value and performing binarization, the sharpness is detected, the sharpness of the smallest circumference in the stored area is calculated, and the width, the whiteness of the sharpness of the largest circumference are calculated. The portions A11 and A12, the circumferential widths A21 and A22 of the next iris portion, and the outer pupil widths A31 and A32 are determined (step ST20).
When the information of A11 to A32 cannot be specified with the above threshold value, the threshold value is incremented or decremented again, and the same processing is performed again (step ST21).
[0020]
Further, the center point coordinate P1 which is the actual gazing point of the pupil is obtained from (A31 + A32) / 2 based on the coordinate information of the outer pupil widths A31 and A32 (step ST22).
[0021]
Next, with respect to the center point coordinates P1, the position is corrected using the result of the calibration described later (step ST23).
[0022]
When the corrected center point coordinate P1 does not extremely move or exceed the threshold from the previous state, a position character display is generated from the result of the center point coordinate P1, and is displayed via the display control unit 21. The information is displayed on the LCD 22, which is a display (steps ST24 and ST25).
[0023]
The calibration is for associating the relationship between the point of regard of the eye and the display area of the LCD 22. At the time of calibration, as shown in FIGS. 6 and 7, the index (Po (127, 127)) and the LCD 22 are centered. The characters (Py (0, 255), Px (255, 0)) are displayed at the four corners of the pupil, and the photographer is noticed, so that the pupil center reference coordinates (Po (0, 0)) and the coordinates of the four corners are obtained. This calibration will be described later.
[0024]
Then, the center point is set to (0.0), and each of the four corners is calculated as a specified offset value therefrom. The result is held in the central controller 18. The recording information can be stored in a memory such as an EEPROM.
[0025]
Next, when the position of the pupil of the photographer changes, as shown in FIG. 5, in the same manner as in the processing shown in FIG. 4, the human eye area has different shades of white, iris, and pupil. In order to accurately detect the shading, the central control device 18 changes the binarization threshold value for each step in the stored area. After changing the threshold value and performing binarization, the sharpness is detected, the sharpness of the smallest circumference in the stored area is calculated, and the width, the whiteness of the sharpness of the largest circumference are calculated. By determining the portions A11 and A12, the circumferential widths A21 and A22 of the next iris portion, and the outer pupil widths A31 and A32, the center point coordinates P2 of the moved pupil can be obtained. If the extracted designated area is extremely displaced when the center point coordinates P2 are detected, the immediately preceding extracted area is compared with Z (see FIG. 3), and after finalizing the position, the final center point coordinates are applied. P2.
[0026]
As shown in FIG. 7, the center position determines the moving position in the screen by the ratio with the calibrated center position. Here, if the information of the center point coordinates P2 exceeds the calibrated offset values of the four corners, the position character display on the LCD 22 remains at the immediately preceding position.
If it is within the range, as shown in FIG. 8, the central control unit generates a character display (P1 (10, -20)) at the position of the center point coordinate P2, and sends it to the LCD 22 via the display control unit 21. indicate.
If the coordinate position is within the character display area where the function is turned on, the central controller 18 starts counting. The central controller 18 decrements the count and, even when the count reaches zero, if there is a gazing point in the area, the function in the area is turned on to change the setting or the like.
[0027]
Obviously, it is possible to focus on a certain area using this method. That is, when the captured image is displayed on the LCD 22, focusing is performed on the specific area determined by the point of regard. In this way, for example, when the distant position of the subject is set as the gazing point, the distant region is focused.
This will be described later.
When the user turns off the LCD 22, the central control unit 18 determines that the LCD 22 has been turned off, and the power supply to the second CCD unit 36 and the second front end unit 37 for capturing an image of the photographer is stopped or the power saving mode is set. Move to
[0028]
When the central control unit 18 detects that the LCD 22 is turned on again by a user operation, the power supply to the second CCD unit 36 and the second front end unit 37 is restored, and the switch is turned on again. If so, the above process is restarted.
[0029]
Next, calibration will be described with reference to the flowchart shown in FIG.
First, when the gazing point input switch is turned on, a calibration character is displayed on the LCD 22 (steps ST31 and ST32).
Next, the processing of the gazing point described in the flowchart of FIG. 10 is started, the counter decrement is started, and the setting position is notified to the user by changing the state of the character (steps ST33 and ST34).
[0030]
When the counter is not zero, the counter is decremented at regular intervals until the counter becomes zero (steps ST35 and ST36).
[0031]
When the counter reaches zero, the processing result of the gazing point is registered in association with the designated position (step ST37).
[0032]
If there is a next calibration position, the counter is preset, and the results of the gazing point processing in steps ST34 to ST37 are registered in association with the designated position (steps ST38 and ST39).
[0033]
Next, when there is no calibration position, after completion display is performed, the display returns to normal display (step ST40).
[0034]
Calibration is performed in this manner. For example, as shown in FIG. 9, the position of the center is (127, 127) on the LCD, while the position of the user is (30, 30). The position of the left top is (0, 255) in the LCD, the position of the user is (20, 40), and the position of the right top is (255, 255) in the LCD, whereas the position of the user is (255, 255). The position is (40, 40), the position of the left bottom is (0, 0) on the LCD, while the position of the user is (20, 20), and the position of the right bottom is (0, 255) on the LCD. In contrast, the position of the user can be associated as (40, 20).
[0035]
Next, focusing of a captured image by obtaining the point of regard will be described with reference to a flowchart shown in FIG.
First, the switch for inputting the point of interest is turned on, the counter is reset, and the processing of the point of interest is started (steps ST41 and ST42).
[0036]
Then, it is confirmed whether or not the point of interest is in the designated area (step ST43).
[0037]
If the point of interest exists in the designated area, the counter starts decrementing (step ST44).
[0038]
If the counter is not zero, the counter is decremented at regular intervals (steps ST45 and ST46).
[0039]
When the counter is zero, it is checked whether or not the point of gaze still exists in the designated area (step ST47).
[0040]
When the point of gaze exists in the specified area, the processing assigned to the specified area, that is, the processing for focusing on the specified area is executed (step ST48).
[0041]
As described above, the image pickup apparatus of the present invention has the second image pickup means 38 for imaging the face of the photographer, particularly the eyes, separately from the first image pickup means 16 for picking up an object, and performs photoelectric conversion therefrom. By detecting the position of the pupil from the photographer's face information from the signal thus obtained, calculating the gazing point of the eye from the result, and displaying the gazing point on the LCD 22, the button is not used for the user. It is possible to provide an input method that can be operated.
In addition, by performing a so-called calibration in which the relationship between the position of the eyes and the display area of the LCD 22 is performed, versatility can be given to the moving range of the eyes of the user.
Further, since the settings and the like can be changed by looking at the character on the LCD 22 by the above input method, the number of buttons and switches can be reduced, and the size can be further reduced.
Further, since it is not necessary to look into the electronic viewfinder as in the related art, it is possible to provide a downsized and general-purpose line-of-sight input device.
[0042]
【The invention's effect】
As described above, the imaging apparatus of the present invention includes the second imaging unit for imaging the face of the photographer, particularly the eyes, in addition to the first imaging unit for imaging the subject. By setting the imaging function of the captured image displayed on the LCD from the fixation point, which is the position of the pupil of the imaged eye, various imaging conditions can be changed according to the eyes of the photographer. As a result, operation members such as buttons and switches can be omitted, and there is an effect that the size and size of the device itself can be reduced.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating an imaging apparatus according to the present invention.
FIG. 2 is an explanatory view showing a portion of a photographer's face captured by a second image capturing means on an image capturing screen of a display unit.
FIG. 3 is an explanatory diagram showing a state in which a white portion, an iris, and a pupil portion of the eye are extracted.
FIG. 4 is an explanatory view showing a state in which the pupil of the eye has moved.
FIG. 5 is an explanatory diagram showing a state of calibration indicating a relationship between an LCD area of a display unit and a degree of eye movement.
FIG. 6 is an explanatory diagram showing a relationship between a display area of the LCD and eyes.
FIG. 7 is an explanatory diagram showing the relationship between the display area of the LCD and the eyes.
FIG. 8 is a table showing a relationship between a calibrated LCD position and an eye position.
FIG. 9 is a table showing an image of calibration data.
FIG. 10 is a flowchart for inputting a gazing point.
FIG. 11 is a flowchart of calibration in inputting a gazing point.
FIG. 12 is a flow chart of a focus process in inputting a gazing point.
[Explanation of symbols]
11; first lens unit, 12; first shutter, 13; first iris, 14; first CCD unit, 15; first front end unit, 16; first imaging means, 17; Central processing unit, 19; RAM unit, 20; ROM unit, 21; display control unit, 22; LCD, 23; memory card, 24; memory interface unit, 25; timing generator, 26; 27; lens / iris driver; 28; power supply control unit; 29; temperature sensor; 30; external operation unit; 31; first network interface unit; 32; second network interface unit; 34; second shutter, 35; second iris, 36; second CCD section, 37; second front end section, 38; Second imaging means, 39; bus line, 40; SDRAM controller, 41; SDRAM memory.

Claims (12)

A first image pickup unit for picking up an image of a subject, a display for displaying an image picked up by the first image pickup unit, a second image pickup unit for picking up an eye of a photographer looking at the display, Gazing point detecting means for detecting a gazing point of the photographer's eye from information on the eyes imaged by the second imaging means;
An imaging apparatus, wherein an imaging function of a subject is set based on positional information of a gazing point detected by the gazing point detection unit. 2. The imaging apparatus according to claim 1, wherein the position information of the gazing point is information based on calibration for aligning a gazing point calculated based on a designated position of the display with a display area of the display. 3. apparatus. The imaging apparatus according to claim 1, wherein the position information of the point of gazing point of the photographer is displayed on the display. 2. The imaging apparatus according to claim 1, wherein the setting of the imaging function of the subject based on the position information of the gazing point detected by the gazing point detection unit is to make the focus follow a display area gazed by the display. apparatus. 2. The imaging apparatus according to claim 1, wherein the setting of the imaging function is stopped when a photographer does not look at the display. The imaging apparatus according to claim 5, wherein stopping the setting of the imaging function is to stop the power supply of the second imaging unit or to set the power to a power saving mode. A first imaging unit for imaging a subject, a display on which an image captured by the first imaging unit is displayed, an operation mode display unit for displaying various operation modes on the display with a predetermined character, A second imaging unit configured to capture an image of a photographer's eyes looking at a display; and a fixation point detection unit configured to detect a fixation point of the photographer's eyes from information of the eyes captured by the second imaging unit. ,
An imaging mode wherein a corresponding operation mode is set when it is detected that any one of the characters is gazing based on positional information of a gazing point detected by the gazing point detecting means. apparatus. The imaging apparatus according to claim 7, wherein the position information of the gazing point is information obtained by calibration that positions a gazing point calculated by looking at a designated position of the display on a display area of the display. apparatus. The imaging apparatus according to claim 7, wherein the position information of the gazing point of the photographer is displayed on the display. When the position information of the gazing point detected by the gazing point detection unit is the area of the image displayed on the display, the focus is made to follow the display area watched by the display. The imaging device according to claim 7, wherein: The imaging apparatus according to claim 7, wherein the function of the second imaging unit is stopped when a photographer does not look at the display. 12. The imaging apparatus according to claim 11, wherein stopping the function of the second imaging unit stops power supply of the second imaging unit or sets the power supply mode.

Images