JPH0782539B2 - Pupil imager - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a pupil image capturing apparatus, and more particularly to a pupil image capturing apparatus capable of stably extracting a pupil, which is an eyeball feature point, with illumination with a small amount of light.

[Prior Art] In recent years, computers have come to be able to realize complicated and abundant functions due to the progress thereof, and the range of application thereof is expanding more and more. On the other hand, its users are also expanding from specialists to non-professionals. Here, human interface technology that makes it easier to use more complex systems becomes important. In human-to-human communication, smooth dialogue is performed by inferring the other person's intention not only from words but also from facial expressions and gestures. In particular, eye movements play an important role in communication.

That is, it is considered that the intention of the person is largely reflected in the movement of the viewpoint. Since a user faces a certain display device for a long time in a computer or a communication system interface, if the movement of the viewpoint on the drawing of the display device can be constantly detected, it is considered to be a great clue for extracting the user's intention. In short, what users are looking at, what they are interested in, and what hesitation is.

Therefore, the inventors of the present application collect the operation intention of the user,
To realize an interface function that returns a flexible response,
Furthermore, in the field of intelligent communication, by extracting the visual target of interest from the recipient from the movement of the recipient's line of sight and feeding it back to the sender, it is possible to realize a realistic image communication of the recipient. I have pointed out the importance of an efficient method of detecting the line of sight.

An eye camera has been conventionally known as a device for detecting the line of sight as described above. However, the eye camera
Since the user must wear eyeglasses, and the user's head must be fixed in order to obtain the viewpoint in the coordinate system of the display that the user is facing, the line of sight is used for the interface and image communication. It is not always suitable for the purpose. In other words, without making people wear special items,
In order to detect the line of sight without contact, the method of detecting by visual processing is more advantageous.

FIG. 23 is a diagram showing an example conceivable as the structure of the non-contact visual line detection device. Referring to FIG. 23, display 1
Cameras 2 and 3 and lighting devices 4 and 5 are provided on both sides of. In such a non-contact line-of-sight detection device, the first problem is that the images of the user illuminated by the illumination devices 4 and 5 are captured by the cameras 2, 3
Is performed, and a plurality of feature points required for eye gaze detection are extracted from the captured image. The second problem is to measure the spatial positions of those feature points at high speed and with high accuracy. The third problem is to find the direction of the line of sight and the position of the point of gaze on the display from the positions of the feature points.

Since the movement of a person, especially the movement of the line of sight, is fast, it is necessary to capture a clear image first and to be able to extract the feature points by simple image processing in order to accurately detect the movement by following this movement. There is. However, in an actual room, the user emits light from the display, and the external lighting such as a fluorescent lamp influences the lighting condition of the user. Therefore, a stable image is not always obtained. If the quality of the input image is poor, it takes time to remove noise, etc., and speedup is impossible.

As a means for solving this, a method of illuminating a person who is a photographing target can be considered, but this method has the following problems. That is, one problem is that it is difficult to obtain a natural interface environment. In other words, incandescent lamps, xenon lamps, halogen lamps, etc. have been well known as lighting devices, but since these have a wide wavelength range centered around the visible range, this light illuminates the user from the front. The method to do is not suitable in terms of a natural interface.

The second problem is that the device becomes large and heat is generated. In other words, considering the conditions used for the interface, and trying to improve the lighting conditions of conventional lighting devices,
It is necessary to install optical components such as bandpass filters and polarizing plates in front of the illumination source. For example, in the case of using near-infrared illumination that is not perceived by a person and capturing the reflected light, it is necessary to block visible light. However, the above-described conventionally known illuminating device has a low light emission efficiency and a large amount of heat generation, so that the ambient temperature rises. For this reason,
Since the light source and the optical element cannot be integrated into a small size, the illumination device must be a large one.

Next, although illumination produces a clear image depending on the method of use and is effective in extracting characteristic points, on the other hand, if it is not used in accordance with the purpose, it becomes a noise source and has the opposite effect. A case of extracting the feature points of a person or an eyeball will be described in detail as an example.

FIG. 24 is a diagram showing an example of an experiment in which a blue mark matching a person's face is extracted as a feature point using a conventional lighting device and a photographing device, and FIGS. 25A and 25B are shown. FIG. 25 is a diagram showing an example in which facial feature points are extracted by the experiment shown in FIG. 24.

As shown in FIG. 25A, blue markers 6 are attached to four points on the person's face. The reference light 9 is emitted from the illumination device 8 shown in FIG.
The image shown in FIG. 25B is the result of binarization by extracting the blue component from the image captured and captured in. As is clear from FIG. 25B, the noise component 7 is captured in addition to the blue mark 6, which shows that there is a problem. The cause of this is considered as follows. That is, when the object 10 to be photographed is irradiated with the reference light 9, the reflected light is roughly divided into two types of components. One is the light 11 which is diffusely reflected on the surface of the object to be photographed 10.
And reflects the optical properties of the reflective material. Therefore, among the feature points necessary for detecting the line of sight, elements of the face (mouth, eyelashes, eyes,
It is a useful component in extracting feature points such as the corners of the eyes and nose) and pupils. The other is the component 12 that is specularly reflected on the surface of the object 10 to be photographed, which directly reflects the optical properties of the light source. That is, since it is a component that does not reflect the nature of the object to be photographed 10, it is likely to become noise. The latter is often included in the smooth part of the subject 10. Specifically, when the object to be photographed is a person, the part of sweat attached to the face, the edge of the glasses, the spectacle lens, and structures such as plastic and glass around the person correspond to this. In the example shown in FIG. 25B, noise 7 is a place of sweat.

In the example shown in FIGS. 25A and 25B, the case of extracting the blue mark attached to the face has been described, but the same applies to the case of using this mark of another color and extracting this color component. Also,
Even when parts such as eyes, nose, mouth, and eyelashes are extracted from a natural image without using marks, the specular reflection component 12 often becomes noise. Further, as another example of detecting the shape of a person's face without using a mark, a pre-controlled shape pattern is irradiated on the object to be photographed, as represented by the moire method and the light section method, and 2 Extracting a binarized image (reflection pattern) and featuring the shape of this reflection pattern,
There is so-called active stereo vision that measures the three-dimensional shape of the object to be photographed. Even in this method, when the object to be photographed is easily specularly reflected, the image created by this regular reflection becomes noise, which is a characteristic of the object to be photographed. There are often problems in extracting the reflection pattern.

Next, problems with the arrangement of the conventional illumination device and the extraction efficiency of the pupil feature points will be described. In order to detect the user's line of sight in a contactless manner without wearing anything on the user, it is necessary to extract a plurality of feature points by image processing as described later.
The pupil is the opening of the iris. The size of the pupil is appropriate, it is not easily affected by the eyelids, and it is convenient for conversion to the line of sight. Therefore, it can be said that it has a wide applicability if there is an appropriate extraction method. However, since the pupil is generally observed to be black, when the iris is blackish brown, it is necessary to distinguish this and extract only the pupil.

There are some examples of eye extraction in eye cameras, and several methods are known. For example, USP4,102,564, USP4,14
5,122, USP3,689,135, USP4,075,657, USP4,755,045, U
It is described in SP4,303,394, USP4,651,145, USP4,702,575 and the like. Some types of eye cameras use a method in which a light source is incorporated in eyeglasses, light is emitted to the eyeball, the reflected light is photographed, and the intensities of the reflected light of the pupil and the iris are measured. When the device is worn on the head like an eye camera, the lighting device is close to the shooting device and is linked to the movement of the head, so the lighting device only needs to illuminate the eyeballs. The device need only image the eyeball. That is, the influence of noise is small, and therefore the pupil can be extracted only by the intensity difference between the reflected light of the pupil and the reflected light of the iris.

However, in gaze detection considering the use of the interface,
Since it is necessary to be non-contact as described above, it is desired that the photographing range is not limited to the eyeball portion but a wide range that allows the movement of the head. For example, it is necessary to extract the pupil from an image of the entire face. In such a case, it is difficult to separate the pupil part from the background noise by the above method.

FIG. 26 is a diagram showing another method of extracting the pupil, in which light is incident from the pupil and light reflected by the retina is captured. First
Referring to FIG. 26, a half mirror 23 is provided on the optical axis 22 of the taking lens 21 of the camera 20, and a conventional lighting device 24 is arranged so that the optical axis is aligned using the half mirror 23. It is to be noted that such an half mirror 23 is used in order to obtain a pupil image in which the distribution of the reflection intensity in the pupil is uniform in the illumination device using one light source, that is, in which the intensity distribution is less biased. Is essential.

A visible cutoff filter 25 is arranged in front of the lighting device 24,
The visible cutoff filter 25 cuts off the visible wavelength component of the light from the illumination device 24, and the half mirror 23 forms a lens.
It is aligned with the optical axis 22 of 21 and is illuminated on the person 26. This light enters from the pupil of the person 26, is reflected by the retina, passes through the half mirror 23 again, and is captured by the camera 20. Therefore, the pupil is photographed brightly with respect to the iris.

[Problems to be Solved by the Invention] However, the method shown in FIG. 26 has the following problems. That is, since the half mirror 23 is used, the device becomes large. When the photographing range is widened not around the eyes but around the entire face, for example, the influence of noise also increases in this method, and stable pupil extraction becomes difficult. In order to reduce the influence of noise, it is conceivable to increase the illumination, but the light amount is reduced by the visible cutoff filter 25, the half mirror 23 further reduces the light amount to 1/2, and the reflected light is reflected by the half mirror 23. Since the light of the lighting device 24 is captured by the camera 20, the loss is large and the efficiency is low. If the lighting is strengthened, power consumption increases and heat generation also increases. In order to reduce this effect, it is necessary to devise a method in which each component is mounted separately, which further increases the size of the device. In addition, increasing the intensity of light imposes a physiological burden on the eyes of the user, which is not desirable. Therefore, this method is also not suitable for the purpose of attaching the sight line detecting camera to the side of the display and using it as an interface.

The corneal reflection image is a virtual image created by the light regularly reflected by the convex surface of the cornea when the eyeball is irradiated with light, and since it moves in the same direction as the eyeball in almost proportion to the movement of the line of sight, it is an effective feature point for detecting the line of sight. is there. The problem with extraction is the separation from background noise.

FIG. 27 is a diagram showing an example of an apparatus for photographing a corneal reflection image with a camera. Referring to FIG. 27, when the light from the reference light source 31 is applied to the eyes of the user 30, the corneal reflection image necessary for detecting the line of sight is captured by the camera 32. But user 30
Not only the light from the reference light source 31 but also the light from the display surface of the display 33 facing the user and the light from the external illumination 34 such as a fluorescent lamp are sources of light to the eyes of the cornea. To create a virtual image. Therefore, in the image captured by the camera, a number of corneal reflection images are generated in the eyeball and become noise, which makes it difficult to extract the corneal reflection image by the light from the reference light source 31.

When shooting a moving object such as a person or an eyeball, the shooting time is preferably short in order to obtain an image without blurring.
For this reason, recently, a camera with an electronic shutter has been put into practical use. Strong illumination is required for such cameras with short shooting times. However, considering that a conventional lighting device is used to illuminate a person in the interface, heat and strong light irradiate the interface for a long period of time, so that the influence on the eyeball and the living body becomes a problem.

On the other hand, for measuring the spatial position of feature points, a stereo image measurement method is known, but it has been difficult to extract feature points in the past as described above. There is no study example of real-time measurement of the spatial position of.

Conventionally, several methods have been proposed for visual axis detection by image processing. However, because of the fact that it is difficult to extract feature points efficiently, and it is difficult to detect spatial position at high speed, many conditions are set for eye gaze detection, and the applicable range may be limited. Many. As a configuration,
Many systems use one black and white camera. An example thereof will be described below.

FIG. 28 is a diagram for explaining a method of detecting the position of the black eye in the white eye and converting it into the line of sight on the captured image. Referring to FIG. 28, the face of a person is photographed by the camera 41, the eye portion 42 is extracted from the photographed face image,
The areas of the black eye 43 and the white eye 44 in the eye are separated. Next, the length a of the region of the white eye 44 and the length x from the end of the white eye 44 to the center of the black eye 43 are obtained. The direction of the line of sight can be calculated as approximately proportional to x / a. Real-time detection has not been realized because it is difficult to speed up the extraction processing of the regions of the white eye 44 and the black eye 43. In this method, the degree of freedom is limited to the rotational movement of the eyeball unless the position and direction of the face are obtained by some method. Further, with respect to the rotational movement of the eyeball, the size of the eye changes depending on the facial expression, so high detection accuracy cannot be obtained. In particular, it is difficult to detect the movement of the upper and lower eyes because the area of the black eye 43 changes due to the effect of the eyelids.

FIG. 29 is a diagram for explaining a method of detecting with one camera using the pupil and the corneal reflection image as feature points. 7th
Referring to the figure, the position of reference light source 51 is known with respect to the coordinate system of camera 50. The spatial position of the corneal reflection image 53 and the spatial position of the pupil 54 generated by the reference light from the reference light source 51 are independently determined by the position of the center of rotation of the eyeball 52 and the vertical and horizontal rotation angles α, β of the eyeball. Therefore, the position of the corneal reflex image, the position of the pupil 54, the radius a of the eyeball which is a structural parameter of the eyeball 52, the radius of curvature c of the cornea, and the distance b between the center of the eyeball and the center of corneal curvature are determined. Eyeball
The rotation center of 52 and the rotation angle of the eyeball 52 are determined, and therefore the line of sight is obtained. The positions of the corneal reflection image 53 and the pupil 54 can be obtained from one projection image if the condition that the distance from the camera 50 to the eyeball 52 is substantially constant is provided. In this way, the line of sight can be obtained for the left and right rotation angles α and the upper and lower rotation angles β of the eyeball 52, but there is a drawback that the accuracy deteriorates when the face moves in the z-axis direction due to the above constraint condition. is there.

Further, in the examples shown in FIGS. 28 and 29 described above, since there is only one camera, it is difficult to find the gazing point. When using any of the above methods for the interface, it is necessary to know where to look on the display and therefore to find the point of gaze in the display coordinate system.
In order to know where the user is looking in the display coordinate system, it is necessary to let the user see a predetermined point on the display surface of the display and calibrate based on the data. However, when there is only one camera, the number of calibration parameters increases, which complicates the calibration method and deteriorates the accuracy.

Therefore, a main object of the present invention is to provide a pupil image photographing device capable of efficiently extracting the pupil by photographing the pupil brightly or darkly.

[Means for Solving the Problem] The invention according to claim 1 is a pupil image photographing device for photographing a pupil image, which is provided outside an effective diameter of a photographing lens and irradiates the direction of the eye. No. 1 light source and a first photographing means for forming an image of reflected light of the first light source through a photographing lens and outputting a first image signal, and an eye direction provided in the vicinity of the optical axis of the photographing lens. A second light source for irradiating the second light source, and a second photographing means for forming an image of reflected light of the second light source through a photographing lens and outputting a second image signal; and outputs of the first and second photographing means. And a processing means for outputting a differential signal of the image signal as a third image signal, and the second light source is a line segment connecting the light source and the eyeball to be photographed, and a line segment connecting the center of the photographing lens to the eyeball. If the angle between and is θth2, θth2 ≤ 5
The first light source is installed at an angle of ° and the angle between the line segment connecting the light source and the eye to be photographed and the line segment connecting the center of the photographing lens to the eye is θth1.
The first and second photographing means are arranged so as to form an angle of ≧ θth2 + 1 °, and are configured to alternately and intermittently photograph images based on the reflected light from the first and second light sources.

The invention according to claim 2 is a pupil image photographing device for photographing a pupil image, the first light source being provided outside an effective diameter of a photographing lens, and illuminating the direction of the eye; A first photographing means for forming an image of the reflected light of the image through a photographing lens and outputting a first video signal; and a second light source which is provided near the optical axis of the photographing lens and illuminates the direction of the eye. A second photographing means for forming an image of reflected light of the second light source through a photographing lens and outputting a second image signal, and a difference signal between the image signals output from the first and second photographing means for the third Of the first light source incident on the photographing lens and the image based on the second reflected light from the second light source are separated to separate the first image signal from the first light source. And a prism optical system for individually providing to the second photographing means, Preliminary second optical system, which is constituted by a respective different emission wavelengths to the extent possible separation prism optical system emitting diode.

[Operation] In the invention according to claim 1, from the first light source provided outside the effective diameter of the taking lens with respect to the center of the taking lens, θ
The eye direction is illuminated at an angle of th1 ≧ θth2 + 1 °, and the eye direction is illuminated at an angle of θth2 ≦ 5 ° with respect to the center of the taking lens from a second light source provided near the optical axis of the taking lens. The reflected light from the first and second light sources is intermittently imaged by the first and second photographing means via the photographing lens, and the difference between the image signals output from each of them is obtained and output.

In the invention according to claim 2, while irradiating the direction of the eye from the first light source provided outside the effective diameter of the taking lens, the direction of the eye from the second light source provided near the optical axis of the taking lens. And the images based on the reflected light from the first and second light sources are separated by the prism optical system and individually given to the first and second photographing means, and output from the first and second photographing means. The difference between the image signals to be processed is calculated and output.

[Embodiment of the Invention] FIG. 1 is a diagram for explaining the concept of the present invention, and FIG. 2 is a diagram showing the types and characteristics of feature points necessary for line-of-sight detection.

First, the concept of the present invention will be described with reference to FIGS. 1 and 2. A face image is captured by the camera 60, and a feature point reflecting the head and a feature point of the eye are extracted from the image signal output from the camera 60. As shown in FIG. 2, the feature points that reflect the head can be detected by the difference in color between the outer corners of the eyes, inner corners of the eyes, lips, and the like. The characteristic points of the eye are the corneal reflex image, the black eye, and the pupil, and the corneal reflex image is a virtual image created by the light regularly reflected by the convex surface of the cornea, and moves in the same direction as the eyeball in proportion to the movement of the line of sight. There is. The black eye is observed due to the difference in reflection intensity between the iris and the sclera, and it has the property that it can be extracted by indoor lighting without using reference light.
The pupil is observed by the intensity difference of the reflected light from the retina at the iris and this opening, and has the property that it is not easily affected by the eyelids.

The position and direction of the head are detected from the extracted feature points that reflect the head, and the result is converted into the eyeball center position. On the other hand, the center of the pupil or the black eye is detected from the extracted eye feature points, and the direction of the line of sight is detected in response to the result and the converted eyeball center position.

In addition, the head is modeled. This modeling is performed by at least three feature points of the face with little skin movement on the face, and the position and direction of the head are detected from the positions of the feature points of each face.

Further, in modeling the head, at least three virtual feature points with little movement with respect to the skeleton of the head are obtained from the positions of at least four face feature points on the face, and from the positions of the virtual feature points. The position and orientation of the head are detected.

Further, a mark object whose feature extraction is easy is provided in at least three places around the frame of the eyeglass, and by wearing the eyeglass, the head is modeled with the mark object,
The position and direction of the head are detected from the position of the marked object.

FIG. 3 is a view showing an embodiment of the image taking apparatus of the present invention, FIG. 4 is a view seen from the front of the taking lens shown in FIG. 3, and FIG. 5 is a side view. Is.

Referring to FIG. 3, a lighting device 61 and a lens 65 are arranged in front of the camera unit 60. As shown in FIG. 4, the lighting device 61 includes a light emitting diode 66 arranged around the lens.
including. A linear polarizing plate 63 is arranged in front of the light emitting diode 66, and a visible cutoff filter 68 is arranged in front of the lens 65. The linear polarizing plate 63 polarizes the light of the light emitting diode 66, and the visible cutoff filter 68 blocks visible light emitted from external lighting such as a fluorescent lamp. Further, the illumination device 62 is arranged at a position away from the optical axis of the lens 65. The lighting device 62 includes a light emitting diode 64 and a linear polarization plate 67 provided in front of the light emitting diode 64.

The camera unit 60 includes three prisms 607, 608 and 609.
A wavelength separation surface 610 is formed between the prisms 607 and 608, and a half mirror 611 is arranged between the prisms 608 and 609. Further, a polarizing plate 615 and a CCD image pickup device 612 are arranged facing the light emission face of the prism 607, a CCD image pickup device 614 is arranged on the light emission face of the prism 608, and a light emission face of the prism 609 is arranged. Polarizer 616 and CCD
The image sensor 613 is arranged. Polarizing planes of the polarizing plates 615 and 616 are orthogonal to the linear polarizing plates 63 and 67 provided in front of the illumination devices 61 and 62. CCD image sensors 613 and 614
Is output to the difference image calculation means 617, and the corneal reflection image is extracted. The outputs of the CCD image pickup devices 612 and 613 are given to the difference calculation means 618 to extract the pupil.

FIG. 6 is a characteristic diagram showing the optical property of the optical component used in one embodiment of the present invention. Illumination device shown in FIG.
The light emitting diodes 64 and 66 used for 61 and 62 are near infrared rays which are not perceived by the human eye. In this example,
The light emitting diode 64 has a wavelength λ1 of 850 nm, and the light emitting diode 66 has a wavelength λ2 of 950 nm. The wavelength characteristics of these light emitting diodes 64 and 66 are shown in FIG. 6, and the characteristics of the visible cutoff filter 68 provided in front of the lens 65 are shown in FIG.
The sensitive characteristic of 614 is as shown in FIG. As is clear from FIG. 6, since the light-emitting diodes 64 and 66 have narrow half-widths of wavelengths, two lights emitted from the respective light-emitting diodes 64 and 66 can be obtained by using those whose central wavelengths are separated by about 100 nm. Do not interfere. Also, at this wavelength, it is hardly perceived by the eyes of the day. If it is perceived by the human eye, the emission wavelength may be shifted to the longer wavelength side. The silicon CCD image pickup devices 612, 613, 614 are
As shown in the figure, there is a sufficient sensitivity characteristic in this wavelength range.

FIG. 7 is an external perspective view showing an example of a lighting device in which a light emitting diode and a polarizing plate are integrated. Referring to FIG. 7,
A light emitting diode 72 is mounted on the substrate 71, and a frame 74 for mounting an optical element is attached to the substrate 71 via a spacer 73 at a constant distance. In this frame 74,
Optical elements such as polarizing plates and visible blocking / near infrared transmission filters
75 is installed. Further, a hole for a camera lens is formed in the substrate 71 and the optical element 75, and a cylinder 76 is provided to connect these two holes.

As described above, in the example shown in FIG. 7, by covering the periphery of the optical element 75 with the frame 74, the mechanical strength can be increased and distortion of the optical element 75 can be prevented. This structure is an optical element 75
Has anisotropy in the refractive index, and is particularly important when the characteristics change depending on the wavelength. That is, when strain such as bending is applied, a predetermined reference light may not be obtained. Further, the cylinder 76 prevents the reference light emitted from the light emitting diode 72 from directly entering the lens. That is, the tube 76 separates the reference light and the reflected light.

Next, the operation of the embodiment of the present invention will be described with reference to FIG. Light emitting diodes 64, 66 of lighting devices 61, 62
The lights respectively emitted by (1) and (2) are polarized by the polarizing plates 67 and 63, and are irradiated to a person, and the reflected light is incident on the prism 607 from the visible cutoff filter 68 via the lens 65. Here, the light of wavelength λ2 (950 mm) is separated by the wavelength separation film 610 and is incident on the CCD image pickup device 612 via the polarizing plate 615 to form an image. If the polarized light of the illumination by the illuminating device 62 is in the horizontal direction, the polarization direction of the polarizing plate 615 is shifted by 90 ° to the vertical direction, so that the image in which the specular reflection component is blocked on the photosensitive surface of the CCD image sensor 612 is cut off. That is, an image of the diffuse reflection component is formed. Since specular reflection occurs on the smooth surface of the subject, the specular component strongly maintains the optical properties of the light source.

On the other hand, the diffuse reflection component is modulated by the absorption / reflection characteristics of the object to be photographed, and for the extraction of facial feature points, pupils, black eyes, etc., diffuse reflection component is used to suppress noise and improve S / N. It is preferred to use images. On the other hand, the reflected light of the wavelength λ1 passes through the visible cutoff filter 68, the lens 65 and the prism 607, and is split by the half mirror 611 provided at the interface between the prisms 608 and 609. One of the split lights is incident on the CCD image sensor 613 through the polarizing plate 616 to form an image, and the other light is directly incident on the CCD image sensor 614 to form an image. Therefore, the output image of the CCD image sensor 614 includes both specular reflection components and diffuse reflection components, and the output image of the CCD image sensor 613 includes only diffuse reflection components.
Since the intensity of light passing through the polarizing plates 615 and 616 is halved, the split ratio of the half mirror 611 is set to 1: 2 so that the image intensities of the CCD image pickup devices 613 and 614 are almost the same. There is.

FIG. 8A is a diagram showing a diffuse reflection image photographed by an embodiment of the present invention, and FIG. 8B is a diagram showing an effect of feature point extraction. In one embodiment of the present invention, the blue mark attached to the face of a person was used as a feature point, and an experiment was conducted to extract this, and the principle of the present invention was confirmed. For the convenience of the experiment comparing with the conventional photographing method, a white light source (halogen light) was used as the light source instead of the light emitting diode, and the light was polarized and irradiated.
An ordinary three-plate color camera was used as a camera, and a polarizing plate whose polarization direction was orthogonal to that of the polarizing plate used for illumination was inserted on the image forming optical axis for photographing. FIG. 8A shows an example in which the diffuse reflection component is extracted from the reflected light captured by the camera by the above method. FIG. 8B is an image in which the blue component is extracted from this pixel and binarized. This FIG. 8B and the above-mentioned conventional example, 25B.
Comparing with the figure, it can be seen that the noise is small and the blue mark 77 is efficiently extracted as a feature point.

Next, extraction of the corneal reflection image will be described. The corneal reflection image is a virtual image created by specularly reflecting the light of the reference light source on the surface of the cornea. Therefore, in this case, the specular reflection image becomes a desired signal. The corneal reflection image is extracted by extracting the difference between the images of the CCD image pickup devices 613 and 614 in FIG. 3 by the difference calculating means 617. In the extraction of the corneal reflection image, the conventional problem has been that the light from the external surface or the display is reflected by the cornea to form an image and overlaps with the image of the reference light source as described above. However, in the embodiment of the present invention, since the light of the light emitting diode 64 is polarized, the corneal reflection image due to the reflection image of the light from the light emitting diode 64 is blocked by the polarizing plate 616 orthogonal thereto, and diffuse reflection is performed. The statue remains. Therefore, as described above, the difference calculation means 617 extracts the corneal reflection image from the background.

However, external lighting is generally unpolarized, so the CCD
Similar images are taken by the image pickup devices 613 and 614 and disappear when the difference is taken. In this way, the difference calculation means 61
Since the difference image obtained from 7 has little noise, it is possible to extract a corneal reflection image with high S / N.

9A to 9C are diagrams showing an example of extracting a corneal reflection image.
The examples shown in FIGS. 9A to 9C show the results of an experiment for extracting a corneal reflection image using the same imaging system as in FIG. 3 in order to confirm the above-mentioned principle. , FIG. 9A is an image obtained from the CCD image pickup device 614 and includes a specular reflection component and a diffuse reflection component. Fig. 9B shows CCD image sensor 6
The image obtained from 13 contains only diffuse reflection components. In the difference image output from the difference calculation means 617, it can be seen that the corneal reflection image is clearly extracted from the background as shown in FIG. 9C.

Next, the extraction of the pupil will be described. As shown in FIG. 3, the first lighting device 61 has a lens 65 as shown in FIG.
The light emitting diode 66 is arranged so as to surround the optical axis of. The reflection image by the illumination from this illumination device 61, that is,
In the image output from the CCD image sensor 612, the pupil is photographed brightly against the background. Although it cannot be said that the pupil enhancement for the background is sufficient by itself, it is possible to extract the pupil by limiting the application range. On the other hand, the second lighting device 62 is arranged away from the optical axis of the lens 65, and
62 reflected image by illumination, that is, CCD image sensor 613 or
The image obtained from 614 has a dark pupil. Therefore, the image obtained from the CCD image sensor 612 and the CCD image sensor 6
By the difference processing of the image obtained from 13 or the difference processing between the image obtained from the CCD image pickup element 612 and the image obtained from the CCD image pickup element 614, the pupil is further emphasized from the background, and extraction is facilitated. The second lighting device 62 may be replaced by external lighting such as a fluorescent lamp because the installation conditions are not so strict as described above. However, in this case, it is necessary for the reflected light of the illumination to reach the CCD image pickup device 613 or 614, and therefore external illumination must have a near-infrared component or the lens visible cutoff filter 68 must be removed. is there.

FIG. 10, FIG. 11A to FIG. 11C, and FIG. 12 are views for explaining the arrangement of the light sources for photographing the pupil brightly.

Next, regarding the relationship between the position of the light source and the brightness of the pupil,
This will be described in detail with reference to FIG. The eye 80 can be modeled as a composite sphere in which transparent spheres having different radii overlap each other with a center therebetween. Consider a case where light is emitted to such an eyeball 80 on the optical axis of the taking lens 86 and the reflected light is captured. The opening of the iris 81 of the eye 80 is the pupil 85. The light emitted from the light source 87 is refracted by the cornea 84, enters the eyeball 80, passes through the pupil 85, and reaches the diffuse reflection surface 82 corresponding to the retina. Some light is absorbed by the retinal cells, while the rest is diffusely reflected here. Of the reflected light, the light that can pass through the pupil 85 again returns to the direction of the light source. Therefore, if there is the taking lens 86 in this direction and the reflected light can be captured, the portion of the pupil 85 becomes a bright and bright image in the taken image.

However, when the taking lens 86 is separated from the light source 87, the reflected light cannot be sufficiently captured, and the bright and bright area in the pupil 85 portion has a peculiar shape. If the light source 87 is near the optical axis of the taking lens 86 as shown in FIG.
As shown in the figure, a bright and bright area in the pupil 85 is widened. However, when the light source 88 is separated from the taking lens 86, the bright and bright areas are largely biased, as shown in FIG. 11A. In this way, the pupil image with uneven brightness is binarized,
Even if the process of obtaining the center of gravity cannot be performed, the center of the pupil 85 cannot be accurately obtained, and the error in the line-of-sight detection increases. Therefore, from the viewpoint of improving the accuracy of line-of-sight detection, it is particularly desirable that the pupil 85 be photographed with uniform brightness.

Therefore, in one embodiment of the present invention, in order to uniformly and brightly photograph the entire area of the pupil 85, it is desirable to dispose the light source 87 near the optical axis of the photographing lens 86 and around the optical axis. I am trying. FIG. 11C schematically shows a pupil image photographed by using the illumination device 61 shown in FIG. The light emitting diode 66 shown in FIG.
As shown in FIG. 11A, it acts to brightly image a specific portion of the pupil 85, but the entire pupil image is photographed as the sum of the actions of the light emitting diodes 66 arranged around the optical axis. . Note that the light source 87 is located near the lens optical axis.
The arrangement can be realized only when a small and efficient light source such as the light emitting diode 66 is used.

Next, with reference to FIG. 12, the conditions for uniformly brightening the pupil will be described. The main parameters that influence the shooting conditions are as follows.

Eyeball position d, g relative to the lens center position
And the angle of rotation of the eyeball ξ The position of the light source Among the above parameters, the parameter of is determined by the environmental conditions using the eye gaze detection method. In the interface environment, the eyeball position d is about 50 cm to 200 cm, the eyeball position g is about ± 30 cm, and the rotation angle ξ is about ± 20 °. Under such a condition, as shown in FIGS. 11A and 11B, when an arrangement condition of light sources for brightly photographing at least a part of the pupil is obtained by an experiment, this condition connects each light source and an object to be photographed. The angle θ formed by the line segments l ₁ and l ₁ ′ and the line segments l ₂ and l ₂ ′ connecting the center of the lens and the eyeball was almost determined, and the value of θ was less than θth1 (5 °).

The angle at which the brightly photographed part of the pupil did not appear was θth2 (6 °) or more. In other words, the pupil is divided into bright and dark images with about 5 ° as a boundary. This θth1, θth2
The value of is set according to environmental conditions. Therefore, in the illumination device 61 for brightly photographing the pupil, each light source surrounding the lens optical axis may be set to θth1 or less. The larger the number of light sources, the better, but it is preferable that the number of light sources is three or more.

Under the above conditions, the pupil portion is emphasized with respect to the background by the difference image between the image in which the pupil is photographed brightly and the image in which the pupil is photographed darkly.

13A to 13C are diagrams showing an example of an image taken by using the apparatus shown in FIG. 3, and particularly, FIG. 13A is an image taken by the CCD image pickup device 612, and FIG. 13B. Is CCD
It is an image captured by the image sensor 614. When the difference image is binarized, it can be seen that the pupil part is extracted with a high S / N as shown in FIG. 13C.

In one embodiment of the present invention, the light reflected by the retina of the eyeball is extracted from the pupil to photograph the pupil brightly, but this technique can also be used for feature point extraction. That is, this is a method of using a transparent sphere for the mark in a method of detecting the position and direction of the face by attaching a mark to the face and extracting the mark.

FIG. 14 is a diagram showing an example of a mark object applied to the apparatus of one embodiment of the present invention. Referring to FIG. 14, a paint such as white paint is applied on the glass ball 90 to form a diffuse reflection layer.
91 is formed, and a paint such as black paint is further applied thereon to form the light absorption layer 92. The light enters through the opening of the glass ball 90, is refracted by the curvature of the rain opening, and reaches the diffusion surface. Although the incident light does not form an image on the diffusion surface, the curvature of the opening acts as a lens, so that the incident light is condensed as shown in FIG. When the light reflected by the diffusion surface passes through the opening, the curvature acts as a lens, so that the light is reflected generally in the direction of the incident light.

As described above, since the direction of reflected light is close to the direction of incident light, when the reflected light is photographed by the image pickup apparatus of the embodiment shown in FIG. 3, for example, the image obtained by the CCD image pickup element 612 has an aperture. The picture is taken brightly against the surroundings. In addition, shooting lens 65
Since the camera 60 cannot capture the reflected light from the illumination device 62 provided at a position away from, as in the case of the eyeball, the openings of the images of the CCD image pickup devices 613 and 614 are dark. Therefore, these two types of images, CCD
When the difference between the image of the image pickup device 612 and the image of the CCD image pickup device 613 or 614 is taken, the opening is emphasized as a feature point and extracted. In this way, the opening of the glass ball 90 can be extracted in the same manner as the condition for extracting the pupil. As the glass bulb 90, one having a diameter of about 3 to 5 mm is prepared, and it is attached to a part of the face where the skin does not move much or is embedded in a frame of an eyeglass frame so that the user can wear it. Good.

FIG. 15 is a sectional view showing another example of the marked object. 15th
Referring to the drawing, the mark object 93 has a circular shape with a diameter of about 2 to 5 mm and is formed by processing glass. A film 94 that transmits at a wavelength of 900 nm or more and reflects at a wavelength of 900 nm or less is formed on the front surface, and the back surface is a diffuse reflection surface 95. Therefore,
Light of 900 nm or more is diffusely reflected on the back surface and captured by the camera. Light of 900 nm or less is reflected on the surface of the mark object 93, but if the reflection direction is different from that of the photographing lens, it is not photographed. Since this light is specularly reflected light, it can be blocked by using a polarizing plate. In the image obtained by the CCD image pickup device 612 shown in FIG. 3, the mark portion becomes bright. Further, in the image obtained by the CCD image pickup device 613, the mark portion becomes dark. Therefore, the mark portion is extracted together with the pupil by taking these differences.

FIG. 16 is a diagram showing an embodiment in which the present invention is applied to an electronic shutter type photographing apparatus. For example, in FIG.
This corresponds to a camera in which the image sensor 614 is omitted and which has an electronic shutter function. That is, the image sensor 105 corresponds to the CCD image sensor 612 in FIG. 3, and the image sensor 105 is the CCD image sensor.
Equivalent to 613. Referring to FIG. 16, an electronic shutter camera 101 includes a lens 102, an electronic shutter 103, an optical device 104 for separating an image according to a wavelength, an image sensor 105 for capturing an image of wavelength λ ₁ , and a wavelength The image pickup device 106 for picking up an image of λ ₂ is included. The output of the image sensor 105 is given to the memory 107, and the image output of the image sensor 106 is given to the memory 108. The memories 107 and 108 store image pickup outputs, respectively.
The image processing device 109 processes the respective image outputs stored in the memories 107 and 108 and outputs, for example, a difference. The electronic shutter 103 and the image pickup devices 105 and 106 are controlled by a controller 110 including, for example, a microcomputer. The illuminating device 111 is an illuminating device 113 that emits light of wavelength λ _1.
And an illuminating device 115 that emits light of wavelength λ ₂ , each of which is driven by drive circuits 112 and 114 under the control of the controller 110.

FIG. 17 and FIG. 18 are diagrams showing drive timings of the image pickup device and the illumination device shown in FIG. The electronic shutter camera 101 shown in FIG. 16 described above can shoot a moving object with less blurring. The principle is that the electronic shutter 103 is opened for a short time to expose the image pickup devices 105 and 106, and the image signal is temporarily stored in the image pickup devices 105 and 106 during a short time 116 of, for example, 1 msec shown in FIG. 17 (b). Then the 17th
The image signals output from the image pickup devices 105 and 106 are transferred to the memories 107 and 108 during the timing 117 shown in FIG. 7A, the electronic shutter 103 is opened again at the timing when the transfer is completed, the image pickup devices 105 and 106 are exposed, and this operation is performed. Is repeated.

In the electronic shutter type camera 101 shown in FIG. 16 described above,
It is necessary to use high-sensitivity image pickup elements 105 and 106, but a larger amount of light is required as the shutter speed increases. In such a device, the lighting device 111
By using, the following advantages are obtained.

That is, not only can the light emitting diode be turned on and off at high speed, but under an intermittent driving condition, a larger current can be passed than in continuous driving, and thus a large amount of light can be obtained. As a result, sufficient light can be emitted during the exposure time of the image pickup devices 105 and 106. As described above, the illumination device 111 using the light emitting diode meets the condition that the electronic shutter camera 101 needs strong illumination in a short time.

A width of about 33 msec to 0.5 msec is set for one screen exposure time of the image pickup devices 105 and 106. Here, for example, when taking 30 images per second at the minimum exposure time of 1 msec or 0.5 msec, the irradiation time is longer than that of continuous irradiation.
It becomes 1/1000 or less. In this case, even a light emitting diode that can only supply about 50 mA in continuous energization can energize a current of several A by intermittent driving. Therefore, the illumination required for exposure can be secured.

In the image pickup apparatus according to the present invention, it is preferable to shut off the external illumination because it causes noise. The effect of shortening the exposure time is that the light quantity of external illumination is relatively small with respect to the light quantity of the lighting device. As a result, the reflection component due to external illumination is relatively small, the noise component is small, and the S / N is improved. Also, since the necessary amount of light is illuminated only during the required illumination time, wasteful power is reduced and heat generation is reduced.

Next, as another example of intermittent driving, another method of removing the reflection component due to external illumination will be described.

FIG. 18 is a timing diagram for driving the lighting device. In this example, the images are taken in a time-division manner by a set of two images, and the image taking time for one screen is, for example, 33 msec.
Shown in 2. When shooting A1, the lighting system is turned on and A
The illumination device is turned off when shooting 2. Reflected light from external illumination is superimposed on both images A1 and A2. If the external lighting conditions do not change during A1 and A2 shooting, from the A1 image to A
By subtracting the two images, the image when the lighting device is turned on can be extracted.

Next, as an application example of the present invention, an image processing apparatus for extracting the feature points of the photographing target in real time, for example, 30 screens / second will be described. When trying to detect human movements in real time, it is necessary to select a shooting method that makes it easy to extract feature points, such as putting a mark on the face or body or applying reference light, as described above. This alone is not enough, and it is essential to speed up the image processing apparatus.
In the present invention, the above-mentioned feature points can be easily extracted by the difference processing of two types of images.

FIG. 19A is a schematic block diagram showing an overall configuration of an image processing apparatus for detecting feature points in real time by using the image capturing apparatus according to the present invention, and FIG. 19B is a schematic block diagram of the image processing apparatus shown in FIG. 19A. It is a concrete block diagram.

Referring to FIG. 19A, the image signals of CCD1 to CCD3 of each camera photographed by near infrared cameras 121 and 122 are applied to image processing devices 123 and 124, respectively, subjected to image processing and applied to host computer 125. The image processing devices 123 and 124 are each configured as shown in FIG. 19B. That is, the image processing apparatus 123 is basically based on the module 130 that generates a timing signal for pipeline processing from an input signal, and other input modules 131, 132 and various pipeline processor modules 133, 134, 135 and 136 connected to this module 130 operate in parallel. To do. Near infrared camera
121 includes CCD image sensors 612, 613 and 614, and the respective image signals are individually applied to the modules 130, 131 and 132.

Modules 130, 131 and 132 are pixel data by A / D converting the image signals in synchronization with the respective output signals of the CCD image pickup devices 612, 613 and 614, and the pixel data are sequentially pipeline processor modules 133, 134, via a video bus.
Given to 135 and 136. Each of the pipeline processor modules 133, 134, 135 and 136 has a plurality of inputs. For example, in the case of two types of differential image processing, the pipeline processor module 133 performs the following processing (image of CCD614-image of CCD613) and binarization. The pipeline processor 134 (image of CCD612-CCD
613 image) and binarization processing are performed, and the pipeline processor 135
The image signals processed in and are packed, and the pipeline processor 136 calculates the center of gravity of the extracted feature points.

The calculated barycentric coordinates are transmitted to the host computer 125 via the repeater 138, and the line of sight is detected. In this way, while one image is being captured, the A / D converted pixel data can be converted into multiple pipeline processor modules.
Calculated through 133-136. The image processing time for one image is 33 msec, which is the time until the last pixel is captured, plus the delay time for pipeline processing. Also, the input modules 130, 131 and 132 can be ready for the next image capture after sending to the last image processor. Therefore, as in the case of the object of the present invention, in the case of image processing (difference processing between images) which can be performed by simple four arithmetic operations, real-time processing can be performed with 33 sheets / second and a delay time of 33 msec + α.

Next, as an application example other than line-of-sight detection, an application to active stereo vision, in which a shape pattern is irradiated as a reference light to a shooting target and a three-dimensional shape of the shooting target is obtained from a change in a reflection pattern corresponding to the irradiated pattern, will be described. To do.

20, 21, and 22 are diagrams showing an example in which the present invention is applied to an active stereo image measuring method. 20th
Referring to the drawing, a lighting device using a light source of a light emitting diode
141 to lens system 142, shape pattern 143 and polarizing plate 144
The reference light is applied to the imaging target 145 via. The shooting target is, for example, the face of a person. 21st to shape pattern 143
A pattern 145 having a pattern as shown in the figure is formed,
The reference light from the light source 141 couples the pattern 146 of the shape pattern 143 to the object 145 to be photographed by the lens system 142. Target 1
The reflected light from 45 passes through the taking lens 151 and the polarizing plate 152 and is imaged on the image sensor 153. The polarization plane of the polarizing plate 152 is assumed to be rotated by 90 ° with respect to the polarization plane of the polarizing plate 144.

In FIG. 22, the pattern projected on the object 145 is the image sensor 1.
It is an image taken by 53. Generally, 14
5 has a partially smooth surface, and the reference light is specularly reflected here, so in the conventional method that does not use polarized light, the projection image shown in FIG. 29 may interfere with the projection pattern. A specular reflection image appears. However, in this method, in which the reference light is polarized and irradiated, and the specular reflection component is blocked from the reflected light to shoot, the noise component due to the specular reflection image is small, so as shown in FIG. A shape pattern image reflecting the characteristic information can be efficiently extracted.

[Effects of the Invention] As described above, according to the present invention, the pupil can be emphasized for photographing. This means that even if the lighting is weakened, it is resistant to noise. Although strong illumination has a physiological effect and is not desirable, according to the present invention, a pupil can be stably extracted even with weak illumination. Therefore, a human-friendly device can be provided when detecting a line of sight. That is, it is possible to realize a device that does not get tired even if it is used for a long time.

[Brief description of drawings]

FIG. 1 is a diagram for explaining the concept of the present invention. FIG. 2 is a diagram showing types and characteristics of feature points necessary for line-of-sight detection. FIG. 3 is a diagram showing an embodiment of the image capturing apparatus of the present invention. FIG. 4 is a front view of the taking lens shown in FIG. FIG. 5 is a view seen from the same side. FIG. 6 is a characteristic diagram showing the optical property of the optical component used in one embodiment of the present invention. FIG. 7 is an external perspective view showing an example of a lighting device in which a light emitting diode and a polarizing plate are integrated. FIG. 8A is a diagram showing a diffuse reflection image photographed by an embodiment of the present invention, and FIG. 8B is a diagram showing an effect of feature point extraction. 9A to 9C are diagrams showing an example of extracting a corneal reflection image. FIG. 10, FIG. 11A to FIG. 11C, and FIG. 12 are views for explaining the arrangement of the light sources for photographing the pupil brightly. FIGS. 13A to 13C are diagrams showing an example of an image taken by using the apparatus shown in FIG. FIG. 14 is a diagram showing an example of a mark object applied to the apparatus of one embodiment of the present invention. FIG. 15 is a sectional view showing another example of the marked object. FIG. 16 is a diagram showing an embodiment in which the present invention is applied to an electronic shutter type photographing apparatus. FIG. 17 and FIG. 18 are diagrams showing drive timings of the photographing device and the illumination device shown in FIG. FIG. 19A is a schematic block diagram showing the overall configuration of an image processing apparatus for detecting feature points in real time using the image capturing apparatus according to the present invention, and FIG. 19B is a specific example of the image processing apparatus shown in FIG. 19A. It is a typical block diagram. 20, 21, and 22 are diagrams showing an example in which the present invention is applied to an active stereo image measuring method. FIG. 23 is a diagram showing an example of a conventional non-contact visual line detection device. FIG. 24 is a diagram showing an example in which an experiment for extracting a blue mark matching a person's face as a feature point was performed using a conventional lighting device and a photographing device. 25A and 25B are diagrams showing an example in which the feature points of the face are extracted by the experiment shown in FIG. FIG. 26 is a diagram showing another method of extracting the pupil, in which light is incident from the pupil and light reflected by the retina is captured. FIG. 27 is a diagram showing an example of an apparatus for photographing a corneal reflection image with a camera. FIG. 28 is a diagram for explaining a method of detecting the position of the black eye in the white eye and converting it into the line of sight on the captured image. FIG. 29 is a diagram for explaining a method of extracting with one camera using a pupil and a corneal reflection image as feature points. In the figure, 60 is a camera unit, 61, 62, 111, 113, 115
Is a lighting device, 64, 66, 72 are light emitting diodes, 63, 67, 615, 61
6 is a polarizing plate, 65 and 86 are lenses, 68 is a visible cutoff filter, and 75
Is an optical element, 87 is a light source, 105 and 106 are image pickup elements, 107 and 108 are memories, 109, 123 and 124 are image processing devices, 110 is a controller, 112 and 114 are drive circuits, 121 and 122 are near-infrared cameras, and 125.
Is a host computer, 130, 131, 132 are modules, 15
3,154,155,156 are pipeline processor modules, 6
07, 608 and 609 are prisms, 610 is a wavelength separation film, 611 is a half mirror, 612, 613 and 614 are CCD image pickup devices, and 617 and 618 are difference image calculation means.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification number Internal reference number FI Technical display area A61B 3/10 B (56) References JP 62-149252 (JP, A) JP 60 -197063 (JP, A) JP-A-58-114667 (JP, A) JP-A-61-268226 (JP, A)

Claims (2)

[Claims] 1. A pupil image photographing device for photographing a pupil image, the first light source being provided outside an effective diameter of a photographing lens and illuminating the direction of an eye, and reflected light of the first light source. A first photographing means for forming an image through the photographing lens and outputting a first image signal; a second light source which is provided near the optical axis of the photographing lens and illuminates an eye direction; A second image capturing unit that forms an image of reflected light of the second light source through the image capturing lens and outputs a second image signal, and a differential signal of the image signals output from the first and second image capturing units. The second light source includes a processing unit that outputs as a third image signal, and the second light source forms an angle between a line segment that connects the light source and an eyeball that is an imaging target, and a line segment that connects the center of the imaging lens and the eyeball. Is set to θth2, θth2 ≦ 5 ° is installed. The light source is installed so that an angle formed by a line segment connecting the light source and the eyeball to be photographed and a line segment connecting the center of the photographing lens and the eyeball is θth1 ≧ θth2 + 1 °. The first and second photographing means are configured to operate the first and second photographing means.
2. A pupil image photographing device, characterized by alternately and intermittently photographing images based on reflected light from the light source. 2. A pupil image photographing device for photographing a pupil image, the first light source being provided outside an effective diameter of a photographing lens and irradiating the direction of an eye, and reflected light from the first light source. A first photographing means for forming an image through the photographing lens and outputting a first video signal; a second light source provided near the optical axis of the photographing lens for illuminating the direction of the eye; A second image capturing unit that forms an image of reflected light of the second light source through the image capturing lens and outputs a second image signal, and a differential signal of the image signals output from the first and second image capturing units. A processing unit for outputting as a third image signal is further provided, and an image based on the reflected light from the first light source and an image based on the reflected light from the second light source is incident on the taking lens. Separated from each other and individually for the first and second photographing means. A pupil image including a prism optical system for giving light, wherein the first and second optical systems are configured by light emitting diodes whose emission wavelengths are different from each other so as to be separable by the prism optical system. Imaging device.