JP2009147708A - Image pickup device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image pickup device which outputs image the pickup information of a subject and distance information simultaneously from a sensor, and sends a signal inputted by an audio terminal in the vicinity of the subject to a body image pickup device as an infrared signal. <P>SOLUTION: An irradiation light from an infrared-emitting device captures reflected light from the subject on a far-red light absorbing film of each pixel of the sensor by using the sensor having films absorbing RGB and a film absorbing far-red light, and the infrared-emitting device attached near the sensor. Thus, when a distance to the subject exceeds a fixed value at the same time as the distance information is obtained, audio information in the vicinity of the subject inputted into the audio terminal provided in the subject is modulated to the infrared signal and sent to the image pickup device as complementary audio information. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging device having an electromagnetic wave absorbing film that absorbs visible light and a near-infrared absorbing film that absorbs near-infrared light, and particularly receives an audio signal transmitted by near-infrared light via an audio terminal device. The present invention relates to an imaging apparatus.

  In recent years, video cameras and digital still cameras have also been able to output image information and distance information at the same time in order to achieve high-definition images and improve focus accuracy and strobe light emission adjustment accuracy. The demand for equipment is also increasing.

  Furthermore, in order to reduce the size of such an image pickup apparatus, there is a demand for a single image pickup element that can use incident light more effectively in place of a conventional color filter instead of a mosaic image pickup element (Bayer array).

  In view of such a situation, for each pixel unit of the solid-state imaging device, RGB, which are the three primary colors of light, and an electromagnetic wave absorption film that absorbs light of each wavelength are laminated to improve the utilization efficiency of incident light. A technique for realizing high resolution without increasing the size of the image sensor has been proposed (Patent Document 1).

  There has also been proposed an imaging apparatus that simultaneously acquires a high-resolution image and distance information to a subject using two imaging elements (Patent Document 2).

Also, it has an audio input device such as a microphone, modulates the input audio signal, emits it into the space as an infrared ray optical signal, images the optical signal simultaneously with the image of the subject, An image pickup apparatus that can record a voice and a human image has been proposed (Patent Document 3).
JP 2003-234460 A Japanese Patent Laid-Open No. 11-41521 JP-A-63-256061

  As described above, in a technique for realizing a high resolution by multilayering an electromagnetic wave absorption film that absorbs the wavelengths of RGB, which are the three primary colors of light, for each pixel unit, a mosaic-shaped color filter type solid-state imaging device and In comparison, although the light utilization efficiency is improved three times, high resolution can be realized, but on the other hand, the electromagnetic wave absorbing film corresponding to the three primary colors RGB of light has a laminated structure, so Was the output of only image information.

  In the method of simultaneously acquiring high-resolution image quality and distance information using two image sensors, the image sensor for image and the image sensor for distance information acquisition pass through a half mirror that separates incident light from one lens. Then, there is a method in which light is incident and an image and distance information are acquired by one optical device such as a lens.

  However, since the incident light is separated from one optical device via the half mirror, the mechanical attachment accuracy of the half mirror is required so that the incident angle of the light to the two image sensors does not deviate. Since two image sensors and half mirrors are used, there is a problem that it is difficult to reduce the size and power consumption of the image pickup apparatus, and further increase the cost.

In Patent Document 3, a sound signal input from a sound input device of a microphone is modulated and emitted into a space as an infrared light signal, and the light signal is imaged at the same time as an image of a subject. For example, an imaging device capable of recording a person's voice and person image has been disclosed.
In this case, since the distance to the subject cannot be determined, the voice is not picked up when the subject D is located outside the screen as in the subject D of FIG. 5, for example. Since audio levels equivalent to those of the subjects A, B, and C are recorded at the time, the audio becomes smaller in proportion to the distance of the subject, whereas there are cases where the video and audio have a sense of incongruity.

The imaging device provided with the voice terminal device of the present invention,
From visible light, first to third electromagnetic wave absorbing films that can absorb R, G, and B, which are the three primary colors of light, and electromagnetic waves that are not absorbed by the first to third electromagnetic wave absorbing films, near infrared light A multilayer laminated imaging device having a fourth electromagnetic wave absorbing film capable of absorbing an electromagnetic wave corresponding to the wavelength, and an infrared light emitting device installed in the vicinity of the imaging device, and absorbed by the first to third electromagnetic wave absorbing films The object image is taken from the charge, and the reflected light from the infrared light emitting element is reflected by the fourth electromagnetic wave absorbing film, and the distance information to the subject is calculated from the time difference between the irradiated light and the reflected reflected light. An imaging apparatus comprising: means for outputting simultaneously a subject image and distance information corresponding to the subject image,
An audio terminal device that modulates an audio signal and emits it with near-infrared rays is provided on a subject. Based on the near-infrared rays output from the audio terminal device and the near-infrared rays emitted from the imaging device In addition, the sound input / output means of the imaging apparatus is switched.

Moreover, the further characteristic of the imaging device provided with the audio | voice terminal device of this invention is the following.
The first electromagnetic wave absorbing film has a characteristic of absorbing an electromagnetic wave of 400 to 500 nm, the second electromagnetic wave absorbing film has a characteristic of absorbing an electromagnetic wave of 500 to 600 nm, and the third electromagnetic wave absorbing film is 600 to 700 nm. In addition, the fourth electromagnetic wave absorbing film has an absorption characteristic equivalent to the wavelength of near-infrared light, particularly an infrared light-emitting element installed near the imaging element, It has the 1st-4th electromagnetic wave absorption film | membrane characterized by having the absorption characteristic equivalent to the wavelength of near-infrared light, It is characterized by the above-mentioned.

As described above, according to the present invention, the solid-state imaging device includes an electromagnetic wave absorbing film that absorbs the three primary colors RGB of light, and an electrode for reading out charges from the film,
From the electromagnetic wave that is not absorbed by the RGB electromagnetic wave absorbing film in the lowermost layer portion, a fourth electromagnetic wave absorbing film that absorbs near-infrared wavelengths, and an electrode for reading out the charge absorbed by the absorbing film,
Irradiate near-infrared rays from an infrared light emitting element disposed near the image sensor, and absorb reflected light from a target subject by a fourth electromagnetic wave absorbing film included in the image sensor disposed for each pixel;
By converting the absorbed electromagnetic wave into a signal and reading out from the connected electrode, the irradiation light from the infrared light emitting element is provided with means for calculating the arrival time for each pixel of the reflected light through the subject,
Means for calculating distance information for each pixel corresponding to the subject image from the arrival time for each pixel;
The distance information for all pixels is read simultaneously after obtaining the distance information for each pixel, and includes means for generating distance information corresponding to the subject image,
Means for synthesizing the distance information and the main image of the subject image;
A recording means for recording the synthesized image as image data, for example, RAW data;
The first to third electromagnetic wave absorbing films provided in the solid-state imaging device by the above means generate an object image by absorbing electromagnetic waves corresponding to visible light,
By detecting the light reflected from the subject by the irradiation light from the infrared light emitting element with the fourth electromagnetic wave absorbing film provided in the imaging device, the distance information to the subject is acquired,
The voice terminal device provided for the subject includes means for inputting the voice of the subject and modulating and outputting with near infrared rays,
By including means for demodulating the audio signal input by the imaging device and adding it to the audio signal, and having means for changing the addition ratio of the audio signal based on the information obtained by calculating the distance to the subject, It is now possible to change the size of the sound in proportion to the distance to the subject according to the addition ratio, and to obtain images and sounds that do not feel uncomfortable.

  Next, details of the present invention will be described in accordance with the description of the embodiments.

  A mode for carrying out the present invention will be described.

  FIG. 1 is a drawing to which the present invention is applied. First, specific operations of the imaging apparatus of the present invention will be described with reference to FIG.

  In the figure, an optical signal that has passed through a lens 1 is input to a CMOS sensor 3 via a diaphragm 2, and the CMOS sensor is driven by a timing pulse output from a TG 7 to photoelectrically convert the incident optical signal. Thus, an electrical signal is converted and output.

  The TG 7 is controlled by the CPU 8. For example, the TG 7 outputs a timing pulse having an amplitude of 3V in synchronization with the horizontal synchronization signal HD at a constant cycle.

  The output signal of the CMOS sensor 3 is composed of “S” indicating a signal level and “N” indicating a noise level, and “S−N” indicates a value obtained by converting incident light into an electrical signal.

  The output signal of the CMOS sensor 3 is input to an AFE 4 composed of a CDS that performs correlated double sampling for calculating the difference value of “SN” and an A / D converter that converts an analog signal into a digital signal. After being converted into a digital signal, it is input to the multiplex 5.

  Charges converted from electromagnetic waves absorbed by each layer in the multiplex 5 and the CMOS sensor are synthesized, and an output signal is input to the camera signal processing circuit 9 via the AGC 6.

  The camera signal processing circuit 9 performs YC separation, video signal processing, color signal processing, and gamma correction to generate an image.

  The output of the camera signal processing circuit 9 is input to the video signal processing circuit 10, performs video signal processing and outputs a signal, and the monitor 11 displays an image.

  In addition, the camera signal processing circuit 9 outputs an evaluation value for performing exposure correction of the imaging device to the luminance level detection unit 12.

  The luminance level detection unit 12 calculates a luminance level detection value from the evaluation value and outputs it to the CPU 8.

  The CPU 8 calculates the difference value of the exposure correction amount from the output correction table prepared in advance from the output signal from the luminance level detector 12, and if there is a difference, controls the aperture 2, the CMOS sensor 3, and the AGC 6 to control. A signal is output and exposure compensation is performed.

  The audio signal input to the main body built-in microphone 20 is amplified by the amplifier 21, subjected to signal processing by the audio circuit 22, superimposed on the video signal by the video signal processing circuit 10, and output to the monitor 11.

  The audio modulation signal portion is extracted by the multiplex circuit 5 and demodulated into an audio signal by the audio demodulation circuit 23. The demodulated audio signal is superimposed on the video signal by the video signal processing circuit 10 and output to the monitor 11.

  Next, the audio signal terminal device 30 will be described. The audio signal terminal device 30 includes a built-in microphone 31, a modulation circuit 32, an infrared diode drive amplifier 33, an infrared light emitting diode 34, and an oscillator 35. Modulation is performed by the modulation circuit 32 at the oscillation frequency.

  The modulated audio signal is amplified by an infrared light emitting diode drive amplifier 33, converted into infrared rays by an infrared light emitting diode 34, and emitted as infrared rays.

  In the audio signal terminal device, an audio signal modulated into infrared light is input to the CMOS sensor 3 through the lens 1. Among them, the infrared ray including the audio signal is separated from the video signal by the AFE 4 together with the other infrared light, and is input from the multiplex circuit 5 to the audio demodulation circuit 23, and from the signal modulated by the audio demodulation circuit 23. The audio signal is taken out, adjusted in level by the audio amplification control amplifier 24 under the control of the CPU 8, then superimposed on the video signal by the video signal processing circuit 10 and output to the monitor 11.

  Next, the charge readout configuration from the multilayer laminated film included in the CMOS sensor 3 will be described with reference to FIGS. 2, 3, and 4.

  First, FIG. 2 shows the wavelength of an electromagnetic wave input to the CMOS sensor 3 and the transmittance of each electromagnetic wave absorbing film. The input signal wavelength is, for example, an optical LPF (Low) inserted between the lens 1 and the CMOS sensor 3. An electromagnetic wave wavelength of 400 nm or more is input due to ultraviolet (UV) cut characteristics provided in the Pass Filter.

  Next, in the electromagnetic wave absorbing film having a multilayer structure that constitutes the CMOS sensor 3, the first electromagnetic wave absorbing film is formed of an organic film that absorbs an electromagnetic wave having an electromagnetic wave wavelength of 400 to 500 nm of incident light.

  The first electromagnetic wave absorbing film is connected to an electrode 300 for converting electromagnetic waves absorbed by the absorbing film into electric charges and reading out the converted electric charges.

  The electric charge read from the electrode 300 is output to the AFE 4 as DATA1 through the AMP 304.

  DATA1 is subjected to correlated double sampling and A / D conversion by CDS 400 inside AFE 4 and A / D 404.

  On the other hand, electromagnetic waves other than the electromagnetic wave absorbed by the first electromagnetic wave absorbing film pass through the first electromagnetic wave absorbing film and permeate the second electromagnetic wave absorbing film.

  The second electromagnetic wave absorbing film is formed of an organic film that absorbs an electromagnetic wave having an electromagnetic wave wavelength of 500 to 600 nm of incident light.

  Connected to the second electromagnetic wave absorbing film is an electrode 301 for converting electromagnetic waves absorbed by the absorbing film into electric charges and reading out the converted electric charges.

  The electric charge read from the electrode 301 is output to the AFE 4 as DATA 2 via the AMP 305.

  DATA 2 is subjected to correlated double sampling and A / D conversion by CDS 401 inside AFE 4 and A / D 405.

  On the other hand, electromagnetic waves other than the electromagnetic wave absorbed by the second electromagnetic wave absorbing film pass through the second electromagnetic wave absorbing film and permeate the third electromagnetic wave absorbing film.

  The third electromagnetic wave absorbing film is formed of an organic film that absorbs an electromagnetic wave having an electromagnetic wave wavelength of 600 to 700 nm of incident light.

  The third electromagnetic wave absorbing film is connected to an electrode 302 for converting electromagnetic waves absorbed by the absorbing film into electric charges and reading out the converted electric charges.

  The electric charge read from the electrode 302 is output to the AFE 4 as DATA 3 via the AMP 306.

  DATA3 is subjected to correlation double sampling and A / D conversion by CDS 402 inside AFE and A / D 406.

  Next, a mechanism for calculating distance information to a subject using an LED arranged near the image sensor will be described with reference to FIGS. 3, 4, and 5. FIG.

  During imaging of a subject by the imaging apparatus 1000, infrared light having a wavelength of 800 to 1200 nm is irradiated from the LED 1004 constituting the imaging unit 1002 to the imaging subject 1001 (irradiation light 1005).

  Irradiation light 1005 is applied to a person or an object projected in the photographic subject 1001, and when there is a person or object at a distance that the irradiation light 1005 reaches, the irradiation light 1005 depends on the person or object. Is reflected (reflected light 1006).

  The reflected light 1006 is incident on the image sensor 1003 as an optical signal of the photographic subject 1001.

  Since the reflected light 1006 incident on the image sensor 1003 is a signal having a wavelength of 800 to 1200 nm, it passes through the first to third electromagnetic wave absorbing films included in the image sensor without being absorbed and is arranged in the lowermost layer. It penetrates into the formed fourth electromagnetic wave absorbing film.

  Since the fourth electromagnetic wave absorbing film is formed of an organic film that absorbs an electromagnetic wave having an electromagnetic wave wavelength of 800 to 1200 nm of incident light, the reflected light 1006 is absorbed by the absorbing film and converted into electric charges.

  The converted charge is output to the AFE 4 as DATA 4 through the charge reading electrode 303.

  DATA 4 is subjected to correlated double sampling and A / D conversion in CDS 403 inside AFE and A / D 407, and is output to multiplex 5.

  Next, a method for calculating the distance to the subject by the irradiation light from the LED and the reflected light from the subject will be described with reference to FIGS.

  FIG. 6 shows, for example, pixel information of an imaging element included in the imaging device, and all pixels have address information.

  FIG. 7 shows an LED irradiation timing for acquiring distance information and a distance calculation method, and the irradiation pulse timing of the LED starts irradiation within the V ranking period in the video signal.

  The pulse emitted from the LED is incident on each pixel of the image sensor via the subject, and the arrival time of the reflected pulse is detected in each pixel.

  Here, as a method for detecting the arrival time of the reflected pulse, a time difference is calculated by comparing the rising edge portion of the irradiation pulse with the rising edge portion of the reflected pulse.

  In FIG. 7, the reflected pulse first reaches the sensor 23 pixel and the sensor 33 pixel, and the time difference Φ1 is calculated by comparing the pulse with the irradiation pulse.

  Next, the reflected pulse reaches the sensor 13 pixel, the sensor 22 pixel, and the sensor 32 pixel, and the pulse is compared with the irradiation pulse to calculate the time difference Φ2.

  Next, the reflected pulse reaches the sensor 12 pixel, and the pulse is compared with the irradiation pulse to calculate the time difference Φ3.

  Finally, since the distance to the subject is very far, the reflected pulse does not reach the sensor 11 pixel, the sensor 21 pixel, and the sensor 31 pixel.

  As described above, the arrival time of the reflected pulse is detected in all the pixels included in the image sensor, and the distance information to the subject is calculated in all the pixels from the arrival time.

  Here, the method of calculating the distance information from the detected arrival time is calculated by converting the flight time from the arrival time of the reflected light and multiplying the flight time by the speed of light.

  In addition, the readout frequency of the pixels included in the image sensor is as follows.

F = 1 / (X / A * C) (1)
C: Speed of light X: Distance detection accuracy A: Number of pixels of the image sensor The distance information is obtained for all pixels of the image sensor by driving the image sensor at the pixel readout frequency calculated by the equation (1). Can be calculated.

  As described above, the subject image is output as charges from the first to third electromagnetic wave absorption films included in the imaging element, and the distance information to the subject is output as charge from the fourth electromagnetic wave absorption film included in the imaging element. The

  Each output charge is converted into a digital signal via the AFE and then input to the camera signal processing circuit via the multiplex and AGC gain.

  The camera signal processing circuit synthesizes the subject image and distance information to the subject, outputs them to the camera data memory, and records them as RAW data.

  As a result, an image pickup apparatus capable of simultaneously outputting a subject image and distance information to the subject without affecting the subject image at the time of photographing by one solid-state image sensor and an LED disposed near the image sensor. Can be realized.

  FIG. 8 is a diagram showing an embodiment of a means for calculating the distance information to the subject while preventing the influence on the subject image in the imaging apparatus of the present invention.

The structure of one Example in the imaging device of this invention is shown. One Example of the electromagnetic wave wavelength and the transmittance | permeability in each charge absorption film of the solid-state image sensor which the imaging device of this invention has is shown. An example of each charge absorption film and electrode configuration of a solid-state imaging device included in the imaging device of the present invention is shown. 1 shows an embodiment of an internal circuit configuration of an AFE included in an imaging apparatus of the present invention. In the imaging apparatus of the present invention, an embodiment of a distance calculation method using irradiation light and reflected light from an LED will be shown. In the imaging device of the present invention, an embodiment for representing position information of all pixels of the imaging device will be shown. In the imaging apparatus of the present invention, an example of timing for irradiating the LED within the video V blanking period and distance information calculating means for each pixel is shown. In the imaging apparatus of the present invention, an example of means for preventing the influence on the subject image and calculating the distance information to the subject is shown.

Explanation of symbols

1 Lens 2 Aperture 3 CMOS sensor 4 AFE
5 Multiplex 6 AGC
7 TG
8 CPU
DESCRIPTION OF SYMBOLS 9 Camera signal processing circuit 10 Video signal processing circuit 11 Monitor 12 Luminance level detection 13 Memory 14 Camera data memory 20 Microphone 21 Amplifier 22 Audio | voice processing circuit 23 Audio | voice demodulation circuit 24 Audio | voice amplification control amplifier 30 Audio | voice terminal device 31 Microphone 32 Modulation circuit 33 Amplifier 34 Infrared light emitting diode 35 Oscillator circuit 300 Electrode 300
301 Electrode 301
302 Electrode 302
303 Electrode 303
304 AMP304
305 AMP305
306 AMP306
307 AMP307
1000 Imaging Device 1001 Shooting Subject 1002 Imaging Unit 1003 Imaging Element 1004 LED
1005 Irradiation light 1006 Reflected light

Claims (2)

From visible light, first to third electromagnetic wave absorbing films that can absorb R, G, and B, which are the three primary colors of light, and electromagnetic waves that are not absorbed by the first to third electromagnetic wave absorbing films, near infrared light A multilayer laminated imaging device having a fourth electromagnetic wave absorbing film capable of absorbing an electromagnetic wave corresponding to the wavelength, and an infrared light emitting device installed in the vicinity of the imaging device, and absorbed by the first to third electromagnetic wave absorbing films The object image is taken from the charge, and the reflected light from the infrared light emitting element is reflected by the fourth electromagnetic wave absorbing film, and the distance information to the subject is calculated from the time difference between the irradiated light and the reflected reflected light. And an image pickup apparatus comprising: means for outputting simultaneously a subject image and distance information corresponding to the subject image;
An audio terminal device that modulates an audio signal and emits it with near-infrared rays is provided on a subject. Based on the near-infrared rays output from the audio terminal device and the near-infrared rays emitted from the imaging device And an audio input / output means of the imaging apparatus.   The first electromagnetic wave absorbing film has a characteristic of absorbing an electromagnetic wave of 400 to 500 nm, the second electromagnetic wave absorbing film has a characteristic of absorbing an electromagnetic wave of 500 to 600 nm, and the third electromagnetic wave absorbing film is The fourth electromagnetic wave absorbing film has a characteristic of absorbing electromagnetic waves of 600 to 700 nm, and the fourth electromagnetic wave absorbing film has an absorption characteristic equivalent to the wavelength of near-infrared light, in particular, an LED installed near an image sensor, and the voice terminal The imaging apparatus according to claim 1, wherein the imaging apparatus has an absorption characteristic equivalent to a wavelength of near infrared light of the apparatus.

Images