JP2011147469A - Biological information acquisition device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a biological information acquisition device which can be easily used in daily life. <P>SOLUTION: This invention provides a cellular phone which can perform skin moisture measurement, skin photographing, vein pattern detection, pulse wave detection and blood oxygen saturation measurement, etc., on the basis of the received light information of a photoelectric conversion part having a plurality of light receiving parts provided with quantum efficiency ≥60% in a visible light region and an infrared light region respectively, and regularly arrayed in order to pick up images by the light of a plurality of wavelength regions. The erroneous measurement due to sweat or the like is prevented. One kind of a light emitting diode is used also as a light source for detecting the plurality of wavelength regions in the photoelectric conversion part. The wavelength region of a solar blind region is adopted. A focus is determined on the basis of the images of the plurality of wavelength regions. Display brightness is adjusted by the output of the photoelectric conversion part. A display backlight is used also for imaging illumination. When a measuring wavelength light source is lighted, the backlight is put out for skin proximity detection. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

The present invention relates to a biological information acquisition apparatus.

  Various proposals have been made regarding biometric information acquisition apparatuses, and interest from the aspect of health check is increasing. For example, various skin moisture detection devices are proposed in Japanese Patent Application Laid-Open No. 11-19060 (Patent Document 1) and Japanese Patent Application Laid-Open No. 2003-169788 (Patent Document 2).

Japanese Patent Laid-Open No. 11-19060 JP 2003-169788 A

However, there are many problems that should be further studied in order to provide an easy-to-use biometric information acquisition apparatus for daily life.

In view of the above, an object of the present invention is to propose a biological information acquisition apparatus that can be easily used in daily life.

In order to achieve the above object, the present invention is detected by a photoelectric conversion unit that detects light in a plurality of wavelength regions having a quantum efficiency of 60% or more in a visible light region and an infrared light region, respectively, and a light beam conversion unit. There is provided a biological information acquisition apparatus having a processing unit that acquires biological information based on received light information. This makes it possible to acquire biological information effectively. According to a specific feature of the present invention, the photoelectric conversion unit includes a plurality of light receiving units regularly arranged to capture images of light in a plurality of wavelength ranges. According to another specific feature, a light-emitting diode is used as a light source, and a light-emitting diode having one peak wavelength is used as a light source for detecting a plurality of wavelength ranges in the photoelectric conversion unit. The characteristics of the quantum efficiency in the photoelectric conversion unit described above are useful for this specific characteristic. The biometric information acquisition apparatus according to the present invention is suitable for being built in a mobile phone.

According to a specific feature of the present invention, skin moisture information is acquired by the photoelectric conversion unit. According to a more specific feature, the skin moisture information is acquired by the photoelectric conversion unit in a non-contact manner, and has means for determining that the acquired skin moisture is outside the normal measurement range. Thereby, erroneous measurement due to the influence of sweat or the like can be prevented. According to another specific feature, vein pattern information is acquired by the photoelectric conversion unit. By configuring in this way, the light beam conversion unit can be used for both the measurement of skin moisture and the acquisition of vein pattern information for information acquisition in a situation where the living body is in close proximity. According to another specific feature, pulse wave information is acquired by the photoelectric conversion unit, and blood oxygen saturation is acquired based on the pulse wave information acquired by the photoelectric conversion unit. By configuring in this way, the light beam conversion unit can be used for both information acquisition in the situation where the living body is in close proximity, that is, measurement of skin moisture and acquisition of pulse wave information. When the acquisition of vein pattern information and the acquisition of pulse wave information are combined, the health condition can be checked together with the acquisition of the vein pattern for biometric authentication. According to still another specific feature, imaging of the skin is performed by the light beam conversion unit, and the skin condition can be checked in a multifaceted manner together with the skin moisture value.

According to another specific feature of the invention, one of the plurality of wavelength regions is set to a region where sunlight does not reach the ground surface. As a result, measurement without the influence of sunlight disturbance can be performed. According to another specific feature, the biometric information acquisition apparatus includes a determination unit that determines a focus state on an object based on an image of a plurality of wavelength regions by light of a wavelength region. The feature of high quantum efficiency over a wide wavelength range in the above-described photoelectric conversion unit is useful for this specific feature.

According to another feature of the present invention, a photoelectric conversion unit that detects light in a plurality of wavelength ranges, a processing unit that acquires biological information based on light reception information detected by the photoelectric conversion unit, and the acquired biological information A biological information acquisition device is provided that includes a display unit that displays and a control unit that adjusts the display brightness of the display unit based on the output of the photoelectric conversion unit. Accordingly, the output of the photoelectric conversion unit can be effectively used even when displaying information after information is acquired.

According to another feature of the present invention, a photoelectric conversion unit that detects light in a plurality of wavelength ranges, a processing unit that acquires biological information based on light reception information detected by the photoelectric conversion unit, and the acquired biological information There is provided a biological information acquisition apparatus that includes a display unit that displays and includes a backlight, and a control unit that turns on the backlight to illuminate an object when biological information is acquired by the photoelectric conversion unit. According to this configuration, for example, when the skin is imaged with the photoelectric conversion unit in proximity, the skin can be illuminated in the proximity state using the backlight of the display unit facing the skin.

According to another feature of the present invention, a photoelectric conversion unit that detects light in a plurality of wavelength ranges, a processing unit that acquires biological information based on light reception information detected by the light beam conversion unit, and the acquired biological information A display unit that displays and has a backlight, a light source unit for detection, a proximity detection unit that detects whether the photoelectric conversion unit is close to the object, and a proximity detection unit that detects the proximity of the object Then, a biological information acquisition apparatus having a control unit that turns on the light source unit and turns off the backlight is provided. According to this configuration, for example, when measuring skin moisture using light of a specific wavelength from the light source unit, it is possible to prevent the backlight from adversely affecting the measurement.

  As described above, according to the present invention, a biological information acquisition device that can be easily used in daily life can be provided.

It is a block diagram which shows 1st Example of this invention. (Example 1) FIG. 2 is a block diagram showing in detail the configuration of the photometry / AF sensor and the live view sensor of FIG. 1 together with related portions. It is the graph which compared the spectral sensitivity of the CIGS sensor used for the CIGS imaging sensor and CIGSAF sensor of FIG. 2 with the silicon CMOS sensor. It is a flowchart of operation | movement of the camera control part in 1st Example. It is a block diagram which shows 2nd Example of this invention. (Example 2) It is a flowchart of operation | movement of the camera control part in 2nd Example. It is a 1st example of the color filter arrangement | sequence of the CIGS imaging sensor used for the live view sensor of FIG. 2 or FIG. It is a 2nd example of the filter arrangement | sequence of a CIGS image sensor. It is a schematic cross section of the CIGS sensor which employ | adopted the filter arrangement | sequence of FIG. It is a 3rd example of the filter arrangement | sequence of a CIGS image sensor. It is a flowchart which shows operation | movement of the camera control part at the time of recording the image of a live view sensor in 2nd Example of FIG. It is a flowchart which shows the detail of the process which can be utilized in common with step S108 of FIG. 11, and step S114. It is a block diagram which shows 3rd Example of this invention. Example 3 It is a front view which shows the 1st example of arrangement | positioning of LED employable as a 3rd Example. It is a front view which shows the 2nd example of arrangement | positioning of LED employable as a 3rd Example. It is a front view which shows the 3rd example of arrangement | positioning of LED employable as a 3rd Example. It is a front view which shows the 4th example of arrangement | positioning of LED employable as a 3rd Example. It is an operation | movement timing chart in the color and infrared mode of 3rd Example. 19 is a timing chart showing the relationship between the operation of FIG. 18 and color image creation. It is an operation | movement timing chart in the fine color mode of 3rd Example. 21 is a timing chart showing the relationship between the operation of FIG. 20 and color image creation. It is an operation | movement timing chart in the infrared mode of 3rd Example. It is a flowchart of operation | movement of the endoscope control part in 3rd Example. It is a flowchart which shows the detail of step S170 of FIG. It is a flowchart which shows the detail of step S208 of FIG. It is a flowchart which shows the detail of step S172 of FIG. It is a flowchart which shows operation | movement of the monitor control part in 3rd Example. It is an operation | movement timing chart in the color and infrared mode of 4th Example of this invention. Example 4 It is an operation | movement timing chart in the fine color mode of 4th Example. It is a block diagram which shows 5th Example of this invention. (Example 5) It is the block diagram which showed the detail of the monitoring apparatus for vehicles of 5th Example with the distance detection principle. It is a conceptual diagram of the image imaged by the CIGS image sensor of 5th Example. It is a graph of the spectral transmission characteristic of the filter of the CIGS image sensor in 5th Example. It is a flowchart of operation | movement of the monitoring recording control part in 5th Example. It is a flowchart which shows the detail of step S336 of FIG. It is a flowchart which shows the detail of step S308 of FIG. 34, and step S346 of FIG. It is the block diagram which showed the detail of the monitoring apparatus for vehicles of 6th Example of this invention with the distance detection principle. (Example 6) It is a graph of the spectral transmission characteristic of the filter of the CIGS imaging sensor in 6th Example. It is a flowchart of operation | movement of the monitoring recording control part in 6th Example of FIG. It is a 4th example of the filter arrangement | sequence of a CIGS image sensor, and is used for the 7th Example of this invention. (Example 7) It is a graph of the spectral transmission characteristic of the filter of the CIGS image sensor in 7th Example. It is a graph of the spectral transmission characteristic of the color filter of the CIGS imaging sensor used for the front of the 8th Example of this invention, and the monitoring apparatus for vehicles. (Example 8) It is an upper surface external view in the mobile telephone of 9th Example of this invention. Example 9 It is a block diagram of Example 9 of FIG. It is a graph for demonstrating the spectral transmission characteristic of the filter of the CIGS image sensor in 9th Example, and the peak wavelength of an image sensor light source part. It is an example of the color filter arrangement | sequence of the CIGS imaging sensor in 9th Example. It is a front view which shows the example of arrangement | positioning of LED employable for 9th Example. It is a flowchart of operation | movement of the control part in 9th Example. It is a flowchart which shows the detail of step 458 of FIG. It is a flowchart which shows the detail of step S464 of FIG. It is a graph of the spectral transmission characteristic of the filter of the CIGS image sensor in 10th Example of this invention implemented in a mobile telephone, and the peak wavelength of an image sensor light source part. (Example 10) It is an example of the color filter arrangement | sequence of the CIGS imaging sensor in 10th Example. It is a front view which shows the example of arrangement | positioning of LED employable as a 10th Example. It is a graph of the spectral transmission characteristic of the filter of the CIGS image sensor in 11th Example of this invention implemented in a mobile telephone, and the peak wavelength of an image sensor light source part. (Example 11) It is an example of the color filter arrangement | sequence of the CIGS imaging sensor in 11th Example. It is a front view which shows the example of arrangement | positioning of LED employable as 11th Example. It is a graph of the spectral transmission characteristic of the filter of the CIGS image sensor in 12th Example of this invention implemented in a mobile telephone, and the peak wavelength of an image sensor light source part. Example 12 It is an example of the color filter arrangement | sequence of the CIGS imaging sensor in 12th Example. It is a front view which shows the example of arrangement | positioning of LED employable as a 12th Example. (Example 13) It is a graph of the spectral transmission characteristic of the filter of the silicon image sensor in the 13th Example of this invention implemented in a mobile telephone, and the peak wavelength of an image sensor light source part.

FIG. 1 is a block diagram showing a first example of an autofocus digital single-lens reflex camera according to an embodiment of the present invention. The autofocus digital single-lens reflex camera has a camera body 2 and an interchangeable lens 4 that is detachably attached to the camera body 2. The subject light incident from the lens optical system 6 of the interchangeable lens 4 is reflected upward by the mirror 8 at the observation position and forms an image at the position of the focusing screen 10. After this image is reflected by the pentarhythm 12, it is observed by the eyepiece 14, and a composition for imaging is determined.

  At the time of shooting, by operating the shutter release button of the operation unit 15, the mirror 8 is retracted to the shooting position together with the auto-focusing sub mirror 16 and the focal plane shutter 18 is opened, and the lens optical system 6 of the interchangeable lens 4 is opened. The incident subject light is imaged and imaged on the imaging unit 20. Image information captured by the image capturing unit 20 is subjected to image processing by the image processing unit 22 and then stored in the image storage unit 26 under the control of the camera control unit 24. The image information stored in the image storage unit 26 is transferred to a storage medium such as a memory card inserted into the medium slot 28 as appropriate. The image information stored in the image storage unit 26 can be appropriately transferred from the input / output unit 30 to the outside under the control of the camera control unit 24. In addition, since the image information immediately after photographing is sent from the camera control unit 24 to the display unit 32 and automatically displayed, the operator can check the captured image.

  When the image is reproduced, the image information stored in the image storage unit 26 or the medium slot 28 is read by the camera control unit 24 by the operation of the operation unit 15, and is read from the liquid crystal provided on the back surface of the camera body 2. Is displayed on the display unit 32. The above is the basic configuration and basic functions related to imaging and playback in the autofocus digital single-lens reflex camera of FIG. As apparent from the above, when the mirror 8 is at the observation position, the imaging unit 20 does not capture the subject image, so the real-time subject image that can be observed with the eyepiece 14 with the above configuration is not displayed on the display unit. 32 is not displayed and can only be confirmed after shooting. This is the special characteristic of a digital single lens reflex camera, which is different from a normal compact digital camera that can determine the composition while observing the image on the display unit 32.

  Next, the configuration and functions related to autofocus in the autofocus digital single lens reflex camera of FIG. 1 will be described. Part of the subject light incident from the lens optical system 6 of the interchangeable lens 4 is transmitted through the semi-transmissive portion at the center of the mirror 8 at the observation position, and reflected downward by the sub mirror 16 to be used for photometry / autofocus (hereinafter referred to as “AF”). ]) Guided to the sensor. The photometry / use AF sensor 34 re-images the light incident from the sub-mirror 16 on the AF sensor, analyzes it, and sends the result to the camera control unit 24. This analysis is performed, for example, by analyzing the shift direction and the degree of the image formation position between the imaging surface of the imaging unit 20 and the lens optical system 6 by a well-known phase difference detection method using pupil division. The camera control unit 24 drives the lens optical system 6 to eliminate the deviation of the imaging position based on the information on the direction and degree of deviation of the imaging position by the lens optical system 6 obtained from the AF sensor 34 for photometry. Information on the amount and the driving direction is sent to the AF control unit 36. The AF driving unit 38 drives the lens optical system 6 on the basis of information on the driving amount and driving direction transmitted from the AF control unit 36 through a mechanical or electrical interface between the camera body 2 and the interchangeable lens 4 and performs automatic focusing. I do. The details of the configuration of the photometry / AF sensor 34 will be described later.

  The live view sensor 40 is configured for a “live view” function in a digital single-lens reflex camera so that the composition can be determined while observing the image on the display unit 32 in the same manner as an ordinary compact digital camera. is there. The entire reflecting surface 12a of the pentaprism 12 is semi-transmissive, and the live view sensor 40 can capture the entire image of the focal plane 10 by re-imaging the image of the focusing screen 10 on the CIGS imaging sensor. It is like that. The CIGS imaging sensor is an optical sensor made of copper (Cu), indium (In), gallium (Ga), and selenium (Se), and details thereof will be described later.

The reflection surface 12a of the pentaprism 12 transmits light almost entirely except in the visible light region, and has a spectral transmission characteristic in which most of the visible light region is reflected by only slightly transmitting light. When the image of the focal plate 10 is observed at 14, the image is not substantially darkened. Further, since the CIGS imaging sensor employed in the live view sensor 40 has high sensitivity in the visible light region as described later, the light transmittance of the reflecting surface 12a in the visible light region is slight. In addition, it is possible to sufficiently capture an image of the focal plate 10 in the visible light region. The distribution of the visible light to the live view sensor 40 is set to a level at which the light quantity is insufficient for the live view imaging by the CIGS imaging sensor when it becomes dark that it is difficult to optically observe the subject by the eyepiece 14. Details of the live view sensor 40 using the CIGS imaging sensor will be described later. Since the image captured by the live view sensor 40 is sent to the camera control unit 24 and displayed on the display unit 32, the autofocus digital single-lens reflex camera shown in FIG. 1 is normal as long as the subject has normal brightness. The composition can be determined while observing the image on the display unit 32 in the same manner as the compact digital camera.

FIG. 2 is a block diagram showing in detail the configuration of the photometry / use AF sensor 34 and the live view sensor 40 in the first embodiment of the autofocus digital single-lens reflex camera of FIG. As described above, the reflecting surface 12a of the pentaprism 12 transmits light almost entirely except in the visible light region, and in the visible light region, has a spectral transmission characteristic in which most of the light is reflected by only slightly transmitting light. However, the multilayer film 52 is coated on the reflecting surface 12a in order to realize such spectral transmission characteristics.

The live view sensor 40 includes a re-imaging lens 56 for re-imaging the light beam from the focal plane 10 that has passed through the multilayer film 52 on the imaging surface of the CIGS imaging sensor 54. The infrared light cut filter 58 substantially cuts light outside the visible light region that has been transmitted through the multilayer film 52 to approximate the wavelength sensitivity characteristic of the CIGS imaging sensor 54 to the wavelength sensitivity characteristic of the imaging unit 20. Then, a subject image that matches the visibility is picked up and sent to the camera control unit 24 to enable live view on the display unit 32 of FIG. The “infrared light” in the present invention mainly refers to infrared light in a region that is relatively close to visible light, which is called “near infrared light”. Abbreviated as “infrared light”.

The live view sensor 40 further enables full-screen photometry for measuring the brightness on the focusing screen 10. That is, the image information output from the CIGS imaging sensor 54 is processed by the camera control unit 24 as photometric information over the entire screen, and is combined with the output of the AF-compatible partial photometric sensor 72 as necessary. Based on these processing results, automatic exposure control is performed to control the aperture diameter of the interchangeable lens 4, the shutter speed by the focal plane shutter 18, the sensitivity of the imaging unit 20, and the like.

The visible light cut filter 60 is inserted in the optical path to the CIGS imaging sensor 54 in place of the infrared light cut filter 58, and is used in the “long wavelength mode”. Since the multilayer film 52 transmits almost the entire region other than the visible light region, when the visible light cut filter 60 is inserted in the optical path instead of the infrared light cut filter 58 in the long wavelength mode setting, it is visible. Light having a wavelength longer than that of the light enters the CIGS imaging sensor 54. As will be described later, the CIGS imaging sensor 54 has a spectral sensitivity of 1300 nm on the long wavelength side. Therefore, by inserting the visible light cut filter 60, the live view sensor 40 becomes an image sensor suitable for photographing with light in these long wavelength regions. Then, such an image output in the long wavelength region can be observed in real time on the display unit 32 or recorded in the image storage unit 26.

The mirror / filter drive unit 62 drives replacement of the visible light cut filter 60 and the infrared light cut filter 58 under the control of the camera control unit 24 according to mode switching by the operation unit 15. In FIG. 2, the mirror 8a and the sub mirror 16a retracted to the photographing position are shown by two-dot chain lines. However, the driving between the observation position and the photographing position in the mirror 8 and the sub mirror 16 is also performed by the camera control unit. The L / filter driving unit 62 performs the control of 24.

The re-imaging lens 64 of the AF metering sensor 34 re-images the subject light incident from the interchangeable lens 4 and transmitted through the semi-transmissive portion at the center of the mirror 8 at the observation position and reflected downward by the sub-mirror 16. It is for making it happen. The light beam from the re-imaging lens 64 passes through the movable semi-transmissive mirror 66 and the infrared light cut filter 68 having no wavelength selectivity and forms an image on the CIGSAF sensor 70. The CIGSAF sensor also has high sensitivity in the visible light region, as will be described later, and can perform automatic focus detection without auxiliary light even for a dark subject. The infrared light cut filter 68 cuts the wavelength of the harmful infrared light region in order to make the CIGSAF sensor 70 work as an AF sensor. What is the infrared light cut filter 58 for the CIGS imaging sensor 54? The characteristics are not necessarily the same. For example, the infrared light cut filter 8 is set to have a transmission spectral characteristic narrower than that of the infrared light cut filter 58.

For this reason, when the subject has normal brightness, the movable semi-transparent mirror 66 is inserted at the position shown in the figure for dimming, and the amount of light incident on the CIGSAF sensor is adjusted to the sensitivity dynamic range of the CIGSAF sensor. On the other hand, when the subject becomes dark enough to require auxiliary light in a normal AF sensor, the movable semi-transparent mirror is retracted to the position 64a, and the subject image is formed on the CIGS sensor without dimming. At this time, it is necessary to compensate for the optical path length depending on the presence or absence of the movable semi-transmissive mirror 66. For example, when the movable translucent mirror 66 is retracted, a totally transmissive parallel plate having the same optical path length is inserted into the optical path instead. Of course, when the movable semi-transmissive mirror is retracted to the position 64a, photometry by the AF-compatible partial photometry sensor 72 cannot be performed.

When the movable semi-transmissive mirror 66 is inserted in the re-imaging optical path for dimming the light to the CIGSAF sensor 70, the light reflected therefrom enters the AF corresponding partial photometric sensor 72. The AF-corresponding partial photometric sensor 72 measures the brightness of the portion of the focus detected by the CIGSAF sensor, and selectively selects the brightness of the portion of the entire screen that is the focus detection target. By measuring the light, it is used as information for automatic exposure control so that a portion of high interest in photographing is appropriately exposed. As described above, the dimming portion to the CIGSAF sensor 70 that becomes excessive when the subject is bright is not discarded but is effectively used as photometric information.

The partial photometric information from the AF corresponding partial photometric sensor 72 is processed by the camera control unit 24 in combination with the photometric information on the entire screen from the CIGS imaging sensor 54 of the live view sensor 40, and finally the aperture diameter of the interchangeable lens 4. The shutter speed by the focal plane shutter 18 and the sensitivity of the imaging unit 20 are controlled.

The sensor control unit 74 controls the automatic focus adjustment by performing the light reception integration time and gain control of the CIGSAF sensor 70 regardless of whether the movable semi-transparent mirror 66 is inserted or retracted. To do. In order to perform the light receiving integration time and gain control without confusion, information on whether the movable semi-transparent mirror 66 is inserted or retracted is also used. The sensor control unit 74 also issues an instruction to the CIGSAF sensor 70 and the AF-corresponding partial photometric sensor 72, and performs control to match the portion to be subjected to focus detection with the portion to be subject to focus detection in the entire screen, The camera control unit 24 outputs corresponding focus detection information and photometry information.

On the other hand, when the live view sensor 40 is set to the “long wavelength mode” and the visible light cut filter 60 is inserted in the optical path instead of the infrared light cut filter 58 in the optical path from the multilayer film 52 to the CIGS imaging sensor 54. The photometric AF sensor 34 also performs filter replacement corresponding to this. Specifically, in the “long wavelength mode”, the infrared light cut filter 68 is replaced with the visible light cut filter 76 on the premise that the semi-transmissive mirror 66 is retracted. Thereby, the focus detection for imaging in the long wavelength range by the CIGS imaging sensor 54 is performed by the CIGSAF sensor 70 without dimming. At this time, not only adjustment of the wavelength sensitivity range but also compensation for a change in optical path length due to a difference in wavelength and a difference in chromatic aberration during focus detection, etc.

The movement of the movable semi-transmissive mirror 66 and the replacement of the infrared light cut filter 68 and the visible light cut filter 76 as described above are driven by the mirror / filter by the control of the camera control unit 24 based on the mode switching operation by the operation unit 15. Department 78 governs.

The first embodiment shown in FIGS. 1 and 2 can provide a “composite AF function” in addition to the basic functions as described above. When the “composite AF function” is selected by the operation of the operation unit 15, the live view function is stopped and the start of the “composite AF function” is instructed. Specifically, when the “composite AF function” is selected by operating the operation unit 15, the camera control unit 24 replaces the infrared light cut filter 58 with the visible light cut filter 60 to the CIGS imaging sensor 54. An instruction to set the optical path is given to the mirror / filter driving unit 62 and the live view display on the display unit 32 based on the output of the CIGS imaging sensor 54 is stopped.

Instead, the image signal of the CIGS imaging sensor 54 that has become a sensitivity region on the long wavelength side by the visible light cut filter 60 is combined with the output of the CIGSAF sensor in the visible light sensitivity region by the infrared light cut filter 68, The “composite AF function” is executed. Specifically, image analysis of the subject is performed by image processing based on the image signal of the CIGS imaging sensor 54, and a focus detection area by the CIGSAF sensor is determined based on the result.

FIG. 3 compares the spectral sensitivity (quantum efficiency) of the CIGS sensor used in the CIGS imaging sensor 54 and the CIGSAF sensor of FIG. 2 with a silicon CMOS sensor. FIG. 3A shows the quantum efficiency (%) of the CIGS sensor at each wavelength, which is clear compared to the similar quantum efficiency (%) for the silicon CMOS sensor in FIG. 3B. It shows high sensitivity and broadband characteristics. Specifically, the CIGS sensor shown in FIG. 3A has a wide sensitivity range over a wavelength of about 1300 nm. Furthermore, it has a spectral sensitivity exceeding 50% quantum efficiency over a wide wavelength range from about 400 nm to about 1200 nm, and shows particularly remarkable high quantum efficiency in the visible light and the infrared light region adjacent thereto. Such a high-sensitivity and broadband spectral sensitivity characteristic having a quantum efficiency of 60% or more in the visible light region and the infrared light region cannot be expected with a silicon CMOS sensor as shown in FIG.

FIG. 4 is a flowchart of the operation of the camera control unit 24 in the first embodiment of FIGS. 1 and 2. When the main switch of the camera is turned on by the operation unit 15, the flow starts. In step S2, it is checked whether the autofocus digital single-lens reflex camera is set to the reproduction mode by the operation unit 15. If the reproduction mode setting is not detected, the mode is the shooting mode, and the process proceeds to step S4, where the movable semi-transparent mirror 66 is set in the optical path to the CIGSAF sensor 70 to instruct the mirror / filter driving unit 78 to reduce the amount of incident light.

It should be noted that a delay period is provided in the mechanical execution of the setting of the movable semi-transmissive mirror 66 by the mirror / filter driving unit 78 in response to the instruction in step S4. For example, the movable semi-transmissive mirror 66 is provided in the optical path to the CIGSAF sensor 70. Instructed to cancel the dimming to retract the movable semi-transparent mirror 66 from the optical path in the set state, and then instructed to set the movable semi-transparent mirror 66 in the optical path within the delay time. In the case where it is performed subsequently, the mirror / filter driving unit 78 does not actually drive the movable semi-transmissive mirror 66, and the state where the movable semi-transmissive mirror 66 is set in the optical path continues. . In other words, the mirror / filter driving unit 78 executes the driving of the movable semi-transmissive mirror 66 only after the instruction to drive the movable semi-transmissive mirror 66 to a different state is repeatedly performed within the delay time. In addition, when the instruction in step S4 is performed in a state where the movable semi-transmissive mirror 66 is already set in the optical path of the CIGSAF sensor 70, the mirror / filter driving unit 78 naturally has no effect on the movable semi-transmissive mirror 66. Does not drive. These are common to various “instructions” in the following steps.

Next, in step S6, the mirror / filter driving unit 62 is instructed to set the live view infrared light filter 58 in the optical path to the CIGS image sensor 54. Note that the mirror replacement operation of the mirror / filter drive unit 62 is also provided with a delay time for drive execution in response to the same instruction as described in the mirror / filter drive unit 78 above.

In step S8, the mirror / filter driving unit 78 is instructed to set the infrared cut filter 68 for AF in the optical path to the CIGSAF sensor 70. In step S10, based on the output of the CIGSAF sensor 70, it is checked whether or not the subject is dark enough to cancel the dimming. If applicable, the process proceeds to step S12, where an instruction to cancel the light reduction for retracting the movable semi-transmissive mirror 66 from the optical path is given, and the process proceeds to step S14. On the other hand, if the subject is sufficiently bright, the process proceeds directly to step S14.

In step S14, it is checked whether or not the “composite AF mode” has been selected by the operation unit 15. If there is a selection, the process proceeds to step S16, and an instruction to set the visible light cut filter 60 in the optical path to the CIGS imaging sensor 54 in place of the infrared light filter 58 in order to perform the composite AF is given to the mirror / filter drive unit 62. Do it. Further, in step S18, the live view display on the display unit 32 based on the output of the CIGS imaging sensor 54 is stopped, and the CIGSAF sensor in the sensitivity range of the visible light and the image signal of the CIGS imaging sensor 54 in the sensitivity region on the long wavelength side is stopped. The start of the “composite AF function” to be combined with the output is instructed, and the process proceeds to step S20. On the other hand, if the selection of “AF mode” is not detected in step S14, the process directly proceeds to step S20.

In step S <b> 20, it is checked whether the image picked up by the image pickup unit 20 is so dark that the light quantity becomes insufficient. Usually, when the subject becomes dark to this level, photographing using auxiliary light such as a flash is required. When the light quantity shortage is detected in step S20, the process proceeds to step S22, and it is checked whether or not the “long wavelength mode” is selected by the operation of the operation unit 15. Then, if applicable, the process proceeds to step S 24, and an instruction to set the live view visible light cut filter 60 in the optical path to the CIGS imaging sensor 54 instead of the infrared light filter 58 is given to the mirror / filter drive unit 62. Further, in step S26, the mirror / filter drive unit 78 is instructed to set the visible light cut filter 76 for AF in the optical path to the CIGSAF sensor 70 instead of the infrared light cut filter 68, and the process proceeds to step S28. To do.

On the other hand, if it is not detected in step S20 that the light intensity of the imaging unit is insufficient, the process directly proceeds to step S28. As described above, when the image is not dark enough to cause the light amount of the image capturing unit to be insufficient, the process cannot normally proceed to step S22, and the “long wavelength mode” is prohibited. This is to prevent confusion of settings. Even when the subject is bright, it is possible to proceed to step S22 by performing a special operation with the operation unit 15 in particular when it is desired to select the “long wavelength mode”. Further, when the “long wavelength mode” setting is not detected in step S22, the process directly proceeds to step S28.

In step S28, it is checked whether or not a release operation using the shutter release button of the operation unit 15 has been performed. If the release operation cannot be detected, the process proceeds to step S30, and it is checked whether an operation to turn off the main switch of the camera is performed by the operation unit 15. If no camera-off operation is detected, the flow returns to step S2, and thereafter, step S2 to step S30 are repeated unless a playback mode operation is detected in step S2 or a release operation is not detected in step S28.

The above repetition is performed at a sufficiently high speed and is repeated many times within the delay time provided in the mirror / filter driving units 62 and 78 described above. Accordingly, when the detection result based on step S10, step S14, step S20 and step S22 changes, the same instruction based on this change is repeatedly performed within the delay time of the mirror / filter driving units 62 and 78, and the mirror / filter driving unit 62 78, an appropriate instruction is executed. Accordingly, the setting / cancellation of the light reduction based on the change in the brightness of the subject, the switching of the wavelength band cut filter, and the switching of the wavelength band cut filter based on the mode switching are executed smoothly.

When a playback mode setting operation by the operation unit 15 is detected in step S2, the process proceeds to the playback mode process in step S32. When the shooting mode is selected by the function in the playback mode process, the flow returns to step S4. Also, when a camera-off operation is detected by a function within the playback mode process, the flow ends.

On the other hand, when the release fixed operation by the shutter release button of the operation unit 15 is detected in step S28, the process proceeds to the imaging recording process in step S34. When the imaging record and the imaging result display on the display unit are finished, the flow automatically returns to step S2. Note that when the camera-off operation is detected in step S30, the flow of FIG. 4 ends.

FIG. 5 is a block diagram showing a second example of the autofocus digital single-lens reflex camera according to the embodiment of the present invention. Since most of the configuration is the same as that of the first embodiment of FIG. 1, the same reference numerals are given to common portions, and the description is omitted unless particularly required. The second embodiment shown in FIG. 5 differs from the first embodiment shown in FIG. 1 in the camera body 100. In particular, the live view sensor 102 and the configuration and functions related thereto are different from those in the first embodiment.

The live view sensor 40 according to the first embodiment is configured to receive light through the semi-transmissive reflective surface 12a, and suppresses light in the visible light region that is transmitted through the reflective surface 12a. This is because the object image can be optically observed with the eyepiece 14 without hindrance, and at the same time, a live view is always possible. Since the CIGS imaging sensor is used for the live view sensor 40, even if the light in the visible light region that is transmitted through the reflecting surface 12a is suppressed, it is sufficient for live view of a subject with normal brightness. However, in the case of a dark subject that cannot be sufficiently observed with the eyepiece 14, the live view sensor 40 has insufficient light quantity. On the other hand, the second embodiment of FIG. 5 is configured so that the live view can be performed by the cIGS imaging sensor employed in the live view sensor 102 even in the case of a dark subject that cannot be sufficiently observed by the eyepiece 14. The detailed structure of the live view sensor 102 in FIG. 5 is basically the same as the live view sensor 40 in FIG. 2, and includes a re-imaging optical system and a CIGS imaging sensor. However, since the arrangement position of the live view sensor 102 with respect to the pentaprism 104 is different, the re-imaging optical system is different from the re-imaging lens 56 of FIG.

Based on the above concept, in the second embodiment, the normal pentaprism 104 is employed, and when the live view mode is not set, all the light from the pentaprism 104 goes to the eyepiece. At this time, the movable total reflection mirror 106 is retracted from the optical path to the eyepiece 14 as shown in FIG. Therefore, live view is not possible in this state.

When the live pure mode is selected by operating the operation unit 15, the movable total reflection mirror is lowered to a position 106 a and all the light from the pentaprism 104 is reflected in the direction of the live view sensor 102. Therefore, the optical viewfinder image cannot be observed by the eyepiece 14. The movable neutral density filter 108 is inserted in the optical path to the live view sensor as shown in FIG. 5 when the subject has normal brightness, and matches the amount of light incident on the live pure sensor 102 to the sensitivity dynamic range of the CIGS imaging sensor. Come on. On the other hand, when the subject becomes so dark that it is difficult to observe with the eyepiece, the movable neutral density filter 108 retracts from the optical path to the live view sensor 102 and guides the subject image to the live view sensor without dimming. At this time, it is necessary to compensate the optical path length depending on the presence / absence of the movable neutral density filter 108. For example, when the movable neutral density filter is retracted, a parallel plate having the same optical path length as that of the movable neutral density filter is used instead. Insert into. In this way, even in the case of a dark subject that is difficult to observe optically, in the case of the second embodiment of FIG. 5, a live view is possible by the CIGS imaging sensor. The image in the visible light range from the live view sensor 102 can be recorded not only in the live view on the display unit 32 but also in the image storage unit 26. It becomes possible to do.

The infrared light cut filter 110 cuts light outside the visible light region reflected from the movable total reflection mirror 106a in the live view mode, and approximates the wavelength sensitivity characteristic of the CIGS imaging sensor to the wavelength sensitivity characteristic of the imaging unit 20. A subject image that matches the visibility is captured and sent to the camera control unit 116 to enable a natural live view.

The visible light cut filter 112 is inserted in the optical path to the live view sensor 102 in place of the infrared light cut filter 110 and is used in the “long wavelength mode”. Since the movable total reflection mirror 106a reflects almost the entire area other than the visible light region, the visible light cut filter 112 replaces the infrared light cut filter 110 in the long wavelength mode setting, and the optical path to the live view sensor 102 is changed. Is inserted into the CIGS imaging sensor of the live view sensor 102, light having a wavelength longer than visible light is incident on the CIGS imaging sensor. Accordingly, as in the first embodiment, the image output of the image in the long wavelength region can be observed in real time on the display unit 32 or recorded in the image storage unit 26. In the long wavelength mode using the visible light cut filter 112, the movable neutral density filter 108 is retracted from the optical path to the live view sensor. The movable total reflection mirror 106, the movable neutral density filter 108, the infrared light cut filter 110, and the visible light cut filter 112 are driven by the mirror / filter drive unit 114 controlled by the camera control unit 116. .

FIG. 6 is a flowchart of the operation of the camera control unit 116 in the second embodiment of FIG. As in the first embodiment, when the main switch of the camera is turned on by the operation unit 15, the flow starts. In step S42, it is checked whether or not the autofocus digital single-lens reflex camera is set to the reproduction mode by the operation unit 15. If the reproduction mode setting is not detected, the shooting mode is set, and the process proceeds to step S44 to instruct to set the optical viewfinder optical path. Specifically, the mirror / filter driving unit 114 is instructed so that the movable total reflection mirror 106 is retracted from the optical path to the eyepiece 14. In step S44, an instruction to reduce the amount of incident light by inserting the movable neutral density filter 108 into the optical path to the live view sensor 102 is sent to the mirror / filter driving unit 114, and the movable semi-transmissive mirror 66 in FIG. The mirror / filter driving unit 78 is instructed to reduce the amount of incident light by setting in the optical path to the CIGSAF sensor 70.

In step S46, it is checked whether the “live pew mode” is set by the operation unit 15. If applicable, the process proceeds to step S48, and an instruction to switch the optical path to the live view is issued. Specifically, the mirror / filter driving unit 114 is instructed to advance the movable total reflection mirror 106 in the optical path to the eyepiece 14, and the process proceeds to step S50. When this instruction is executed, the finder image cannot be optically observed from the eyepiece 14, and a live view on the display unit 32 based on the output of the live view sensor 102 is enabled instead. In step S48, the mirror / filter driving unit 114 is further instructed to set the infrared light cut filter 110 for live view in the optical path to the live view sensor 102. Note that the mirror / filter driving unit 114 is also provided with a delay time for executing the driving in response to the instruction as described in the first embodiment. On the other hand, when the setting to “live view mode” is not detected in step S46, the process directly proceeds to step S50.

In step S50, the mirror / filter driving unit 78 is instructed to set the infrared cut filter 68 for AF in the optical path to the CIGSAF sensor 70. In step S52, based on the output of the CIGSAF sensor 70, it is checked whether or not the subject is dark enough to cancel the dimming. If applicable, the process proceeds to step S54, and it is checked whether the “live pew mode” is set by the operation unit 15. If applicable, the process proceeds to step S56, where an instruction to cancel the dimming is performed to retract the movable neutral density filter 108 from the optical path to the live view sensor 102, and the process proceeds to step S58. On the other hand, if the setting of “live view mode” is not detected in step S54, the process directly proceeds to step S58. In step S58, the movable semi-transparent mirror 66 is retracted from the optical path to the CIGSAF sensor 70 to instruct to cancel the AF dimming. Thus, if it is detected in step S52 that the subject is dark, the AF dimming is canceled regardless of the “live view mode” setting.

Next, in step S60, it is checked whether the image picked up by the image pickup unit 20 is so dark that the light quantity becomes insufficient. If applicable, the process proceeds to step S62, and it is checked whether or not the long wavelength mode is selected by operating the operation unit 15. Then, if applicable, the process proceeds to step S 64, and an instruction to set the live view visible light cut filter 112 in the optical path to the live view sensor 102 instead of the infrared light filter 110 is given to the mirror / filter driving unit 114. In step S 66, the mirror / filter driving unit 78 is instructed to set the visible light cut filter 76 for AF in the optical path to the CIGSAF sensor 70 instead of the infrared light cut filter 68.

Through the above steps, the flow proceeds to step S68. On the other hand, if it is not detected in step S52 that the subject is dark enough to cancel the dimming, if it is not detected in step S60 that the subject is dark enough that the imaging by the imaging unit 20 is insufficient, and step If no selection for the long wavelength mode is detected in S62, the process directly proceeds to step S68.

In step S68, it is checked whether or not a release operation using the shutter release button of the operation unit 15 has been performed. If the release operation cannot be detected, the process proceeds to step S70 to check whether the operation unit 15 has performed an operation to turn off the main switch of the camera. If no camera-off operation is detected, the flow returns to step S42, and thereafter, step S42 to step S70 are repeated unless a playback mode operation is detected in step S42 or a release operation is not detected in step S68.

Similar to the first embodiment, the above repetition is performed at a sufficiently high speed, and is repeated many times within the delay time provided in the mirror / filter driving units 78 and 114 described above. Accordingly, when the detection result based on step S46, step S52, step S54, step S60 and step S62 changes, the same instruction based on this change is repeatedly performed within the delay time of the mirror / filter drive units 78 and 114, and the mirror / filter Appropriate instructions are executed by the drive units 78 and 114. Accordingly, the setting / cancellation of the light reduction based on the change in the brightness of the subject, the switching of the wavelength band cut filter, and the switching of the wavelength band cut filter based on the mode switching are executed smoothly.

As in the first embodiment, when a reproduction mode setting operation by the operation unit 15 is detected in step S42, the process proceeds to a reproduction mode process in step S72. When the shooting mode is selected by the function within the playback mode process, the flow returns to step S44. Also, when a camera-off operation is detected by a function within the playback mode process, the flow ends.

Further, when the release fixed operation by the shutter release button of the operation unit 15 is detected in step S68, the process proceeds to the imaging recording process in step S74. When the imaging record and the imaging result display on the display unit are finished, the flow automatically returns to step S42. Note that when the camera-off operation is detected in step S70, the flow of FIG. 6 ends.

The various features of the present invention described above are not limited to the embodiments and can be widely used. For example, in the first embodiment, the visible light cut filter 60 combines the CIGS imaging sensor 54 having sensitivity in the long wavelength region and the infrared light cut filter 68 with the output of the CIGSAF sensor having sensitivity to visible light. The “composite AF function” has been described as being implemented. However, the implementation of the “composite AF function” is not limited to this. For example, the movable semi-transparent mirror 66 having no wavelength selectivity in FIG. 2 is formed of a dichroic mirror, transmits visible light and guides it onto the CIGSAF sensor 70, and reflects the long wavelength region to guide to the AF-compatible partial photometric sensor 72. Like that. A CIGS sensor is also used for the AF-compatible partial photometric sensor 72. In this case, the infrared light cut filter 68 is not necessary.

If comprised as mentioned above, it becomes possible to estimate where a person exists in the AF corresponding part by the AF corresponding partial photometric sensor 72 having sensitivity in the long wavelength region, and the focus by the CIGSAF sensor 70 is applied to that part. Detection can be performed.

Furthermore, the implementation of the “composite AF function” is not limited to the one using two CIGS sensors as described above. For example, in FIG. 2, in the state where the visible light cut filter 76 is inserted in the optical path of the CIGSAF sensor, the CIGSAF sensor itself estimates where a person exists in the AF-corresponding portion, and the infrared light cut filter 18. It is also possible to perform focus detection by the CIGSAF sensor 70 on the portion in a state where the is inserted in the optical path. It is also possible to realize the “composite AF function” by using one CIGS having a wide sensitivity region in different sensitivity regions in a time division manner and combining their outputs.

In the above embodiment, the description has been made on the assumption that the movable semi-transmissive mirror or the filter is put in and out of the optical path for dimming. For example, a plurality of neutral density filters with different transmittances may be prepared, and one of these may be inserted into the optical path to change the degree of attenuation finely in a stepwise manner. You may comprise so that the degree of light attenuation may be changed continuously using the light reduction means from which the transmittance | permeability changes continuously.

In the above embodiment, the CIGS sensor is used as a sensor having a high sensitivity and a broadband spectral sensitivity characteristic having a quantum efficiency of 60% or more in the visible light region and the infrared light region. The CIGS sensor is a photoelectric sensor using a polycrystalline CIGS thin film made of copper, indium, gallium, and selenium, and the absorption wavelength region can be controlled by changing the band gap by controlling the composition thereof. Among them, the gallium content of zero is also referred to as “CIS thin film”, but in this specification “CIGS sensor” refers to such a “CIS thin film” that does not contain gallium. It shall also mean a photoelectric sensor.

FIG. 7 is a first example of the color filter array of the CIGS imaging sensor used in the live view sensor 40 in the first embodiment of FIG. 2 or the live view sensor 102 in the second embodiment of FIG. In this first example, an infrared light transmission filter R11, a blue transmission filter B12, a green transmission filter G22, and a red transmission filter R21 are arranged as shown in the figure, and this is repeated as a unit. Since the CIGS imaging sensor of the present invention has a wide spectral sensitivity range from the visible light range to the infrared light as shown in FIG. 3, the color filter for visible light and infrared light can be provided in one sensor in this way. . The arrangement in FIG. 7 is obtained by adding an infrared light transmission filter to the primary color filter. Unlike the typical Bayer arrangement in the primary color filter, the green light receiving area is the same as that of blue and blue. This point can be corrected by later circuit processing.

Here, the interpolation of the infrared light image regarding the pixel in which the infrared light transmission filter is not disposed will be described. First, for the pixel corresponding to the blue transmission filter B12, basically, the average value of the pixel data corresponding to the infrared light transmission filter IR11 and the pixel data corresponding to the infrared light transmission filter IR13 on both sides thereof is determined. Interpolation is performed. The same applies to the interpolation of the infrared light image in the pixels corresponding to the other blue transmission fills. On the other hand, the pixel corresponding to the red transmission filter R21 is similarly interpolated by the average value of the pixel data corresponding to the infrared light transmission filter IR11 above and below the pixel data corresponding to the infrared light transmission filter IR31. The The same applies to the interpolation of the infrared light image in the pixels corresponding to the other red transmission filters. For the pixel corresponding to the green transmission filter G22, the pixel data corresponding to the infrared light transmission filter IR11, the pixel data corresponding to the infrared light transmission filter IR13, and the infrared light transmission filter IR33 around it. Interpolation is performed based on the average value of the corresponding pixel data and the pixel data corresponding to the infrared light transmission filter IR31. The same applies to the interpolation of the infrared light image in the pixels corresponding to the other green transmission filters.

Note that with simple interpolation as described above, an infrared light image different from the actual subject may be obtained. To prevent this, instead of simply interpolating an infrared light image based only on the data corresponding to the nearby infrared light transmission filter, the visible light data affecting the pixel being interpolated is also included. It is effective to perform interpolation with consideration. For example, in the interpolation of the infrared light image of the pixel corresponding to the red transmission filter R21, the red light data actually received by the pixel corresponding to the red transmission filter R21 is also taken into consideration. The presence or absence of such visible light data and the degree of addition are determined based on the correlation between visible light data and infrared light data or the correlation between other visible light data of surrounding pixels. .

FIG. 8 is a second example of the filter array of the CIGS imaging sensor used in the live view sensor 40 in the first embodiment of FIG. 2 or the live view sensor 102 in the second embodiment of FIG. In the second example, the color filter array itself is common to the first example of FIG. 7, but the light receiving area of each color filter is different. In other words, the infrared light transmission filter IR11 and the green transmission filter G22 secure the maximum light receiving area allowed for the pixel, but the blue transmission filter B12 is provided with the light shielding portion 202 so that the light reception area is the green transmission filter. It is about half of G22. Similarly, with respect to the red transmission filter R21, the light receiving area is about half that of the green transmission filter G22 by providing the light shielding portion 204. This corresponds to the visibility of human eyes with respect to red and blue being about half that of green.

Since the CIGS imaging sensor of the present invention has high sensitivity in the visible light region as shown in FIG. 3, it can sufficiently cope with reducing the light receiving areas of the blue transmission filter B12 and the red transmission filter R21 as described above. In addition, since the light receiving area of each pixel is changed by the light shielding portion, finer adjustments can be made as compared with approximation to human visual sensitivity by the ratio of the number of pixels as in the Bayer array. Accordingly, the light receiving area ratio of the blue transmission filter B12 and the red transmission filter R21 can be changed.

FIG. 9 is a schematic cross-sectional view of a CIGS sensor that employs a second example of the filter arrangement of FIG. As shown in FIG. 9A, the CIGS imaging sensor of the present invention has a structure in which a CIGS thin film 402 is laminated on an LSI 400, and the aperture ratio for one pixel is very large. A color filter 404 is placed thereon. The basic structure itself in the schematic cross-sectional view of FIG. 9A is not limited to the second example of the filter array, and is common to the CIGS sensor of the present invention.

FIG. 9B is an enlarged schematic cross-sectional view of the portion 406 of FIG. 9A, and conceptually shows a cross section of the second example of the filter array of FIG. In FIG. 8 and FIG. 9 (A), the same numbers are assigned to corresponding parts. As is clear from FIG. 9B, the CIGS thin film 402 is divided into photodiodes 408, 410, etc. that form pixels, and an infrared light transmission filter IR11 is placed on the photodiode 408. Yes. On the photodiode 410, a light shielding portion 202 and a blue transmission filter B12 for reducing the light receiving area are placed.

FIG. 10 is a third example of the filter array of the CIGS imaging sensor used in the live view sensor 40 in the first embodiment of FIG. 2 or the live view sensor 102 in the second embodiment of FIG. In this example, a quarter of the total number of green transmission filters in the Bayer array is regularly replaced with infrared light transmission filters IR11, IR33, IR51, and the like. The remaining three quarters are green transmission filters G13, G22, G31, G42, G44, G35, G24, etc., as in the Bayer array.

As a result, the ratio of the total number of green transmission filters G13 and the like is 1.5 times the ratio of the total number of red transmission filters R23 and blue transmission filters B32. As a result, the number of pixels corresponding to the green transmission filter is increased in the same manner as the Bayer array, so that the light receiving area of the green transmission filter is increased to approximate the visual sensitivity of the human eye. Also in the filter arrangement of FIG. 10, taking into account the concept of the filter arrangement of FIG. It is also possible to adjust the light receiving area for approximation.

On the other hand, since the infrared light transmission filters IR11 and the like are arranged as described above, the arrangement is sparse and the ratio of the total number is half of the ratio of the total number of the red transmission filter R23 and the blue transmission filter B32. . Since the CIGS imaging sensor of the present invention has high sensitivity in the infrared light region as shown in FIG. 3, it can sufficiently cope with a small proportion of the total number of pixels, and the infrared light has a long wavelength, so the pixel arrangement is visible light. Even if it is sparser than this, it can be handled.

Next, interpolation of an infrared light image relating to a pixel in which an infrared light transmission filter is not arranged in the filter array of FIG. 10 will be described. First, with respect to the pixel corresponding to the green transmission filter G35, the pixel data corresponding to the infrared light transmission filter IR15 which is two above and the pixel data corresponding to the infrared light transmission filter IR33 which are two to the left. The interpolation is performed by the average value of the pixel data corresponding to the two infrared light transmission filters IR55 below and the pixel data corresponding to the two infrared light transmission filters IR37 on the right. For the pixel corresponding to the green transmission filter G24, the average value of the pixel data corresponding to the infrared light transmission filter IR15 at the upper right and the pixel data corresponding to the infrared light transmission filter IR33 at the lower left. Is interpolated. Further, for the pixel corresponding to the green transmission filter G26, the average value of the pixel data corresponding to the infrared light transmission filter IR15 in the upper left and the pixel data corresponding to the infrared light transmission filter IR37 in the lower right is determined. Interpolation is performed.

The pixels corresponding to the red transmission filter R25 correspond to the infrared light image data and the infrared light transmission filter IR15 corresponding to the green transmission filters G35, G24, and G26 obtained by interpolation as described above. Interpolation is based on the average value of pixel data. When this is arranged, the weighted average of infrared light image data corresponding to IR15, IR33, IR55, and IR37 is obtained as follows.
{(IR15 + IR33 + IR55 + IR37) / 4 + (IR15 + IR33) / 2
+ (IR15 + IR37) / 2 + IR15} / 4
= (9IR15 + 3IR33 + IR55 + 3IR37) / 16
In the same manner, the infrared light image data of the pixels corresponding to each visible light filter is complemented.

For the interpolation of the green image for the pixels where the green transmission filter is not arranged, first, the average of the pixels corresponding to the green transmission filters G22, G42, G44, and G24 around the image corresponding to the infrared light transmission filter IR33 is calculated. Interpolate. Then, the average of the green image data of the image corresponding to the infrared light transmission filter IR33 obtained by interpolation as described above and the green image data of the green transmission filters G22, G31, and G42 is taken and is at the center thereof. The image corresponding to the red transmission filter B32 also interpolates the green image data. When this is arranged, the weighted average of G22, G31, G42, G44 and G24 is obtained as follows.
{(G22 + G42 + G44 + G24) / 4 + G22 + G31 + G42} / 4
= (5G22 + 4G31 + 5G42 + G44 + G24) / 16
In the same manner, the green image data of the pixels corresponding to the infrared light transmission filter, the red transmission filter, and the blue transmission filter are complemented. Note that the interpolation of the red image and the blue image is the same as in FIG.

When interpolation is repeated when the arrangement is sparse like the above red transmission filter, or when interpolation is performed using asymmetric arrangement data such as the above green transmission filter, as described above Interpolation is further performed using data created by interpolation, and an image different from the actual one may be obtained. Even in such a case, in addition to the interpolation using only the pixel data corresponding to the infrared light transmission filter as described above, or the interpolation using only the image data corresponding to the green transmission filter, as described in FIG. It is effective to perform interpolation in consideration of data of other colors affecting the pixel to be interpolated.

FIG. 11 is a flowchart showing the operation of the camera control unit 116 when the image of the live view sensor 102 is recorded in the image storage unit 26 in the second embodiment. When the flow is started by performing an operation of selecting this function by the operation unit 15, first, the movable total reflection mirror is lowered to the position 106a in step S82, and all the light from the pentaprism 104 is directed to the live view sensor 102. The optical path switching is instructed to reflect. In step S84, the mirror 8 is fixed at the observation position so that it does not move up to the photographing position even when the release is performed. In step S86, display on the display unit 32 is performed.

Next, in step S88, it is checked whether or not the infrared light mode has been selected. 78. In step S92, the mirror / filter driving unit 114 is instructed to set the live view visible light cut filter 112 in the optical path to the live view sensor 102, and the process proceeds to step S94.

On the other hand, if it is detected in step S88 that the infrared light mode is not selected, the process proceeds to step S96, and an instruction to set the infrared light cut filter 68 for AF to the optical path to the CIGSAF sensor 70 is given by the mirror / filter. This is performed for the drive unit 78. In step S98, it is checked whether the “image fusion mode” is selected. Although it is known to analyze plant vegetation and detect pest damage by fusing infrared and visible light images of the same subject, the “image fusion mode” is applied to the same subject almost simultaneously. On the other hand, it is possible to obtain an infrared light image and a visible light image, and an infrared light image and a visible light image that are not shifted from each other can be obtained even for a moving subject.

If the setting to the image fusion mode is not detected in step S98, it means that the visible light mode is selected, so the process proceeds to step S100, and the live view infrared light cut filter 110 is connected to the live view sensor 102. An instruction to set the optical path is given to the mirror / filter driving unit 114, and the process proceeds to step S94.

On the other hand, when the setting to the image fusion mode is detected in step S98, the process proceeds to step S102, and an instruction to remove both the infrared light cut filter 110 and the visible light cut filter 112 from the optical path to the live view sensor 102 is given. Is performed on the mirror / filter driving unit 114. This is because the live view sensor 102 acquires both the infrared light image and the visible light image almost simultaneously.

Next, in step S104, an instruction to prohibit live view display by the display unit 32 is issued, and the process proceeds to step S94. This is because the imaging positions of visible light and infrared light by the interchangeable lens 4 are different, so that when the infrared light image and the visible light image are displayed as they are on the display unit 32, the image that does not match the focused image overlaps, making it difficult to see. Because it becomes. In step S104, instead of completely prohibiting display by the display unit 32, only the pixel information of the infrared light image and the visible light image that are in focus (usually the visible light image) is extracted. May be displayed. In this case, because the filter is removed, light of image information that is not in focus is incident on these pixels, but display is possible because the light intensity of the image information that is in focus is dominant. is there. Further, it may be possible to select in advance whether to prohibit the entire display as described above or display only pixel information for an image that is not in focus, and to issue an instruction of the one selected in step S104.

In step S94, it is checked whether or not a release operation has been performed. If there is no release operation, the process returns to step S86. Thereafter, steps S86 to S104 are repeated until a release operation is detected. This corresponds to mode switching by the operation unit 15. As described with reference to FIG. 4, since a delay period is provided for the mechanical execution of the “instruction”, even in FIG. In this case, driving of the filter does not occur. The same applies to the display and prohibition switching of the display unit 32 in step S86 and step S104.

In step S94, when a release operation is detected, the process proceeds to step S106, where AF driving of the lens optical system 6 is performed, and when the pin is aligned, a live view recording process of step S108 is performed. This AF drive may be for an infrared light image or a visible light image. Next, in step S110, it is detected whether the image fusion mode is set. If not, the flow is immediately terminated.

On the other hand, when the setting of the image fusion mode is detected in step S110, the process proceeds to step S112, and the AF control unit 36 drives the lens optical system 6 for infrared light correction. In other words, when the image fusion mode is set, the AF driving in step S106 and the live view recording process in step S108 are for the visible light image, but in step S112, the infrared light is shifted from the focus position for visible light. The lens optical system 6 is caused to perform predetermined correction driving up to the focus position for the AF driving unit 38. This infrared light correction drive is performed in a very short time, and immediately enters the live view infrared light recording process of step S114. Then, when the process is completed, the flow ends. Although details of the above functions will be described later, basically, in the image fusion mode, a visible light image is recorded in step S108, and subsequently, an infrared light image is recorded almost simultaneously in step S114. It is.

FIG. 12 is a flowchart showing details of the live view recording process in step S108 in FIG. 11 and the live view infrared light recording process in step S114, which can be used in common for both. When the flow starts, it is checked in step S122 whether or not the infrared light photographing mode is set. If not, the process proceeds to step S124, and pixel information of the visible light image is obtained by reading out RGB pixel data. In step S126, RGB interpolation processing is performed. Next, in step S128, it is checked whether or not the image fusion mode is set. If not, the process proceeds to step S130 to perform image processing on the RGB visible light color image. In step S132, the image is recorded in the image recording unit 26, and the flow ends.

On the other hand, when the setting to the infrared light mode is detected in step S122, the process proceeds to step S134, and the pixel information of the infrared light image is obtained by reading the IR pixel data. In step S136, IR interpolation processing is performed. Next, in step S138, it is checked whether or not the image fusion mode is set. In this case, since the infrared light mode is not applicable, the process proceeds to step S140, and image processing is performed on the infrared light image. In step S132, the image is recorded in the image recording unit 26, and the flow ends.

Further, when the setting to the image fusion mode is detected in step S128, the process proceeds to step S134, and the pixel information of the infrared light image is obtained by reading the IR pixel data. In step S136, IR interpolation processing is performed. By these, in addition to visible light image information in steps S124 and 126, infrared light image information can also be obtained. Next, in step S138, it is checked whether or not the image fusion mode is set. In this case, since it is the image fusion mode, the process proceeds to step S142, and the infrared light image obtained by the processing in steps S134 and S136 is red. It is checked whether it is after external light correction driving.

Here, the meaning of whether or not the step corresponds to step S142 will be supplemented. First, when it does not correspond to step S142, it corresponds to the case where the flow of FIG. 12 is performed by step S108 of FIG. The visible light image obtained in step S124 and step S126 is in focus by AF driving for visible light in step S106 in FIG. 11, and the infrared light image obtained in step S134 and step S136 is out of focus. It has become a thing. On the other hand, the case corresponding to step S142 corresponds to the case where the flow of FIG. 12 is executed in step S114 of FIG. The infrared light images obtained in steps S134 and S136 are in focus by the infrared light correction drive in step S112 in FIG. 11, and the visible light images obtained in steps S124 and S126 are out of focus. It has become a thing.

Therefore, if it is determined in step S142 that the obtained image does not correspond to the image after infrared light correction driving, the process proceeds to step S144, and image processing is performed on the focused RGB visible light image. Next, in step S146, auxiliary infrared image processing is performed on the in-focus infrared light image. In step S148, an image obtained by correcting the visible light image with the infrared light image information is created. This image is basically a visible light image, but it improves the image quality of the visible light image by the low-pass filter effect, etc., by adding the information of the infrared light image that is out of focus and defocused. is there. The process proceeds to step S132 through the above processing, and the images obtained in steps S144, S146, and S148 are recorded, and the flow is terminated.

On the other hand, if it is determined in step S142 that the obtained image corresponds to an image after infrared light correction driving, the process proceeds to step S150, and image processing is performed on the focused infrared light image. Next, in step S152, image processing is performed supplementarily for the visible RGB light image that is not in focus. In step S156, an image obtained by correcting the infrared light image with the visible light image information is created. This image is basically an infrared light image, but it improves the image quality of the infrared light image by the low-pass filter effect, etc. by adding the information of the visible light image that is out of focus and defocused. It is.

Further, in step S156, the image before infrared light correction driving recorded in step S108 of FIG. As a result, a focused infrared light image, a focused visible light image, an out-of-focus infrared light image, an out-of-focus visible light image, a corrected infrared light image, and a corrected visible light image are aligned. In the next step S158, image fusion processing based on these images is performed.

The content of the image fusion processing in step S158 is basically the fusion of the focused infrared light image and the focused visible light image, thereby enabling image diagnosis that cannot be determined by one alone. In place of this, image diagnosis by fusing a corrected infrared light image and a corrected visible light image is also possible. Furthermore, a soft-focused visible light image with a tight core can be obtained by fusing a focused visible light image and an out-of-focus visible light image. Similarly, a soft focus infrared light image can be obtained from an in-focus infrared light image and an out-of-focus infrared light image. Furthermore, by correcting the focused visible light image with the focused infrared light image, an image according to the visible light image obtained by applying the infrared light cut filter can be obtained. Conversely, by correcting the focused infrared light image with the focused visible light image, it is also possible to obtain an image according to the infrared light image obtained by applying the visible light cut filter. Which of these processes is selected can be set by the operation unit 15, but can be automatically selected according to the subject.

When the process of step S158 as described above is completed, the process proceeds to step S132, the image obtained by the process is recorded, and the flow ends. Note that the image to be recorded in step S132 may be all the images processed in step S158, or may be an image selected in the process of step S158 and finally determined to be necessary. This selection can also be set by the operation unit 15, but it can be configured to automatically select in step S158 according to the processing result.

The application target of the flowchart relating to the image recording function of FIG. 11 and FIG. 12 is not limited to the image recording of the live view sensor 102 of the single-lens reflex camera as in the second embodiment. For example, the image recording to the CIGS sensor The present invention can also be applied to a dedicated visible / infrared light image recording camera. Such a visible light / infrared light image recording camera removes the optical finder system such as the focusing screen 10, the pentaprism 104, and the eyepiece 14 in FIG. 5 and replaces the imaging unit 20 with a CIGS imaging sensor. Can be configured. At that time, an infrared light cut filter 110 and a visible light cut filter 112 are provided so that they can be taken in and out of an optical path that goes straight from the lens optical system 6 toward the CIGS image sensor. Further, instead of the movable mirror 8, a fixed half mirror is provided in which most of the incident light travels straight as a transmission component and the reflected light travels to the lower photometry / AF sensor 34.

FIG. 13 is a block diagram of a third embodiment of the present invention, which constitutes an endoscope system. The endoscope system includes a capsule endoscope 502 that is swallowed into the body, images the inside of the digestive organs, and transmits image data to the outside of the body, and an extracorporeal monitor 504 that receives and monitors the image data transmitted outside the body. The capsule endoscope 502 has a sealed structure having a transparent protective window 506, and an image inside the digestive organ formed by the imaging lens 508 through the protective window 506 is captured by the CIGS imaging sensor 510. The CIGS image sensor 510 has the spectral sensitivity as described with reference to FIG. 3 and can capture an image in the visible light range with high sensitivity and can also perform image capture with high sensitivity even in infrared light. The angle of view and the focus position of the imaging lens 508 can be adjusted by the lens driving unit 512.

The CIGS sensor 510 of the third embodiment does not have a color filter as shown in FIGS. 7 to 10, and a wide range of light ranging from the visible light region to the infrared light region can be incident on all pixels. That is, the light decomposition in the imaging in the third embodiment is not performed by the color filter on the light receiving side, but is performed by switching the light on the light source side. Specifically, red, green, blue, and infrared light emitting diodes (hereinafter referred to as “LEDs” as appropriate) are used as light sources, and these emit light sequentially in a time-sharing manner, whereby CIGS sensor 510 imaging output at each light emission timing. Is the image data for each color.

A large number of LEDs are concentrically provided around the optical axis of the imaging lens 508. In FIG. 13, for the sake of simplicity, one green LED 514 and one infrared LED 516 are shown as an example. For example, the imaging output of the CIGS imaging sensor 510 when the green LED 514 emits light becomes green image data, and the imaging output of the CIGS imaging sensor 510 when the infrared LED 516 emits light becomes infrared image data. In addition, since there is a deviation in the imaging position between visible light and infrared light, the lens driving unit 512 adjusts the imaging position as necessary. The third embodiment is an endoscope, and the inside of the body to be photographed is sufficiently dark, so that the light can be decomposed by time division of the light source light in this way. The relationship between the light source, imaging, imaging lens, etc. will be described in detail later.

The LED driver 518 controls the lighting timing of the LEDs 514 and 516 based on an instruction from the endoscope control unit 520. The endoscope control unit 520 controls the entire capsule endoscope 502, and its function follows a program stored in the storage unit 522. The storage unit 522 further temporarily stores data necessary for the function of the endoscope control unit 520 as needed.

The sensor driver 524 controls the CIGS imaging sensor 510 based on an instruction from the endoscope control unit 520 and stores each color-specific image RAW data from the CIGS imaging sensor in the image buffer 526. The image buffer 526 can store the image RAW data for each color for a predetermined number of times, and the wireless communication unit 528 extracts the image RAW data for each color in the image buffer 526 using a FIFO and transmits it from the antenna 530 to the outside of the body. The battery 532 is configured by a button battery or the like, and supplies power to the entire capsule endoscope 502.

The extracorporeal monitor 504 includes a wireless communication unit 534, receives each color image RAW data transmitted from the capsule endoscope 502 by the antenna 536, and stores the received image RAW data in the image buffer 538. These functions are controlled by the monitor control unit 540. The monitor control unit 540 controls the entire external monitor 504 according to the program stored in the storage unit 542. The storage unit 542 further temporarily stores data and the like necessary for the function of the monitor control unit 540 as necessary.

The image processing unit 544 performs image processing on each color-specific RAW data stored in the image buffer 548 based on an instruction from the monitor control unit 540 into an image signal, and from the red image signal, the green image signal, and the blue image signal, Is stored in the recorder 546. The infrared image signal is also recorded in the recorder 546. The recorded data can be monitored by the display unit 548 as appropriate. Further, the color image signal or infrared image signal from the image processing unit can be directly monitored by the display unit 548 in real time.

FIG. 14 is a front view showing a first example of an LED arrangement that can be employed in the capsule endoscope 502 of the third embodiment. Parts corresponding to those in FIG. As is apparent from FIG. 14, four green LEDs 514 are provided around the imaging lens 508 inside the transparent protective window 506 and are to be rotated 90 degrees apart from each other. A line 550 connecting them is a square. In addition, four infrared LEDs 516 are provided at the apex portion of the square 552 rotated by 45 degrees from the green LED 514 to be rotated objects that are separated from each other by 90 degrees. Further, four red LEDs 556 are provided at the apex portion of the vertically long rectangle 554, and four blue LEDs 560 are provided at the apex portion of the horizontally long rectangle 558. As a result, the red, green, and blue LEDs are arranged in line symmetry in both the vertical and horizontal directions as shown in FIG. 14, and the symmetry of illumination is maintained in both the vertical and horizontal directions for each color. Be drunk.

FIG. 15 is a front view showing a second example of the LED arrangement that can be employed in the capsule endoscope 502 of the third embodiment. Also in FIG. 15, the same reference numerals are given to the portions corresponding to those in FIG. The arrangement of the green LED 514 and the infrared LED 516 in FIG. 15 is the same as that in FIG. On the other hand, four red LEDs 562 are provided on the object to be rotated 90 degrees apart from each other at the apex of a square 564 rotated 22.5 degrees to the left from the green LED 514. In addition, four blue diodes 566 are provided on the top of a square 568 that is rotated 22.5 degrees to the right from the green LED 514 on a rotation target that is 90 degrees apart from each other. As a result, the red, green, and blue LEDs are densely arranged in the four directions, up, down, left, and right, respectively, as seen in FIG. Color unevenness is reduced. Further, since each color is arranged at the apex of the square, the rotation targets are arranged around the optical axis of the imaging lens 508.

FIG. 16 is a front view showing a third example of the LED arrangement that can be employed in the capsule endoscope 502 of the third embodiment. Also in FIG. 16, the same reference numerals are given to the portions corresponding to FIG. The arrangement of the green LED 514 and the infrared LED 516 in FIG. 16 is the same as that in FIG. In contrast, four red LEDs 572 are provided at the apex portion of the vertical rectangle 570 inclined 45 degrees to the left, and four blue LEDs 576 are provided at the apex portion of the vertical rectangle 574 inclined 45 degrees to the right. ing. As a result, the red, green, and blue LEDs are arranged in line symmetry with respect to both the 45 ° left line and the right 45 ° line connecting the opposing infrared LEDs as seen in FIG. The illumination symmetry is maintained in any of these directions. Further, the red, green, and blue LEDs are densely arranged in the four directions of top, bottom, left, and right as seen in FIG. Unevenness is reduced.

As described above, in the LED arrangement examples shown in FIGS. 14 to 16, the green LEDs are arranged vertically and horizontally and the infrared LEDs are arranged at a position rotated 45 degrees from now on. The arrangement is not limited to this, and the entire arrangement may be appropriately rotated in relation to the grid direction of the pixel arrangement of the CIGS image sensor. For example, in the LED arrangement examples shown in FIGS. 14 to 16, the green LEDs are arranged vertically and horizontally with reference to the vertical and horizontal directions of the pixel arrangement of the CIGS imaging sensor. The infrared LED may be arranged vertically and horizontally according to the grid direction of the pixel arrangement by rotating. In this case, the green LED is arranged at a position rotated 45 degrees from now.

FIG. 17 is a front view showing a fourth example of the LED arrangement that can be employed in the capsule endoscope 502 of the third embodiment. When red, green, blue, and infrared LEDs are arranged around the optical axis of the imaging lens 508, the number of LEDs of all colors is not limited to the same number. FIG. 17 employs four green LEDs 514, two red LEDs 578, and two blue LEDs 580 as examples of such. The reason why the number of green LEDs 514 is twice that of red LEDs 578 and blue LEDs 580 is to relatively increase the amount of emitted green light to match the visibility. In FIG. 17, eight infrared LEDs 582 are arranged to increase the amount of infrared light, and the in-vivo observation ability with infrared light is enhanced.

In addition, in FIG. 14 to FIG. 17, a total of 16 LEDs are employed, but the present invention is not limited to this. If arrangement is possible, the total number of LEDs can be further increased to reduce illumination unevenness. Moreover, in order to maintain the minimum symmetry of illumination, it is also possible to adopt a pair of LEDs for each color and to adopt a total of eight LEDs to simplify the configuration. In this case, the line connecting the pair of green LEDs and the line connecting the pair of infrared LEDs are crossed at 90 degrees, and for the pair of red LEDs and the pair of blue LEDs, the line connecting them connects the pair of green LEDs. It is desirable that the red and blue LEDs be adjacent to both sides of the green LED as an arrangement in a state rotated 45 degrees to the left and right of the line. If the arrangement space permits, instead of arranging the eight LEDs at regular intervals in this way, the red and blue LEDs are brought into close contact with both sides of the green LED, and the red, green and blue LEDs are mutually connected. An arrangement in which the positional deviation is as small as possible is also possible.

FIG. 18 shows the light emission timing of each color LED, the operation timing of the photoelectric conversion unit, the operation timing of the AD trunk, and the operation timing of the wireless communication unit in the color / infrared mode of operation in the capsule endoscope 502 of the third embodiment. It is a timing chart which shows the relationship. In the color / infrared mode, a visible color image and an infrared image are acquired in parallel. As is clear from FIG. 18, all red LEDs overlap each other at the timing from t1 to t2, all green LEDs from t3 to t4, all blue LEDs from t5 to t6, and all infrared LEDs from t7 to t8. It lights up in time division without any. When the four color LEDs are turned on, all red LEDs are turned on again at the timing from t9 to t10, and the green, blue, and infrared LEDs are turned on in the same manner, and time-division lighting is performed in the same manner. repeat. The time from t1 to t8 is about one frame time of a normal color moving image, and the light emission amount of each color is less than a quarter of the time without time division, but the CIGS sensor is a normal CMOS as shown in FIG. Compared with the sensor, it has high sensitivity and broadband characteristics, so that even a short amount of light emission provides sufficient light source light.

As shown in FIG. 18, in the color / infrared mode, when visible light and infrared light are emitted almost simultaneously in a time division manner, the angle of view of the imaging lens 508 in FIG. 13 is set to a wide angle by the control of the lens driving unit 512, The depth of focus is set to be deep, and the focus position is also set to be in a pan-focus state that covers visible light to infrared light under the control of the lens driving unit 512. As described above, the color / infrared mode is suitable for observing the overall state of the body roughly.

As is clear from the timing chart of the photoelectric conversion unit in FIG. 18, the photoelectric conversion unit starts red exposure immediately after the start of light emission of the red LED and accumulates charges. Since the charge accumulation time is set immediately before the end of light emission of the red LED, the exposure ends here and the charge is read out. Further, when the charge reading is completed, the residual charge is swept away. When the charge sweeping is completed, the next green exposure is started. As is apparent from FIG. 18, light emission of the green diode starts immediately before the green exposure. Also for the green exposure, after the charge accumulation time is over, the charge read-out and the residual charge sweep out follow. Similarly, blue and infrared charge accumulation, charge readout, and residual charge sweeping are performed in synchronization with the light emission of the blue LED and the light emission of the infrared LED, respectively. These operations circulate. In the above description, the function of the photoelectric conversion unit has been described for each color. However, the photoelectric conversion unit itself does not have a function of performing photoelectric conversion by separating each color, and the photoelectric conversion unit itself simply performs charge accumulation, charge read, and It only repeats the same operation of residual charge sweeping. The read charge amount has information on each color depending solely on the light source color at the time of charge accumulation.

As is clear from the timing chart of the AD conversion unit in FIG. 18, the photoelectric conversion unit starts AD conversion immediately after the charge reading of each color. For example, red AD conversion is started immediately after completion of red charge reading. Then, using the time zone during the next green exposure, the red AD conversion is continued in parallel with this. As is clear from the timing chart (A) of the wireless communication unit in FIG. 18, the wireless communication unit can start communication of digital signals of the resulting color immediately after the completion of photoelectric conversion of each color. For example, communication of a red digital signal is started immediately after the end of red AD conversion. The next green AD conversion time zone is also used, and in parallel with this, red communication is continued. Similarly, AD conversion and communication are performed for green, blue, and infrared.

Regarding communication, depending on the relationship between the capsule endoscope 502 and the extracorporeal monitor 504, there may be a case where the communication cannot be performed successfully immediately after AD conversion. In such a case, communication is executed at the timing when the communication environment becomes sufficient as shown in the timing chart (B) of the wireless communication unit in FIG. For example, IR data transmission 592 is executed later than the timing chart (A), and is executed immediately before the next R data. The G data transmission 594 and the B data transmission 596 are also executed with a delay. However, the adjustment of the communication time causes the FIFO to fail because the capacity of the image buffer 526 in FIG. 13 is full. It is possible as long as there is not.

FIG. 19 is a timing chart showing the relationship between the light emission timing of each color LED and the color image creation in the operation of the capsule endoscope 502 of the third embodiment in the color / infrared mode shown in FIG. As shown in FIG. 19, a red image based on red LED emission starting at t1, a green image based on green LED emission starting at t3, and a blue image based on blue LED emission starting at t5. , F1 is displayed as a color image. Strictly speaking, since there is a time difference in light emission of each color, the images of each color are not of the same time. Similarly, a red image based on red LED emission starting at t9, a green image based on green LED emission starting at t11, and a blue image based on blue LED emission starting at t13, at F2. A one-frame color image shown is created. In the same manner, a color image of one frame is created, and each color moving image can be recorded as a still image, and can also be recorded as a color moving image by connecting them. Note that these color processes are performed by the image processing unit 544 of the extracorporeal monitor 504 shown in FIG. In addition, as shown in the timing chart (B) of the wireless communication unit in FIG. 18, the reception of each color data by the extracorporeal monitor 504 is not necessarily at regular intervals, but the image acquisition timing is determined by the light emission timing of each color LED. A relationship is established.

In addition, as shown in FIG. 19, based on the green image based on the green LED emission starting at t3, the blue image based on the blue LED emission starting at t5, and the red LED emission starting at t9. Since the red image also contains RGB three-color data, a one-frame color interpolated image indicated by I1 is created. Similarly, with a blue image based on blue LED emission starting at t5, a red image based on red LED emission starting at t9, and a green image based on green LED emission starting at t11, I2 A one-frame color interpolated image shown is created. In these interpolated images, the emission of infrared LEDs is intervened until the RGB colors are aligned, and the time until RGB is aligned becomes slightly longer, and the RGB emissions are not equally spaced, so the image quality of the color image is poor. . Therefore, it is adopted as an interpolation image for obtaining a smooth moving image.

On the other hand, as shown in FIG. 19, for the infrared image, an image IR1 based on the emission of the infrared LED starting at t7, an image IR2 based on the emission of the infrared LED starting at t15, and the like are still images. It can be recorded, and can also be recorded as a color video by connecting them. In color / infrared mode, color images and infrared images can be acquired in parallel as described above, so both images can be taken in parallel for endoscopic diagnosis, and both images can be combined. It becomes possible. Also, when a quantity image is synthesized as a still image, the acquisition time of the infrared image is included in the acquisition time zone of the color interpolation image, so the color interpolation image is used for the synthesis with the infrared image. It is also possible. Specifically, since the acquisition time zones of the color interpolation images I2 and I2 both include the acquisition time of the infrared image IR1, it is possible to synthesize the color interpolation images I1 and I2 or their average with the infrared image IR1. Is possible.

FIG. 20 shows the relationship between the light emission timing of each color LED, the operation timing of the photoelectric conversion unit, the operation timing of the AD trunk and the operation timing of the wireless communication unit in the fine color mode in the capsule endoscope 502 of the third embodiment. It is a timing chart which shows. In the fine color mode, only a visible color image is acquired, and the infrared LED does not emit light. As is clear from FIG. 20, all red LEDs are lit in a time-division manner without overlapping each other at the timings t1 to t2, all green LEDs are t3 to t4, and all blue LEDs are t5 to t6. Then, when the lighting of the RGB three-color LEDs is completed, all the red LEDs are turned on again at the timing from t7 to t8. Thereafter, the green, blue, and LEDs are turned on in the same manner, and time-division lighting is repeated in the same manner. In this case, the time from t1 to t6 required for one cycle is shorter than t1 to t8 in FIG. The video becomes fine. The timing chart of the wireless communication unit (B) in FIG. 20 shows a state in which communication is continuously performed after the communication environment has not been prepared for a while.

As shown in FIG. 20, in the fine color mode, when only visible light is emitted almost simultaneously in a time division manner, the angle of view of the imaging lens 508 in FIG. The focus position is also set so that the focus of visible light is imaged on the imaging surface under the control of the lens driving unit 512. This is because the focus position shifts of red, green, and blue are small and can be dealt with by aberration correction in the design of the imaging lens, so that optimum focus position adjustment is possible. Thus, the fine color mode is suitable for observing the state of the body in detail with high definition.

FIG. 21 is a timing chart showing the relationship between the light emission timing of each color LED and the color image creation in the operation of the capsule endoscope 502 of the third embodiment in the fine color mode shown in FIG. As shown in FIG. 21, a red image based on red LED emission starting at t1, a green image based on green LED emission starting at t3, and a blue image based on blue LED emission starting at t5. , F1 is displayed as a color image. Next, indicated by F2 by a green image based on the green LED emission starting at t3, a blue image based on the blue LED emission starting at t5, and a red image based on the red LED emission starting at t7. A color image of one frame is created. Similarly, one frame indicated by F3 by a blue image based on blue LED emission starting at t5, a red image based on red LED emission starting at t7, and a green image based on green LED emission starting at t9. A color image is created. In the same manner, a color image of one frame or less indicated by F4 is created. In this manner, in the fine color mode, a new one-frame color image is created every time the LED of each color emits light, so that smooth fine moving images can be recorded.

FIG. 22 shows the infrared LED emission timing, the photoelectric conversion unit operation timing, the AD trunk unit operation timing, and the wireless communication unit operation timing in the infrared mode of the capsule endoscope 502 of the third embodiment. It is a timing chart which shows a relationship. In the infrared color mode, only an infrared image is acquired, and LEDs other than the infrared LED do not emit light. As is clear from FIG. 22, all the infrared LEDs emit light at every exposure in the photoelectric conversion unit from t1 to t2 and from t3 to t4. In response to this, an infrared image of one frame is created every time. This makes it possible to record a smooth infrared moving image.

As shown in FIG. 22, when only infrared light is emitted in the infrared mode, the angle of view of the imaging lens 508 in FIG. The focus of the infrared light is set to form an image on the imaging surface by the control of the lens driving unit 512. Thus, the infrared mode is also suitable for observing the state of the body in detail with high definition.

FIG. 23 is a flowchart of the operation of the endoscope control unit 520 in the third embodiment of FIG. When the battery 532 is set in the capsule endoscope 502, the flow starts. In step S162, the color / infrared mode is initialized. In response to this, in step S154, the imaging lens 508 is set to the wide-angle and pan-focus state. Next, in step S166, all red, green, blue and infrared LEDs are set to emit light sequentially in a predetermined order. In step S168, the fact that these settings have been made is externally transmitted and reported to the external monitor 504.

Next, an imaging process is performed in step S170, and a transmission process is performed in step S172. Details thereof will be described later. When the transmission process ends, the process proceeds to step S174, and it is checked whether an operation stop signal is received from the extracorporeal monitor 504. If there is reception, the flow is immediately terminated. On the other hand, if no stop signal has been received, the process proceeds to step S176 to check whether a mode change signal has been received. If there is reception, the process advances to step S178 to check whether the changed mode is the color / infrared mode. If it is in the color / infrared mode, the process returns to step S164, the lens is set to the wide angle and pan focus state, and the operation proceeds to the operation after step S166 already described.

On the other hand, if the mode changed in step S178 is not the color / infrared mode, the process proceeds to step S180 to check whether the mode is the fine color mode. In the fine color mode, in step S182, the angle of view of the lens is set to a narrow angle (telephoto) and set to the visible light focus state, and in step S184, only visible light LEDs are set to sequentially emit light with a rotation number. Then, after these setting states are transmitted to the outside in step S186, the process returns to step S170.

If the mode changed in step S180 is not the fine color mode, it means that the changed mode is the infrared mode, so that the process proceeds to step S188, and the angle of view of the lens is narrowed (telephoto). And set the infrared light focus state. Furthermore, it sets so that only infrared LED may light-emit in step S190. Then, the process proceeds to step S186, and after these setting states are transmitted to the outside, the process returns to step S170.

FIG. 24 is a flowchart showing details of the imaging process in step S170 of FIG. When the flow starts, it is checked in step S192 whether a mode has been selected or changed. If no mode has been selected or changed, the process proceeds to step S194 to check whether the exposure time has been completed. If the completion is detected, the process proceeds to step S196, where the stored charge readout start process is performed. Further, in step S198, an instruction to stop LED emission is given. Further, in step S200, it is checked whether or not the accumulated charge reading is completed. If it is not completed, the completion is waited while repeating step S200.

When the completion of reading is detected in step S200, the process proceeds to step S206, a residual charge sweeping start process is performed, and the process proceeds to the LED selection process in step S208. This is a process of selecting the next LED to emit light, the details of which will be described later. Further, in step S210, the start of light emission of the LED selected in step S208 is instructed. Next, in step S212, it is checked whether or not the stored charge has been completely purged.

When it is detected in step S212 that the accumulated charge has been completely removed, the process proceeds to step S214 to start exposure, and in step S216, the exposure time starts to be counted, and the flow ends. On the other hand, if the exposure time is not completed in step S194, the flow is immediately terminated. If selection or change of the mode is detected in step S192, the process proceeds to step S218, the imaging process is initialized, and the process proceeds to the LED selection process in step S208.

FIG. 25 is a flowchart showing details of the LED selection processing in step S208 of FIG. When the flow starts, it is checked whether or not the infrared mode is set in step S222. If not applicable, it means color / infrared mode or fine color mode. In this case, the process proceeds to step S224, and it is checked whether or not the imaging process has been initialized in step S218 of FIG. If this is not the case, the process advances to step S226 to read the previously selected LED memory. Then, based on the memory read in step S228, it is checked whether or not the last light emission was a red LED. If not, it is further checked in step S230 whether the last time it was a green LED.

If it is determined in step S230 that the previous light emission was not a green LED, the process proceeds to step S232 to check whether the color mode is the fine color mode. If this is the case, the red, green, and blue LEDs are emitting light on the wheel. In this case, if the previous light emission is neither red nor green, it means that the light was blue, the process proceeds from step S232 to step S234, and the red LED corresponding to the next order is selected. Then, the selection result is stored in step S236, and the flow ends.

On the other hand, if the fine color mode is not detected in step S232, it means the color / infrared mode. In this case, red, green, blue, and infrared LEDs emit light in a ring. In this case, it means that if the previous light emission is neither red nor green, it means that the light was blue, so the process proceeds from step S232 to step S238, and the infrared LED corresponding to the next order is selected. Then, the selection result is stored in step S236, and the flow ends.

If it is the red LED that previously emitted light in step S228, the process proceeds to step S242, and the green LED corresponding to the next order is selected. Then, the selection result is stored in step S236, and the flow ends. Further, when the infrared mode is set in step S222, the process proceeds to step S244, and an infrared LED is selected. In the infrared mode, it is always the infrared LED that is selected, so that the selection result need not be stored in particular and the flow is immediately terminated. If the imaging process has been initialized in step S224, the process proceeds to step S242 to select the green LED as the first light emitting LED.

FIG. 26 is a flowchart showing details of the transmission processing in step S172 of FIG. When the flow starts, it is checked in step S252 whether data is being transmitted. If transmission is not in progress, the process advances to step S254 to check whether there is data that has been successfully transmitted. If there is a corresponding item, it is erased from the image buffer in step S256, and the process proceeds to step S258. On the other hand, if there is no successful transmission data in step S254, the process directly proceeds to step S258.

In step S258, it is checked whether or not AD conversion has been completed. If applicable, the AD converted data is stored in the image buffer, and the process proceeds to step S262. On the other hand, if AD conversion is not completed, the process directly proceeds to step S262. In step S262, it is checked whether there is data stored in the image buffer. If there is data, it is checked in step S264 whether the communication state is OK. If it is OK, data is read from the image buffer by FIFO (first-in first-out) in step S266, the transmission of the read data is instructed in step S268, and the flow is terminated. It should be noted that when the data is being transmitted in step S252, when no data is stored in the image buffer in step S262, or when the communication state is not OK in step S264, the flow is immediately terminated.

FIG. 27 is a flowchart showing the operation of the monitor control unit 540 of the extracorporeal monitor 504 in the third embodiment of FIG. 13, and the flow starts when communication with the capsule endoscope 502 is started. When the flow starts, it is checked in step S272 whether there is a new arrival of image data. If there is new arrival data, the process proceeds to step S274 to check whether the new arrival data is complete. If it is complete, the process proceeds to step S276, where it is stored in the image buffer 538, and the process proceeds to step S278. If there is no new arrival of data in step S272 or new arrival data is not complete in step S274, the process returns to step S272.

In step S278, it is checked whether the infrared mode is selected. If not, the color / infrared mode is checked in step S280. If it is the color / infrared mode, the process advances to step S282 to check whether the newly arrived data is infrared image data. If not, it means that the image data is one of red, green, and blue, so the process proceeds to step S284, and an instruction to create an interpolation auxiliary image of an infrared image is given from these data, and the process proceeds to step S286. . The instruction in step S284 uses information of visible light image data obtained at a timing between infrared images as an auxiliary when performing interpolation based on infrared image data in order to obtain a smooth infrared moving image. It is an instruction for.

In step S286, it is checked whether the new inverse data is blue image data. If it does not correspond, it means red image data or green image data, so the process proceeds to step S290, and it is checked whether or not the visible two-color data immediately before the new arrival data is stored. If it is stored, red, green, and blue are available together with newly arrived data, so the process proceeds to step S292, and an instruction to create a storage color image is issued, and the process returns to step S272. The image created by the instruction in step S292 corresponds to the color interpolation image I1 or I2 in FIG.

When it is the infrared mode in step S278, the newly arrived data is infrared image data, so the process proceeds to step S294, the creation of the infrared image is instructed, and the process returns to step S272. Also, when the newly arrived data is infrared image data in step S282, the process proceeds to step S294. On the other hand, if the newly arrived data is blue image data in step S286, the process proceeds to step S296, and it is checked whether the image data of the immediately preceding two colors (in this case, red and green) are stored in the image buffer. If these are stored, three consecutive colors are prepared, so the process proceeds to step S298, where an instruction to create a color image is issued, and the process returns to step S272. The image created by this instruction corresponds to the color image F1 or F2 in FIG.

If the color / infrared mode is not set in step S280, it means that the mode is the fine color mode. Therefore, the process proceeds to step S296, where it is checked whether the previous visible two-color data is stored. If there are two colors, an instruction to create a color image in step S298 is given. Images created by this instruction correspond to the color images F1, F2, F3, etc. in FIG. If no previous visible two-color data is stored in step S290 or step S296, the process immediately returns to step S272.

FIG. 28 is a timing chart showing the relationship of the operation timing of the fourth embodiment of the present invention. Since the fourth embodiment is basically configured in common with the endoscope system shown in FIGS. 13 to 17, the following description will be made using the reference numerals in the block diagram of FIG. 13 as appropriate. The fourth embodiment differs from the third embodiment in the configuration of the CIGS imaging sensor and the light emission timing of the LEDs. That is, the CIGS sensor 510 of the third embodiment does not have a color filter, and the color separation is based on the time-division light emission of the LED, but the CIGS imaging sensor 510 of the fourth embodiment is as shown in FIGS. The CIGS imaging sensor itself performs color separation in the same manner as in the first and second embodiments. The light emission of the LEDs is not performed in a time division manner, but is performed simultaneously for all colors.

FIG. 28 shows the LED light emission timing, the photoelectric conversion unit operation timing, the AD trunk unit operation timing, and the wireless communication unit operation in the color / infrared mode in the capsule endoscope 502 of the fourth embodiment. It is a timing chart which shows the relationship of operation timing. As described above, all the red LEDs, all the green LEDs, all the blue LEDs, and all the infrared LEDs all emit light simultaneously at the exposure timing of the photoelectric conversion unit. Each LED may emit light continuously instead of blinking as shown in FIG. In the color / infrared mode of FIG. 28, the angle of view of the imaging lens 508 is set to a wide angle by the control of the lens driving unit 512, and the focal depth is set to be deep, and the focus position is also lens driven. Under the control of the unit 512, the pan focus state is set so as to cover visible light to infrared light. Such imaging lens control is common to the color / infrared mode of FIG.

FIG. 29 shows the relationship among the LED light emission timing, the photoelectric conversion unit operation timing, the AD trunk unit operation timing, and the wireless communication unit operation timing in the fine color mode operation in the capsule endoscope 502 of the fourth embodiment. It is a timing chart which shows. As is clear from FIG. 29, all red LEDs, all green LEDs, and all blue LEDs all emit light simultaneously at the exposure timing of the photoelectric conversion unit. The infrared LED does not emit light. As shown in FIG. 29, when only visible light is emitted simultaneously in the fine color mode, the angle of view of the imaging lens 508 in FIG. The focus of visible light is set to form an image on the imaging surface under the control of the driving unit 512. Such imaging lens control is common to the fine color mode in FIG.

The timing chart of the infrared mode in the fourth embodiment is the same as that in FIG. 22 in the third embodiment. In addition, the angle of view of the imaging lens 508 in the infrared mode is set to a narrow angle (telephoto) by the control of the lens driving unit 512, and the focus position of the imaging lens 508 is also controlled by the lens driving unit 512 to focus the infrared light on the imaging surface. The point set to image is also common to the infrared mode in FIG. The color / infrared mode in the fourth embodiment is suitable for observing the overall state of the body roughly, while the fine color mode and infrared mode are suitable for observing the state of the body in detail with high definition. This is also the same as the third embodiment.

In addition, although the endoscope system in the said 3rd Example and 4th Example was comprised as what has a capsule endoscope and an external monitor, implementation of this invention is not restricted to this. For example, it can be configured as a normal endoscope in which the inside and outside of the body are connected by a tube. In this case, wireless communication by the antennas 530 and 546 in FIG. 13 is wired communication using a cable in the tube, and a known vent pipe, water conduit, tube bending mechanism, and the like are provided in the tube. Further, instead of performing image information transmission between the inside and outside of the body using an electrical signal, an image acquired inside the body may be taken out of the body by optical means such as a fiber. In this case, the CIGS image sensor is provided outside the body. Further, the light source can also be provided outside the body and guided to the inside by a light guide. In such a configuration, the light source arrangement in FIGS. 14 to 17 is understood not as the arrangement of the light emitting portion but as the arrangement of the light source light emitting portions. Furthermore, when using a light guide, it is not always necessary to provide a light source light emitting portion for each color, and light from each color light emitting portion is guided into the body using a common light guide and is emitted from a common outlet. Also good. In addition, the implementation of the various features of the present invention shown in the third and fourth embodiments is not limited to an endoscope, but may be used as appropriate for imaging / observation / recording equipment using various imaging sensors. Is also possible.

FIG. 30 is a block diagram of a fifth embodiment of the present invention, which constitutes a vehicle monitoring device. The vehicle monitoring function is a function for detecting the distance between vehicles before and after the vehicle for preventing a collision and a function for capturing and recording images before and after the vehicle as a drive recorder. The detected inter-vehicle distance is recorded in a drive recorder and also used for focus adjustment for image shooting. In FIG. 30, a vehicle 602 is configured as either a gasoline engine vehicle, an electric vehicle, or a so-called hybrid vehicle that uses a gasoline engine and a motor together.

The vehicle 602 includes a vehicle control unit 604 including a computer that controls the entire vehicle, and controls the vehicle function unit 610 having a power 606 and a brake 608 in accordance with an operation by a driver of the vehicle to cause the vehicle 602 to travel. . The power 606 is configured to include a gasoline engine and / or motor. The vehicle control unit 604 includes a storage unit that stores software and data necessary for controlling the vehicle 602. In addition, the vehicle control unit 604 controls the in-vehicle output unit 612 and causes the image output unit 614 to display a GUI display necessary for operation of the vehicle and display the control result. The voice output unit 616 also makes an announcement to the driver.

The GPS unit 618 obtains latitude, longitude, and altitude information, which are absolute position information of the vehicle 602, from the satellite and the nearest broadcasting station based on the GPS system and sends the information to the vehicle control unit 604. The car navigation function unit 620 processes absolute position information from the GPS unit 618 obtained via the vehicle control unit 604 and causes the image output unit 614 to display the position of the vehicle 602 on the map.

Next, the configuration relating to the vehicle monitoring device will be described in association with the above configuration as necessary. The fifth embodiment includes substantially the same monitoring device for monitoring the front and rear of the vehicle. First, a monitoring device behind the vehicle will be described. The rear camera / sensor 622 includes a camera and a microphone 628 including an imaging lens 624 and a CIGS imaging sensor 626. When the vehicle 602 is in a traveling state, an image such as the front of the vehicle and surrounding sounds are always input.

The CIGS imaging sensor 626 basically has the characteristics as shown in FIG. 3 described in the embodiments so far, and has the color filter array as described with reference to FIGS. However, the spectral transmittance of the color filter has the characteristics peculiar to the fifth embodiment, and the red transmission filter, the blue transmission filter, and the green transmission filter are band-pass filters that transmit light in each narrow visible light region. In addition, the infrared light transmission filter is a band-pass filter that transmits infrared light in a narrow area of the solar blind. A solar blind is a wavelength region of sunlight that reaches the earth that cannot reach the surface of the earth due to atmospheric absorption or is weak even if it reaches the surface. It has been. In the fifth embodiment, a visible light image is captured by red, blue, and green pixels and the presence or absence of sunlight by infrared pixels by combining the light receiving sensitivity of the CIGS sensor as shown in FIG. 3 and the filter as described above. There is no distance detection. Details thereof will be described later.

The infrared beam scanner 630 projects an infrared light beam in the solar blind area under the control of the monitoring recording control unit 632 and scans the rear of the vehicle. When the projected infrared light beam is reflected by the following vehicle, it is received by the infrared pixels of the CIGS imaging sensor 626. The image signal of the CIGS sensor 626 is processed by the rear image processing unit 636 via the rear sensor driver 634, and the reflection position of the infrared beam is detected. The monitoring and recording control unit 632 calculates the inter-vehicle distance to the following vehicle that reflects the infrared light beam from the reflection position information detected by the rear image processing unit 636 and the infrared light beam projection angle at that time. The monitoring recording control unit 632 controls the rear AF (autofocus) control unit 638 according to the calculated inter-vehicle distance, and drives the imaging lens 624 by the AF driving unit 640 to reflect the infrared light beam with respect to the following vehicle. Focus on visible light.

On the other hand, the visible light image information detected by the visible light pixel of the CIGS imaging sensor 626 based on the focus adjustment as described above is processed into a visible image by the rear image processing unit 636 via the rear sensor driver 634, and is monitored and recorded. The sound information from the microphone 628 is input to the FIFO (first-in first-out recording unit) 642 via the unit 632. The FIFO (first-in first-out recording unit) 642 includes a nonvolatile buffer memory having a capacity of about 20 seconds, and stores image and sound information input from the rear image processing unit 636 and the microphone 628 in a first-in first-out manner. That is, the latest 20-second information is always overwritten and stored.

Then, when the abnormal acceleration detection unit 644 detects a large acceleration change due to a collision or the like, or particularly when a manual operation for instructing recording is performed, the recording of the FIFO 642 is maintained as evidence. Specifically, the monitoring record control unit 632 is connected to the vehicle control unit 604, and abnormal acceleration detection or manual operation by the abnormal acceleration detection unit 644 is transmitted to the monitoring record control unit 632 via the vehicle control unit 604. If the rear image processing unit 636 and the microphone 628 are continuously input without being destroyed, the storage in the FIFO 0642 is continued for about 10 seconds from that point. Thereby, the FIFO 642 finally holds image and sound information of about 10 seconds before and after the time of detection of acceleration change or manual operation. Then, the image and sound information of about 10 seconds before and after this is automatically transferred to the nonvolatile recording unit 646 and stored as evidence. This achieves the drive recorder function. Even if most of the drive recorder function is destroyed due to a collision or the like, if even the non-volatile FIFO 642 is safe, the recording of about 20 seconds before the accident is maintained.

When recording the image of the following vehicle, when the inter-vehicle distance is shortened, not only the number of the following vehicle but also the face of the driver of the front seat is clearly recorded through the windshield. These images will be discarded after 20 seconds in the FIFO unless there is an abnormality. However, in order to protect privacy, the monitoring and recording control unit confirms that the following vehicle has narrowed the inter-vehicle distance more than a predetermined distance. If 632 detects, this will be sent to the vehicle control part 604. In response to this, the vehicle control unit 604 causes the rear vehicle drive recorder (drive recorder) notification display unit 648 to display a notification such as “being rear-photographed” on the rear window of the vehicle 602. This is an advance notice for alleviating troubles, and it can be expected that a subsequent vehicle that dislikes photographing will voluntarily take an appropriate inter-vehicle distance. further,

In the fifth embodiment, a monitoring device similar to the rear monitoring device described above is provided in front of the vehicle. Since the configuration is the same as that of the rear monitoring device, the corresponding configuration is numbered in the 700s with the same number at the 10's and 1's, and the description is omitted unless particularly necessary. Note that the front visible light image information from the front image processing unit 736 of the front monitoring device is recorded in the FIFO 642 and the recording unit 646 in the same manner as the rear visible light image. For this reason, the FIFO 642 and the recording unit 646 have recording areas for the rear side and the front side, respectively.

In the case of the front monitoring device, when the monitoring record control unit 632 detects that the own vehicle 602 has narrowed the inter-vehicle distance to the front vehicle more than a predetermined value and this is sent to the vehicle control unit 604, When the voice output unit 616 of the output unit 612 makes a warning announcement such as “take a distance between vehicles”, and when the distance between the vehicles is reduced to a dangerous area due to falling asleep, the brake 608 is automatically activated, Prevent collisions.

FIG. 31 is a block diagram showing the details of the rear camera / sensor 622 together with the distance detection principle, and is an arrangement conceptual diagram viewed from the top. The same parts as those in FIG. 30 are denoted by the same reference numerals. In FIG. 31, an infrared beam scanner 630 includes a solar blind beam light source 802 made of a laser or a light emitting diode and projecting a beam, a polygon mirror for reflecting the projected beam and scanning in parallel with the road surface, and a resonant galvanometer mirror. Alternatively, a two-dimensional scanning system 804 including an acousto-optic deflection element is included. The beam from the solar blind beam light source 802 has a vertically long cross section as will be described later in order to cover the following vehicle corresponding to the up and down of the road. For the same purpose, the two-dimensional scanning system 804 is configured as a spatial scanning system including a vertical scanning in order to cover a succeeding vehicle that may be positioned in the horizontal and vertical directions corresponding to the up and down of the road. May be.

The optical axis 806 of the CIGS imaging sensor 626 is disposed substantially at the center of the vehicle 602, but the rotation center of the scanning beam by the two-dimensional scanning system 804 is provided at a position away from the optical axis 806. Since the distance between the rotation center of the projection beam and the optical axis 806 is a so-called triangulation baseline length, in principle, it is desirable that the two be separated as much as possible. Therefore, if the arrangement is possible, for example, the infrared beam scanner 630 is preferably provided in a part of the taillight. However, on the other hand, when priority is given to assembling the rear camera / sensor 622 as compact as possible, it is preferable to arrange the infrared beam scanner 630 and the CIGS imaging sensor close to each other as long as accuracy can be guaranteed. It is.

Since the rear camera / sensor 622 is configured as described above, for example, when the following vehicle exists at a position 810 immediately behind the vehicle 622, the reflected light 814 of the projection beam at the angle 812 is directed to the CIGS imaging sensor 626. That is, the image of the reflected beam 814 is imaged on the optical axis 806 of the CIGS imaging sensor 626, that is, at the center in the left-right direction of the screen. The combination of the image position of the reflected beam 814 and the angle of the projection beam is uniquely determined by the position 810 of the following vehicle, and the angle 812 of the projection beam at that time is grasped by the monitoring recording control unit. The distance from the combination of the imaging position of the reflected beam 814 to the following vehicle at the position 810 is obtained.

Similarly, when the following vehicle is present at a position 816 immediately behind the vehicle 622, the reflected beam 820 by the projection beam at the angle 818 is directed to the CIGS imaging sensor 626. At this time as well, the image of the reflected beam 820 is captured at the center of the screen of the CIGS imaging sensor 626 in the left-right direction. When the succeeding vehicle is at position 822, the reflected light beam 824 image when the projection beam is at an angle 818 is picked up by the CIGS image sensor 626 as in the case of the position 816. However, the imaging position of the reflected light beam 824 at this time is not the center of the left and right sides of the screen, but is imaged on the right side of the center of the screen in FIG. Will never be done.

FIG. 32 is a conceptual diagram when an image captured by the CIGS imaging sensor 624 is viewed by the image output unit 614 of the in-vehicle output unit 612. FIG. 32A corresponds to the case where the following vehicle is present at the position as shown in FIG. 31, and the succeeding vehicle at the nearest rearmost position 810 and the reflected beam 814 thereby are observed. Since the cross section of the projection beam is vertically long as described above, the image of the reflected beam 814 from the succeeding vehicle has a vertically long shape. The reflected beam 814 is shown only in the center of the following vehicle at the position 810 for the sake of simplicity, but the reflected beam is actually banded while the projection beam is scanned from the left end to the right end of the front surface of the following vehicle. Detected. Since the scan speed is sufficiently higher than the vehicle speed, the movement of the vehicle position during the scan can be ignored. Therefore, in order to determine the angle of the projection beam corresponding to the image of the reflected beam 814 as shown in FIG. 32 (A) from the image of the reflected beam extending over the entire front surface of the vehicle in this manner. A process for obtaining the center of gravity of the belt-like reflected beam image extending over the front of the vehicle is necessary. Finding and processing such a center of gravity will be described later.

In FIG. 32 (A), in the same manner as the following vehicle at the position 810 as described above, the vehicle follows the following vehicle at the far rear position 816, the center of gravity of the reflected beam 820, and the position 822 shifted to the right side. The following vehicle and the center of gravity of the reflected beam 824 are shown in the image output unit 614. Note that FIG. 32 is a conceptual diagram, and for convenience of explanation, a plurality of succeeding vehicles are shown simultaneously in a bird's-eye view, but an actual image is different from this. That is, when the vehicle 602 and each subsequent vehicle are traveling on a flat ground, the optical axis 806 of the CIGS imaging sensor is set in parallel to the road surface, so that the subsequent vehicle at the position 816 is at the position 810 just before it. If there is a following vehicle, it is hidden behind and cannot be seen at all. In addition, the following vehicle that is traveling at an obliquely rear position 822 is imaged so as to overlap the following vehicle at the forward position 810. Note that processing in the case where a plurality of succeeding vehicles overlap in this way will be described later.

FIG. 32B illustrates a case where the following vehicle has abnormally approached as indicated by a position 826. As described above, for such a succeeding vehicle, a “rear vehicle driving information display” 648 indicates “rearward”. An indication such as “Shooting is in progress” is displayed. For privacy reasons, unless there is an abnormal state, the image is subjected to a windshield mosaic process 828 and a license plate mosaic process 830. This mosaic processing is canceled when the abnormal acceleration is detected by the abnormal acceleration detector 644 due to an accident or the like. Therefore, in order to preserve evidence, an image showing the driver's face, license plate, etc. is recorded.

FIG. 33 is a graph for explaining the spectral transmission characteristics of the filter of the CIGS imaging sensor 626 in the fifth embodiment. The spectral sensitivity (quantum efficiency) 832 of the CIGS sensor is conceptually the same as that shown in FIG. On the other hand, the spectral irradiance 834 of sunlight on the ground surface has a solar blind region where the spectral irradiance of sunlight is reduced on the comb teeth upon absorption of the atmosphere. This corresponds to around 1100 nm and around 1400 nm. Comparing the two, it can be seen that the CIGS sensor has high spectral sensitivity in the solar blind region of sunlight around 1100 nm. Therefore, in the fifth embodiment, a bandpass filter having a spectral transmittance 836 having a peak in the solar blind region of sunlight near 1100 nm as an infrared light transmitting filter in the color filter array as described with reference to FIGS. Is adopted. The solar blind beam light source of FIG. 31 has a strong spectral intensity in the transmission region of this bandpass filter. In this way, the number of infrared pixels of the CIGS image sensor 626 can detect the reflected light of the beam projected from the infrared beam scanner regardless of the presence of sunlight.

FIG. 33 further shows spectral transmittances 838, 840, and 842 of bandpass filters of a narrow visible light region that are respectively employed for a red transmission filter, a green transmission filter, and a blue transmission filter. In these wavelength ranges, the spectral irradiance 834 of sunlight on the ground surface is sufficient, and the spectral sensitivity (quantum efficiency) 832 of the CIGS sensor is extremely high, so that visible light can be imaged. As described above, in the fifth embodiment, since all the filters including the spectral transmittance 838 of the red transmission filter are band pass filters that transmit a narrow wavelength region, imaging of visible light and solar blind region are possible. It is possible to simultaneously detect the distance by the projection beam. Note that the focus position of the imaging lens 604 is determined based on visible light. In this case, the infrared pixel is not in focus, but there is no problem for the purpose of obtaining the center of gravity of the reflected beam.

FIG. 34 is a flowchart of the operation of the monitoring record control unit 632 in the fifth embodiment of FIG. The flow starts when the vehicle 602 starts traveling. In step S302, the start of drive recorder recording is instructed. In step S304, the start of scanning with a solar blind infrared beam is instructed, and the process proceeds to step S306. In step S306, based on the above-described instructions, the visual images sent from the front and rear image processing units, the sound from the front and rear microphones, and the FIFO recording of the front and front detection distances are instructed.

Next, in step S308, an inter-vehicle distance detection process based on the beam angle from the infrared beam scanner 630 and the solar blind area captured image of the CIGS imaging sensor is performed. This inter-vehicle distance detection is performed for both the front vehicle and the rear vehicle, details of which will be described later. When the front and rear inter-vehicle distances are detected in step S308, in step S310, the front camera / sensor 722 and the rear camera / sensor 622 are automatically focused on the camera for visible light shooting, respectively. .

In the next step S312, the presence or absence of an abnormal approach to the preceding vehicle is checked based on the front inter-vehicle distance detected in step S308. When an abnormal approach between the front vehicles is detected, the process proceeds to step S314, where an instruction to decelerate by automatically operating the brake with a safe strength is given, and an announcement of an abnormal inter-vehicle distance is given at step S316, and the process proceeds to step S318. On the other hand, if no forward inter-vehicle abnormal approach is detected in step S312, the process proceeds directly to step S318. Here, when more detailed processing according to the inter-vehicle distance is desired, the inter-vehicle distance determination in step S312 is made in two stages, and when the inter-vehicle distance is relatively large, only the warning announcement instruction in step S316 is kept, and the inter-vehicle distance is shorter. Only when this occurs, the brake automatic operation of step S314 is applied. Further, it can be configured to select in advance whether such a fine treatment or one-step processing as shown in FIG.

In step S318, based on the distance between the rear vehicles detected in step S308, it is checked whether the rear vehicle is abnormally approaching. When an abnormal approach between the rear vehicles is detected, the process proceeds to step S3320, and an announcement of the abnormal inter-vehicle distance is instructed. Further, in step S322, a notice display such as “being photographed in the rear” is displayed on the rear window of the vehicle 602 so that it can be seen by the driver of the rear vehicle. In step S324, an instruction is given to perform windshield mosaic processing 828 and license plate mosaic processing 830 on the rear vehicle image captured by the rear camera / sensor 622, and the process proceeds to step S326. On the other hand, if an abnormal approach between the rear vehicles is not detected in step S318, the process proceeds to step S328, the countermeasure for the rear vehicle corresponding to step S324 is canceled from step S320, and the process proceeds to step S326. This is a process that is necessary when the inter-vehicle distance from the rear vehicle increases corresponding to the detection of the abnormal approach state and the vehicle returns to the normal state.

In step S326, it is checked whether or not an abnormal acceleration is detected. If there is a detection, the rear vehicle front / number plate mosaic process is canceled in step S330. In step S332, the image / sound / distance for 10 seconds after the detection of the abnormal acceleration is waited to be input to the FIFO 642, and the image / sound / distance for 10 seconds before and after the detection of the abnormal acceleration in the FIFO 642 thus secured is obtained. Recording is performed in the recording unit 646, and the process proceeds to step S334. On the other hand, if no abnormal acceleration is detected in step S326, the process directly proceeds to step S334.

In step S334, it is checked whether or not the vehicle has stopped. If not stopped, the process returns to step S306, and thereafter steps S306 to S334 are repeated to continue recording and detection by the front camera / sensor 722 and the rear camera / sensor 622. While dealing with various conditions. On the other hand, when the stop of the vehicle is detected in step S334, the process proceeds to the stoppage process in step S336. There are cases in which a vehicle accident occurs not only when the vehicle is running, but also when the vehicle crashes from behind while the vehicle is stopped by a signal, for example. The stop processing is for preserving evidence even for such an accident while the vehicle is stopped.

FIG. 35 is a flowchart showing details of the stopping process of FIG. When the process proceeds from the step S334 in FIG. 34 to the stoppage process in FIG. 35, it is first checked in step S342 whether or not the vehicle is traveling. If it is detected that the vehicle is traveling, the process returns to step S306 in FIG. On the other hand, if it is confirmed that the stop is continued, the process proceeds to step S344, and the visible image continuously transmitted from the front and rear image processing units, the sound from the front and rear microphones, the detection of the front and front are detected. Instructs FIFO recording of distance.

Next, in step S336, an inter-vehicle distance detection process based on the beam angle from the infrared beam scanner 630 and the solar blind area captured image of the CIGS imaging sensor is performed. This is the same process as step S308 in FIG. When the front and rear inter-vehicle distances are detected in step S346, in step S348, the front camera / sensor 722 and the rear camera / sensor 622 are automatically focused on the camera for visible light shooting based on this. . This is also the same as step S310 in FIG.

In the next step S350, based on the front inter-vehicle distance detected in step S308, the presence or absence of abnormal approach on the premise of stopping is checked for the preceding vehicle. The criterion for determining the abnormal approach is a distance closer than that in step S312 of FIG. Since the vehicle 602 has already stopped, such a case corresponds to a case where the preceding vehicle has come back without noticing. Accordingly, when an abnormal approach between the vehicles on the premise of stopping is detected, the process proceeds to step S352, the horn is automatically actuated to call attention to the preceding vehicle, and the process proceeds to step S354. On the other hand, if it is determined in step S3520 that an abnormal approach between the vehicles on the premise of stopping is not detected, the process directly proceeds to step S354.

In step S354, based on the distance between the rear vehicles detected in step S308, it is checked whether the rear vehicle is abnormally approaching. This check criterion is the same as that in step S318 in FIG. However, when the rear vehicle abnormal approach is detected here, the warning announcement instruction and the rear car dorareko notification display instruction as shown in FIG. 34 are not performed, and only the rear car front / number mosaic process of step S356 is performed and the process proceeds to step S358. Transition. This is because it is natural that the distance between the vehicle and the rear vehicle is shortened when waiting for a signal, etc., and the warning announcement instruction in such a state is unnecessary, and the rear vehicle dorareko notification display instruction is This is a cause of trouble. Therefore, it is limited to privacy considerations by the rear car front / number mosaic processing. On the other hand, if an abnormal approach between the rear vehicles is not detected in step S356, the process proceeds to step S360, the rear vehicle countermeasure is canceled, and the process proceeds to step S358. This is to cancel the rear vehicle front / number mosaic process in a state where the rear vehicle is far away when the host vehicle is stopped.

In step S358, on the basis of the front inter-vehicle distance detected in step S308, it is checked whether or not there is an abnormal approach on the assumption that the rear vehicle is stopped. The criterion for determining the abnormal approach is a closer distance than in step S354. When such an abnormal approach is detected, the process proceeds to step S362, the foot brake is automatically operated, and the process proceeds to step S364. This is because, when the vehicle is stopped only by the side brake, in the unlikely event of a rear-end collision, the stopping ability of the vehicle is enhanced to reduce the shock as much as possible. On the other hand, if it is not detected in step S358 that an abnormal approach between the vehicles behind the stop is detected, the process directly proceeds to step S364.

In step S362, it is checked whether or not an abnormal acceleration is detected. If there is a detection, the rear vehicle front / number plate mosaic process is canceled in step S366. In step S368, the image / sound / distance for 10 seconds after the abnormal acceleration is detected is input to the FIFO 642, and the image / sound / distance for 10 seconds before and after the abnormal acceleration detection in the FIFO 642 thus secured is determined. Recording is performed in the recording unit 646, and the process proceeds to step S334. These are the same as step S330 and step S332 of FIG. On the other hand, if no abnormal acceleration is detected in step S364, the process directly proceeds to step S370.

In step S370, it is checked whether the power is stopped. If it is stopped, the process proceeds to step S372, and the front image processing unit 736 and the rear image processing unit based on the signals from the front camera / sensor 722 and the rear camera / sensor 622 are checked. By the image processing at 636, it is checked whether or not a moving object is not captured in the image for a predetermined time or more. If a moving object is detected, the process returns to step S342, and thereafter, steps S342 to S372 are repeated to cope with various states while continuing recording and detection by the front camera / sensor 722 and the rear camera / sensor 622. When a moving object is detected in step S372, the process returns to step S342 because there is a possibility that the vehicle 602 may be involved in an accident if there is a running vehicle even if the vehicle 602 is stopped and the power is stopped. This is to maintain evidence. On the other hand, if no moving object is detected for a predetermined time or more in step S372, it means that the vehicle 602 has been stored in a safe position such as a garage, and thus the flow ends. When it is detected in step S370 that the power is not stopped, the process returns to step S342, and various states are dealt with while recording and detection by the front camera / sensor 722 and the rear camera / sensor 622 are continued.

FIG. 36 is a flowchart showing details of the inter-vehicle distance detection process in step S308 of FIG. 34 and step S346 of FIG. This flow is performed for each of the image of the front camera / sensor 722 and the rear camera / sensor 622, but in the following description, for the sake of simplicity, it will be described as processing for the image from the rear camera / sensor 622. When the flow starts, a visible light image is processed in step S382. Based on this processing, it is checked in step S384 whether there is a vehicle image behind the vehicle 602 in the same lane. If there is an image, the process proceeds to step S386, and the objectivity of the image is checked. This means checking whether a plurality of vehicles overlap each other. If there is no target, the process proceeds to step S388, and matching is performed with several patterns of vehicle overlap images prepared in advance.

Next, in step S386, a matching pattern is found, and it is checked whether images of only the closest one of a plurality of overlapping vehicles can be separated based on the pattern. If separation is possible, the process proceeds to step S392. On the other hand, if there is symmetry in the vehicle image in step S386, it is determined that there is no overlap, and the process directly proceeds to step S392. Then, the distance of the vehicle is determined from the size of the image of the single vehicle separated in step S392. However, this distance determination is only rough because the vehicle itself has variations in size ranging from light cars to large buses. Further, in step S394, the center of gravity in the visible light image of the separated single vehicle is detected, and the process proceeds to step S396.

In step S396, solar blind pixel image processing is performed. In step S398, it is checked whether or not there is an image group whose output is greater than or equal to a predetermined value. This is equivalent to checking whether the projected beam in the solar blind area scanning backward is reflected by the vehicle behind and is detected by the solar blind pixel of the rear camera / sensor 622. This is because the reflected light of the projection beam does not have a predetermined intensity or more unless there is a close object such as a vehicle.

If there is a pixel group having a predetermined output or more in step S398, the process proceeds to step S400, and it is checked whether or not there is a portion in the pixel group that matches the center of the visible light image detected in step S494. If there is a corresponding portion, this is regarded as the center of gravity of the reflected beam, and in step S402, the angle information of the projection beam that is the origin of the center of the reflected beam is acquired. Thus, in step S404, the distance to the vehicle that reflected the projection beam is provisionally determined from the projection beam angle obtained in step S402 and the image centroid position obtained in step S394. It is checked whether the distance temporarily determined in this way is consistent with the distance based on the image size obtained in step S392 in the next step S406. If there is no contradiction, the distance temporarily determined in step S404 is officially determined as the distance to the detected vehicle, and the flow ends.

On the other hand, if there is a discrepancy in the distance in step S406, the process proceeds to step S410, where it is checked whether the image processing has been performed a predetermined number of times or more. If the predetermined number of times has not yet been reached, the process returns to step S382, and the processing from the visible light image processing is performed. Try again. In this way, unless it is confirmed in step S406 that a solar blind detection distance that is consistent with the image size distance has been obtained or it is not detected in step S410 that the image processing has been performed a predetermined number of times or more, steps S382 to S382 are performed. S410 is repeated. In addition, when a vehicle image in the same lane is not detected in step S384, or when a single vehicle cannot be separated from the overlapping image in step S390, or when there is no solar blind pixel whose output is greater than or equal to a predetermined value in step S398, Alternatively, when there is no portion in the solar blind pixel group that coincides with the visible light image centroid in step S400, the process immediately proceeds to step S410, and the image processing is repeated.

On the other hand, when it is detected in step S410 that the image processing has been repeated a predetermined number of times or more without being able to determine the distance, the process proceeds to step S412 and whether or not there is an image size distance is checked in step S392. If there is an image size distance, the process proceeds to step S414, the latest image size distance among them is adopted, and the flow ends. On the other hand, if there is no image size distance in step S412, the process proceeds to step S416, and a pan focus distance that focuses on most of the rear of the vehicle from infinity to the middle distance is adopted, and the flow ends. Accordingly, at least some distance for the camera AF in step S310 of FIG. 34 or step S348 of FIG. 35 is determined, and the flow is ended.

FIG. 37 is a block diagram showing details of the rear camera / sensor of the vehicle monitoring apparatus according to the sixth embodiment of the present invention together with the distance detection principle. Although the sixth embodiment basically has the same configuration as that of the fifth embodiment including the entire configuration block of FIG. 30, the configuration around the CIGS sensor in the front camera sensor and the rear camera / sensor is slightly different. It will be. FIG. 37 shows the rear camera / sensor 922 as a representative in the same manner as the fifth embodiment of FIG. 31, but parts common to the fifth embodiment are numbered the same as in FIG. 31. Description is omitted unless particularly necessary.

The CIGS imaging sensor 926 of the sixth embodiment has a different color filter configuration, and the red transmission filter is not a band-pass filter that transmits a narrow region but a low-pass filter that transmits longer wavelengths than red. Also, the infrared light transmission filter is not a bandpass filter that transmits infrared light in a narrow area of the solar blind, but a low-pass filter that transmits light in the solar blind area and other near infrared areas. . These low-pass filters are simpler in construction than band-pass filters. For this reason, first, a fixed high-pass filter 901 for cutting the longer wavelength side than the solar blind region is provided in the optical path to the CIGS sensor 926. Further, a movable band cut filter 903 that cuts light that is longer than the red region and shorter than the solar blind region is provided in the optical path to the CIGS sensor 926. As a result of these combinations, light in a narrow area is incident on each of the red pixel, the green pixel, the blue pixel, and the solar blind pixel, as in FIG.

In the sixth embodiment, infrared imaging is further possible. For this purpose, the movable band cut filter 903 is retracted from the optical path to the CIGS imaging sensor 926, and instead the movable low-pass filter 905 that cuts the visible light region and transmits the infrared region is the optical path to the CIGS sensor 926. Provided inside. The filter drive unit 907 replaces the movable band cut filter 903 and the movable low-pass filter 905. Based on the visible light output of the CIGS sensor, the average illuminance detection unit 909 determines that the average illuminance has decreased, that is, the situation corresponding to the evening to the night, and instructs the filter driving unit 907 to move. The band cut filter 903 is automatically switched to the movable low pass filter 905. Further, based on the visible light output of the CIGS sensor, the contrast detector 909 determines that the field of view has deteriorated due to rain or fog when the contrast decreases, and instructs the filter driver 907 to cut the movable band. The filter 903 is automatically switched to the movable low-pass filter 905.

As described above, according to the sixth embodiment, visible light photographing can be automatically switched to infrared light photographing when it is dark or when visibility is poor. The average illuminance detection unit 909 and the contrast detection unit 909 automatically switch the movable low-pass filter 905 to the movable band cut filter 903 when the average illuminance or contrast increases based on the infrared light output of the CIGS sensor. Switch from light photography to visible light photography. The AF driving unit 940 works in conjunction with the above-described filter replacement, and adds infrared correction to focus when infrared imaging is performed.

38A and 38B are graphs for explaining the spectral transmission characteristics of the filter of the CIGS image sensor 626 in the sixth embodiment. FIG. 38A is a visible light imaging state, and FIG. 38B is an infrared light imaging. Each state is shown. The spectral sensitivity 832 of the CIGS sensor, the spectral irradiance 834 of sunlight, the spectral transmission factor 840 of the green transmission filter, and the spectral transmission factor 842 of the blue transmission filter are the same as in FIG. On the other hand, the spectral transmittance 932 of the red transmission filter is a low-pass filter that transmits longer wavelengths than red. In addition, the spectral transmittance 936 of the infrared light transmission filter is a low-pass filter that transmits light in the solar blind region and other near infrared regions. As can be seen from FIG. Also, the near-infrared region on the short wavelength side is transmitted relatively widely. Furthermore, the spectral sensitivity of the fixed high-pass filter 901 is a high-pass filter that cuts the longer wavelength side than the solar blind region.

In the above configuration, in FIG. 38A, the movable band cut filter 903 cuts light on the longer wavelength side than the red region and on the shorter wavelength side than the solar blind region, as indicated by the hatched portion. ing. As a result, similar to FIG. 33, a combined spectral transmittance is achieved in which light in a narrow area is incident on each of the red pixel, the green pixel, the blue pixel, and the solar blind pixel. In the state of FIG. 38A, the reflected light of the solar blind projection beam is sensed by superimposing it on the red image even in the red pixel. Since this reflected light is detected only at a previously known beam projection timing, the projected beam reflected light can be detected from the output change at this timing and used for distance detection information. Even if the red pixel has sensitivity in the solar blind region, there is no influence of sunlight, so that there is no problem with red visible light imaging unless the projection beam is involved.

On the other hand, in FIG. 38B, the movable band cut filter 903 is removed, and the movable low-pass filter 905 cuts the visible light region as indicated by the hatched portion. As a result, light in a fairly wide band in the near infrared is incident on the infrared pixel. As a result, imaging using infrared light including that using sunlight as a light source becomes possible. In addition, since light in the solar blind region can be incident, distance measurement using a projection beam is also possible. Here, in the state of FIG. 38B, the red pixel has almost the same spectral transmittance as that of the infrared pixel. Therefore, in infrared imaging, the red pixel is regarded as the infrared pixel and used for imaging information. can do.

In the sixth embodiment, when aberration due to incidence of near-infrared light in a wide band and shift in focus position become problems, the cutoff wavelength of the spectral transmittance 936 of the infrared light transmission filter is set to about 1000 nm, for example. As a result, the cut-off wavelength of the movable low-pass filter 905 is shifted to about 1000 nm correspondingly to cut near-infrared light on the short wavelength side. As a result, the infrared imaging band can be narrowed to reduce aberrations and focus position shifts. However, since this reduces the amount of light for infrared light photography, the transmission bandwidth for infrared light photography is determined based on the balance between the two. Also, it is possible to select where the peak is depending on the purpose.

FIG. 39 is a flowchart of the operation of the monitoring record controller 632 in the sixth embodiment of FIG. Since the contents are often in common with the flowchart in the fifth embodiment of FIG. 34, the common steps are given the same step numbers and will be described in relation to FIG. First, step S302 to step S308 and step S334 in FIG. 39 are the same as the step with the same number in FIG. Further, the vehicle abnormal approach process in step S422 in FIG. 39 is a combination of steps S312 to S324 and S328 in FIG. 34, and the contents thereof are the same. Similarly, the abnormal acceleration detection process in step S424 in FIG. 39 is a combination of step S326, step S330, and step S332 in FIG. 34, and the contents thereof are the same. 34 is replaced by step S442 from step S426 in FIG. The contents of the stopping process in step S444 in FIG. 39 are obtained by replacing step S348 in FIG. 35 from step S426 in FIG. 39 to step S442.

Based on the above assumptions, steps S426 to S442 in FIG. 39 will be described. These portions relate to automatic switching between the movable low-pass filter 905 and the movable band cut filter 903 described with reference to FIG. 37, and infrared correction of the AF driving unit 940 in conjunction therewith. In FIG. 39, when the front and rear inter-vehicle distances are detected in step S308, it is checked in step S426 whether infrared imaging has been manually selected. If there is no selection, the process proceeds to step S428, and it is checked whether the illuminance detected by the average illuminance detection unit 909 is equal to or less than a predetermined value. If the illuminance is not less than the predetermined value, it is further checked in step S430 if the contrast detected by the contrast detection unit 911 is less than the predetermined value.

When it is determined in step S430 that the contrast is equal to or lower than the predetermined value, the process proceeds to step S432, and an instruction to insert a movable low-pass filter 905 that cuts the visible light band is given to the filter control unit 907. When manual infrared imaging selection is detected in step S426, or when it is detected in step S428 that the illuminance is not more than a predetermined value, the process immediately proceeds to step S432. Next, in step S434, the AF driving unit 940 is instructed to perform focusing by camera AF with infrared correction. In step S436, the image processing unit is instructed to select infrared image output, and the process proceeds to step S422.

On the other hand, if the contrast is not less than or equal to the predetermined value in step S430, the process proceeds to step S438, and the filter controller 907 is instructed to insert a movable band cut filter 903 that cuts the infrared band other than the solar blind. Next, in step S440, the AF drive unit 940 is instructed to perform focusing by camera AF without infrared correction. In step S436, the image processing unit is instructed to select visible light image output, and the process proceeds to step S422.

FIG. 40 is a color filter array of a CIGS imaging sensor used for the front and rear cameras / sensors of the vehicle monitoring apparatus according to the seventh embodiment of the present invention. This filter arrangement is basically the same as a typical Bayer arrangement in primary color filters, and includes a red transmission filter R21 (IR21), green transmission filters G11 and G22, and a blue transmission filter B12. However, in the seventh embodiment, the output of the pixel corresponding to the red transmission filter R21 (IR21) is also used as the infrared light output by switching the filter as described later. The configuration of the front and rear cameras / sensors of the vehicle monitoring apparatus of the seventh embodiment of the present invention is the same as that of the sixth embodiment shown in FIG. 37, and only the filter array of the CIGS image sensor 926 is as described above. Different.

FIG. 41 is a graph for explaining the spectral transmission characteristics of the filter of the CIGS imaging sensor 626 in the seventh embodiment. FIG. 41A shows the state of visible light imaging, as in FIG. 41 (B) shows the state of infrared light photography. CIGS sensor spectral sensitivity 832, solar spectral irradiance 834, green transmission filter spectral transmission factor 840, blue transmission filter spectral transmission factor 842, red transmission filter spectral transmission factor 932, fixed high-pass filter 901 spectral sensitivity and The spectral sensitivity of the movable band cut filter 903 is the same as that in FIG. That is, the seventh embodiment differs from the sixth embodiment only in that the infrared-only pixels are removed by the color filter array of the CIGS imaging sensor.

As a result, in the state of FIG. 41A, the reflected light of the solar blind projection beam is detected by superimposing the red image on the red pixel of the spectral transmittance 932 corresponding to R21 (IR21) of FIG. . In the seventh embodiment, since there is no dedicated image for detecting the solar blind reflected light, the projection beam is intermittently interrupted in order to separate the solar blind reflected light component from the red image output of the red pixel. Since this intermittent timing is known in advance, if there is a red pixel output that increases in synchronization with the projection timing, it is separated as solar blind reflected light. Note that even if the red pixel has sensitivity in the solar blind area, it is not affected by sunlight. Therefore, as long as the projection beam is not involved, there is no problem with red visible light imaging as in the sixth embodiment.

On the other hand, in FIG. 41B, the movable band cut filter 903 is removed, and the movable low-pass filter 905 cuts the visible light region and the near-infrared region up to about 1000 nm as shown by the hatched portion. As a result, red light having a spectral transmittance of 932 detects infrared light based on sunlight from 1000 nm to around 1100 nm and reflected light of the solar blind projection beam. Again, in order to separate the solar blind reflected light component from the red pixel infrared light output, the projection beam is intermittently interrupted. Since the intermittent timing is known in advance, as in the case of FIG. 41A, if there is a red pixel output that increases in synchronization with the projection timing, it can be separated as solar blind reflected light.

FIG. 41B shows an example in which the movable low-pass filter 905 cuts the visible light region and the near-infrared region up to about 1000 nm, but this is not unique to the seventh embodiment. In the sixth embodiment, it has already been described that the cut-off wavelength of the movable low-pass filter 905 is shifted to about 1000 nm in order to cope with the problem of aberration and focus position. FIG. 41B adopts such a configuration. This example is specifically shown. Also in the seventh embodiment, when there is no problem in terms of separation of reflected light components of the solar blind, aberration, and focus position shift, the movable low-pass filter 905 in the infrared imaging state is the same as in the sixth embodiment. The cutoff wavelength may be shifted to the short wavelength side.

FIG. 42 is a graph for explaining the spectral transmission characteristics of the color filter of the CIGS image sensor used in the front and rear cameras / sensors of the vehicle monitoring apparatus according to the eighth embodiment of the present invention. The eighth embodiment has basically the same configuration as the seventh embodiment. That is, the filter arrangement of the CIGS imaging sensor is a Bayer arrangement as shown in FIG. 40, and the output of the pixel corresponding to the red transmission filter R21 is also used as the infrared light output by switching the filter as in the seventh embodiment. The Similarly to the seventh embodiment, the configuration of the front and rear cameras / sensors of the vehicle monitoring apparatus is the same as that of the sixth embodiment shown in FIG. The eighth embodiment of FIG. 42 differs from the seventh embodiment in that the distance detection function by solar blind beam reflected light detection is omitted, and the visible light drive recorder in the normal daytime with only visible light and the infrared light in the nighttime or in fog. The function is limited to automatic switching of the optical drive recorder.

Also in the eighth embodiment, similarly to FIGS. 38 and 41, FIG. 42 (A) shows the state of visible light photographing, and FIG. 42 (B) shows the state of infrared light photographing, respectively. 42, the spectral sensitivity 832 of the CIGS sensor, the spectral irradiance 834 of sunlight, the spectral transmission factor 840 of the green transmission filter, the spectral transmission factor 842 of the blue transmission filter, the spectral transmission factor 932 of the red transmission filter, and the movable The spectral sensitivity of the band cut filter 903 is the same as that in FIG. FIG. 42 differs from FIG. 41 in that the fixed high-pass filter 901 has a spectral transmittance that cuts a longer wavelength range including the solar blind region near 1100 nm, and uses the sensitivity of the CIGS sensor in the solar blind region. That is not the point.

Thus, in the state of FIG. 42A in which the movable band cut filter 903 is inserted, only the red image is sensed by the red pixel having the spectral transmittance 932. On the other hand, in FIG. 42B, the movable band cut filter 903 is removed, and the movable low-pass filter 905 cuts the visible light region and the near-infrared region up to about 1000 nm as shown by the hatched portion. As a result, infrared light based on sunlight from 1000 nm to around 1100 nm is detected by the red pixel having the spectral transmittance of 932. In this way, the wide spectral sensitivity of the CIGS sensor is used, and the red pixels in the Bayer array are switched to red image detection in visible light photography and infrared image detection in infrared light photography. The CIGS sensor of the eighth embodiment is not limited to the drive recorder, but can be used in the cameras in the first and second embodiments, and in the endoscope in the fourth embodiment. is there.

FIG. 43 is a top external view of a ninth embodiment of the present invention, which is configured as a mobile phone 1. The cellular phone 1 is configured so that the upper part 7 having the display part 5 can be folded on the lower part 11 having the operation part 9 such as a numeric keypad by the hinge part 3. The upper portion 7 is provided with an earpiece 13 that constitutes a telephone function. In addition, when the mobile phone 1 is used as a video phone, the face of the operator who is looking at the display unit 5 can be shown and when taking a selfie. An inner camera 17 is also used. Further, the upper portion 7 includes an infrared light emitting unit 19 for detecting that the mobile phone 1 is close to the face for a call, and an infrared light proximity sensor 21 for receiving infrared reflected light from the face. When the approach of the face is detected when the output of the infrared light proximity sensor 21 is greater than or equal to a predetermined value, the backlight of the display unit 5 is turned off for power saving. Although not shown in FIG. 43, a rear camera is provided on the rear surface of the upper portion 7, and a subject that is on the rear surface side of the mobile phone 1 and is monitored by the display unit 5 can be photographed.

On the other hand, the lower part 11 is provided with a mouthpiece 23 constituting a telephone function below, and a CIGS imaging sensor 25 is arranged for information. As shown in FIG. 3A, the CIGS imaging sensor 25 has a wide sensitivity range over a wavelength of about 1300 nm and has a spectral sensitivity exceeding 50% quantum efficiency over a wide wavelength range from about 400 nm to about 1200 nm. As will be described later, it has various functions such as cheek skin imaging, cheek skin moisture measurement, thumb vein authentication, and health check by pulse wave. Around that, a light source unit 27 having LEDs for various purposes described above is arranged. Since the CIGS imaging sensor 25 and the light source unit 27 are provided above the lower portion 11, that is, near the center of the cellular phone 1 as a whole, the thumb of the hand holding the cellular phone 1 is used for finger vein authentication. Authentication is possible with one-handed operation.

The cheek skin is photographed and the cheek skin moisture is measured while the mobile phone 1 is in close proximity to the face for a call. When the call ends, the skin moisture display 29 is automatically set on the display unit 5. Done for hours. On the other hand, vein authentication or pulse wave detection using the thumb is possible by holding the thumb of the hand holding the mobile phone 1 in the vicinity of the CIGS image sensor 25 in a non-contact manner. At this time, based on the focus detection function based on the output of the CIGS imaging sensor 25, an instruction 31 for bringing the thumb close or an instruction 32 for releasing the thumb is displayed on the display unit 5. At the same time as thumb authentication, the pulse wave of the thumb is detected, and health check information 35 such as a blood vessel health check result based on the detected pulse wave shape and blood oxygen saturation of the pulse oximeter based on the pulse wave is provided. It is displayed on the display unit 5. In FIG. 43, for convenience of explanation, the instruction 31 currently displayed on the display unit 5 is shown by being surrounded by practice, and other information that can be appropriately displayed is surrounded by a broken line. Further, the display layout is not limited to that shown in FIG. 43, and each display may be displayed in full on the display unit 5 as long as it is not necessary to display them simultaneously.

FIG. 44 is a block diagram of the ninth embodiment. The same parts as those in FIG. 43 are denoted by the same reference numerals, and description thereof is omitted unless necessary. The mobile phone 1 is controlled by a control unit 39 that operates according to a program stored in the storage unit 37. The storage unit 37 temporarily stores data necessary for the control of the control unit 39 and can also store various measurement data and images. Display on the display unit 5 is performed based on display data held by the display driver 41 under the control of the control unit 39. The display unit 5 has a display backlight 43, and the brightness of the backlight is adjusted by the control 39 based on the ambient brightness detected by the CIGS image sensor 25.

The telephone function unit 45 including the earpiece 13 and the mouthpiece 23 can be connected to a wireless telephone line by a telephone communication unit 47 under the control of the control unit 39. In addition, the mobile phone 1 can wirelessly perform data communication with a nearby device based on a short-range communication system different from the telephone line by the short-range communication unit 49. The speaker 51 performs ringtones and various guidance under the control of the control unit 39 and outputs the other party's voice during a videophone call. In addition, the image processing unit 53 processes images captured by the inner camera 17 and the rear camera 55 under the control of the control unit 39, processes images of the CIGS imaging sensor 25, and stores images of these processing results. Input to the unit 37.

FIG. 45 is a graph for explaining the spectral transmission characteristics of the filter of the CIGS image sensor 25 and the peak wavelength of the image sensor light source 27 in the ninth embodiment. The spectral sensitivity (quantum efficiency) 101 of the CIGS imaging sensor 25 conceptually shows the same one as that in FIG. 3A, and has a wide sensitivity range over a wavelength of about 1300 nm and a wide wavelength from about 400 nm to about 1200 nm. It shows high quantum efficiency across the region. FIG. 45A shows the water absorbance characteristic 103 with the horizontal axis indicating the wavelength in common with the spectral sensitivity 101 of the CIGS imaging sensor 25 as described above. Note that the vertical axis merely indicates the wavelength-dependent relative changes of the spectral sensitivity 101 and the water absorbance property 103, and the absolute value between the two is meaningless. As is clear from FIG. 45A, the absorbance property 103 of water has peaks at 970 nm, 1200 nm, 1450 nm, and the like.

On the other hand, FIG. 45B shows the absorbance characteristic 105 of oxyhemoglobin and the absorbance characteristic 107 of reduced hemoglobin with the horizontal axis indicating the wavelength superimposed on the spectral sensitivity 101 of the CIGS imaging sensor 25. In FIG. 45B as well, the vertical axis only shows the wavelength-dependent relative changes of the spectral sensitivity 101, the absorbance characteristic 105 of oxyhemoglobin, and the absorbance characteristic 107 of reduced hemoglobin. The value has no meaning. As is apparent from FIG. 45B, the absorbance characteristic 105 of oxyhemoglobin gradually increases from around 660 nm to around 900 nm. On the other hand, the absorbance characteristic 10 of reduced hemoglobin exhibits a large absorbance near 660 nm, and the absorbance gradually decreases as a whole near 900 nm. During this time, there is once a peak of absorbance near 760 nm, and crosses the absorbance characteristic 105 of oxyhemoglobin near 805 nm, so that both absorbances are equal.

FIG. 45 shows a plurality of types of color filters of the CIGS imaging sensor 25 set on the premise of the spectral sensitivity 101 of the CIGS imaging sensor 25, the water absorbance characteristic 103 of water, the absorbance characteristic 105 of oxyhemoglobin, and the absorbance characteristic 107 of reduced hemoglobin. 2 shows the spectral transmittance of the band-pass filter employed respectively, and the peak wavelengths of a plurality of types of LEDs used as the light source unit 27. First, what is related to the measurement of skin moisture will be described with reference to FIG. The bandpass filter 109 is set to have a peak wavelength of 970 nm in accordance with the peak near 970 nm shown in the water absorbance characteristic 103. As a light source that covers this wavelength region, an LED 111 having a peak wavelength of 940 nm (only the peak wavelength is indicated by a one-dot chain line for simplification of the drawing, the same applies hereinafter) is provided. On the other hand, the bandpass filter 113 is set with a peak wavelength of 805 nm in a low absorbance region as indicated by the water absorbance characteristic 103. An LED having a peak wavelength of 850 nm is provided as a light source covering this wavelength region. As described above, as long as a desired output can be obtained, it is not necessary to make the peak wavelengths of the LED and the bandpass filter exactly coincide with each other in consideration of cost. The use of the LED 115 having a peak wavelength slightly deviated from the peak of the bandpass filter 113 is an example. Since CIGS image sensors have high quantum efficiency and good sensitivity, there is a large tolerance for this.

Next, the authentication of the thumb vein will be described with reference to FIG. In addition, the peak wavelength of each band pass filter and each LED is common in FIG. 45 (A) and FIG. 45 (B). The bandpass filter 117 is set with a peak wavelength of 760 nm in accordance with the absorbance peak near 760 nm in the absorbance characteristic 107 of reduced hemoglobin. In the vicinity of 760 nm, as is apparent from the absorbance characteristic 105, the absorbance of oxidized hemoglobin is sufficiently small. An LED 119 having a peak wavelength of 750 nm is provided as a light source covering this wavelength region. The thumb vein image for thumb vein authentication is picked up based on the pixel output of a CIGS imaging sensor in which the LED 119 having a peak wavelength of 750 nm is used as a light source and a band pass filter 117 having a peak wavelength of 760 nm is disposed. As a result, a vein image rich in reduced hemoglobin can be extracted. At this time, in order to acquire a reference image, the pixel output of the CIGS imaging sensor in which the bandpass filter 113 having a peak wavelength of 805 nm is disposed using the LED 115 having the peak wavelength of 850 nm described in the above-described skin moisture detection as a light source is used. An image with a peak wavelength of 805 nm is suitable as a reference image because the absorbances of oxyhemoglobin and reduced hemoglobin are equal.

Furthermore, detection of a pulse wave and measurement of blood oxygen saturation based on this will be described with reference to FIG. The band-pass filter 201 is set with a peak wavelength of 660 nm having a high absorbance in the absorbance characteristic 107 of reduced hemoglobin. In the vicinity of 660 nm, as is apparent from the absorbance characteristic 105, the absorbance of oxidized hemoglobin is the smallest, and the difference in absorbance between oxidized hemoglobin and reduced hemoglobin is large. An LED 203 having a peak wavelength of 660 nm is provided as a light source covering this wavelength region. The pulse wave is detected by detecting the pixel output of the CIGS imaging sensor in which the bandpass filter 201 having the peak wavelength of 660 nm is disposed using the LED 203 having the peak wavelength of 660 nm and the peak wavelength of 805 nm using the LED 115 having the peak wavelength of 850 nm as the light source. Is detected based on the pixel output of the CIGS imaging sensor in which the bandpass filter 113 is arranged. That is, since the increase or decrease in arterial blood in the thumb is known based on both outputs, the pulse wave is detected based on this. Furthermore, based on the change in the thickness of arterial blood corresponding to the pulse wave, the oxygen saturation of arterial blood can be obtained by a known theory.

The use of the pixel output of the CIGS imaging sensor 25 in skin moisture measurement, thumb vein authentication, and pulse wave detection and blood oxygen saturation measurement is not limited to the two-wavelength pixel output as described above. It is also possible to combine three or more wavelengths as appropriate. For example, in the measurement of skin moisture, the pixel output of a CIGS image sensor provided with a bandpass filter 201 having a peak wavelength of 660 nm or the pixel output of a CIGS image sensor provided with a bandpass filter 117 having a peak wavelength of 760 nm is used. can do. Similarly, in the authentication of the thumb vein, the pixel output of the CIGS image sensor provided with the bandpass filter 201 having the peak wavelength of 660 nm and / or the pixel output of the CIGS image sensor provided with the bandpass filter 109 having the peak wavelength of 970 nm are used. Can be used. Furthermore, in pulse wave detection or blood oxygen saturation measurement, the pixel output of a CIGS image sensor having a bandpass filter 117 having a peak wavelength of 760 nm or the CIGS image sensor having a bandpass filter 109 having a peak wavelength of 970 nm is arranged. Pixel output or both can be used. On the other hand, depending on the design conditions, the bandpass filter and LED types can be selected by combining the wavelength range used for skin moisture measurement, thumb vein authentication, and pulse wave detection and blood oxygen saturation measurement. It can also be reduced.

FIG. 46 is an example of the color filter array of the CIGS imaging sensor 25 in the ninth embodiment in FIGS. 43 to 45. In the example of FIG. 46, an infrared light transmission filter IRref11 having a peak wavelength of 805 nm, a red transmission filter R12 having a peak wavelength of 660 nm, an infrared transmission filter IR21 having a peak wavelength of 970 nm, and an infrared transmission filter bIR22 having a peak wavelength of 760 nm are illustrated. It is the arrangement which repeats this as one unit. These correspond to the bandpass filter 113, the bandpass filter 201, the bandpass filter 109, and the bandpass filter 117 in FIG. In addition, for interpolating image information other than the wavelength region of the color filter in each pixel in FIG. 46, the configuration described in the other embodiments in FIG. In addition, when taking a skin image together with skin moisture measurement, a multi-wavelength image with red and multiple-wavelength infrared images can be obtained. The state can be visually confirmed by an image.

FIG. 47 is a front view showing an example of the arrangement of LEDs that can be employed in the ninth embodiment, and shows the detailed configuration of the CIGS image sensor 25 shown in FIG. 45 and the image sensor light source 27 around it. In FIG. 47, the same reference numerals are given to the portions corresponding to those in FIG. As is apparent from FIG. 47, a pair of LED 111 having a peak wavelength of 940 nm, an LED 115 having a peak wavelength of 850 nm, and an LED 119 having a peak wavelength of 750 nm are disposed around the lens 205 of the CIGS image sensor at 60 degrees across the optical axis of the lens 205. It is provided on the rotation object shifted by one. On the other hand, six LEDs 203 having a peak wavelength of 660 nm are arranged on the rotation target in a gap between the LEDs 111, 115, and 119, shifted by 60 degrees around the optical axis of the lens 205. Since the LED 203 having a peak wavelength of 660 nm can be configured to be relatively small, such mounting is possible. Further, by increasing the number, it is possible to use an LED having a small light emission intensity, thereby reducing the cost. In the case of a design in which the objectivity around the optical axis is not a problem, the number of LEDs can be reduced to one for each wavelength.

FIG. 48 is a flowchart of the operation of the control unit 39 in the ninth embodiment of FIG. The flow starts when the main power supply is turned on by the operation unit 9 of the mobile phone 1, and at step S452, initial startup and function check of each unit are performed and screen display on the display unit 5 is started. Next, in step S454, the imaging sensor light source unit 27 is turned off. Note that nothing is done here when the light is originally off. Further, in step S456, it is checked whether or not the mobile phone 1 is in a no-operation state such as being left for a predetermined time. In this case, it is assumed that the mobile phone 1 is not in a no-operation state even when the mobile phone 1 functions by itself, such as detecting an incoming call. If no operation for a predetermined time is not detected in step S456, the process proceeds to step S458, and the display backlight 43 is turned on. At the same time, the CIGS imaging sensor 25 is activated in step S460. Thereby, when passing through step S460 from step S454, a CIGS image sensor will be in the state which receives light without a light source.

Next, in step S462, the brightness of the display backlight is adjusted based on the output of the CIGS imaging sensor 25. Specifically, based on the output of the LED 203 mainly in the red region of the CIGS imaging sensor, the backlight is brightened when this is large, and the backlight is darkened when the output of the LED 203 is small. Then, the flow moves to step S464. On the other hand, when no operation is detected for a predetermined time in step S456, the process proceeds to step S466, the display backlight is turned off, and the CIGS imaging sensor 25 is deactivated in step S468, and the process proceeds to step S464. In this case, since the display backlight is turned off, the brightness is not adjusted.

In step S464, it is checked whether the mobile phone 1 has received an incoming call. If not, it is checked in step S450 whether or not there has been an operation for transmission such as a telephone number operation. When a call operation is detected, the process proceeds to step S452. Note that the process also proceeds to step S452 when an incoming call is detected in step S464. Step S452 restores the backlight when it is turned off and activates the image sensor for adjusting the brightness (the same processing as Step S458 to Step S462), and incoming or This is a step of performing processing for starting a call based on outgoing calls.

When the backlight restoration / call start processing in step S452 is completed, the process proceeds to step S454, and it is checked whether the cheek is close to the mobile phone 1 based on the output of the infrared light proximity sensor 21. This state occurs when the mobile phone 1 is brought close to the ear and mouth to enter a talking posture. If cheek proximity is not detected, the process proceeds to step S456. This corresponds to the case where the mobile phone 1 is not yet brought close to the ear and mouth even when the call is started, or a call is made by a videophone. On the other hand, when a call operation is not detected in step S450, the process directly proceeds to step S456.

In step S456, it is checked whether or not a manual operation for measuring skin moisture has been performed. If this operation is detected, the flow proceeds to skin moisture measurement / skin imaging processing in step S458. On the other hand, when cheek proximity is detected in step S454, the process proceeds to step S460, the display backlight is turned off, and the process proceeds to step S458. Thus, when the proximity of the cheek is detected, the skin moisture measurement process is automatically performed without any operation. Details of the skin moisture measurement / skin imaging process in step S458 will be described later. Note that turning off the display backlight in step S460 means power saving when the display unit 5 cannot be seen, and omits adjustment of the measurement posture in the case of automatic measurement of skin moisture by approaching the cheek as described later. Therefore, it is meaningful to prevent the illumination state of the skin by the backlight from affecting the measurement due to variations in the measurement posture. On the other hand, in the case of manual measurement, as described later, the skin is also photographed and the measurement conditions such as focusing are adjusted. Therefore, the display backlight is actively used for skin photographing (at least of the CIGS imaging sensor). Used as auxiliary illumination light (for visible light pixels).

When the skin moisture measurement process in step S458 is completed, it is checked in step S462 if a call is in progress. Moreover, also when the skin moisture measurement manual operation is not detected by step S456, it transfers to step S462. When it is detected in step S462 that a call is in progress, the process returns to step S454, and steps S454 to S462 are repeated as long as it is detected in step S462 that a call is in progress. This responds to various situation changes during a call (for example, manually performs skin moisture measurement during a videophone call), and increases the amount of measurement information by repeating the skin moisture measurement process in step S458.

If it is not detected in step 462 that the call is in progress, the process proceeds to step S464, and the backlight recovery / skin moisture display process is started. Details thereof will be described later. As described above, when there is information by the skin moisture measurement / skin photographing process 458 in step S458, the result is automatically displayed on the display unit 5 when the call is finished. When the backlight restoration / skin moisture display process in step S464 is completed, it is checked in step S466 whether an operation requiring authentication (for example, reading of mail or changing of phone book information) has been performed, and these operations should not be detected. If so, the process proceeds to step S468. On the other hand, when an authentication-required operation is detected in step S466, the process proceeds to step S468 through step S470. Step S470 is a process of performing a pulse wave simultaneously with the thumb vein authentication process and performing a health check based on the pulse wave and blood oxygen saturation, and the health check result is automatically displayed on the display unit 5 upon successful authentication. Is displayed. In step S468, it is checked whether or not the main power supply is turned off by the operation unit 9. If main power off is not detected, the process returns to step S454. Thereafter, unless the main power supply is turned off, steps S454 to S470 are repeated. Corresponds to various situations. On the other hand, if it is detected in step S468 that the main power is off, the flow is immediately terminated.

FIG. 49 is a flowchart showing details of the skin moisture measurement / skin imaging process in step 458 of FIG. When the flow starts, first, in step S472, it is checked whether or not the skin moisture measurement / skin photographing process has been entered manually. If it is via manual operation, the process proceeds to step S474, and all of the LED 111 with a peak wavelength of 940 nm, the LED 115 with a peak wavelength of 850 nm, the LED 119 with a peak wavelength of 750 nm, and the LED 203 with a peak wavelength of 660 nm are turned on, and the process proceeds to step S476. . On the other hand, if it is not via manual operation in step S472, it means via cheek proximity detection, so that the process proceeds to step S478, the LED 111 having a peak wavelength of 940 nm and the LED 115 having a peak wavelength of 850 nm are turned on, and step S476 is performed. Proceed to In this case, the other LEDs cannot be turned on.

In step S476, the CIGS imaging sensor image is read and the process proceeds to step S480, where it is checked again whether the manual operation has been performed. If it is via manual operation, the process proceeds to step S482, where it is checked whether the skin image has been successfully captured and the storage has already been completed. If the skin image storage is not completed at this time, the process proceeds to step S484, where the bandpass filter 109 having a peak wavelength of 970 nm, the bandpass filter 113 having a peak wavelength of 805 nm, the bandpass filter 117 having a peak wavelength of 760 nm, and the band having a peak wavelength of 660 nm are used. An image of each pixel of the CIGS imaging sensor 25 to which the pass filter 201 is applied is extracted. In step S486, the contrast of the image by each pixel is detected and compared with each other, and the process proceeds to step S488.

In step S488, whether the contrast of the image (hereinafter abbreviated as “970 nm image”, the same applies to other images) of the pixel of the CIGS imaging sensor 25 to which the bandpass filter 109 having the peak wavelength of 970 nm is applied is larger than the contrast of the 760 nm image. Is checked. If not, the process proceeds to step S490, where it is checked whether the contrast of the 660 nm image is greater than the contrast of the 805 nm image. If this is not the case, there is a high possibility that the focus of the image by the light between the wavelength of 760 nm and the wavelength of 660 nm matches the imaged image CIGS imaging sensor, so the process proceeds to step S492, and each of the images extracted in step S484 is displayed. Interpolation processing of each pixel is performed based on the image information of the wavelength pixel.

On the other hand, when it is detected in step S488 that the contrast of the 970 nm image is larger than the contrast of the 760 nm image, the process proceeds to step S494, and “the skin is being checked. Please give an instruction to make an announcement such as “Please,” and the process returns to step S476. If it is detected in step S490 that the contrast of the 660 nm image is greater than the contrast of the 805 nm image, the process proceeds to step S496, and the earpiece 13 says “Skin is being checked. Is instructed to make an announcement, etc., and the process returns to step S476. Thereafter, step S476 to step S490, step S494, and step S496 are repeated until the state does not correspond to either step S488 or step S490, and the change of the distance between the mobile phone and the skin and the reading of the CIGS imaging sensor image are repeated. It is. Note that steps S488 and S490 have been described as comparing the contrast information itself for the sake of simplicity. In practice, however, the magnitude comparison is performed after adding a predetermined bias experimentally obtained by actual measurement of the focus state. .

When each pixel image interpolation process in step S492 is completed, the process proceeds to step S498, where a visualization image of the cheek skin is created and stored. Four images picked up by four wavelengths are obtained by the interpolation processing in step S492. Since these are images of red and three infrared wavelengths, in step S498, these four images are appropriately replaced with four visible light images. , Convert to a “color image” that can be observed with the eyes. The “color” image visualized in this way is not an actual skin color, but it is possible to visually observe the health state of the skin by imaging with four wavelengths. Since the skin imaging is a macro photography in which the mobile phone 1 is in close proximity to the cheek, an enlarged image in which the details of the skin can be observed is obtained.

Next, in step S500, all the outputs of the 970 nm pixels and all the outputs of the 805 nm pixels are added, respectively, to obtain the sum of the received light outputs of 970 nm and 805 nm, respectively. If it is not detected in step S480 that a manual operation is being performed, the process proceeds to step S500 immediately without acquiring skin image information and adjusting the focus because it is an automatic measurement based on cheek proximity detection. Further, if it is detected in step S482 that the creation and storage of the skin image in step S498 has already been completed, the process immediately proceeds to step S500. Next, in step S502, skin moisture is calculated based on the received light output of 970 nm and 805 nm. In step S502, if there is information on the skin moisture calculated so far through the repetition of steps S454 to S462 in FIG. 48, the skin moisture calculated this time is accumulated and statistically processed to calculate an average value. To do.

Next, in step S504, it is checked whether the skin moisture determined in step S502 is 20% or more. If not, the process proceeds to step S506, the skin moisture obtained in step S502 is stored, and the flow ends. On the other hand, when it is detected in step S504 that the skin moisture is 20% or more, the process proceeds to step S508, an error message is stored, and the flow ends. This is because it is generally known that the skin moisture is about 10 to 15%. Therefore, when the skin moisture is 20% or more, it is considered an error due to the influence of sweat.

FIG. 50 is a flowchart showing details of the backlight restoration / skin moisture display process in step S464 of FIG. When the flow starts, it is first checked in step S512 whether the cheek proximity detection result has changed from detection to non-detection. This corresponds to checking whether or not there has been an action of moving the mobile phone 1 away from the cheek. If this change is not detected, the flow is immediately terminated. In this case, nothing is performed in step S464 in FIG.

On the other hand, when the corresponding change in cheek construction detection is confirmed in step S512 of FIG. 50, the process proceeds to step S514, and whether there is a memory of skin moisture or an error message based on the function of step S506 or step S508 of FIG. Will be checked. If there is any memory, the display backlight is turned on in step S516 and the image sensor 25 in step S518 is activated. In step S520, the brightness of the display backlight is adjusted based on the output of the CIGS imaging sensor 25. As a result, the lighting of the backlight and its adjustment state are restored.

Next, in step S522, the storage of the skin moisture or the storage of the error message is read and the storage read in step S524 is displayed. In step S526, it is checked whether there is an operation by the operation unit. If no operation is detected, the process proceeds to step S528, and it is checked whether a predetermined time has elapsed since the skin moisture or error message was started by the function of step S524. If the predetermined time has not elapsed, the process returns to step S524. Thereafter, unless the operation unit operation is detected in step S526 or the elapse of the predetermined time in step S528 is not detected, steps S524 to S528 are repeated, and the display of skin moisture or an error message is continued. On the other hand, when the elapse of the predetermined time is detected in step S528, the process proceeds to step S530, the display of the skin moisture or the error message is terminated, and the process proceeds to step S532. If operation of the operation unit is detected in step S526, the process immediately proceeds to step S530 and the display is terminated. On the other hand, if the storage of skin moisture or error message is not detected in step S514, the process proceeds directly to step S532. In this manner, when there is a memory of skin moisture or an error message, these are automatically displayed on the display unit 5 for a predetermined time by speaking the mobile phone 1 from the skin.

In step S532, it is checked whether a visualized image of the cheek skin based on the function of step S498 in FIG. 49 is stored. If it is stored, the process proceeds to step S534, and it is checked whether or not an operation for performing the reproduction is performed within a predetermined time. If an operation is detected within a predetermined time, the process proceeds to step S536, where the visualized image memory of the cheek skin is read and the memory read in step S538 is displayed. In step S540, it is checked whether there has been an operation by the operation unit for ending the display. If no operation is detected, the process returns to step S538, and the display is continued by repeating steps S538 and S540 unless a display end operation is detected in step S540. On the other hand, when a display end operation is detected in step S540, the flow ends. Note that if it is not detected in step S532 that a visualized image of the cheek skin is stored, or if a reproduction operation of the visualized image of the cheek is not detected within a predetermined time in step S534, the flow is immediately terminated. .

FIG. 51 is a graph for explaining the spectral transmission characteristics of the filter of the CIGS image sensor and the peak wavelength of the image sensor light source section in the tenth embodiment of the present invention implemented in a mobile phone. Since the tenth embodiment basically has the same configuration as the ninth embodiment, it can be understood by using the external view of FIG. 43, the block diagram of FIG. 44, and the flows of FIGS. The tenth embodiment differs from the ninth embodiment in the details of the filter configuration of the CIGS image sensor 25 and the details of the configuration of the image sensor light source unit 27. FIG. 51 can be understood according to FIG. 45, and common portions are denoted by common numbers. Specifically, the spectral sensitivity 101 of the CIGS image sensor 25 in FIG. 51, the water absorbance characteristic 103, the oxygenated hemoglobin absorbance characteristic 105, and the reduced hemoglobin absorbance characteristic 107 are the same as those in FIG.

51, the bandpass filter 109 having a peak wavelength of 970 nm, the LED 111 having a peak wavelength of 940 nm used as the light source, the bandpass filter 113 having a peak wavelength of 805 nm, and the plateau thereof are used. The LED 115 having a peak wavelength of 850 nm, the band-pass filter 201 having a peak wavelength of 660 nm, and the LED 203 having a peak wavelength of 660 nm used as the light source thereof are also common to the ninth embodiment of FIG. However, the bandpass filter 117 having a peak wavelength of 760 nm and the LED 119 having a peak wavelength of 750 nm used as its light source are omitted in the tenth embodiment of FIG.

In the tenth embodiment, the wavelength band used for vein authentication and pulse wave detection is combined in the above configuration. In other words, the thumb vein image for thumb vein authentication is captured based on the pixel output of the CIGS imaging sensor in which the LED 203 having the peak wavelength of 660 nm is used as the light source and the bandpass filter 201 having the peak wavelength of 660 nm is arranged. It is possible to extract a vein image rich in reduced hemoglobin. The reference image uses a pixel output of a CIGS imaging sensor in which an LED 115 having a peak wavelength of 850 nm is used as a light source and a bandpass filter 113 having a peak wavelength of 805 nm is arranged.

Detection of pulse waves and measurement of blood oxygen saturation based on this in the tenth embodiment are performed in the same manner as in the ninth embodiment. That is, the pulse wave is detected by the pixel output of the CIGS imaging sensor in which the bandpass filter 201 having the peak wavelength of 660 nm is disposed with the LED 203 having the peak wavelength of 660 nm as the light source, and the peak wavelength having the LED 115 having the peak wavelength of 850 nm as the light source. Detection is based on the pixel output of a CIGS imaging sensor provided with a bandpass filter 113 of 805 nm.

FIG. 52 is an example of the color filter array of the CIGS imaging sensor 25 in the tenth embodiment. In the example of FIG. 52, red transmission filters R11 and R22 having a peak wavelength of 660 nm, an infrared transmission filter IRref12 having a peak wavelength of 805 nm, and an infrared transmission filter IR21 having a peak wavelength of 970 nm are arranged as shown in FIG. It is an array that repeats as These correspond to the bandpass filter 201, the bandpass filter 113, and the bandpass filter 109 in FIG. 51, respectively. In FIG. 52, the number of pixels to which a color filter having a peak wavelength of 660 nm is applied is twice that of other pixels. This corresponds to adopting a LED having a low emission intensity as an LED having a peak wavelength of 660 nm as described later. This is not limited to the peak wavelength of 660 nm, but generally for the purpose of cost reduction, increasing the number of pixels in the specific wavelength region in the CIGS imaging sensor to increase the sensitivity, thereby reducing the light emission intensity of the corresponding wavelength region as the light source. It is possible to use.

FIG. 53 is a front view showing an example of the LED arrangement that can be adopted in the tenth embodiment. Like the embodiment 9 in FIG. 47, the CIGS image sensor 25 shown in FIG. 45 and the surrounding image sensors are shown. The detailed structure of the light source part 27 is shown. In FIG. 53, the same reference numerals are given to the portions corresponding to FIG. As is apparent from FIG. 53, a pair of LED 111 having a peak wavelength of 940 nm, an LED 115 having a peak wavelength of 850 nm, and an LED 203 having a peak wavelength of 660 nm are disposed around the lens 205 of the CIGS image sensor at 60 degrees across the optical axis of the lens 205. It is provided on the rotation object shifted by one. Since the LED 203 having a peak wavelength of 660 nm can be made relatively small, the arrangement space shown in FIG. 53 can reduce the vertical mounting space. Further, as shown in FIG. 52, since the number of pixels of the CIGS imaging sensor that receives the wavelength region of the peak wavelength 660 is doubled and the sensitivity is high, the LED having a low emission intensity is used as the LED of the peak wavelength 660 to reduce the cost. be able to.

The “color” imaging of the skin in the tenth embodiment is performed based on the 660 nm image, the 805 nm image, and the 970 nm image. At this time, regarding focus detection, step S488 in FIG. 49 is read as “970 nm image> 805 nm image?”. Thus, when the contrast of the 805 nm image is the highest, the process proceeds to step S492.

FIG. 54 is a graph for explaining the spectral transmission characteristics of the filter of the CIGS imaging sensor and the peak wavelength of the imaging sensor light source section in the eleventh embodiment of the present invention implemented in the mobile phone. Since the eleventh embodiment basically has the same configuration as the ninth embodiment, it can be understood by using the external view of FIG. 43, the block diagram of FIG. 44, and the flows of FIGS. The eleventh embodiment is different from the ninth embodiment in the details of the filter configuration of the CIGS image sensor 25 and the details of the configuration of the image sensor light source unit 27. Also in FIG. 54, the spectral sensitivity 101 of the CIGS imaging sensor 25, the water absorbance characteristic 103, the oxygenated hemoglobin absorbance characteristic 105, and the reduced hemoglobin absorbance characteristic 107 are the same as those in FIG.

54, the bandpass filter 109 having a peak wavelength of 970 nm, the LED 111 having a peak wavelength of 940 nm used as its light source, the bandpass filter 201 having a peak wavelength of 660 nm, and the peak wavelength used as its light source. The LED 203 having a wavelength of 660 nm is common to the ninth embodiment of FIG. The band-pass filter 113 having a peak wavelength of 805 nm used in the ninth embodiment, the LED 115 having a peak wavelength of 850 nm used as its plateau, the band-pass filter 117 having a peak wavelength of 760 nm, and the light source thereof. The LED 119 having a peak wavelength of 750 nm is omitted in the eleventh embodiment shown in FIG. Instead, a bandpass filter 207 having a peak wavelength of 900 nm is added. As the light source, the LED 111 having a peak wavelength of 940 nm is also used. Further, a band pass filter 209 having a peak wavelength of 540 nm in the green region of visible light and an LED 211 having a peak wavelength of 550 nm used as the light source are added.

In the eleventh embodiment, in the above-described configuration, skin moisture is measured by the CIGS imaging sensor provided with the bandpass filter 201 having the peak wavelength of 970 nm and the CIGS imaging having the bandpass filter 207 having the peak wavelength of 900 nm. The pixel output of the sensor is used. The LED 111 having a peak wavelength of 940 nm is also used as both light sources. On the other hand, in the eleventh embodiment, vein authentication, pulse wave detection, and blood oxygen saturation measurement are performed using a CIGS imaging sensor in which the LED 203 having a peak wavelength of 660 nm is used as a light source and the bandpass filter 201 having a peak wavelength of 660 nm is disposed. A pixel output and a pixel output of a CIGS imaging sensor in which a bandpass filter 207 having a peak wavelength of 900 nm is disposed using an LED 115 having a peak wavelength of 940 nm as a light source are used.

Note that skin imaging in the eleventh embodiment is basically performed as a simple color image with two wavelengths of visible light based on a 540 nm image in the green region and a 660 nm image in the red region. At this time, regarding focus detection, step S488 in FIG. 49 is read as “970 nm image> 660 nm image?”, And step S490 is read as “900 nm image> 540 nm image?”. Further, if neither of step S488 and step S490 is applicable, the state where the focus of the image by the light between the wavelength of 660 nm and the wavelength of 900 nm is in alignment with the formed image CIGS image sensor is detected. Therefore, a magnitude comparison is performed after adding a bias that is expected to shift the focus position from about 700 nm, which is the middle of the wavelength of 660 nm and 900 nm, to about 600 nm, which is the middle of red and green, to Step S488 and Step S490. .

As described above, it is possible to detect a state where the focus of the image by the light whose focus position is between the visible wavelength 540 nm and the wavelength 660 nm is in alignment with the formed image CIGS imaging sensor, and to proceed to step S492. In the case of the eleventh embodiment, since the image to be picked up itself is a visible light image and the cheek visualized image creation process is unnecessary, step S498 in FIG. 49 is basically replaced with a simple “image storage process”. Can be understood. It should be noted that the infrared 900 nm image and the 970 image can be supplementarily added to the skin imaging in the eleventh embodiment. In this case, “cheek skin visualized image creation processing” is required in step S498 of FIG.

FIG. 55 is an example of the color filter array of the CIGS imaging sensor 25 in the eleventh embodiment. In the example of FIG. 55, an infrared transmission filter IRref11 having a peak wavelength of 900 nm, a green transmission filter G12 having a peak wavelength of 540 nm, an infrared transmission filter IR21 having a peak wavelength of 970 nm, and a red transmission filter R22 having a peak wavelength of 660 nm are arranged as illustrated. It is an array that repeats this as one unit. These correspond to the bandpass filter 207, the bandpass filter 209, the bandpass filter 109, and the bandpass filter 201 in FIG.

FIG. 56 is a front view showing an example of the LED arrangement that can be adopted in the eleventh embodiment. As in the case of the ninth embodiment in FIG. 47, the CIGS image sensor 25 shown in FIG. The detailed structure of the light source part 27 is shown. In FIG. 56, the same reference numerals are given to the portions corresponding to FIG. As is clear from FIG. 56, a pair of LEDs 111 having a peak wavelength of 940 nm are provided around the lens 205 of the CIGS imaging sensor with the optical axis of the lens 205 interposed therebetween. In addition, between the LEDs 111, six LEDs 203 having a peak wavelength of 660 nm and six LEDs 211 having a peak wavelength of 550 nm are alternately provided with the optical axis of the lens 205 interposed therebetween. As described above, the LED 203 having the peak wavelength of 660 nm and the LED 209 having the peak wavelength of 550 nm are divided into those having a relatively small area and small individual outputs. Such an arrangement can reduce the vertical mounting space as in the tenth embodiment of FIG.

In the eleventh embodiment, as shown in FIG. 54, the bandpass filter 109 having a peak wavelength of 970 nm and the bandpass filter 207 having a peak wavelength of 900 nm share the LED 111 having a peak wavelength of 940 nm. This is because the CIGS imaging sensor has high quantum efficiency and high sensitivity. However, as a modified example of the eleventh embodiment, a dedicated peak wavelength 900 nm LED and a peak wavelength 950 nm LED are used for the band pass filter 109 having a peak wavelength of 970 nm and the band pass filter 207 having a peak wavelength of 900 nm, respectively. It can also be configured to be used. In this case, although the color filter arrangement of FIG. 55 can be employed as it is, the number of types of LEDs increases to four, and thus, for example, a mounting arrangement as in the ninth embodiment in FIG. 47 can be employed.

FIG. 57 is a graph for explaining the spectral transmission characteristics of the filter of the CIGS image sensor and the peak wavelength of the image sensor light source section in the twelfth embodiment of the present invention implemented in a mobile phone. Since the twelfth embodiment basically has the same configuration as the ninth embodiment, it can be understood by using the external view of FIG. 43, the block diagram of FIG. 44, and the flows of FIGS. The twelfth embodiment is different from the ninth embodiment in the details of the filter configuration of the CIGS image sensor 25 and the details of the configuration of the image sensor light source unit 27. Also in FIG. 57, the spectral sensitivity 101 of the CIGS imaging sensor 25, the water absorbance characteristic 103, the oxygenated hemoglobin absorbance characteristic 105, and the reduced hemoglobin absorbance characteristic 107 are the same as those in FIG. In addition, in FIG. 57 (A), the spectral irradiance 834 of sunlight on the ground surface similar to FIGS. 41 and 42 is added.

In the twelfth embodiment of FIG. 57, a bandpass filter 201 having a peak wavelength of 660 nm and an LED 203 having a peak wavelength of 660 nm used as the light source are provided, and a bandpass filter 113 having a peak wavelength of 805 nm and peaks at 900 nm. A band-pass filter 207 having a wavelength and an LED 115 having a peak wavelength of 850 nm that is also used for these light sources are provided. The twelfth embodiment of FIG. 57 further includes a bandpass filter 213 having a peak wavelength of 1200 nm matched to a peak of water absorbance of 1200 nm, an LED 215 having a peak wavelength of 1200 nm as its light source, and a solar blind region in the vicinity of a wavelength of 1100 nm. A band-pass filter 217 having a peak wavelength of 1120 nm matched to a wavelength at which the absorbance of water is low, and an LED 219 having a peak wavelength of 1070 nm as the light source are provided.

In the twelfth embodiment, based on the above configuration, the pixel output of the CIGS imaging sensor in which the bandpass filter 213 having a peak wavelength of 1200 nm having a large water absorbance is arranged, and the peak wavelength at which the water absorbance is small and a solar blind is formed. Skin moisture is measured based on the pixel output of the CIGS imaging sensor provided with the 1120 nm band-pass filter 217. In this way, reference wavelength information can be acquired without the influence of sunlight. On the other hand, the vein authentication, pulse wave detection and blood oxygen saturation measurement in the twelfth embodiment are performed for the pixel output of the CIGS imaging sensor provided with the bandpass filter 201 having a peak wavelength of 660 nm and the bandpass filter having a peak wavelength of 805 nm. The pixel output of the CIGS imaging sensor in which 113 is arranged and the pixel output of the CIGS imaging sensor in which the bandpass filter 207 having a peak wavelength of 900 nm is arranged are used. By using three wavelengths in this way, information for vein authentication and pulse wave detection is enriched.

Note that skin imaging in the twelfth embodiment is performed in five wavelength regions of a 660 nm image, a 805 nm image, a 900 nm image, a 1120 nm image, and a 1200 nm image. At this time, regarding focus detection, step S488 in FIG. 49 is read as “1200 nm image> 805 nm image?”, And step S490 is read as “1120 nm image> 660 nm image?”. As a result, it is possible to detect a state where the focus position of the image by the light having a wavelength near 900 nm is in alignment with the formed image CIGS imaging sensor, and to proceed to step S492.

FIG. 58 is an example of the color filter array of the CIGS imaging sensor 25 in the twelfth embodiment. In the example of FIG. 58, infrared transmission filters mIRref11 and mIRref33 having a peak wavelength of 1120 nm, infrared overfilters sIRref12, sIRref14, sIRref32 and sIRref34 having a peak wavelength of 805 nm, infrared transmission filters cIRref13 and cIRref31 having a peak wavelength of 900 nm, and a peak wavelength of 1200 Infrared transmission filters mIR21, mIR23, mIR41 and mIR43, and red transmission filters R22, R24, R42 and R44 having a peak wavelength of 660 nm are arranged as shown in the figure, and these 12 pixels are repeated as one unit. . These correspond to the bandpass filter 217, the bandpass filter 113, the bandpass filter 217, the bandpass filter 201, and the bandpass filter 213 in FIG. 57, respectively.

FIG. 59 is a front view showing an example of the arrangement of LEDs that can be employed in the twelfth embodiment, and shows the detailed configuration of the CIGS image sensor 25 shown in FIG. 45 and the image sensor light source 27 around it. In FIG. 59, the same reference numerals are given to the portions corresponding to FIG. As is clear from FIG. 57, a pair of LED 115 having a peak wavelength of 850 nm, LED 215 having a peak wavelength of 1200 nm, and LED 217 having a peak wavelength of 1070 nm are disposed around the lens 205 of the CIGS image sensor, and the optical axis of the lens 205 is sandwiched by 60 degrees. It is provided on the rotation object shifted by one. On the other hand, six LEDs 203 having a peak wavelength of 660 nm are arranged on the object to be rotated by being shifted by 60 degrees around the optical axis of the lens 205 and in the gaps between the LEDs 115, 215, and 217. Since the LED 203 having a peak wavelength of 660 nm can be configured to be relatively small, such mounting is possible. Further, by increasing the number, it is possible to use an LED having a small light emission intensity, thereby reducing the cost. In the case of a design in which the objectivity around the optical axis is not a problem, the number of LEDs can be reduced to one for each wavelength.

In the twelfth embodiment, as shown in FIG. 57, the bandpass filter 113 having a peak wavelength of 805 nm and the bandpass filter 207 having a peak wavelength of 900 nm share the LED 115 having a peak wavelength of 850 nm. This is because the CIGS imaging sensor has high quantum efficiency and high sensitivity. However, as a modified embodiment of the twelfth embodiment, a dedicated peak wavelength 800 nm LED and a peak wavelength 900 nm LED are used for the band pass filter 113 having a peak wavelength of 805 nm and the band pass filter 207 having a peak wavelength of 900 nm, respectively. It can also be configured to be used. In this case, the color filter arrangement of FIG. 55 can be used as it is, but since the number of types of LEDs increases to five, FIG. 59 is slightly modified to shift the LED pairs to which a large area is allocated by 46 degrees instead of 60 degrees. In addition to the type arrangement, eight fifth-type LEDs are arranged in the gap.

FIG. 60 is a graph for explaining the spectral transmission characteristics of the filter and the peak wavelength of the imaging sensor light source section in the thirteenth embodiment of the present invention implemented in a cellular phone. The thirteenth embodiment basically has the same configuration as that of the ninth embodiment, and can be understood by using the external view of FIG. 43, the block diagram of FIG. 44, and the flows of FIGS. 48 to 50. The thirteenth embodiment is different from the other embodiments in the image sensor. That is, the imaging sensor for skin moisture measurement and the like in the mobile phones from the ninth embodiment to the twelfth embodiment is composed of a CIGS imaging sensor, whereas the imaging sensor of the thirteenth embodiment is a silicon imaging sensor. . Therefore, in FIG. 60, the spectral sensitivity (quantum efficiency) of the image sensor is not that of the CIGS image sensor of FIG. 3A, but the spectral sensitivity (quantum efficiency) 221 of the silicon image sensor corresponding to FIG. Conceptually illustrated. Note that the spectral sensitivity 221 merely indicates a relative concept, and the absolute value on the vertical axis has no meaning.

The thirteenth embodiment shown in FIG. 60 has the same structure as the tenth embodiment except for the spectral sensitivity of the image sensor. Therefore, the color filter array of the image sensor can basically adopt the one shown in FIG. Also, the arrangement of LEDs as the light source can basically adopt the example of FIG. Since the thirteenth embodiment employs a silicon imaging sensor, the overall quantum efficiency is low, and the sensitivity at the peak wavelength of 790 nm is slightly insufficient. However, depending on the design conditions, it is possible to reduce the cost by employing a silicon imaging sensor as in the thirteenth embodiment. In order to compensate for the lack of sensitivity at the peak wavelength of 790 nm, the color filter array in FIG. 52 can be slightly changed, and the wavelength region where the number of pixels is doubled can be changed from 660 nm to 970 nm. Also, with regard to the mounting of the LED of FIG. 53, it is possible to change the number of 940 nm LEDs that may be insufficient in light quantity to be larger than other LEDs.

The various features of the present invention are not limited to the above-described embodiments, and various other implementations are possible. For example, in the above embodiment, the skin moisture measuring function is incorporated in the mobile phone, and there are various advantages. However, more generally, it can be configured as a dedicated measuring device, It can also be implemented. Further, even when incorporated in a mobile phone, the infrared light emitting unit 19 and the infrared light proximity sensor 21 in FIGS. 43 and 44 are omitted, and the skin proximity detection function that they are responsible for is also used as the CIGS imaging sensor. You can also. At this time, an acceleration sensor is provided in the mobile phone 1 in order to prevent erroneous detection of the approach of a finger during operation, and the posture of the mobile phone 1 is detected by detecting gravitational acceleration. Thus, skin proximity detection is performed only when proximity to the CIGS imaging sensor is detected in a state where the acceleration sensor detects that the mobile phone 1 is standing, and the mobile phone 1 is operated in a state where the mobile phone 1 is almost horizontal. The backlight of the display unit 5 is not turned off even if the finger is close to the imaging sensor while the unit 9 is being operated.

The present invention can be applied to a biological information measuring device such as a skin moisture measuring device.

25 Photoelectric conversion unit 39 Processing unit 27 Light source unit 111, 115, 119, 203, 211, 215, 219 Light-emitting diode 39 Determination unit 5 Display unit 43 Backlight 39 Determination unit 19, 21 Proximity detection unit 1 Mobile phone 39 Control unit

Claims (21)

A photoelectric conversion unit that has a quantum efficiency of 60% or more in each of the visible light region and the infrared light region, and detects biological light based on light reception information detected by the light beam conversion unit. A biological information acquisition apparatus. The biological information acquisition apparatus according to claim 1, wherein the photoelectric conversion unit includes a plurality of light receiving units regularly arranged to capture images of light in a plurality of wavelength ranges. 3. The biological information acquisition apparatus according to claim 1, wherein a light emitting diode is used as a light source, and a light emitting diode having one peak wavelength is used as a light source for detecting a plurality of wavelength ranges in the photoelectric conversion unit. The biological information acquisition apparatus according to claim 1, wherein skin photoelectric information is acquired by the photoelectric conversion unit. The living body according to claim 4, wherein the skin moisture information is acquired by the photoelectric conversion unit in a non-contact manner, and has a determination unit that determines that the acquired skin moisture is outside a normal measurement range. Information acquisition device. The biological information acquisition apparatus according to claim 4 or 5, wherein information on a vein pattern is acquired by the photoelectric conversion unit. The biological information acquisition apparatus according to claim 4, wherein pulse wave information is acquired by the photoelectric conversion unit. The biological information acquisition apparatus according to claim 7, wherein blood oxygen saturation is acquired based on pulse wave information acquired by the photoelectric conversion unit. The biological information acquisition apparatus according to claim 4, wherein skin imaging is performed. The biometric information acquisition apparatus according to claim 1, further comprising a display unit that displays the acquired biometric information, wherein the display brightness of the display unit is adjusted by an output of the photoelectric conversion unit. . The biological information acquisition apparatus according to claim 10, wherein the display unit includes a backlight that adjusts display brightness, and the backlight is controlled by an output of the photoelectric conversion unit. The biological information acquisition apparatus according to claim 1, wherein one of the plurality of wavelength ranges is set in a region where sunlight does not reach the ground surface. The biological information acquisition apparatus according to claim 2, further comprising: a determination unit that determines a focus state with respect to the object based on the images of the plurality of wavelength regions by the light of the plurality of wavelength regions. A display unit that displays a captured image and includes a backlight, and when the image is captured by the photoelectric conversion unit, the backlight is turned on to illuminate an object. 2. The biological information acquisition device according to 2. It has a light source unit for detection and a proximity detection unit that detects whether the photoelectric conversion unit is close to the object, and the proximity detection unit detects the proximity of the object when imaging is not performed The biological information acquisition apparatus according to claim 14, wherein the light source unit is turned on and the backlight is turned off. The biological information acquiring apparatus according to claim 1, wherein the biological information acquiring apparatus is configured as a mobile phone. A biological information acquisition apparatus comprising a photoelectric conversion unit that detects light in a plurality of wavelength ranges, and acquires skin moisture information and vein pattern information based on light reception information detected by the light beam conversion unit. 18. The biological information acquisition apparatus according to claim 17, wherein health information is acquired by the photoelectric conversion unit. A photoelectric conversion unit that detects light in a plurality of wavelength ranges, a processing unit that acquires biological information based on light reception information detected by the light beam conversion unit, a display unit that displays the acquired biological information, and the photoelectric conversion And a control unit that adjusts the display brightness of the display unit according to the output of the unit. A photoelectric conversion unit that detects light in a plurality of wavelength ranges, a processing unit that acquires biological information based on light reception information detected by the light beam conversion unit, a display that displays the acquired biological information and includes a backlight And a control unit that turns on the backlight to illuminate the object when the biological information is acquired by the photoelectric conversion unit. A photoelectric conversion unit that detects light in a plurality of wavelength ranges, a processing unit that acquires biological information based on light reception information detected by the light beam conversion unit, a display that displays the acquired biological information and includes a backlight , A light source unit for detection, a proximity detection unit that detects whether or not the photoelectric conversion unit is close to the object, and turns on the light source unit when the proximity detection unit detects the proximity of the object And a biological information acquisition apparatus comprising: a controller that turns off the backlight.

Images