JP5537870B2 - Vehicle monitoring device - Google Patents

Description

The present invention relates to an image sensor and its utilization.

As photoelectric conversion means, those having various characteristics are proposed every day. Conventionally, various image sensors for color images and image sensors for infrared images have been proposed. Furthermore, various proposals have been made on a vehicle monitoring apparatus for preventing traffic accidents using these image sensors. For example, Japanese Patent Laid-Open No. 2001-18738 (Patent Document 1) proposes to capture and display an object outside the irradiation range of the headlight with an infrared camera. Japanese Patent Application Laid-Open No. 2005-202815 (Patent Document 2) irradiates a band-pass filter that irradiates near-infrared light in a specific wavelength band where sunlight does not have a spectral irradiation degree and transmits only the specific wavelength band. It has been proposed to provide a driver with information at an intersection without being affected by sunlight even by daylight by taking an image with an infrared camera arranged on the axis.

JP 2001-18738 A JP 2005-202815 A

However, the various characteristics proposed in the photoelectric conversion means are not yet fully utilized, and there are many problems to be further examined.

In view of the above, an object of the present invention is to propose a suitable use for a vehicle monitoring device having characteristics proposed in photoelectric conversion means.

In order to achieve the above object, the present invention has a quantum efficiency of 60% or more in the visible light region and also has an effective quantum efficiency in the infrared light region where sunlight becomes weak on the ground surface due to atmospheric absorption. A photoelectric conversion unit in which a plurality of light receiving units are regularly arranged, an imaging lens that forms a subject image on the photoelectric conversion unit, and infrared light in an infrared region where sunlight is weak on the ground surface There is provided a vehicle monitoring apparatus having an infrared light projection unit and an image processing unit that acquires an image in a visible light range from a photoelectric conversion unit and a reflection image of infrared light projected from the infrared projection unit. Thereby, based on information from the photoelectric conversion unit, an image in the visible light region and a reflected image in the infrared light region where sunlight is weak on the ground surface can be acquired in association with each other.

According to the specific feature of the present invention, the filters having different spectral transmittances are arranged in the plurality of light receiving units, respectively, so that the assigned wavelength region can be shared by the plurality of light receiving units. According to another specific feature, the image processing unit acquires a reflected image of infrared light that has a predetermined positional relationship with an image in the visible light range, so the position of the reflected image of infrared light to be acquired Can be specified. According to a more specific feature, the image processing unit extracts the image of the vehicle from the image in the visible light range, so that the position of the reflected image of the infrared light from the vehicle can be specified. According to a more specific feature, the image processing unit acquires the reflected image of the infrared light from the center of gravity of the extracted vehicle image, so the position of the reflected image of the infrared light from the vehicle is more clearly defined. Can be identified. According to an even more specific feature, the image processing unit extracts the image of the closest single vehicle when a plurality of vehicle images overlap, so that the image from the closest vehicle in the case where the plurality of vehicles overlap each other is extracted. The position of the reflected image of infrared light can be clearly specified.

According to another specific feature of the present invention, the infrared light projection unit projects beam-shaped infrared light, so that the position of the reflected image can be clearly identified. According to another specific feature of the present invention, the distance to the reflection target is calculated from the reflected light image position of the infrared light acquired by the image processing unit and the infrared light projection direction from the infrared light projection unit. According to another specific feature of the present invention, the direction of infrared light projection by the infrared light projection unit is variable, which allows a wide range of reflection targets to be scanned.

According to another aspect of the present invention, an image acquisition unit that acquires an image of a target region, an extraction unit that extracts a vehicle image from the image acquisition unit, and sunlight on the target region is weak on the ground due to absorption of air. An infrared light projection unit for projecting infrared light in a band, a detection unit for detecting infrared light reflected from the center of gravity of the vehicle image extracted by the extraction unit, and a vehicle image extracted by the extraction unit A vehicle monitoring device is provided that includes a control unit that calculates a distance to an extracted vehicle based on the position of the center of gravity and the direction of infrared light projection from the infrared light projection unit. This provides a vehicle monitoring apparatus that can measure the inter-vehicle distance that identifies the target vehicle.

According to another aspect of the present invention, a photoelectric conversion unit that is sensitive to infrared light in a visible region and a band in which sunlight is weakened on the ground surface due to atmospheric absorption, and infrared light in which sunlight is weakened on the ground surface. Infrared light projection unit that projects infrared light in a region at a predetermined timing, visible light information acquisition unit that obtains light reception information in a visible light region from a photoelectric conversion unit, and infrared light projection timing by the infrared projection unit There is provided a vehicle monitoring apparatus including an infrared light information acquisition unit that acquires reflected light information of infrared light in an infrared light region where sunlight is weak on the ground surface from a photoelectric conversion unit. Imaging of the visible light region and acquisition of infrared light can be performed by effectively utilizing the characteristics of the photoelectric conversion unit that is sensitive to infrared light in the visible region and in the band where the sunlight becomes weak due to atmospheric absorption.

According to another aspect of the present invention, a plurality of light receiving units having sensitivity in a blue region, a plurality of light receiving units having sensitivity in a green region, and a red region and red in which sunlight is weakened on the ground surface due to atmospheric absorption. A photoelectric conversion unit in which a plurality of light receiving units having sensitivity in the outside light region are regularly arranged, an imaging lens that forms a subject image on the photoelectric conversion unit, and an infrared light region in which sunlight is weak on the ground surface An infrared light projection unit that projects infrared light, a visible light information acquisition unit that acquires image information in the visible light region from a plurality of light receiving units in the photoelectric conversion unit, a red region and sunlight in the photoelectric conversion unit in the atmosphere An infrared light information acquisition unit for acquiring reflected light information of infrared light in the infrared light region where sunlight is weak on the ground surface from a plurality of light receiving units having sensitivity in the infrared light region that is weak on the ground surface due to absorption; There is provided a monitoring device for vehicles. Thus, the same light receiving unit can be used for both the red region and the infrared light region where sunlight is weak on the ground surface due to atmospheric absorption. According to the above-mentioned specific feature of the present invention, the band cut filter that cuts the shorter wavelength side than the infrared light region that is longer wavelength side than the red region and the sunlight becomes weak on the ground surface due to atmospheric absorption is photoelectric conversion. The red region and the sunlight are selectively received in the infrared light region that is weak on the ground surface due to atmospheric absorption. According to a more specific feature, in place of the band cut filter, there is provided a filter driving unit that arranges a low-pass filter that cuts at least the visible light region in the optical path to the photoelectric conversion unit. It is possible to selectively receive an infrared region in which light can reach the ground surface and an infrared light region in which sunlight becomes weak on the ground surface due to atmospheric absorption.

  As described above, according to the present invention, it is possible to provide a vehicle monitoring apparatus that makes use of various characteristics proposed in the photoelectric conversion means.

It is a block diagram which shows 1st Example of this invention. Example 1 FIG. 2 is a block diagram showing in detail the configuration of the photometry / AF sensor and the live view sensor of FIG. 1 together with related portions. It is the graph which compared the spectral sensitivity of the CIGS sensor used for the CIGS imaging sensor and CIGSAF sensor of FIG. 2 with the silicon CMOS sensor. It is a flowchart of operation | movement of the camera control part in 1st Example. It is a block diagram which shows 2nd Example of this invention. (Example 2) It is a flowchart of operation | movement of the camera control part in 2nd Example. It is a 1st example of the color filter arrangement | sequence of the CIGS imaging sensor used for the live view sensor of FIG. 2 or FIG. It is a 2nd example of the filter arrangement | sequence of a CIGS image sensor. It is a schematic cross section of the CIGS sensor which employ | adopted the filter arrangement | sequence of FIG. It is a 3rd example of the filter arrangement | sequence of a CIGS image sensor. It is a flowchart which shows operation | movement of the camera control part at the time of recording the image of a live view sensor in 2nd Example of FIG. It is a flowchart which shows the detail of the process which can be utilized in common with step S108 of FIG. 11, and step S114. It is a block diagram which shows 3rd Example of this invention. (Example 3) It is a front view which shows the 1st example of arrangement | positioning of LED employable as a 3rd Example. It is a front view which shows the 2nd example of arrangement | positioning of LED employable as a 3rd Example. It is a front view which shows the 3rd example of arrangement | positioning of LED employable as a 3rd Example. It is a front view which shows the 4th example of arrangement | positioning of LED employable as a 3rd Example. It is an operation | movement timing chart in the color and infrared mode of 3rd Example. 19 is a timing chart showing the relationship between the operation of FIG. 18 and color image creation. It is an operation | movement timing chart in the fine color mode of 3rd Example. 21 is a timing chart showing the relationship between the operation of FIG. 20 and color image creation. It is an operation | movement timing chart in the infrared mode of 3rd Example. It is a flowchart of operation | movement of the endoscope control part in 3rd Example. It is a flowchart which shows the detail of step S170 of FIG. It is a flowchart which shows the detail of step S208 of FIG. It is a flowchart which shows the detail of step S172 of FIG. It is a flowchart which shows operation | movement of the monitor control part in 3rd Example. It is an operation | movement timing chart in the color and infrared mode of 4th Example of this invention. Example 4 It is an operation | movement timing chart in the fine color mode of 4th Example. It is a block diagram which shows 5th Example of this invention. (Example 5) It is the block diagram which showed the detail of the monitoring apparatus for vehicles of 5th Example with the distance detection principle. It is a conceptual diagram of the image imaged by the CIGS image sensor of 5th Example. It is a graph of the spectral transmission characteristic of the filter of the CIGS image sensor in 5th Example. It is a flowchart of operation | movement of the monitoring recording control part in 5th Example. It is a flowchart which shows the detail of step S336 of FIG. It is a flowchart which shows the detail of step S308 of FIG. 34, and step S346 of FIG. It is the block diagram which showed the detail of the monitoring apparatus for vehicles of 6th Example of this invention with the distance detection principle. (Example 6) It is a graph of the spectral transmission characteristic of the filter of the CIGS imaging sensor in 6th Example. It is a flowchart of operation | movement of the monitoring recording control part in 6th Example of FIG. It is a 4th example of the filter arrangement | sequence of a CIGS image sensor, and is used for the 7th Example of this invention. (Example 7) It is a graph of the spectral transmission characteristic of the filter of the CIGS image sensor in 7th Example. It is a graph of the spectral transmission characteristic of the color filter of the CIGS imaging sensor used for the front of the 8th Example of this invention, and the monitoring apparatus for vehicles. (Example 8)

FIG. 1 is a block diagram showing a first example of an autofocus digital single-lens reflex camera according to an embodiment of the present invention. The autofocus digital single-lens reflex camera has a camera body 2 and an interchangeable lens 4 that is detachably attached to the camera body 2. The subject light incident from the lens optical system 6 of the interchangeable lens 4 is reflected upward by the mirror 8 at the observation position and forms an image at the position of the focusing screen 10. After this image is reflected by the pentarhythm 12, it is observed by the eyepiece 14, and a composition for imaging is determined.

  At the time of shooting, by operating the shutter release button of the operation unit 15, the mirror 8 is retracted to the shooting position together with the auto-focusing sub mirror 16 and the focal plane shutter 18 is opened, and the lens optical system 6 of the interchangeable lens 4 is opened. The incident subject light is imaged and imaged on the imaging unit 20. Image information captured by the image capturing unit 20 is subjected to image processing by the image processing unit 22 and then stored in the image storage unit 26 under the control of the camera control unit 24. The image information stored in the image storage unit 26 is transferred to a storage medium such as a memory card inserted into the medium slot 28 as appropriate. The image information stored in the image storage unit 26 can be appropriately transferred from the input / output unit 30 to the outside under the control of the camera control unit 24. In addition, since the image information immediately after photographing is sent from the camera control unit 24 to the display unit 32 and automatically displayed, the operator can check the captured image.

  When the image is reproduced, the image information stored in the image storage unit 26 or the medium slot 28 is read by the camera control unit 24 by the operation of the operation unit 15, and is read from the liquid crystal provided on the back surface of the camera body 2. Is displayed on the display unit 32. The above is the basic configuration and basic functions related to imaging and playback in the autofocus digital single-lens reflex camera of FIG. As apparent from the above, when the mirror 8 is at the observation position, the imaging unit 20 does not capture the subject image, so the real-time subject image that can be observed with the eyepiece 14 with the above configuration is not displayed on the display unit. 32 is not displayed and can only be confirmed after shooting. This is the special characteristic of a digital single lens reflex camera, which is different from a normal compact digital camera that can determine the composition while observing the image on the display unit 32.

  Next, the configuration and functions related to autofocus in the autofocus digital single lens reflex camera of FIG. 1 will be described. Part of the subject light incident from the lens optical system 6 of the interchangeable lens 4 is transmitted through the semi-transmissive portion at the center of the mirror 8 at the observation position, and reflected downward by the sub mirror 16 to be used for photometry / autofocus (hereinafter referred to as “AF”). ]) Guided to the sensor. The photometry / use AF sensor 34 re-images the light incident from the sub-mirror 16 on the AF sensor, analyzes it, and sends the result to the camera control unit 24. This analysis is performed, for example, by analyzing the shift direction and the degree of the image formation position between the imaging surface of the imaging unit 20 and the lens optical system 6 by a well-known phase difference detection method using pupil division. The camera control unit 24 drives the lens optical system 6 to eliminate the deviation of the imaging position based on the information on the direction and degree of deviation of the imaging position by the lens optical system 6 obtained from the AF sensor 34 for photometry. Information on the amount and the driving direction is sent to the AF control unit 36. The AF driving unit 38 drives the lens optical system 6 on the basis of information on the driving amount and driving direction transmitted from the AF control unit 36 through a mechanical or electrical interface between the camera body 2 and the interchangeable lens 4 and performs automatic focusing. I do. The details of the configuration of the photometry / AF sensor 34 will be described later.

  The live view sensor 40 is configured for a “live view” function in a digital single-lens reflex camera so that the composition can be determined while observing the image on the display unit 32 in the same manner as an ordinary compact digital camera. is there. The entire reflecting surface 12a of the pentaprism 12 is semi-transmissive, and the live view sensor 40 can capture the entire image of the focal plane 10 by re-imaging the image of the focusing screen 10 on the CIGS imaging sensor. It is like that. The CIGS imaging sensor is an optical sensor made of copper (Cu), indium (In), gallium (Ga), and selenium (Se), and details thereof will be described later.

The reflection surface 12a of the pentaprism 12 transmits light almost entirely except in the visible light region, and has a spectral transmission characteristic in which most of the visible light region is reflected by only slightly transmitting light. When the image of the focal plate 10 is observed at 14, the image is not substantially darkened. Further, since the CIGS imaging sensor employed in the live view sensor 40 has high sensitivity in the visible light region as described later, the light transmittance of the reflecting surface 12a in the visible light region is slight. In addition, it is possible to sufficiently capture an image of the focal plate 10 in the visible light region. The distribution of the visible light to the live view sensor 40 is set to a level at which the light quantity is insufficient for the live view imaging by the CIGS imaging sensor when it becomes dark that it is difficult to optically observe the subject by the eyepiece 14. Details of the live view sensor 40 using the CIGS imaging sensor will be described later. Since the image captured by the live view sensor 40 is sent to the camera control unit 24 and displayed on the display unit 32, the autofocus digital single-lens reflex camera shown in FIG. 1 is normal as long as the subject has normal brightness. The composition can be determined while observing the image on the display unit 32 in the same manner as the compact digital camera.

FIG. 2 is a block diagram showing in detail the configuration of the photometry / use AF sensor 34 and the live view sensor 40 in the first embodiment of the autofocus digital single-lens reflex camera of FIG. As described above, the reflecting surface 12a of the pentaprism 12 transmits light almost entirely except in the visible light region, and in the visible light region, has a spectral transmission characteristic in which most of the light is reflected by only slightly transmitting light. However, the multilayer film 52 is coated on the reflecting surface 12a in order to realize such spectral transmission characteristics.

The live view sensor 40 includes a re-imaging lens 56 for re-imaging the light beam from the focal plane 10 that has passed through the multilayer film 52 on the imaging surface of the CIGS imaging sensor 54. The infrared light cut filter 58 substantially cuts light outside the visible light region that has been transmitted through the multilayer film 52 to approximate the wavelength sensitivity characteristic of the CIGS imaging sensor 54 to the wavelength sensitivity characteristic of the imaging unit 20. Then, a subject image that matches the visibility is picked up and sent to the camera control unit 24 to enable live view on the display unit 32 of FIG. The “infrared light” in the present invention mainly refers to infrared light in a region that is relatively close to visible light, which is called “near infrared light”. Abbreviated as “infrared light”.

The live view sensor 40 further enables full-screen photometry for measuring the brightness on the focusing screen 10. That is, the image information output from the CIGS imaging sensor 54 is processed by the camera control unit 24 as photometric information over the entire screen, and is combined with the output of the AF-compatible partial photometric sensor 72 as necessary. Based on these processing results, automatic exposure control is performed to control the aperture diameter of the interchangeable lens 4, the shutter speed by the focal plane shutter 18, the sensitivity of the imaging unit 20, and the like.

The visible light cut filter 60 is inserted in the optical path to the CIGS imaging sensor 54 in place of the infrared light cut filter 58, and is used in the “long wavelength mode”. Since the multilayer film 52 transmits almost the entire region other than the visible light region, when the visible light cut filter 60 is inserted in the optical path instead of the infrared light cut filter 58 in the long wavelength mode setting, it is visible. Light having a wavelength longer than that of the light enters the CIGS imaging sensor 54. As will be described later, the CIGS imaging sensor 54 has a spectral sensitivity of 1300 nm on the long wavelength side. Therefore, by inserting the visible light cut filter 60, the live view sensor 40 becomes an image sensor suitable for photographing with light in these long wavelength regions. Then, such an image output in the long wavelength region can be observed in real time on the display unit 32 or recorded in the image storage unit 26.

The mirror / filter drive unit 62 drives replacement of the visible light cut filter 60 and the infrared light cut filter 58 under the control of the camera control unit 24 according to mode switching by the operation unit 15. In FIG. 2, the mirror 8a and the sub mirror 16a retracted to the photographing position are shown by two-dot chain lines. However, the driving between the observation position and the photographing position in the mirror 8 and the sub mirror 16 is also performed by the camera control unit. The L / filter driving unit 62 performs the control of 24.

The re-imaging lens 64 of the AF metering sensor 34 re-images the subject light incident from the interchangeable lens 4 and transmitted through the semi-transmissive portion at the center of the mirror 8 at the observation position and reflected downward by the sub-mirror 16. It is for making it happen. The light beam from the re-imaging lens 64 passes through the movable semi-transmissive mirror 66 and the infrared light cut filter 68 having no wavelength selectivity and forms an image on the CIGSAF sensor 70. The CIGSAF sensor also has high sensitivity in the visible light region, as will be described later, and can perform automatic focus detection without auxiliary light even for a dark subject. The infrared light cut filter 68 cuts the wavelength of the harmful infrared light region in order to make the CIGSAF sensor 70 work as an AF sensor. What is the infrared light cut filter 58 for the CIGS imaging sensor 54? The characteristics are not necessarily the same. For example, the infrared light cut filter 8 is set to have a transmission spectral characteristic narrower than that of the infrared light cut filter 58.

For this reason, when the subject has normal brightness, the movable semi-transparent mirror 66 is inserted at the position shown in the figure for dimming, and the amount of light incident on the CIGSAF sensor is adjusted to the sensitivity dynamic range of the CIGSAF sensor. On the other hand, when the subject becomes dark enough to require auxiliary light in a normal AF sensor, the movable semi-transparent mirror is retracted to the position 64a, and the subject image is formed on the CIGS sensor without dimming. At this time, it is necessary to compensate for the optical path length depending on the presence or absence of the movable semi-transmissive mirror 66. For example, when the movable translucent mirror 66 is retracted, a totally transmissive parallel plate having the same optical path length is inserted into the optical path instead. Of course, when the movable semi-transmissive mirror is retracted to the position 64a, photometry by the AF-compatible partial photometry sensor 72 cannot be performed.

When the movable semi-transmissive mirror 66 is inserted in the re-imaging optical path for dimming the light to the CIGSAF sensor 70, the light reflected therefrom enters the AF corresponding partial photometric sensor 72. The AF-corresponding partial photometric sensor 72 measures the brightness of the portion of the focus detected by the CIGSAF sensor, and selectively selects the brightness of the portion of the entire screen that is the focus detection target. By measuring the light, it is used as information for automatic exposure control so that a portion of high interest in photographing is appropriately exposed. As described above, the dimming portion to the CIGSAF sensor 70 that becomes excessive when the subject is bright is not discarded but is effectively used as photometric information.

The partial photometric information from the AF corresponding partial photometric sensor 72 is processed by the camera control unit 24 in combination with the photometric information on the entire screen from the CIGS imaging sensor 54 of the live view sensor 40, and finally the aperture diameter of the interchangeable lens 4. The shutter speed by the focal plane shutter 18 and the sensitivity of the imaging unit 20 are controlled.

The sensor control unit 74 controls the automatic focus adjustment by performing the light reception integration time and gain control of the CIGSAF sensor 70 regardless of whether the movable semi-transparent mirror 66 is inserted or retracted. To do. In order to perform the light receiving integration time and gain control without confusion, information on whether the movable semi-transparent mirror 66 is inserted or retracted is also used. The sensor control unit 74 also issues an instruction to the CIGSAF sensor 70 and the AF-corresponding partial photometric sensor 72, and performs control to match the portion to be subjected to focus detection with the portion to be subject to focus detection in the entire screen, The camera control unit 24 outputs corresponding focus detection information and photometry information.

On the other hand, when the live view sensor 40 is set to the “long wavelength mode” and the visible light cut filter 60 is inserted in the optical path instead of the infrared light cut filter 58 in the optical path from the multilayer film 52 to the CIGS imaging sensor 54. The photometric AF sensor 34 also performs filter replacement corresponding to this. Specifically, in the “long wavelength mode”, the infrared light cut filter 68 is replaced with the visible light cut filter 76 on the premise that the semi-transmissive mirror 66 is retracted. Thereby, the focus detection for imaging in the long wavelength range by the CIGS imaging sensor 54 is performed by the CIGSAF sensor 70 without dimming. At this time, not only adjustment of the wavelength sensitivity range but also compensation for a change in optical path length due to a difference in wavelength and a difference in chromatic aberration during focus detection, etc.

The movement of the movable semi-transmissive mirror 66 and the replacement of the infrared light cut filter 68 and the visible light cut filter 76 as described above are driven by the mirror / filter by the control of the camera control unit 24 based on the mode switching operation by the operation unit 15. Department 78 governs.

The first embodiment shown in FIGS. 1 and 2 can provide a “composite AF function” in addition to the basic functions as described above. When the “composite AF function” is selected by the operation of the operation unit 15, the live view function is stopped and the start of the “composite AF function” is instructed. Specifically, when the “composite AF function” is selected by operating the operation unit 15, the camera control unit 24 replaces the infrared light cut filter 58 with the visible light cut filter 60 to the CIGS imaging sensor 54. An instruction to set the optical path is given to the mirror / filter driving unit 62 and the live view display on the display unit 32 based on the output of the CIGS imaging sensor 54 is stopped.

Instead, the image signal of the CIGS imaging sensor 54 that has become a sensitivity region on the long wavelength side by the visible light cut filter 60 is combined with the output of the CIGSAF sensor in the visible light sensitivity region by the infrared light cut filter 68, The “composite AF function” is executed. Specifically, image analysis of the subject is performed by image processing based on the image signal of the CIGS imaging sensor 54, and a focus detection area by the CIGSAF sensor is determined based on the result.

FIG. 3 compares the spectral sensitivity (quantum efficiency) of the CIGS sensor used in the CIGS imaging sensor 54 and the CIGSAF sensor of FIG. 2 with a silicon CMOS sensor. FIG. 3A shows the quantum efficiency (%) of the CIGS sensor at each wavelength, which is clear compared to the similar quantum efficiency (%) for the silicon CMOS sensor in FIG. 3B. It shows high sensitivity and broadband characteristics. Specifically, the CIGS sensor shown in FIG. 3A has a wide sensitivity range over a wavelength of about 1300 nm. Furthermore, it has a spectral sensitivity exceeding 50% quantum efficiency over a wide wavelength range from about 400 nm to about 1200 nm, and shows particularly remarkable high quantum efficiency in the visible light and the infrared light region adjacent thereto. Such a high-sensitivity and broadband spectral sensitivity characteristic having a quantum efficiency of 60% or more in the visible light region and the infrared light region cannot be expected with a silicon CMOS sensor as shown in FIG.

FIG. 4 is a flowchart of the operation of the camera control unit 24 in the first embodiment of FIGS. 1 and 2. When the main switch of the camera is turned on by the operation unit 15, the flow starts. In step S2, it is checked whether the autofocus digital single-lens reflex camera is set to the reproduction mode by the operation unit 15. If the reproduction mode setting is not detected, the mode is the shooting mode, and the process proceeds to step S4, where the movable semi-transparent mirror 66 is set in the optical path to the CIGSAF sensor 70 to instruct the mirror / filter driving unit 78 to reduce the amount of incident light.

It should be noted that a delay period is provided in the mechanical execution of the setting of the movable semi-transmissive mirror 66 by the mirror / filter driving unit 78 in response to the instruction in step S4. For example, the movable semi-transmissive mirror 66 is provided in the optical path to the CIGSAF sensor 70. Instructed to cancel the dimming to retract the movable semi-transparent mirror 66 from the optical path in the set state, and then instructed to set the movable semi-transparent mirror 66 in the optical path within the delay time. In the case where it is performed subsequently, the mirror / filter driving unit 78 does not actually drive the movable semi-transmissive mirror 66, and the state where the movable semi-transmissive mirror 66 is set in the optical path continues. . In other words, the mirror / filter driving unit 78 executes the driving of the movable semi-transmissive mirror 66 only after the instruction to drive the movable semi-transmissive mirror 66 to a different state is repeatedly performed within the delay time. In addition, when the instruction in step S4 is performed in a state where the movable semi-transmissive mirror 66 is already set in the optical path of the CIGSAF sensor 70, the mirror / filter driving unit 78 naturally has no effect on the movable semi-transmissive mirror 66. Also does not drive. These are common to various “instructions” in the following steps.

Next, in step S6, the mirror / filter driving unit 62 is instructed to set the live view infrared light filter 58 in the optical path to the CIGS image sensor 54. Note that the mirror replacement operation of the mirror / filter drive unit 62 is also provided with a delay time for drive execution in response to the same instruction as described in the mirror / filter drive unit 78 above.

In step S8, the mirror / filter driving unit 78 is instructed to set the infrared cut filter 68 for AF in the optical path to the CIGSAF sensor 70. In step S10, based on the output of the CIGSAF sensor 70, it is checked whether or not the subject is dark enough to cancel the dimming. If applicable, the process proceeds to step S12, where an instruction to cancel the light reduction for retracting the movable semi-transmissive mirror 66 from the optical path is given, and the process proceeds to step S14. On the other hand, if the subject is sufficiently bright, the process proceeds directly to step S14.

In step S14, it is checked whether or not the “composite AF mode” has been selected by the operation unit 15. If there is a selection, the process proceeds to step S16, and an instruction to set the visible light cut filter 60 in the optical path to the CIGS imaging sensor 54 in place of the infrared light filter 58 in order to perform the composite AF is given to the mirror / filter drive unit 62. Do it. Further, in step S18, the live view display on the display unit 32 based on the output of the CIGS imaging sensor 54 is stopped, and the CIGSAF sensor in the sensitivity range of the visible light and the image signal of the CIGS imaging sensor 54 in the sensitivity region on the long wavelength side is stopped. The start of the “composite AF function” to be combined with the output is instructed, and the process proceeds to step S20. On the other hand, if the selection of “AF mode” is not detected in step S14, the process directly proceeds to step S20.

In step S <b> 20, it is checked whether the image picked up by the image pickup unit 20 is so dark that the light quantity becomes insufficient. Usually, when the subject becomes dark to this level, photographing using auxiliary light such as a flash is required. When the light quantity shortage is detected in step S20, the process proceeds to step S22, and it is checked whether or not the “long wavelength mode” is selected by the operation of the operation unit 15. Then, if applicable, the process proceeds to step S 24, and an instruction to set the live view visible light cut filter 60 in the optical path to the CIGS imaging sensor 54 instead of the infrared light filter 58 is given to the mirror / filter drive unit 62. Further, in step S26, the mirror / filter drive unit 78 is instructed to set the visible light cut filter 76 for AF in the optical path to the CIGSAF sensor 70 instead of the infrared light cut filter 68, and the process proceeds to step S28. To do.

On the other hand, if it is not detected in step S20 that the amount of light in the imaging unit is insufficient, the process directly proceeds to step S28. As described above, when the image is not dark enough to cause the light amount of the image capturing unit to be insufficient, the process cannot normally proceed to step S22, and the “long wavelength mode” is prohibited. This is to prevent confusion of settings. Even when the subject is bright, it is possible to proceed to step S22 by performing a special operation with the operation unit 15 in particular when it is desired to select the “long wavelength mode”. Further, when the “long wavelength mode” setting is not detected in step S22, the process directly proceeds to step S28.

In step S28, it is checked whether or not a release operation using the shutter release button of the operation unit 15 has been performed. If the release operation cannot be detected, the process proceeds to step S30, and it is checked whether an operation to turn off the main switch of the camera is performed by the operation unit 15. If no camera-off operation is detected, the flow returns to step S2, and thereafter, step S2 to step S30 are repeated unless a playback mode operation is detected in step S2 or a release operation is not detected in step S28.

The above repetition is performed at a sufficiently high speed and is repeated many times within the delay time provided in the mirror / filter driving units 62 and 78 described above. Accordingly, when the detection result based on step S10, step S14, step S20 and step S22 changes, the same instruction based on this change is repeatedly performed within the delay time of the mirror / filter driving units 62 and 78, and the mirror / filter driving unit 62 78, an appropriate instruction is executed. Accordingly, the setting / cancellation of the light reduction based on the change in the brightness of the subject, the switching of the wavelength band cut filter, and the switching of the wavelength band cut filter based on the mode switching are executed smoothly.

When a playback mode setting operation by the operation unit 15 is detected in step S2, the process proceeds to the playback mode process in step S32. When the shooting mode is selected by the function in the playback mode process, the flow returns to step S4. Also, when a camera-off operation is detected by a function within the playback mode process, the flow ends.

On the other hand, when the release fixed operation by the shutter release button of the operation unit 15 is detected in step S28, the process proceeds to the imaging recording process in step S34. When the imaging record and the imaging result display on the display unit are finished, the flow automatically returns to step S2. Note that when the camera-off operation is detected in step S30, the flow of FIG. 4 ends.

FIG. 5 is a block diagram showing a second example of the autofocus digital single-lens reflex camera according to the embodiment of the present invention. Since most of the configuration is the same as that of the first embodiment of FIG. 1, the same reference numerals are given to common portions, and the description is omitted unless particularly required. The second embodiment shown in FIG. 5 differs from the first embodiment shown in FIG. 1 in the camera body 100. In particular, the live view sensor 102 and the configuration and functions related thereto are different from those in the first embodiment.

The live view sensor 40 according to the first embodiment is configured to receive light through the semi-transmissive reflective surface 12a, and suppresses light in the visible light region that is transmitted through the reflective surface 12a. This is because the object image can be optically observed with the eyepiece 14 without hindrance, and at the same time, a live view is always possible. Since the CIGS imaging sensor is used for the live view sensor 40, even if the light in the visible light region that is transmitted through the reflecting surface 12a is suppressed, it is sufficient for live view of a subject with normal brightness. However, in the case of a dark subject that cannot be sufficiently observed with the eyepiece 14, the live view sensor 40 has insufficient light quantity. On the other hand, the second embodiment of FIG. 5 is configured so that the live view can be performed by the cIGS imaging sensor employed in the live view sensor 102 even in the case of a dark subject that cannot be sufficiently observed by the eyepiece 14. The detailed structure of the live view sensor 102 in FIG. 5 is basically the same as the live view sensor 40 in FIG. 2, and includes a re-imaging optical system and a CIGS imaging sensor. However, since the arrangement position of the live view sensor 102 with respect to the pentaprism 104 is different, the re-imaging optical system is different from the re-imaging lens 56 of FIG.

Based on the above concept, in the second embodiment, the normal pentaprism 104 is employed, and when the live view mode is not set, all the light from the pentaprism 104 goes to the eyepiece. At this time, the movable total reflection mirror 106 is retracted from the optical path to the eyepiece 14 as shown in FIG. Therefore, live view is not possible in this state.

When the live pure mode is selected by operating the operation unit 15, the movable total reflection mirror is lowered to a position 106 a and all the light from the pentaprism 104 is reflected in the direction of the live view sensor 102. Therefore, the optical viewfinder image cannot be observed by the eyepiece 14. The movable neutral density filter 108 is inserted in the optical path to the live view sensor as shown in FIG. 5 when the subject has normal brightness, and matches the amount of light incident on the live pure sensor 102 to the sensitivity dynamic range of the CIGS imaging sensor. Come on. On the other hand, when the subject becomes so dark that it is difficult to observe with the eyepiece, the movable neutral density filter 108 retracts from the optical path to the live view sensor 102 and guides the subject image to the live view sensor without dimming. At this time, it is necessary to compensate the optical path length depending on the presence / absence of the movable neutral density filter 108. For example, when the movable neutral density filter is retracted, a parallel plate having the same optical path length as that of the movable neutral density filter is used instead. Insert into. In this way, even in the case of a dark subject that is difficult to observe optically, in the case of the second embodiment of FIG. 5, a live view is possible by the CIGS imaging sensor. The image in the visible light range from the live view sensor 102 can be recorded not only in the live view on the display unit 32 but also in the image storage unit 26. It becomes possible to do.

The infrared light cut filter 110 cuts light outside the visible light region reflected from the movable total reflection mirror 106a in the live view mode, and approximates the wavelength sensitivity characteristic of the CIGS imaging sensor to the wavelength sensitivity characteristic of the imaging unit 20. A subject image that matches the visibility is captured and sent to the camera control unit 116 to enable a natural live view.

The visible light cut filter 112 is inserted in the optical path to the live view sensor 102 in place of the infrared light cut filter 110 and is used in the “long wavelength mode”. Since the movable total reflection mirror 106a reflects almost the entire area other than the visible light region, the visible light cut filter 112 replaces the infrared light cut filter 110 in the long wavelength mode setting, and the optical path to the live view sensor 102 is changed. Is inserted into the CIGS imaging sensor of the live view sensor 102, light having a wavelength longer than visible light is incident on the CIGS imaging sensor. Accordingly, as in the first embodiment, the image output of the image in the long wavelength region can be observed in real time on the display unit 32 or recorded in the image storage unit 26. In the long wavelength mode using the visible light cut filter 112, the movable neutral density filter 108 is retracted from the optical path to the live view sensor. The movable total reflection mirror 106, the movable neutral density filter 108, the infrared light cut filter 110, and the visible light cut filter 112 are driven by the mirror / filter drive unit 114 controlled by the camera control unit 116. .

FIG. 6 is a flowchart of the operation of the camera control unit 116 in the second embodiment of FIG. As in the first embodiment, when the main switch of the camera is turned on by the operation unit 15, the flow starts. In step S42, it is checked whether or not the autofocus digital single-lens reflex camera is set to the reproduction mode by the operation unit 15. If the reproduction mode setting is not detected, the shooting mode is set, and the process proceeds to step S44 to instruct to set the optical viewfinder optical path. Specifically, the mirror / filter driving unit 114 is instructed so that the movable total reflection mirror 106 is retracted from the optical path to the eyepiece 14. In step S44, an instruction to reduce the amount of incident light by inserting the movable neutral density filter 108 into the optical path to the live view sensor 102 is sent to the mirror / filter driving unit 114, and the movable semi-transmissive mirror 66 in FIG. The mirror / filter driving unit 78 is instructed to reduce the amount of incident light by setting in the optical path to the CIGSAF sensor 70.

In step S46, it is checked whether the “live pew mode” is set by the operation unit 15. If applicable, the process proceeds to step S48, and an instruction to switch the optical path to the live view is issued. Specifically, the mirror / filter driving unit 114 is instructed to advance the movable total reflection mirror 106 in the optical path to the eyepiece 14, and the process proceeds to step S50. When this instruction is executed, the finder image cannot be optically observed from the eyepiece 14, and a live view on the display unit 32 based on the output of the live view sensor 102 is enabled instead. In step S48, the mirror / filter driving unit 114 is further instructed to set the infrared light cut filter 110 for live view in the optical path to the live view sensor 102. Note that the mirror / filter driving unit 114 is also provided with a delay time for executing the driving in response to the instruction as described in the first embodiment. On the other hand, when the setting to “live view mode” is not detected in step S46, the process directly proceeds to step S50.

In step S50, the mirror / filter driving unit 78 is instructed to set the infrared cut filter 68 for AF in the optical path to the CIGSAF sensor 70. In step S52, based on the output of the CIGSAF sensor 70, it is checked whether or not the subject is dark enough to cancel the dimming. If applicable, the process proceeds to step S54, and it is checked whether the “live pew mode” is set by the operation unit 15. If applicable, the process proceeds to step S56, where an instruction to cancel the dimming is performed to retract the movable neutral density filter 108 from the optical path to the live view sensor 102, and the process proceeds to step S58. On the other hand, if the setting of “live view mode” is not detected in step S54, the process directly proceeds to step S58. In step S58, the movable semi-transparent mirror 66 is retracted from the optical path to the CIGSAF sensor 70 to instruct to cancel the AF dimming. Thus, if it is detected in step S52 that the subject is dark, the AF dimming is canceled regardless of the “live view mode” setting.

Next, in step S60, it is checked whether the image picked up by the image pickup unit 20 is so dark that the light quantity becomes insufficient. If applicable, the process proceeds to step S62, and it is checked whether or not the long wavelength mode is selected by operating the operation unit 15. Then, if applicable, the process proceeds to step S 64, and an instruction to set the live view visible light cut filter 112 in the optical path to the live view sensor 102 instead of the infrared light filter 110 is given to the mirror / filter driving unit 114. In step S 66, the mirror / filter driving unit 78 is instructed to set the visible light cut filter 76 for AF in the optical path to the CIGSAF sensor 70 instead of the infrared light cut filter 68.

Through the above steps, the flow proceeds to step S68. On the other hand, if it is not detected in step S52 that the subject is dark enough to cancel the dimming, if it is not detected in step S60 that the subject is dark enough that the imaging by the imaging unit 20 is insufficient, and step If no selection for the long wavelength mode is detected in S62, the process directly proceeds to step S68.

In step S68, it is checked whether or not a release operation using the shutter release button of the operation unit 15 has been performed. If the release operation cannot be detected, the process proceeds to step S70 to check whether the operation unit 15 has performed an operation to turn off the main switch of the camera. If no camera-off operation is detected, the flow returns to step S42, and thereafter, step S42 to step S70 are repeated unless a playback mode operation is detected in step S42 or a release operation is not detected in step S68.

Similar to the first embodiment, the above repetition is performed at a sufficiently high speed, and is repeated many times within the delay time provided in the mirror / filter driving units 78 and 114 described above. Accordingly, when the detection result based on step S46, step S52, step S54, step S60 and step S62 changes, the same instruction based on this change is repeatedly performed within the delay time of the mirror / filter drive units 78 and 114, and the mirror / filter Appropriate instructions are executed by the drive units 78 and 114. Accordingly, the setting / cancellation of the light reduction based on the change in the brightness of the subject, the switching of the wavelength band cut filter, and the switching of the wavelength band cut filter based on the mode switching are executed smoothly.

As in the first embodiment, when a reproduction mode setting operation by the operation unit 15 is detected in step S42, the process proceeds to a reproduction mode process in step S72. When the shooting mode is selected by the function within the playback mode process, the flow returns to step S44. Also, when a camera-off operation is detected by a function within the playback mode process, the flow ends.

Further, when the release fixed operation by the shutter release button of the operation unit 15 is detected in step S68, the process proceeds to the imaging recording process in step S74. When the imaging record and the imaging result display on the display unit are finished, the flow automatically returns to step S42. Note that when the camera-off operation is detected in step S70, the flow of FIG. 6 ends.

The various features of the present invention described above are not limited to the embodiments and can be widely used. For example, in the first embodiment, the visible light cut filter 60 combines the CIGS imaging sensor 54 having sensitivity in the long wavelength region and the infrared light cut filter 68 with the output of the CIGSAF sensor having sensitivity to visible light. The “composite AF function” has been described as being implemented. However, the implementation of the “composite AF function” is not limited to this. For example, the movable semi-transparent mirror 66 having no wavelength selectivity in FIG. 2 is formed of a dichroic mirror, transmits visible light and guides it onto the CIGSAF sensor 70, and reflects the long wavelength region to guide to the AF-compatible partial photometric sensor 72. Like that. A CIGS sensor is also used for the AF-compatible partial photometric sensor 72. In this case, the infrared light cut filter 68 is not necessary.

If comprised as mentioned above, it becomes possible to estimate where a person exists in the AF corresponding part by the AF corresponding partial photometric sensor 72 having sensitivity in the long wavelength region, and the focus by the CIGSAF sensor 70 is applied to that part. Detection can be performed.

Furthermore, the implementation of the “composite AF function” is not limited to the one using two CIGS sensors as described above. For example, in FIG. 2, in the state where the visible light cut filter 76 is inserted in the optical path of the CIGSAF sensor, the CIGSAF sensor itself estimates where a person exists in the AF-corresponding portion, and the infrared light cut filter 18. It is also possible to perform focus detection by the CIGSAF sensor 70 on the portion in a state where the is inserted in the optical path. It is also possible to realize the “composite AF function” by using one CIGS having a wide sensitivity region in different sensitivity regions in a time division manner and combining their outputs.

In the above embodiment, the description has been made on the assumption that the movable semi-transmissive mirror or the filter is put in and out of the optical path for dimming. However, the adjustment of the incident light quantity is not limited to such two steps. For example, a plurality of neutral density filters with different transmittances may be prepared, and one of these may be inserted into the optical path to change the degree of attenuation finely in a stepwise manner. You may comprise so that the degree of light attenuation may be changed continuously using the light reduction means from which the transmittance | permeability changes continuously.

In the above embodiment, the CIGS sensor is used as a sensor having a high sensitivity and a broadband spectral sensitivity characteristic having a quantum efficiency of 60% or more in the visible light region and the infrared light region. The CIGS sensor is a photoelectric sensor using a polycrystalline CIGS thin film made of copper, indium, gallium, and selenium, and the absorption wavelength region can be controlled by changing the band gap by controlling the composition thereof. Among them, the gallium content is zero, which is also referred to as “CIS thin film”. However, in the present specification, the “CIGS sensor” refers to such a “CIS thin film” that does not contain gallium. It shall also mean a photoelectric sensor.

FIG. 7 is a first example of the color filter array of the CIGS imaging sensor used in the live view sensor 40 in the first embodiment of FIG. 2 or the live view sensor 102 in the second embodiment of FIG. In this first example, an infrared light transmission filter R11, a blue transmission filter B12, a green transmission filter G22, and a red transmission filter R21 are arranged as shown in the figure, and this is repeated as a unit. Since the CIGS imaging sensor of the present invention has a wide spectral sensitivity range from the visible light range to the infrared light as shown in FIG. 3, the color filter for visible light and infrared light can be provided in one sensor in this way. . The arrangement in FIG. 7 is obtained by adding an infrared light transmission filter to the primary color filter. Unlike the typical Bayer arrangement in the primary color filter, the green light receiving area is the same as that of blue and blue. This point can be corrected by later circuit processing.

Here, the interpolation of the infrared light image regarding the pixel in which the infrared light transmission filter is not disposed will be described. First, for the pixel corresponding to the blue transmission filter B12, basically, the average value of the pixel data corresponding to the infrared light transmission filter IR11 and the pixel data corresponding to the infrared light transmission filter IR13 on both sides thereof is determined. Interpolation is performed. The same applies to the interpolation of the infrared light image in the pixels corresponding to the other blue transmission fills. On the other hand, the pixel corresponding to the red transmission filter R21 is similarly interpolated by the average value of the pixel data corresponding to the infrared light transmission filter IR11 above and below the pixel data corresponding to the infrared light transmission filter IR31. The The same applies to the interpolation of the infrared light image in the pixels corresponding to the other red transmission filters. For the pixel corresponding to the green transmission filter G22, the pixel data corresponding to the infrared light transmission filter IR11, the pixel data corresponding to the infrared light transmission filter IR13, and the infrared light transmission filter IR33 around it. Interpolation is performed based on the average value of the corresponding pixel data and the pixel data corresponding to the infrared light transmission filter IR31. The same applies to the interpolation of the infrared light image in the pixels corresponding to the other green transmission filters.

Note that with simple interpolation as described above, an infrared light image different from the actual subject may be obtained. To prevent this, instead of simply interpolating an infrared light image based only on the data corresponding to the nearby infrared light transmission filter, the visible light data affecting the pixel being interpolated is also included. It is effective to perform interpolation with consideration. For example, in the interpolation of the infrared light image of the pixel corresponding to the red transmission filter R21, the red light data actually received by the pixel corresponding to the red transmission filter R21 is also taken into consideration. The presence or absence of such visible light data and the degree of addition are determined based on the correlation between visible light data and infrared light data or the correlation between other visible light data of surrounding pixels. .

FIG. 8 is a second example of the filter array of the CIGS imaging sensor used in the live view sensor 40 in the first embodiment of FIG. 2 or the live view sensor 102 in the second embodiment of FIG. In the second example, the color filter array itself is common to the first example of FIG. 7, but the light receiving area of each color filter is different. In other words, the infrared light transmission filter IR11 and the green transmission filter G22 secure the maximum light receiving area allowed for the pixel, but the blue transmission filter B12 is provided with the light shielding portion 202 so that the light reception area is the green transmission filter. It is about half of G22. Similarly, with respect to the red transmission filter R21, the light receiving area is about half that of the green transmission filter G22 by providing the light shielding portion 204. This corresponds to the visibility of human eyes with respect to red and blue being about half that of green.

Since the CIGS imaging sensor of the present invention has high sensitivity in the visible light region as shown in FIG. 3, it can sufficiently cope with reducing the light receiving areas of the blue transmission filter B12 and the red transmission filter R21 as described above. In addition, since the light receiving area of each pixel is changed by the light shielding portion, finer adjustments can be made as compared with approximation to human visual sensitivity by the ratio of the number of pixels as in the Bayer array. Accordingly, the light receiving area ratio of the blue transmission filter B12 and the red transmission filter R21 can be changed.

FIG. 9 is a schematic cross-sectional view of a CIGS sensor that employs a second example of the filter arrangement of FIG. As shown in FIG. 9A, the CIGS imaging sensor of the present invention has a structure in which a CIGS thin film 402 is laminated on an LSI 400, and the aperture ratio for one pixel is very large. A color filter 404 is placed thereon. The basic structure itself in the schematic cross-sectional view of FIG. 9A is not limited to the second example of the filter array, and is common to the CIGS sensor of the present invention.

FIG. 9B is an enlarged schematic cross-sectional view of the portion 406 of FIG. 9A, and conceptually shows a cross section of the second example of the filter array of FIG. In FIG. 8 and FIG. 9 (A), the same numbers are assigned to corresponding parts. As is clear from FIG. 9B, the CIGS thin film 402 is divided into photodiodes 408, 410, etc. that form pixels, and an infrared light transmission filter IR11 is placed on the photodiode 408. Yes. On the photodiode 410, a light shielding portion 202 and a blue transmission filter B12 for reducing the light receiving area are placed.

FIG. 10 is a third example of the filter array of the CIGS imaging sensor used in the live view sensor 40 in the first embodiment of FIG. 2 or the live view sensor 102 in the second embodiment of FIG. In this example, a quarter of the total number of green transmission filters in the Bayer array is regularly replaced with infrared light transmission filters IR11, IR33, IR51, and the like. The remaining three quarters are green transmission filters G13, G22, G31, G42, G44, G35, G24, etc., as in the Bayer array.

As a result, the ratio of the total number of green transmission filters G13 and the like is 1.5 times the ratio of the total number of red transmission filters R23 and blue transmission filters B32. As a result, the number of pixels corresponding to the green transmission filter is increased in the same manner as the Bayer array, so that the light receiving area of the green transmission filter is increased to approximate the visual sensitivity of the human eye. Also in the filter arrangement of FIG. 10, taking into consideration the concept of the filter arrangement of FIG. 8, a light-shielding portion is provided in the red transmission filter R23 and the blue transmission filter B32 to reduce the light receiving area, thereby improving the visibility. It is also possible to adjust the light receiving area for approximation.

On the other hand, since the infrared light transmission filters IR11 and the like are arranged as described above, the arrangement is sparse and the ratio of the total number is half of the ratio of the total number of the red transmission filter R23 and the blue transmission filter B32. . Since the CIGS imaging sensor of the present invention has high sensitivity in the infrared light region as shown in FIG. 3, it can sufficiently cope with a small proportion of the total number of pixels, and the infrared light has a long wavelength, so the pixel arrangement is visible light. Even if it is sparse compared to, it is possible to cope.

Next, interpolation of an infrared light image relating to a pixel in which an infrared light transmission filter is not arranged in the filter array of FIG. 10 will be described. First, with respect to the pixel corresponding to the green transmission filter G35, the pixel data corresponding to the infrared light transmission filter IR15 which is two above and the pixel data corresponding to the infrared light transmission filter IR33 which are two to the left. The interpolation is performed by the average value of the pixel data corresponding to the two infrared light transmission filters IR55 below and the pixel data corresponding to the two infrared light transmission filters IR37 on the right. For the pixel corresponding to the green transmission filter G24, the average value of the pixel data corresponding to the infrared light transmission filter IR15 at the upper right and the pixel data corresponding to the infrared light transmission filter IR33 at the lower left. Is interpolated. Further, for the pixel corresponding to the green transmission filter G26, the average value of the pixel data corresponding to the infrared light transmission filter IR15 in the upper left and the pixel data corresponding to the infrared light transmission filter IR37 in the lower right is determined. Interpolation is performed.

The pixels corresponding to the red transmission filter R25 correspond to the infrared light image data and the infrared light transmission filter IR15 corresponding to the green transmission filters G35, G24, and G26 obtained by interpolation as described above. Interpolation is based on the average value of pixel data. When this is arranged, the weighted average of infrared light image data corresponding to IR15, IR33, IR55, and IR37 is obtained as follows.
{(IR15 + IR33 + IR55 + IR37) / 4 + (IR15 + IR33) / 2
+ (IR15 + IR37) / 2 + IR15} / 4
= (9IR15 + 3IR33 + IR55 + 3IR37) / 16
In the same manner, the infrared light image data of the pixels corresponding to each visible light filter is complemented.

For the interpolation of the green image for the pixels where the green transmission filter is not arranged, first, the average of the pixels corresponding to the green transmission filters G22, G42, G44, and G24 around the image corresponding to the infrared light transmission filter IR33 is calculated. Interpolate. Then, the average of the green image data of the image corresponding to the infrared light transmission filter IR33 obtained by interpolation as described above and the green image data of the green transmission filters G22, G31, and G42 is taken and is at the center thereof. The image corresponding to the red transmission filter B32 also interpolates the green image data. When this is arranged, the weighted average of G22, G31, G42, G44 and G24 is obtained as follows.
{(G22 + G42 + G44 + G24) / 4 + G22 + G31 + G42} / 4
= (5G22 + 4G31 + 5G42 + G44 + G24) / 16
In the same manner, the green image data of the pixels corresponding to the infrared light transmission filter, the red transmission filter, and the blue transmission filter are complemented. Note that the interpolation of the red image and the blue image is the same as in FIG.

When interpolation is repeated when the arrangement is sparse like the above red transmission filter, or when interpolation is performed using asymmetric arrangement data such as the above green transmission filter, as described above Interpolation is further performed using data created by interpolation, and an image different from the actual one may be obtained. Even in such a case, in addition to the interpolation using only the pixel data corresponding to the infrared light transmission filter as described above, or the interpolation using only the image data corresponding to the green transmission filter, as described in FIG. It is effective to perform interpolation in consideration of data of other colors affecting the pixel to be interpolated.

FIG. 11 is a flowchart showing the operation of the camera control unit 116 when the image of the live view sensor 102 is recorded in the image storage unit 26 in the second embodiment. When the flow starts by performing an operation of selecting this function by the operation unit 15, first, the movable total reflection mirror is lowered to the position 106a in step S82, and all the light from the pentaprism 104 is directed to the live view sensor 102. The optical path switching is instructed to reflect. In step S84, the mirror 8 is fixed at the observation position so that it does not move up to the photographing position even when the release is performed. In step S86, display on the display unit 32 is performed.

Next, in step S88, it is checked whether or not the infrared light mode is selected, and if it is applicable, the process proceeds to step S90, and an instruction to set the visible light cut filter 76 for AF in the optical path to the CIGSAF sensor 70 is given. 78. In step S92, the mirror / filter driving unit 114 is instructed to set the live view visible light cut filter 112 as the optical path to the live view sensor 102, and the process proceeds to step S94.

On the other hand, if it is detected in step S88 that the infrared light mode is not selected, the process proceeds to step S96, and an instruction to set the infrared light cut filter 68 for AF to the optical path to the CIGSAF sensor 70 is given by the mirror / filter. This is performed for the drive unit 78. In step S98, it is checked whether the “image fusion mode” is selected. Although it is known to analyze plant vegetation and detect pest damage by fusing infrared and visible light images of the same subject, the “image fusion mode” is applied to the same subject almost simultaneously. On the other hand, it is possible to obtain an infrared light image and a visible light image, and an infrared light image and a visible light image that are not shifted from each other can be obtained even for a moving subject.

If the setting to the image fusion mode is not detected in step S98, it means that the visible light mode is selected, so the process proceeds to step S100, and the live view infrared light cut filter 110 is connected to the live view sensor 102. An instruction to set the optical path is given to the mirror / filter driving unit 114, and the process proceeds to step S94.

On the other hand, when the setting to the image fusion mode is detected in step S98, the process proceeds to step S102, and an instruction to remove both the infrared light cut filter 110 and the visible light cut filter 112 from the optical path to the live view sensor 102 is given. Is performed on the mirror / filter driving unit 114. This is because the live view sensor 102 acquires both the infrared light image and the visible light image almost simultaneously.

Next, in step S104, an instruction to prohibit live view display by the display unit 32 is issued, and the process proceeds to step S94. This is because the imaging positions of visible light and infrared light by the interchangeable lens 4 are different, so that when the infrared light image and the visible light image are displayed as they are on the display unit 32, the image that does not match the focused image overlaps, making it difficult to see. Because it becomes. In step S104, instead of completely prohibiting display by the display unit 32, only the pixel information of the infrared light image and the visible light image that are in focus (usually the visible light image) is extracted. May be displayed. In this case, because the filter is removed, light of image information that is not in focus is incident on these pixels, but display is possible because the light intensity of the image information that is in focus is dominant. is there. Further, it may be possible to select in advance whether to prohibit the entire display as described above or display only pixel information for an image that is not in focus, and to issue an instruction of the one selected in step S104.

In step S94, it is checked whether or not a release operation has been performed. If there is no release operation, the process returns to step S86, and thereafter, steps S86 to S104 are repeated until a release operation is detected. This corresponds to mode switching by the operation unit 15. As described with reference to FIG. 4, a delay period is provided for the mechanical execution of “instruction”. Therefore, in FIG. 11 as well, steps S86 to S104 are repeated unless the mode is actually switched. In this case, driving of the filter does not occur. The same applies to the display and prohibition switching of the display unit 32 in step S86 and step S104.

In step S94, when a release operation is detected, the process proceeds to step S106, where AF driving of the lens optical system 6 is performed, and when the pin is aligned, a live view recording process of step S108 is performed. This AF drive may be for an infrared light image or a visible light image. Next, in step S110, it is detected whether the image fusion mode is set. If not, the flow is immediately terminated.

On the other hand, when the setting of the image fusion mode is detected in step S110, the process proceeds to step S112, and the AF control unit 36 drives the lens optical system 6 for infrared light correction. In other words, when the image fusion mode is set, the AF driving in step S106 and the live view recording process in step S108 are for the visible light image, but in step S112, the infrared light is shifted from the focus position for visible light. The lens optical system 6 is caused to perform predetermined correction driving up to the focus position for the AF driving unit 38. This infrared light correction drive is performed in a very short time, and immediately enters the live view infrared light recording process of step S114. Then, when the process is completed, the flow ends. Although details of the above functions will be described later, basically, in the image fusion mode, a visible light image is recorded in step S108, and subsequently, an infrared light image is recorded almost simultaneously in step S114. It is.

FIG. 12 is a flowchart showing details of the live view recording process in step S108 in FIG. 11 and the live view infrared light recording process in step S114, which can be used in common for both. When the flow starts, it is checked in step S122 whether or not the infrared light photographing mode is set. If not, the process proceeds to step S124, and pixel information of the visible light image is obtained by reading out RGB pixel data. In step S126, RGB interpolation processing is performed. Next, in step S128, it is checked whether or not the image fusion mode is set. If not, the process proceeds to step S130 to perform image processing on the RGB visible light color image. In step S132, the image is recorded in the image recording unit 26, and the flow ends.

On the other hand, when the setting to the infrared light mode is detected in step S122, the process proceeds to step S134, and the pixel information of the infrared light image is obtained by reading the IR pixel data. In step S136, IR interpolation processing is performed. Next, in step S138, it is checked whether or not the image fusion mode is set. In this case, since the infrared light mode is not applicable, the process proceeds to step S140, and image processing is performed on the infrared light image. In step S132, the image is recorded in the image recording unit 26, and the flow ends.

Further, when the setting to the image fusion mode is detected in step S128, the process proceeds to step S134 to read out IR pixel data to obtain pixel information of the infrared light image. In step S136, IR interpolation processing is performed. By these, in addition to visible light image information in steps S124 and 126, infrared light image information can also be obtained. Next, in step S138, it is checked whether or not the image fusion mode is set. In this case, since it is the image fusion mode, the process proceeds to step S142, and the infrared light image obtained by the processing in steps S134 and S136 is red. It is checked whether it is after external light correction driving.

Here, the meaning of whether or not the step corresponds to step S142 will be supplemented. First, when it does not correspond to step S142, it corresponds to the case where the flow of FIG. 12 is performed by step S108 of FIG. The visible light image obtained in step S124 and step S126 is in focus by AF driving for visible light in step S106 in FIG. 11, and the infrared light image obtained in step S134 and step S136 is out of focus. It has become a thing. On the other hand, the case corresponding to step S142 corresponds to the case where the flow of FIG. 12 is executed in step S114 of FIG. The infrared light images obtained in steps S134 and S136 are in focus by the infrared light correction drive in step S112 in FIG. 11, and the visible light images obtained in steps S124 and S126 are out of focus. It has become a thing.

Therefore, if it is determined in step S142 that the obtained image does not correspond to the image after infrared light correction driving, the process proceeds to step S144, and image processing is performed on the focused RGB visible light image. Next, in step S146, auxiliary infrared image processing is performed on the in-focus infrared light image. In step S148, an image obtained by correcting the visible light image with the infrared light image information is created. This image is basically a visible light image, but it improves the image quality of the visible light image by the low-pass filter effect, etc., by adding the information of the infrared light image that is out of focus and defocused. is there. Through the above processing, the process proceeds to step S132, the images obtained in step S144, step S146, and step S148 are recorded, and the flow ends.

On the other hand, if it is determined in step S142 that the obtained image corresponds to an image after infrared light correction driving, the process proceeds to step S150, and image processing is performed on the focused infrared light image. Next, in step S152, image processing is performed supplementarily for the visible RGB light image that is not in focus. In step S156, an image obtained by correcting the infrared light image with the visible light image information is created. This image is basically an infrared light image, but it improves the image quality of the infrared light image by the low-pass filter effect, etc. by adding the information of the visible light image that is out of focus and defocused. It is.

Further, in step S156, the pre-infrared light correction driving image recorded in step S108 of FIG. 11 is read. As a result, a focused infrared light image, a focused visible light image, an out-of-focus infrared light image, an out-of-focus visible light image, a corrected infrared light image, and a corrected visible light image are aligned. In the next step S158, image fusion processing based on these images is performed.

The content of the image fusion processing in step S158 is basically the fusion of the focused infrared light image and the focused visible light image, thereby enabling image diagnosis that cannot be determined by one alone. In place of this, image diagnosis by fusing a corrected infrared light image and a corrected visible light image is also possible. Furthermore, a soft-focused visible light image with a tight core can be obtained by fusing a focused visible light image and an out-of-focus visible light image. Similarly, a soft focus infrared light image can be obtained from an in-focus infrared light image and an out-of-focus infrared light image. Furthermore, by correcting the focused visible light image with the focused infrared light image, an image according to the visible light image obtained by applying the infrared light cut filter can be obtained. Conversely, by correcting the focused infrared light image with the focused visible light image, it is also possible to obtain an image according to the infrared light image obtained by applying the visible light cut filter. Which of these processes is selected can be set by the operation unit 15, but can be automatically selected according to the subject.

When the process of step S158 as described above is completed, the process proceeds to step S132, the image obtained by the process is recorded, and the flow ends. Note that the image to be recorded in step S132 may be all the images processed in step S158, or may be an image selected in the process of step S158 and finally determined to be necessary. This selection can also be set by the operation unit 15, but it can be configured to automatically select in step S158 according to the processing result.

The application target of the flowchart relating to the image recording function of FIG. 11 and FIG. 12 is not limited to the image recording of the live view sensor 102 of the single-lens reflex camera as in the second embodiment. For example, the image recording to the CIGS sensor The present invention can also be applied to a dedicated visible / infrared light image recording camera. Such a visible light / infrared light image recording camera removes the optical finder system such as the focusing screen 10, the pentaprism 104, and the eyepiece 14 in FIG. 5 and replaces the imaging unit 20 with a CIGS imaging sensor. Can be configured. At that time, an infrared light cut filter 110 and a visible light cut filter 112 are provided so that they can be taken in and out of an optical path that goes straight from the lens optical system 6 toward the CIGS image sensor. Further, instead of the movable mirror 8, a fixed half mirror is provided in which most of the incident light travels straight as a transmission component and the reflected light travels to the lower photometry / AF sensor 34.

FIG. 13 is a block diagram of a third embodiment of the present invention, which constitutes an endoscope system. The endoscope system includes a capsule endoscope 502 that is swallowed into the body, images the inside of the digestive organs, and transmits image data to the outside of the body, and an extracorporeal monitor 504 that receives and monitors the image data transmitted outside the body. The capsule endoscope 502 has a sealed structure having a transparent protective window 506, and an image inside the digestive organ formed by the imaging lens 508 through the protective window 506 is captured by the CIGS imaging sensor 510. The CIGS image sensor 510 has the spectral sensitivity as described with reference to FIG. 3 and can capture an image in the visible light range with high sensitivity and can also perform image capture with high sensitivity even in infrared light. The angle of view and the focus position of the imaging lens 508 can be adjusted by the lens driving unit 512.

The CIGS sensor 510 of the third embodiment does not have a color filter as shown in FIGS. 7 to 10, and a wide range of light ranging from the visible light region to the infrared light region can be incident on all pixels. That is, the light decomposition in the imaging in the third embodiment is not performed by the color filter on the light receiving side, but is performed by switching the light on the light source side. Specifically, red, green, blue, and infrared light emitting diodes (hereinafter referred to as “LEDs” as appropriate) are used as light sources, and these emit light sequentially in a time-sharing manner, whereby CIGS sensor 510 imaging output at each light emission timing Is the image data for each color.

A large number of LEDs are concentrically provided around the optical axis of the imaging lens 508. In FIG. 13, for the sake of simplicity, one green LED 514 and one infrared LED 516 are shown as an example. For example, the imaging output of the CIGS imaging sensor 510 when the green LED 514 emits light becomes green image data, and the imaging output of the CIGS imaging sensor 510 when the infrared LED 516 emits light becomes infrared image data. In addition, since there is a deviation in the imaging position between visible light and infrared light, the lens driving unit 512 adjusts the imaging position as necessary. The third embodiment is an endoscope, and the inside of the body to be photographed is sufficiently dark, so that the light can be decomposed by time division of the light source light in this way. The relationship between the light source, imaging, imaging lens, etc. will be described in detail later.

The LED driver 518 controls the lighting timing of the LEDs 514 and 516 based on an instruction from the endoscope control unit 520. The endoscope control unit 520 controls the entire capsule endoscope 502, and its function follows a program stored in the storage unit 522. The storage unit 522 further temporarily stores data necessary for the function of the endoscope control unit 520 as needed.

The sensor driver 524 controls the CIGS imaging sensor 510 based on an instruction from the endoscope control unit 520 and stores each color-specific image RAW data from the CIGS imaging sensor in the image buffer 526. The image buffer 526 can store the image RAW data for each color for a predetermined number of times, and the wireless communication unit 528 extracts the image RAW data for each color in the image buffer 526 using a FIFO and transmits it from the antenna 530 to the outside of the body. The battery 532 is configured by a button battery or the like, and supplies power to the entire capsule endoscope 502.

The extracorporeal monitor 504 includes a wireless communication unit 534, receives each color image RAW data transmitted from the capsule endoscope 502 by the antenna 536, and stores the received image RAW data in the image buffer 538. These functions are controlled by the monitor control unit 540. The monitor control unit 540 controls the entire external monitor 504 according to the program stored in the storage unit 542. The storage unit 542 further temporarily stores data and the like necessary for the function of the monitor control unit 540 as necessary.

The image processing unit 544 performs image processing on each color-specific RAW data stored in the image buffer 548 based on an instruction from the monitor control unit 540 into an image signal, and outputs a color image signal from the red image signal, the green image signal, and the blue image signal. Is stored in the recorder 546. The infrared image signal is also recorded in the recorder 546. The recorded data can be monitored by the display unit 548 as appropriate. Further, the color image signal or infrared image signal from the image processing unit can be directly monitored by the display unit 548 in real time.

FIG. 14 is a front view showing a first example of an LED arrangement that can be employed in the capsule endoscope 502 of the third embodiment. Parts corresponding to those in FIG. As is apparent from FIG. 14, four green LEDs 514 are provided around the imaging lens 508 inside the transparent protective window 506 and are to be rotated 90 degrees apart from each other. A line 550 connecting them is a square. In addition, four infrared LEDs 516 are provided at the apex portion of the square 552 rotated by 45 degrees from the green LED 514 to be rotated objects that are separated from each other by 90 degrees. Further, four red LEDs 556 are provided at the apex portion of the vertically long rectangle 554, and four blue LEDs 560 are provided at the apex portion of the horizontally long rectangle 558. As a result, the red, green, and blue LEDs are arranged in line symmetry in both the vertical and horizontal directions as shown in FIG. 14, and the symmetry of illumination is maintained in both the vertical and horizontal directions for each color. Be drunk.

FIG. 15 is a front view showing a second example of the LED arrangement that can be employed in the capsule endoscope 502 of the third embodiment. Also in FIG. 15, the same reference numerals are given to the portions corresponding to those in FIG. The arrangement of the green LED 514 and the infrared LED 516 in FIG. 15 is the same as that in FIG. On the other hand, four red LEDs 562 are provided on the object to be rotated 90 degrees apart from each other at the apex of a square 564 rotated 22.5 degrees to the left from the green LED 514. In addition, four blue diodes 566 are provided on the top of a square 568 that is rotated 22.5 degrees to the right from the green LED 514 on a rotation target that is 90 degrees apart from each other. As a result, the red, green, and blue LEDs are densely arranged in the four directions, up, down, left, and right, respectively, as seen in FIG. Color unevenness is reduced. Further, since each color is arranged at the apex of the square, the rotation targets are arranged around the optical axis of the imaging lens 508.

FIG. 16 is a front view showing a third example of the LED arrangement that can be employed in the capsule endoscope 502 of the third embodiment. Also in FIG. 16, the same reference numerals are given to the portions corresponding to FIG. The arrangement of the green LED 514 and the infrared LED 516 in FIG. 16 is the same as that in FIG. In contrast, four red LEDs 572 are provided at the apex portion of the vertical rectangle 570 inclined 45 degrees to the left, and four blue LEDs 576 are provided at the apex portion of the vertical rectangle 574 inclined 45 degrees to the right. ing. As a result, the red, green, and blue LEDs are arranged in line symmetry with respect to both the 45 ° left line and the right 45 ° line connecting the opposing infrared LEDs as seen in FIG. The illumination symmetry is maintained in any of these directions. Further, the red, green, and blue LEDs are densely arranged in the four directions of top, bottom, left, and right as seen in FIG. Unevenness is reduced.

As described above, in the LED arrangement examples shown in FIGS. 14 to 16, the green LEDs are arranged vertically and horizontally and the infrared LEDs are arranged at a position rotated 45 degrees from now on. The arrangement is not limited to this, and the entire arrangement may be appropriately rotated in relation to the grid direction of the pixel arrangement of the CIGS image sensor. For example, in the LED arrangement examples shown in FIGS. 14 to 16, the green LEDs are arranged vertically and horizontally with reference to the vertical and horizontal directions of the pixel arrangement of the CIGS imaging sensor. The infrared LED may be arranged vertically and horizontally according to the grid direction of the pixel arrangement by rotating. In this case, the green LED is arranged at a position rotated 45 degrees from now.

FIG. 17 is a front view showing a fourth example of the LED arrangement that can be employed in the capsule endoscope 502 of the third embodiment. When red, green, blue, and infrared LEDs are arranged around the optical axis of the imaging lens 508, the number of LEDs of all colors is not limited to the same number. FIG. 17 employs four green LEDs 514, two red LEDs 578, and two blue LEDs 580 as examples of such. The reason why the number of green LEDs 514 is twice that of red LEDs 578 and blue LEDs 580 is to relatively increase the amount of emitted green light to match the visibility. In FIG. 17, eight infrared LEDs 582 are arranged to increase the amount of infrared light, and the in-vivo observation ability with infrared light is enhanced.

In addition, in FIG. 14 to FIG. 17, a total of 16 LEDs are employed, but the present invention is not limited to this. If arrangement is possible, the total number of LEDs can be further increased to reduce illumination unevenness. Moreover, in order to maintain the minimum symmetry of illumination, it is also possible to adopt a pair of LEDs for each color and to employ a total of 8 LEDs to simplify the configuration. In this case, the line connecting the pair of green LEDs and the line connecting the pair of infrared LEDs are crossed at 90 degrees, and for the pair of red LEDs and the pair of blue LEDs, the line connecting them connects the pair of green LEDs. It is desirable that the red and blue LEDs be adjacent to both sides of the green LED as an arrangement in a state rotated 45 degrees to the left and right of the line. If the arrangement space permits, instead of arranging the eight LEDs at regular intervals in this way, the red and blue LEDs are brought into close contact with both sides of the green LED, and the red, green and blue LEDs are mutually connected. An arrangement in which the positional deviation is minimized is also possible.

FIG. 18 shows the light emission timing of each color LED, the operation timing of the photoelectric conversion unit, the operation timing of the AD trunk, and the operation timing of the wireless communication unit in the color / infrared mode of operation in the capsule endoscope 502 of the third embodiment. It is a timing chart which shows the relationship. In the color / infrared mode, a visible color image and an infrared image are acquired in parallel. As is clear from FIG. 18, all red LEDs overlap each other at the timing from t1 to t2, all green LEDs from t3 to t4, all blue LEDs from t5 to t6, and all infrared LEDs from t7 to t8. It lights up in time division without any. When the four color LEDs are turned on, all red LEDs are turned on again at the timing from t9 to t10, and the green, blue, and infrared LEDs are turned on in the same manner, and time-division lighting is performed in the same manner. repeat. The time from t1 to t8 is about one frame time of a normal color moving image, and the light emission amount of each color is less than a quarter of the time without time division, but the CIGS sensor is a normal CMOS as shown in FIG. Compared with the sensor, it has high sensitivity and broadband characteristics, so that even a short amount of light emission provides sufficient light source light.

As shown in FIG. 18, in the color / infrared mode, when visible light and infrared light are emitted almost simultaneously in a time division manner, the angle of view of the imaging lens 508 in FIG. The depth of focus is set to be deep, and the focus position is also set to be in a pan-focus state that covers visible light to infrared light under the control of the lens driving unit 512. As described above, the color / infrared mode is suitable for observing the overall state of the body roughly.

As is clear from the timing chart of the photoelectric conversion unit in FIG. 18, the photoelectric conversion unit starts red exposure immediately after the start of light emission of the red LED and accumulates charges. Since the charge accumulation time is set immediately before the end of light emission of the red LED, the exposure ends here and the charge is read out. Further, when the charge reading is completed, the residual charge is swept away. When the charge sweeping is completed, the next green exposure is started. As is apparent from FIG. 18, light emission of the green diode starts immediately before the green exposure. Also for the green exposure, after the charge accumulation time is over, the charge read-out and the residual charge sweep out follow. Similarly, blue and infrared charge accumulation, charge readout, and residual charge sweeping are performed in synchronization with the light emission of the blue LED and the light emission of the infrared LED, respectively. These operations circulate. In the above description, the function of the photoelectric conversion unit has been described for each color. However, the photoelectric conversion unit itself does not have a function of performing photoelectric conversion by separating each color, and the photoelectric conversion unit itself simply performs charge accumulation, charge read, and It only repeats the same operation of residual charge sweeping. The read charge amount has information on each color depending solely on the light source color at the time of charge accumulation.

As is clear from the timing chart of the AD conversion unit in FIG. 18, the photoelectric conversion unit starts AD conversion immediately after the charge reading of each color. For example, red AD conversion is started immediately after completion of red charge reading. Then, using the time zone during the next green exposure, the red AD conversion is continued in parallel with this. As is clear from the timing chart (A) of the wireless communication unit in FIG. 18, the wireless communication unit can start communication of digital signals of the resulting color immediately after the completion of photoelectric conversion of each color. For example, communication of a red digital signal is started immediately after the end of red AD conversion. The next green AD conversion time zone is also used, and in parallel with this, red communication is continued. Similarly, AD conversion and communication are performed for green, blue, and infrared.

Regarding communication, depending on the relationship between the capsule endoscope 502 and the extracorporeal monitor 504, there may be a case where the communication cannot be performed successfully immediately after AD conversion. In such a case, communication is executed at the timing when the communication environment becomes sufficient as shown in the timing chart (B) of the wireless communication unit in FIG. For example, IR data transmission 592 is executed later than the timing chart (A), and is executed immediately before the next R data. The G data transmission 594 and the B data transmission 596 are also executed with a delay. However, the adjustment of the communication time may cause the FIFO to fail because the capacity of the image buffer 526 in FIG. 13 is full. It is possible as long as there is not.

FIG. 19 is a timing chart showing the relationship between the light emission timing of each color LED and the color image creation in the operation of the capsule endoscope 502 of the third embodiment in the color / infrared mode shown in FIG. As shown in FIG. 19, a red image based on red LED emission starting at t1, a green image based on green LED emission starting at t3, and a blue image based on blue LED emission starting at t5. , F1 is displayed as a color image. Strictly speaking, since there is a time difference in light emission of each color, the images of each color are not the same time, but since the time difference is small, there is no problem in acquiring each color image by such time division unless the subject moves at high speed. Similarly, a red image based on red LED emission starting at t9, a green image based on green LED emission starting at t11, and a blue image based on blue LED emission starting at t13, at F2. A one-frame color image shown is created. In the same manner, a color image of one frame is created, and each color moving image can be recorded as a still image, and can also be recorded as a color moving image by connecting them. Note that these color processes are performed by the image processing unit 544 of the extracorporeal monitor 504 shown in FIG. In addition, as shown in the timing chart (B) of the wireless communication unit in FIG. 18, the reception of each color data by the extracorporeal monitor 504 is not necessarily at regular intervals, but the image acquisition timing is determined by the light emission timing of each color LED. A relationship is established.

In addition, as shown in FIG. 19, based on the green image based on the green LED emission starting at t3, the blue image based on the blue LED emission starting at t5, and the red LED emission starting at t9. Since the red image also contains RGB three-color data, a one-frame color interpolated image indicated by I1 is created. Similarly, with a blue image based on blue LED emission starting at t5, a red image based on red LED emission starting at t9, and a green image based on green LED emission starting at t11, I2 A one-frame color interpolated image shown is created. In these interpolated images, the emission of infrared LEDs is intervened until the RGB colors are aligned, and the time until RGB is aligned becomes slightly longer, and the RGB emissions are not equally spaced, so the image quality of the color image is poor. . Therefore, it is adopted as an interpolation image for obtaining a smooth moving image.

On the other hand, as shown in FIG. 19, for the infrared image, an image IR1 based on the emission of the infrared LED starting at t7, an image IR2 based on the emission of the infrared LED starting at t15, and the like are still images. It can be recorded, and can also be recorded as a color video by connecting them. In color / infrared mode, color images and infrared images can be acquired in parallel as described above, so both images can be taken in parallel for endoscopic diagnosis, and both images can be combined. It becomes possible. Also, when a quantity image is synthesized as a still image, the acquisition time of the infrared image is included in the acquisition time zone of the color interpolation image, so the color interpolation image is used for the synthesis with the infrared image. It is also possible. Specifically, since the acquisition time zones of the color interpolation images I2 and I2 both include the acquisition time of the infrared image IR1, it is possible to synthesize the color interpolation images I1 and I2 or their average with the infrared image IR1. Is possible.

FIG. 20 shows the relationship between the light emission timing of each color LED, the operation timing of the photoelectric conversion unit, the operation timing of the AD trunk and the operation timing of the wireless communication unit in the fine color mode in the capsule endoscope 502 of the third embodiment. It is a timing chart which shows. In the fine color mode, only a visible color image is acquired, and the infrared LED does not emit light. As is clear from FIG. 20, all red LEDs are lit in a time-division manner without overlapping each other at the timings t1 to t2, all green LEDs are t3 to t4, and all blue LEDs are t5 to t6. Then, when the lighting of the RGB three-color LEDs is completed, all the red LEDs are turned on again at the timing from t7 to t8. Thereafter, the green, blue, and LEDs are turned on in the same manner, and time-division lighting is repeated in the same manner. In this case, the time from t1 to t6 required for one cycle is shorter than t1 to t8 in FIG. The video becomes fine. The timing chart of the wireless communication unit (B) in FIG. 20 shows a state in which communication is continuously performed after the communication environment has not been prepared for a while.

As shown in FIG. 20, in the fine color mode, when only visible light is emitted almost simultaneously in a time division manner, the angle of view of the imaging lens 508 in FIG. The focus position is also set so that the focus of visible light is imaged on the imaging surface under the control of the lens driving unit 512. This is because red, green, and blue focus position shifts are small and can be handled by aberration correction in the design of the imaging lens, so that optimum focus position adjustment is possible. Thus, the fine color mode is suitable for observing the state of the body in detail with high definition.

FIG. 21 is a timing chart showing the relationship between the light emission timing of each color LED and the color image creation in the operation of the capsule endoscope 502 of the third embodiment in the fine color mode shown in FIG. As shown in FIG. 21, a red image based on red LED emission starting at t1, a green image based on green LED emission starting at t3, and a blue image based on blue LED emission starting at t5. , F1 is displayed as a color image. Next, indicated by F2 by a green image based on the green LED emission starting at t3, a blue image based on the blue LED emission starting at t5, and a red image based on the red LED emission starting at t7. A color image of one frame is created. Similarly, one frame indicated by F3 by a blue image based on blue LED emission starting at t5, a red image based on red LED emission starting at t7, and a green image based on green LED emission starting at t9. A color image is created. In the same manner, a color image of one frame or less indicated by F4 is created. In this manner, in the fine color mode, a new one-frame color image is created every time the LED of each color emits light, so that smooth fine moving images can be recorded.

FIG. 22 shows the infrared LED emission timing, the photoelectric conversion unit operation timing, the AD trunk unit operation timing, and the wireless communication unit operation timing in the infrared mode of the capsule endoscope 502 of the third embodiment. It is a timing chart which shows a relationship. In the infrared color mode, only an infrared image is acquired, and LEDs other than the infrared LED do not emit light. As is clear from FIG. 22, all the infrared LEDs emit light at every exposure in the photoelectric conversion unit from t1 to t2 and from t3 to t4. In response to this, an infrared image of one frame is created every time. This makes it possible to record a smooth infrared moving image.

As shown in FIG. 22, when only infrared light is emitted in the infrared mode, the angle of view of the imaging lens 508 in FIG. 13 is set to a narrow angle (telephoto) by the control of the lens driving unit 512, and the focus position is also set. The focus of the infrared light is set to form an image on the imaging surface by the control of the lens driving unit 512. Thus, the infrared mode is also suitable for observing the state of the body in detail with high definition.

FIG. 23 is a flowchart of the operation of the endoscope control unit 520 in the third embodiment of FIG. When the battery 532 is set in the capsule endoscope 502, the flow starts. In step S162, the color / infrared mode is initialized. In response to this, in step S154, the imaging lens 508 is set to the wide-angle and pan-focus state. Next, in step S166, all red, green, blue and infrared LEDs are set to emit light sequentially in a predetermined order. In step S168, the fact that these settings have been made is externally transmitted and reported to the external monitor 504.

Next, an imaging process is performed in step S170, and a transmission process is performed in step S172. Details thereof will be described later. When the transmission process ends, the process proceeds to step S174, and it is checked whether an operation stop signal is received from the extracorporeal monitor 504. If there is reception, the flow is immediately terminated. On the other hand, if no stop signal has been received, the process proceeds to step S176 to check whether a mode change signal has been received. If there is reception, the process advances to step S178 to check whether the changed mode is the color / infrared mode. If it is in the color / infrared mode, the process returns to step S164, the lens is set to the wide angle and pan focus state, and the operation proceeds to the operation after step S166 already described.

On the other hand, if the mode changed in step S178 is not the color / infrared mode, the process proceeds to step S180 to check whether the mode is the fine color mode. In the fine color mode, in step S182, the angle of view of the lens is set to a narrow angle (telephoto) and set to the visible light focus state, and in step S184, only visible light LEDs are set to sequentially emit light with a rotation number. Then, after these setting states are transmitted to the outside in step S186, the process returns to step S170.

If the mode changed in step S180 is not the fine color mode, it means that the changed mode is the infrared mode, so that the process proceeds to step S188, and the field angle of the lens is narrowed (telephoto). And set to the infrared light focus state. Furthermore, it sets so that only infrared LED may light-emit in step S190. Then, the process proceeds to step S186, and after these setting states are transmitted to the outside, the process returns to step S170.

FIG. 24 is a flowchart showing details of the imaging process in step S170 of FIG. When the flow starts, it is checked in step S192 whether a mode has been selected or changed. If no mode has been selected or changed, the process proceeds to step S194 to check whether the exposure time has been completed. If the completion is detected, the process proceeds to step S196, where the stored charge readout start process is performed. Further, in step S198, an instruction to stop LED emission is given. Further, in step S200, it is checked whether or not the accumulated charge reading is completed. If it is not completed, the completion is waited while repeating step S200.

When the completion of reading is detected in step S200, the process proceeds to step S206, a residual charge sweeping start process is performed, and the process proceeds to the LED selection process in step S208. This is a process of selecting the next LED to emit light, the details of which will be described later. Further, in step S210, the start of light emission of the LED selected in step S208 is instructed. Next, in step S212, it is checked whether or not the stored charge has been completely purged.

When it is detected in step S212 that the accumulated charge has been completely removed, the process proceeds to step S214 to start exposure, and in step S216, the exposure time starts to be counted, and the flow ends. On the other hand, if the exposure time is not completed in step S194, the flow is immediately terminated. If selection or change of the mode is detected in step S192, the process proceeds to step S218, the imaging process is initialized, and the process proceeds to the LED selection process in step S208.

FIG. 25 is a flowchart showing details of the LED selection processing in step S208 of FIG. When the flow starts, it is checked whether or not the infrared mode is set in step S222. If not applicable, it means color / infrared mode or fine color mode. In this case, the process proceeds to step S224, and it is checked whether or not the imaging process has been initialized in step S218 of FIG. If this is not the case, the process advances to step S226 to read the previously selected LED memory. Then, based on the memory read in step S228, it is checked whether or not the last light emission was a red LED. If not, it is further checked in step S230 whether the last time it was a green LED.

If it is determined in step S230 that the previous light emission was not a green LED, the process proceeds to step S232 to check whether the color mode is the fine color mode. If this is the case, the red, green, and blue LEDs are emitting light on the wheel. In this case, if the previous light emission is neither red nor green, it means that the light is blue, so the process proceeds from step S232 to step S234, and the red LED corresponding to the next order is selected. Then, the selection result is stored in step S236, and the flow ends.

On the other hand, if the fine color mode is not detected in step S232, it means the color / infrared mode. In this case, red, green, blue, and infrared LEDs emit light in a ring. In this case, it means that if the previous light emission is neither red nor green, it means that the light was blue, so the process proceeds from step S232 to step S238, and the infrared LED corresponding to the next order is selected. Then, the selection result is stored in step S236, and the flow ends.

If it is the red LED that previously emitted light in step S228, the process proceeds to step S242, and the green LED corresponding to the next order is selected. Then, the selection result is stored in step S236, and the flow ends. Furthermore, when it is in the infrared mode in step S222, the process proceeds to step S244, and an infrared LED is selected. In the infrared mode, it is always the infrared LED that is selected, so that the selection result need not be stored and the flow is immediately terminated. If the imaging process has been initialized in step S224, the process proceeds to step S242 to select the green LED as the first light emitting LED.

FIG. 26 is a flowchart showing details of the transmission processing in step S172 of FIG. When the flow starts, it is checked in step S252 whether data is being transmitted. If transmission is not in progress, the process advances to step S254 to check whether there is data that has been successfully transmitted. If there is a corresponding item, it is erased from the image buffer in step S256, and the process proceeds to step S258. On the other hand, if there is no successful transmission data in step S254, the process directly proceeds to step S258.

In step S258, it is checked whether or not AD conversion has been completed. If applicable, the AD converted data is stored in the image buffer, and the process proceeds to step S262. On the other hand, if AD conversion is not completed, the process directly proceeds to step S262. In step S262, it is checked whether there is data stored in the image buffer. If there is data, it is checked in step S264 whether the communication state is OK. If it is OK, data is read from the image buffer by FIFO (first-in first-out) in step S266, the transmission of the read data is instructed in step S268, and the flow is terminated. It should be noted that when the data is being transmitted at step S252, when no data is stored in the image buffer at step S262, or when the communication state is not OK at step S264, the flow is immediately terminated.

FIG. 27 is a flowchart showing the operation of the monitor control unit 540 of the extracorporeal monitor 504 in the third embodiment of FIG. 13, and the flow starts when communication with the capsule endoscope 502 is started. When the flow starts, it is checked in step S272 whether there is a new arrival of image data. If there is new arrival data, the process proceeds to step S274 to check whether the new arrival data is complete. If it is complete, the process proceeds to step S276, where it is stored in the image buffer 538, and the process proceeds to step S278. If there is no new arrival of data in step S272 or new arrival data is not complete in step S274, the process returns to step S272.

In step S278, it is checked whether the infrared mode is selected. If not, the color / infrared mode is checked in step S280. If it is the color / infrared mode, the process advances to step S282 to check whether the newly arrived data is infrared image data. If not, it means that the image data is one of red, green, and blue, so the process proceeds to step S284, and an instruction to create an interpolation auxiliary image of an infrared image is given from these data, and the process proceeds to step S286. . The instruction in step S284 uses information of visible light image data obtained at a timing between infrared images as an auxiliary when performing interpolation based on infrared image data in order to obtain a smooth infrared moving image. It is an instruction for.

In step S286, it is checked whether the new inverse data is blue image data. If it does not correspond, it means red image data or green image data, so the process proceeds to step S290, and it is checked whether or not the visible two-color data immediately before the new arrival data is stored. If it is stored, red, green, and blue are available together with newly arrived data, so the process proceeds to step S292, and an instruction to create a storage color image is issued, and the process returns to step S272. The image created by the instruction in step S292 corresponds to the color interpolation image I1 or I2 in FIG.

When it is the infrared mode in step S278, the newly arrived data is infrared image data, so the process proceeds to step S294, the creation of the infrared image is instructed, and the process returns to step S272. Also, when the newly arrived data is infrared image data in step S282, the process proceeds to step S294. On the other hand, if the newly arrived data is blue image data in step S286, the process proceeds to step S296, and it is checked whether the image data of the immediately preceding two colors (in this case, red and green) are stored in the image buffer. If these are stored, three consecutive colors are prepared, so the process proceeds to step S298, where an instruction to create a color image is issued, and the process returns to step S272. The image created by this instruction corresponds to the color image F1 or F2 in FIG.

If the color / infrared mode is not set in step S280, it means that the mode is the fine color mode. Therefore, the process proceeds to step S296, where it is checked whether the previous visible two-color data is stored. If there are two colors, an instruction to create a color image in step S298 is given. Images created by this instruction correspond to the color images F1, F2, F3, etc. in FIG. If no previous visible two-color data is stored in step S290 or step S296, the process immediately returns to step S272.

FIG. 28 is a timing chart showing the relationship of the operation timing of the fourth embodiment of the present invention. Since the fourth embodiment is basically configured in common with the endoscope system shown in FIGS. 13 to 17, the following description will be made using the reference numerals in the block diagram of FIG. 13 as appropriate. The fourth embodiment differs from the third embodiment in the configuration of the CIGS imaging sensor and the light emission timing of the LEDs. That is, the CIGS sensor 510 of the third embodiment does not have a color filter, and the color separation is based on the time-division light emission of the LED, but the CIGS imaging sensor 510 of the fourth embodiment is as shown in FIGS. The CIGS imaging sensor itself performs color separation in the same manner as in the first and second embodiments. The light emission of the LEDs is not performed in a time division manner, but is performed simultaneously for all colors.

FIG. 28 shows the LED light emission timing, the photoelectric conversion unit operation timing, the AD trunk unit operation timing, and the wireless communication unit operation in the color / infrared mode in the capsule endoscope 502 of the fourth embodiment. It is a timing chart which shows the relationship of an operation timing. As described above, all red LEDs, all green LEDs, all blue LEDs, and all infrared LEDs all emit light simultaneously at the exposure timing of the photoelectric conversion unit. Each LED may emit light continuously instead of blinking as shown in FIG. In the color / infrared mode of FIG. 28, the angle of view of the imaging lens 508 is set to a wide angle by the control of the lens driving unit 512, and the focal depth is set to be deep, and the focus position is also lens driven. Under the control of the unit 512, the pan focus state is set so as to cover visible light to infrared light. Such imaging lens control is common to the color / infrared mode of FIG.

FIG. 29 shows the relationship among the LED light emission timing, the photoelectric conversion unit operation timing, the AD trunk unit operation timing, and the wireless communication unit operation timing in the fine color mode operation in the capsule endoscope 502 of the fourth embodiment. It is a timing chart which shows. As is clear from FIG. 29, all red LEDs, all green LEDs, and all blue LEDs all emit light simultaneously at the exposure timing of the photoelectric conversion unit. The infrared LED does not emit light. As shown in FIG. 29, when only visible light is emitted simultaneously in the fine color mode, the field angle of the imaging lens 508 in FIG. 13 is set to a narrow angle (telephoto) under the control of the lens driving unit 512, and the focus position is also the lens. The focus of visible light is set to form an image on the imaging surface under the control of the driving unit 512. Such imaging lens control is common to the fine color mode in FIG.

The infrared mode timing chart in the fourth embodiment is the same as that in FIG. 22 in the third embodiment. In addition, the angle of view of the imaging lens 508 in the infrared mode is set to a narrow angle (telephoto) by the control of the lens driving unit 512, and the focus position of the imaging lens 508 is also controlled by the lens driving unit 512 to focus the infrared light on the imaging surface. The point set to image is also common to the infrared mode in FIG. The color / infrared mode in the fourth embodiment is suitable for observing the overall state of the body roughly, while the fine color mode and infrared mode are suitable for observing the state of the body in detail with high definition. This is also the same as the third embodiment.

In addition, although the endoscope system in the said 3rd Example and 4th Example was comprised as what has a capsule endoscope and an external monitor, implementation of this invention is not restricted to this. For example, it can be configured as a normal endoscope in which the inside and outside of the body are connected by a tube. In this case, wireless communication by the antennas 530 and 546 in FIG. 13 is wired communication using a cable in the tube, and a known vent pipe, water conduit, tube bending mechanism, and the like are provided in the tube. Further, instead of performing image information transmission between the inside and outside of the body using an electrical signal, an image acquired inside the body may be taken out of the body by optical means such as a fiber. In this case, the CIGS image sensor is provided outside the body. Further, the light source can also be provided outside the body and guided to the inside by a light guide. In such a configuration, the light source arrangement in FIGS. 14 to 17 is understood not as the arrangement of the light emitting portion but as the arrangement of the light source light emitting portions. Furthermore, when using a light guide, it is not always necessary to provide a light source light emitting portion for each color, and light from each color light emitting portion is guided into the body using a common light guide and is emitted from a common outlet. Also good. In addition, the implementation of the various features of the present invention shown in the third and fourth embodiments is not limited to an endoscope, but may be used as appropriate for imaging / observation / recording equipment using various imaging sensors. Is also possible.

FIG. 30 is a block diagram of a fifth embodiment of the present invention, which constitutes a vehicle monitoring device. The vehicle monitoring function is a function for detecting the distance between vehicles before and after the vehicle for preventing a collision and a function for capturing and recording images before and after the vehicle as a drive recorder. The detected inter-vehicle distance is recorded in a drive recorder and also used for focus adjustment for image shooting. In FIG. 30, a vehicle 602 is configured as either a gasoline engine vehicle, an electric vehicle, or a so-called hybrid vehicle that uses a gasoline engine and a motor together.

The vehicle 602 includes a vehicle control unit 604 including a computer that controls the entire vehicle, and controls the vehicle function unit 610 having a power 606 and a brake 608 in accordance with an operation by a driver of the vehicle to cause the vehicle 602 to travel. . The power 606 is configured to include a gasoline engine and / or motor. The vehicle control unit 604 includes a storage unit that stores software and data necessary for controlling the vehicle 602. In addition, the vehicle control unit 604 controls the in-vehicle output unit 612 and causes the image output unit 614 to display a GUI display necessary for operation of the vehicle and display the control result. The voice output unit 616 also makes an announcement to the driver.

The GPS unit 618 obtains latitude, longitude, and altitude information, which are absolute position information of the vehicle 602, from the satellite and the nearest broadcasting station based on the GPS system and sends the information to the vehicle control unit 604. The car navigation function unit 620 processes absolute position information from the GPS unit 618 obtained via the vehicle control unit 604 and causes the image output unit 614 to display the position of the vehicle 602 on the map.

Next, the configuration relating to the vehicle monitoring device will be described in association with the above configuration as necessary. The fifth embodiment includes substantially the same monitoring device for monitoring the front and rear of the vehicle. First, a monitoring device behind the vehicle will be described. The rear camera / sensor 622 includes a camera and a microphone 628 including an imaging lens 624 and a CIGS imaging sensor 626. When the vehicle 602 is in a traveling state, an image such as the front of the vehicle and surrounding sounds are always input.

The CIGS imaging sensor 626 basically has the characteristics as shown in FIG. 3 described in the embodiments so far, and has the color filter array as described with reference to FIGS. However, the spectral transmittance of the color filter has the characteristics peculiar to the fifth embodiment, and the red transmission filter, the blue transmission filter, and the green transmission filter are band-pass filters that transmit light in each narrow visible light region. In addition, the infrared light transmission filter is a band-pass filter that transmits infrared light in a narrow area of the solar blind. A solar blind is a wavelength region of sunlight that reaches the earth that cannot reach the ground surface due to atmospheric absorption or is weak even if it reaches the surface. For example, it is known that such a region exists in the vicinity of 1100 nm. It has been. In the fifth embodiment, a visible light image is captured by red, blue, and green pixels and the presence or absence of sunlight by infrared pixels by combining the light receiving sensitivity of the CIGS sensor as shown in FIG. 3 and the filter as described above. There is no distance detection. Details thereof will be described later.

The infrared beam scanner 630 projects an infrared light beam in the solar blind area under the control of the monitoring recording control unit 632 and scans the rear of the vehicle. When the projected infrared light beam is reflected by the following vehicle, it is received by the infrared pixels of the CIGS imaging sensor 626. The image signal of the CIGS sensor 626 is processed by the rear image processing unit 636 via the rear sensor driver 634, and the reflection position of the infrared beam is detected. The monitoring and recording control unit 632 calculates the inter-vehicle distance to the following vehicle that reflects the infrared light beam from the reflection position information detected by the rear image processing unit 636 and the infrared light beam projection angle at that time. The monitoring recording control unit 632 controls the rear AF (autofocus) control unit 638 according to the calculated inter-vehicle distance, and drives the imaging lens 624 by the AF driving unit 640 to reflect the infrared light beam with respect to the following vehicle. Focus on visible light.

On the other hand, the visible light image information detected by the visible light pixel of the CIGS imaging sensor 626 based on the focus adjustment as described above is processed into a visible image by the rear image processing unit 636 via the rear sensor driver 634, and is monitored and recorded. The sound information from the microphone 628 is input to the FIFO (first-in first-out recording unit) 642 via the unit 632. The FIFO (first-in first-out recording unit) 642 includes a nonvolatile buffer memory having a capacity of about 20 seconds, and stores image and sound information input from the rear image processing unit 636 and the microphone 628 in a first-in first-out manner. That is, the latest 20-second information is always overwritten and stored.

Then, when the abnormal acceleration detection unit 644 detects a large acceleration change due to a collision or the like, or particularly when a manual operation for instructing recording is performed, the recording of the FIFO 642 is maintained as evidence. Specifically, the monitoring record control unit 632 is connected to the vehicle control unit 604, and abnormal acceleration detection or manual operation by the abnormal acceleration detection unit 644 is transmitted to the monitoring record control unit 632 via the vehicle control unit 604. If the rear image processing unit 636 and the microphone 628 are continuously input without being destroyed, the storage in the FIFO 0642 is continued for about 10 seconds from that point. Thereby, the FIFO 642 finally holds image and sound information of about 10 seconds before and after the time of detection of acceleration change or manual operation. Then, the image and sound information of about 10 seconds before and after this is automatically transferred to the nonvolatile recording unit 646 and stored as evidence. This achieves the drive recorder function. Even if most of the drive recorder function is destroyed due to a collision or the like, if even the non-volatile FIFO 642 is safe, the recording of about 20 seconds before the accident is maintained.

When recording the image of the following vehicle, when the inter-vehicle distance is shortened, not only the number of the following vehicle but also the face of the driver of the front seat is clearly recorded through the windshield. These images will be discarded after 20 seconds in the FIFO unless there is an abnormality. However, in order to protect privacy, the monitoring and recording control unit confirms that the following vehicle has narrowed the inter-vehicle distance more than a predetermined distance. If 632 detects, this will be sent to the vehicle control part 604. In response to this, the vehicle control unit 604 causes the rear vehicle drive recorder (drive recorder) notification display unit 648 to display a notification such as “being rear-photographed” on the rear window of the vehicle 602. This is an advance notice for alleviating troubles, and it can be expected that a subsequent vehicle that dislikes photographing will voluntarily take an appropriate inter-vehicle distance. further,

In the fifth embodiment, a monitoring device similar to the rear monitoring device described above is provided in front of the vehicle. Since the configuration is the same as that of the rear monitoring device, the corresponding configuration is numbered in the 700s with the same number at the 10's and 1's, and the description is omitted unless particularly necessary. Note that the front visible light image information from the front image processing unit 736 of the front monitoring device is recorded in the FIFO 642 and the recording unit 646 in the same manner as the rear visible light image. For this reason, the FIFO 642 and the recording unit 646 have recording areas for the rear side and the front side, respectively.

In the case of the front monitoring device, when the monitoring record control unit 632 detects that the own vehicle 602 has narrowed the inter-vehicle distance to the front vehicle more than a predetermined value and this is sent to the vehicle control unit 604, When the voice output unit 616 of the output unit 612 makes a warning announcement such as “take a distance between vehicles”, and when the distance between the vehicles is reduced to a dangerous area due to falling asleep, the brake 608 is automatically activated, Prevent collisions.

FIG. 31 is a block diagram showing the details of the rear camera / sensor 622 together with the distance detection principle, and is an arrangement conceptual diagram viewed from the top. The same parts as those in FIG. 30 are denoted by the same reference numerals. In FIG. 31, an infrared beam scanner 630 includes a solar blind beam light source 802 made of a laser or a light emitting diode and projecting a beam, a polygon mirror for reflecting the projected beam to scan parallel to the road surface, and a resonant galvanometer mirror. Alternatively, a two-dimensional scanning system 804 including an acousto-optic deflection element is included. The beam from the solar blind beam light source 802 has a vertically long cross section as will be described later in order to cover the following vehicle corresponding to the up and down of the road. For the same purpose, the two-dimensional scanning system 804 is configured as a spatial scanning system including a vertical scanning in order to cover a succeeding vehicle that may be positioned in the horizontal and vertical directions corresponding to the up and down of the road. May be.

The optical axis 806 of the CIGS imaging sensor 626 is disposed substantially at the center of the vehicle 602, but the rotation center of the scanning beam by the two-dimensional scanning system 804 is provided at a position away from the optical axis 806. Since the distance between the rotation center of the projection beam and the optical axis 806 is a so-called triangulation baseline length, in principle, it is desirable that the two be separated as much as possible. Therefore, if the arrangement is possible, for example, the infrared beam scanner 630 is preferably provided in a part of the taillight. However, on the other hand, when priority is given to assembling the rear camera / sensor 622 as compact as possible, it is preferable to arrange the infrared beam scanner 630 and the CIGS imaging sensor close to each other as long as accuracy can be guaranteed. It is.

Since the rear camera / sensor 622 is configured as described above, for example, when the following vehicle exists at a position 810 immediately behind the vehicle 622, the reflected light 814 of the projection beam at the angle 812 is directed to the CIGS imaging sensor 626. That is, the image of the reflected beam 814 is imaged on the optical axis 806 of the CIGS imaging sensor 626, that is, at the center in the left-right direction of the screen. The combination of the image position of the reflected beam 814 and the angle of the projection beam is uniquely determined by the position 810 of the following vehicle, and the angle 812 of the projection beam at that time is grasped by the monitoring recording control unit. The distance from the combination of the imaging position of the reflected beam 814 to the following vehicle at the position 810 is obtained.

Similarly, when the following vehicle is present at a position 816 immediately behind the vehicle 622, the reflected beam 820 by the projection beam at the angle 818 is directed to the CIGS imaging sensor 626. At this time as well, the image of the reflected beam 820 is captured at the center of the screen of the CIGS imaging sensor 626 in the left-right direction. However, since the projection beam is at a different angle 818, it can be seen that the following vehicle is at a farther position 816. When the succeeding vehicle is at position 822, the reflected light beam 824 image when the projection beam is at an angle 818 is picked up by the CIGS image sensor 626 as in the case of the position 816. However, the imaging position of the reflected light beam 824 at this time is not the center of the left and right sides of the screen, but is imaged on the right side of the center of the screen in FIG. It will never be done.

FIG. 32 is a conceptual diagram when an image captured by the CIGS imaging sensor 624 is viewed by the image output unit 614 of the in-vehicle output unit 612. FIG. 32A corresponds to the case where the following vehicle is present at the position as shown in FIG. 31, and the succeeding vehicle at the nearest rearmost position 810 and the reflected beam 814 thereby are observed. Since the cross section of the projection beam is vertically long as described above, the image of the reflected beam 814 from the succeeding vehicle has a vertically long shape. The reflected beam 814 is shown only in the center of the following vehicle at the position 810 for the sake of simplicity, but the reflected beam is actually banded while the projection beam is scanned from the left end to the right end of the front surface of the following vehicle. Detected. Since the scan speed is sufficiently higher than the vehicle speed, the movement of the vehicle position during the scan can be ignored. Therefore, in order to determine the angle of the projection beam corresponding to the image of the reflected beam 814 as shown in FIG. 32 (A) from the image of the reflected beam extending over the entire front surface of the vehicle in this manner. A process for obtaining the center of gravity of the belt-like reflected beam image extending over the front of the vehicle is necessary. Finding and processing such a center of gravity will be described later.

In FIG. 32 (A), in the same manner as the following vehicle at the position 810 as described above, the vehicle follows the following vehicle at the far rear position 816, the center of gravity of the reflected beam 820, and the position 822 shifted to the right side. The following vehicle and the center of gravity of the reflected beam 824 are shown in the image output unit 614. Note that FIG. 32 is a conceptual diagram, and for convenience of explanation, a plurality of succeeding vehicles are shown simultaneously in a bird's eye view, but an actual image is different from this. That is, when the vehicle 602 and each subsequent vehicle are traveling on a flat ground, the optical axis 806 of the CIGS imaging sensor is set in parallel to the road surface, so that the subsequent vehicle at the position 816 is at the position 810 just before it. If there is a following vehicle, it is hidden behind and cannot be seen at all. In addition, the following vehicle that is traveling at an obliquely rear position 822 is imaged so as to overlap the following vehicle at the forward position 810. Note that processing in the case where a plurality of succeeding vehicles overlap in this way will be described later.

FIG. 32B illustrates a case where the following vehicle has abnormally approached as indicated by a position 826. As described above, for such a succeeding vehicle, a “rear vehicle driving information display” 648 indicates “rearward”. An indication such as “Shooting is in progress” is displayed. For privacy reasons, unless there is an abnormal state, the image is subjected to a windshield mosaic process 828 and a license plate mosaic process 830. This mosaic processing is canceled when the abnormal acceleration is detected by the abnormal acceleration detector 644 due to an accident or the like. Therefore, in order to preserve evidence, an image showing the driver's face, license plate, etc. is recorded.

FIG. 33 is a graph for explaining the spectral transmission characteristics of the filter of the CIGS imaging sensor 626 in the fifth embodiment. The spectral sensitivity (quantum efficiency) 832 of the CIGS sensor is conceptually the same as that shown in FIG. On the other hand, the spectral irradiance 834 of sunlight on the ground surface has a solar blind region where the spectral irradiance of sunlight is reduced on the comb teeth upon absorption of the atmosphere. This corresponds to around 1100 nm and around 1400 nm. Comparing the two, it can be seen that the CIGS sensor has high spectral sensitivity in the solar blind region of sunlight around 1100 nm. Therefore, in the fifth embodiment, a bandpass filter having a spectral transmittance 836 having a peak in the solar blind region of sunlight near 1100 nm as an infrared light transmitting filter in the color filter array as described with reference to FIGS. Is adopted. The solar blind beam light source of FIG. 31 has a strong spectral intensity in the transmission region of this bandpass filter. In this way, the number of infrared pixels of the CIGS image sensor 626 can detect the reflected light of the beam projected from the infrared beam scanner regardless of the presence of sunlight.

FIG. 33 further shows spectral transmittances 838, 840, and 842 of bandpass filters of a narrow visible light region that are respectively employed for a red transmission filter, a green transmission filter, and a blue transmission filter. In these wavelength ranges, the spectral irradiance 834 of sunlight on the ground surface is sufficient, and the spectral sensitivity (quantum efficiency) 832 of the CIGS sensor is extremely high, so that visible light can be imaged. As described above, in the fifth embodiment, since all the filters including the spectral transmittance 838 of the red transmission filter are band pass filters that transmit a narrow wavelength region, imaging of visible light and solar blind region are possible. It is possible to simultaneously detect the distance by the projection beam. Note that the focus position of the imaging lens 604 is determined based on visible light. In this case, the infrared pixel is not in focus, but there is no problem for the purpose of obtaining the center of gravity of the reflected beam.

FIG. 34 is a flowchart of the operation of the monitoring record control unit 632 in the fifth embodiment of FIG. The flow starts when the vehicle 602 starts to travel. In step S302, the start of drive recorder recording is instructed. In step S304, the start of scanning with a solar blind infrared beam is instructed, and the process proceeds to step S306. In step S306, based on the above-described instructions, the visual images sent from the front and rear image processing units, the sound from the front and rear microphones, and the FIFO recording of the front and front detection distances are instructed.

Next, in step S308, an inter-vehicle distance detection process based on the beam angle from the infrared beam scanner 630 and the solar blind area captured image of the CIGS imaging sensor is performed. This inter-vehicle distance detection is performed for both the front vehicle and the rear vehicle, details of which will be described later. When the front and rear inter-vehicle distances are detected in step S308, based on this, in step S310, the front camera / sensor 722 and the rear camera / sensor 622 are automatically focused on the camera for visible light shooting, respectively. .

In the next step S312, the presence or absence of an abnormal approach to the preceding vehicle is checked based on the front inter-vehicle distance detected in step S308. When an abnormal approach between the front vehicles is detected, the process proceeds to step S314, where an instruction to decelerate by automatically operating the brake with a safe strength is given, and an announcement of an abnormal inter-vehicle distance is given at step S316, and the process proceeds to step S318. On the other hand, if no forward inter-vehicle abnormal approach is detected in step S312, the process proceeds directly to step S318. Here, when more detailed processing according to the inter-vehicle distance is desired, the inter-vehicle distance determination in step S312 is made in two stages, and when the inter-vehicle distance is relatively large, only the warning announcement instruction in step S316 is kept, and the inter-vehicle distance is shorter. Only when this occurs, the brake automatic operation of step S314 is applied. Further, it can be configured to select in advance whether such a fine treatment or one-step processing as shown in FIG.

In step S318, based on the distance between the rear vehicles detected in step S308, it is checked whether the rear vehicle is abnormally approaching. When an abnormal approach between the rear vehicles is detected, the process proceeds to step S3320, and an announcement of the abnormal inter-vehicle distance is instructed. Further, in step S322, a notice display such as “being photographed in the rear” is displayed on the rear window of the vehicle 602 so that it can be seen by the driver of the rear vehicle. In step S324, an instruction is given to perform windshield mosaic processing 828 and license plate mosaic processing 830 on the rear vehicle image captured by the rear camera / sensor 622, and the process proceeds to step S326. On the other hand, if an abnormal approach between the rear vehicles is not detected in step S318, the process proceeds to step S328, the countermeasure for the rear vehicle corresponding to step S324 is canceled from step S320, and the process proceeds to step S326. This is a process that is necessary when the inter-vehicle distance from the rear vehicle increases corresponding to the detection of the abnormal approach state and the vehicle returns to the normal state.

In step S326, it is checked whether or not an abnormal acceleration is detected. If there is a detection, the rear vehicle front / number plate mosaic process is canceled in step S330. In step S332, the image / sound / distance for 10 seconds after the detection of the abnormal acceleration is waited to be input to the FIFO 642, and the image / sound / distance for 10 seconds before and after the detection of the abnormal acceleration in the FIFO 642 thus secured is obtained. Recording is performed in the recording unit 646, and the process proceeds to step S334. On the other hand, if no abnormal acceleration is detected in step S326, the process directly proceeds to step S334.

In step S334, it is checked whether or not the vehicle has stopped. If not stopped, the process returns to step S306, and thereafter steps S306 to S334 are repeated to continue recording and detection by the front camera / sensor 722 and the rear camera / sensor 622. While dealing with various conditions. On the other hand, when the stop of the vehicle is detected in step S334, the process proceeds to the stoppage process in step S336. There are cases in which a vehicle accident occurs not only when the vehicle is running, but also when the vehicle crashes from behind while the vehicle is stopped by a signal, for example. The stop processing is for preserving evidence even for such an accident while the vehicle is stopped.

FIG. 35 is a flowchart showing details of the stopping process of FIG. When the process proceeds from the step S334 in FIG. 34 to the stoppage process in FIG. If it is detected that the vehicle is traveling, the process returns to step S306 in FIG. On the other hand, if it is confirmed that the stop is continued, the process proceeds to step S344, and the visible image continuously transmitted from the front and rear image processing units, the sound from the front and rear microphones, the detection of the front and front are detected. Instructs FIFO recording of distance.

Next, in step S336, an inter-vehicle distance detection process based on the beam angle from the infrared beam scanner 630 and the solar blind area captured image of the CIGS imaging sensor is performed. This is the same process as step S308 in FIG. When the front and rear inter-vehicle distances are detected in step S346, in step S348, the front camera / sensor 722 and the rear camera / sensor 622 are automatically focused on the camera for visible light shooting based on the distance. . This is also the same as step S310 in FIG.

In the next step S350, based on the front inter-vehicle distance detected in step S308, the presence or absence of abnormal approach on the premise of stopping is checked for the preceding vehicle. The criterion for determining the abnormal approach is a distance closer than that in step S312 of FIG. Since the vehicle 602 has already stopped, such a case corresponds to a case in which the vehicle ahead is not aware of it and is back. Accordingly, when an abnormal approach between the vehicles on the premise of stopping is detected, the process proceeds to step S352, the horn is automatically actuated to call attention to the preceding vehicle, and the process proceeds to step S354. On the other hand, if it is determined in step S3520 that an abnormal approach between the vehicles on the premise of stopping is not detected, the process directly proceeds to step S354.

In step S354, based on the distance between the rear vehicles detected in step S308, it is checked whether the rear vehicle is abnormally approaching. This check criterion is the same as that in step S318 in FIG. However, when the rear vehicle abnormal approach is detected here, the warning announcement instruction and the rear car dorareko notification display instruction as shown in FIG. 34 are not performed, and only the rear car front / number mosaic process of step S356 is performed and the process proceeds to step S358. Transition. This is because it is natural that the distance between the vehicle and the rear vehicle is shortened when waiting for a signal, etc., and the warning announcement instruction in such a state is unnecessary, and the rear vehicle dorareko notification display instruction is This is a cause of trouble. Therefore, it is limited to privacy considerations by the rear car front / number mosaic processing. On the other hand, if an abnormal approach between the rear vehicles is not detected in step S356, the process proceeds to step S360, the rear vehicle countermeasure is canceled, and the process proceeds to step S358. This is to cancel the rear vehicle front / number mosaic process in a state where the rear vehicle is far away when the host vehicle is stopped.

In step S358, on the basis of the front inter-vehicle distance detected in step S308, it is checked whether or not there is an abnormal approach on the assumption that the rear vehicle is stopped. The criterion for determining the abnormal approach is a closer distance than in step S354. When such an abnormal approach is detected, the process proceeds to step S362, the foot brake is automatically operated, and the process proceeds to step S364. This is because, when the vehicle is stopped only by the side brake, in the unlikely event of a rear-end collision, the stopping ability of the vehicle is enhanced to reduce the shock as much as possible. On the other hand, if it is not detected in step S358 that an abnormal approach between the vehicles behind the stop is detected, the process directly proceeds to step S364.

In step S362, it is checked whether or not an abnormal acceleration is detected. If there is a detection, the rear vehicle front / number plate mosaic process is canceled in step S366. In step S368, the image / sound / distance for 10 seconds after the abnormal acceleration is detected is input to the FIFO 642, and the image / sound / distance for 10 seconds before and after the abnormal acceleration detection in the FIFO 642 thus secured is determined. Recording is performed in the recording unit 646, and the process proceeds to step S334. These are the same as step S330 and step S332 of FIG. On the other hand, if no abnormal acceleration is detected in step S364, the process directly proceeds to step S370.

In step S370, it is checked whether the power is stopped. If it is stopped, the process proceeds to step S372, and the front image processing unit 736 and the rear image processing unit based on the signals from the front camera / sensor 722 and the rear camera / sensor 622 are checked. By the image processing at 636, it is checked whether or not a moving object is not captured in the image for a predetermined time or more. If a moving object is detected, the process returns to step S342, and thereafter, steps S342 to S372 are repeated to cope with various states while continuing recording and detection by the front camera / sensor 722 and the rear camera / sensor 622. When a moving object is detected in step S372, the process returns to step S342 because there is a possibility that the vehicle 602 may be involved in an accident if there is a running vehicle even if the vehicle 602 is stopped and the power is stopped. This is to maintain evidence. On the other hand, if no moving object is detected for a predetermined time or more in step S372, it means that the vehicle 602 has been stored in a safe position such as a garage, and thus the flow ends. When it is detected in step S370 that the power is not stopped, the process returns to step S342, and various states are dealt with while recording and detection by the front camera / sensor 722 and the rear camera / sensor 622 are continued.

FIG. 36 is a flowchart showing details of the inter-vehicle distance detection process in step S308 of FIG. 34 and step S346 of FIG. This flow is performed for each of the image of the front camera / sensor 722 and the rear camera / sensor 622. For the sake of simplicity, the flow will be described as processing for the image from the rear camera / sensor 622. When the flow starts, a visible light image is processed in step S382. Based on this processing, it is checked in step S384 whether there is a vehicle image behind the vehicle 602 in the same lane. If there is an image, the process proceeds to step S386, and the objectivity of the image is checked. This means checking whether a plurality of vehicles overlap each other. If there is no target, the process proceeds to step S388, and matching is performed with several patterns of vehicle overlap images prepared in advance.

Next, in step S386, a matching pattern is found, and it is checked whether images of only the closest one of a plurality of overlapping vehicles can be separated based on the pattern. If separation is possible, the process proceeds to step S392. On the other hand, if there is symmetry in the vehicle image in step S386, it is determined that there is no overlap, and the process directly proceeds to step S392. Then, the distance of the vehicle is determined from the size of the image of the single vehicle separated in step S392. However, this distance determination is only rough because the vehicle itself has variations in size ranging from light cars to large buses. Further, in step S394, the center of gravity in the visible light image of the separated single vehicle is detected, and the process proceeds to step S396.

In step S396, solar blind pixel image processing is performed. In step S398, it is checked whether or not there is an image group whose output is greater than or equal to a predetermined value. This corresponds to checking whether the projected beam of the solar blind area scanning backward is reflected by the vehicle behind and is detected by the solar blind pixel of the rear camera / sensor 622. This is because the reflected light of the projection beam does not have a predetermined intensity or more unless there is a close object such as a vehicle.

If there is a pixel group having a predetermined output or more in step S398, the process proceeds to step S400, and it is checked whether or not there is a portion in the pixel group that matches the center of the visible light image detected in step S494. If there is a corresponding portion, this is regarded as the center of gravity of the reflected beam, and in step S402, the angle information of the projection beam that is the origin of the center of the reflected beam is acquired. Thus, in step S404, the distance to the vehicle that reflected the projection beam is provisionally determined from the projection beam angle obtained in step S402 and the image centroid position obtained in step S394. It is checked whether the distance temporarily determined in this way is consistent with the distance based on the image size obtained in step S392 in the next step S406. If there is no contradiction, the distance temporarily determined in step S404 is officially determined as the distance to the detected vehicle, and the flow ends.

On the other hand, if there is a discrepancy in the distance in step S406, the process proceeds to step S410, where it is checked whether the image processing has been performed a predetermined number of times or more. Try again. In this way, unless it is confirmed in step S406 that the solar blind detection distance that is consistent with the image size distance has been obtained or it is not detected in step S410 that the image processing has been performed a predetermined number of times or more, steps S382 to S382 are performed. S410 is repeated. In addition, when a vehicle image in the same lane is not detected in step S384, or when a single vehicle cannot be separated from the overlapping image in step S390, or when there is no solar blind pixel whose output is greater than or equal to a predetermined value in step S398, Alternatively, in step S400, if there is no portion in the solar blind pixel group that coincides with the visible light image centroid, the process immediately proceeds to step S410, and the image processing is repeated.

On the other hand, when it is detected in step S410 that the image processing has been repeated a predetermined number of times or more without being able to determine the distance, the process proceeds to step S412 and whether or not there is an image size distance is checked in step S392. If there is an image size distance, the process proceeds to step S414, the latest image size distance among them is adopted, and the flow ends. On the other hand, if there is no image size distance in step S412, the process proceeds to step S416, and a pan focus distance that focuses on most of the rear of the vehicle from infinity to the middle distance is adopted, and the flow ends. Accordingly, at least some distance for the camera AF in step S310 of FIG. 34 or step S348 of FIG. 35 is determined, and the flow is ended.

FIG. 37 is a block diagram showing details of the rear camera / sensor of the vehicle monitoring apparatus according to the sixth embodiment of the present invention together with the distance detection principle. Although the sixth embodiment basically has the same configuration as that of the fifth embodiment including the entire configuration block of FIG. 30, the configuration around the CIGS sensor in the front camera sensor and the rear camera / sensor is slightly different. It will be. FIG. 37 shows the rear camera / sensor 922 as a representative in the same manner as the fifth embodiment of FIG. 31, but parts common to the fifth embodiment are numbered the same as in FIG. 31. Description is omitted unless particularly necessary.

The CIGS imaging sensor 926 of the sixth embodiment has a different color filter configuration, and the red transmission filter is not a band-pass filter that transmits a narrow region but a low-pass filter that transmits longer wavelengths than red. Also, the infrared light transmission filter is not a bandpass filter that transmits infrared light in a narrow area of the solar blind, but a low-pass filter that transmits light in the solar blind area and other near infrared areas. . These low-pass filters are simpler in construction than band-pass filters. For this reason, first, a fixed high-pass filter 901 for cutting the longer wavelength side than the solar blind region is provided in the optical path to the CIGS sensor 926. Further, a movable band cut filter 903 that cuts light that is longer than the red region and shorter than the solar blind region is provided in the optical path to the CIGS sensor 926. As a result of these combinations, light in a narrow area is incident on each of the red pixel, the green pixel, the blue pixel, and the solar blind pixel, as in FIG.

In the sixth embodiment, infrared imaging is further possible. For this purpose, the movable band cut filter 903 is retracted from the optical path to the CIGS imaging sensor 926, and instead the movable low-pass filter 905 that cuts the visible light region and transmits the infrared region is the optical path to the CIGS sensor 926. Provided inside. The filter drive unit 907 replaces the movable band cut filter 903 and the movable low-pass filter 905. Based on the visible light output of the CIGS sensor, the average illuminance detection unit 909 determines that the average illuminance has decreased, that is, the situation corresponding to the evening to the night, and instructs the filter driving unit 907 to move. The band cut filter 903 is automatically switched to the movable low pass filter 905. Further, based on the visible light output of the CIGS sensor, the contrast detector 909 determines that the field of view has deteriorated due to rain or fog when the contrast decreases, and instructs the filter driver 907 to cut the movable band. The filter 903 is automatically switched to the movable low-pass filter 905.

As described above, according to the sixth embodiment, visible light photographing can be automatically switched to infrared light photographing when it is dark or when visibility is poor. The average illuminance detection unit 909 and the contrast detection unit 909 automatically switch the movable low-pass filter 905 to the movable band cut filter 903 when the average illuminance or contrast increases based on the infrared light output of the CIGS sensor. Switch from light photography to visible light photography. The AF driving unit 940 works in conjunction with the above-described filter replacement, and adds infrared correction to focus when infrared imaging is performed.

38A and 38B are graphs for explaining the spectral transmission characteristics of the filter of the CIGS image sensor 626 in the sixth embodiment. FIG. 38A is a visible light imaging state, and FIG. 38B is an infrared light imaging. Each state is shown. The spectral sensitivity 832 of the CIGS sensor, the spectral irradiance 834 of sunlight, the spectral transmission factor 840 of the green transmission filter, and the spectral transmission factor 842 of the blue transmission filter are the same as in FIG. On the other hand, the spectral transmittance 932 of the red transmission filter is a low-pass filter that transmits longer wavelengths than red. In addition, the spectral transmittance 936 of the infrared light transmission filter is a low-pass filter that transmits light in the solar blind region and other near infrared regions. As can be seen from FIG. Also, the near-infrared region on the short wavelength side is transmitted relatively widely. Furthermore, the spectral sensitivity of the fixed high-pass filter 901 is a high-pass filter that cuts the longer wavelength side than the solar blind region.

In the above configuration, in FIG. 38A, the movable band cut filter 903 cuts light on the longer wavelength side than the red region and on the shorter wavelength side than the solar blind region, as indicated by the hatched portion. ing. As a result, similar to FIG. 33, a combined spectral transmittance is achieved in which light in a narrow area is incident on each of the red pixel, the green pixel, the blue pixel, and the solar blind pixel. In the state of FIG. 38A, the reflected light of the solar blind projection beam is sensed by superimposing it on the red image even in the red pixel. Since this reflected light is detected only at a previously known beam projection timing, the projected beam reflected light can be detected from the output change at this timing and used for distance detection information. Even if the red pixel has sensitivity in the solar blind region, there is no influence of sunlight, so that there is no problem with red visible light imaging unless the projection beam is involved.

On the other hand, in FIG. 38B, the movable band cut filter 903 is removed, and the movable low-pass filter 905 cuts the visible light region as indicated by the hatched portion. As a result, light in a fairly wide band in the near infrared is incident on the infrared pixel. As a result, imaging using infrared light including that using sunlight as a light source becomes possible. In addition, since light in the solar blind region can be incident, distance measurement using a projection beam is also possible. Here, in the state of FIG. 38B, the red pixel has almost the same spectral transmittance as that of the infrared pixel. Therefore, in infrared imaging, the red pixel is regarded as the infrared pixel and used for imaging information. can do.

In the sixth embodiment, when aberration due to incidence of near-infrared light in a wide band and shift in focus position become problems, the cutoff wavelength of the spectral transmittance 936 of the infrared light transmission filter is set to about 1000 nm, for example. As a result, the cut-off wavelength of the movable low-pass filter 905 is shifted to about 1000 nm correspondingly to cut near-infrared light on the short wavelength side. As a result, the infrared imaging band can be narrowed to reduce aberrations and focus position shifts. However, since this reduces the amount of light for infrared light photography, the transmission bandwidth for infrared light photography is determined based on the balance between the two. Also, it is possible to select where the peak is depending on the purpose.

FIG. 39 is a flowchart of the operation of the monitoring record controller 632 in the sixth embodiment of FIG. Since the contents are often in common with the flowchart in the fifth embodiment of FIG. 34, the common steps are given the same step numbers and will be described in relation to FIG. First, step S302 to step S308 and step S334 in FIG. 39 are the same as the step with the same number in FIG. Further, the vehicle abnormal approach process in step S422 in FIG. 39 is a combination of steps S312 to S324 and S328 in FIG. 34, and the contents thereof are the same. Similarly, the abnormal acceleration detection process in step S424 in FIG. 39 is a combination of step S326, step S330, and step S332 in FIG. 34, and the contents thereof are the same. 34 is replaced by step S442 from step S426 in FIG. The contents of the stopping process in step S444 in FIG. 39 are obtained by replacing step S348 in FIG. 35 from step S426 in FIG. 39 to step S442.

Based on the above assumptions, steps S426 to S442 in FIG. 39 will be described. These portions relate to automatic switching between the movable low-pass filter 905 and the movable band cut filter 903 described with reference to FIG. 37, and infrared correction of the AF driving unit 940 in conjunction therewith. In FIG. 39, when the front and rear inter-vehicle distances are detected in step S308, it is checked in step S426 whether infrared imaging has been manually selected. If there is no selection, the process proceeds to step S428, and it is checked whether the illuminance detected by the average illuminance detection unit 909 is equal to or less than a predetermined value. If the illuminance is not less than the predetermined value, it is further checked in step S430 if the contrast detected by the contrast detection unit 911 is less than the predetermined value.

When it is determined in step S430 that the contrast is equal to or lower than the predetermined value, the process proceeds to step S432, and an instruction to insert a movable low-pass filter 905 that cuts the visible light band is given to the filter control unit 907. When manual infrared imaging selection is detected in step S426, or when it is detected in step S428 that the illuminance is not more than a predetermined value, the process immediately proceeds to step S432. Next, in step S434, the AF driving unit 940 is instructed to perform focusing by camera AF with infrared correction. In step S436, the image processing unit is instructed to select infrared image output, and the process proceeds to step S422.

On the other hand, if the contrast is not less than or equal to the predetermined value in step S430, the process proceeds to step S438, and the filter controller 907 is instructed to insert a movable band cut filter 903 that cuts the infrared band other than the solar blind. Next, in step S440, the AF drive unit 940 is instructed to perform focusing by camera AF without infrared correction. In step S436, the image processing unit is instructed to select visible light image output, and the process proceeds to step S422.

FIG. 40 is a color filter array of a CIGS imaging sensor used for the front and rear cameras / sensors of the vehicle monitoring apparatus according to the seventh embodiment of the present invention. This filter arrangement is basically the same as a typical Bayer arrangement in primary color filters, and includes a red transmission filter R21 (IR21), green transmission filters G11 and G22, and a blue transmission filter B12. However, in the seventh embodiment, the output of the pixel corresponding to the red transmission filter R21 (IR21) is also used as the infrared light output by switching the filter as described later. The configuration of the front and rear cameras / sensors of the vehicle monitoring apparatus of the seventh embodiment of the present invention is the same as that of the sixth embodiment shown in FIG. 37, and only the filter array of the CIGS image sensor 926 is as described above. Different.

FIG. 41 is a graph for explaining the spectral transmission characteristics of the filter of the CIGS imaging sensor 626 in the seventh embodiment. FIG. 41A shows the state of visible light imaging, as in FIG. 41 (B) shows the state of infrared light photography. CIGS sensor spectral sensitivity 832, solar spectral irradiance 834, green transmission filter spectral transmission factor 840, blue transmission filter spectral transmission factor 842, red transmission filter spectral transmission factor 932, fixed high-pass filter 901 spectral sensitivity and The spectral sensitivity of the movable band cut filter 903 is the same as that in FIG. That is, the seventh embodiment differs from the sixth embodiment only in that the infrared-only pixels are removed by the color filter array of the CIGS imaging sensor.

As a result, in the state of FIG. 41A, the reflected light of the solar blind projection beam is detected by superimposing the red image on the red pixel of the spectral transmittance 932 corresponding to R21 (IR21) of FIG. . In the seventh embodiment, since there is no dedicated image for detecting the solar blind reflected light, the projection beam is intermittently interrupted in order to separate the solar blind reflected light component from the red image output of the red pixel. Since this intermittent timing is known in advance, if there is a red pixel output that increases in synchronization with the projection timing, it is separated as solar blind reflected light. Note that even if the red pixel has sensitivity in the solar blind area, it is not affected by sunlight. Therefore, as long as the projection beam is not involved, there is no problem with red visible light imaging as in the sixth embodiment.

On the other hand, in FIG. 41B, the movable band cut filter 903 is removed, and the movable low-pass filter 905 cuts the visible light region and the near-infrared region up to about 1000 nm as shown by the hatched portion. As a result, red light having a spectral transmittance of 932 detects infrared light based on sunlight from 1000 nm to around 1100 nm and reflected light of the solar blind projection beam. Again, in order to separate the solar blind reflected light component from the red pixel infrared light output, the projection beam is intermittently interrupted. Since the intermittent timing is known in advance, as in the case of FIG. 41A, if there is a red pixel output that increases in synchronization with the projection timing, it can be separated as solar blind reflected light.

FIG. 41B shows an example in which the movable low-pass filter 905 cuts the visible light region and the near-infrared region up to about 1000 nm, but this is not unique to the seventh embodiment. In the sixth embodiment, it has already been described that the cut-off wavelength of the movable low-pass filter 905 is shifted to about 1000 nm in order to cope with the problem of aberration and focus position. FIG. 41B adopts such a configuration. This example is specifically shown. Also in the seventh embodiment, when there is no problem in terms of separation of reflected light components of the solar blind, aberration, and focus position shift, the movable low-pass filter 905 in the infrared imaging state is the same as in the sixth embodiment. The cutoff wavelength may be shifted to the short wavelength side.

FIG. 42 is a graph for explaining the spectral transmission characteristics of the color filter of the CIGS image sensor used in the front and rear cameras / sensors of the vehicle monitoring apparatus according to the eighth embodiment of the present invention. The eighth embodiment has basically the same configuration as the seventh embodiment. That is, the filter arrangement of the CIGS imaging sensor is a Bayer arrangement as shown in FIG. 40, and the output of the pixel corresponding to the red transmission filter R21 is also used as the infrared light output by switching the filter as in the seventh embodiment. The Similarly to the seventh embodiment, the configuration of the front and rear cameras / sensors of the vehicle monitoring apparatus is the same as that of the sixth embodiment shown in FIG. The eighth embodiment of FIG. 42 differs from the seventh embodiment in that the distance detection function by solar blind beam reflected light detection is omitted, and the visible light drive recorder in the normal daytime with only visible light and the infrared light in the nighttime or in fog. The function is limited to automatic switching of the optical drive recorder.

Also in the eighth embodiment, similarly to FIGS. 38 and 41, FIG. 42 (A) shows the state of visible light photographing, and FIG. 42 (B) shows the state of infrared light photographing, respectively. 42, the spectral sensitivity 832 of the CIGS sensor, the spectral irradiance 834 of sunlight, the spectral transmission factor 840 of the green transmission filter, the spectral transmission factor 842 of the blue transmission filter, the spectral transmission factor 932 of the red transmission filter, and the movable The spectral sensitivity of the band cut filter 903 is the same as that in FIG. FIG. 42 differs from FIG. 41 in that the fixed high-pass filter 901 has a spectral transmittance that cuts a longer wavelength range including the solar blind region near 1100 nm, and uses the sensitivity of the CIGS sensor in the solar blind region. That is not the point.

Thus, in the state of FIG. 42A in which the movable band cut filter 903 is inserted, only the red image is sensed by the red pixel having the spectral transmittance 932. On the other hand, in FIG. 42B, the movable band cut filter 903 is removed, and the movable low-pass filter 905 cuts the visible light region and the near-infrared region up to about 1000 nm as shown by the hatched portion. As a result, infrared light based on sunlight from 1000 nm to around 1100 nm is detected by the red pixel having the spectral transmittance of 932. In this way, the wide spectral sensitivity of the CIGS sensor is used, and the red pixels in the Bayer array are switched to red image detection in visible light photography and infrared image detection in infrared light photography. The CIGS sensor of the eighth embodiment is not limited to the drive recorder, but can be used in the cameras in the first and second embodiments, and in the endoscope in the fourth embodiment. is there.

The present invention provides a monitoring device for a vehicle.

626 Photoelectric conversion unit 624 Imaging lens 630 Infrared light projection unit 636, 736 Image processing unit IR11, G11, B12, R21, R21 (IR21) Filter 632 Control unit 622, 722 Image acquisition unit 636 Extraction unit 636 Detection unit 636 Visible information Acquisition Unit 636 Infrared Light Information Acquisition Unit 903 Band Cut Filter 905 Low Pass Filter

Claims (5)

A plurality of light-receiving portions having a quantum efficiency of 60% or more in the visible light region and having an effective quantum efficiency in the infrared light region where sunlight is weak on the ground surface due to atmospheric absorption are regularly arranged. A photoelectric conversion unit, an imaging lens that forms a subject image on the photoelectric conversion unit, an infrared light projection unit that projects infrared light in an infrared region where the sunlight is weak on the ground surface, and the photoelectric conversion unit A visible light information acquisition unit that acquires an image in the visible light range from the conversion unit, and an infrared light information acquisition unit that acquires a reflected image of the infrared light projected from the infrared light projection unit from the photoelectric conversion unit. An image processing unit , wherein the image processing unit extracts a vehicle image from the image in the visible light range and extracts red from the center of gravity of the extracted vehicle image. Obtaining external light reflection images The vehicle monitoring system according to claim and. Wherein the image processing unit, when the image of the vehicle is overlapped plural, the vehicle monitoring system according to claim 1, wherein extracting the image of the closest single vehicle. 2. A control unit that calculates a distance to a reflection target based on a reflected light image position of infrared light acquired by the image processing unit and an infrared light projection direction from the infrared light projection unit. Or the monitoring apparatus for vehicles of 2 . A photoelectric conversion unit having sensitivity to infrared light in a band in which the visible region and sunlight are weak on the ground surface due to atmospheric absorption, and infrared light in the infrared light region in which the sunlight is weak on the ground surface at a predetermined timing Infrared light projection unit for projecting, visible light information acquisition unit for acquiring an image in the visible light region from the photoelectric conversion unit, and the sunlight on the ground surface at the infrared light projection timing by the infrared light projection unit An infrared light information acquisition unit that acquires reflected light information of infrared light in the infrared light region from the photoelectric conversion unit, and extracts a vehicle image from the visible light region image and the extracted vehicle image And an image processing unit that acquires an infrared reflected image from the center of gravity of the vehicle. From claim 1 characterized by having a recording unit for recording at least one of infrared light information visible information and the infrared light information acquiring section the visible light information acquisition unit acquires acquires the 4 The monitoring apparatus for vehicles in any one.