JP6426753B2 - Imaging system - Google Patents

Description

  The present invention relates to an imaging system that acquires information by simultaneously performing imaging with visible light and imaging with infrared light in the same imaging range.

In general, imaging sensors (image sensors) have sensitivity to light reception not only in visible light but also in near infrared, and with surveillance cameras etc., visible light is photographed in the daytime and infrared light is used at night in the nighttime. It was known to shoot light.
Further, it has been proposed to improve the accuracy of image recognition by using imaging with visible light and imaging with infrared light in combination (see, for example, Patent Document 1).

  For example, in the case of face recognition, the face of a photo such as a poster may be erroneously recognized as the face of a person. So, in patent document 1, a face is recognized from the image recognition by visible light, and the heat detection (body temperature detection) by infrared light.

  In this case, photographing with visible light and photographing with infrared light are performed by one camera, and the temperature of the position of the face specified by face recognition using the image data of visible light is calculated from the image data of infrared light Since the recognition and face recognition and the area where the temperature is high can be recognized as the face, the face photograph of the poster whose temperature is lower than the body temperature is not erroneously recognized as the face of the person.

  Further, a proposal has been made to measure the distance using near-infrared light and an imaging sensor (see, for example, Patent Document 2).

Patent No. 4702441 JP, 2014-36801, A

  By the way, in the case of an imaging device that simultaneously captures both an image of visible light and an image of infrared light, an imaging device using an infrared cut filter for visible light and a visible light cut filter for infrared light In the case of using both the imaging device and the imaging device, at least two imaging devices are required, which makes it difficult to miniaturize and to reduce the cost.

Further, in the case of adding a configuration that enables photographing of a visible image to a configuration that measures a distance using an infrared image, it is necessary to project a pattern set for the subject with infrared light, There is a possibility that the imaging device may become large, and it is desirable to make a portion that simultaneously captures a visible image and an infrared image compact.
When two image sensors are used, a slight positional deviation occurs in two moving image data captured by each image sensor, so when performing image processing using the same coordinate system for two image data, It is necessary to correct misalignment.

  In addition, in the case of an imaging device that switches between an infrared cut filter and a visible light cut filter, the structure for switching the filter is costly, and it is difficult to simultaneously capture a visible image and an infrared image. In the case of a structure in which the movement of the filter is repeated, the number of frames captured per time is limited.

  The present invention has been made in view of the above-mentioned circumstances, and simultaneously performs photographing of a visible image and photographing of an infrared image, and for example, makes data of the distance to the object available with the visible image of the object It aims at providing an imaging system.

In order to solve the above-mentioned subject, the imaging system of the present invention
In the second wavelength band, which has a transmission characteristic in the visible light band, has a blocking characteristic in the first wavelength band adjacent to the long wavelength side of the visible light band, and is a portion within the first wavelength band An optical filter having transmission characteristics;
The imaging sensor main body in which the light receiving element is disposed for each pixel, and the four or more types of regions having different transmission characteristics in the visible light band and having the transmission characteristics in the second wavelength band, An imaging sensor including a color filter disposed in each pixel of the imaging sensor main body in a predetermined array;
An optical system having a lens that forms an image on the imaging sensor;
Based on the signals output from the imaging sensor, signals of components of plural colors of visible light and signals of components of infrared light corresponding to the second wavelength band are determined, and based on these signals, Signal processing means capable of simultaneously outputting a visible image signal and an infrared image signal of the same imaging range;
Pattern projection means including a light source of the infrared light and intermittently projecting a predetermined projection pattern onto a subject;
First image data captured when the predetermined projection pattern is projected is acquired from the infrared image signal output from the signal processing means, and imaging is performed when the predetermined projection pattern is not projected Differential image data acquisition means for acquiring acquired second image data and acquiring third image data including only reflected light of the predetermined projection pattern from the difference between the first image data and the second image data;
An autocorrelation coefficient acquisition unit that calculates an autocorrelation coefficient between pixels of the third image data for a predetermined predetermined pixel;
A distance acquisition unit may be provided to acquire the distance to the subject based on the autocorrelation coefficient of the predetermined pixel.

  According to such a configuration, for example, color filters having four or more types of regions having transmission characteristics in the infrared corresponding to the second wavelength band of the optical filter while having different transmission characteristics in the visible light band are provided. Each color component except the infrared light by providing the imaging sensor and the optical filter having the transmission characteristic in the visible light band and the second wavelength band which is a near infrared band apart from the visible light band. And the signal of the infrared light component can be obtained. In this case, the visible image signal output without using the infrared cut filter does not include the infrared light component. The infrared light passing through the second wavelength band of the optical filter passes through the regions of the color filter. In addition, each region of the color filter includes an infrared (IR) region that transmits visible light but does not transmit visible light and a white (W) region that transmits white light and infrared light. May be Hereinafter, an optical filter having light transmission characteristics in the visible light band and the infrared second wavelength band may be referred to as DBPF (double band pass filter).

  Moreover, the infrared light component contained in the visible light component of each color which has passed through the area of each color of visible light is only the infrared light which has passed through the second wavelength band of DBPF. Also, for example, the infrared light component that has passed through the IR region of the color filter is the component of the infrared light that has passed through the second wavelength band of the optical filter. Therefore, the wavelength range of the infrared component contained in the visible light is limited by the DBPF, and the wavelength range of the infrared light which can pass through the IR region blocking the visible light of the color filter substantially includes the second wavelength band. If so, it is possible to remove the infrared component whose wavelength range is limited as described above from the visible light component of each color, and more accurately remove the infrared light component contained in the visible light component of each color It will be possible to

  This makes it possible to obtain a natural visible image similar to the case of using a conventional infrared cut filter, even without the infrared cut filter, so that the visible image signal can be obtained without switching the presence or absence of the infrared cut filter. The infrared image signal can be simultaneously output, and the distance to the subject on the visible image can be determined from the infrared image. That is, it becomes possible to perform the measurement of the distance to the subject based on the visible image shooting of the subject and the infrared image substantially simultaneously.

  Moreover, since imaging is performed using the above-described color filter with one imaging element, the imaging range does not deviate between the visible image and the infrared image, and basically each pixel of the visible image and the infrared image The respective pixels of each correspond to each other. Therefore, it is not necessary to correct positional deviation as in the case of using two imaging elements.

  In addition, the correlation between the autocorrelation coefficient between the pixels obtained for each pixel of the third image data obtained by photographing the reflected light of the predetermined projection pattern projected onto the subject and the distance to the subject corresponding to each pixel The distance to the subject can be obtained based on the autocorrelation coefficient of the pixel. According to the present invention, since the distance can be measured based on the near-infrared first image data and the second image data obtained by imaging the subject from one viewpoint, two imaging devices are required to measure the distance. There is no need to use it. In addition, in order to obtain the first image data and the second image data, it is sufficient to form a state in which the projection pattern is projected on the subject and a state in which the pattern is not projected on the subject. Basically, the projection pattern itself is changed There is no need to Thus, the pattern projection apparatus can be configured simply. Furthermore, it is not necessary to use a special imaging sensor at the time of imaging, and a general-purpose imaging sensor can be used. Therefore, the manufacturing cost of the imaging system as an imaging system can be held down. In addition, in order to project a predetermined | prescribed projection pattern on a to-be-photographed object, the element which permeate | transmits or reflects a part of light ray from the said light source can be used. As such an element, there is a space modulation element such as a photomask, a transmissive liquid crystal panel, a reflective liquid crystal panel, and a digital micro mirror device. Further, the pattern projection apparatus may not only project the projection pattern fixed by these elements, but also may change the projection pattern as needed. Furthermore, although the autocorrelation coefficient may be calculated for all pixels as a predetermined pixel, the distance of only the representative point may be calculated depending on the size of the object to be measured, and in that case, for discrete pixels An autocorrelation coefficient may be calculated.

  The visible light band is a wavelength band of about 400 nm to 700 nm, the first wavelength band is a wavelength band of about 700 nm or more as a near infrared wavelength band, and the second wavelength band is, for example, 800 nm to It is a wavelength band of about 1100 nm or a wavelength band included in this wavelength band and narrower than this wavelength band.

In the above configuration of the present invention, it has a table that stores and holds in a form in which the autocorrelation coefficient of the pixel of the third image data and the distance to the subject are associated.
It is preferable that the distance acquisition unit acquires the distance to the subject by referring to the table based on the autocorrelation coefficient of the pixel.

  According to such a configuration, it is possible to simplify the process of obtaining the distance from the autocorrelation coefficient to the subject based on the correlation between the autocorrelation coefficient and the distance to the subject.

  In the configuration of the present invention, a monitor for displaying a visible image based on the visible image signal is provided, and the distance to the acquired subject is displayed corresponding to the subject displayed on the monitor. Is preferred.

  According to such a configuration, since the distance is displayed on the monitor in a state where the subject is displayed, it becomes clear which distance is being measured. For example, when there is another object in the vicinity of the subject whose distance is to be measured, it may be difficult to understand whether the distance to the symmetrical subject or the distance to an object nearby was measured. By displaying the distance in the state of being displayed, the object to be measured becomes clear, and false recognition can be prevented. In this case, when there are a plurality of subjects in the imaging range, it is preferable that the distances are displayed corresponding to the respective subjects.

  In the configuration of the present invention, at least two of the signal processing means, the difference image data acquisition means, the autocorrelation coefficient acquisition means, and the distance acquisition means are composed of an electronic circuit on one chip. Preferably, it is made into one chip.

  According to such a configuration, all or two or more of the signal processing means, the difference image data acquisition means, the autocorrelation coefficient acquisition means and the distance acquisition means are integrated into one chip. By reducing the number of parts of the imaging system, it is possible to achieve cost reduction, simplification of the apparatus configuration, simplification of assembly work, downsizing, and the like.

In the configuration of the present invention, the region of each type of the color filter includes a third wavelength band whose transmittances approximate each other on the longer wavelength side than the visible light band,
The spectral transmission characteristics of the optical filter and the types of the color filter such that the second wavelength band of the optical filter is included in the third wavelength band of the region of each type of the color filter The spectral transmission characteristics of the region are set
In the third wavelength band, the difference between the transmittances of the regions of each type is preferably 20% or less of the transmittance.

  According to such a configuration, infrared light passing through the second wavelength band of the optical filter reaches the light receiving element of each pixel of the imaging sensor main body, and imaging is performed by increasing the number of electrons generated by the photoelectric effect. It is preferable to suppress the influence of the infrared light which has passed through the optical filter by processing the image signal because it affects the image.

  In this case, when the color of each type of area of the color filter is, for example, red, green, blue, infrared (or white (including infrared)), the boundary portion between the visible light band and the infrared band The transmittance of the red region is substantially maximum, and the transmittance is substantially maximum even on the longer wavelength side. Moreover, in the green and blue regions, the transmittance is low at the boundary between the visible light band and the infrared band, but the transmittance increases as the wavelength becomes longer than the visible light band, and the transmissivity finally reaches a maximum The transmittance is substantially maximum even on the longer wavelength side than the wavelength at which the transmittance is substantially maximum.

  In this case, in the infrared band adjacent to the visible light band, that is, in the region where the wavelength of the infrared band is short, the transmittance of the blue and green regions is significantly different from the red region where the transmittance is maximum. The transmittances of blue and green are different. Further, in the blue and green regions, the transmittance tends to increase toward the long wavelength side, and the transmittance varies depending on the wavelength until the transmittance is substantially maximized. Therefore, when the wavelength band of this portion is included in the second wavelength band, it is difficult to prevent the infrared light passing through the second wavelength low band from affecting the visible image. That is, when correcting an image signal of red, green and blue based on light passing through each area of red, green and blue using an infrared image signal based on light passing through each area, The amount of infrared component to be corrected is different for each of the green and blue image signals, and it is difficult to improve color reproducibility by subtracting the signal of the same infrared component from the signal of each color.

  Therefore, the third wavelength is such that the transmittances of the red, green, and blue regions are approximately approximated because the transmittances of the red, green, and blue regions are substantially maximum in the second wavelength band of the optical filter. When included in the band, the transmittances of light passing through the second wavelength band and passing through the red, green, and blue regions become substantially the same in the red, green, and blue regions. That is, wavelength bands with largely different transmittances in the red, green, and blue regions overlap with the wavelength band having the cutoff characteristic between the visible light band having the transmission characteristic of the optical filter and the second wavelength band.

  As a result, light of wavelength bands whose transmittances greatly differ depending on the color of the region can not pass through the optical filter, and on the infrared side, light of wavelength bands whose transmittances approximate in red, green and blue regions pass through the optical filter It becomes possible.

  In addition, each area is not limited to red, green, blue and infrared, and different color areas may be used, for example, colors other than red, green and blue in the visible light band instead of infrared. Alternatively, it may be white (clear (C) or white (W)) or the like that transmits light in substantially the entire wavelength band of the visible light band. Also, red, green and blue may use different colors. Also, white may have the light transmittance almost equally low throughout the visible light band.

  Further, as described above, it is possible to suppress the influence on the visible image by the infrared light passing through the second wavelength band of the optical filter, and the color reproducibility can be improved. Basically, it is preferable that the transmittances of the respective color regions in the second wavelength band be substantially equal, but if the transmittances differ by 20% or less, image processing capable of suppressing the above-mentioned influence of infrared light is possible. . The condition that the difference in transmittance in the second wavelength band is 20% or less in terms of transmittance is the highest transmittance (%) to the lowest transmittance (%) in each color region in the second wavelength band. Is a case where the difference in transmittance at the time of subtraction is 20 or less in%. The difference in transmittance is more preferably 10% or less.

  Further, in the configuration of the present invention, the color filter has three types of transmission characteristics of the visible light band as regions having different transmission characteristics of the visible light band and having transmission characteristics in the second wavelength band. It is preferable to provide the area | region of a mutually different color component, and the area | region which becomes white.

  According to such a configuration, the color filter has areas of three different color components and a white area. The three color components may be, for example, three color components of the RGB color space, or may be each color component of another color space. In addition, white is a color obtained by combining the respective colors in the color space described above. For example, white W can be W = R + G + B. In addition, the white area may be an area in a transparent state in which substantially the entire color of visible light is transmitted. A visible image and an infrared image can be obtained by using such a color filter. As described above, it is possible to use a color filter provided with regions of each color component of RGB and a region of IR that blocks light in the visible light band and transmits infrared, but IR filters are visible. While little light is transmitted, a white filter transmits a large amount of visible light, which is an area showing luminance.

  When the IR region is added to the color filter consisting of three regions of RGB, some of the visible light information including the luminance information is lost, while when the white region is added, the luminance information increases. As a result, the resolution of the visible image can be improved.

  According to the imaging system of the present invention, it is possible to obtain a visible image and an infrared image which do not contain an infrared component, without using an infrared cut filter in one imaging device. Therefore, it is possible to measure the distance to the subject using the infrared image simultaneously with capturing of the visible image, and to know the distance of the subject on the visible image.

It is a schematic block diagram of the imaging system of the 1st Embodiment of this invention. Similarly, it is an example of a pattern projected by a photomask. 10 is a timing chart showing a vertical synchronization signal of a CCD image sensor of an imaging device and a control signal for causing a light source of a pattern projection device to blink. It is a schematic diagram of 3rd image data for demonstrating the calculation method of an autocorrelation coefficient equally. Similarly, it is a graph showing the relationship between the autocorrelation coefficient of the pixel and the distance to the subject. It is a general | schematic flowchart which similarly shows the distance measuring method of an optical distance measuring device. It is explanatory drawing of 1st image data, 2nd image data, 3rd image data, and distance image data equally. It is a figure for demonstrating the sequence pattern of the color filter which has an infrared area similarly. It is a graph which similarly shows the transmittance | permeability spectrum of DBPF of the said imaging sensor, and a color filter. It is a figure for demonstrating the arrangement | sequence of the color filter of the imaging sensor of the 2nd Embodiment of this invention. It is a graph which similarly shows the transmittance | permeability spectrum of DBPF of the said imaging sensor, and a color filter. It is a figure for demonstrating the arrangement | sequence of the color filter of an image pick-up sensor equally. It is a figure for demonstrating the arrangement | sequence of a color filter equally. It is a figure for demonstrating the arrangement | sequence of a color filter equally. It is a figure for demonstrating the characteristic of each color filter equally. It is a figure for demonstrating the arrangement | sequence of the color filter of the 3rd Embodiment of this invention. It is a graph which similarly shows DBPF of the said imaging sensor, and the transmittance | permeability spectrum of blue B of a color filter. It is a graph which similarly shows the transmission factor spectrum of green G of DBPF of the said imaging sensor, and a color filter. It is a graph which similarly shows DBPF of the said imaging sensor, and the transmittance | permeability spectrum of red R of a color filter. It is a graph which similarly shows the transmittance | permeability spectrum of DBPF of the said imaging sensor, and clear C of a color filter.

Hereinafter, a first embodiment of the present invention will be described.
FIG. 1 is a schematic block diagram of an imaging system having an optical distance measurement function to which the present invention is applied. The imaging system drives and controls a pattern projection device (pattern projection unit) 2 that intermittently projects a predetermined projection pattern on the subject 100, the imaging device 3, the pattern projection device 2 and the imaging device 3, and captures an image. The control device 4 is provided to calculate the distance D to the subject 100 by performing arithmetic processing on the image data acquired by the device 3. The subject in this example is a cylinder 100a and a wall 100b.

  The pattern projection device 2 includes a light emitting element (LED) that emits near infrared light having a wavelength of 0.85 μm as the light source 5. Further, a photomask (element for transmitting a part of a light beam from the light source) 6 for projecting a predetermined projection pattern onto the optical path of projection light from the light source 5 toward the subject 100, a prism (optical element) 7, and An objective lens 8 is provided.

  FIG. 2 is an example of a pattern projected by the photomask 6. As shown in FIG. 2, the photomask 6 is provided with a random pattern for partially passing the projection light emitted from the light source 5. The photomask 6 is movable in the optical axis direction by a photomask moving mechanism 9 having a piezoelectric element.

  The imaging device 3 has an image sensor 10 mounted thereon as an imaging element. Further, on the optical path between the CCD image sensor 10 and the subject 100, a prism 7 as an optical system for imaging and an objective lens 8 are provided. The imaging device 3 shares the prism 7 and the objective lens 8 with the pattern projection device 2. The image sensor 10 is composed of a sensor body 10a, which is a CCD image sensor of an interlace scanning method, and a color filter 10b. The optical system having the objective lens 8 and the prism 7 is provided with a DBPF 7 d which is an optical filter. For example, the DBPF 7 d is provided on the image sensor 10 side of the prism 7. Further, the imaging device 3 includes a signal processing unit (signal processing means) 21 as described later, and the signal processing unit 21 is configured to output a visible image signal 22 and an infrared image signal 23. The infrared image signal 23 is output from the image sensor 10 to the control device 4 through the signal processing unit 21.

  Next, a process of calculating the distance to the subject 100 using an infrared image signal is described above, and a visible image signal and an infrared image signal are output using the color filter 10b, the DBPF 7d, and the signal processing unit 21. The processing will be described later. Here, when it is described as an image, a near infrared image is shown, and in the case of a visible image, it is called a visible image.

  The prism 7 is formed by bonding two triangular prisms 7a and 7b, and projection light from the light source 5 of the pattern projection device 2 is transmitted through the bonding surface 7c and guided to the subject 100, and from the subject 100 A thin film is formed which has optical characteristics for reflecting the reflected light in a direction different from the light path of the projection light and guiding it to the CCD image sensor 10. Thus, the optical path of the projection light from the prism 7 to the subject 100 via the objective lens 8 and the optical path of the reflected light from the subject 100 to the prism 7 via the objective lens 8 are common. The projection illumination angle of view by the pattern projection device 2 and the imaging angle of view by the imaging device 3 are the same and overlap.

  The control device 4 includes a drive control unit 12 that controls driving of the light source 5 of the pattern projection device 2 via the driver 11 and controls driving of the CCD image sensor 10 of the imaging device 3. The control device 4 also includes an image memory 13 for temporarily storing infrared image data acquired by the CCD image sensor 10 by imaging the subject 100. Furthermore, the control device 4 includes a difference image data acquisition unit (difference image data acquisition means) 14, an autocorrelation coefficient acquisition unit (autocorrelation coefficient acquisition means) 15, a distance acquisition unit (distance acquisition means) 16 and a table storage unit (Table storage means) 17 is provided.

  In the imaging system, the signal processing unit 21 and the control device 4 are configured by, for example, one chip. That is, the signal processing unit 21, the differential image data acquisition unit 14 of the control device 4, the autocorrelation coefficient acquisition unit 15, the distance acquisition unit 16, the image memory 13 and the table storage unit are general purpose microcomputers or dedicated gate arrays. It consists of electronic circuits (integrated circuits), but these are mounted on one chip and made into one chip. The signal processing unit 21, the differential image data acquisition unit 14 of the control device 4, the autocorrelation coefficient acquisition unit 15, the distance acquisition unit 16, the image memory 13 and the table storage unit are both general purpose microcomputers and dedicated gate arrays. It is preferable to make it one chip also when it is composed of Further, it is not necessary to make all of the signal processing unit 21, the differential image data acquisition unit 14 of the control device 4, the autocorrelation coefficient acquisition unit 15, the distance acquisition unit 16, the image memory 13 and the table storage unit into one chip. It is preferable that as many as possible be made into one chip by two or more of these. By making it into one chip, it is possible to reduce the cost, reduce the size, and facilitate the assembly.

  FIG. 3 is a timing chart showing a vertical synchronization signal of the CCD image sensor 10 of the imaging device 3 and a control signal for causing the light source 5 of the pattern projection device 2 to blink. As shown in FIG. 3, the drive control unit 12 drives and controls the light source 5 to project a predetermined projection pattern in synchronization with the odd field of the CCD image sensor 10. Further, the drive control unit 12 controls the drive of the CCD image sensor 10 to acquire the first image data obtained by imaging the subject 100 in the state where the projection pattern is projected as an odd field, and a predetermined projection pattern. The second image data obtained by imaging the subject 100 in a state where the image is not projected is acquired in the even field. Furthermore, the drive control unit 12 drives and controls the imaging device 3 to transfer the first image data and the second image data from the imaging device 3 to the control device 4.

  The image memory 13 includes a first image memory 13a that stores and holds the first image data in a two-dimensional expansion state, and a second image memory 13b that stores and holds the second image data in a two-dimensional expansion state. The first image data expanded in the first image memory 13 a and the second image data expanded in the second image memory 13 b are sequentially updated along with imaging by the CCD image sensor 10. The second image data expanded in the second memory can be sequentially output to the monitor 18 connected to the imaging system, and can be switched and displayed on the monitor 18 with a visible image described later.

  The differential image data acquisition unit 14 acquires the first image data two-dimensionally expanded in the first image memory 13a and the second image data two-dimensionally expanded in the second image memory 13b, and these first images The difference between the data and the second image data is acquired as third image data including only the reflected light of a predetermined projection pattern. In this example, after the first image data and the second image data are converted to monochrome based on the luminance of each pixel, the second image data is subtracted from the first image data to acquire the third image data. For example, the difference image data acquisition unit 14 controls the image memory 13 to acquire the first image data and the second image data.

  The autocorrelation coefficient acquisition unit 15 calculates an autocorrelation coefficient between pixels of the third image data for each pixel P. FIG. 4 is a schematic view of third image data for explaining a method of calculating an autocorrelation coefficient. The autocorrelation coefficient R (x, y) is defined by the following equation with N × N pixels including the pixel P (x, y) at the coordinates (x, y) of the third image data as one block Q.

R (x, y): autocorrelation coefficient I (x, y) at coordinates (x, y): luminance N: block size <I>: average luminance in block

N is preferably about 8 pixels, and this block Q is slid for each pixel to obtain R (x, y). If computation time is limited, or if the definition as a distance image may be low, block Q may be slid each time autocorrelation coefficient R (x, y) is calculated for a plurality of pixels. Good. In the above equation, <I> ² is used as a normalization factor for correcting the light amount due to the reflectance of the object to be measured.

  Here, the inventors of the present invention calculated the autocorrelation coefficient R (x, y) between the pixels obtained for each pixel P (x, y) and the distance to the subject 100 corresponding to each pixel P. It has been found that there is a correlation with D, and the distance D to the subject 100 can be obtained based on the autocorrelation coefficient R (x, y) of each pixel P (x, y). That is, if the pattern projection apparatus 2 constitutes an imaging optical system and the imaging point is set at a certain distance from the projection unit, the size of the image of the light spot by the photomask is inversely proportional to the distance, The light spots projected to the close objects overlap each other, and the autocorrelation coefficient becomes large. Here, mathematically, the Fourier transform of the autocorrelation coefficient is uniquely equivalent to the direct Fourier transform of the image. Therefore, the Fourier transform of the third image may be correlated with the distance.

  FIG. 5 is a graph showing the relationship between the autocorrelation coefficient R (x, y) of the pixel P (x, y) and the distance D to the subject 100 corresponding to the pixel P (x, y). FIG. 5 shows the autocorrelation coefficient R (x, y) and the distance D when the imaging system is fixed and the distance from the imaging system to the cylinder 100a is changed between 2 m and 40 cm in front of the wall 100 b. It shows the relationship. The correlation distance between pixels at the time of obtaining the autocorrelation coefficient R (x, y) is 4 pixels. As shown in FIG. 5, as the distance D between the imaging system and the cylinder 100 a decreases, the correlation between each pixel P (x, y) and the adjacent pixel P becomes high, and each pixel P The value of the autocorrelation coefficient R (x, y) of (x, y) is high. Further, it is recognized that there is a correspondence between the autocorrelation coefficient R (x, y) and the distance D to the subject 100. Therefore, if the autocorrelation coefficient R (x, y) of each pixel P (x, y) of the third image data is calculated, the autocorrelation coefficient R (x, y) of each pixel P (x, y) is calculated. It is possible to grasp the distance D to the subject 100 based on

  Next, when the autocorrelation coefficient R (x, y) of each pixel P (x, y) is calculated, the distance acquisition unit 16 refers to the table stored and held in the table storage unit 17, The distance D to the part of the subject 100 corresponding to each pixel P (x, y) is acquired. In addition, the distance acquisition unit 16 performs coloring processing to color each pixel P (x, y) to be associated with the acquired distance D to generate distance image data, and the distance image data is monitored by the monitor 18. Output, and can be switched and displayed with a visible image described later.

  Here, the table shows the autocorrelation coefficient R (x, y) calculated for each pixel P (x, y) of the third image data, and the subject 100 corresponding to each pixel P (x, y) from the imaging system. The distance D to the part of the subject is measured in advance, and stored in the form in which the autocorrelation coefficient R (x, y) and the distance D to the subject 100 are associated with each other. In addition, as shown in the graph of the distance to the cylinder in FIG. 5, when there is a range where a linear correspondence can be recognized between the autocorrelation coefficient R (x, y) and the distance D to the subject 100. The distance acquisition unit 16 can calculate the distance D to the subject 100 based on the autocorrelation coefficient R (x, y) of each pixel P (x, y) without referring to the table. .

  As described above, the color filter 10b is provided on the sensor body 10a. Here, the Bayer-arranged color filter having red R, green G, and blue B regions but no infrared IR region has 16 regions of 4 × 4 serving as a basic pattern. Regions are G regions, 4 regions are R, and 4 regions are B. On the other hand, in the color filter 10b of this embodiment, as shown in FIG. 8, four R among the eight G regions in the Bayer arrangement are used as the IR region, There are four G, four B, and four IR. The color filters including the IR region are not limited to the color filter 10b shown in FIG. 8, and color filters of various arrangements can be used. However, it is necessary to include both the region of each color of visible light and the region of IR. In addition, although each region of RGB is a general RGB filter, since it has a peak of transmittance in the wavelength range of each color and has transmittance in the near infrared wavelength region, in FIG. The region of R is IR, the region of green is G + IR, and the region of blue is B + IR.

  The transmittance spectra of the R region, the G region, and the B region in the present embodiment are as shown in the graph of FIG. That is, the transmittance spectra of the red (R), green (G), blue (B), and infrared (IR) filters of the color filter 10b are shown, the vertical axis indicates the transmittance, and the horizontal axis is the horizontal axis. It is a wavelength. The wavelength range in the graph includes a part of the visible light band and the near infrared band, and indicates, for example, a wavelength range of 300 nm to 1100 nm.

  For example, as shown by R (double line) in the graph, the R region has a substantially maximum transmittance at a wavelength of 600 nm, and the long wavelength side maintains a substantially maximum transmittance even if it exceeds 1000 nm. It will be in the The region G has a peak at which the wavelength reaches a maximum of about 540 nm, as shown by G in the graph (long broken line), and the peak of the transmittance at the long wavelength side of about 620 nm has a minimum There is a part that becomes. Further, in the G region, the longer wavelength side tends to increase from the portion where the transmittance becomes minimum, and the transmittance becomes substantially maximum at about 850 nm. On the longer wavelength side, the transmittance is substantially maximized even if it exceeds 1000 nm. The region B has a peak at which the transmittance is maximum at a wavelength of about 460 nm as shown by B (a narrow broken line in the graph) of the graph, and the peak of about 630 nm on the longer wavelength side is transmitted. There is a part where the rate is minimal. Further, the longer wavelength side tends to increase from that, and the transmittance is substantially maximized at about 860 nm, and the transmittance is substantially maximized at more than 1000 nm on the longer wavelength side. The region of IR blocks light on the short wavelength side from about 780 nm and blocks light on the long wavelength side from about 1020 nm, and the portion of about 820 nm to 920 nm has a substantially maximum transmittance.

  The transmittance spectrum of each of the R, G, B, and IR regions is not limited to that shown in FIG. 9, but the color spectrum 10b generally used at present is a transmittance spectrum close to this. It seems to show. In addition, 1 of the vertical axis | shaft which shows the transmittance | permeability does not mean transmitting 100% of light, but in the color filter 10b, it shows the largest transmittance | permeability, for example.

The DBPF 7d has a transmission characteristic in the visible light band, has a blocking characteristic in the first wavelength band adjacent to the long wavelength side of the visible light band, and is a second wavelength band that is a portion within the first wavelength band Is an optical filter having transmission characteristics.
As shown in the graph of FIG. 9, the DBPF 7 d has a visible light band indicated by DBPF (VR) and a DBPF at a position slightly away from the visible light band on the long wavelength side as indicated by DBPF (solid line) in the graph. The transmittance of the two bands in the infrared band (second wavelength band) indicated by (IR) is high. Moreover, DBPF (VR) as a zone | band where the transmittance | permeability of a visible light zone | band is high has a wavelength zone | band of about 370 nm-700 nm, for example. Further, DBPF (IR) as a second wavelength band having a high transmittance on the infrared side is, for example, a band of about 830 nm to 970 nm.

In the present embodiment, the relationship between the transmittance spectrum of each region of the color filter 10b described above and the transmittance spectrum of the DBPF 7d is defined as follows.
That is, DBPF (IR), which is a second wavelength band that transmits infrared light of the transmittance spectrum of DBPF 7d, has almost maximum transmittance in all of the R, G, and B regions. It is included in the wavelength band A shown in FIG. 9 in which the transmittance is substantially the same in the region, and is included in the wavelength band B that transmits light with the substantially maximum transmittance in the region of IR.

Here, the wavelength band A in which the transmittances of the R, G, and B regions are substantially the same is a portion where the difference in transmittance of each region is 20% or less, preferably 10% or less in transmittance. Do.
Note that on the shorter wavelength side than this wavelength band A (wavelength band C), the transmittance of the G and B regions is lower than the region of R where the transmittance is substantially maximum. In DBPF 7d, the portion where there is a difference in transmittance in each of the R, G, and B regions is DBPF (VR), which is a portion with high transmittance in the visible light band, and the second wavelength band in the infrared light band. This corresponds to a portion where the transmittance substantially blocking the light of the DBPF 7 d between the DBPF (IR) which is a portion with high transmittance is minimized. That is, on the infrared side, transmission of light in a portion where the difference in transmittance in each of the R, G, and B regions is large is cut, and the transmittance in each region is substantially maximized on the longer wavelength side than that The light is transmitted in the wavelength band A in which is substantially the same.

  From the above, in the present embodiment, in the DBPF 7d used instead of the infrared light cut filter, there is a region where light is transmitted not only in the visible light band but also in the second wavelength band on the infrared light side. Therefore, in the case of color photography with visible light, it is affected by the light passing through the second wavelength band, but as described above, the second wavelength band has different transmittance in each of R, G, and B regions Only the light of the wavelength band where the transmittance of each region is substantially maximum and the transmittance is substantially the same is not transmitted.

  In the second wavelength low region of the DBPF 7d, light in a portion where the transmittance is substantially maximum in the IR region is transmitted. Therefore, assuming that regions of R, G, B, and IR are respectively provided in four pixels in close proximity to each other irradiated with substantially the same light, in the second wavelength band, the region of R, G In the area B, the area B, and the area IR, the light passes in the same manner. As the light on the infrared side, the light of approximately the same light quantity in each area including the IR It will lead to the photodiode. That is, the amount of light passing through the second wavelength band on the infrared side among the light passing through the R, G, and B filters is the same as the amount of light passing through the IR region. Under the assumption described above, basically, the filter signal of the pixel assumed as described above from the sensor main body 10a that has received the light transmitted through the R, G, and B filters and the IR filter is transmitted. The difference from the above-described assumed output signal of the pixel from the sensor main body 10a that has received the light is each of R, G, and B obtained by cutting the light on the infrared side that has passed through each of the R, G, and B regions. It becomes an output signal of the visible light part.

  In practice, as shown in the pattern of the color filter 10b, one of R, G, B and IR regions is disposed for each pixel of the sensor body 10a, and each color irradiated to each pixel is It is highly likely that the amount of light of each of the lights will be different. For example, in each pixel, the brightness of each color of each pixel is determined using a known interpolation method (interpolation method), The differences between the luminances of R, G, and B of the pixel and the luminance of IR similarly interpolated can be used as the luminances of R, G, and B, respectively. The image processing method for removing the infrared light component from the luminance of each color of R, G and B is not limited to this, and finally the second wavelength band is passed from each luminance of R, G and B. Any method can be used as long as the method can cut off the influence of light. In any of the methods, DBPF 7d cuts the portion where the transmittance of the R, G, and B regions on the infrared side differs from 20% (preferably 10%), ie, the portion where the transmittance differs from a predetermined ratio Therefore, in each pixel, the process of removing the influence of infrared light is facilitated.

  When infrared light imaging is used at night, as in the case of visible light, the amount of light at night is insufficient even at infrared light, so infrared light illumination is required. The transmittance spectrum of DBPF 7d shown in FIG. 9 takes into consideration the transmittance spectrum of each of R, G, B, and IR, and the light emission for infrared light illumination, for example, the emission spectrum of infrared light LED for illumination. decide.

  In such an imaging sensor, the second wavelength band transmitting light on the infrared side of the DBPF 7d is on the infrared side of each of the R, G, B, and IR regions, and the transmittance of each region is substantially maximum. Thus, it is included in the wavelength band A in which the transmittance of each region is substantially the same, and is included in the wavelength band B in which the transmittance of the IR region is substantially maximum. In other words, on the longer wavelength side than the visible light band, the transmittances of the R, G, and B filters are substantially maximized by the region of R, and the transmittances of the G and B regions are not substantially maximized. Thus, the light of different portions is not cut by the DBPF 7 d without the transmittances of the R, G, and B regions being substantially the same.

  That is, in each of the R, G, B, and IR regions, the light of the second wavelength band is transmitted on the infrared side, and the transmittance on the infrared side in each of the regions is substantially the same. If the light in the wavelength band of (1) is irradiated with the same amount of light, the amount of transmitted light in each of the R, G, B, and IR regions becomes the same. Thereby, as described above, the color based on the output signal from the pixel corresponding to each of R, G, and B is corrected, and the influence by the infrared light passing through the second wavelength band of the color at the time of color photographing is A suppressed image can be easily obtained.

  Further, by making the second wavelength band correspond to the peak of the emission spectrum of the infrared light illumination included in the above-mentioned wavelength band A and wavelength band B, the light of the infrared illumination can be efficiently used, By narrowing the width of the two wavelength bands, the influence of infrared light passing through the second wavelength band can be reduced at the time of color photographing.

  That is, by using the DBPF 7d, it is possible to perform highly accurate correction by subtracting the value of the IR signal from the value of each signal of RGB of the image sensor 10. For example, as shown below, the light reception components of the pixels of each color of the image sensor 10 are in a state in which the components of the IR are added to the components of each color.

R pixel R + IR
G pixel G + IR
B pixel B + IR
IR pixel IR

  Therefore, as described below, IR correction is performed to remove the IR component from the light reception component of each pixel of RGB excluding the IR pixel.

R signal (R pixel output)-(IR pixel output) = (R + IR)-IR = R
G pixel (R pixel output)-(IR pixel output) = (G + IR)-IR = G
B pixel (R pixel output)-(IR pixel output) = (B + IR)-IR = B
As a result, it is possible to exclude the IR component transmitting through the DBPF 7 d and transmitting through the color filter from the area of each color other than the IR of the color filter.

The signal processing unit 21 performs the synchronization processing (interpolation processing) as described above, and in the image data as a moving image, in each frame, a frame of all pixels is an R frame, a G frame, a B frame, IR Frame is generated. That is, in the color filter, there are only four R, G, B, and IR pixels in the 4 × 4 basic pattern described above, but this is interpolated by interpolation processing, and all pixels of the basic pattern are R Generate an image, an image of G, an image of B, and an image of IR.
Also. The signal processing unit 21 performs image processing such as gamma correction, white balance, and RGB matrix correction on the visible image signal as described above.

  The signal processing unit 21 outputs a visible image signal 22 composed of RGB signals from which IR components have been removed and an infrared image signal 23 composed of IR signals. The visible image signal 22 is output to the monitor 18 as a color image, for example. In addition, the infrared image signal 23 is output to the control device 4 as described above and used for measuring the distance.

  FIG. 6 is a schematic flowchart showing the distance measurement method of the imaging system, and the distance measurement method will be described with reference to this flowchart. FIGS. 7A to 7D are explanatory diagrams of first image data, second image data, third image data, and distance image data, respectively. In each of FIGS. 7A to 7D, the subject 100 is the cylinder 100a and the wall 100b shown in FIG. Further, in each of FIGS. 7A to 7D, the image data at the left end is obtained by imaging the first state in which the cylinder 100a is disposed at a position close to the wall 100b, and the image data at the center Is obtained by imaging the second state in which the cylinder 100a is separated from the wall 100b and disposed at a position closer to the imaging system than the first state, and the image data of the right end further corresponds to the cylinder 100a from the wall 100b It is obtained by imaging the third state placed at a position closer to the imaging system than the second state.

  When measuring the distance D to the object 100, the imaging device 3 starts imaging by the CCD image sensor 10, and the pattern projection device 2 projects a predetermined projection pattern in synchronization with the odd field of the CCD image sensor 10. . As a result, as shown in FIG. 7A, the CCD image sensor 10 acquires the first image data of the subject 100 in a state in which the predetermined projection pattern is projected as an odd field. Further, as shown in FIG. 7B, the second image data of the subject 100 in a state in which the predetermined projection pattern is not projected is acquired in the even field (step ST1).

  When the first image data and the second image data are acquired, the differential image data acquisition unit 14 subtracts the second image data from the first image data, and as shown in FIG. 7C, a predetermined projection pattern is obtained. The third image data including only the reflected light is acquired (step ST2).

  When the third image data is acquired, the autocorrelation coefficient acquisition unit 15 compares the autocorrelation coefficient R (x, y) between the pixels of the third image data with each pixel P (x, y) of the third image data. y) is calculated (step ST3). Also, when the autocorrelation coefficient R (x, y) of each pixel P (x, y) is calculated, the distance acquisition unit 16 calculates a table based on the calculated autocorrelation coefficient R (x, y). Referring to the distance D to the subject 100 is acquired (step ST4). Thereafter, each pixel P (x, y) is colored to correspond to the distance D, and distance image data is generated as shown in FIG. 7D and can be output to the monitor 18 (step ST5). .

  In such an imaging system, since the distance D to the subject can be measured based on the first image data and the second image data obtained by imaging the subject 100 from one viewpoint, the imaging system has the imaging device 3 You only need to have one. Further, in order to obtain the first image data and the second image data, it is sufficient to form a state in which the projection pattern is projected on the subject 100 and a state in which the pattern is not projected on the subject 100. There is no Therefore, the pattern projection apparatus 2 can be made into a simple structure. Furthermore, it is not necessary to use a special image sensor at the time of imaging, and a general-purpose CCD image sensor 10 can be used. Therefore, the manufacturing cost of the imaging system can be reduced.

  Further, according to the present embodiment, it has a table that stores and holds the autocorrelation coefficient R (x, y) of each pixel P (x, y) and the distance D to the subject 100 in association with each other. The distance acquiring unit 16 acquires the distance D to the subject 100 with reference to the table based on the autocorrelation coefficient R (x, y) of each pixel P (x, y). Therefore, regardless of the projection pattern projected by the photomask 6, the distance D to the subject 100 can be acquired.

  Furthermore, in this example, since the distance D is acquired for each pixel P (x, y) of the third image data, the shape of the subject 100 can be grasped.

  Further, according to this example, a part of the optical path of the projection light from the pattern projection device 2 to the subject 100 and a part of the optical path of the reflected light from the subject 100 to the imaging device 3 are common. Therefore, the imaging system can be easily miniaturized.

Furthermore, in the present example, since the projection illumination angle of view and the imaging angle of view are the same and overlap, it is possible to suppress the occurrence of a shadow on the subject 100 when acquiring the first image data. Therefore, it is possible to accurately acquire the third image data including only the reflected light of the predetermined projection pattern.

  Further, in this example, the pattern projection device 2 projects the predetermined projection pattern in synchronization with the odd field of the CCD image sensor 10, and the imaging device 3 acquires the first image data in the odd field and The second image data is acquired. Therefore, it is possible to acquire the first image data and the second image data substantially in real time. Also, the infrared image of the subject 100 can be output from the monitor 18 using the second image data.

  Furthermore, the present example is provided with a photomask moving mechanism 9 for moving the photomask 6 in the optical axis direction by the pattern projection device 2. Therefore, by moving the photomask 6 to change the position of the imaging point of the projection pattern, it is possible to adjust the sensitivity to the distance at the time of acquisition of the first image data.

  In addition, since the light source 5 of the pattern projection apparatus 2 for projecting a predetermined projection pattern emits near-infrared rays, the projection pattern to the person being the subject 100 or to the person positioned in the vicinity of the subject 100 The distance D to the subject 100 can be measured without being aware of the projection of the image.

  Moreover, although it becomes possible to perform imaging of a visible image and measurement of the distance by an infrared image with one imaging system, at this time, there is one optical system and one imaging element and DBPF 7d is used However, since the infrared cut filter is not used, the imaging system can be manufactured more compact and at low cost. Basically, by displaying the distance on the visible image, it becomes clear which object the distance is being measured, and it is possible to reduce an error in distance setting due to a false recognition. Further, since the distance to each pixel of the infrared image can be obtained, the distance to each pixel of the visible image can be known. The position of each pixel in three dimensions can be known from the visible image and the distance to each pixel, and it becomes possible to create a 3D image. In addition, by image recognition of three-dimensional movement of each pixel, in the operation input of the electronic device by hand movement and body movement, without using two cameras, it approaches in the depth direction (longitudinal direction, away direction) Direction) can be used.

  Further, if the imaging position and imaging direction of the imaging device can be detected by the GPS and the compass sensor, the position of the subject can be specified. For example, the position of the subject can be displayed on a visible image. Also. In a drive recorder for capturing and storing an image in front of a car, it is possible to record an inter-vehicle distance from the front car.

  In the above example, the first image data obtained by imaging the subject 100 in a state where the projection pattern is projected is acquired as an odd field, and the subject 100 is imaged in a state where a predetermined projection pattern is not projected. The second image data obtained from the first image data is acquired from the even field adjacent to the odd field from which the first image data was acquired, but the first image data and the second image data are not adjacent fields but discrete fields It does not matter if it is obtained from.

  Further, in the above example, the imaging device 3 is equipped with the CCD image sensor 10 of the interlace system as the imaging element, but the first image data and the second image data are acquired using the image sensor of the progressive scanning method. May be Alternatively, a CMOS image sensor may be used.

  Furthermore, in the above example, the autocorrelation coefficient is calculated for all pixels, but depending on the size of the object to be measured, the distance of only the representative point may be calculated. In that case, for discrete pixels An autocorrelation coefficient may be calculated.

  Although the light source 5 emits a light beam having a wavelength of 0.85 μm in the above example, the wavelength of the light emitted from the light source 5 is not limited to this, and the CCD image sensor 10 or CMOS type In the case of near infrared light of a wavelength that can be imaged by the image sensor, for example, a light beam with a wavelength of 0.7 to 1.1 μm, it can be appropriately selected. Here, even in the case of a general image sensor distributed as a general-purpose product, the wavelength sensitivity is extended to about 1.0 μm. In addition, when acquiring 1st image data using the light ray with a longer wavelength, it is desirable to make light source intensity | strength of the pattern projector 2 sufficient. In addition, it is desirable to presuppose use in an environment where a sufficient SN ratio can be obtained. Here, it is needless to say that, when near infrared light is used, it is necessary to appropriately change the wavelength selectivity of the infrared blocking filter disposed in front of the light receiving surface of the image sensor. There is also an image sensor which has enhanced sensitivity to infrared wavelengths and a wider wavelength range, and when it is used, a light source with a longer wavelength can be used. Since background light of the same wavelength is always present no matter which wavelength is selected, it is an effective means to calculate the difference between the first and second image data described above in order to remove the background light.

  Furthermore, the predetermined projection pattern projected by the photomask 6 may be a regular projection pattern such as a stripe. Also, the projection pattern may be different depending on the subject 100 and the imaging environment. In this case, instead of the photomask 6, a spatial modulation element that transmits or reflects a part of the light beam from a light source such as a transmissive liquid crystal panel, a reflective liquid crystal panel, or a digital micromirror device is used. , The projection pattern can be changed.

  Further, in applications where accuracy is not required, it is not necessary to make the optical paths of the pattern projection device 2 and the imaging device 3 common, and it goes without saying that optical systems independent of each other may be used.

  Next, an imaging system according to a second embodiment of the present invention will be described. In the imaging system of the second embodiment, the configuration of part of the color filter 10b, the method of removing the IR component from each signal of RGB in the signal processing unit 21, and the IR component in the first embodiment. However, the other configuration is the same as that of the first embodiment, and the color filter 10b and the method of removing the IR component will be described below.

  In the present embodiment, for example, as shown in FIG. 10, in the color filter 10be (arrangement 1 of RGBC), two out of the four B of the color filter pattern of the Bayer arrangement described above are C, and four are Two of R are C, and four of eight G are C. That is, in the basic arrangement of 4 rows and 4 columns, the color filter 10be has four G filter units, eight C filter units, and four R filter units of R, G, B, and C. While two filter units and two filter units of B are arranged, filter units of the same type are arranged so as not to be adjacent to each other in the row direction and the column direction. Therefore, the eight C filter sections are arranged in a checkered pattern. Here, C indicates a transparent state as a clear filter section, and basically has transmission characteristics from the visible light band to the near infrared wavelength band, and here, the visible light band And C = R + G + B in FIG. C to be clear can be called white light, that is, white (W) because three colors of RGB are transmitted, and C = W = R + G + B. Therefore, C corresponds to the light quantity of substantially the entire wavelength band of the visible light band.

  Here, as shown in the graph showing the transmittance spectra (spectral transmission characteristics) of the color filter 10b and the DBPF 7d in FIG. 11, there are transmission characteristics on the long wavelength side of the visible light band in each of R, G and B filter parts Also in the filter section C which is a clear filter section, light is transmitted in the long wavelength side of the visible light band. On the other hand, by using DBPF 7d, as in the first embodiment, the infrared light transmitting on the longer wavelength side than the visible light band is limited to the second wavelength band, R, The amount of light passing through the G, B, and C filter sections and the DBPF 7 d becomes almost the same (approximate) in each of the R, G, B, and C filter sections, and in the visible light band, the R, G, B, and C filters Transmission characteristics differ depending on the wavelength of the part.

  Also in the first and second embodiments, the infrared rays transmitting the longer wavelength side than the visible light band are limited to be the second wavelength band, and R, G, B, IR The amount of light passing through the filter unit and the DBPF 7d is almost the same in the R, G, B, and IR filter units, and in the visible light band, the transmission characteristics according to the wavelengths of the R, G, B, and IR filter units are It is different.

  As a result, even in the third embodiment, IR correction of each pixel can be accurately performed, and a visible image with high color reproducibility can be generated. That is, it does not include the filter part of the above-mentioned IR having the blocking characteristic in substantially the entire wavelength range of the visible light band as in the first embodiment and the transmission characteristic in the infrared side longer than the visible light band. However, by providing the C filter unit, the IR signal can be calculated by the following equation.

In the following description, C (W), R, G, B, and IR indicate the levels of the output signal from the image sensor 10, but C (W), R, G, and B indicate the levels of the visible light band. Indicated and shall not contain infrared components.
Here, the color filter 10 b is designed as C = W ≒ R + G + B, and an IR signal to be removed from each of RGB signals is IR ′,
IR '= ((R + IR) + (G + IR) + (B + IR)-(C + IR)) / 2 = IR + (R + G + B-C) / 2
IR '≒ IR. In addition, IR shows the actual value calculated | required by measurement etc., and IR 'shows the value calculated | required by calculation. By subtracting IR ′ from each filter, IR correction can be performed.
That is,
R filter (R + IR):
R '= (R + IR) -IR' = R- (R + G + B-C) / 2
G filter (G + IR):
G '= (G + IR) -IR' = G- (R + G + B-C) / 2
B filter (B + IR):
B '= (B + IR) -IR' = B- (R + G + B-C) / 2
C (= W) filter (W + IR):
W '= (C + IR) -IR' = C- (R + G + B-C) / 2
It becomes.

  As a result, even if a clear C filter is used instead of the IR filter in the color filter 10b, the IR transmittance of each filter can be approximated by the DBPF 7d, and the IR component is determined as described above. By removing this from the signal of each filter section, color reproducibility can be improved.

  In addition, such calculation is performed by obtaining R + IR, G + IR, B + IR, and C + IR in each pixel, for example, by interpolation as described above, and the above calculation is performed in each pixel. Although C = W ≒ R + G + B is designed, it is not necessary to exactly conform to this formula almost exactly, and if it is approximate, there may be a difference due to an error or other factors, for example, 10 There may be a deviation of about%.

  In addition, since C becomes R + G + B, a large amount of light received by the pixel is likely to be saturated, so that the amount of light received in the visible light band can be reduced in the filter portion of C, or the infrared wavelength band and visible light band The amount of light received may be reduced over a wavelength band including the above, or the charge accumulated with respect to the amount of light received may be reduced in the element portion constituting the pixel in each pixel. In that case, it is necessary to change the above-mentioned formula accordingly.

FIG. 12 shows another arrangement of R, G, B and C color filters, in which R, G, B and C are equally arranged one by one in a 2 × 2 arrangement. is there.
Further, in the case of the conventional Bayer arrangement of R, G, and B, one R and one B and two Gs are arranged in the 2 × 2 array as shown in FIG.
In addition, as shown in FIG. 14, a 2 × 2 array of R, G, B, and IR color filters in which one G of the conventional C- and IR-free sequences is changed to IR, as shown in FIG. It becomes an array in which G, B and IR are arranged one by one.

As such color filters, configuration 1 of RGB-C shown in FIG. 10, configuration 2 of RGB-C shown in FIG. 12, conventional RGB arrangement (Bayer arrangement) shown in FIG. 13, RGB-IR arrangement shown in FIG. In one example, there is a difference in characteristics as shown in FIG. C does not include information of colors such as RGB, but includes information of luminance as a light amount.
Therefore, the RGB-C (Constitution 1) sensor has a high luminance resolution due to the checkered arrangement of C, but the RGB pixels are sparse and asymmetric arrangement, so the resolution is low and moiré is likely to occur. However, there is no problem in the color signal because the resolution required for the luminance signal is 1/2 or less and lower. Also the sensitivity is high.

RGB-C (Configuration 2) has the same luminance resolution and color resolution as the conventional RGB sensor, and the sensitivity is higher than that of the RGB sensor. The sensitivity is higher in the visible image and the resolution of the luminance is also higher in the visible image than in the RGB-IR sensor in which the IR having no transmission characteristic in the visible light band is provided.
That is, the color filter having C is more likely to be advantageous in resolution and sensitivity than the color filter having IR according to the first embodiment described above.

  Next, an imaging sensor and an imaging apparatus according to a third embodiment of the present invention will be described. The third embodiment is a generalization of each color of the color filter, and shows that the color filter of the present invention is not limited to RGB-IR or RGB-C. Below, the removal method of IR component in the imaging sensor provided with the color filter which has a four-color filter part generalized is demonstrated. The four-color (four types) filter unit basically has different transmission characteristics according to the wavelength in the visible light band, and the visible light band in the wavelength band including the second wavelength band of the above-mentioned DBPF 7d. It has a third wavelength band in which the difference in transmittance with the other filter part on the longer wavelength side is 20% or less, preferably 10% or less, and in this third wavelength band, the second wavelength band of DBPF 7d Has been included. As a result, when the color filter and the DBPF 7 d are used, the transmission characteristics according to the wavelength on the infrared side of the visible light band are approximated by the filter units of the respective colors.

Furthermore, in the filter arrangement of four types of pixels, IR can be separated by designing a color filter under the following conditions.
In the filter arrangement, it is preferable that four kinds of filter parts A, B, C, and D be provided in a 2 × 2 arrangement as shown in FIG.
Moreover, it is preferable to design each filter part of A, B, C, and D so that the following relationship may be realized in the visible wavelength band as much as possible.
That is, in the visible light band,
KaA + KbB + KcC + KdD ≒ 0
I assume. In addition, A, B, C, and D show the level of the output signal from the image sensor 10 of the visible light zone | band of each filter part.

In the IR region on the longer wavelength side than the visible light band, the transmittance of IR is substantially constant in the above-described third wavelength band of each of the filter portions A, B, C, and D. The IR transmittance may be approximately an integral multiple of an IR transmittance in each of the A, B, C, and D filter portions. When designed in this way (here, the transmittance of IR is constant as described above),
Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR) ≒ IR (Ka + Kb + Kc + Kd)
Therefore, the IR signal is
IR '= (Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR)) / (Ka + Kb + Kc + Kd)
It can be calculated by
The IR component contained in each pixel of A, B, C and D can be corrected by the following calculation.
A '= (A + IR)-(Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR) / (Ka + Kb + Kc + Kd) = A- (KaA + KbB + KcC + KdD) / (Ka + Kb + Kc + Kd)
B '= B + IR- (Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR) / (Ka + Kb + Kc + Kd) = B- (KaA + KbB + KcC + KdD) / (Ka + Kb + Kc + Kd)
C '= C + IR- (Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR) / (Ka + Kb + Kc + Kd) = C- (KaA + KbB + KcC + KdD) / (Ka + Kb + Kc + Kd)
D '= D + IR- (Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR) / (Ka + Kb + Kc + Kd) = D- (KaA + KbB + KcC + KdD) / (Ka + Kb + Kc + Kd)
Here, the error is
It is (KaA + KbB + KcC + KdD) / (Ka + Kb + Kc + Kd). This error can be corrected in the RGB matrix.
In fact, since the transmittance of the IR component for each filter section is somewhat different, it is corrected with a coefficient-corrected signal as described below.

  A '= A + IR * KIRa-KIRa (Ka (A + IR * KIRa) + Kb (B + IR * KIRb) + Kc (C + IR * KIRc) + Kd (D + IR * KIRd)) / (Ka * KIRa + Kb * KIRb + Kc * KIRc + Kd * KIRd)

  B '= B + IR * KIRb-KIRb (Ka (A + IR * KIRa) + Kb (B + IR * KIRb) + Kc (C + IR * KIRc) + Kd (D + IR * KIRd)) / (Ka * KIRa + Kb * KIRb + Kc * KIRc + Kd * KIRd)

  C '= C + IR * KIRc-KIRc (Ka (A + IR * KIRa) + Kb (B + IR * KIRb) + Kc (C + IR * KIRc) + Kd (D + IR * KIRd)) / (Ka * KIRa + Kb * KIRb + Kc * KIRc + Kd * KIRd)

  D '= D + IR * KIRd-KIRd (Ka (A + IR * KIRa) + Kb (B + IR * KIRb) + Kc (C + IR * KIRc) + Kd (D + IR * KIRd)) / (Ka * KIRa + Kb * KIRb + Kc * KIRc + Kd * KIRd)

  The spectral transmission characteristic of each filter when DBPF is used is as shown in FIG. Although the filter section is an example using four types of filter sections of R + IR, G + IR, B + IR, and C + IR, the IR parts are constant or have an integral multiple of each other, such that KaA + KbB + KcC + KdD ≒ 0. As long as a color filter is designed, each filter section is not limited to R + IR, G + IR, B + IR, and C + IR.

  FIG. 17 shows the spectral transmission obtained by combining the filter unit of B and the DBPF 7 d, FIG. 18 shows the spectral transmission obtained by combining the filter unit of G and the DBPF 7 d, and the spectrum obtained by combining the filter unit of B and DBPF 7 d in FIG. FIG. 20 shows transmission, and FIG. 20 shows spectral transmission obtained by combining the filter part of C (W) and DBPF 7 d.

  Each spectral transmission characteristic is, as shown in the above-mentioned formulas, a sum of four transmissions of a visible R transmission region, a visible G transmission region, a visible B transmission region, and an IR transmission region. . From this, it is possible to calculate signal values of each visible R transmission region, visible G transmission region, visible B transmission region, and IR transmission region from the values of four or more types of filters. The spectral transmission characteristics are determined on the basis of the above-mentioned formulas showing each of the branch transmission characteristics of A, B, C, and D. Six combinations of spectral transmission characteristics of two of these filters are determined.

2 Pattern projection device (pattern projection means)
3 Imaging device 7d DBPF (optical filter)
10 Image sensor (imaging sensor)
10a Image sensor main body 10b color filter 14 difference image data acquisition unit (difference image data acquisition means)
15 Autocorrelation coefficient acquisition unit (Autocorrelation coefficient acquisition means)
16 distance acquisition unit (distance acquisition means)
17 Table storage unit (table storage means)
18 Monitor 21 Signal processing unit (signal processing means)
22 visible image signal 23 infrared image signal IR first wavelength band DBPF (IR) second wavelength band DBPF (VR) visible light band

Claims (6)

In the second wavelength band, which has a transmission characteristic in the visible light band, has a blocking characteristic in the first wavelength band adjacent to the long wavelength side of the visible light band, and is a portion within the first wavelength band An optical filter having transmission characteristics;
The imaging sensor main body in which the light receiving element is arranged for each pixel, and the transmission characteristics of the visible light band are different from each other, and the transmittances in the third wavelength band longer than the visible light band approximate each other An imaging sensor including four or more types of regions, and the regions including a color filter disposed in each pixel of the imaging sensor main body in a predetermined array;
An optical system having a lens that forms an image on the imaging sensor;
Based on the signals output from the imaging sensor, signals of components of plural colors of visible light and signals of components of infrared light corresponding to the second wavelength band are determined, and based on these signals, Signal processing means capable of simultaneously outputting a visible image signal and an infrared image signal of the same imaging range;
Pattern projection means including a light source of the infrared light and intermittently projecting a predetermined projection pattern onto a subject;
First image data captured when the predetermined projection pattern is projected is acquired from the infrared image signal output from the signal processing means, and imaging is performed when the predetermined projection pattern is not projected Differential image data acquisition means for acquiring acquired second image data and acquiring third image data including only reflected light of the predetermined projection pattern from the difference between the first image data and the second image data;
An autocorrelation coefficient acquisition unit that calculates an autocorrelation coefficient between pixels of the third image data for a predetermined predetermined pixel;
Distance acquisition means for acquiring the distance to the subject based on the autocorrelation coefficient of the predetermined pixel ,
The spectral transmission characteristics of the optical filter and the spectral transmission characteristics of the regions of each type of the color filter are such that the second wavelength band of the optical filter is included in the third wavelength band of the color filter An imaging system characterized by being set . A table storing and holding an autocorrelation coefficient of pixels of the third image data and a distance to the subject in association with each other;
The imaging system according to claim 1, wherein the distance acquiring unit acquires the distance to the subject with reference to the table based on an autocorrelation coefficient of the pixel.   2. A monitor according to claim 1, further comprising: a monitor for displaying a visible image based on the visible image signal, wherein the distance to the acquired subject is displayed corresponding to the subject displayed on the monitor. Imaging system as described.   At least two of the signal processing means, the difference image data acquisition means, the autocorrelation coefficient acquisition means, and the distance acquisition means may be configured as an electronic circuit on one chip to be one chip The imaging system according to claim 1, characterized in that: Before SL in the third wavelength band, imaging system of claim 1, the difference between each other of the transmittance of each type of the region is characterized that it is within 20% in the transmittance.   The color filters are different from each other in transmission characteristics of the visible light band and have transmission characteristics in the second wavelength band, and regions of three different color components according to the transmission characteristics of the visible light band, The imaging system according to any one of claims 1 to 5, further comprising: a white area.