JP6720971B2 - Video shooting device, video shooting method, and video shooting program - Google Patents

Description

The present invention relates to an image capturing device and an image capturing method.

In a photographing device such as a digital camera or a video camera, a red (R), green (G), and blue (B) three-color optical filter is usually incorporated in its image sensor. The light incident on the camera is decomposed by the three-color optical filter and converted into a video signal by the image sensor to generate RGB video data.

By the way, when the image sensor used in the image capturing device is a silicon-based sensor, it has sensitivity to light in the near infrared region as well as the visible light region. Therefore, when near-infrared light (Near InfraRed, NIR) is incident on the image sensor, the output by NIR is added to the output by each light of RGB. As a result, color reproducibility is reduced. Therefore, in a general digital camera or digital video camera, near-infrared light is removed by a near-infrared cut filter to ensure highly accurate color reproducibility. In a general definition, near-infrared light is light having a wavelength of approximately 0.7 to 2 μm.

On the other hand, there is a demand for taking a near-infrared light image by utilizing the above-mentioned near-infrared light sensitivity. In order to meet this demand, various methods of performing visible light photography and near-infrared light photography with a single image capturing apparatus have been studied.

The simplest method is to provide a mechanism for mechanically moving an IR cut filter that removes infrared light (InfraRed, IR). In this case, the IR cut filter is set in the optical system for visible light photography, and the near infrared light photography is performed by removing the IR cut filter from the optical system for photography outdoors at night or in a dark place. In this way, it is possible to obtain a visible light image with good color reproducibility and a near-infrared light image with one image capturing device.

In addition, a method of capturing an RGB image and an NIR image without requiring a mechanical operation has also been proposed. For example, Non-Patent Document 1 discloses a method of using a four-color optical filter in which an IR transmission filter that removes visible light and transmits IR is added in addition to an RGB three-color optical filter. That is, an IR dedicated pixel is provided to generate an IR signal. Then, using the generated IR signal, the contribution of IR is subtracted from the R, G, B sensor outputs, and the correct R, G, B signals are calculated. This ensures a high color reproduction line. Further, in photographing in a dark place, R, G, and B image sensors are used as near-infrared sensors to generate near-infrared light monochrome images.

Further, for example, Patent Document 1 discloses a method of measuring the intensity of light incident on a semiconductor photosensor in two steps, that is, measuring the intensity of visible light and measuring the intensity of near-infrared light. The visible light intensity is measured using a three-color optical filter that transmits each of R, G, and B colors. These optical filters also transmit NIR. Then, the NIR intensity is measured by the NIR sensor provided in the deep portion (the portion far from the light receiving surface) of each of the R, G, and B photosensors. This utilizes the phenomenon that light having a longer wavelength penetrates deeper into the surface of the semiconductor and is absorbed. That is, the intensity of the NIR light transmitted through the visible light sensing unit is measured by the NIR dedicated sensor. With this configuration, one image capturing device can capture both visible light images and near infrared light images.

JP, 2011-243862, A

Kayama, Tanaka, Hirose, “Image sensor for surveillance cameras used both day and night”, Panasonic Technical Journal Vol.54, No.4, Jan.2009

However, each of the above techniques has problems.

The technique of Non-Patent Document 1 is provided with a pixel dedicated to IR. Therefore, when applied to an image sensor having the same area, the number of RGB pixels or the pixel area is reduced as compared with the method of measuring only RGB three colors. As a result, there is a problem that the resolution or sensitivity is lowered.

Further, in the technique of Patent Document 1, the NIR sensor is provided below the photosensor portion for each of the RGB colors (on the side far from the surface). For this reason, there is a problem that the structure and process are more complicated and the manufacturing cost is higher than the structure in which one type of photosensor is arranged in a plane.

The present invention has been made in view of the above problems, and an object thereof is to provide an image capturing device capable of capturing both visible light images and near-infrared light images, while having a simple configuration. There is.

In order to solve the above-mentioned problems, an image capturing apparatus of the present invention generates an incident light image signal generating an incident light image signal according to the intensities of a visible light component and a near infrared light component of incident light incident on a light receiving surface. Means, near-infrared light diffracting means arranged on the incident side of the light-receiving surface for diffracting the near-infrared light component of the incident light, and diffraction for extracting a diffraction pattern generated by the near-infrared light on the light-receiving surface. Pattern extraction means, near-infrared light image signal generation means for generating a near-infrared light image signal according to the intensity of the near-infrared light component calculated based on the diffraction pattern, the incident light image signal and the near-infrared light image signal And a visible light image signal generating means for generating a visible light image signal corresponding to the intensity of the visible light component calculated based on the near infrared light image signal.

An advantage of the present invention is that it is possible to provide an image capturing device capable of capturing an image with both visible light and infrared light with a simple configuration.

It is a block diagram which shows the video imaging device of 1st Embodiment. It is a block diagram which shows the video imaging device of 2nd Embodiment. It is a graph which shows an example of the brightness|luminance distribution of the diffraction pattern by a near infrared light. It is a perspective view which shows the geometric information of a near-infrared light diffracting means and an incident light image signal generation means. It is a graph which shows the example of the center distance and intensity|strength in a near infrared light diffraction model. It is a top view showing an example of distribution of a circular pattern on a photosensor array. It is a figure which shows the example of conversion from rectangular coordinates to polar coordinates. It is an example of a histogram regarding a brightness gradient ratio. It is an example of a histogram regarding a luminance gradient ratio and a wavelength of near-infrared light. It is a flowchart which shows a near-infrared light component estimation operation. It is a block diagram which shows the video imaging device of 3rd Embodiment. It is a top view showing an example of a code type IR cut filter. It is a top view showing an example of a photo sensor array with which a Bayer array type color filter was incorporated. It is a flow chart which shows an outline of operation of a 3rd embodiment. It is a top view for explaining an example of demosaicing processing. It is a plane schematic diagram which shows an example of the video signal which added the circular pattern by a near infrared light to the visible light video signal. It is a block diagram which shows the video imaging device of 4th Embodiment. It is a block diagram which shows the video imaging device of 5th Embodiment. It is a block diagram which shows the video imaging device of 6th Embodiment.

Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing an image capturing device 100 according to the first embodiment. The image capturing apparatus 100 includes a near infrared light diffracting means 110, an incident light image signal generating means 120, a diffraction pattern extracting means 130, a near infrared light image signal generating means 140, and a visible light image signal generating means 150. ,have.

The near-infrared light diffracting means 110 diffracts the near-infrared light included in the incident light 10 that has entered the image capturing apparatus 100. The near infrared light diffracted by the near infrared light diffracting means 110 forms a diffraction pattern on the light receiving surface 120 a of the incident light image signal generating means 120.

The incident light image signal generation means 120 generates an incident light image signal according to the intensities of visible light and near infrared light.

The diffraction pattern extracting means 130 extracts the diffraction pattern formed by the near infrared light on the light receiving surface 120a based on the incident light image signal.

The near-infrared light image signal generation means 140 calculates the intensity of the near-infrared light incident on the image capturing device 100 based on the extracted diffraction pattern of the near-infrared light, and generates the near-infrared light image signal.

The visible light image signal generating means 150 generates a visible light image signal based on the incident light image signal and the near infrared light image signal. With the above configuration, it is possible to generate a near-infrared light image signal and a visible light image signal.

As described above, according to the present embodiment, it is possible to capture a visible light image and a near infrared light image by using one image capturing device having a simple configuration.

(Second embodiment)
In the present embodiment, details of the video image capturing apparatus 100 of the first embodiment will be described. FIG. 2 is a block diagram showing an image capturing device 100 according to the second embodiment.

The image capturing apparatus 100 includes a near infrared light diffracting means 110, an incident light image signal generating means 120, a diffraction pattern extracting means 130, a near infrared light image signal generating means 140, and a visible light image signal generating means 150. ,have.

The diffraction pattern extraction means 130 has a diffraction model generation means 131, a brightness gradient analysis means 132, and a suitable model determination means 133. Details of each element will be described later.

The near infrared light image signal generation means 140 analyzes the diffraction pattern extracted by the diffraction pattern extraction means 130, calculates the intensity of the incident near infrared light, and generates a near infrared light image signal.

The visible light image signal generating means 150 generates a visible light signal based on the incident image signal and the near infrared light image signal. Specifically, a visible light image signal can be obtained by subtracting the near infrared light image signal from the incident image signal in each pixel.

The details of each element will be described below.

The near-infrared light diffracting means 110 has a near-infrared light cutting portion that cuts near-infrared light and a near-infrared light transmitting portion that transmits near-infrared light. , Acts as a slit. The shape of the near-infrared light transmitting portion can be, for example, a circular shape or a shape close to a circular shape. The near-infrared light component of the incident light 10 that has entered the image capturing apparatus 100 is diffracted by passing through the near-infrared light diffracting means 110, and is bright and dark on the light receiving surface 120a of the incident light image signal generating means 120. Results in a diffraction pattern of When the near infrared light transmitting portion is circular, the diffraction pattern is a circular pattern composed of bright and dark concentric circles.

FIG. 3 is a graph showing an example of the luminance distribution of one circular pattern. Here, (x, y) is a rectangular coordinate in the incident light image signal generating means 120, and the z axis represents the luminance at that coordinate. This circular pattern can be approximated as Fraunhofer diffraction. The bright part in the center of the circular pattern is called the Airy disk.

The incident light image signal generating means 120 generates an incident light image signal corresponding to the intensities of visible light and near infrared light incident on the light receiving surface 120a. The incident light image signal generation means 120 is, for example, a photo sensor array in which photo sensors having sensitivity to visible light and near infrared light are used as pixels, and the pixels are two-dimensionally arranged.

The diffraction pattern extraction means 130 analyzes the incident light image signal and extracts the diffraction pattern formed by the near infrared light. For that purpose, it has a diffraction model generation means 131, a brightness gradient analysis means 132, and an adapted model determination means 133.

The diffraction model generation means 131 generates a diffraction model of near-infrared light by theoretical calculation, for example, Fraunhofer approximation.

The brightness gradient analysis unit 132 analyzes the brightness gradient at each coordinate of the incident light image signal. In the analysis, a process of calculating the ratio between the brightness gradient of the incident light image signal at a certain coordinate and the brightness gradient at the position of the generated diffraction model corresponding to the coordinate is performed. Details will be described later.

The adapted model determination means 133 determines a diffractive model adapted to the circular pattern included in the incident light image signal based on the diffractive model and the result of the brightness gradient analysis. Thereby, the diffraction pattern is extracted.

The near-infrared light image signal generation means 140 estimates the intensity and wavelength of the near-infrared light incident on the coordinates based on the matching diffraction model. A near infrared image signal is generated from the estimation result.

Next, a method in which the diffraction pattern extraction unit 130 extracts the diffraction pattern and the near-infrared light image signal generation unit 140 generates the near-infrared light image signal will be described.

Now, it is assumed that the near-infrared light having the wavelength λ and the incident intensity I ₀ is incident on one near-infrared light transmitting portion of the near-infrared light diffracting means 110. At that time, the intensity I(x) on the light receiving surface 120a of the incident light image signal generating means 120 is expressed by the following formula.

Here, J ₁ (x) is a Bessel function of the first kind of order 1, C is a correction coefficient, and x is as in Expression (2).

Here, a is the radius of the portion of the near-infrared light diffracting means 110 that transmits near-infrared light. q is the distance on the light-receiving surface 120a from the point perpendicular to the incident light image signal generating means 120 to the point from the center of the portion. R represents the distance from the point perpendicular to the light receiving surface 120a from the center of the portion that transmits near infrared light to the point on the light receiving surface 120a that is a distance q away. These pieces of geometric information are shown in FIG.

Now, the diffraction pattern that can be expressed by Expression (1) and Expression (2) by normalizing the incident intensity I ₀ will be referred to as a diffraction model.

FIG. 5 is an example of a diffraction model when calculated assuming a certain wavelength. The diffraction model represents the relationship between the intensity I of the circular pattern generated on the light receiving surface 120a and the distance q from the center of the circular pattern. Here, the correction coefficient C in Formula (1) shows two examples of C1 (solid line) and C2 (broken line). The correction coefficient C is a coefficient that is adjusted so as to match the circular pattern generated in the actual video signal.

Next, a method of estimating the near-infrared light component from the incident light image signal generated by the incident light image signal generating means 120 will be described. Here, it is assumed that the incident light image signal generating means 120 is a photosensor array 121 in which a plurality of pixels are two-dimensionally arranged with a photosensor having sensitivity to visible light and near infrared light as a pixel. Further, the near infrared light diffracting means 110 is assumed to generate a plurality of circular patterns on the photo sensor array 121. At this time, the circular patterns are prevented from overlapping each other. FIG. 6 schematically shows this state.

In estimating the near-infrared light component, first, as shown by the broken line in FIG. 6, the area of the incident light image signal is divided so that one circular pattern 20 is included. Next, with respect to the incident light image signal of each region, the intensity I(x, y) of the rectangular coordinate system is represented by the radius r and the angle of deviation θ with the center of the circular pattern 20 as a pole. Convert to (r, θ). FIG. 7 illustrates a process of converting an incident light image signal from a rectangular coordinate system to a polar coordinate system. The intensity I(x, y) can be restated as the brightness received by the incident light image signal generation means 120. Hereinafter, intensity and brightness will be used interchangeably.

Next, the brightness gradient in the center direction of the circular pattern is calculated for each coordinate (r, θ) in the polar coordinate system. The brightness gradient is a slope of brightness detected by an adjacent photosensor (pixel), and is defined for each coordinate where the pixel is located.

On the other hand, the brightness gradient is also calculated for the diffraction model. First, assuming a certain wavelength, a diffraction model pattern expressed by the equations (1) and (2) is created. Next, the gradient of intensity (which may be regarded as luminance) with the distance from the center as a variable is calculated. Here, since the diffraction model is concentric, the intensity is determined only by the distance from the center. Therefore, it is not necessary to consider the angular direction.

Next, a method of estimating the near infrared light component using the brightness gradient of the polar coordinate system will be described. The incident light image signal includes both a visible light component and a near infrared light component. Therefore, the absolute value of the brightness is greatly influenced by the visible light component. However, the influence of the visible light component can be suppressed by taking the brightness gradient in the polar coordinate system. Then, by obtaining the luminance gradient for all coordinates in the circular pattern and performing statistical processing, it becomes possible to extract the circular pattern generated only by the near-infrared light component.

Specifically, the brightness gradient of the incident light image signal in the polar coordinate system is calculated at predetermined intervals in all directions, and a diffraction model suitable for the brightness gradient is obtained. Then, the intensity and wavelength of near infrared light are estimated from the diffraction model. The method will be described below.

<Estimation of incident near infrared light intensity>
First, assuming a certain wavelength in the near-infrared light region, a diffraction model is created using equations (1) and (2). The unit of the intensity is arbitrary, but the normalized incident intensity I ₀ can be set to 255 by representing it by 8 bits, for example.

Next, the ratio b between the brightness gradient at each coordinate position (r, θ) of the incident light image signal and the brightness gradient of the diffraction model at the point where the distance from the center coincides is calculated. The brightness gradient ratio is calculated in all directions at predetermined intervals. Also, this is performed for all the pixels in the area.

Then, all the calculated brightness gradient ratios b are aggregated to create a histogram relating to the number of appearances of the brightness gradient ratio as shown in FIG. From the histogram, determining the intensity gradient ratio b _m indicating the most frequently. This brightness gradient ratio b _m can be regarded as equivalent to the ratio of the incident near infrared light intensity I ₁ and the maximum intensity I ₀ of the diffraction model pattern. Therefore, the equation (4) can be established. Then, the intensity I ₁ of the near infrared light estimated by the equation (5) is calculated.

As described above, the intensity of the incident near-infrared light component can be estimated. Then, by using the brightness gradient, it is possible to suppress the influence of a component other than the near-infrared component in the target region, that is, the discontinuity (edge or the like) of the brightness of the video signal due to visible light.

The above calculation process is performed for one area set to include one circular pattern. The same calculation is executed for all the regions of the incident light image signal to calculate the intensity of the near infrared component in each region.

<Estimation of incident near-infrared light wavelength>
Next, in order to estimate the wavelength of the incident near-infrared light, the calculation for obtaining the near-infrared light intensity similar to the above is performed using the wavelength λ as a parameter.

First, for a wavelength range that covers the wavelength range of near-infrared light, a diffraction model is generated based on equations (1) and (2). Here, the wavelength region covering the wavelength region of near-infrared light may be discretely set at intervals of 10 nm from 700 nm, for example. By this calculation, a diffraction model is obtained for each predetermined wavelength.

Then, the brightness gradient ratio between the incident light image signal and the diffraction model is calculated for one region set in the incident light image signal so as to include one circular pattern. Here, the ratio between the brightness gradient at each coordinate (r, θ) of the incident light image signal and the brightness gradient in the diffraction model having the same distance from the center is calculated based on the equation (3). Next, a histogram of the brightness gradient ratio is created. One histogram is created for one wavelength. Then, by changing the wavelength and creating a similar histogram, as shown in FIG. 9, histograms are created by the number of selected wavelengths.

Next, the created histogram is analyzed, and one or more combinations of wavelength and luminance gradient ratios are selected from the largest of the histograms. Then, the selected wavelength is substituted into the equation (5) to calculate the incident intensity of near infrared light in the target region. In this way, the wavelength and the incident light intensity of the near infrared light component in one region can be calculated. When there are a plurality of sets of wavelengths and brightness gradients to be selected, the incident light intensity of the near infrared light component may be calculated and added for each set.

The same calculation is performed for each area set to include one circular pattern with the incident light image signal, and the wavelength and the incident intensity of near infrared light in each area are calculated.

From the above, it is possible to calculate the wavelength and the intensity of the near-infrared light incident on all the near-infrared light transmitting parts provided in the near-infrared light diffracting means. That is, a near infrared light image signal can be generated.

FIG. 10 is a flowchart summarizing the near-infrared light image signal generation operation in one area described above. It should be noted that this flowchart shows the operation after performing the region division and the polar coordinate conversion. The flowchart will be described below.

First, a diffraction model is created assuming a certain wavelength in the near infrared light region (S1). Next, the brightness gradient ratio between the incident light image signal and the diffraction model is calculated for each coordinate (S2). Next, a histogram of the obtained brightness gradient ratio is created (S3). Next, the incident intensity of near-infrared light is estimated from the brightness gradient giving the mode value (S4). Next, a diffraction model is created with a plurality of predetermined wavelengths covering the near infrared light region (S5). Next, the brightness gradient ratio between the diffraction model of each wavelength and the incident light image signal is calculated at each coordinate (S6). Next, a histogram of the brightness gradient is created for each wavelength for which the brightness gradient ratio has been calculated (S7). Next, the wavelength of the incident near-infrared light is estimated from the brightness gradient showing the peak of the histogram (S8). From the above, the wavelength and intensity of near infrared light are estimated.

As described above, according to the present embodiment, one visible-light image and near-infrared-light image are captured by a single image capturing device using a photosensor array having a general simple configuration. can do.

(Third Embodiment)
FIG. 11 is a block diagram showing an image capturing device 100A according to the third embodiment of the present invention. The image capturing apparatus 100A of the present embodiment generates both a color image signal of visible light and a near infrared light image. The image capturing apparatus 100A has a code type IR cut filter 110a as a near infrared light diffracting means. Further, the incident light image signal generating means 120 includes a photo sensor array 121 in which a plurality of photo sensors are arranged in a plane, and a color filter array 122 provided on the light incident side of the photo sensor array 121. Further, it has a diffraction pattern extraction means 130, a near infrared light image signal generation means 140, and a visible light image signal generation means 150. R, G, B, and NIR in FIG. 11 represent red, green, blue, and near infrared signals, respectively. Here, arrows show how the photosensor array 121 generates R+NIR, G+NIR, and B+NIR signals, and the near-infrared light image signal generation means 140 generates NIR image signals. Also, the way in which the visible light image signal generating means 150 generates R, G, B signals is indicated by arrows. Details of the operation will be described later.

Next, a specific configuration of each element will be described.

FIG. 12 is a plan view showing the code type IR cut filter 110a. The coded IR cut filter 110a includes a near infrared light cut section 111 and a near infrared light transmission section 112. That is, the code type means a binary value of transmission and cut. The coded IR cut filter 110a is provided on the front side of the color filter array 122 in the light traveling direction. As a result, when the incident light passes through the infrared transmission section 12, diffraction of near infrared light occurs.

The near-infrared light cut unit 111 does not transmit near-infrared light but transmits visible light.

The near infrared light transmitting unit 112 transmits near infrared light. As described above, the near infrared light transmitting portion 112 has a circular shape or a shape close thereto. The circular patterns formed by the near-infrared light on the photosensor array surface should not overlap each other. Based on the equations (1) and (2), the approximate spread of the diffracted light can be predicted. The near-infrared light transmitting portion 112 is arranged so that the above condition is satisfied, and the distance between the code type IR cut filter 110a and the incident light image signal generating means 120 is set. The arrangement of the near-infrared light transmitting portion is arbitrary as long as this condition is satisfied.

By using such a code type IR cut filter 110a, a plurality of circular patterns having a predetermined size can be formed at a predetermined position on the light receiving surface of the incident light image signal generating means 120. Note that the size of the near-infrared light cut unit 112 does not necessarily have to match the size of one pixel of the photo sensor. Further, although it is desirable that the near-infrared light transmitting portion 112 also transmit visible light, the present embodiment is established even if visible light is not transmitted.

The photosensor array 121 is one in which photosensors are two-dimensionally arranged as in the second embodiment, and each photosensor (pixel) is sensitive to visible light and near infrared light.

As the color filter 122, for example, color filters that transmit R (red), G (green), and B (blue), respectively, are arranged in an array provided at positions corresponding to the photosensors. Further, each color filter also transmits near infrared light. The color filter may be of a type that uses complementary colors such as C (cyan), M (magenta), and Y (yellow). Here, an example of R, G, and B will be described.

FIG. 13 is a plan view showing an example of the configuration of the color filter array 122. As described above, each color filter also transmits near-infrared light in addition to visible light of the color transmitted by each color filter. In order to clarify this, in FIG. 13, each pixel is described in the form of “color+NIR”. R+NIR is a color filter transmitting red and NIR, G+NIR is transmitting green and NIR, and B+NIR is a color filter transmitting blue and NIR. In the incident light image signal generating means 120, a photo sensor is provided at a position corresponding to each color filter. Note that the number of photosensors corresponding to one color filter may be one or more. The array of color filters of each color shown in FIG. 13 is called a Bayer array, and one R pixel, two G pixels, and one B pixel form one unit, and this unit is arranged periodically. It was done. In FIG. 13, 6 units of vertical 2×horizontal 3 are illustrated.

The light transmitted through the code type IR cut filter 110a and the color filter 122 is converted by the photosensor into three color signals of R+NIR, G+NIR, and B+NIR.

With the above configuration, an incident light image signal in which a diffraction pattern of near infrared light by the code type IR cut filter 110a is added to each color is formed.

In this embodiment, similar to the second embodiment, the diffraction pattern extraction unit 130 extracts the above diffraction pattern, and the near infrared light image signal generation unit 140 generates the near infrared light image signal. To do. Details of the operation will be described later.

The visible light image signal generating means 150 generates R, G, B, and color signals based on the incident light image signal and the near infrared light image signal.

Next, the operation will be described. FIG. 14 is a flowchart showing the outline of the operation. The flowchart will be described below.

First, the incident light image signal generating means generates an incident light image signal (S101). The incident light image signal is an image signal of R, G, B, and three colors, and each signal includes a near infrared light component. Next, interpolation (demosaicing processing) of the missing color in each pixel is performed to generate progressive video signals of R+NIR, G+NIR, and B+NIR (S102). Here, the progressive video signal is defined as a video signal of each color obtained by the above demosaicing processing. Next, the progressive video signals of each color are analyzed in the same manner as in the second embodiment, and the near-infrared light video signals included in these are extracted and generated (S103). Next, based on each progressive video signal and near-infrared light video signal, a monochromatic video signal of R, G, B, or each color is generated (S104). As described above, it is possible to generate a visible light image signal of three colors and a near infrared light image.

Next, the details of each operation will be described.

<Demosaicing processing>
First, the demosaicing process for interpolating a missing color will be described. The demosaicing processing described below is merely an example, and other methods may be used.

FIG. 15 is a schematic plan view for explaining the demosaicing process performed by one unit in the Bayer array. One unit is composed of four R pixels, two G pixels, and one B pixel. Note that the pixel here is a set of a color filter and a photo sensor. Light of R+NIR, G+NIR, and B+NIR is detected in each pixel. Here, focusing on one pixel, no color other than the target color is detected. Therefore, the luminance of the peripheral pixels is used to interpolate the signal of the missing color. Note that the R, G, B color signals at this point also include NIR components, but in order to simplify the description, they will be described as R, G, B color signals.

The pixel at the coordinates (1, 1) corresponds to R and directly generates the R signal.

R(1,1)=R(1,1) (6)

The G signal and the B signal which do not exist in the pixel of the coordinates (1, 1) are calculated by interpolating from the color signals of the peripheral pixels as follows, for example.

G(1,1)=(G(2,1)+G(1,2))/2 (7)
B(1,1)=B(2,2) (8)

Next, the video signals (R, G, B color signals) of the pixel at the coordinates (1, 2) are generated.

The pixel at the coordinate (1,2) corresponds to G and directly generates the G signal.
G(1,2)=G(1,2) (9)

The R signal and the B signal that do not exist in the pixel at the coordinates (1, 2) are also calculated by interpolating from the color information of the peripheral pixels, as in the case of the R pixel.

R(1,2)=R(1,1) (10)
B(1,2)=B(2,2) (11)

The same processing as above is repeated to generate video data (R, G, B color signals) for all pixels. The demosaicing process is not limited to the above method, and various methods can be used. As described above, a progressive video signal in which R, G, and B color information is set in all pixels can be obtained.

Here, the visible light components R, G, and B are not affected (not diffracted) by the infrared transmission part 112 in the code type IR cut filter 110a. Therefore, the information of the shooting scene is directly applied to the color filter array 122 and the photo sensor array 121. Then, an incident light image signal composed of three color signals of R+NIR, G+NIR, and B+NIR is generated from the irradiated light. The progressive signal generated at this time, in which signals of R, G, B, and three colors are set in all pixels, includes a near-infrared light component in a circular pattern. Therefore, these R+NIR, G+NIR, and B+NIR signal components are represented by I _R+NIR , I _G+NIR , and I _B+NIR .

FIG. 16 is a schematic plan view showing progressive signals obtained for R, G, and B colors. The progressive R video signal 1R, the progressive G video signal 1G, and the progressive B video signal 1B each include a circular pattern of near infrared light. This is schematically shown in the figure.

<Generation of near infrared image signal component>
As described above, the progressive signals of R, G, B, and colors include a circular pattern of near-infrared light components. The estimation of each near infrared light component is performed in the same manner as in the second embodiment. That is, the same processing as that in the flowchart of FIG. 10 is performed using the R, G, and B progressive signals. As a result, three near-infrared light image signals obtained from the three progressive signals are extracted and generated. Here, the NIR signal obtained from the R progressive signal 1R will be referred to as NIR_R signal. Similarly, the NIR signal obtained from the G progressive signal 1G is called the NIR_G signal, and the NIR signal obtained from the B progressive signal 1B is called the NIR_B signal. Then, each signal component is represented by I _{NIR_R} , I _{NIR_G} , and I _{NIR_B} .

<Generation of visible light image signal and near infrared image signal>
In each pixel, R, G, B, and monochromatic color components can be obtained by subtracting the near-infrared light component from the progressive signal component of each color. When the respective color components are called I _R , I _G , and I _B , these components can be calculated by the following formulas.

I _R =I _{R +NIR} −I _{NIR_R} (12)
I _G =I _{G +NIR} −I _{NIR_G} (13)
_{I B = I B + NIR -I} NIR_B (14)

Further, the near-infrared light component monochromatic component I _NIR can be expressed by the following equation.

I _NIR =I _{NIR_R} +I _{NIR_G} +I _{NIR_B} (15)

It should be noted that each item of Expression (15) may be weighted to reflect the method of the demosaicing process.

As described above, R, G, B, NIR, and four color video signal components can be extracted from each pixel. Then, by using the signal components of all pixels, it is possible to generate the video data of the above four colors.

As described above, according to the present embodiment, a visible light color video signal and a near-light color video signal and a near-light color video signal are obtained with a video camera having a simple structure without mechanically moving the IR filter or using a special photo sensor. Infrared video signals can be generated at the same time.

<Modification>
In the above-described embodiment, the near infrared light component I _NIR is calculated using three progressive signals, I _R+NIR , I _G+NIR , and I _B+NIR , but it is also possible to calculate using a signal of a specific color. .. In particular, when the pixels are of the RGB Bayer array type, the information of the G channel is twice as large as that of the other color channels. Therefore, it is effective to use only the G channel after the demosaicing process.

That is, the progressive signal R component _{R + NIR,} the B component I _{B + NIR,} although strictly speaking include NIR information, ignore it because it is very small, I _{R + NIR} an R signal _I R, I _{Consider B+NIR} as B signal I _B. Then, the component I _NIR of only _NIR is calculated by the following equation.

I _NIR =I _{NIR_G} (16)

The video data I _G of the G signal is calculated by the equation (13).

The coded IR cut filter 1 can control the position where the circular pattern is formed. Therefore, if the near-infrared light transmitting part is arranged so that the circular pattern is formed centering on the G pixel, the NIR components contained in IR _+NIR and _{IB +NI} will be small, and even if these are ignored, there is an effect. Becomes smaller. Further, the influence can be further reduced by setting the shape and arrangement of the near-infrared light transmitting portion to about the size of one pixel of the color filter and the photo sensor.

(Fourth Embodiment)
FIG. 17 is a block diagram of the image capturing device 100B according to the fourth embodiment. The video image capturing apparatus 100B has a configuration in which a coded information memory 134 is added to the video image capturing apparatus 100A of the third embodiment. The encoded information memory 134 is connected to the diffraction pattern extraction means 130. Since the coded IR cut filter 110a, the color filter array 122, and the photo sensor array 121 are the same as those in the image capturing device 100A, the coded information memory 134 and the diffraction pattern extraction means 130 will be described here.

The coded information memory 134 records information about a circular pattern formed by near infrared light. Specifically, the center coordinates of the circular pattern, the size, and the distance between the coded IR cut filter 110a and the photosensor array 121 are recorded.

When the near-infrared light passes through the near-infrared light transmitting portion 112 arranged in the code type IR cut filter 110a, a circular pattern is formed on the photo sensor 3 by diffraction.

This circular pattern is determined by the wavelength of near-infrared light, the size of the infrared transmission part 12, and the distance between the code type IR cut filter 110a and the photosensor array 121. Therefore, if these parameters are determined, the center coordinates and the size of the circular pattern can be grasped in advance by performing calculation or calibration.

The coded information having the size of the infrared transmission part 12, the coordinate position of the center of the circular pattern on the photo sensor array 121, the size of the circular pattern, and the like, which is grasped by the above method, is coded information. Record in the memory 5. By using the recorded coded information, it is possible to estimate the position on the light receiving surface where the circular pattern is formed and determine the region for analysis.

With this configuration, a visible light image signal and a near infrared light image signal are generated. The generation method is the same as in the third embodiment. At this time, the processing can be simplified by using the coded information recorded in the coded information memory 134.

As described above, according to this embodiment, the same effect as that of the third embodiment can be easily obtained.

(Fifth Embodiment)
In the third and fourth embodiments, the image capturing device that performs the spectrum by the Bayer array type optical filter has been described, but the present invention is not limited to the Bayer array type. In the present embodiment, an example applied to a three-plate image capturing device is shown.

FIG. 18 is a block diagram of a three-plate type image capturing device 101. The image capturing device 101 includes a code type IR cut filter 110a, a prism 160, and three types of incident light image signal generating means 120R, G, and B. Further, it has a diffraction pattern extraction means 130, a near infrared light image signal generation means 140, and a visible light image signal generation means 150.

The prism 160 is a color separation unit, and in the example of FIG. 18, G light travels straight, B light travels downward in the drawing, and R+NIR light travels upward in the drawing. Note that the shape of the prism 160 drawn in FIG. 18 does not represent the actual shape, but is conceptual. Further, the prism 160 may include an optical system that collimates light, and a camera lens 170 may be provided on the incident side of the prism 160. The prism 160 and the camera lens 170 may be those generally used in the three-plate type image capturing device.

In the path of the light of each color, an incident light image signal generating means dedicated to that color is provided. The incident light image signal generating means 120G generates a G image signal. Incident light image signal generating means 120B generates a B image signal. Then, the incident light image signal generating means 120R generates an R+NIR image signal. A code type IR cut filter 110a is provided on the light incident side of 120R. With this configuration, the video signal generated at 120R includes a circular pattern of near infrared light. The photosensor used for each incident light image signal generating means may be a photosensor generally used in a three-plate image capturing device.

As the code type IR cut filter 110a, for example, the one used in any of the second to fourth embodiments is applied. The code type IR cut filter 110a is provided on the front side in the light traveling direction with respect to at least one of the three incident light image signal generating means. In the example of FIG. 18, it is provided corresponding to the incident light image signal generating means 120R corresponding to R. This causes diffraction of near infrared light.

A normal near-infrared cut filter is provided for the remaining two incident light image signal generating means not provided with the coded IR cut filter in order to cut near-infrared light that may leak from the prism 160. May be installed. Thereby, color reproducibility can be secured.

Next, the operation will be described. Here, it is assumed that the G and B lights split by the prism 160 do not include near infrared light.

Light incident on the image capturing apparatus 101 through the camera lens 170 is decomposed into R, G, and B lights having different wavelength bands by the prism 160. The light corresponding to R is incident on the incident light image signal generating means 120R, the light corresponding to G is incident on the incident light image signal generating means 120G, and the light corresponding to B is incident on the incident light image signal generating means 120B. ..

At this time, the light corresponding to R has its near-infrared light component diffracted by the near-infrared light transmitting portion of the code type IR cut filter 110a. Then, the incident light image signal generating means 120R generates an R+NIR image signal including NIR. That is, with the above configuration, an R+NIR video signal to which a plurality of circular patterns of near infrared light are added is generated.

The diffraction pattern extraction means 130 and the near infrared light video signal generation means 140 receive the R+NIR video signal and generate an NIR video signal. Then, the NIR video signal and the R+NIR video signal are output. The visible light image generating means 150 subtracts the NIR image signal from the R+NIR image signal to generate a monochromatic R image signal. As described above, 4-color video data (R, G, B, NIR) can be generated.

The video image capturing apparatus 101 of the present embodiment has a configuration of a general three-plate type image capturing apparatus and does not require a special device. Further, the coded IR cut filter 1 is a simple cut filter that is a simple modification and has a simple structure. That is, both a visible light image signal and a near infrared light image signal can be generated with a simple configuration. Therefore, reduction in production cost and reduction in failure rate can be expected as compared with the method using a complicated configuration or a special device. Further, even in a situation where near-infrared light is strongly saturated, it is possible to suppress saturation by dispersing near-infrared light by diffraction and widen the apparent dynamic range.

(Sixth Embodiment)
In the fifth embodiment, an example in which the present invention is applied to a three-plate type image capturing device has been shown, but the present invention can also be applied to an image capturing device having a laminated photo sensor.

FIG. 19 is a block diagram of the video image capturing apparatus 102 according to this embodiment.
The image capturing device 102 includes a coded IR cut filter 110a and a laminated incident light image signal generation unit 120S in which incident light image signal generation units 120R, 120G and 120B dedicated to R, G and B are laminated. ing. Further, it has a diffraction pattern extraction means 130, a near infrared light image signal generation means 140, and a visible light image signal generation means 150. As the laminated incident light image signal generating means 120S, for example, a laminated sensor used in a general laminated sensor type image capturing device can be used. In the example of FIG. 19, the photosensors are stacked in the order of 120B, 120G, and 120R in the light traveling direction.

As the code type IR cut filter 110a, the one used in any of the second to third embodiments can be applied. 110a is provided on the front side in the light traveling direction with respect to 120S.

The light incident on the image capturing device 102 includes R, G, B, and NIR lights having different wavelength bands. Light corresponding to B is converted to a B+NIR video signal by 120B, light corresponding to G is converted to a G+NIR video signal by 120G, and light corresponding to R and NIR is converted to an R+NIR video signal by 120R. These R, G, B+NIR video signals include a circular pattern formed by NIR diffracted by the coded IR cut filter 110a.

Then, as in the other embodiments, the circular pattern is extracted by the diffraction pattern extraction means 130, and the near infrared light image signal is generated by the near infrared light image generation means 130. That is, the NIR video signal is generated based on the R+NIR, G+NIR, and B+NIR video signals input from the laminated incident light video signal generation means 120S. The method is similar to the methods of the third and fourth embodiments.

With the above configuration, video signals of R, G, B, NIR, and four colors can be generated for all pixels.

As described above, according to the present embodiment, the same effect as that of the fifth embodiment can be obtained.

In the above embodiment, each unit may be configured by hardware or may be realized by a computer program. In this case, a processor operating with a program stored in the program memory realizes the same function and operation as described above. Further, only some of the functions may be realized by a computer program.

A program that causes a computer to execute the processes of the above first to sixth embodiments and a recording medium that stores the program are also included in the scope of the present invention. As the recording medium, for example, a magnetic disk, a magnetic tape, an optical disk, a magneto-optical disk, a semiconductor memory, or the like can be used.

The present invention has been described above using the above-described embodiment as an exemplary example. However, the present invention is not limited to the above embodiment. That is, the present invention can apply various aspects that can be understood by those skilled in the art within the scope of the present invention.

This application claims the priority on the basis of Japanese application Japanese Patent Application No. 2015-121588 for which it applied on June 17, 2015, and takes in those the indications of all here.
(Appendix 1)
Incident light image signal generating means for generating an incident light image signal according to the intensities of the visible light component and the near infrared light component of the incident light incident on the light receiving surface, and the incident light image signal generating means arranged on the incident side of the light receiving surface. Near-infrared light diffracting means for diffracting a near-infrared light component, diffraction pattern extracting means for extracting a diffraction pattern generated by the near-infrared light on the light-receiving surface, and the near-red light calculated based on the diffraction pattern Near-infrared light image signal generating means for generating a near-infrared light image signal according to the intensity of the external light component, and the visible light component calculated based on the incident light image signal and the near-infrared light image signal And a visible light image signal generating means for generating a visible light image signal according to the intensity of the image capturing device.
(Appendix 2)
The near-infrared light diffracting means includes a near-infrared light cutting portion that cuts near-infrared light and transmits visible light, and a near-infrared light transmitting portion that transmits near-infrared light. The image capturing device according to appendix 1.
(Appendix 3)
The image capturing apparatus according to appendix 2, wherein the near infrared light diffracting unit has a plurality of the near infrared light transmitting portions.
(Appendix 4)
4. The image capturing device according to attachment 2 or attachment 3, wherein the near-infrared light transmitting portion has a substantially circular shape.
(Appendix 5)
An additional information memory for storing information on the near-infrared light diffracting means and information on a position where the diffraction pattern is generated on the light receiving surface is added. The image capturing device described.
(Appendix 6)
The near-infrared light image signal generation means is a diffraction model generation means for generating a diffraction model of near-infrared light, a brightness gradient analysis means for analyzing a brightness gradient of the incident light image signal, and an analysis result of the brightness gradient. And an incident near-infrared light intensity estimating unit for estimating the intensity of incident near-infrared light based on the adaptive diffraction model. The image capturing device according to any one of appendices 1 to 5.
(Appendix 7)
7. The near-infrared light image signal generation means includes incident near-infrared light wavelength estimation means for estimating the wavelength of near-infrared light that has entered based on the adaptive diffraction model. Video shooting device.
(Appendix 8)
The brightness gradient analyzing means converts the incident light image signal into area dividing means for dividing the incident light image signal into areas including one of the diffraction patterns, and coordinates of the divided areas into polar coordinates having the center of the diffraction pattern as an origin. The image capturing device according to appendix 6 or 7, further comprising: coordinate conversion means.
(Appendix 9)
The brightness gradient analysis means calculates a brightness gradient ratio which is a ratio of a brightness gradient at a coordinate of the incident light image signal and a brightness gradient at a position corresponding to the coordinate of the diffraction model. 9. The image capturing device according to appendix 8, further comprising a calculating unit.
(Appendix 10)
10. The image capturing apparatus according to appendix 9, wherein the brightness gradient analysis means has a brightness gradient ratio statistical processing means for statistically processing the brightness gradient ratio generated at a plurality of coordinates in the divided area.
(Appendix 11)
11. The image processing apparatus according to appendix 10, wherein the adaptive diffraction model determination unit makes a determination based on a statistical processing result of the activation gradient ratio statistical processing unit.
(Appendix 12)
12. The image capturing apparatus according to any one of appendices 1 to 11, wherein the incident light image signal generation unit separates the incident light into visible light of a plurality of colors and generates a signal according to intensity.
(Appendix 13)
13. The image capturing device according to appendix 12, wherein the near-infrared light image signal generating means generates a near-infrared light image signal for each of the plurality of colors.
(Appendix 14)
14. The image capturing device according to any one of appendices 1 to 13, wherein the first incident light image signal generating means includes a silicon-based photo sensor.
(Appendix 15)
The near infrared light component of the incident light is diffracted by the near infrared light diffracting means arranged on the incident side of the light receiving surface, and the visible light component and the near infrared light component of the incident light incident on the light receiving surface are intensified. A near-infrared light corresponding to the intensity of the near-infrared light component, which is generated based on the diffraction pattern, is generated by generating an incident light image signal according to the near-infrared light and extracting a diffraction pattern generated on the light-receiving surface An image capturing method, comprising: generating an optical image signal, and generating a visible light image signal corresponding to the intensity of the visible light component based on the incident light image signal and the near infrared light image signal.
(Appendix 16)
Coded IR having a near-infrared light cut portion that cuts near-infrared light and transmits visible light, and a near-infrared light transmitting portion that transmits near-infrared light, in the near-infrared light component of the incident light. 16. The image capturing method described in appendix 15, wherein the image is diffracted by a cut filter.
(Appendix 17)
17. The image capturing method according to appendix 16, wherein the code type IR cut filter having a plurality of the near infrared light transmitting portions is used.
(Appendix 18)
18. The image capturing method according to appendix 16 or 17, wherein the coded IR cut filter in which the near-infrared light transmitting portion has a substantially circular shape is used.
(Appendix 19)
The information of the near-infrared light diffracting means and the position information of the diffraction pattern generated on the incident light image signal generating means are stored in a memory, according to any one of appendices 15 to 18. Video shooting method.
(Appendix 20)
A diffraction model of near-infrared light is generated, the brightness gradient of the incident light image signal is analyzed, a diffraction model suitable for the analysis result of the brightness gradient is determined, and the incident near-red light is based on the suitable diffraction model. The image capturing method according to any one of appendices 15 to 19, wherein the intensity of outside light is estimated.
(Appendix 21)
21. The image capturing method according to appendix 20, wherein the wavelength of the incident near-infrared light is estimated based on the adapted diffraction model.
(Appendix 22)
The incident light image signal is divided into regions each including one of the diffraction patterns, and the coordinates of the divided regions are converted into polar coordinates whose origin is the center of the diffraction pattern. The video shooting method described in any one of the appendices.
(Appendix 23)
23. The image according to appendix 22, wherein a brightness gradient ratio, which is a ratio of a brightness gradient at a coordinate of the incident light image signal and a brightness gradient at a position corresponding to the coordinate of the diffraction model, is calculated. How to shoot.
(Appendix 24)
24. The image capturing method according to appendix 23, wherein the brightness gradient ratio generated at a plurality of coordinates in the divided area is statistically processed.
(Appendix 25)
25. The image capturing method according to appendix 24, wherein the matching diffraction model is determined based on a result of the statistical processing of the brightness gradient ratio.
(Appendix 26)
26. The image capturing method according to any one of appendices 15 to 25, wherein incident light is separated into visible light of a plurality of colors to generate a signal according to intensity.
(Appendix 27)
27. The image capturing method according to appendix 26, wherein a near infrared light image signal is generated for each of the plurality of colors.
(Appendix 28)
28. The image capturing method according to any one of appendices 15 to 27, wherein the incident light image signal is generated by a silicon-based photosensor.
(Appendix 29)
Diffracting the near-infrared light component of the incident light by the near-infrared light diffracting means arranged on the incident side of the light receiving surface, and the visible light component and the near infrared light component of the incident light incident on the light receiving surface. Generating an incident light image signal according to intensity, extracting a diffraction pattern generated by the near infrared light on the light receiving surface, and intensity of the near infrared light component calculated based on the diffraction pattern Generating a near-infrared light image signal according to the above, and generating a visible light image signal according to the intensity of the visible light component based on the incident light image signal and the near-infrared light image signal. A video recording program comprising:
(Appendix 30)
Generating a diffraction model of near-infrared light; analyzing a brightness gradient of the incident light image signal; determining a diffraction model that matches the analysis result of the brightness gradient; and 30. An image capturing program as set forth in appendix 29, further comprising the step of estimating the intensity of incident near-infrared light based on the above.
(Appendix 31)
A step of dividing the incident light image signal into areas each including one of the diffraction patterns; and a step of converting coordinates of the divided areas into polar coordinates having a center of the diffraction pattern as an origin. 29. The video recording program according to supplement 29 or supplement 30.
(Appendix 32)
The method further comprises: calculating a brightness gradient ratio, which is a ratio of a brightness gradient at a coordinate of the incident light image signal and a brightness gradient at a position corresponding to the coordinate of the diffraction model. The described video shooting program.
(Appendix 33)
A step of statistically processing the luminance gradient ratio generated at a plurality of coordinates in the divided area; and a step of determining the suitable diffraction model based on a result of the statistical processing. The video recording program according to attachment 32.
(Appendix 34)
The video recording program according to any one of appendices 29 to 33, further comprising the step of separating incident light into visible lights of a plurality of colors and sensing the visible light.

10 incident light 20 circular pattern 100 image capturing device 110 near-infrared light diffracting means 110a coded IR cut filter 120 incident light image signal generating means 121 photosensor array 122 color filter array 131 diffraction model generating means 132 luminance gradient analyzing means 133 Adaptable model determination means 134 Encoded information memory 140 Near infrared light image signal generation means 150 Visible light image signal generation means 160 Prism 170 Camera lens

Claims (10)

Incident light image signal generating means for generating an incident light image signal according to the intensities of the visible light component and the near infrared light component of the incident light incident on the light receiving surface, and the incident light image signal generating means arranged on the incident side of the light receiving surface. Near-infrared light diffracting means for diffracting the near-infrared light component, diffraction pattern extracting means for extracting a diffraction pattern generated by the near-infrared light component on the light receiving surface, and the near-infrared light component calculated based on the diffraction pattern. Near-infrared light image signal generation means for generating a near-infrared light image signal according to the intensity of an infrared light component, and the visible light calculated based on the incident light image signal and the near-infrared light image signal Film image shooting apparatus characterized by comprising: a visible light image signal generating means for generating a visible light image signals corresponding to the intensity of the component, a. The near-infrared light diffracting means includes a near-infrared light cutting portion that cuts near-infrared light and transmits visible light, and a near-infrared light transmitting portion that transmits near-infrared light. The image capturing device according to claim 1. The image capturing apparatus according to claim 2, wherein the near-infrared light transmitting portion has a substantially circular shape. The diffraction pattern extraction means adapts to a diffraction model generation means for generating a near infrared light diffraction model, a luminance gradient analysis means for analyzing a luminance gradient of the incident light image signal, and an analysis result of the luminance gradient. 3. An adaptive near-infrared light intensity estimating means for estimating the intensity of near-infrared light incident on the basis of the adaptive diffraction model, and adaptive diffraction model determining means for determining a diffraction model. The image capturing device according to any one of claims 3 to 4. The near-infrared light image signal generation means includes an incident near-infrared light wavelength estimation means for estimating the wavelength of the incident near-infrared light based on the adaptive diffraction model. Video shooting device. The brightness gradient analyzing means converts the incident light image signal into area dividing means for dividing the incident light image signal into areas including one of the diffraction patterns, and coordinates of the divided areas into polar coordinates having the center of the diffraction pattern as an origin. The image capturing apparatus according to claim 4, further comprising: coordinate conversion means. The brightness gradient analysis means calculates a brightness gradient ratio which is a ratio of a brightness gradient at a coordinate of the incident light image signal and a brightness gradient at a position corresponding to the coordinate of the diffraction model. The image capturing apparatus according to claim 6, further comprising a calculating unit. The brightness gradient analysis means has a brightness gradient ratio statistical processing means for performing statistical processing of the brightness gradient ratio generated at a plurality of coordinates in the divided area,
The image capturing apparatus according to claim 7, wherein the adaptive diffraction model determination unit determines the adaptive diffraction model based on a result of the statistical processing. Diffract the near-infrared light component of the incident light by the near-infrared light diffracting means arranged on the incident side of the light-receiving surface to obtain the intensity of the visible light component and the near-infrared light component of the incident light that has entered the light-receiving surface. A near-infrared light corresponding to the intensity of the near-infrared light component, which is calculated based on the diffraction pattern, is generated by generating an incident light image signal according to the near-infrared light component and extracting a diffraction pattern generated on the light receiving surface by the near-infrared light component. An image capturing method, wherein an external light image signal is generated, and a visible light image signal corresponding to the intensity of the visible light component is generated based on the incident light image signal and the near infrared light image signal. .. A process of diffracting the near-infrared light component of the incident light by the near-infrared light diffracting means arranged on the incident side of the light-receiving surface, and the visible light component and the near-infrared light component of the incident light incident on the light-receiving surface. A process of generating an incident light image signal according to intensity, a step of extracting a diffraction pattern generated by the near-infrared light component on the light receiving surface, and the near-infrared light component of the near-infrared light component calculated based on the diffraction pattern. Processing for generating a near-infrared light image signal according to intensity, and processing for generating a visible-light image signal according to the intensity of the visible-light component based on the incident light image signal and the near-infrared light image signal When,
A video recording program that causes a computer to execute .

Images