JP2013187574A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device that is capable of, using a camera having an imaging element constructed by arranging color light receiving pixels and transparent light receiving pixels, obtaining images for presenting a video of an external environment and detecting a signal in a predetermined wavelength region.SOLUTION: An image processing device includes: external environment image generation means 32 for generating external environment images 42, 43, and 44 for presenting a video of an external environment from an original image 41 imaged by a camera 2 having an imaging element 22 where color light receiving pixels and transparent light receiving pixels are arranged; signal detection image generation means 34 for generating an image 45, in which a light reception component in a predetermined wavelength region (wavelength region in an infrared region, for instance) is emphasized, based on a gradation value of the color pixel and a gradation value of the transparent pixel in a detection target area of the original image 41; and signal detection means 33 for detecting, from the image 45, a signal component in the predetermined wavelength region received in the detection target area.

Description

  The present invention relates to an image processing apparatus for processing imaging data of a camera having an imaging element in which a plurality of color light receiving pixels that receive light through a color filter and a plurality of transparent light receiving pixels that receive light without passing through a color filter are arranged. About.

  Conventionally, the sensitivity of a color image is obtained by using an image sensor configured by arranging pixels that receive light through a color filter (color light receiving pixels) and pixels that receive light without passing through a color filter (transparent light receiving pixels). A method for improving the above has been proposed (see, for example, Patent Document 1).

  In addition, for example, a device that mounts a camera that captures the front of the vehicle or the like on the vehicle and monitors a situation around the vehicle using a captured image of the mounted camera is known (for example, Patent Document 2). See

Special table 2008-507908 gazette Japanese Patent Laid-Open No. 2004-247721

  By the way, for example, in a vehicle, a communication signal output from another vehicle or a roadside communication facility, a sensing signal related to environmental information of the vehicle, or the like is received by a wireless signal in a predetermined wavelength band such as an infrared region. There is.

  In such a case, a dedicated receiver for receiving these signals is usually mounted on the vehicle. However, when a dedicated receiver is mounted on a vehicle in this way, there is a disadvantage that the vehicle is reduced in weight and cost.

  On the other hand, the inventor of the present application can not only acquire an image of an external image such as a visible image around the vehicle, but also the imaging data of the camera (output data of each pixel) in the camera having the color light receiving pixels and the transparent light receiving pixels. It has been found that by using the signal, it is possible to detect a signal in a predetermined wavelength region such as the communication signal without using a dedicated receiver.

  The present invention has been made in view of such a background, using a camera having an image sensor configured by arranging color light-receiving pixels and transparent light-receiving pixels, obtaining an image showing an image of the outside world, An object of the present invention is to provide an image processing apparatus capable of detecting a signal in a predetermined wavelength range.

In order to achieve the above object, an image device according to the present invention includes a plurality of color light receiving pixels that receive light via a color filter and a plurality of transparent light receiving pixels that receive light without passing through a color filter. A camera having
In the original image captured by the camera, a plurality of color pixels each assigned a gradation value corresponding to the light receiving level of each color light receiving pixel and a gradation value corresponding to the light receiving level of each transparent light receiving pixel are respectively An external image generating means for generating a color or monotone external image indicating an external image from an original image formed by arranging a plurality of assigned transparent pixels;
Based on the gradation value of the color pixel in the detection target area, which is at least a partial area of the original image, and the gradation value of the transparent pixel in the detection target area, the light receiving component in the detection target area Among them, an image in which each of the plurality of color light receiving pixels emphasizes a light receiving component in a predetermined wavelength region near the end of the entire wavelength region or a predetermined wavelength region that deviates from the entire wavelength region. Signal detection image generation means for generating a signal detection image,
Signal detecting means for detecting a signal component included in the predetermined wavelength region from the signal detection image is provided (first invention).

  According to the first aspect of the invention, the external image generation unit generates an external image indicating an external image from the original image. It becomes possible to recognize the situation of the outside world from this outside world image. In this case, the external image may be a color image or a monotone image. In addition, since the image sensor of the camera has transparent light receiving pixels, it is possible to generate a highly sensitive external image or an external image with a wide dynamic range.

  Furthermore, in the first invention, separately from the external image, the signal detection image generating means detects at least a part of the original image (that is, a whole area or a part of the original image). Based on the gradation value of the color pixel in the target area and the gradation value of the transparent pixel in the detection target area, a signal detection image that is an image in which the light receiving component in the predetermined wavelength range is emphasized is generated. The

  Then, the signal detection means detects a signal component included in the predetermined wavelength range from the signal detection image.

  More specifically, the image (signal detection image) in which the light receiving component in the predetermined wavelength region is emphasized is a portion corresponding to the detection target region (the whole of the imaging device) in the whole of the imaging device. Among the light (electromagnetic waves) received by the color light-receiving pixels and the transparent light-receiving pixels included in the color light-receiving pixels or the transparent light-receiving pixels, the sensitivity to the light (electromagnetic waves) in the predetermined wavelength range is relative to other wavelength ranges. This means an enhanced image.

  Here, since each color light receiving pixel receives light via a color filter, the wavelength range that can be detected by the color light receiving pixel is limited to a wavelength range that can be transmitted through the color filter.

  On the other hand, since each transparent light receiving pixel receives light without passing through a color filter, the wavelength range that can be detected by the transparent light receiving pixel is the entire wavelength range that can be detected by each color light receiving pixel. It includes the wavelength range that deviates from the range.

  The wavelength range that can be detected by the color light receiving pixel or the transparent light receiving pixel means the light receiving sensitivity of each pixel (when the pixel receives light through a color filter or a transparent filter, the light receiving sensitivity including the filter). ) And a wavelength characteristic indicating a relationship between the received light wavelength and a wavelength range in which the received light sensitivity is relatively high.

  The entire wavelength range that can be detected by each color light receiving pixel means a wavelength range obtained by combining the wavelength ranges that can be detected by each of the plurality of color light receiving pixels included in the image sensor.

  The predetermined wavelength range is a predetermined wavelength range close to an end of an entire wavelength range that can be detected by each of the plurality of color light receiving pixels or a predetermined wavelength range deviating from the entire wavelength range. Therefore, in the predetermined wavelength region, the difference from the light receiving sensitivity of the color light receiving pixels is larger than the wavelength region near the center of the wavelength region that can be detected by each color light receiving pixel.

  Therefore, comparing the case where light in the predetermined wavelength region is received with the case where light is not received in the detection target region, the light reception level of the color light receiving pixel in the detection target region (and hence the gradation value of the galler pixel). ) Is less likely to cause a large difference in both cases, while the light receiving level of the transparent light receiving pixel in the detection target region (and thus the gradation value of the transparent pixel) is relatively different in both cases. Prone to occur.

  Therefore, an image in which the light receiving component in the predetermined wavelength region is emphasized based on the gradation value of the color pixel in the detection target area of the original image and the gradation value of the transparent pixel in the detection target area. A certain signal detection image can be generated. As a result, a signal component included in the predetermined wavelength region can be detected from the signal detection image.

  Therefore, according to the present invention, using a camera having an image sensor configured by arranging color light-receiving pixels and transparent light-receiving pixels, an image showing the external image (the external image) is acquired, It is possible to detect a signal in the wavelength region.

  In the first invention, the predetermined wavelength region is, for example, a wavelength region in the infrared region (second invention). Since the color filter generally does not easily transmit light having a part of the wavelength in the infrared region, such as the near infrared region, the light receiving sensitivity of the color light receiving pixels in the part of the wavelength region of the infrared region is low.

  On the other hand, the light receiving sensitivity of the transparent light receiving pixel is usually sufficiently higher than the light receiving sensitivity of the color light receiving pixel.

  Therefore, by adopting a wavelength region in the infrared region where the light receiving sensitivity is different depending on the presence or absence of a color filter, the signal component in the wavelength region in the infrared region can be appropriately detected as the predetermined wavelength region. It becomes.

  The wavelength region in the infrared region is a wavelength region that can be widely used for communication, sensing (for example, raindrop detection, infrared image detection, etc.), and the like. For this reason, according to the second aspect of the invention, it is possible to perform communication with the outside, sensing, and the like in addition to acquiring the outside world image.

  In the first invention or the second invention, the signal detection image generation means can generate the signal detection image by the following method, for example.

  That is, the signal detection image generation means sets the gradation value of each pixel in the detection target area according to at least the gradation values of the plurality of transparent pixels included in the detection target area. First reference image generation means for generating one reference image, and gradation values of each pixel in the detection target area are set according to at least gradation values of the plurality of color pixels included in the detection target area. Second reference image generation means for generating the second reference image, and generates an image of a difference in gradation value between the first reference image and the second reference image as the signal detection image (third invention).

  According to the third aspect of the invention, with respect to the entire wavelength range that can be detected by each of the color light receiving pixels, an image of the outside world that is incident on the image sensor of the camera (an image having a wavelength that belongs to the entire wavelength range). Regardless, the first reference image and the second reference image can be generated so that the degree of coincidence of the gradation values of the first reference image and the second reference image is as high as possible. Therefore, an image of a difference in gradation value between the first reference image and the second reference image (specifically, a difference in gradation value between the first reference image and the second reference image is assigned as the gradation value of each pixel. The image) is highly dependent on the received light intensity in the predetermined wavelength range. Therefore, the difference image can be obtained as the signal detection image in which the light receiving component in the predetermined wavelength region is emphasized.

1 is a diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 2A is a diagram illustrating an arrangement example of the color filter of the camera in the embodiment, and FIG. 2B is a diagram for explaining a captured image (original image) obtained from the image sensor of the camera of the embodiment. The flowchart which shows the process regarding the external field image generation part shown in FIG. The figure for demonstrating the image according to color (G image) produced | generated by the external field image generation part shown in FIG. The figure for demonstrating the image according to color (R image) produced | generated by the external field image generation part shown in FIG. The figure for demonstrating the image according to color (B image) produced | generated by the external field image generation part shown in FIG. The figure for demonstrating the image according to color (W image) produced | generated by the external field image generation part shown in FIG. The figure for demonstrating the color image produced | generated by the external field image generation part shown in FIG. The figure for demonstrating the wide dynamic range image produced | generated by the external field image generation part shown in FIG. The graph which illustrates the sensitivity of the light reception pixel of the camera of an embodiment. The flowchart which shows the process regarding the communication signal generation part shown in FIG. FIGS. 12A, 12B, and 12C are diagrams for explaining the first reference image, the second reference image, and the signal detection image generated by the signal detection image generation unit illustrated in FIG. 1, respectively. . FIGS. 13A and 13B are diagrams showing other arrangement examples of the color filter of the camera.

  An embodiment of an image processing apparatus of the present invention will be described with reference to FIGS. With reference to FIG. 1, the image processing apparatus of the present embodiment includes a camera 2 mounted on a vehicle 1 and an image controller 3 connected to the camera 2.

  The camera 2 images the surroundings of the vehicle 1 by an image sensor 22 (CCD, CMOS, etc.) in which a filter 21 is incorporated, and outputs the image data to the control circuit 30 of the image controller 3. The imaging element 22 is configured by arranging a plurality of m × n light receiving elements in an array pattern of m rows and n columns in a two-dimensional manner.

  Referring to FIG. 2A, the filter 21 of the camera 2 is a color filter of three primary colors R (red), G (green), and B (blue), that is, light in the R, G, and B wavelength ranges. It is composed of three types of color filters that can each transmit (visible light). The color filter of any one of these three primary color filters is attached to the light receiving path side of a predetermined number of light receiving pixels among the m × n light receiving pixels of the image sensor 22. . Thereby, each light receiving pixel to which the color filter is attached receives light through the color filter of the corresponding color.

  Note that the entire color filters mounted on these light receiving pixels (the predetermined number of light receiving pixels) include color filters of all the colors R, G, and B. Accordingly, the entire light-receiving pixel to which the color filter is attached is divided into a light-receiving pixel (hereinafter referred to as an R light-receiving pixel) to which an R color filter (hereinafter referred to as an R filter) is attached, and a G color filter ( Hereinafter, a light receiving pixel (hereinafter referred to as G light receiving pixel) to which a G filter is attached, and a light receiving pixel (hereinafter referred to as B light receiving pixel) to which a B color filter (hereinafter referred to as B filter) is attached. Consists of These R light receiving pixel, G light receiving pixel, and B light receiving pixel correspond to the color light receiving pixels in the present invention.

  Of the m × n light receiving pixels, the remaining light receiving pixels have a color filter omitted (not mounted). For this reason, each of the light receiving pixels receives light without passing through a color filter. Hereinafter, this light receiving pixel is referred to as a W light receiving pixel. The W light receiving pixel corresponds to the transparent light receiving pixel in the present invention.

  The camera 2 includes R light receiving pixels (indicated by R11, R15,...), G light receiving pixels (indicated by G12, G14,...), And B light receiving pixels (B13, B31,. And output signals of Rx light receiving pixels (indicated by Rx22, Rx24,... In the figure) are output to the image controller 3 as imaging data. The output signal of each light receiving pixel is a signal having an intensity (magnitude) corresponding to the light receiving level per predetermined time at the light receiving pixel.

  The light receiving level of each light receiving pixel corresponds to the intensity of the light received by the light receiving pixel among the light incident on the image sensor 22 of the camera 2.

  As the color filter, other types of color filters other than RGB (Cy (cyan), Mg (magenta), Ye (yellow) complementary primary color filters, etc.) may be used.

  The image controller 3 includes a control circuit 30 including a CPU, a memory, an input / output circuit, and the like (not shown), an image memory 40, and a CAN (Controller Area Network) driver 50.

  The control circuit 30 functions as an original image acquisition unit 31, an external image generation unit 32, a communication signal detection unit 33, and an object detection unit 35 by executing an image processing program stored in the memory by the CPU. Note that some or all of the original image acquisition unit 31, the external image generation unit 32, the communication signal detection unit 33, and the object detection unit 35 may be configured by hardware.

  The original image acquisition unit 31 outputs a control signal to the camera 2 to image the surroundings of the vehicle 1, and acquires the data of the original image 41 from the imaging data (output signal of each light receiving pixel) output from the camera 2. And stored in the image memory 40.

  As shown in FIG. 2B, the original image 41 is output from each light receiving pixel (R light receiving pixel, G light receiving pixel, B light receiving pixel, and W light receiving pixel) of the image sensor 22 shown in FIG. The gradation values indicating the signal intensity are individually assigned as the gradation values of the pixels at the corresponding arrangement positions (pixels having the same arrangement position). In FIG. 2B, the gradation value of each pixel is set to S (upper case), one of lower case letters r, g, b, w, and subscripts i, j (i = 1, 2,..., M, j = 1, 2,..., N).

  Here, Srij represents a gradation value of a pixel at an arrangement position corresponding to the R light receiving pixel in FIG. 2A (hereinafter referred to as R pixel), and Sgij represents an arrangement corresponding to the G light receiving pixel in FIG. 2 indicates the gradation value of the pixel at the position (hereinafter referred to as G pixel), Sbij indicates the gradation value of the pixel at the arrangement position (hereinafter referred to as B pixel) corresponding to the B light receiving pixel in FIG. Indicates a gradation value of a pixel (hereinafter referred to as a W pixel) at an arrangement position corresponding to the W light receiving pixel in FIG.

  The W pixel corresponds to the transparent pixel in the present invention, and the R pixel, G pixel, and B pixel correspond to the color pixel in the present invention.

  The external image generation unit 32 is a functional unit corresponding to the external image generation unit of the present invention. The external image generation unit 32 generates four types of color-specific images 42 corresponding to each of the R light receiving pixel, the G light receiving pixel, the B light receiving pixel, and the W light receiving pixel, a color image 43, and a wide dynamic range image 44. Images are generated and stored in the image memory 40.

  The communication signal detector 33 is a functional unit corresponding to the communication signal detector of the present invention. In the present embodiment, the communication signal detection unit 33 detects a communication signal for inter-vehicle communication output from another vehicle outside the vehicle 1 or a communication signal output from roadside communication equipment such as a beacon. . These communication signals are, for example, communication signals in the near-infrared wavelength region (signals composed of ON / OFF pulse trains).

  In this case, the communication signal detection unit 33 includes a function as a signal detection image generation unit 34 corresponding to the signal detection image generation means in the present invention, and is generated by the signal detection image generation unit 34 as described later. The signal detection image 45 is stored in the image memory 40. Then, the communication signal detection unit 33 detects a communication signal from the signal detection image 45.

  The signal detection image generation unit 34 of the communication signal detection unit 33 includes functions as a first reference image generation unit and a second reference image generation unit in the present invention.

  The object detection unit 35 uses the color-specific image 42, the color image 43, and the wide dynamic range image 44 to detect lane marks, other vehicles, traffic lights, pedestrians, and the like laid on the road on which the vehicle 1 is traveling. Then, various control signals are transmitted to the vehicle controller 6 according to the detection result.

  The vehicle controller 6 is an electronic circuit unit that includes a CPU, a memory, an input / output circuit, and the like (not shown). The vehicle controller 6 executes the control program for the vehicle 1 held in the memory by the CPU, thereby controlling the operation of the steering device 71, and the braking control unit for controlling the operation of the braking device 72. 62 and a display display control unit 63 that controls display on the display 73.

  The image controller 3 and the vehicle controller 6 communicate with each other via the CAN drivers 50 and 64.

  Next, processing of the external image generation unit 32 (including processing of the original image acquisition unit 31) will be described in detail with reference to FIGS. The external world image generation unit 32 generates the color-specific image 42, the color image 43, and the wide dynamic range image 44 as the external world image by executing the processing shown in the flowchart of FIG. 3 including the processing of the original image acquisition unit 31. To do.

  STEP 1 in FIG. 3 is processing by the original image acquisition unit 31. The original image acquisition unit 31 acquires the original image 41 (see FIG. 2B) from the imaging data output from the camera 2 and holds it in the image memory 40.

  In subsequent STEP 2, the external image generation unit 32 generates four types of color-specific images 42 from the original image, and holds these color-specific images 42 in the image memory 40.

  More specifically, the four types of color-specific images 42 correspond to an R image (hereinafter referred to as reference numeral 42r) that is a color-specific image 42 of a color (red) corresponding to the R light receiving pixel, and a G light receiving pixel. A G image (hereinafter referred to as reference numeral 42g) that is a color-specific image 42 of the color (green) to be performed, and a B image (hereinafter referred to as reference numeral 42) that is a color-specific image 42 of the color (green) corresponding to the B light receiving pixel. 42b) and a W image (hereinafter referred to as reference numeral 42w) which is a color-specific image 42 of a color (gray) corresponding to the W light receiving pixel.

  The R image 42r has a red tone monotone image (a red luminance distribution is determined by a demosaicing process) in which the tone value of each pixel is determined based on at least the tone value of the R pixel in the original image 41. Image).

  The G image 42g has a green tone monotone image (green luminance) in which the tone value of each pixel is determined by demosaicing processing based on at least the tone value of the G pixel in the original image 41. Image showing the distribution).

  Further, the B image 42b has a blue tone monotone image (blue luminance) in which the tone value of each pixel is determined by demosaicing processing based on at least the tone value of the B pixel in the original image 41. Image showing the distribution).

  Further, the W image 42w is a gray-tone monotone image (gray luminance) in which the gradation value of each pixel is determined by demosaicing processing based on at least the gradation value of the W pixel in the original image 41. Image showing the distribution).

  These four types of color-specific images 42 (42r, 42g, 42b, 42w) are generated as follows in this embodiment.

  First, one color-specific image 42 among the R image 42r, the G image 42r, the B image 42b, and the W image 42w is generated as a reference luminance distribution image. This reference luminance distribution image means an image having a luminance distribution state in which the degree of matching with the actual luminance distribution of the photographing target of the camera 2 is relatively high.

  In the present embodiment, for example, the G image 42g corresponding to the largest number of G light receiving pixels in the image sensor 22 is generated as the reference luminance distribution image. As shown in FIG. 4, the G image 42g is an image to which a G value (G (green) luminance value) is assigned as a gradation value of each pixel. In FIG. 4, the gradation value (G value) of each pixel is represented by G′i, j (i = 1, 2,..., M, j = 1, 2,..., N).

  The G value G′i, j of each pixel of the G image 42g as the reference luminance distribution image is determined as follows.

  Among the pixels of the G image 42g, the pixel at the arrangement position (the pixel having the same arrangement position) corresponding to the G pixel of the original image 41 (the pixel whose gradation value is Sgi, j) is expressed by the following equation (1). As described above, the gradation value Sgi, j of the G pixel of the corresponding original image 41 is directly determined as the G value (G′i, j) of the pixel of the G image 42g. For example, the gradation value (Sg2,3) of the pixel at the position of (i, j) = (2,3) in the original image 41 shown in FIG. 2B is (i, j) = It is determined as the G value (G′2, 3) of the pixel at the arrangement position of (2, 3).

G'i, j = Sgi, j (1)

In addition, among the pixels of the G image 42g, for the pixels at the arrangement positions corresponding to the pixels (R pixels, B pixels, or W pixels) other than the G pixels of the original image 41, the G pixels at the arrangement positions around the pixels are changed. It is determined complementarily using at least the gradation value.

  Specifically, among the pixels of the G image 42g, for the pixels at the arrangement positions corresponding to the R pixel or the B pixel of the original image 41, the gradation values Sgi of the four G pixels adjacent to the pixel in the vertical and horizontal directions. On the basis of + 1, j, Sgi-1, j, Sgi, j + 1, Sgi, j-1, depending on the magnitude relationship between Ig and Jg in the following formulas (2) and (3), formula (4) Alternatively, the G value (G′i, j) is determined by (5) or (6).

Ig = | Sgi + 1, j−Sgi-1, j | (2)
Jg = | Sgi, j + 1−Sgi, j-1 | (3)
When Ig <Jg G'i, j = (Sgi + 1, j + Sgi-1, j) / 2 (4)
When Ig> Jg G'i, j = (Sgi, j + 1 + Sgi, j-1) / 2 (5)
When Ig = Jg G'i, j = (Sgi, j + 1 + Sgi, j-1 + Sgi + 1, j + Sgi-1, j) / 4 (6)

Further, among the pixels of the G image 42g, for the pixels at the arrangement positions corresponding to the W pixels of the original image 41, the gradation values Sgi + 1, j, Sgi of the four G pixels adjacent to the pixel above and below and to the left and right. -1, j, Sgi, j + 1, Sgi, j-1 and the corresponding gradation value Swi, j of the W pixel, and four W pixels existing at intervals of one pixel above and below and on the left and right. Based on the gradation values Swi + 2, j, Swi-2, j, Swi, j + 2, Swi, j-2, depending on the magnitude relationship between Ig and Jg in the following formulas (7) and (8) The G value (G′i, j) is determined by the equation (9) or (10) or (11).

Ig = | Sgi + 1, j-Sgi-1, j | + | 2Srxi, j-Srxi + 2, j-Srxi-2, j | (7)
Jg = | Sgi, j + 1−Sgi, j−1 | + | 2Srxi, j−Srxi, j + 2−Srxi, j−2 | (8)
When Ig <Jg G'i, j = (Sgi + 1, j + Sgi-1, j) / 2
+ (2Srxi, j-Srxi + 2, j-Srxi-2, j) / 4 (9)
When Ig> Jg G'i, j = (Sgi, j + 1 + Sgi, j-1) / 2
+ (2Srxi, j-Srxi, j + 2-Srxi, j-2) / 4 (10)
When Ig = Jg G'i, j = (Sgi, j + 1 + Sgi, j-1 + Sgi + 1, j + Sgi-1, j) / 4
+ (4Srxi, j-Srxi + 2, j-Srxi-2, j-Srxi, j + 2-Srxi, j-2) / 8
...... (11)

Through the above processing (demosaicing processing), using the gradation value (G value) of the G pixel of the original image 41, each pixel of the G image 42g has a G value indicating a green gradation value (luminance value). Assigned. As a result, a G image 42g (green tone luminance distribution image) as a reference luminance distribution image is generated.

  In STEP 2, the external image generation unit 32 then generates the remaining three color-specific images 42r, 42b, and 42w other than the G image 42g generated as the reference luminance distribution image. In this case, in the present embodiment, the remaining three color-specific images 42r, 42b, and 42w are generated by reflecting the luminance distribution of the reference luminance distribution image as described below.

  First, as shown in FIG. 5, the R image 42r is an image to which an R value (red luminance value) is assigned as the gradation value of each pixel. In FIG. 5, the gradation value (R value) of each pixel is represented by R′i, j (i = 1, 2,..., M, j = 1, 2,..., N). The R value (R′i, j) of each pixel of the R image 42r is determined as follows.

  That is, among the pixels of the R image 42r, the pixel at the arrangement position corresponding to the R pixel of the original image 41 (pixels having the same arrangement position) is represented by the R pixel of the corresponding original image 41 as shown in Expression (12). Gradation value Sri, j is determined as the R value (R′i, j) of the pixel of the R image 42r.

R'i, j = Sri, j (12)

In addition, among the pixels of the R image 42r, the R value (R′i, j) of the pixel at the arrangement position corresponding to a pixel (G pixel, B pixel, or W pixel) other than the R pixel of the original image 41 is Using the gradation value of the R pixel around the pixel (gradation value in the original image 41) and the G value of the pixel and the surrounding pixels (gradation value (G value) in the G image 42g). It is determined.

  Specifically, among the pixels of the R image 42r, with respect to the pixel at the arrangement position corresponding to the B pixel of the original image 41, four R pixels (one pixel interval above and below and right and left of the pixel) ( The gradation value Sri + 2, j, Sri-2, j, Sri, j + 2, Sri, j-2 of the original image 41) and the corresponding pixel (pixel in the G image 42g) G Value (luminance value) G′i, j, and G values (luminance values) G′i + 2, of four pixels (pixels in the G image 42g) existing at intervals of one pixel above and below and on the left and right. Based on j, G'i-2, j, G'i, j + 2, G'i, j-2, depending on the magnitude relationship between Ir and Jr in the following formulas (13) and (14), The R value (R′i, j) is determined by the equation (15) or (16) or (17).

Ir = | Sri + 2, j−Sri−2, j | + | 2G′i, j−G′i + 2, j−G′i−2, j | (13)
Jr = | Sri, j + 2-Sri, j-2 | + | 2G'i, j-G'i, j + 2-G'i, j-2 | (14)
When Ir <Jr R'i, j = (Sri + 2, j + Sri-2, j) / 2
+ (2G'i, j-G'i + 2, j-G'i-2, j) / 2 (15)
When Ir> Jr R'i, j = (Sri, j + 2 + Sri, j-2) / 2
+ (2G′i, j−G′i, j + 2−G′i, j−2) / 2 (16)
When Ir = Jr R'i, j = (Sri, j + 2 + Sri, j-2 + Sri + 2, j + Sri-2, j) / 4
+ (4G'i, j-G'i + 2, j-G'i-2, j-G'i, j + 2-G'i, j-2) / 4
...... (17)

Further, among the pixels of the R image 42r, for the pixel at the arrangement position corresponding to the G pixel of the original image 41, one R pixel (a pixel in the original image 41) adjacent to any one of the upper, lower, left, and right of the pixel. The gradation value Sri + 1, j or Sri-1, j or Sri, j + 1 or Sri, j-1 and the G value (luminance value) G′i, of the corresponding pixel (the pixel in the G image 42g) j and the G value (luminance value) G′i + 1, j, G′i−1, j, G′i, of one pixel (a pixel in the G image 42g) adjacent to either the top, bottom, left, or right thereof Based on j + 1 and G′i, j−1, the R value (R′i, j) is determined by the following formula (18) or (19) or (20) or (21).

When R pixel is adjacent to the lower side R′i, j = Sri + 1, j + (G′i, j−G′i + 1, j) (18)
When R pixel is adjacent on the upper side R′i, j = Sri−1, j + (G′i, j−G′i−1, j) (19)
When R pixel is adjacent to the right side R′i, j = Sri, j + 1 + (G′i, j−G′i, j + 1) (20)
When R pixel is adjacent to the left side R'i, j = Sri, j-1 + (G'i, j-G'i, j-1) (21)

Among the pixels of the R image 42r, the pixels at the arrangement positions corresponding to the W pixels of the original image 41 are adjacent to the pixels on the upper right side and the lower left side or the upper left side and the lower right side. The gradation values (Sri-1, j + 1, Sri + 1, j-1) or (Sri-1, j-1, Sri + 1, j + 1) of two R pixels (pixels in the original image 41) And the G value (luminance value) G′i, j of the corresponding pixel (the pixel in the G image 42g) and two adjacent pixels on the upper right and lower left sides, or on the upper left and lower right sides. G value (luminance value) (G′i−1, j + 1, G′i + 1, j−1) or (G′i−1, j−1, G−1) of a pixel (pixel in the G image 42g) Based on 'i + 1, j + 1), the R value (R'i, j) is determined by the following equation (22) or (23).

R'i, j = (Sri + 1, j-1 + Sri-1, j + 1) / 2 when R pixels are adjacent to the upper right and lower left
+ (2G′i, j−G′i + 1, j−1−G′i−1, j + 1) / 2 (22)
R'i, j = (Sri + 1, j + 1 + Sri-1, j-1) / 2 when R pixel is adjacent to upper left and lower right
+ (2G'i, j-G'i + 1, j + 1-G'i-1, j-1) / 2 (23)

As described above, among the pixels of the R image 42r, the R value (R′i, j) of each pixel other than the pixel at the arrangement position corresponding to the R pixel of the original image 41 is the R around the pixel. The basic value (the first term on the right side of each of the equations (15) to (23)) determined according to the tone value of the pixel (the tone value in the original image 41) is the pixel and the surrounding pixels. It is determined by correcting with a correction value (second term on the right side of each of the equations (15) to (23)) determined according to the G value (G value in the G image 42g).

  The R value (R′i, j) of each pixel at the arrangement position corresponding to the R pixel of the original image 41 among the pixels of the R image 42r is the level of the R pixel of the original image 41 corresponding to the pixel. It is determined so as to coincide with the key value Sri, j.

  Thereby, the R image 42r is generated by reflecting the luminance distribution of the G image 42g as the reference luminance distribution image.

  Next, as shown in FIG. 6, the B image 42b is an image to which a B value (blue luminance value) is assigned as the gradation value of each pixel. In FIG. 6, the gradation value (B value) of each pixel is represented by B′i, j (i = 1, 2,..., M, j = 1, 2,..., N). The B value (B′i, j) of each pixel of the B image 42b is determined in the same manner as the R value (R′i, j) of each pixel of the R image 42r.

  That is, among the pixels of the B image 42b, the pixels at the arrangement positions corresponding to the B pixels of the original image 41 (the pixels at the same arrangement position) are represented by the B pixels of the corresponding original image 41 as shown in Expression (24). Gradation value Sbi, j is determined as the B value (B′i, j) of the pixel of the B image 42b.

B'i, j = Sbi, j (24)

In addition, among the pixels of the B image 42b, the B value (B′i, j) of the pixel at the arrangement position corresponding to a pixel other than the B pixel of the original image 41 (R pixel, G pixel, or W pixel) is Using the gradation value of the B pixel around the pixel (gradation value in the original image 41) and the G value of the pixel and the surrounding pixels (gradation value (G value) in the G image 42g) It is determined.

  Specifically, among the pixels of the B image 42b, with respect to the pixel at the arrangement position corresponding to the R pixel of the original image 41, four B pixels (one pixel interval above and below and right and left of the pixel) ( The gradation value Sbi + 2, j, Sbi-2, j, Sbi, j + 2, Sbi, j-2 of the original image 41) and the corresponding pixel (pixel in the G image 42g) G Value (luminance value) G′i, j, and G values (luminance values) G′i + 2, of four pixels (pixels in the G image 42g) existing at intervals of one pixel above and below and on the left and right. Based on j, G'i-2, j, G'i, j + 2, and G'i, j-2, depending on the magnitude relationship between Ib and Jb in the following equations (25) and (26), The B value (B′i, j) is determined by the equation (27), (28) or (29).

Ib = | Sbi + 2, j−Sbi−2, j | + | 2G′i, j−G′i + 2, j−G′i−2, j | (25)
Jb = | Sbi, j + 2-Sbi, j-2 | + | 2G'i, j-G'i, j + 2-G'i, j-2 | (26)
When Ib <Jb B'i, j = (Sbi + 2, j + Sbi-2, j) / 2
+ (2G'i, j-G'i + 2, j-G'i-2, j) / 2 (27)
When Ib> Jb B′i, j = (Sbi, j + 2 + Sbi, j−2) / 2
+ (2G'i, j-G'i, j + 2-G'i, j-2) / 2 (28)
When Ib = Jb B'i, j = (Sbi, j + 2 + Sbi, j-2 + Sbi + 2, j + Sbi-2, j) / 4
+ (4G'i, j-G'i + 2, j-G'i-2, j-G'i, j + 2-G'i, j-2) / 4
...... (29)

In addition, among the pixels of the B image 42 b, for the pixel at the arrangement position corresponding to the G pixel of the original image 41, one B pixel adjacent to either the upper, lower, left, or right of the pixel (B pixel in the original image 41) Gradation value Sbi + 1, j or Sbi-1, j or Sbi, j + 1 or Sbi, j-1 and the G value (luminance value) G′i of the corresponding pixel (the pixel in the G image 42g). , j and the G value (luminance value) G′i + 1, j, G′i−1, j, G′i of one pixel (pixel in the G image 42g) adjacent to any one of the top, bottom, left, and right , j + 1, G′i, j−1, the B value (B′i, j) is determined by the following equation (30) or (31) or (32) or (33).

When B pixel is adjacent to the lower side B′i, j = Sbi + 1, j + (G′i, j−G′i + 1, j) (30)
When B pixel is adjacent on the upper side B′i, j = Sbi−1, j + (G′i, j−G′i−1, j) (31)
When B pixel is adjacent to the right side B′i, j = Sbi, j + 1 + (G′i, j−G′i, j + 1) (32)
When R pixel is adjacent to the left side B′i, j = Sbi, j−1 + (G′i, j−G′i, j−1) (33)

Among the pixels of the B image 42b, the pixels at the arrangement positions corresponding to the W pixels of the original image 41 are adjacent to the pixel on the upper right side and the lower left side, or on the upper left side and the lower right side. The gradation values (Sbi-1, j + 1, Sbi + 1, j-1) or (Sbi-1, j-1, Sbi + 1, j + 1) of two B pixels (B pixels in the original image 41) ) And the G value (luminance value) G′i, j of the corresponding pixel (the pixel in the G image 42 g), 2 adjacent to the upper right side and the lower left side, or the upper left side and the lower right side. G values (luminance values) (G′i−1, j + 1, G′i + 1, j−1) or (G′i−1, j−1,) of two pixels (pixels in the G image 42g) Based on G′i + 1, j + 1), the B value (B′i, j) is determined by the following equation (34) or (35).

When B pixels are adjacent to the upper right and lower left, B'i, j = (Sbi + 1, j-1 + Sbi-1, j + 1) / 2
+ (2G′i, j−G′i + 1, j−1−G′i−1, j + 1) / 2 (34)
When B pixel is adjacent to upper left and lower right B'i, j = (Sbi + 1, j + 1 + Sbi-1, j-1) / 2
+ (2G'i, j-G'i + 1, j + 1-G'i-1, j-1) / 2 (35)

As described above, among the pixels of the B image 42b, the B value (B′i, j) of each pixel other than the pixel at the arrangement position corresponding to the B pixel of the original image 41 is the B pixel around the pixel. The basic value (the first term on the right side of each of the equations (27) to (35)) determined according to the tone value (the tone value in the original image 41) of the pixel and its surrounding pixels It is determined by correcting with a correction value (second term on the right side of each of the equations (27) to (35)) determined according to the value (G value in the G image 42g).

  Then, among the pixels of the B image 42b, the B value (B′i, j) of each pixel at the arrangement position corresponding to the B pixel of the original image 41 is the level of the B pixel of the original image 41 corresponding to the pixel. It is determined so as to coincide with the key value Sbi, j.

  Thereby, the B image 42b is generated by reflecting the luminance distribution of the G image 42g as the reference luminance distribution image.

  Next, as shown in FIG. 7, the W image 42w is an image in which a W value (gray gradation luminance value) is assigned as the gradation value of each pixel. In FIG. 7, the gradation value (W value) of each pixel is represented by W′i, j (i = 1, 2,..., M, j = 1, 2,..., N). The W value (W′i, j) of each pixel of the W image 42w is determined as follows.

  That is, among the pixels of the W image 42w, the pixel at the arrangement position corresponding to the W pixel of the original image 41 (pixels having the same arrangement position) is represented by the W pixel of the corresponding original image 41 as shown in Expression (36). Gradation value Swi, j is determined as the W value (W′i, j) of the pixel of the W image 42w.

W'i, j = Swi, j (36)

In addition, among the pixels of the W image 42w, the W value (W′i, j) of the pixel at the arrangement position corresponding to a pixel other than the W pixel of the original image 41 (R pixel, G pixel, or B pixel) is Using the gradation value of the W pixel around the pixel (gradation value in the original image 41) and the G value of the pixel and the surrounding pixels (gradation value (G value) in the G image 42g). It is determined.

  Specifically, among the pixels of the W image 42w, with respect to the pixel at the arrangement position corresponding to the R pixel or the B pixel of the original image 41, the pixel is located on the upper right side, the lower right side, the upper left side, and the lower left side. Gradation values Swi-1, j + 1, Swi + 1, j + 1, Swi-1, j-1, Swi + 1, j-1 of four W pixels adjacent to (W pixels in the original image) And the G value G′i, j of the corresponding pixel (pixel in the G image) and four pixels (in the G image 42g) that are adjacent to the upper right side, the lower right side, the upper left side, and the lower left side. Pixel) G values G′i−1, j + 1, G′i + 1, j + 1, G′i−1, j−1, G′i + 1, j−1 The W value (W′i, j) is determined by the formula (39), (40) or (41) according to the magnitude relationship between Iw and Jw in the formulas (37) and (38).

Iw = | Swi + 1, j + 1−Swi-1, j−1 |
+ | 2G'i, j-G'i + 1, j + 1-G'i-1, j-1 | (37)
Jw = | Swi + 1, j-1-Swi-1, j + 1 |
+ | 2G′i, j−G′i + 1, j−1−G′i−1, j + 1 | (38)
When Iw <Jw W′i, j = (Swi + 1, j + 1 + Swi-1, j−1) / 2
+ (2G'i, j-G'i + 1, j + 1-G'i-1, j-1) / 2 (39)
When Iw> Jw W'i, j = (Swi + 1, j-1 + Swi-1, j + 1) / 2
+ (2G′i, j−G′i + 1, j−1−G′i−1, j + 1) / 2 (40)
When Iw = Jw W'i, j = (Swi + 1, j + 1 + Swi + 1, j-1 + Swi-1, j + 1 + Swi-1, j-1) / 4
+ (4G′i, j−G′i + 1, j + 1−G′i + 1, j−1−G′i−1, j + 1−G′i−1, j−1) / 4
...... (41)

Of the pixels of the W image 42w, for the pixel at the arrangement position corresponding to the G pixel of the original image 41, the gradation of two W pixels (pixels in the original image 41) adjacent to the pixel in the vertical and horizontal directions. G value (luminance value) G ′ of the value (Swi-1, j, Swi + 1, j) or (Swi, j−1, Swi, j + 1) and the corresponding pixel (the pixel in the G image 42g) i, j and G values (luminance values) (G′i−1, j, G′i + 1, j) or (G Based on 'i, j-1, G'i, j + 1), the W value (W'i, j) is determined by the following equation (42) or (43).

W'i, j = (Swi + 1, j + Swi-1, j) / 2
+ (2G′i, j−G′i + 1, j−G′i−1, j) / 2 (42)
When W pixels are adjacent to the left and right W'i, j = (Swi, j + 1 + Swi, j-1) / 2
+ (2G'i, j-G'i, j + 1-G'i, j-1) / 2 (43)

As described above, among the pixels of the W image 42w, the W value (W′i, j) of each pixel other than the pixel at the arrangement position corresponding to the W pixel of the original image 41 is the W pixel around the pixel. The basic value (the first term on the right side of each of the equations (39) to (43)) determined according to the tone value (the tone value in the original image 41) of the pixel and its surrounding pixels It is determined by correcting with the correction value (the second term on the right side of each of the equations (39) to (43)) determined according to the value (G value in the G image 42g).

  Then, among the pixels of the W image 42w, the W value (W′i, j) of each pixel at the arrangement position corresponding to the W pixel of the original image 41 is the level of the W pixel of the original image 41 corresponding to the pixel. It is determined to match the key value Swi, j.

  Thereby, the W image 42w is generated by reflecting the luminance distribution of the G image 42g as the reference luminance distribution image.

  The above is the details of the process (the demosaicing process) in which the external image generation unit 32 generates the four color-specific images 42r, 42g, 42b, and 42w in STEP2.

  Supplementally, a color-specific image 42 other than the G image 42g may be generated as a reference luminance distribution image, and the other three types of color-specific images may be generated by reflecting the luminance distribution of the reference luminance distribution image. For example, in a case where there are a relatively large number of G pixels (so-called blackened G pixels) whose gradation value is a minute value equal to or less than a predetermined value among all the G pixels in the original image 41, W The image 42w is generated as a reference luminance distribution image, and the other three types of color-specific images (R image 42r, G image increase 42g, and B image 42b) are generated by reflecting the luminance distribution of the reference luminance distribution image. May be.

  Further, the entire original image 41 is divided into a plurality of regions, and for each region, one of the four types of color-specific images 42 is generated as a reference luminance distribution image, and the luminance of the reference luminance distribution image is generated. The remaining three types of color-specific images 42 may be generated by reflecting the distribution.

  In STEP 3 following STEP 2, the external image generation unit 32 generates a color image 43. As shown in FIG. 8, the color image 43 includes an R value (Ci, j_r) which is a gradation value of R (red) and a G value (G) which is a gradation value of G (green). Ci, j_g) and a gradation value Ci, j configured as a set of B values (Ci, j_b) that is a gradation value of B (blue) (a set of gradation values of three colors) are assigned. It is an image.

  The color image 43 is generated as a composite image of the R image 42r, the G image increase 42b, and the B image 42b among the color-specific images 42. Specifically, as shown by the following expressions (44a) to (44c), the R value (Ci, j_r), G value (Ci, j_g) of the gradation values Ci, j of each pixel of the color image 43, As the B value (Ci, j_b), the R value (R′i, j) of the corresponding pixel of the R image 42r, the G value (G′i, j) of the corresponding pixel of the G image 42g, and the correspondence of the B image 42b, respectively. The B value (B′i, j) of the pixel is set as it is.

Ci, j_r = R'i, j (44a)
Ci, j_g = G'i, j (44b)
Ci, j_b = B'i, j (44c)

Thereby, a color image 43 is generated.

  In STEP 4 following STEP 3, the external image generation unit 32 generates a wide dynamic range image 44.

  The wide dynamic range image 44 is an image in which the dynamic range is expanded as compared with the color-specific image 42 and the color image 43. As shown in FIG. 9, the wide dynamic range image 44 is a monotone image in which gradation values Di, j are assigned to the respective pixels.

  In this case, the gradation value Di, j of each pixel of the wide dynamic range image 44 is determined as follows.

  First, the external image generation unit 32 has three kinds of gradation values (Ci, j_r, Ci, j_g, Ci, j_b) (or R image 42r, G image 42g, and B image 42b) of each pixel of the color image 43. From the gradation values R′i, j, G′i, j, B′i, j) of the corresponding pixels, the reference gradation values Yi, j corresponding to the luminance values of the pixels of the color image 43 are expressed as follows: Calculated according to equation (45).

Yi, j = 0.3 × Ci, jr + 0.59 × Ci, jg + 0.11 × Ci, jb (45)

Therefore, Yi, j is calculated as a linear combination value of Ci, jr (= R'i, j), Ci, jg (= G'i, j) and Ci, jb (= B'i, j). . In this case, the weighting coefficient (0.3, 0.59, 0.11) of each term is determined by experiments or the like. However, other values may be used for these weighting factors.

  Next, the external image generation unit 32 uses the following equation (46) to calculate the gradation value Swi, j of each W pixel of the original image 41 and the pixel reference floor of the color image 43 at the arrangement position corresponding to each W pixel. A ratio a i, j with the tone value Y ij is calculated as a sensitivity difference correction coefficient.

ai, j = Swi, j / Yi, j (46)

Then, the external image generation unit 32 calculates the gradation value (Di, j) to be assigned to each pixel of the wide dynamic range image 44 using the weighting function w (x) of the following equation (47).

w (x) = 1 / (1 + exp (g × (0.5−x))) (47)

Here, w (x) is a sigmoid function, g is a gain, and exp () is a natural logarithm base e exponential function.

  Specifically, the external image generation unit 32 sets the gradation value (W′i, j) of the W image 42w in the color-specific image 42 to the maximum gradation value (255 or 10 bits if the resolution is 8 bits). For example, the normalized gradation value (hi, j) normalized with respect to 1023) and the reference gradation value (Yi, j) calculated from the color image 43 by the above equation (45) are standardized with the maximum gradation value. From the normalized gradation value (yi, j), the normalized composite gradation value (hdri, j) is calculated by the following equation (48). Note that normalizing the gradation value with respect to the maximum gradation value means dividing the gradation value by the maximum gradation value. For example, when W′i, j = 200, Yi, j = 65 and the maximum gradation value is 255, hi, j = 200/255, yi, j = 65/255.

hdri, j = ((1-w (hi, j)) * hi, j + w (hi, j) * a '* yi, j) / a'
...... (48)

However, when a ′ is a pixel corresponding to the W pixel of the original image 41, the sensitivity difference correction coefficient a i, j of the pixel calculated by the above equation (46), the R pixel of the original image 41, or G When the pixel corresponding to the pixel or the B pixel is targeted, the sensitivity difference correction coefficient a i, j of the pixel corresponding to the W pixel arranged in the periphery is used.

  Then, in order to maintain a low gradation contrast, the external image generation unit 32 performs a γ conversion process on the normalized composite gradation value (hdri, j) according to the following equation (49), so that a wide dynamic range image is obtained. The gradation value (Di, j) assigned to the 44 target pixels is determined.

Di, j = Mb × (hdri, j) ^{(1 / γ)} (49)

However, Mb is the maximum gradation value of the wide dynamic range image 44.

  Thus, the wide dynamic range image 44 is generated by determining the gradation value (Di, j) of each pixel.

  The sigmoid function of the above equation (47) used for generating the wide dynamic range image 44 is an example of a weighting function, and other weighting functions may be used.

  In the present embodiment, four types of color-specific images 42 (42r, 42g, 42b, 42w), a color image 43, and a wide dynamic range image 44 are formed as external images by the processing of the external image generation unit 32 described above. Generated. In this case, the W image 42w of the color-specific images 42 is an image generated by using the gradation value Swi, j of the W pixel indicating the intensity of the output signal of the W light receiving pixel that receives light without passing through the color filter. Therefore, it has a meaning as an image having higher sensitivity than the R image 42r, the G image 42g, and the B image 42b.

  The external image generation unit 32 does not have to generate all the four types of color-specific images 42 (42r, 42g, 42b, 42w), the color image 43, and the wide dynamic range image 44. The generation of the external image may be omitted. For example, the generation of the wide dynamic range image 44 may be omitted.

  Next, the processing of the communication signal detection unit 33 will be described with reference to FIGS.

  First, the basic matters regarding the processing of the communication signal detection unit 33 will be described with reference to FIG. 10. The W light receiving pixel of the image sensor 22 of the camera 2 is incident on the camera 2 without passing through the color filter (this is In general, it includes electromagnetic waves in a wavelength range other than visible light. For this reason, the relationship (spectral characteristics) between the light receiving sensitivity (the ratio of the light receiving level (output signal intensity) of the light receiving pixel to the intensity of the incident light) and the wavelength of the incident light constitutes the W light receiving pixel. It depends on the structure and material of the light receiving element itself.

  Therefore, the spectral characteristic of the light receiving sensitivity of the W light receiving pixel is generally a characteristic as shown by a solid line graph aw in FIG. 10, for example. In this case, the W light receiving pixel has sensitivity in a wide wavelength range including a wavelength in the near infrared region as well as a wavelength in the visible light region, and can detect incident light in the wide wavelength region.

  On the other hand, R light receiving pixels, G light receiving pixels, and B light receiving pixels that receive light incident on the camera 2 through color filters of R, G, and B colors respectively correspond to the light incident on the camera 2. Only light in the wavelength range that can pass through the color filter is received.

  For this reason, the relationship (spectral characteristics) between the light receiving sensitivity of each of the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel and the wavelength of the incident light depends on the characteristics of the light receiving element itself and the transmission characteristics of the corresponding color filters. It depends on and.

  In this case, the spectral characteristics of the light receiving sensitivities of the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel are generally characteristics as illustrated by solid line graphs a r, a g, and a b in FIG. In this case, the light receiving sensitivities of the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel have characteristics of a convex waveform that has sensitivity in a relatively narrow wavelength region near the red wavelength, the green wavelength, and the blue wavelength, respectively. Become.

  Therefore, the wavelength range in which each of the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel can be detected (the wavelength range in which the light receiving sensitivity of each light receiving pixel is relatively high) is narrower than that of the W light receiving pixel. Most of the wavelengths are in the visible light range. The light receiving sensitivity of each of the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel is smaller than the maximum light receiving sensitivity in the visible light region in a wavelength region that deviates from the visible light region, such as the near infrared region. It will be a thing.

  For this reason, attention is paid to the near infrared region as the wavelength region of the communication signal in the present embodiment in the wavelength region that can be detected by the W light receiving pixel (the wavelength region in which the light receiving sensitivity of the W light receiving pixel is relatively high). Then, the near-infrared region is a wavelength region (WL shown in FIG. 10) in which the entire R light receiving pixel, G light receiving pixel, and B light receiving pixel can be detected, that is, the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel. Each becomes a wavelength region near the end of the wavelength region WL obtained by synthesizing the detectable wavelength region, or a wavelength region deviating from the wavelength region WL.

  Therefore, at the wavelength in the near infrared region, the difference (degree of difference) between the light receiving sensitivity of the W light receiving pixel and the light receiving sensitivity of the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel is as follows. The light receiving sensitivity of each of the pixels and the B light receiving pixels is remarkable as compared with the wavelength region where the light receiving sensitivity is relatively large.

  The communication signal detector 33 detects a communication signal in the near-infrared wavelength region using the characteristics as described above. The processing for the detection will be described below according to the flowchart of FIG.

  Note that the processing of the flowchart of FIG. 11 related to the communication signal detection unit 33 includes the same processing as part of the processing of the flowchart of FIG. In this case, the same processes in FIG. 3 and FIG. 11 do not need to be executed separately by both the communication signal detector 33 and the external image generator 32, but may be executed by only one of them. In this case, the communication signal detection unit 33 and the external image generation unit 32 have a common processing function.

  In the following, the processing of the flowchart of FIG. 11 will be described. First, in STEP 11, the original image 41 (see FIG. 2B) is acquired by the original image acquisition unit 31 and held in the image memory 40. The processing of STEP 11 is the same as the processing of STEP 1 described above. Therefore, when the processing of STEP1 is already executed, it is not necessary to perform the processing of STEP11 again.

  Next, the processing of STEPs 12 to 16 is executed by the signal detection image generation unit 34 of the communication signal detection unit 33.

  In STEP 12, the signal detection image generator 34 generates four types of color-specific images 42 (R image 42r, G image 42g, B image 42b, and W image 42w) from the original image by the same processing as in STEP 2. Those color-specific images 42 are stored in the image memory 40. If the color-specific image 42 has already been generated by the processing of STEP2, it is not necessary to perform the processing of STEP12 again.

  Next, in STEP 13, the signal detection image generation unit 34 sets a detection target region in which the communication signal is to be detected in the original image 41. For example, when a communication signal transmitted from a preceding vehicle is to be received, an area in the original image 41 where the preceding vehicle is shown or is likely to be shown (almost the same height as the host vehicle 1). Are set as detection target areas.

  In addition, when receiving a communication signal transmitted from a communication facility on the side of the road, a region where the transmission unit of the communication facility is captured or is likely to be captured is set as a detection target region. The

  The detection target area may be a predetermined area determined in advance according to the type of the transmission source of the communication signal to be received.

  Alternatively, transmission of a communication signal to be received from any one or more of the external images (color-specific image 42, color image 43, wide dynamic range image 44) generated by the external image generation unit 32. After specifying the original location, the detection target area may be set accordingly. Alternatively, the entire area of the original image 41 may be set as the detection target area.

  Next, the signal detection image generation unit 34 executes the processing of STEP14. In this STEP 14, the signal detection image generating unit 34 selects the W image 42w in the detection target region (specifically, the image in the detection target region in the W image 42w previously generated in STEP 12 (or STEP 2)). Obtained as one reference image. Therefore, as shown in FIG. 12A, the first reference image is an image to which the grayscale gradation value W′i, j determined as described above is assigned as the gradation value of each pixel. is there.

  Supplementally, the detection target region may be set before the color-specific image 42 is generated. In that case, in the process of the signal detection image generation unit 34, the color-specific image 42 may be generated only in the set detection target region. In this case, the W image 42w of the color-specific images 42 is generated as the first reference image.

  Next, the signal detection image generation unit 34 executes the processing of STEP15. In this STEP 15, the signal detection image generating unit 34 performs the R image 42r, the G image 42g, and the B image 42b (specifically, the R image 42r and the G image 42g previously generated in STEP 12 (or STEP 2)) in the detection target region. And a second reference image in which a gradation value (monotone gradation value) is assigned to each pixel in the detection target area. Hereinafter, as shown in FIG. 12B, the gradation value of each pixel of the second reference image (the pixel whose arrangement position is (i, j)) is denoted as PYi, j.

  For example, the second reference image is generated as follows. First, the signal detection image generation unit 34 generates a base second reference image of monotone gradation as a previous image of the second reference image. Hereinafter, the gradation value of each pixel (the pixel whose arrangement position is (i, j)) of the base second reference image is denoted as PY0i, j.

  The gradation value PY0i, j of each pixel of the base second reference image is equal to the gradation value R'i, j of the corresponding pixel (image having the same arrangement position) in the R image 42r and the corresponding pixel in the G image 42g. Is determined by linearly combining the gradation value G′i, j of the pixel and the gradation value B′i, j of the corresponding pixel in the B image 42b by the following equation (50).

PY0i, j = αr × R′i, j + αg × G′i, j + αb × B′i, j (50)

αr, αg, and αb are predetermined weighting coefficients. As those values, for example, the same values (0.3, 0.59, 0.11) as those in the formula (45) can be used. However, the values of αr, αg, and αb may be different from the values shown in Expression (45).

  The gradation values PY0i, j calculated in this way are the gradation values R′i, j, G′i, j of the pixels at the arrangement positions corresponding to each of the R image 42r, the G image 42g, and the B image 42b. Since this is a linear combination value of j and B′i, j, the sensitivity characteristic (spectral characteristic with respect to the wavelength of the incident light) of the gradation value PY0i, j with respect to the incident light of the camera 2 is R light receiving pixel, G light receiving pixel, The light receiving sensitivity characteristic of each of the B light receiving pixels is combined. For example, the sensitivity characteristic of the gradation value PY0i, j with respect to the wavelength of the incident light is a characteristic illustrated by a solid line graph ay in FIG. In this case, the gradation value PY0i, j has sensitivity in the wavelength range WL where the entire R light receiving pixel, G light receiving pixel, and B light receiving pixel can be detected.

  Then, the signal detection image generation unit 34 sets, as the second reference image, an image obtained by determining the gradation value PYi, j of each pixel in the detection target region according to PY0i, j by the following equation (51). Generate. That is, the signal detection image generation unit 34 sets the gradation value PYi, j of each pixel of the second reference image to the gradation value PY0i, j of the pixel of the base second reference image corresponding to the arrangement position. The second reference image is generated by multiplying the gain coefficient β of the second reference image.

PYi, j = β × PY0i, j (51)

Here, the gain coefficient β is a coefficient having a value determined in advance based on experiments or the like. The value is the component corresponding to the wavelength of the visible light region included in the gradation value W′i, j of each pixel of the first reference image and the gradation value PYi, j of each pixel of the second reference image. It is set so that the component included and corresponding to the wavelength in the visible light region can be substantially matched.

  As described above, in the second reference image, the gradation value PYi, j of each pixel is the gradation of the pixel at the arrangement position corresponding to each of the R image 42r, the G image 42g, and the B image 42b in the detection target region. A value obtained by multiplying a value obtained by linearly combining values R′i, j, G′i, j, and B′i, j (a value corresponding to the combined luminance value of the pixel) by a gain coefficient β. Is generated.

  Note that the second reference image may be generated before the first reference image.

  Next, the signal detection image generation unit 34 executes the processing of STEP16. In STEP 16, the signal detection image generation unit 34 generates an image of the difference between the first reference image and the second reference image generated as described above as the signal detection image 45. Hereinafter, as shown in FIG. 12C, the gradation value of each pixel (the pixel whose arrangement position is (i, j)) of the signal detection image 45 is denoted as Dfi, j.

  In the signal detection image 45, the gradation value Dfi, j of each pixel (the pixel at the position (i, j)) is set to the pixel at the corresponding position in the first reference image as shown in the following equation (52). The tone value W′i, j of the second reference image and the tone value PYi, j of the pixel at the corresponding position in the second reference image are matched with each other.

Dfi, j = W'i, j-PYi, j (52)

The signal detection image 45 is generated by the processing of the signal detection image generation unit 34 in STEPs 12 to 16 described above.

  Next, in STEP 17, the communication signal detector 33 detects a communication signal based on the signal detection image 45 generated as described above.

  Here, the gradation value W′i, j of each pixel of the first reference image and the gradation value PYi, j of each pixel of the second reference image corresponding thereto correspond to the visible light region included in each of them. Since the components corresponding to the wavelengths substantially match, when a communication signal having a wavelength in the near-infrared region is incident on the position of the image sensor 22 of the camera 2 corresponding to the detection target region of the original image 41, the above signal In the detection image 45, the gradation value Dfi, j of the pixel where the communication signal is incident is relatively significantly larger than that of the surrounding area.

  On the other hand, when the communication signal is not incident on the portion corresponding to the detection target region, generally, the gradation value Dfi, j is remarkably large in the signal detection image 45. The place is hard to occur.

  Therefore, the signal detection image 45 is an image in which a light receiving component having a wavelength in the near infrared region is emphasized.

  Therefore, the communication signal detection unit 33 detects whether or not a communication signal is incident on a location corresponding to the detection target area of the image sensor 22 based on the distribution of the gradation values Dfi, j in the signal detection image 45. To do. For example, when a portion having a predetermined area or larger such that the gradation value Dfi, j is equal to or larger than a predetermined value exists in the signal detection image 45, the portion corresponding to the detection target region of the image sensor 22 is displayed. It detects that a communication signal has entered. Then, when such a location is not detected in the signal detection image 45, the communication signal detection unit 33 determines that the communication signal is not incident on the location corresponding to the detection target region of the image sensor 22. .

  Thereby, it is possible to detect a communication signal having a wavelength in the near-infrared region constituted by an ON / OFF pulse train with high reliability.

  Note that the communication signal incident on the image sensor 22 of the camera 2 is based on the change in the magnitude of the gradation value Dfi, j at a location having a gradation value Dfi, j that is equal to or greater than a predetermined value in the signal detection image 45. It is also possible to detect a change in intensity.

  Further, a target to be communicated is detected with a visible image such as the color-specific image 42, the color image 43, and the wide dynamic range image 44, and the gradation value of the corresponding position of the signal detection image 45 (communication signal emphasized image) A communication signal may be detected using a change in gradation value.

  The details of the processing of the communication signal detection unit 33 have been described above.

  The gain coefficient β in the equation (51) is a constant value in the present embodiment, but may be changed as appropriate according to the imaging conditions of the camera 2. For example, the value of the gain coefficient β may be changed according to the ambient light environment state (daytime, evening, nighttime, shade, sunlit, etc.), the exposure of the camera 2, and the like. In addition, at night when it is difficult to be affected by visible light, the components of the visible light range included in the gradation values of these pixels are minute in both the first reference image and the second reference image. For example, it is possible to detect a communication signal having a wavelength in the near-infrared region using only the first reference image (in other words, β in Expression (51) is set to “0”).

  Further, the gradation value of the first reference image, the gradation value R′i, j of the R image 42r, the gradation value G′i, j of the G image 42g, or the gradation value B′i, j of the B image 42b. It is also possible to detect a communication signal by comparing.

  Next, processing of the object detection unit 35 will be described. The object detection unit 35 selects the color-specific image 42, the color image 43, and the wide dynamic according to the type of the object to be detected, whether it is nighttime (or whether the surrounding environment is dark), and the like. One or more external images of the range image 44 are selected, and an object existing around the vehicle 1 is detected using the selected external image.

  For example, since pedestrians often have relatively low luminance, the object detection unit 35 detects pedestrians from a W image 42w that is a highly sensitive image.

  In addition, since it is necessary to detect a wide range of luminance objects from dark objects to bright objects at night, the object detection unit 35 detects pedestrians, other vehicles, etc. from the wide dynamic range image 44. To do.

  In addition, when it is not nighttime, the color of the object can be recognized from the color image 43, so that lane marks, other vehicles, traffic lights, and the like are detected from the color image 43.

  As described above, by selecting an image to be used for detection of an object according to the type of the object or the brightness of the surrounding environment, various kinds of objects can be highly reliable in various environments. Can be detected.

  When the object detection unit 35 detects a pedestrian or other vehicle, the object detection unit 35 determines the possibility of contact between the pedestrian or other vehicle and the vehicle 1 (own vehicle). If it is determined, a control signal for instructing the vehicle controller 6 to perform the contact avoidance process is transmitted. At this time, the display display control unit 63 of the vehicle controller 6 displays an alarm on the display 73. Moreover, the braking control part 62 performs the process which operates the braking device 72 and avoids a contact as needed.

  Further, the object detection unit 35 transmits to the vehicle controller 6 a control signal for lane keeping control that causes the vehicle 1 (own vehicle) to travel while maintaining the lane mark from the detection position of the lane mark. At this time, the steering control unit 61 controls the operation of the steering device 71 to perform lane keep control.

  According to the embodiment described above, the color-specific images 42 as a plurality of types of external images from the original image 41 composed of W pixels that are transparent pixels and R pixels, G pixels, and B pixels that are color pixels. A color image 43 and a wide dynamic range image 44 can be generated.

  Then, while generating the external image in the visible light region in this way, the signal detection image 45 is generated, and the communication signal in the near-infrared wavelength region is highly reliable based on the signal detection image 45. Can be detected by sex.

  In the embodiment described above, a communication signal in the near-infrared wavelength range is detected by the communication signal detection unit 33, but a wavelength other than the near-infrared region, for example, a wavelength that is higher than that in the near-infrared region. A communication signal in a long infrared region may be detected.

  In this embodiment, the case of detecting a communication signal has been described as an example. However, it is also possible to detect a signal for a purpose other than communication. For example, the camera 2 can be arranged in the vehicle interior of the vehicle 1 to detect a signal for detecting raindrops attached to the window glass of the vehicle 1.

  In this case, for example, the near infrared light is irradiated from the light source such as the near infrared LED to the raindrop detection target region of the window glass, and the reflected light of the near infrared light from the region is incident on the camera 2. Thus, the camera 2 is arranged in the vehicle interior. And the near-infrared light (reflected light) which injected into the camera 2 is detected by the method similar to the communication signal detection part 33 of the said embodiment.

  In this case, since the reflectance of near-infrared light is different between the case where raindrops are attached to the raindrop detection target region of the window glass and the case where no raindrops are attached, the detection of raindrops is performed by detecting the reflected light. The presence or absence can be detected.

  Further, at night or the like, by detecting light in an invisible region that is incident on the camera 2, for example, light in an infrared region such as near-infrared light, using the same method as the communication signal detection unit 33, an infrared camera is connected to the camera 2. It can also be made to have the function as. Thereby, an infrared image (an image similar to the communication signal detection image 45) in which light components in the infrared region are emphasized, and a visible image (the color-specific image 42, the color image 43, the wide dynamic range image 44, etc.). And can be acquired at the same time. At this time, for example, infrared light may be projected onto a part or the whole of the imaging region of the camera 2 (the range of the angle of view of the camera 2) using a light source such as an infrared projector.

  Further, at this time, by synchronizing the exposure control of the camera 2 and the light emission cycle of the light source, the camera 2 may be used as a distance image sensor using a TOF (Time of Flight) method to perform distance detection. Good.

  Further, the arrangement pattern of the light receiving pixels of the image sensor 22 of the camera 2 is not limited to the pattern shown in FIG. For example, a pattern as shown in FIG. 13A or FIG. 13B may be adopted as the array pattern of the light receiving pixels of the image sensor 22 of the camera 2.

  DESCRIPTION OF SYMBOLS 2 ... Camera, 22 ... Image pick-up element, 32 ... Outside world image generation part (outside world image generation means), 33 ... Communication signal detection part (communication signal detection means), 34 ... Signal detection image generation part (Signal detection image generation means) ), STEP12, 14... First reference image generating means, STEP12, 15... Second reference image generating means.

Claims (3)

A camera having an imaging device in which a plurality of color light receiving pixels that receive light through a color filter and a plurality of transparent light receiving pixels that receive light without passing through a color filter;
In the original image captured by the camera, a plurality of color pixels each assigned a gradation value corresponding to the light receiving level of each color light receiving pixel and a gradation value corresponding to the light receiving level of each transparent light receiving pixel are respectively An external image generating means for generating a color or monotone external image indicating an external image from an original image formed by arranging a plurality of assigned transparent pixels;
Based on the gradation value of the color pixel in the detection target area, which is at least a partial area of the original image, and the gradation value of the transparent pixel in the detection target area, the light receiving component in the detection target area Among them, an image in which each of the plurality of color light receiving pixels emphasizes a light receiving component in a predetermined wavelength region near the end of the entire wavelength region or a predetermined wavelength region that deviates from the entire wavelength region. Signal detection image generation means for generating a signal detection image,
An image processing apparatus comprising: signal detection means for detecting a signal component included in the predetermined wavelength region from the signal detection image. The image processing apparatus according to claim 1.
The image processing apparatus according to claim 1, wherein the predetermined wavelength region is a wavelength region in an infrared region. The image processing apparatus according to claim 1 or 2,
The signal detection image generation means sets a gradation value of each pixel in the detection target area according to at least gradation values of a plurality of the transparent pixels included in the detection target area. And a first reference image generating means for generating the first reference image generating means, and a gradation value of each pixel in the detection target area is set according to at least the gradation values of the plurality of color pixels included in the detection target area. A second reference image generating means for generating a two reference image, wherein an image having a difference in gradation value between the first reference image and the second reference image is generated as the signal detection image. Processing equipment.

Images