JP7113283B2 - Imaging device, imaging system, vehicle driving control system, and image processing device - Google Patents

Description

The present disclosure relates to imaging devices, imaging systems, vehicle travel control systems, and image processing devices.

An image reading method is known as a method of acquiring information using an imaging device. The image reading method is a technique for acquiring information from a one-dimensional or two-dimensional code typified by a bar code or QR code (registered trademark). This method can acquire information from a single captured image. This method can also acquire information from multiple codes existing in one image. Therefore, this method is useful when information is acquired while an object such as a vehicle is moving, and control or judgment is performed based on the acquired information. For example, a method has been proposed in which a code is displayed on a tail lamp or a traffic light and the information obtained from the code is used to control the vehicle (see Patent Documents 1 and 2, for example).

JP 2012-221303 A Japanese Patent Application Laid-Open No. 2014-93700

In order to acquire information by the image reading method, it is desired to be able to extract the code with high accuracy.

According to certain non-limiting exemplary embodiments of the present disclosure, the following are provided.

an imaging unit that acquires first image data and second image data including an image of an object displaying a code with invisible light; and extracting the code based on the first image data and the second image data. and an image processing unit that acquires the first image data using a plurality of first pixels that are sensitive to visible light, and obtains the second image data using a plurality of second pixels that are sensitive to invisible light. The image processing unit acquires image data, performs correction processing for multiplying at least one of the first image data and the second image data by a correction coefficient, and converts the first image data and the second image data after the correction processing. An imaging device that generates third image data from a difference in image data and extracts the code based on the third image data.

A generic or specific aspect may be embodied in an element, device, module, system, integrated circuit or method. Also, the generic or specific aspects may be implemented by any combination of elements, devices, modules, systems, integrated circuits and methods.

Additional effects and advantages of the disclosed embodiments will become apparent from the specification and drawings. Benefits and/or advantages are provided individually by the various embodiments or features disclosed in the specification and drawings, and not all are required to obtain one or more of these.

The present disclosure can provide an imaging device, an imaging system, a vehicle travel control system, or an image processing device that can extract codes with high accuracy.

FIG. 4 is a diagram showing an example of an image including a code displayed with infrared light according to the first embodiment; It is a figure which shows an example of the 3rd image which concerns on 1st Embodiment. 1 is a diagram showing a configuration example of a vehicle travel control system according to a first embodiment; FIG. 3 is a diagram illustrating a configuration example of an imaging unit according to the first embodiment; FIG. 3 is a diagram showing an example of a circuit configuration of a pixel according to the first embodiment; FIG. 3 is a diagram showing an example of a circuit configuration of a pixel according to the first embodiment; FIG. 2 is a cross-sectional view showing an example of a photoelectric conversion unit according to the first embodiment; FIG. 4 is a diagram showing an example of a color arrangement of unit pixels according to the first embodiment; FIG. 4 is a diagram showing an example of a color arrangement of unit pixels according to the first embodiment; FIG. 4 is a diagram showing an example of a color arrangement of unit pixels according to the first embodiment; FIG. 4 is a diagram showing an example of a color arrangement of unit pixels according to the first embodiment; FIG. It is a figure which shows an example of the code|cord|chord which concerns on 1st Embodiment. It is a figure which shows an example of the code|cord|chord which concerns on 1st Embodiment. It is a figure which shows an example of the code|cord|chord which concerns on 1st Embodiment. It is a figure which shows an example of the code|cord|chord which concerns on 1st Embodiment. 1 is a block diagram of a vehicle cruise control system according to a first embodiment; FIG. 2 is a block diagram of an imaging system according to a second embodiment; FIG. 10 is a flowchart of information acquisition processing when acquiring a first image and a second image at the same time according to the second embodiment; FIG. 10 is a flowchart of information acquisition processing when sequentially acquiring a first image and a second image according to the second embodiment; FIG. FIG. 11 is a block diagram of an image processing apparatus according to a third embodiment; FIG. FIG. 11 is a flowchart of information acquisition processing when acquiring a first image and a second image at the same time according to the third embodiment; FIG. It is a figure which shows an example of the spectral sensitivity characteristic of R pixel, G pixel, and B pixel which concern on 3rd Embodiment. FIG. 11 is a diagram showing an example of spectral sensitivity characteristics of IR pixels according to the third embodiment; FIG. 11 is a flowchart of information acquisition processing when sequentially acquiring a first image and a second image according to the third embodiment; FIG. It is a figure which shows an example of the spectral sensitivity characteristic of G pixel which concerns on 3rd Embodiment. FIG. 11 is a diagram showing an example of spectral sensitivity characteristics of G+IR pixels according to the third embodiment; It is a figure which shows an example of the spectrum of sunlight which concerns on 4th Embodiment. FIG. 14 is a diagram showing a result of multiplication of spectral sensitivity characteristics of R pixels, G pixels, and B pixels and the spectrum of sunlight according to the fourth embodiment; FIG. 11 is a diagram showing a result of multiplication of spectral sensitivity characteristics of IR pixels and the spectrum of sunlight according to the fourth embodiment; FIG. 11 is a block diagram of an image processing apparatus according to a fifth embodiment; FIG. FIG. 14 is a flowchart of information acquisition processing when acquiring a first image and a second image simultaneously according to the fifth embodiment; FIG. FIG. 4 is a diagram showing an example of a code image captured by a rolling shutter operation according to the embodiment; FIG. 10 is a diagram showing an example of an image of a code imaged by global shutter operation according to the embodiment;

(Knowledge leading to this disclosure)
In the acquisition of information by the image reading method, it is a problem to extract the code with high accuracy. If multiple codes existing in the captured image can be extracted at high speed and with high accuracy, it can be used as a means of automating vehicle control or providing new information to passengers for ADAS (Advanced Driver Assistance System) technology or automatic driving technology. Useful.

A summary of one aspect of the present disclosure follows.

An imaging device according to an aspect of the present disclosure includes an imaging unit that acquires first image data and second image data including an image of an object that displays a code with invisible light; 2, an image processing unit for extracting the code based on image data, wherein the imaging unit acquires the first image data by a plurality of first pixels sensitive to visible light, and sensitive to non-visible light; and the image processing unit performs a correction process of multiplying at least one of the first image data and the second image data by a correction coefficient, and corrects the Third image data is generated from the difference between the first image data and the second image data, and the code is extracted based on the third image data.

According to this, the imaging device generates the third image data by difference processing between the first image data in which visible light is captured and the second image data in which non-visible light is captured. As a result, the contrast of the code in the third image data can be increased, so that the imaging device can extract the code with high precision.

Further, according to this, the imaging device can reduce the difference in signal intensity between the first image data and the second image data, so that the code contrast in the third image data can be made higher. This allows the imaging device to extract the code with higher accuracy.

For example, the image processing unit processes the first image data such that the signal intensity of the image other than the code in the second image data is greater than the signal intensity of the image other than the code in the first image data. and the correction process of multiplying at least one of the second image data by the correction coefficient.

For example, a portion of the object may emit invisible light, and the object may display the code by emitting invisible light at the portion.

For example, a part of the object has a higher reflectance for invisible light than a reflectance for visible light, and the part of the object reflects the invisible light to display the code. good.

For example, in the second image data, the highest signal intensity of the image of the code is set as the first intensity, the highest signal intensity of the image other than the code is set as the second intensity, and the third image. In the data, the highest signal intensity of the code image is set as the third intensity, and the highest signal intensity of the image other than the code is set as the fourth intensity. The intensity ratio may be greater than the ratio of said first intensity to said second intensity.

For example, the image processing unit may reduce the difference between the signal intensity of the image other than the code in the first image data and the signal intensity of the image other than the code in the second image data. The correction process may be performed by multiplying at least one of the image data and the second image data by the correction coefficient.

According to this, since the difference in signal strength between the first image data and the second image data can be reduced, the code contrast in the third image data can be increased. This allows the imaging device to extract the code with higher accuracy.

For example, the correction coefficient may be calculated based on a first spectral sensitivity characteristic of the first pixel and a second spectral sensitivity characteristic of the second pixel.

For example, the correction coefficient is a first characteristic obtained by multiplying the first spectral sensitivity characteristic by the sunlight spectrum, and a second characteristic obtained by multiplying the second spectral sensitivity characteristic by the sunlight spectrum. and may be calculated based on.

According to this, the contrast of the code in the third image data can be made higher by considering the spectrum of sunlight. This allows the imaging device to extract the code with higher accuracy.

For example, the second pixels may also be sensitive to visible light.

For example, a day/night determination unit for determining whether it is daytime or nighttime is further provided, and when the day/night determination unit determines that it is daytime, the image processing unit extracts the code based on the third image data, The image processing unit may extract the code based on the second image data when the day/night determination unit determines that it is nighttime.

According to this, it is possible to reduce the processing amount of the imaging device at night when the influence of invisible light contained in sunlight is small.

For example, the image processing section may acquire information associated with the extracted code. Further, for example, the information may include traffic information.

For example, the first pixel and the second pixel may be separate pixels.

According to this, the imaging device can acquire the first image data and the second image data at the same time. Therefore, the imaging device can reduce the shift of the image between the first image data and the second image data caused by the movement, so that the code can be extracted with higher accuracy.

For example, the first pixel and the second pixel are the same third pixel, and the third pixel has a first state sensitive to visible light and a second state sensitive to invisible light. wherein the photographing unit acquires the first image data from the third pixels in the first state and acquires the second image data from the third pixels in the second state .

According to this, the imaging device can acquire the first image data and the second image data with the same pixels. Therefore, the imaging device can reduce the number of pixels.

For example, the timing of the start and end of the exposure period when the first pixel acquires the first image data is the timing of the start and end of the exposure period when the second pixel acquires the second image data. may be the same as

According to this, the imaging device can reduce the distortion of the image of the code caused by movement, so that the code can be extracted with higher accuracy.

For example, the imaging unit acquires the first image data and the second image data in each of a plurality of frame periods, and the image processing unit acquires the first image data and the second image data in each of the plurality of frame periods. may extract the code only.

An imaging system according to an aspect of the present disclosure includes the imaging device and a notification unit that notifies a user of the information.

A vehicle cruise control system according to an aspect of the present disclosure includes the imaging device and a vehicle control unit that controls braking and acceleration of a vehicle based on the information.

An image processing device according to an aspect of the present disclosure is configured such that first image data including a visible light image of an object for displaying a code with invisible light and second image data including an invisible light image of the object are an image processing unit for extracting the code based on the code, the image processing unit performs correction processing for multiplying at least one of the first image data and the second image data by a correction coefficient, and obtains the code by the correction processing. Third image data is generated from the difference between the two image data, and the code is extracted based on the third image data.

According to this, the image processing device generates the third image data by difference processing between the first image data in which visible light is captured and the second image data in which non-visible light is captured. As a result, the image processing device can increase the contrast of the image of the code in the third image data, so that the code can be extracted with high accuracy.

For example, the image processing section may acquire information associated with the extracted code.

In addition, these general or specific aspects may be realized by a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. and any combination of recording media.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In addition, all the embodiments described below show comprehensive or specific examples. Numerical values, shapes, materials, components, arrangement and connection forms of components, steps, order of steps, and the like shown in the following embodiments are examples, and are not intended to limit the present disclosure. The various aspects described herein are combinable with each other unless inconsistent. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in independent claims representing the highest concept will be described as optional constituent elements. In the following description, constituent elements having substantially the same functions are denoted by common reference numerals, and their description may be omitted. As used herein, "image data" means data containing image information. Also, "image data" may be simply referred to as "image".

(First embodiment)
First, the configuration and operation of the vehicle travel control system 100 according to this embodiment will be described.

A vehicle travel control system 100 includes an imaging device 101 mounted on a vehicle. The imaging device 101 reads the code and acquires information associated with the code.

FIG. 3 is a diagram showing an example of the hardware configuration of the vehicle cruise control system 100 according to this embodiment. A vehicle travel control system 100 shown in FIG. 3 includes an imaging device 101, an optical system 102, a display section 103, a network communication section 104, and an ECU 105 as an electronic control unit. The imaging device 101 includes an imaging unit 111 , an ISP 112 , a system controller 115 , an image transmission unit 113 and an information acquisition unit 114 . The ISP 112 is an image processor that performs image processing on output data from the imaging unit 111 . System controller 115 controls how ISP 112 performs image processing. The image transmission unit 113 transmits the data output from the ISP 112 to the outside.

The optical system 102 has well-known lens groups. In the lens group, for example, the focus lens may move in the optical axis direction. Thereby, the focus position of the subject image can be adjusted in the imaging unit 111 .

The imaging unit 111 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor. FIG. 4 is a diagram showing a configuration example of the imaging unit 111. As shown in FIG. As shown in FIG. 4, the imaging unit 111 includes a pixel array including a plurality of pixels 201 arranged in rows and columns, a row scanning circuit 202, a column scanning circuit 203, and current sources 204 provided for each column. , and an AD conversion circuit 205 provided for each column.

Each pixel 201 is connected to a horizontal signal line 207 in a corresponding row and a vertical signal line 208 in a corresponding row. Each pixel 201 is individually selected by a row scanning circuit 202 and a column scanning circuit 203 . A power supply voltage is supplied to each pixel 201 through a common power supply line 206 . Each pixel 201 converts an input optical signal into an electrical signal. The AD conversion circuit 205 converts the electrical signal obtained by the pixel 201 into a digital signal. This digital signal is output from the column scanning circuit 203 to the output signal line 209 . Note that the AD conversion circuit 205 may not be provided, and the electric signal may be output as an analog signal. Alternatively, the output signals of a plurality of pixels 201 may be added and subtracted, and the calculated values may be output.

5A and 5B are diagrams showing configurations of pixels 201A and 201B, which are examples of the pixel 201. FIG. The pixel 201A has, as shown in FIG. 5A, a photoelectric conversion unit 210, a charge storage unit 225, a reset transistor M3, an amplification transistor M1, and a selection transistor M2. The photoelectric conversion unit 210 photoelectrically converts incident light. The charge storage unit 225 stores charges. The reset transistor M3 resets the potential of the charge storage unit 225 to the desired reset potential V1. The amplification transistor M1 outputs a signal corresponding to the amount of charge accumulated in the charge accumulation section 225 . The selection transistor M2 selectively outputs a signal to the vertical signal line 208. FIG.

Note that a photodiode 211 may be used as the photoelectric conversion unit as shown in FIG. 5B.

FIG. 6 is a cross-sectional view showing a configuration example of the pixel 201A shown in FIG. 5A. As shown in FIG. 6 , the photoelectric conversion section 210 has a transparent electrode 221 , a pixel electrode 223 and a photoelectric conversion layer 222 . A photoelectric conversion layer 222 is located between the transparent electrode 221 and the pixel electrode 223 . At least a portion of the charge storage section 225 may be located within the semiconductor substrate 226 . The charge storage section 225 may be electrically connected to the pixel electrode 223 through the contact plug 224 . An electric field is generated when a bias voltage is applied between the transparent electrode 221 and the pixel electrode 223 . Due to this electric field, one of positive and negative charges generated by photoelectric conversion in the photoelectric conversion layer 222 is collected by the pixel electrode 223 . The collected charges are stored in the charge storage section 225 .

Pixels 201 are sensitive to visible and invisible light. 7A to 7D show an example arrangement of the pixels 201. FIG. Also, as an example of invisible light, a case of targeting infrared light will be described. In FIGS. 7A-7D, each square represents a single pixel. G written in the pixel is a green wavelength band, R is a red wavelength band, B is a blue wavelength band, IR is an infrared wavelength band, and W is a wavelength band from visible light to infrared light. Indicates that the pixel has sensitivity. Hereinafter, G pixels are pixels that are sensitive to the green wavelength band, R pixels are pixels that are sensitive to the red wavelength band, B pixels are pixels that are sensitive to the blue wavelength band, and sensitivity is to the infrared wavelength band. A pixel having a sensitivity to a wavelength band from visible light to infrared light is called a W pixel. The pixel array has four pixels as one unit, and the same configuration is arranged vertically and horizontally.

FIG. 7A shows a configuration example in which one of the G pixels is replaced with an IR pixel in the Bayer array. An IR pixel is typically realized by forming a visible light cut filter on the pixel. The Bayer array is an array in which four pixels, one R pixel, two G pixels, and one B pixel, are set as one unit and the same configuration is arranged vertically and horizontally. Also, two G pixels are arranged diagonally and are not adjacent to each other vertically or horizontally.

FIG. 7B shows a configuration example in which one of the G pixels is replaced with a W pixel in the Bayer array. A W pixel is typically realized by not forming an on-chip color filter on the pixel.

FIG. 7C shows a configuration example in which all pixels are sensitive to the wavelength band of infrared light in the Bayer array. Typically, the on-chip color filter on each pixel is configured to transmit light in the wavelength band of visible light (red light, green light, or blue light) and light in the wavelength band of infrared light. . Further, an infrared light cut filter is mounted, and the state in which the infrared light cut filter is arranged on the optical path to the photographing unit 111 and the state in which it is not arranged are switched on the front surface of the photographing unit 111 . By controlling in this way, for each pixel, signal acquisition by visible light (red light, green light or blue light) and signal acquisition by visible light (red light, green light or blue light) and infrared light are performed. You can switch.

FIG. 7D shows a configuration example in which each pixel in the Bayer array switches between a state of sensitivity to the wavelength band of infrared light and a state of not having sensitivity to the wavelength band of infrared light. Typically, state switching is accomplished by controlling the voltage applied to the pixel.

For example, in the pixel shown in FIG. 6, by switching the potential difference between the transparent electrode 221 and the pixel electrode 223, the wavelength band of light obtained by each pixel can be changed. For example, the photoelectric conversion layer 222 may be composed of two materials with different absorption spectra. When the potential difference between the transparent electrode 221 and the pixel electrode 223 is larger than a predetermined value, the pixel 201 is sensitive to infrared light, and when it is smaller than the predetermined value, the pixel 201 is sensitive to red light. It may be configured so as not to have sensitivity to external light. Alternatively, in a pixel with a photodiode, for example, the charge trapping region may be varied by changing the bias voltage to the carrier ejection region located below the photodiode. Thereby, the wavelength band of light acquired by each pixel can be changed. Further, for example, by applying a high bias voltage to the carrier discharge region, the pixel 201 is made sensitive only to visible light, and by applying a low bias voltage to the carrier discharge region, sensitivity is also made to infrared light. can be made to have

Specifically, when the photoelectric conversion portion has the configuration shown in FIG. 6, the sensitivity per unit time of the pixel 201 can be changed by changing the potential difference between the transparent electrode 221 and the pixel electrode 223. be. For example, by reducing the potential difference between the transparent electrode 221 and the pixel electrode 223, the charge generated by photoelectric conversion in the photoelectric conversion layer 222 can be prevented from being detected. In other words, the sensitivity of the pixel 201 can be made substantially zero. For example, by making the transparent electrode 221 common to all pixels and simultaneously setting the sensitivity of all pixels to zero, a global shutter operation is possible. Note that when the photoelectric conversion unit is the photodiode 211, a global shutter operation can be performed by providing a charge transfer transistor and a charge storage capacitor. Since the photographing unit 111 has a global shutter function, the photographing unit 111 can photograph the code without distortion even when the vehicle is moving at high speed. As a result, the code recognition accuracy of the imaging device 101 can be improved. Details about the code will be described later.

Please refer to FIG. 3 again. The ISP 112 performs image processing on the data output from the imaging unit 111 as necessary. The data output from the imaging unit 111 is, for example, RAW data. Image processing is, for example, compression processing or correction processing. Note that the imaging unit 111 and the ISP 112 may be provided on the same chip. As a result, it is possible to realize high speed processing and low cost of the device. The ISP 112 outputs the data output from the imaging unit 111 as it is or after image processing to the image transmission unit 113 and the information acquisition unit 114 . The data output from the ISP 112 may be, for example, RAW data that has not been subjected to image processing, or data in a predetermined format in which RAW data has been subjected to image processing.

The image transmission unit 113 outputs the data output from the ISP 112 to an external device such as the ECU 105, for example. The information acquisition unit 114 extracts the code 50 from the captured image. The code 50 is obtained by converting information into code according to certain rules. The information acquisition unit 114 also decodes the extracted code 50 to convert the code 50 into information such as traffic information. 8A-8D are diagrams illustrating examples of code 50. FIG. For example, as shown in Figures 8A-8C, code 50 is a two-dimensional pattern. Specifically, as the code 50, a QR code (registered trademark) as shown in FIG. 8A, an arbitrary pattern as shown in FIG. 8B, or characters or symbols as shown in FIG. 8C can be used. Alternatively, as the code 50, a bar code that is a one-dimensional pattern as shown in FIG. 8D may be used.

In addition, the information acquisition unit 114 performs, for example, decoding processing described in JIS X 0510:2004 for QR Codes (registered trademark) or acquisition of corresponding information by pattern matching as the decoding processing. Further, when the code 50 is information itself, the information acquisition unit 114 performs character recognition or the like as decoding processing. Note that the information acquisition unit 114 may be implemented by the ISP 112 .

The extracted information may, for example, be sent to the ECU 105, which will be described later, and used to control the speed and braking of the vehicle. Alternatively, the extracted information may be sent to the display unit 103 and used for presenting traffic information to the driver, warning of danger by display or sound, or the like.

The system controller 115 includes an input section 122 , an output section 121 , a program memory 123 , a working memory 125 and a control microcomputer 126 . A program 124 is stored in the program memory 123 . The microcomputer 126 executes the program 124 using the working memory 125, thereby functioning as, for example, a control unit of the imaging apparatus 101. FIG. Note that the system controller 115 may control the imaging unit 111 in addition to controlling the ISP 112 . The output unit 121 outputs the control result of the system controller 115 to the imaging unit 111 .

A display unit 103 displays a camera image. The network communication unit 104 is means for wirelessly connecting to the Internet. For example, the network communication unit 104 acquires information necessary for information acquisition processing in the information acquisition unit 114 from the Internet via the network communication unit 104 . The ECU 105 is a unit that controls various vehicle bodies such as the engine, braking, and acceleration.

Note that these components do not need to be configured independently, and may be integrated.

FIG. 9 is a block diagram showing the functional configuration of the vehicle cruise control system 100 according to this embodiment. The vehicle travel control system 100 is mounted on a vehicle, for example. As shown in FIG. 9 , the vehicle travel control system 100 includes an imaging device 101 and a vehicle control section 131 .

The imaging device 101 includes an imaging unit 111 and an image processing device 140 . The image processing device 140 is implemented by, for example, the ISP 112 and the information acquisition unit 114 shown in FIG. The image processing device 140 includes an image processing section 142 .

Note that the image processing device 140 may exist outside the imaging device 101 . For example, the imaging device 101 may transmit the captured image to the external image processing device 140 . Image processor 140 may perform processing on the received image.

The image capturing unit 111 captures a first image 161 containing an image of an object, which is an image captured with visible light, and a second image 162, which is an image captured with non-visible light and includes an image of the object. and Note that the second image 162 may be an image in which visible light and non-visible light are captured. Moreover, invisible light is, for example, infrared light. Also, the first image 161 and the second image 162 captured by the capturing unit 111 are stored in the frame memory. The frame memory may be built in the imaging unit 111 or ISP 112, or may be a discrete memory device.

The object is, for example, a display medium that displays the code 50 . Code 50 is associated with information, for example. The object may be, for example, a sign, billboard, electronic billboard, or building. Code 50 is displayed in non-visible light. As a result, the object can display information without impairing the visibility or design of the appearance observed with visible light. Specifically, the object may display the code 50 by emitting invisible light. For example, the object may be a light source that emits invisible light. Alternatively, the object may display code 50 by reflecting non-visible light. For example, in the portion where the code 50 of the object is displayed, the reflectance for invisible light may be higher than the reflectance for visible light. In this specification, the code 50 is, for example, a pattern composed of a portion that emits or reflects more invisible light and other portions, and is associated with specific information.

In FIG. 1, a display medium 51 is a road sign, a display medium 52 is a signboard, and a display medium 53 is a vehicle. FIG. 1 shows an example of images of a display medium 51, a display medium 52, and a display medium 53 captured by a camera capable of capturing non-visible light components. In FIG. 1, the background portion other than the code 50 is also shown. This is because ordinary objects such as people, vehicles, and buildings also emit invisible light due to the invisible light contained in sunlight outdoors during the day. That is, even if the image is captured by an imaging device having sensitivity in the infrared region, it may be difficult to extract the code 50 with high contrast as shown in FIG. In this embodiment, the code 50 is extracted with high accuracy as shown in FIG. 2 by image processing to be described later.

For example, the imaging unit 111 may include a plurality of first pixels sensitive to visible light and a plurality of second pixels sensitive to invisible light. A first image 161 is obtained by a plurality of first pixels. A second image 162 is obtained by a plurality of second pixels. The first pixels are, for example, the R, G and B pixels shown in FIG. 7A or 7B. The second pixel is, for example, the IR pixel shown in FIG. 7A or the W pixel shown in FIG. 7B.

Alternatively, the imaging unit 111 includes a plurality of pixels 201 and has a first imaging state in which each pixel 201 is sensitive to visible light and a second imaging state in which the plurality of pixels 201 is sensitive to invisible light. may Thereby, the first image 161 is obtained in the first photographing state. A second image 162 is obtained in the second photographing state. That is, the first image 161 and the second image 162 are obtained from the same pixels 201 . For example, as in the configuration shown in FIG. 7C or 7D, the same pixel 201 may be switched between a state of sensitivity to the wavelength band of visible light and a state of sensitivity to the wavelength band of non-visible light. Thereby, the first image 161 and the second image 162 can be obtained from the same pixels 201 .

The image processing unit 142 acquires the first image 161 and the second image 162 from the imaging unit 111 . The image processing unit 142 includes a difference processing unit 151 and an extraction unit 152 .

The difference processing unit 151 generates a third image 163 including the image of the object by difference processing between the first image 161 and the second image 162 . This difference processing generates a third image 163 in which the ratio of the signal intensity of the image of the light emitting portion of the code 50 and the signal intensity of the image other than the code 50 is greater than that of the second image 162 . This facilitates extraction of the image of the code 50 from the third image 163 . That is, in the second image 162, the highest signal intensity of the image of the code 50 is set as the first intensity, and the highest signal intensity of the images other than the code 50 is set as the second intensity. In the third image 163, the highest signal intensity of the image of the code 50 is the third intensity, and the highest signal intensity of the images other than the code 50 is the fourth intensity. In this case, the ratio of the third intensity to the fourth intensity is greater than the ratio of the first intensity to the second intensity.

Extraction unit 152 extracts code 50 from third image 163 . The extraction unit 152 also acquires the information 144 associated with the extracted code 50 . Information 144 is, for example, traffic information. Traffic information is, for example, information indicating the content of road signs. Specifically, for example, it is information indicating prohibition of approach or prohibition of entry.

Vehicle control unit 131 controls braking and acceleration of the vehicle based on information 144 . For example, the vehicle control unit 131 is implemented by the ECU 105 shown in FIG.

Note that the present disclosure can be implemented not only as the vehicle travel control system 100 but also as the imaging device 101 or the image processing device 140 .

(Second embodiment)
In this embodiment, information 144 is communicated to the user. A user is, for example, a driver of a vehicle. FIG. 10 is a block diagram of an imaging system 100A according to this embodiment. The imaging system 100A differs from the vehicle travel control system 100 of the first embodiment in that a notification section 132 is provided instead of the vehicle control section 131 .

The notification unit 132 notifies the user of the information 144 . For example, notification unit 132 displays information 144 . Alternatively, the notification unit 132 may notify the user of the information 144 by voice or the like. For example, the notification unit 132 is implemented by the display unit 103 shown in FIG. The information 144 itself may be notified to the user, or a warning or the like based on the information 144 may be notified to the user. Note that the vehicle travel control system 100 may further include a notification unit 132 .

The operation of the vehicle travel control system 100 configured as described above will be described below.

Information acquisition processing by the imaging unit 111 will be described below based on the flowchart shown in FIG. 11 . In this embodiment, the imaging unit 111 has the pixel configuration shown in FIG. 7A or 7B. That is, the imaging unit 111 can acquire the first image 161 and the second image 162 at the same time.

The imaging unit 111 starts imaging. Imaging may be initiated under manual control by the vehicle occupant. Alternatively, the imaging may be automatically started in response to a starting action such as starting the engine of the vehicle.

Next, the photographing unit 111 photographs the first image 161 and the second image 162 (S101).

Next, the image processing section 142 acquires the first image 161 and the second image 162 captured by the imaging section 111 . The difference processing unit 151 generates a third image 163 by difference processing between the first image 161 and the second image 162 (S102). For example, the difference processing unit 151 calculates the difference between the sum of the output values of R, G and B pixels and the output value of IR or W pixels. Note that the difference processing unit 151 may calculate the difference between the output value of any one of the R, G, and B pixels predetermined and the output value of the IR pixel or W pixel. Further, the difference processing unit 151 performs difference processing between corresponding pixels of the first image 161 and the second image 162 .

Next, the extraction unit 152 generates a binary image by performing binarization processing on the obtained third image 163 using a predetermined threshold value (S103). The threshold is set in advance based on the characteristics of the imaging unit 111 or experimental results.

Next, the extraction unit 152 extracts the code 50 from the binary image using a known method such as pattern matching (S104). Specifically, the extraction unit 152 extracts all the codes 50 in the image, and stores the extracted codes 50 and the types of the codes 50 in memory. Here, the type of code 50 is a bar code, a QR code (registered trademark), or the like.

Finally, the extraction unit 152 acquires the information 144 corresponding to the extracted code 50 (S105). For example, the extractor 152 holds a table showing information 144 corresponding to code 50 . The extraction unit 152 obtains the information 144 corresponding to the code 50 using this table. Note that the extraction unit 152 may acquire the information 144 corresponding to the code 50 by collating with data on the network. Alternatively, the acquisition process itself of the information 144 may be performed on the cloud.

Note that there is a possibility that multiple codes 50 are extracted in the extraction of the code 50 (S104). When multiple codes 50 are extracted, the extraction unit 152 acquires information 144 for each of the multiple codes 50 . In this case, when the first code 50 is extracted, the extraction unit 152 can acquire the information 144 corresponding to the code 50 . Therefore, the extraction unit 152 may execute steps S104 and S105 in parallel.

FIG. 12 shows a flowchart of information acquisition processing when the configuration of the imaging unit 111 is different. In this case, the imaging unit 111 has the configuration shown in FIG. 7C or 7D. That is, the imaging unit 111 acquires the first image 161 and the second image 162 in this order. The flowchart shown in FIG. 12 differs from the flowchart shown in FIG. 9 in that step S101 is replaced with steps S101A and S101B.

The imaging unit 111 captures the first image 161 (S101A). After that, the imaging state is switched, and the second image 162 is captured (S101B). For example, in the configuration shown in FIG. 7C, an infrared light cut filter is placed on the optical path to the imaging unit 111 in step S101A. On the other hand, in step S101B, the infrared light cut filter is removed from the optical path to the imaging unit 111. FIG. As a result, the photographing state is switched. Further, in the configuration shown in FIG. 7D , the imaging state is switched by controlling the voltage applied to the pixel 201 . Specifically, by changing the potential difference between the transparent electrode 221 and the pixel electrode 223, the photographing state can be switched.

Since the subsequent processing is the same as in FIG. 11, the description is omitted. Note that in this example, the imaging order of the first image 161 and the second image 162 may be reversed.

By performing the difference processing as shown in this embodiment, the output values of the second image 162 that are caused by invisible light contained in sunlight can be reduced using the first image 161 . As a result, the code 50 can be made to have a higher contrast than the portions other than the code 50, so that the code can be extracted at high speed and with high accuracy.

The information acquisition process shown in FIG. 11 or 12 may be performed constantly while the engine of the vehicle is running. Alternatively, it may be performed at certain intervals. Also, this information acquisition processing may be performed starting from external information. For example, it may be detected that the vehicle has entered a specific area such as a school zone, and the information acquisition process may be started with that as a starting point. Also, this information acquisition processing may be started with a change in the control information inside the vehicle, such as when the gear is changed.

In the above description, the third image 163 is generated by difference processing between the first image 161 and the second image 162 . However, depending on the situation, it may be difficult to accurately detect the code 50 by simple difference processing alone. For example, the signal intensity of the image of the light emitting portion of the code 50 in the second image 162 is smaller than the signal intensity of the image of the light emitting portion of the code 50 in the first image 161 . In such a case, when the first image 161 is subtracted from the second image 162, the absolute value of the difference between the images of the light emitting portion of the code 50 may become smaller than the absolute value of the difference between the images other than the code 50. . That is, the image of the code 50 may be inverted in the third image 163 .

Here, the signal intensity of the image of the light emitting portion of the code 50 in the second image 162 is the signal intensity of the image of the light emitting portion of the code 50 when it is assumed that the code 50 does not emit invisible light. The signal strength due to non-visible light emitted by the part is added. In addition, the signal intensity of the image of the light emitting portion of the code 50 assuming that the code 50 does not emit invisible light is considered to be equivalent to the signal intensity of the image other than the code 50 . Therefore, if the signal intensity of the images other than the code 50 in the second image 162 is corrected to be substantially the same as or greater than the signal intensity of the images other than the code 50 in the first image 161, the code 50 in the second image 162 can be made higher than the signal intensity of the image of the light emitting portion of the code 50 in the first image 161 . That is, the inversion of the image of the code 50 in the third image 163 can be suppressed.

As such correction, for example, an arbitrary region other than the code 50 is set in the first image 161 and the second image 162, and the minimum value of the signal intensity of the second image 162 in that region is the same as that of the first image in the same region. The correction coefficient may be calculated so as to be greater than the minimum value of the 161 signal strength. Alternatively, based on the spectral sensitivity characteristics of the first pixels that acquire the first image 161 and the spectral sensitivity characteristics of the second pixels that acquire the second image 162, the signal intensity of the images other than the code 50 in the second image 162 However, the correction coefficient may be calculated so that the signal intensity of the image other than the code 50 in the first image 161 becomes substantially equal. Also, the spectrum of sunlight may be taken into account in calculating the correction coefficient. Specific embodiments are described below.

(Third embodiment)
In this embodiment, the image processing device 140A processes the first image 161 and the second image 162 so that the output values of the corresponding pixels of the first image 161 and the second image 162 are the same for light of the same intensity. correct at least one of More specifically, the image processing device 140A performs correction based on differences in the quantum efficiency and spectral sensitivity characteristics of pixels used to capture the first image 161 and the second image 162 . As a result, the signal intensity of the image of the light emitting portion of the code 50 in the second image 162 can be corrected to be higher than the signal intensity of the image of the light emitting portion of the code 50 in the first image 161 .

FIG. 13 is a block diagram showing the configuration of an image processing device 140A according to this embodiment. The image processing device 140A is different from the image processing device 140 in that the image processing section 142A includes a correction section 153 .

Information acquisition processing using the image processing device 140A will be described below with reference to the flowchart shown in FIG. Note that the imaging unit 111 has the pixel configuration shown in FIG. 7A or 7B. That is, the imaging unit 111 acquires the first image 161 and the second image 162 at the same time. The process shown in FIG. 14 differs from the process shown in FIG. 11 in that step S106 is added.

In step S106, the correction unit 153 corrects the first image 161 and the second image 162 so that the difference between the output values of the corresponding pixels in the first image 161 and the second image 162 is small in the area other than the code 50. Multiply the correction factor. That is, the correction unit 153 multiplies the first image 161 by the first correction coefficient to generate the corrected first image 161A. Further, the correction unit 153 multiplies the second image 162 by the second correction coefficient to generate the corrected second image 162A. Note that the correction unit 153 may multiply only one of the first image 161 and the second image 162 by the correction coefficient.

Next, the difference processing unit 151 generates the third image 163 by difference processing between the corrected first image 161A and the corrected second image 162A (S102). It should be noted that subsequent processing is the same as in FIG. 11, and a description thereof will be omitted.

The first correction coefficient and the second correction coefficient, or the correction coefficient, are the first spectral sensitivity characteristics of the pixels 201 used to capture the first image 161 and the second spectral sensitivity characteristics of the pixels 201 used to capture the second image 162 . It is calculated based on the spectral sensitivity characteristics.

FIG. 15A is a diagram showing an example of spectral sensitivity characteristics of R pixels, G pixels, and B pixels. FIG. 15B is a diagram showing an example of spectral sensitivity characteristics of IR pixels. For example, the correction coefficient is set in advance so that the sum of the integral values of the R, G, and B pixel spectral sensitivity characteristic graphs is the same as the integral value of the IR pixel spectral sensitivity characteristic graph. . In this example, the integral value of the R pixel graph is 4.1, the integral value of the G pixel graph is 4.3, the integral value of the B pixel graph is 2.6, and the IR pixel graph has an integral value of 4.3. The integrated value of the graph is 5.4. Therefore, for example, when only the first image 161 is multiplied by the correction coefficient, by using 5.4/(4.1+4.3+2.6)=0.49 as the correction coefficient, the R pixel, the G pixel, and the B pixel Correction can be made so that the sum of the pixel output values and the IR pixel output value are equivalent.

Also, when only the second image 162 is multiplied by the correction coefficient, the correction coefficient is the reciprocal of the numerical value in this example. That is, 1/0.49=2.04 is used as the correction coefficient. When correcting both the first image 161 and the second image 162, the ratio between the first correction coefficient multiplied by the first image 161 and the second correction coefficient multiplied by the second image 162 is , is set to be the same as the ratio between the output value of the second image 162 and the output value of the first image 161 .

In addition, as shown in FIG. 7B, the same applies when W pixels are used instead of IR pixels. Note that the spectral sensitivity characteristic of each pixel may be set in advance, and the correction coefficient may be set in advance during system development.

FIG. 16 shows a flowchart of information acquisition processing when the configuration of the imaging unit 111 is different. In this case, the imaging unit 111 has the pixel configuration shown in FIG. 7C or 7D. That is, the imaging unit 111 acquires the first image 161 and the second image 162 in this order. The flowchart shown in FIG. 16 differs from the flowchart shown in FIG. 14 in that step S101 is replaced with steps S101A and S101B.

The spectral sensitivity characteristics of the G+IR pixels will be described with reference to FIGS. 17A and 17B. G+IR pixels may be sensitive to the green light wavelength band and the infrared light wavelength band. FIG. 17A is a diagram showing an example of spectral sensitivity characteristics when G+IR pixels are set to have sensitivity in the wavelength band of green light. FIG. 17A is hereinafter referred to as the G-state graph. FIG. 17B is a diagram showing an example of spectral sensitivity characteristics when the G+IR pixel is set to have sensitivity in the wavelength band of green light and the wavelength band of infrared light. FIG. 17B is hereinafter referred to as the G+IR state graph. For example, the correction coefficient is set in advance so that the integral value of the G state graph and the integral value of the G+IR state graph are the same. In this example, the G state graph has an integral value of 4.3, and the G+IR state graph has an integral value of 9.7. Therefore, when only the first image 161 is multiplied by the correction coefficient, 9.7/4.3=2.25 is used as the correction coefficient so that the output value of the G pixel when the first image 161 is acquired and the , the output values of the G+IR pixels at the time of acquisition of the second image 162 can be corrected to be equivalent.

Also, when only the second image 162 is multiplied by the correction coefficient, the correction coefficient is the reciprocal of the numerical value in this example. That is, 1/2.25=0.44 is used as the correction coefficient. When correcting both the first image 161 and the second image 162, the ratio between the first correction coefficient multiplied by the first image 161 and the second correction coefficient multiplied by the second image 162 is It is set to be the same as the ratio between the output value of the second image 162 and the output value of the first image 161 .

In this manner, the correction unit 153 performs correction so that the sensitivities of the corresponding pixels of the first image 161 and the second image 162 are the same. This further reduces the output value of the code 50 background. Therefore, the extraction unit 152 can extract only the code 50 with high accuracy.

(Fourth embodiment)
In this embodiment, a modification of the third embodiment will be described. In this embodiment, the spectrum of sunlight is also used to calculate the correction coefficient. More specifically, the spectral sensitivity characteristics of the pixels used to capture the first image 161 and the spectral sensitivity characteristics of the pixels used to capture the second image 162 are multiplied by the spectrum of sunlight. Used as a correction factor. As a result, the signal intensity of the image of the light-emitting portion of the code 50 in the second image 162 can be corrected more accurately so that it becomes higher than the signal intensity of the image of the light-emitting portion of the code 50 in the first image 161 .

FIG. 18 is a diagram showing the spectrum of sunlight. Thus, sunlight has a spectrum with a maximum value near 500 nm and a gentle decrease in intensity toward longer wavelengths. Therefore, by introducing a correction coefficient that considers the influence of the spectrum of sunlight, the influence of sunlight can be removed with higher accuracy. The output value of the imaging unit 111 when sunlight is incident is obtained by multiplying the spectrum of sunlight and the spectral sensitivity characteristic of the imaging unit 111 at each wavelength. That is, in the present embodiment, the correction coefficients, or the first correction coefficient and the second correction coefficient, are obtained by multiplying the first spectral sensitivity characteristics of the pixels 201 when the first image 161 was obtained by the sunlight spectrum. and the second characteristic obtained by multiplying the second spectral sensitivity characteristic of the pixel 201 when the second image 162 was obtained by the sunlight spectrum.

19A is a graph showing the multiplication result of the spectrum of sunlight shown in FIG. 18 and the spectral sensitivity characteristics of the R, G, and B pixels shown in FIG. 15A. 19B is a graph showing the result of multiplication of the spectrum of sunlight shown in FIG. 18 and the spectral sensitivity characteristics of the IR pixels shown in FIG. 15B. As in the third embodiment, for example, the correction coefficient is set so that the sum of the integral values of the graph shown in FIG. 19A is the same as the integral value of the graph shown in FIG. 19B. In this example, the integral value of the graph corresponding to the R pixel shown in FIG. 19A is 3.4, the integral value of the graph corresponding to the G pixel is 3.9, and the integral value of the graph corresponding to the B pixel The value is 2.0. Also, the integrated value of the graph corresponding to the IR pixel shown in FIG. 19B is 3.2. Therefore, when correcting only the first image 161, by using 3.2/(3.4+3.9+2.0)=0.34 as the correction coefficient, the output of the R pixel, the G pixel, and the B pixel is and the output of the IR pixel can be equated. The same applies when W pixels are used instead of IR pixels as shown in FIG. 7B.

Also, when only the second image 162 is corrected, the correction coefficient is the reciprocal of the numerical value in this example. When correcting both the first image 161 and the second image 162, the ratio between the first correction coefficient multiplied by the first image 161 and the second correction coefficient multiplied by the second image 162 is It is set to be the same as the ratio between the output of the second image 162 and the output of the first image 161 .

In this manner, the correction unit 153 performs correction so that the output values of the corresponding pixels of the first image 161 and the second image 162 are the same in consideration of the sunlight spectrum. This allows the output value of the background portion of the code 50 to be further reduced. Therefore, the extraction unit 152 can extract only the code 50 with high accuracy.

(Fifth embodiment)
FIG. 20 is a block diagram showing the configuration of an image processing device 140B according to this embodiment. The image processing device 140B shown in FIG. 20 is different from the image processing device 140 in that it includes a day/night determination unit 143 .

FIG. 21 is a flowchart of information acquisition processing according to this embodiment. The process shown in FIG. 21 differs from the process shown in FIG. 11 in that steps S111 to S113 are added.

First, the day/night determination unit 143 determines whether it is daytime or nighttime (S111). For example, the day/night determination unit 143 determines whether it is daytime or nighttime based on the detection result of an illuminance sensor provided in the vehicle. Alternatively, the day/night determination unit 143 may determine whether it is daytime or nighttime based on the current time.

If it is daytime (Yes in S111), the processes after step S101 are performed in the same manner as the process shown in FIG. That is, the third image 163 is generated by the difference processing section 151 and the extraction section 152 extracts the code 50 from the third image 163 .

On the other hand, if it is now nighttime (No in S111), the image processing device 140B extracts the code 50 from the second image 162 without generating the third image 163. FIG. Specifically, the image capturing unit 111 captures the second image 162, and the image processing unit 142 acquires the captured second image 162 (S112). Next, the extraction unit 152 generates a binary image by performing binarization processing on the second image 162 using a predetermined threshold value (S113). Next, the extraction unit 152 extracts the code 50 from the binary image (S104). Then, the extraction unit 152 acquires the information 144 corresponding to the extracted code 50 (S105).

At night, the influence of the sun, which is a large source of invisible light components, is small. Therefore, the code can be extracted with relatively high accuracy without performing differential processing. The image processing device 140B can reduce the amount of processing and extract the code at high speed by not performing differential processing at night.

Although the example in which the first image 161 and the second image 162 are obtained simultaneously has been described here, the same applies to the case in which the first image 161 and the second image 162 are obtained sequentially. Further, here, the description has been made based on the configuration of the first embodiment, but similar modifications are applied to the configurations of the second, third, and fourth embodiments. can.

Further, in the first to fifth embodiments, the imaging device 101 may have a global shutter function. That is, the imaging unit 111 captures the first image 161 and the second image 162 in a single exposure period within one frame period using multiple first pixels and multiple second pixels. In the exposure period, the timing of starting and ending the exposure of the plurality of first pixels and the plurality of second pixels are the same.

FIG. 22A is a diagram showing an example of an image of code 50 in an image obtained by rolling shutter operation. FIG. 22B is a diagram showing an example of an image of code 50 in an image obtained by global shutter operation. As shown in FIGS. 22A and 22B, global shutter operation can reduce distortion of the image of code 50 . For example, even when the vehicle is moving at high speed, the imaging unit 111 can capture the image of the code 50 without distortion. Therefore, the recognition accuracy of the code 50 can be improved.

Further, in the first to fifth embodiments described above, the image processing unit 142 may extract codes only for some frame periods among a plurality of frame periods. That is, the imaging device 101 acquires the first image 161 and the second image 162 in each of a plurality of frame periods, and the image processing unit 142 extracts the code only for some frame periods. can be

As described above, according to each embodiment, it is possible to extract the code 50 as shown in FIG. 2 with high accuracy.

Although the system and apparatus according to the embodiments have been described above, the present disclosure is not limited to these embodiments.

For example, division of functional blocks in a block diagram is an example, and a plurality of functional blocks can be realized as one functional block, one functional block can be divided into a plurality of functional blocks, and some functions can be moved to other functional blocks. may

Also, the order in which each step in the flowchart is executed is for illustrative purposes in order to specifically describe the present disclosure, and orders other than the above may be used. Also, some of the above steps may be executed concurrently (in parallel) with other steps.

Each processing unit included in each device according to the above embodiments is typically implemented as an LSI, which is an integrated circuit. These may be made into one chip individually, or may be made into one chip so as to include part or all of them.

Further, circuit integration is not limited to LSIs, and may be realized by dedicated circuits or general-purpose processors. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure connections and settings of circuit cells inside the LSI may be used.

Also, in each of the above embodiments, part of each component may be realized by executing a software program suitable for the component. The components may be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded in a recording medium such as a hard disk or semiconductor memory.

Although the system and apparatus according to one or more aspects have been described above based on the embodiments, the present disclosure is not limited to the embodiments. As long as it does not deviate from the spirit of the present disclosure, the scope of one or more embodiments includes various modifications that a person skilled in the art can think of, and a configuration constructed by combining the components of different embodiments. may be included within

INDUSTRIAL APPLICABILITY The present disclosure is useful as various imaging devices. The present disclosure can also be applied to applications such as digital cameras, digital video cameras, mobile phones with cameras, medical cameras such as electronic endoscopes, vehicle-mounted cameras, and robot cameras.

50 code 51, 52, 53 display medium 100 vehicle running control system 100A imaging system 101 imaging device 102 optical system 103 display unit 104 network communication unit 105 electronic control unit (ECU)
111 imaging unit 112 ISP
113 image transmission unit 114 information acquisition unit 115 system controller 121 output unit 122 input unit 123 program memory 124 program 125 working memory 126 microcomputer 131 vehicle control unit 132 notification unit 140, 140A, 140B image processing device 142, 142A image processing unit 143 day and night Determination unit 144 Information 151 Difference processing unit 152 Extraction unit 153 Correction unit 161, 161A First image 162, 162A Second image 163 Third image 201, 201A, 201B Pixel 202 Row scanning circuit 203 Column scanning circuit 204 Current source 205 AD conversion Circuit 206 Power supply line 207 Horizontal signal line 208 Vertical signal line 209 Output signal line 210 Photoelectric conversion unit 211 Photodiode 221 Transparent electrode 222 Photoelectric conversion layer 223 Pixel electrode 224 Contact plug 225 Charge storage unit 226 Semiconductor substrate M1 Amplification transistor M2 Selection transistor M3 reset transistor

Claims (21)

an imaging unit that acquires first image data and second image data including an image of an object that displays a code with invisible light;
an image processing unit that extracts the code based on the first image data and the second image data;
with
The imaging unit includes a plurality of first pixels sensitive to visible light and a plurality of second pixels sensitive to non-visible light, and acquires the first image data from the plurality of first pixels, obtaining the second image data from the plurality of second pixels;
The image processing unit performs correction processing for multiplying at least one of the first image data and the second image data by a correction coefficient, and generates third image data from a difference between the two image data obtained by the correction processing. and extracting the code based on the third image data;
The image processing unit is configured to process the first image data and the performing the correction process of multiplying at least one of the second image data by the correction coefficient;
Imaging device. an imaging unit that acquires first image data and second image data including an image of an object that displays a code with invisible light;
an image processing unit that extracts the code based on the first image data and the second image data;
with
The imaging unit includes a plurality of first pixels sensitive to visible light and a plurality of second pixels sensitive to non-visible light, and acquires the first image data from the plurality of first pixels, obtaining the second image data from the plurality of second pixels;
The image processing unit performs correction processing for multiplying at least one of the first image data and the second image data by a correction coefficient, and generates third image data from a difference between the two image data obtained by the correction processing. and extracting the code based on the third image data;
In the second image data, the highest signal intensity of the image of the code is set as the first intensity, the highest signal intensity of the image other than the code is set as the second intensity, and in the third image data , the highest signal intensity of the image of the code is the third intensity, and the highest signal intensity of the image other than the code is the fourth intensity, the ratio of the third intensity to the fourth intensity is a ratio is greater than the ratio of the first intensity to the second intensity;
Imaging device. an imaging unit that acquires first image data and second image data including an image of an object that displays a code with invisible light;
an image processing unit that extracts the code based on the first image data and the second image data;
with
The imaging unit includes a plurality of first pixels sensitive to visible light and a plurality of second pixels sensitive to non-visible light, and acquires the first image data from the plurality of first pixels, obtaining the second image data from the plurality of second pixels;
The image processing unit performs correction processing for multiplying at least one of the first image data and the second image data by a correction coefficient, and generates third image data from a difference between the two image data obtained by the correction processing. and extracting the code based on the third image data;
The image processing unit processes the first image data so as to reduce a difference between a signal intensity of an image other than the code in the first image data and a signal intensity of an image other than the code in the second image data. and performing the correction process of multiplying at least one of the second image data by the correction coefficient;
Imaging device. an imaging unit that acquires first image data and second image data including an image of an object that displays a code with invisible light;
an image processing unit that extracts the code based on the first image data and the second image data;
with
The imaging unit includes a plurality of first pixels sensitive to visible light and a plurality of second pixels sensitive to non-visible light, and acquires the first image data from the plurality of first pixels, obtaining the second image data from the plurality of second pixels;
The image processing unit performs correction processing for multiplying at least one of the first image data and the second image data by a correction coefficient, and generates third image data from a difference between the two image data obtained by the correction processing. and extracting the code based on the third image data;
It further comprises a day/night determination unit that determines whether it is daytime or nighttime,
When the day/night determination unit determines that it is daytime, the image processing unit extracts the code based on the third image data,
When the day/night determination unit determines that it is nighttime, the image processing unit extracts the code based on the second image data.
Imaging device. 3. The imaging device according to claim 2 , wherein said correction coefficient is calculated based on a first spectral sensitivity characteristic of said first pixel and a second spectral sensitivity characteristic of said second pixel. The correction coefficient is a first characteristic obtained by multiplying the first spectral sensitivity characteristic by the sunlight spectrum, and a second characteristic obtained by multiplying the second spectral sensitivity characteristic by the sunlight spectrum. 6. The imaging device according to claim 5 , wherein the imaging device is calculated based on A portion of the object emits invisible light,
The imaging device according to any one of claims 1 to 4, wherein the object displays the code by emitting invisible light at the part. part of the object has a higher reflectance for invisible light than for visible light,
The imaging device according to any one of claims 1 to 4, wherein the object displays the code by reflecting invisible light at the part. The imaging device according to any one of claims 1 to 4, wherein the second pixels are also sensitive to visible light. 5. The imaging device according to any one of claims 1 to 4, wherein said image processing unit acquires information associated with said extracted code. 11. The imaging device according to claim 10 , wherein said information includes traffic information. the first pixel and the second pixel are the same third pixel;
the third pixel has a first state sensitive to visible light and a second state sensitive to invisible light;
5. The photographing unit acquires the first image data from the third pixels in the first state and acquires the second image data from the third pixels in the second state. The imaging device according to any one of . The timing of the start and end of the exposure period when the first pixel acquires the first image data is the same as the timing of the start and end of the exposure period when the second pixel acquires the second image data. The imaging device according to any one of claims 1 to 4, wherein: the imaging unit acquires the first image data and the second image data in each of a plurality of frame periods;
5. The imaging device according to claim 1, wherein said image processing section extracts said code only for some of said plurality of frame periods. an imaging device according to claim 10 ;
a notification unit that notifies the user of the information;
An imaging system comprising: an imaging device according to claim 10 ;
a vehicle control unit that controls braking and acceleration of the vehicle based on the information;
A vehicle cruise control system comprising: an image processing unit for extracting the code based on first image data including a visible light image of an object for which the code is displayed using invisible light and second image data including an invisible light image of the object; prepared,
The image processing unit performs correction processing for multiplying at least one of the first image data and the second image data by a correction coefficient, and generates third image data from a difference between the two image data obtained by the correction processing. and extracting the code based on the third image data;
The image processing unit is configured to process the first image data and the performing the correction process of multiplying at least one of the second image data by the correction coefficient;
Image processing device. an image processing unit that extracts the code based on first image data including a visible light image of an object for which the code is displayed using invisible light and second image data including an invisible light image of the object; prepared,
The image processing unit performs correction processing for multiplying at least one of the first image data and the second image data by a correction coefficient, and generates third image data from a difference between the two image data obtained by the correction processing. and extracting the code based on the third image data;
In the second image data, the highest signal intensity of the image of the code is set as the first intensity, the highest signal intensity of the image other than the code is set as the second intensity, and in the third image data , the highest signal intensity of the image of the code is the third intensity, and the highest signal intensity of the image other than the code is the fourth intensity, the ratio of the third intensity to the fourth intensity is a ratio is greater than the ratio of the first intensity to the second intensity;
Image processing device. an image processing unit that extracts the code based on first image data including a visible light image of an object for which the code is displayed using invisible light and second image data including an invisible light image of the object; prepared,
The image processing unit performs correction processing for multiplying at least one of the first image data and the second image data by a correction coefficient, and generates third image data from a difference between the two image data obtained by the correction processing. and extracting the code based on the third image data;
The image processing unit processes the first image data so as to reduce a difference between a signal intensity of an image other than the code in the first image data and a signal intensity of an image other than the code in the second image data. and performing the correction process of multiplying at least one of the second image data by the correction coefficient;
Image processing device. an image processing unit that extracts the code based on first image data including a visible light image of an object for which the code is displayed using invisible light and second image data including an invisible light image of the object; prepared,
The image processing unit performs correction processing for multiplying at least one of the first image data and the second image data by a correction coefficient, and generates third image data from a difference between the two image data obtained by the correction processing. and extracting the code based on the third image data;
It further comprises a day/night determination unit that determines whether it is daytime or nighttime,
When the day/night determination unit determines that it is daytime, the image processing unit extracts the code based on the third image data,
When the day/night determination unit determines that it is nighttime, the image processing unit extracts the code based on the second image data.
Image processing device. The image processing device according to any one of claims 17 to 20, wherein said image processing section acquires information associated with said extracted code.

Images