DE102016217282A1 - IMAGE SENSOR, IMAGING DEVICE, DRIVER ASSISTANCE SYSTEM, VEHICLE AND METHOD FOR EVALUATING ELECTROMAGNETIC RADIATION - Google Patents

Abstract

The invention relates to an image sensor (1) with pixels (2) for detecting electromagnetic radiation. The pixels (2) comprise a first set (21) of pixels, which are each designed to detect at least a portion of a visible portion of the electromagnetic radiation and to output respective first sensor data. The pixels (2) further comprise a second set (22) of pixels adapted to detect an infrared portion of the electromagnetic radiation and to output respective second sensor data, and a third set (23) of pixels formed therein to detect the total visible portion of the electromagnetic radiation and to output respective third sensor data. The image sensor (1) comprises a data evaluation device (3) which is designed to output a first evaluation signal based on the first sensor data, output a second evaluation signal based on the second sensor data and output a third evaluation signal based on the third sensor data.

Description

The invention relates to an image sensor with pixels for detecting electromagnetic radiation, an image capture device, a driver assistance system for a vehicle, a vehicle and a method for evaluating electromagnetic radiation by means of an image sensor.

Modern motor vehicles assume certain driving functions of the vehicle, such as accelerating or steering the vehicle partially or completely autonomous. For this purpose, the driver assistance systems rely on an accurate detection and classification of the objects in the vehicle environment of the vehicle. In addition to radar sensors, infrared sensors or lidar systems, especially vehicle cameras are used.

Common image sensors of such vehicle cameras have array-shaped pixels with color filters, so that only certain portions of the visible spectrum are detected by the respective pixel. Based on the brightness values detected by the pixels, a color image is generated. As an example, in the document US 3 971 065 A mentioned so-called Bayer arrangement in which the image sensor comprises a plurality of blocks, each comprising a red pixel, two green pixels and a blue pixel. This arrangement is particularly well adapted to human perception.

However, the large number of environmental sensors installed in modern motor vehicles is disadvantageous on the one hand due to the large amount of space required and on the other hand due to the additional weight.

It is therefore an object of the invention to provide a compact image sensor for accurate detection of the vehicle environment.

This object is achieved by an image sensor with pixels for detecting electromagnetic radiation having the features of patent claim 1, an image capture device having the features of claim 9, a driver assistance system for a vehicle having the features of claim 10, a vehicle having the features of claim 13 and a method for the evaluation of electromagnetic radiation with the features of claim 14 solved.

Further preferred embodiments are subject of the dependent claims.

According to a first aspect, the present invention accordingly provides an image sensor, in particular for a vehicle, with pixels for detecting electromagnetic radiation. The pixels include a first set, a second set, and a third set of pixels. The pixels of the first set are designed in each case to detect at least a partial area of a visible portion of the electromagnetic radiation and to output respective first sensor data. The pixels of the second set are configured to detect an infrared portion of the electromagnetic radiation and output respective second sensor data. The pixels of the third set are configured to detect the entire visible portion of the electromagnetic radiation and to output respective third sensor data. Furthermore, the image sensor comprises a data evaluation device, which is designed to output a first evaluation signal based on the first sensor data, to output a second evaluation signal based on the second sensor data and to output a third evaluation signal based on the third sensor data.

The visible fraction of the electromagnetic radiation is to be understood as the wavelength range which can be detected by the human eye, which is preferably between 380 nm and 780 nm. By infrared radiation is preferably understood electromagnetic radiation having wavelengths greater than 780 nm, more preferably between 780 nm and 1 mm.

According to a second aspect, the invention relates to an image acquisition device with an image sensor and an image processing device, which is coupled to the image sensor, and is configured to generate a color image based on the first evaluation signal and process it, to generate an infrared image based on the second evaluation signal and to process this, and to generate a greyscale image based on the third evaluation signal and to process this.

According to a third aspect, the invention relates to a driver assistance system for a vehicle having an image acquisition device, wherein the image processing device of the image acquisition device is further configured to generate a color image and / or an infrared image and / or a gray scale image from the signals supplied by the data evaluation device of the image acquisition device To extract features of objects, such as pedestrians, and / or environmental properties, such as a surface type. The method of machine vision in this case is supported by the fact that different features of the same structure can be extracted in the individual images of different wavelength ranges, which considerably increases the reliability in recognition and classification.

According to a fourth aspect, the invention relates to a vehicle with a Driver assistance system, which is adapted, depending on the detected objects in the environment of the vehicle and the environmental characteristics to influence the vehicle dynamics, for example by steering or braking.

According to a fifth aspect, the invention relates to a method for evaluating electromagnetic radiation by means of an image sensor. The electromagnetic radiation is detected by the pixels of the first set of pixels and respective first sensor data is output. Further, the electromagnetic radiation is detected by the pixels of the second set of pixels, and respective second sensor data are output. The electromagnetic radiation is detected by the pixels of the third set of pixels and respective third sensor data is output. The data evaluation device outputs a first evaluation signal based on the first sensor data, a second evaluation signal based on the second sensor data, and a third evaluation signal based on the third sensor data.

The invention thus makes it possible to simultaneously generate color images, infrared images and grayscale images on the basis of sensor data of a single image sensor. The image capture device thereby generates a natural color image that may be displayed to the driver and an infrared image that can provide information about heat sources in the vehicle environment, such as pedestrians, cyclists or motorcyclists, as well as a grayscale image, which due to the high photosensitivity of the Pixels of the third set allows a very precise detection and determination of objects.

According to a preferred embodiment of the image sensor, the data evaluation device is designed to directly write the sensor data of the first set of pixels into a first virtual channel during readout of the sensor signals, write the sensor data of the second set of pixels into a second virtual channel, and Write sensor data of the third set of pixels in a third virtual channel. Using virtual channels does not require buffering of the full pixel data on the image sensor. Virtual channels are preferably used according to the Camera Serial Interface 2 (CSI-2) specification of the Mobile Industry Processor Interface (MIPI) alliance.

By using the virtual channels, the division of the sensor data can already be carried out directly on the image sensor, whereby a faster and possibly more accurate evaluation in the image generating device is made possible, as would be the case if the data is transmitted in combination to the image processing device and separated there and would be evaluated.

According to a further embodiment of the image sensor, adjacent pixels of the first set of pixels and / or the second set of pixels and / or the third set of pixels are linked to pixel clusters by means of pixel binning. The linkage is preferably carried out by the data evaluation device on a hardware level. In this case, the pixels are preferably already combined in an analogous manner in the read-out circuits of the pixels themselves. Pixel binning can improve the signal-to-noise ratio because the noise is statistically distributed and precision is increased by averaging over several pixels.

According to a further embodiment of the image sensor, the data evaluation device is designed to combine sensor data of pixels of a pixel cluster. For example, the data evaluation device can average over the brightness values determined by the pixels of a pixel cluster. According to a preferred development of the image sensor, the pixel array has a plurality of blocks, each block having at least one pixel of the first set of pixels, at least one pixel of the second set of pixels and at least one pixel of the third set of pixels. The pixels are thereby evenly distributed on the pixel array to ensure that the areas imaged on the color image, the infrared image and the grayscale image correspond. A parallax effect is avoided, which could occur when using spatially separated image sensors for capturing color images, infrared images and grayscale images.

According to a development of the image sensor, each block has in each case one pixel of the first set of pixels, one pixel of the second set of pixels, and two pixels of the third set of pixels, wherein the two pixels of the third set of pixels are pixel-binned to form a pixel cluster are linked. The block preferably has a square 2 × 2 shape.

According to a further embodiment of the image sensor, the third set of pixels has an infrared cut-off filter which is designed to completely or at least partially filter out the infrared portion of the electromagnetic radiation.

According to a further embodiment of the image sensor, the second set of pixels has a visible-light cut-off filter which is designed to completely or at least partially filter out the visible portion of the electromagnetic radiation.

According to a further embodiment of the driver assistance system, the image processing device is further configured to produce a display image based on the color image and / or on the basis of the extracted objects and / or environmental properties. The driver assistance system further has a display device, which is designed to display the display image to a driver of the vehicle.

According to a further embodiment of the driver assistance system, the latter has a control device which is designed to control at least one driving function of the vehicle on the basis of the objects and / or ambient properties extracted by the image processing device. For example, the control device can steer, accelerate or decelerate the vehicle or activate or deactivate an actuator of the vehicle, such as a turn signal, a windshield wiper or an illumination of the vehicle. In particular, an automatic emergency braking for pedestrians can be performed.

According to a development of the method, adjacent pixels of the first set of pixels and / or the second set of pixels and / or the third set of pixels are linked to pixel clusters by means of pixel binning. Preferably, two pixels are considered adjacent if they have a common edge or vertex in an arrayed arrangement.

The present invention will be explained in more detail with reference to the exemplary embodiments indicated in the schematic figures of the drawings.

Show it:

1 a schematic block diagram of an image sensor according to an embodiment of the invention;

2 a pixel array of an image sensor according to an embodiment of the invention;

3 a schematic virtual first subarray of pixels for explaining the generation of a color image;

4 a schematic virtual second subarray of pixels for explaining the generation of an infrared image;

5 a schematic virtual third subarray of pixels for explaining the generation of a gray scale image;

6 a schematic block diagram of an image capture device according to an embodiment of the invention;

7 a schematic block diagram of a driver assistance system according to an embodiment of the invention;

8th a schematic block diagram of a vehicle according to an embodiment of the invention; and

9 a flowchart for explaining a method for evaluating electromagnetic radiation according to an embodiment of the invention.

If appropriate, the described embodiments and developments can be combined as desired. Further possible refinements, developments and implementations of the invention also include combinations, not explicitly mentioned, of features of the invention described above or below with regard to the exemplary embodiments.

The accompanying drawings are intended to provide further understanding of the embodiments of the invention. They illustrate embodiments and, together with the description, serve to explain principles and concepts of the invention. Other embodiments and many of the stated advantages will become apparent with reference to the drawings. The elements of the drawings are not necessarily shown to scale to each other. The same reference numerals designate the same or similar components.

In 1 is a schematic block diagram of an image sensor 1 illustrated. The image sensor 1 has a plurality of light-sensitive detection units or pixels 2 which are adapted to detect electromagnetic radiation and output a corresponding sensor signal or corresponding sensor data on the basis of a light intensity or a brightness value of the detected radiation. The pixels 2 include a first lot 21 of pixels, which are designed as color pixels to detect at least a portion of a visible portion of the electromagnetic radiation and output corresponding first sensor signals. The first lot 21 of pixels includes red pixels 21a , green pixels 21b and blue pixels 21c , which each have a corresponding color filter, which transmit electromagnetic radiation in a green, blue or red wavelength range and pass in a respective remaining visible wavelength range not or only limited. According to another Embodiment, a single color filter can also be multiple pixels 2 be assigned.

Next include the pixels 2 a second lot 22 of pixels which are designed as infrared pixels to detect an infrared component of the electromagnetic radiation and to output corresponding second sensor signals. Preferably, the pixels are facing 2 the second lot 22 of pixels, a visible light cut-off filter, which filters out the visible portion of the electromagnetic radiation, so that only infrared radiation from the pixels 2 is detected. The one of the first lot 21 of pixels and the second set 22 Partial regions of the electromagnetic radiation detected by pixels are thus preferably disjoint.

Next include the pixels 2 a third lot 23 of pixels, which are designed as clear pixels to detect the entire visible portion of the electromagnetic radiation and to output corresponding third sensor signals. The pixels of the third set 23 may also detect the infrared portion of the electromagnetic radiation. Preferably, the pixels are the third set 23 However, an infrared cut-off filter, which is adapted to filter out the infrared portion of the electromagnetic radiation, so that the pixels of the third set 23 capture only the visible portion of the electromagnetic radiation. Preferably, therefore, one of the third amount 23 pixel portion of the electromagnetic radiation and one of the second quantity 22 of pixels detected partial area of the electromagnetic radiation disjoint.

In 2 is an exemplary pixel distribution of the pixels 2 illustrated. The pixels 2 of the image sensor 1 are arranged in a pixel array A. Four pixels in each case are grouped in a 2 × 2 block B, each block B being a color pixel Ri of the set of red pixels 21a , a color pixel Gia, Give the amount of green pixels 21b or a color pixel Bi of the set of blue pixels 21c includes. Furthermore, each block B comprises an infrared pixel IRi of the second set 22 of pixels as well as two clear pixels Cia, Cib the third lot 23 of pixels. An index i in this case runs from 1 to 16 in accordance with the 16 marked blocks B. The blocks B are in turn arranged in such an array that the pixels of the first set 21 of pixels considered individually arranged in a Bayer pattern with a distribution RGGB.

In the 2 shown distribution of pixels 2 is only an example and a variety of other arrangements is also possible.

The image sensor 1 further comprises a data evaluation device 3 , which is adapted to, which is adapted to output based on the first sensor data, a first evaluation signal. The data evaluation device 3 For this purpose, the color pixels Ri, Gia, Gib, Bi of a virtual first subarray A1 are evaluated 3 is illustrated, and only the pixels of the first set 21 of each block B includes. As described above, the color pixels Ri, Gia, Gib, Bi are the first set 21 of pixels arranged here in a Bayer pattern. However, any further arrangement of the color pixels Ri, Gia, Gib, Bi is also possible.

The data evaluation device 3 is further configured to output a second evaluation signal based on the second sensor data. The data evaluation device 3 For this purpose, the brightness distributions of infrared pixels IRi of a virtual second subarray A2 are evaluated, which in 4 is illustrated, and the pixels of the second set 22 of each block B of the pixel array A.

Next is the data evaluation device 3 designed to output a third evaluation signal based on the third sensor data. The data evaluation device evaluates this 3 the detected brightness values of pixels 2 one in 5 illustrated virtual third subarray A3, which the clear pixels Cia, Cib of the third set 23 of each block of the pixel array A. The two clear pixels Cia, Cib of each block B are hereby linked to a single pixel cluster Ci by means of pixel binning. The clear pixels Cia, Cib are preferably already linked together at hardware level or analogously. However, the clear pixels Cia, Cib can also be obtained from the data evaluator 3 be linked. The pixel cluster Ci consisting of the linked clear pixels Cia, Cib of the third set 23 of pixels, are from the data evaluator 3 evaluated.

The data evaluation device writes to generate the virtual first subarray A1, second subarray A2 and third subarray A3 3 preferably those of the pixels of the first set 21 , the second lot 22 and the third lot 23 generated sensor data into a first virtual channel, a second virtual channel and a third virtual channel, wherein the first virtual subarray A1, the second virtual subarray A2 and the third virtual subarray A3 based on the sensor data of the first virtual channel, the second virtual channel and the third virtual channel can be generated. The first, second and third virtual channels form respective interfaces of the image sensor 1 ,

6 shows a schematic block diagram of an image capture device BEV according to an embodiment of the invention. The image capture device BEV comprises an image sensor 1 according to one of the embodiments described above. Next, the image capture device BEV an image processing device 4 on, which with the data evaluation device 3 of the image sensor 1 is coupled and is adapted to generate based on the first evaluation signal, a color image and process, to generate an infrared image based on the second evaluation signal and process, and to generate based on the third evaluation signal and a grayscale image. The image capture device BEV may preferably be a camera, wherein the image sensor 1 includes an imager chip of the camera. The image capture device BEV may be provided by the image sensor 1 be spatially separated and connected to the image sensor via a physical connection, such as an electrical connection, or wirelessly. The image capture device BEV may be one of the imager chip of the image sensor 1 various computing device, such as in the form of a microchip have. The image capture device BEV can be used together with the image sensor 1 be installed in a common housing, but can also be separated from this. In particular, only the image sensor can 1 Be part of the camera, while the image capture device BEV forms a separate unit of the camera.

In 7 FIG. 2 illustrates a schematic block diagram of a driver assistance system FAS for a vehicle. The driver assistance system FAS has an image acquisition device BEV, which in particular can correspond to the above embodiment. For example, a plurality of image sensors 1 be arranged on the vehicle, wherein the image sensors 1 are coupled to a central image acquisition device BEV of the driver assistance system FAS of the vehicle. The image processing device 4 the image acquisition device BEV is further designed to be based on the data evaluation device 3 The image acquisition device BEV signals provided to generate a color image and / or infrared image and / or gray scale image and extract therefrom features of objects and / or environmental properties with methods of machine vision. Preferably, the image processing device is 4 formed by the means of the very photosensitive pixels of the third set 23 pixelated grayscale image to recognize and classify features, such as shapes or outlines, of objects or environmental properties even in low light conditions, such as at night, in a vehicle environment of the vehicle, and from the infrared image, specific features of heat sources, such as humans extract and classify and classify them in a vehicle environment of the vehicle. The image processing device 4 may be configured to mark or particularly emphasize objects or environmental properties in the color image from the features extracted from at least one of the images, for example shapes and / or outlines. The thus processed color image is processed by the image processing device 4 output as a display image.

The driver assistance system FAS can also be a display device 5 which is adapted to that of the image processing device 4 output display image to a driver of the vehicle F display.

The driver assistance system optionally further comprises a control device 6 on, which is adapted to at least one driving function of the vehicle F on the basis of the image processing device 4 to control detected and classified objects and / or environment properties. So the control device 6 recognize, for example, based on the infrared image features of heat sources, which lead to a classification as a vehicle or person in the vicinity of the vehicle, and on the basis of this information, the control device 6 initiate an evasive maneuver or a braking maneuver of the vehicle F.

In 8th is a schematic block diagram of a vehicle F with a driver assistance system FAS illustrated according to one of the embodiments described above.

In 9 FIG. 3 illustrates a flowchart for explaining a method for evaluating electromagnetic radiation, wherein in a first step S1 the pixels of the first set 21 of pixels of one of the image sensors described above 1 electromagnetic radiation is detected and respective first sensor data are output.

In a second step S2, the electromagnetic radiation is transmitted through the pixels of the second set 22 detected by pixels and respective second sensor data are output.

In a further step S3, the pixels detect the third quantity 23 of pixels, the electromagnetic radiation and output respective third sensor data. The method steps S1 to S3 are preferably carried out simultaneously.

In a further step S4, the data evaluation device is provided 3 a first evaluation signal based on the first sensor data, a second evaluation signal based on the second sensor data and a third evaluation signal based on the third sensor data.

On the basis of the first evaluation signal, a color image and / or an infrared image based on the second evaluation signal and / or a gray scale image based on the third evaluation signal are preferably also generated or created.

Adjacent pixels of the first set 21 of pixels and / or the second set 22 of pixels and / or the third set 23 of pixels may preferably be linked to pixel clusters by means of pixel binning.

LIST OF REFERENCE NUMBERS

1

 image sensor

2

 pixel

21

 first set of pixels

21a

 red pixels

21b

 green pixels

21c

 blue pixels

22

 second set of pixels

23

 third set of pixels

3

 data evaluation

4

 Image processing means

5

 display

6

 control device

A

 pixel array

A1

 virtual first subarray

A2

 virtual second subarray

A3

 virtual third subarray

BEV

 Image capture device

FAS

 Driver assistance system

F

 vehicle

Ri

 red pixels

Gia, give

 green pixels

Bi

 blue pixels

IRi

 infrared pixels

Cia, Cib

 clear pixels

ci

 pixel clusters

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 3971065 A [0003]

Claims (15)

Image sensor ( 1 ) with pixels ( 2 ) for detecting electromagnetic radiation, wherein the pixels ( 2 ) include: a first quantity ( 21 ) of pixels adapted to detect at least a portion of a visible portion of the electromagnetic radiation and to output respective first sensor data; a second set ( 22 ) of pixels adapted to detect an infrared portion of the electromagnetic radiation and to output respective second sensor data; and a third set ( 23 ) of pixels adapted to detect the total visible portion of the electromagnetic radiation and to output respective third sensor data; the image sensor ( 1 ) a data evaluation device ( 3 ), which is designed to output a first evaluation signal based on the first sensor data, to output a second evaluation signal based on the second sensor data, and to output a third evaluation signal based on the third sensor data. Image sensor ( 1 ) according to claim 1, wherein adjacent pixels ( 2 ) of the first quantity ( 21 ) of pixels and / or the second set ( 22 ) of pixels and / or the third set ( 23 ) of pixels are linked by pixel binning to pixel clusters (Ci). Image sensor ( 1 ) according to claim 2, wherein the data evaluation device ( 3 ) is adapted to sensor data of pixels ( 2 ) of a pixel cluster (Ci). Image sensor ( 1 ) according to one of the preceding claims, wherein the first quantity ( 21 ) of pixels, the second set ( 22 ) of pixels and the third set ( 23 ) of pixels each distributed in a common pixel array (A) are arranged. Image sensor ( 1 ) according to claim 4, wherein the pixel array (A) comprises a plurality of blocks (B), each block (B) each comprising at least one pixel of the first set (B) 21 ) of pixels, the second set ( 22 ) of pixels and the third set ( 23 ) of pixels. Image sensor ( 1 ) according to claim 5, wherein each block (B) is in each case a pixel of the first set ( 21 ) of pixels, one pixel of the second set ( 22 ) of pixels and two pixels of the third set ( 23 ) of pixels, the two pixels of the third set ( 23 ) of pixels are linked by pixel binning to a pixel cluster. Image sensor ( 1 ) according to one of the preceding claims, wherein the third quantity ( 23 ) of pixels has an infrared cut-off filter which is designed to at least partially filter out the infrared component of the electromagnetic radiation. Image sensor ( 1 ) according to one of the preceding claims, wherein the second quantity ( 22 ) of pixels has a visible light cutoff filter which is designed to at least partially filter out the visible portion of the electromagnetic radiation. Image capture device (BEV), with an image sensor ( 1 ) according to any one of the preceding claims; and an image processing device ( 4 ), which with the data evaluation device ( 3 ) of the image sensor ( 1 ) and is configured to generate a color image based on the first evaluation signal and process it, to generate an infrared image based on the second evaluation signal and to process this, and to generate a greyscale image based on the third evaluation signal and to process this. Driver assistance system (FAS) for a vehicle (F), with an image capture device (BEV) according to claim 9; and wherein the image processing device ( 4 ) of the image acquisition device (BEV) is further adapted to be evaluated on the basis of the data evaluation device ( 3 ) of the image acquisition device (BEV) supplied to generate a color image and / or an infrared image and / or a gray scale and extract therefrom features of objects and / or environmental properties. Driver assistance system (FAS) according to claim 10, wherein the image processing device (FAS) ( 4 ) is further adapted to create a display image based on the color image and / or on the basis of the extracted objects and / or environmental properties; and wherein the driver assistance system (FAS) further comprises a display device ( 5 ) configured to display the display image to a driver of the vehicle. Driver assistance system (FAS) according to claim 10 or 11, with a control device (FAS). 6 ), which is adapted to at least one driving function of the vehicle (F) on the basis of the image processing device ( 4 ) to control extracted objects and / or environment properties. Vehicle (F) with a driver assistance system (FAS) according to one of Claims 10 to 12. Method for evaluating electromagnetic radiation by means of an image sensor ( 1 ) according to one of claims 1 to 8, comprising the steps of: detecting (S1) the electromagnetic radiation through the pixels of the first set ( 21 ) of pixels and outputting respective first sensor data; Detecting (S2) the electromagnetic radiation through the pixels of the second set ( 22 ) of pixels and outputting respective second sensor data; Detecting (S3) the electromagnetic radiation through the pixels of the third set ( 23 ) of pixels and outputting respective third sensor data; Output (S4), by the data evaluation device ( 3 ) of a first evaluation signal based on the first sensor data, a second evaluation signal based on the second sensor data and a third evaluation signal based on the third sensor data. The method of claim 14, wherein adjacent pixels of the first set ( 21 ) of pixels and / or the second set ( 22 ) of pixels and / or the third set ( 23 ) of pixels by pixel binning to pixel clusters.

Images