EP3437315A1 - Bildsensor für eine multifunktionale beobachtungskamera zum beobachten eines fahrers für ein fahrzeug, bildsensoreinrichtung, beobachtungskamera und verfahren zum erzeugen eines infrarot-bilds und eines intensität-bilds - Google Patents
Bildsensor für eine multifunktionale beobachtungskamera zum beobachten eines fahrers für ein fahrzeug, bildsensoreinrichtung, beobachtungskamera und verfahren zum erzeugen eines infrarot-bilds und eines intensität-bildsInfo
- Publication number
- EP3437315A1 EP3437315A1 EP17711586.2A EP17711586A EP3437315A1 EP 3437315 A1 EP3437315 A1 EP 3437315A1 EP 17711586 A EP17711586 A EP 17711586A EP 3437315 A1 EP3437315 A1 EP 3437315A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- image
- intensity
- infrared
- pixels
- image sensor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
- 238000000034 method Methods 0.000 title claims description 15
- 238000004590 computer program Methods 0.000 claims description 6
- 230000004313 glare Effects 0.000 claims description 5
- 238000005286 illumination Methods 0.000 claims description 4
- 238000013459 approach Methods 0.000 description 8
- 238000001514 detection method Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000003044 adaptive effect Effects 0.000 description 4
- 210000003128 head Anatomy 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000001914 filtration Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 210000004204 blood vessel Anatomy 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
- H04N23/11—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
Definitions
- An image sensor for a multi-functional observation camera for observing a driver for a vehicle image sensor device, observation camera, and method for generating an infrared image and an intensity image
- the approach is based on a device or a method according to the preamble of the independent claims. Subject of the present approach is also a computer program.
- CFA imagers ie image sensors with a special Bayer matrix, each pixel within one of many macro pixels, which typically have a 2x2 matrix grouping of pixels, is each provided with a color filter z.
- a red color filter for example, lets only red light pass through, etc.
- Driver surveillance camera systems typically with single or dual or multi-camera systems, consist of a camera module with the image sensor, an active near-infrared illumination unit, also called the I R module, and a determination device .
- a camera module with the image sensor
- an active near-infrared illumination unit also called the I R module
- a determination device e.g., a camera module with the image sensor
- the image sensor of the camera monochrome and the camera optics is provided with a special band-pass filter to sensitive only for their own infrared lighting and thus insensitive to stray light, z. As sunlight to be.
- an image sensor and an image sensor device further a method for generating an infrared image and an intensity image, further an observation camera, the this method is used, and finally a corresponding
- the advantages that can be achieved with the presented approach are that both an infrared image and an intensity image can be generated by the sensor unit.
- the provided intensity image can be
- the image sensor for a multi-functional observation camera for observing a driver of a vehicle is presented.
- the image sensor has a plurality of macro pixels, wherein a macro pixel has at least one infrared pixel.
- the image sensor such as a semiconductor-based image sensor, may include one of the plurality of macro-pixels.
- the macro-pixels can form a sensor surface of the image sensor and be arranged adjacent to one another like a matrix.
- Each of the macro pixels may comprise a field of pixels, also referred to as pixels.
- Each pixel may be configured to detect a light intensity.
- the intensity pixel can also be referred to as a grayscale pixel.
- An intensity pixel is designed to be one
- an intensity pixel may be formed to detect at least the light intensity of light in the visible range.
- the intensity pixels have no color filter.
- An infrared pixel is designed to detect only infrared light incident on the infrared pixel.
- the infrared pixels have a filter that allows only infrared light to pass.
- the presented approach enables the combination of infrared pixels and intensity pixels in an image sensor to generate both an infrared image and an intensity image, also called gray scale image.
- each of the macro pixels may have more infrared pixels than intensity pixels, for example, three infrared pixels and a single intensity pixel. In this way, a macro pixel can be made up of four equal-sized pixels that can form a square adjacent to one another. According to others
- each of the macro pixels may have a different number of infrared pixels and / or more than one intensity pixel.
- a macro pixel may have another suitable shape, such as a honeycomb shape.
- An image sensor device has the following features:
- One named image sensor and a determiner configured to determine an infrared image using infrared image signals of the infrared pixels and an intensity image using intensity image signals of the intensity pixels.
- the determination device of the presented image sensor device can be, for example, an unprocessed raw image, also called CFA image, which is a common image of both the infrared image signals and the
- the determination device is designed to generate the infrared image by a bilinear interpolation.
- bilinear interpolation positions associated with the intensity pixels in the infrared image are filled by interpolated pixels.
- the determination device can be designed to be used for
- the determining means may be further configured to provide the intensity image as a video image for video telephony.
- the video image for video telephony can be interesting and useful, in particular for highly automated mobile vehicles.
- the determination device can also be designed to provide a dazzling signal indicating dazzling of the driver using the intensity image.
- the glare signal can be determined using an image evaluation method by which, for example, a very high light intensity regions of the intensity image can be evaluated.
- the glare signal can for example be used to control an adaptive sun visor. A video-based detection of glare on the face is possible because there is no optical filtering of the sunlight in the intensity pixel.
- An observation camera for observing a driver has one of the presented image sensor devices and a near-infrared illumination unit for providing an infrared light for illuminating the driver.
- the presented observation camera can be used as a substitute for known
- Driver observation cameras are used, with the difference that the presented observation camera using only a single image sensor can advantageously generate both the infrared image and the intensity image.
- a method for generating an infrared image and an intensity image using one of the presented image sensor devices comprises the following steps:
- Observation camera be executable.
- a computer program product or computer program with program code which can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and for carrying out, implementing and / or controlling the steps of the method for generating an infrared image and an intensity image according to any one of the above
- Embodiments is used, in particular when the program product or program is executed on a computer or a device.
- FIG. 1 shows a schematic representation of an observation camera according to an embodiment
- FIG. 2 is a schematic plan view of an image sensor according to a
- FIG. 3 is a schematic representation of an infrared image according to an embodiment
- FIG. 4 is a schematic representation of an intensity image according to an exemplary embodiment
- FIG. 6 is a flowchart of a method for generating an infrared image and an intensity image according to an exemplary embodiment.
- the observation camera 100 may be arranged to observe a driver of a vehicle in a vehicle and has an image sensor device 105 and a proximity sensor.
- the near-infrared illumination unit 110 is configured to provide an infrared light for illuminating the driver
- the image sensor device 105 has an image sensor 112 and a determination device 115.
- the image sensor 112 has a plurality of macro pixels 120 with four pixels according to this exemplary embodiment.
- the image sensor 112 has three macro-pixels 120 whose four equally sized pixels are each arranged relative to one another in such a way that the macro-pixels 120 have a square shape.
- an image sensor 112 has at least several hundred of macro-pixels 120.
- each macro-pixel 120 has at least one infrared pixel I R and one intensity pixel I.
- Each of the infrared pixels IR is configured to have an intensity of incident on the infrared pixel I R
- Each of the intensity pixels I is configured to be one
- intensity-depicting intensity image signal 130 To detect light intensity of, for example, light in the visible region and provide an intensity-depicting intensity image signal 130.
- intensity-depicting intensity image signal 130 To detect light intensity of, for example, light in the visible region and provide an intensity-depicting intensity image signal 130.
- only one infrared image signal 125 and one intensity image signal 130 are shown for the sake of clarity, however, all other infrared pixels IR shown also generate one each
- Infrared image signal 135 and all other intensity pixels I shown also each generate an intensity image signal 130.
- the determining device 115 is configured to use the infrared image signals 125 of the infrared pixels IR, an infrared image 135 and under Using the intensity image signals 130 of the intensity pixels I to generate an intensity image 140 and to provide an interface of the camera 100 or a processing device integrated in the camera 100.
- the camera 100 is suitable according to an embodiment to previous
- SW functions or algorithms such as head and eye tracking, so a head and eye tracking, face identification, so a face recognition, Driver
- Modeling ie a driver modeling (fatigue, distraction detection) and gaze control, so a view control, are given.
- the advantage here is that the system costs remain low, since no second observation camera is required for the mentioned additional functions.
- the observation camera 100 thereby becomes multifunctional.
- Video telephony with a normal gray value image, ie the intensity image 140 advantageously corresponds to the human
- a video image can only be offered when the vehicle is at a standstill. That the driver moves the face or the head down during video telephony in order to see the video image and thus an unfavorable image perspective on the chin and nostrils of the driver is no longer present, can be easily understood.
- the video telephony can be made available to the driver in partially or highly automated mobile vehicles, since the driver can look down here to the observation camera 100, so that in this case a video image free of nostrils is possible. This typical look down on a display is also very common in smartphone
- FIG. 2 shows a schematic plan view of an image sensor 112 according to an exemplary embodiment. It may be the reference to FIG. 1
- the image sensor 112 shown here has a larger number of macro pixels 120.
- each of the exemplary 36 macro-pixels 120 has three infrared pixels I R and one intensity pixel I.
- FIG. 3 shows a schematic representation of an infrared image 135 according to a
- Embodiment This may be the case with reference to FIG. 1
- Embodiments are associated with the intensity pixels of FIG. 2
- the determination device described with reference to FIG. 1 is designed to carry out a bilinear interpolation which is necessary for this purpose. How the bilinear interpolation can be performed is described below with reference to FIG. 5.
- FIG. 4 shows a schematic representation of an intensity image 140 according to an exemplary embodiment. This may be the intensity image 140 described with reference to FIG. According to this embodiment, the intensity pixels I shown in FIG. 2 are brought together and shown without the infrared pixels. A resolution of the illustrated Intensity image 140 thus has only one quarter of the resolution of the infrared image shown with reference to FIG. 3.
- FIG. 5 shows a schematic representation of a bilinear interpolation 500, as can be carried out by the determination device described with reference to FIG. 1 for determining the infrared image.
- the procedure is described here by way of example with reference to an image sensor 502 of the type R3I, but can be applied correspondingly for an image sensor according to the approach described here.
- the image sensor 502 has a plurality of macro pixels 120 each having three pixels I i
- This image sensor 502 is configured to provide a plurality of driver assistance functions, such as a
- Lane detection for z Lane Keeping Support, to enable object recognition and traffic sign recognition.
- the red pixel R supports the video-based detection of red signs that z. For example, show a speed limit.
- a raw image 505, or CFA image is decomposed into a gray scale image, or intensity image 510, and a red image 515, respectively.
- the fourth missing pixel 520 in the macro pixel is supplemented by bilinear interpolation 500.
- the red image 515 has only a quarter of the resolution of the intensity image 510.
- the bilinear interpolation 500 is also called debayering and means that a colored raster image is reconstructed from the incomplete color values of an image sensor 502 superimposed with mosaic color filters.
- the bilinear interpolation 500 four adjacent pixels I of the missing pixel 520, gradient-based, or up to eight adjacent pixels I, gradient + orientation-based, for interpolation, in the x-y direction and diagonal, are used for this purpose.
- FIG. 6 shows a flow chart of a method 600 for generating an infrared image and an intensity image according to an exemplary embodiment.
- the infrared image and the intensity image are generated using one of the image sensor devices described with reference to the preceding figures.
- a step 605 of reading in the infrared image signals of the infrared pixels and the intensity image signals of the intensity pixels are read.
- a step 610 of generating the infrared image is acquired using the
- an exemplary embodiment comprises a "and / or" link between a first feature and a second feature, then this is to be read so that the embodiment according to one embodiment, both the first feature and the second feature and according to another embodiment either only first feature or only the second feature.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
- Closed-Circuit Television Systems (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE102016205095.8A DE102016205095A1 (de) | 2016-03-29 | 2016-03-29 | Bildsensor für eine multifunktionale Beobachtungskamera zum Beobachten eines Fahrers für ein Fahrzeug, Bildsensoreinrichtung, Beobachtungskamera und Verfahren zum Erzeugen eines Infrarot-Bilds und eines Intensität-Bilds |
PCT/EP2017/055400 WO2017167557A1 (de) | 2016-03-29 | 2017-03-08 | Bildsensor für eine multifunktionale beobachtungskamera zum beobachten eines fahrers für ein fahrzeug, bildsensoreinrichtung, beobachtungskamera und verfahren zum erzeugen eines infrarot-bilds und eines intensität-bilds |
Publications (1)
Publication Number | Publication Date |
---|---|
EP3437315A1 true EP3437315A1 (de) | 2019-02-06 |
Family
ID=58358554
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP17711586.2A Ceased EP3437315A1 (de) | 2016-03-29 | 2017-03-08 | Bildsensor für eine multifunktionale beobachtungskamera zum beobachten eines fahrers für ein fahrzeug, bildsensoreinrichtung, beobachtungskamera und verfahren zum erzeugen eines infrarot-bilds und eines intensität-bilds |
Country Status (4)
Country | Link |
---|---|
EP (1) | EP3437315A1 (de) |
CN (1) | CN108886587B (de) |
DE (1) | DE102016205095A1 (de) |
WO (1) | WO2017167557A1 (de) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE102019111914A1 (de) * | 2019-05-08 | 2020-11-12 | Deutsche Institute Für Textil- Und Faserforschung Denkendorf | Verfahren zur Auswertung einer Infrarotsignatur |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE10046309C1 (de) * | 2000-09-19 | 2001-10-31 | Siemens Ag | Sensoranordnung zur Bilderkennung |
DE10220825A1 (de) * | 2002-05-08 | 2003-07-03 | Audi Ag | Bildaufnahmevorrichtung für Fahrerassistenzsysteme |
CN1698381A (zh) * | 2003-05-08 | 2005-11-16 | 西门子公司 | 检测物体或人员的方法和装置 |
JP2007245911A (ja) * | 2006-03-15 | 2007-09-27 | Omron Corp | 監視装置および方法、記録媒体、並びに、プログラム |
JP6308760B2 (ja) * | 2012-12-20 | 2018-04-11 | キヤノン株式会社 | 光電変換装置および光電変換装置を有する撮像装置 |
CN104224204B (zh) * | 2013-12-24 | 2016-09-07 | 烟台通用照明有限公司 | 一种基于红外检测技术的驾驶员疲劳检测系统 |
-
2016
- 2016-03-29 DE DE102016205095.8A patent/DE102016205095A1/de active Pending
-
2017
- 2017-03-08 CN CN201780021802.2A patent/CN108886587B/zh active Active
- 2017-03-08 EP EP17711586.2A patent/EP3437315A1/de not_active Ceased
- 2017-03-08 WO PCT/EP2017/055400 patent/WO2017167557A1/de active Application Filing
Also Published As
Publication number | Publication date |
---|---|
CN108886587B (zh) | 2022-02-01 |
CN108886587A (zh) | 2018-11-23 |
DE102016205095A1 (de) | 2017-10-05 |
WO2017167557A1 (de) | 2017-10-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP1503580B1 (de) | Sensoranordnung mit einer Mehrzahl von Typen optischer Sensoren | |
DE10046309C1 (de) | Sensoranordnung zur Bilderkennung | |
DE112011103083T5 (de) | Umgebungsbild-Anzeigesystem und Umgebungsbild-Anzeigeverfahren für ein Fahrzeug | |
EP1665815A2 (de) | Verfahren zum erstellen einer raumbildvorlage für abbildungsverfahren mit rumlichen tiefenwirkungen und vorrichtung zum anzeigen einer raumbildvorlage | |
DE102011106072A1 (de) | Schattenentfernung in einem durch eine fahrzeugbasierte kamera erfassten bild unter verwendung einer optimierten ausgerichteten linearen achse | |
DE102015217253A1 (de) | Umfelderfassungseinrichtung für ein Fahrzeug und Verfahren zum Erfassen eines Bilds mittels einer Umfelderfassungseinrichtung | |
DE10203421C1 (de) | Automobiles Infrarot-Nachtsichtgerät und automobiles Display | |
DE102014205519A1 (de) | Verfahren und Vorrichtung zum Anpassen einer Anzeige eines autostereoskopischen Displays für ein Fahrzeug | |
DE102009001122B4 (de) | 1 - 15Kameraanordnung und Verfahren zum Ermitteln von Bildsignalen mit Farbwerten | |
DE102009014437B4 (de) | Objekterkennungssystem und -verfahren | |
DE102007034608A1 (de) | Kamera | |
DE102017011562A1 (de) | Verfahren zur Umwandlung eines Rohbilds in ein Farbbild | |
DE10220825A1 (de) | Bildaufnahmevorrichtung für Fahrerassistenzsysteme | |
EP3437315A1 (de) | Bildsensor für eine multifunktionale beobachtungskamera zum beobachten eines fahrers für ein fahrzeug, bildsensoreinrichtung, beobachtungskamera und verfahren zum erzeugen eines infrarot-bilds und eines intensität-bilds | |
WO2014063701A1 (de) | Sensoranordnung zur bilderfassung | |
EP3073446B1 (de) | Verfahren zum darstellen einer fahrzeugumgebung eines fahrzeuges | |
DE102004033625A1 (de) | Verfahren zur Darstellung des insbesonderen vorderen Sichtfeldes aus einem Kraftfahrzeug | |
DE102015206796A1 (de) | Sichtfeldanzeigevorrichtung für ein Fahrzeug und Verfahren zum Darstellen eines Bildes in einem Sichtbereich einer Person | |
WO2015074915A1 (de) | Filteranordnung und verfahren zum herstellen einer filteranordnung | |
EP3089105A1 (de) | Verfahren zum generieren eines kontrastbildes einer objektbeschaffenheit und diesbezügliche vorrichtungen | |
WO2015090997A1 (de) | Optisches sensorsystem | |
DE102019123740B4 (de) | Verfahren zum Bereitstellen einer Rück- und/oder Seitenansicht eines Fahrzeugs, virtuelleSpiegelvorrichtung und Fahrzeug | |
DE102020215419A1 (de) | Bildsensor für eine Kamera und Verfahren zum Erzeugen eines Kamerabilds | |
DE102005041241A1 (de) | Verfahren und System zum Betreiben eines Kamerasystems eines Fahrzeugs | |
EP2819408A2 (de) | Verfahren und Vorrichtung zum Anpassen einer Darstellung eines Bildes auf einem Bildschirm |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20181029 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: ROBERT BOSCH GMBH |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20210125 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R003 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED |
|
18R | Application refused |
Effective date: 20221230 |