EP4627801A2

EP4627801A2 - A 360-degrees motor vehicle monitoring system

Info

Publication number: EP4627801A2
Application number: EP23806182.4A
Authority: EP
Inventors: Weiquan CONG
Original assignee: Continental Automotive Technologies GmbH
Current assignee: Aumovio Germany GmbH
Priority date: 2022-11-28
Filing date: 2023-11-09
Publication date: 2025-10-08
Also published as: WO2024115067A2; GB2624689A; GB202217773D0; WO2024115067A3

Abstract

A 360-degrees motor vehicle monitoring system is disclosed herein. The motor vehicle monitoring system comprises a camera assembly. The camera assembly comprises a fisheye lens. The camera assembly further comprises an image sensor, more specifically a RGB-IR sensor, operable to generate video images in RGB spectrum and video images in IR spectrum. The camera assembly further comprises an illumination source operable to illuminate in near infrared (NIR) spectrum. In particular, the camera assembly has a field of view operable to monitor at least an area of a passenger cabin and at least partially outside a radius of a motor vehicle.

Description

A 360-DEGREES MOTOR VEHICLE MONITORING SYSTEM

TECHNICAL FIELD

This disclosure relates to a motor vehicle monitoring system, and more in particular a 360 degrees onboard monitoring system.

BACKGROUND

Increasingly, the automotive industry is using camera systems to either monitor a surrounding of the vehicle, or for in-vehicle monitoring purposes. Conventional vehicle monitoring systems either surveillance within a passenger compartment for driver or cabin monitoring purposes or surveillance outside the motor vehicle. Some examples include dash cameras and surround-view monitoring.

However, separate camera solutions are required for a single vehicle to have both interior and exterior surveillance purposes. Having multiple monitoring systems increases the difficulty of fusion of images for different types of automotive applications.

The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

SUMMARY

A purpose of this disclosure is to ameliorate the problem of multiple monitoring systems in a motor vehicle, by providing the subject-matter of the independent claims. Further purposes of this disclosure are set out in the accompanying dependent claims.

In an aspect of this disclosure, a 360-degrees image sensor system for monitoring at least an area of a passenger cabin and simultaneously at least partially an area outside the passenger cabin is provided. The image sensor monitoring system may comprise a camera assembly. The camera assembly may comprise a fisheye lens. The camera assembly may further comprise a RGB-IR sensor. An RGB-IR image sensor is a sensor which is operable to generate video images from electromagnetic waves in the RGB spectrum and IR spectrum. Such sensors are known in the prior art. The camera assembly may further comprise an illumination source operable to illuminate in near infrared (NIR) spectrum. An advantage of the above-described aspect of this disclosure yields a single 360-degree camera assembly operable to monitor both an interior of a motor vehicle and simultaneously at least partially an area outside the passenger cabin of the motor vehicle. The motor vehicle monitoring system disclosed herein eliminates the need for additional dash camera and/or surround-view monitoring system, thereby achieving reduction of number of monitoring systems in a motor vehicle. The use of an illumination light source operable in a NIR spectrum provide unobtrusive, remote sensing of a person’s presence, pupil position and gaze, eyelid movement, and other indicators of driver alertness and occupant safety, without distracting the driver.

The system may further comprise a processing unit operable to convert fisheye projection images into rectilinear projection video images. The system may further comprise a processing unit operable to perform segmentation of the video images. Segmentation of video images is a process that saves videos at a pre-determined set of lengths, for example in sets of 3 minutes, 5 minutes, 10 minutes, etc, instead of one continues video clip. The advantage of performing segmentation of video images reduces data and computational power. The system may further comprise a processing unit operable to execute processing steps for auxiliary applications or combination thereof. Advantageously, vehicle monitoring system is improved through sensing and data fusion using a single monitoring system for multiple automotive applications. The system may further comprise an inertial measurement unit operable to measure a tilting angle of the integrated camera assembly.

The system may further comprise a display device operable to display video images. The camera assembly may be integrated with display device as a single monitoring system. Advantageously, the camera assembly may be incorporated into a display device of a single integrated monitoring system, to solve issue of space constrains. The display device may be used to display image videos and information for other vehicular applications, for example displaying a field of view outside of the motor vehicle.

The camera assembly may be mounted on a windshield of a motor vehicle. The camera assembly may be mounted on at least a side of a rearview mirror of a motor vehicle.

Other objects, features and characteristics, as well as the methods of operation and the functions of the related elements of the structure, the combination of parts and economics of manufacture will become more apparent upon consideration of the following detailed description and appended claims with reference to the accompanying drawings, all of which form a part of this specification. It should be understood that the detailed description and specific examples, while indicating the non-limiting embodiments of the disclosure, are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.

BRIEF DESCRIPTION OF DRAWINGS

The present disclosure will become more fully understood from the detailed description and the accompanying drawings, wherein:

FIG. 1A shows an exemplary aspect of a 360-degree motor vehicle monitoring system implemented on a motor vehicle. FIG. 1 B shows an exemplary aspect of a 360-degree motor vehicle monitoring system implemented on a motor vehicle.

FIG. 2A shows an exemplary integrated 360-degree motor vehicle monitoring system.

FIG. 2B shows an exemplary 360-degree motor vehicle monitoring system.

DETAILED DESCRIPTION

It should be understood that like reference numerals identify corresponding or similar elements throughout the several drawings. It should be understood that although a particular component arrangement is disclosed and illustrated in these exemplary embodiments, other arrangements could also benefit from the teachings of this disclosure.

FIG. 1A and FIG. 1 B shows a 360-degree image sensor system 100a, 100bin accordance with a preferred aspect of this disclosure. The system 100a shown in FIG. 1A includes a camera assembly 102, which is mounted on at least one side of a rearview mirror 104 of a motor vehicle 120. In a typical set up within the rearview mirror is mounted on a windshield 108. The camera assembly 102 includes a fisheye lens (FIG. 2A and FIG. 2B referred) having a field of view (FOV) of more than 180° with an optical axis direction towards at least an area of the passenger cabin 122 and simultaneously at least partially outside a radius of a motor vehicle 120. The fisheye lens has a FOV of 120° or more, preferable 170° or more, more preferable 180° to 360°, even more preferable 190° to 270°, with a suitable image sensor, such as a RGB-IR image sensor, the camera assembly 102 will be able to achieve 360° detection, to double up as a surround-view camera to monitoring surrounding of the motor vehicle 120 and/or a dash camera (dashcam) for surveillance of the motor vehicle 120. An advantage of using a RGB-IR sensor allows a single image sensor system to capture a single image and process the single image captured to produce two separate images, one video image in a RGB spectrum and one video image in an IR spectrum. More advantageously, producing images in IR spectrum allows the monitoring system to identify thermal areas in the video images, thus distinguishing between living beings seated within the passenger cabin 122 and objects, for example chair

FIG. 1 B shows an exemplary aspect of a 360-degree motor vehicle monitoring system 100b implemented on a motor vehicle 120. In this configuration, the camera assembly 102 is mounted on a windshield 108. Similar to the system 100a shown in FIG. 1A, the camera assembly 102 includes a fisheye lens (FIG. 2A and FIG. 2B referred) having a field of view a field of view towards at least an area of the passenger cabin and at least partially outside a radius of a motor vehicle 120. Preferably, the fisheye lens has a field of view of 120° or more, preferable 170° or more, more preferable 180° to 360°, even more preferable 190° to 270°.

As shown in FIG. 1A and FIG. 1 B, the camera assembly 102 is operable to capture images for different zones. This may be achieved by differentiating field of illumination for different zones. Preferably, the camera assembly 102 has a working wavelength range including visible light and near infrared light. This working wavelength range may be between 380nm to 1000 nm. Referring back to FIG. 1A and FIG. 1 B, the zones include a dashcam zone 110 operable to surveillance a front view of the motor vehicle 120, an exterior monitoring zone 112 operable to function as a surround-view monitoring camera relative to a motor vehicle 120, a driver monitoring zone 114 operable to monitor a states of the driver within a motor vehicle 120 and a cabin monitoring zone 114 operable to monitor a passenger cabin 122 of a motor vehicle 120.

Thus, it can be seen, a main advantage of the aforesaid configuration yields a single 360-degree camera assembly operable to monitor both an interior of a motor vehicle and an exterior of the motor vehicle. The motor vehicle monitoring system disclosed herein eliminates the need for additional dash camera and/or surround-view monitoring system, thereby achieving reduction of number of monitoring systems in a motor vehicle. Therefore, an image sensor system proposed herein allows monitoring a vehicular environment with less power consumption. The camera assembly 102 may further include a processing unit 106 (FIG. 2A, FIG. 2B referred). The processing unit 106 may be operable to convert fisheye projection images into rectilinear projection video images. The processing unit 106 may be operable to perform segmentation of the video images. Further, the processing unit 106 may be operable to execute processing steps for auxiliary applications. An advantage of using processing unit 106 to execute such functions improves the system 100a through sensing and data fusion using a single monitoring system for multiple automotive applications. Optionally an electronic control unit 214 may be embedded in the rearview mirror 108. Applicable auxiliary applications may include video images captured by the camera assembly 102 in the dashcam zone 110 and/or the exterior monitoring zone 112 in the format of converted rectilinear projection video images captured by the camera assembly 102.

FIG. 2A shows an exemplary integrated 360-degree motor vehicle monitoring system 200a in accordance with a preferred aspect of this disclosure. The system 200a shown here incorporate the camera assembly 102 with a display device 202, connected by a display connection 204. An advantage of integrating the camera assembly within a display device 202 has the advantage of a field of view of the entire passenger cabin. The display connection 204 may be a hardware element, for example physical wire connections, or software element, for example transmission of data information between different hardware modules. The camera assembly includes a fisheye lens 212 and an image sensor 210. A suitable image sensor 210 will be RGB-IR sensor, operable to generate video images in RGB spectrum and video images in IR spectrum. The camera assembly 102 further includes an illumination source 206 operable to illuminate in a near infrared (NIR) spectrum. The selection of illumination in a NIR spectrum allows the 360-degrees motor vehicle monitoring system to switch between driver monitoring systems application and/or cabin monitoring systems application by controlling the field of illumination, to differentiate the driver monitoring zone 114 and the cabin monitoring zone 116. Suitable illumination source may be vertical cavity surface emitting laser (VCSEL) or light emitting diode (LED). The illumination source 206 may be driven by an illumination driver 208. Optionally, the illumination source 206 may include an illumination lens to adjust the field of illumination and illumination profile, which provides a map of the intensity of light across the entire field of view of the lens.

FIG. 2B shows an exemplary 360-degree motor vehicle monitoring system 200b in accordance with a preferred aspect of this disclosure. In this system 200n, an electronic control unit (ECU) 214 is connected to the camera assembly 102 and display device 202. Communication between ECU 214 uses Serializer/Deserializer high-speed communications. Serializer/Deserializer is a pair of functional blocks used to convert data between serial data and parallel interfaces in each direction.

Optionally, the 360-degrees motor vehicle monitoring system may include an inertial measurement unit (IMU) operable to measure the tilting angle of the camera assembly 102 for measurement of potential compensation value.

Optionally, the 360-degrees motor vehicle monitoring system may further include a read-only-memory (ROM) for image sensor 210 and lens calibration data storage. An example of data to be stored may be lens parameters.

Optionally, the camera assembly may include an optical filter as cover lens. The visible light neural density filter and long pass filter (LPF) may be operable to compensate mismatch between active NIR illumination and ambient illumination. It reduces the visibility to inner side of the cover, to have better appearance.

The foregoing description shall be interpreted as illustrative and not be limited thereto. One of ordinary skill in the art would understand that certain modifications may come within the scope of this disclosure. Although the different non-limiting embodiments are illustrated as having specific components or steps, the embodiments of this disclosure are not limited to those combinations. Some of the components or features from any of the non-limiting embodiments may be used in combination with features or components from any of the other non-limiting embodiments. For these reasons, the appended claims should be studied to determine the true scope and content of this disclosure. List of Reference Signs

Claims

Patent claims

1 . A 360-degrees image sensor system (100a, 100b) for monitoring at least an area of a passenger cabin (122) and simultaneously at least partially an area outside the passenger cabin (122), comprising: a camera assembly (102) comprising: a fisheye lens (212); a RGB-IR image sensor (210); and an illumination source (206) operable to illuminate in near infrared (NIR) spectrum.

2. The system (100a, 100b) of claim 1 , wherein the system (100a, 100b) further comprises a processing unit (106) operable to: convert fisheye projection images into rectilinear projection video images; perform segmentation of the video images execute processing steps for auxiliary applications or combination thereof.

3. The system (100a, 100b) of claims 1 - 2, wherein the system (100a, 100b) further comprises an inertial measurement unit operable to measure a tilting angle of the integrated camera assembly.

4. The system (100a, 100b) of claims 1 -3, wherein the system (100a, 100b) further comprises a display device (202) operable to display video images.

5. The system (100a, 100b) of claims 1 - 4, wherein the camera assembly (102) is operable to monitor one or more occupants within the passenger cabin (122) in response to a field of illumination.

6. The system (100a, 100b) according to any one of the preceding claims, wherein the camera assembly (102) is integrated with the display device (202) as a single monitoring system.

7. The system (100a, 100b) according to any one of the preceding claims, wherein the fisheye lens (212) has a field of view of 120° or more, preferable 170° or more, more preferable 180° to 360°, even more preferable 190° to 270°.

8. The system (100a, 100b) according to any one of the preceding claims, wherein the auxiliary applications comprises: dashcam zone (110); and/or exterior monitoring zone (112).

9. A vehicle (120) comprising a passenger cabin (122) and a system (100a, 100b) according to any one of the preceding claims, wherein the camera assembly (102) has a field of view of at least an area of a passenger cabin (122); and at least partially outside the vehicle (120).

10. The vehicle (120) according to the preceding claim 8, wherein the camera assembly (102) is mounted on a windshield of the vehicle (120).

11. The vehicle (120) according to claim 8, wherein the camera assembly (102) is mounted on at least a side of rearview mirror of a vehicle (120).

12. A method for 360-degrees monitoring for monitoring at least an area of a passenger cabin (122) and simultaneously at least partially an area outside the passenger cabin (122), comprising the steps of: a. capturing images with the system (100a, 100b) according to any one of the preceding claims 1-7 while the system (100a, 100b) is mounted in a passenger cabin (122) of the vehicle; and b. detect objects inside the passenger cabin (122) and/or outside the vehicle (120).