EP3504871A1

EP3504871A1 - Method and device for operating an interior camera

Info

Publication number: EP3504871A1
Application number: EP17757472.0A
Authority: EP
Inventors: Hoang Trinh; Gabor GYURASZ
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2016-08-23
Filing date: 2017-08-03
Publication date: 2019-07-03
Also published as: CN109565549A; DE102016215766A1; WO2018036784A1; CN109565549B

Abstract

The invention relates to a method for operating an interior camera (104) for a vehicle (100), wherein in one control step at least one camera parameter (122) of the interior camera (104) is controlled by using at least one quality parameter (118) of a previously recorded image from the interior camera (104) when a head (110) of a target person (108) is detected in the image, wherein the camera parameter (122) is set to a predefined value when no head (110) is detected.

Description

Description title

Method and device for operating an interior camera Prior art

The invention is based on a device or a method according to the preamble of the independent claims. The subject of the present invention is also a computer program.

An indoor camera captures images in the near infrared range. In order to obtain sufficiently illuminated images, the interior camera has an infrared illumination device. The closer an object is to the

Lighting device is, the greater is an illumination intensity on the object.

For example, a head of a driver of the vehicle at a

Forward displacement have a very short distance to the lighting device, so that the interior camera can no longer detect the head.

Disclosure of the invention

Against this background, with the approach presented here, a method for operating an interior camera for a vehicle, a device that uses this method, and finally a

corresponding computer program presented according to the main claims. The measures listed in the dependent claims advantageous refinements and improvements of the independent claim device are possible. If an object can no longer be detected by a camera, a control of the camera attempts to change camera parameters in order to return to an acquisition-ready state. In the approach presented here at least one camera parameter is the

Set the camera to a reference level immediately after the object has been lost, causing the camera to be high under normal conditions

Has detection probability. It is expected that the designated here as an object head of a driver of a vehicle with a very high probability after a short time is again arranged in an expected position. The reference value is matched to the expected position. When the head is detected again, the camera parameter is tracked according to the detected position of the head.

A method of operating an interior camera for a vehicle is provided, wherein in a step of controlling at least one camera parameter of the indoor camera is controlled using at least one quality parameter of a previously acquired image of the indoor camera when a head of a target person is detected in the image of the

Camera parameter is set to a predefined value if no head is detected.

An interior camera can be understood as a camera directed into an interior of a vehicle. The interior camera can

be aimed in particular at a driver of the vehicle. The

Indoor camera can provide an image sequence of individual images of the interior. The indoor camera can also provide a video signal. A camera parameter can be an adjustable size. The driver can be the target person. A predefined value can be a default value for the

Be camera parameters.

As a quality parameter, a contrast, a brightness and / or a

Brightness distribution of the image can be used. As the camera parameters, an exposure time and / or sensitivity of the indoor camera can be regulated. Alternatively or additionally, the camera parameter can be a Be regulated light intensity of a lighting device of the interior camera. By adjusting these camera parameters, the image can be of high quality.

The method may include a step of detecting the head in a subsequently captured image of the indoor camera. Of the

Quality parameter can be related to a header of the image. The header is the region of interest of the image. By referring the quality parameters to the header area, the header area can be displayed very well.

The method may include a step of adjusting in which a color depth of a raw image signal of the indoor camera is adjusted to obtain a working image signal. The head can be detected in an image of the working image signal. Reduced color depth requires less computation to process the image.

A color depth spectrum can be extracted from the raw image signal to obtain the working image signal. Alternatively or additionally, raw color levels of the raw image signal may be added using a processing protocol

Working color levels of the working image signal are assigned to the

Obtain working image signal. The color depth spectrum can by a

Cutting off irrelevant color levels of the image can be achieved. The

Processing instructions can be an algorithm for converting the color levels. By converting a large information content of the image can be obtained. By extracting, the color depth can be reduced quickly and easily.

This method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit.

The approach presented here also provides a device which is designed to implement the steps of a variant of a method presented here

to implement, control or implement appropriate facilities. Also by this embodiment of the invention in the form of a device, the object underlying the invention can be solved quickly and efficiently.

For this purpose, the device may comprise at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to a sensor or an actuator for reading sensor signals from the sensor or for outputting data or control signals to the sensor Actuator and / or at least one

Communication interface for reading or outputting data embedded in a communication protocol. The arithmetic unit may be, for example, a signal processor, a microcontroller or the like, wherein the memory unit may be a flash memory, an EEPROM or a magnetic memory unit. The communication interface can be designed to read or output data wirelessly and / or by line, wherein a communication interface that can read or output line-bound data, for example, electrically or optically read this data from a corresponding data transmission line or output in a corresponding data transmission line.

In the present case, a device can be understood as meaning an electrical device which processes sensor signals and outputs control and / or data signals in dependence thereon. The device may have an interface, which may be formed in hardware and / or software. In the case of a hardware-based embodiment, the interfaces can be part of a so-called system ASIC, for example, which contains a wide variety of functions of the device. However, it is also possible that the interfaces are their own integrated circuits or at least partially consist of discrete components. In a software training, the interfaces may be software modules that are present, for example, on a microcontroller in addition to other software modules.

Also of advantage is a computer program product or computer program with program code which is stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical disk Memory may be stored and for carrying out, implementing and / or controlling the steps of the method according to one of the above

described embodiments, in particular when the program product or program is executed on a computer or a device.

Embodiments of the approach presented here are shown in the drawings and explained in more detail in the following description. 1 is a block diagram of a vehicle having an apparatus for operating an interior camera according to an embodiment;

FIG. 2 is an illustration of a procedure of operating an indoor camera according to an embodiment; FIG.

3 is a flowchart of a method for operating a

Interior camera according to an embodiment;

4 is a state diagram of a controller for a method of operating an indoor camera according to an embodiment;

5 shows representations of a detection of an object in an image of an interior camera according to an exemplary embodiment; 6 is a flowchart of an algorithm for operating a

Interior camera according to an embodiment;

7 is a block diagram of a controlled system for operating a

Interior camera according to an embodiment; Figures 8 to 11 representations of applications for a

Interior camera according to an embodiment; and

Fig. 12 is a flow chart of a method of operation

Interior camera according to an embodiment. In the following description of favorable embodiments of the present invention are for the in the various figures

represented and similar elements acting the same or similar

Reference numeral used, wherein a repeated description of these elements is omitted.

1 shows a block diagram of a vehicle 100 with a device 102 for operating an interior camera 104 according to one exemplary embodiment. The interior camera 104 is aligned with an expected head area 106 of a driver 108 of the vehicle 100. Here is the head 110 in

Essentially arranged centrally or in the middle range in the head region 106 at a normal position. The interior camera 104 has a lighting device 112. The illumination device 112 comprises at least one infrared light source 114, 116, which is aligned in the head region 106. The first infrared light source 114 is close to

Interior camera 104 is arranged. Light from the first light source 114 is reflected on the retina of the driver's eyes to the interior camera 104 and, similar to the red eye effect, produces bright pupils in images of the camera

Interior camera 104. The second infrared light source 116 is disposed away from the indoor camera 104. Light from the second light source 116 is not reflected via the retina to the indoor camera 104 and produces dark pupils in the images.

When the head 110 is within the head area 106, it may be detected in images of the indoor camera 104. Depending on a position of the head 110 in the header 106, the image has quality parameters 118. When the head 110 is detected, at least one camera parameter 122 of the indoor camera 104 is readjusted in a controller 120 of the apparatus 102 for operation using the quality parameters 118.

In other words, FIG. 1 shows a device 102 for controlling a motor vehicle interior camera 104 with active I R illumination 112.

Driver surveillance camera systems consist of at least one

Camera module 104, an active near-infrared illumination or I R Modules 112 and a computer unit 102. A

Driver surveillance camera system may typically be a one-way, two-way multi-camera system. The I R modules 112 can in

Essentially in Bright Pupil (BP) 114 and Dark Pupil (DP) 116 can be distinguished. Bright pupil (BP) 114 produces bright pupils similar to the "red eye effect" when the illumination 114 is located very close to the camera 104, whereby the light which has hit the retina through the pupil is reflected back into the camera image. DP) 116 occurs when the

Lighting 116 away from the camera 104 is located and thus no emitted light enters directly on the retina or pupil and therefore the pupil in the camera image remains dark.

Using the interior camera 104, software functions or algorithms such as head and eye tracking, face identification, driver modeling (drowsiness, distraction detection) and gaze control can be implemented. The active I R illumination 112 provides a good illumination of the scene or the driver's face. This can be done at all

Conditions such as day and night, in the face area a high-contrast and bright image for the video-based detection performance can be ensured.

The integration times of the imager and the I R illumination can be changed. The I R illumination 112 may be in terms of illumination and

Irradiance can be adjusted for changing distances. Thus, many use cases are covered where the driver or the user is at different distances to the head or eye tracking system.

Without the variability, the captured camera image would be very overexposed and thus unsuitable for tracking, for example, when the driver bends forward or has his head very close to the interior camera 104 during a boarding phase. A camera control, for example by means of the built-in imager auto exposure control (AEC) or a dedicated camera SW module leads to a brightness control on the full picture or a set image area. In the picture area can not between the face and objects, like the sun behind the face, one Obscurity, for example, be distinguished in front of the face with one hand. This causes the face in the controlled image by means of

Image processing methods can no longer be found because the contrast and brightness is not sufficient.

In the approach presented here, an operating point designed for typical situations with a specific integration time or time is used

Duration of illumination determined. At the operating point, the camera 104 and the I R illumination 112 provide an optimal image for the head tracking. Depending on the imager, the image has a quantization of 10 or 12 bits and in the first step becomes, for example, a 10/12 -> 8 bit mapping by means of a function such as a logarithmic characteristic or by simple bit selection or cutting of 8 bits from 10 / 12 bit performed. The raw image is thus reduced to 8-bit resolution and fed to the head tracking algorithm. Once a head 110 is detected, tracking data, such as a 2D head bounding box, a tracking confidence, a face orientation, is detected

Face landmarks and the current state of the tracking state machine supplied to the camera control software module to an adapted

Image control based on the image quality parameters, such as contrast, brightness, and brightness distribution perform. If, however, no face is detected, the system jumps back to the defined operating point. The transition may be made by temporally configuring with a defined period of time, instead of a hard transition to the aforementioned operating point. It is in the use cases mentioned, such as sun behind the head and

Obscurations avoided unfavorable regulation on objects. Even in the case of an approach of the head with previously detected face to the camera 104, adapted to the situation lighting and integration of the image ensure an optimal for the head tracking image.

FIG. 2 shows an illustration of an effect chain of a camera control of an interior camera 104 according to one exemplary embodiment. Starting from the interior camera 104, the chain of effects comprises a frame buffer 200, a preprocessing means 202, a head tracking means 204, an eye tracking means 206, and higher level functions 208. The indoor camera 104 provides raw image data 210 at 12 or 10 bits per pixel for the image buffer 200. The raw image data 210 is in the

Preprocessor 202 by mapping or by a non-linear function, such as log or simply dropping bits, for example, the twelfth and first bits or the two lowest first and second bits of 12 to 8-bit image data 212 reduced. The 8-bit image data 212 is used in the tracker 204 to

To obtain header data 214. The head tracking data 214 is used in the controller 120 in accordance with the approach presented herein to adjust camera and IR exposure control to head tracking data 214. Image quality can be improved for head tracking. The head tracking data 214 is evaluated to determine parameter 122 for controlling the camera / 1 R. In particular, a 2D head boundary box is used as the region of interest (ROI) for calculating image quality parameters 118. A degree of

Quality / Confidence of the head tracking can be considered.

For example, the head tracking data 214 and the 2D header box, respectively, may be used if a

Acceptance level / acceptance threshold is exceeded.

A confidence may be used to set a region of interest for calculating image quality parameters 118. The size of the region of interest is limited in minimum and maximum size to avoid too small or too large a region of interest.

Upon rotation / alignment of the head, the camera / 1R parameters 122 may be frozen if the face is not frontal.

The visibility of features of the head / face, ie whether the entire face is visible or at least both eyes and nose or corner of the mouth are visible. Dealing with possible occlusions caused by proximity to / far from the camera. A head tracking status may be initialized, tracked / tracked, or retrieved / reinitiated when the head tracking is in the tracked mode. In other cases, init or refind, the camera / 1R control will not be changed.

A device exposure and I R illumination time of about 500 microseconds at 5 A and 60 fps. A minimum exposure time of 40 microseconds and a maximum exposure time of three milliseconds for control range.

Possible default settings are a bit shift or bit shift substantially greater than seven, a predetermined gain of one, a predetermined analog gain of one. Control parameters, for example (PID controller). Timing thresholds especially for transition between face captured and not captured.

As a non-linear mapping function, for example, a log function for mapping the 12-bit or 10-bit image to an 8-bit image may be applied.

In the control strategy presented here for addressing different applications, starting from a well-defined operating point 216, the image recording and exposure are adjusted by moving from the operating point 216 to a good image quality achieved, which is suitable for tracking the head and eyes.

FIG. 3 shows a flow chart of a method for operating a

Interior camera according to an embodiment. The method can be carried out on a device for operating, as shown for example in FIG. 1. The flowchart includes a memory block 300, a first functional block 302, a first decision block 304, a second functional block 306, a second decision block 308, and a third functional block 310. In the third functional block 310 is a third

Decision block 312, a fourth function block 314, a fifth

Function block 316, a fourth decision block 318, a sixth

Function block 320 and a seventh function block 322 included. In this case, a start takes place with a default setting or in an operating point from the memory block 300, as shown in Fig. 2. A setting control starts when the face is detected or with an input signal of the Head tracking. The tracking quality / confidence level, tracking refresh rate, estimated distance from the face bounding box in a 2 D image plane may be used to set the region of interest of the image. This region of interest is used to calculate the image quality parameters and to make a review

configured image quality parameter thresholds. These

In the simple case, the region of interest may be a face area within the image displayed by the detected face boundary box.

In order to avoid covert scene control when face capture is no longer feasible, facial features such as visibility, symmetry, or head tracking obscurities are evaluated.

For example, if the steering wheel or a hand is in the picture, and the adaptation of image quality, such as brightness, would be done on the non-face.

The control is only done with (near) frontal view of the face.

Brightness symmetry and / or face rotation or alignment are checked. Control parameters on header profiles are kept. Control occurs only when Head Tracking is in tracking mode. In this mode, only facial features or landmarks such as canthus and nostrils are tracked by applying, for example, a Kalman filter. In the initialization mode, the search is performed over a full frame for head / face candidates. In the recovery mode, try the

Head tracking to find or capture a head within a larger image region than in the tracking mode.

If no face is detected or the conditions are not fulfilled, the default operating point for camera / 1 R control is used.

The control is carried out in two stages. For the exposure, a new exposure time is used using the exposure time +/-

Shutter speed_Step size determined. The step size is used to get a smooth dynamic brightness change to avoid changing a brightness level between frames. The

Control speed is adjusted to head movement toward the camera. A six, seven or eight bit shift bit shift can be used as an optional stage. If the threshold to a good image parameter with a matched image is not met, another bit shift operation may be performed one to two bits left or right. The main control parameter is the

Exposure time. The level and range, span or range can be tested iteratively.

Minimum and maximum range of parameters. A minimum and maximum exposure time is limited, for example between 40 microseconds and three milliseconds. The values are based on heuristics to allow adjustment close to the given operating point. The bit shift takes place with a maximum of 2 bits left and right.

The calculation of img_qpar or image quality parameters such as brightness and contrast and the comparison of actual image parameters with target image quality parameters qpar_thr, ideally equal to an average value, such as image brightness about 128 LSB with 8-bit image.

For example, an exposure time for the next frame may be set above 12C to obtain a result closer to a good image parameter based on an analysis of the current frame. For example, if it is too dark, the exposure time may be increased; if it is too bright, the exposure time may be reduced. The range or range and step or stages can be evaluated heuristically.

Standard control, for example via a PID controller, can be used. The reference variable w is the image quality parameter, the feedback x is the control parameter, the control deviation e is the difference for the controller.

The head tracking quality and the estimated distance corresponding to the detected face bounding box in the 2-D image plane may be used to set a region of interest to the control. The interested parties Region (ROI) is relevant for calculating the image quality parameter and checking against the configured threshold.

In other words, a frame n is recorded with a given exposure time exp_time and a bit shift bitshift. For this purpose, further predetermined camera recording parameters cam_capture_par are used. A query is made as to whether the head is detected in frame n-1.

When the head is detected, quality parameters qpar for frame n are calculated, qpar are image quality parameters such as image brightness or contrast in the full image or the face bounding box. This is followed by another query as to whether qpar is greater or less than a threshold qpar_thr. Where qpar is greater than qpar_thr if qpark is greater than qpar_thrk and qpark + i is greater than qpar_thrk + i and within operational header related parameters and

Thresholds is.

If the query is positive, the exposure time exp_time is set and the bit shift for frame n + 1 is adjusted using a controller. In a first step, the exposure is adjusted. If g_mean is less than g_mean_thr, the exp_time (n + l) is set to clip (exp_time (n) + exp_time_step). If g_mean is greater than g_mean_thr, the exp_time (n + l) is set to clip (exp_time (n) -exp_time_step). The exp_time_range is 0.5 to 3 ms. The exp_time_step is 0.5 ms. In a second stage, the bit shift is set. If qpar (n) - qpar (n-l) is greater than qpar_delta_thr and last_adj_act is greater than

last_adi_act_thr, then bitshift (n + l) is set as bitshift (n) << bitshift_step. If qpar (n) -qpar (n-l) is less than qpar_delta_thr and last_adj_act is less than last_adi_act_thr, bitshift (n + l) is set as bitshift (n) >> bitshift step. The bitshift_range is 6 to 8 and the

Bitshift step is 1.

In one embodiment, the exposure is set based on the first level histogram. If Hist_pix_cnt is greater than pix_cnt thr and the mean gray value, that is, Hist_pix_dark_cnt is greater than pix_dark_cnt_thr and g_mean is less when g_mean_thr is set, the exp_time (n + l) is set to clip (exp_time (n) + exp_time_step). If Hist_pix_cnt is smaller than pix_cnt thr and the mean gray value, ie Hist_pix_bright_cnt is greater than pix bright cnt thr and g_mean is greater than g_mean_thr, the exp_time (n + l) is set to clip (exp_time (n) -exp_time_step).

4 shows a state diagram of a controller 400 for a method for operating an interior camera according to an exemplary embodiment. The controller is a finite automaton or state machine

or Finite State Machine (FSM). The controller essentially corresponds to the controller in FIG. 3. The controller 400 has a first function block 402, a second function block 404 and a third

Function block 406 on. In the process, a global ROI in a global state is used as a default value as long as no head is detected. When the head is detected, the controller 400 changes to a head tracking state within a head tracking ROI. When the head is no longer detected, the controller 400 transitions to a transient state with a transitional time

Transition ROI. If the head is recognized again during the transitional period, the controller 400 returns to the head tracking state in FIG

Head tracking ROI. If the head is not recognized within the transition time, the controller 400 returns to the global state with the global ROI.

5 shows illustrations of a detection of an object 500 in an image of an interior camera according to an exemplary embodiment. At this time, as shown in Fig. 4, a preset setting of the region of interest 502 (ROI) is started. In the default state, the indoor camera with a fixed exposure time or fixed exposure time or with adaptive

Exposure time or operated with adaptive exposure time. The ROI 502, where the probability for the head 500 is highest, is used. The region of interest 502 may be referred to as header 502.

The ROI size in the head tracking depends on the two-dimensional size of the detected head 500, the quality of the head tracking and the Frame rate of the head tracking. The center of the region of interest 502 is the center of the captured two-dimensional head 500, with limitation, to hold the region of interest 502 within the image.

The setpoint is converted linearly over time. If the exposure time is fixed, a low pass filter with a longer rise time is used. if the

Exposure time is adaptive, corners of the region of interest 502 are converted from the head track to linear ROI corners.

FIG. 6 shows a flowchart of an algorithm for operating a

Interior camera 104 according to an embodiment. The controller implements a model-based algorithm. The optimum actuation value or the exposure time u is calculated using an inverse model 600. An input signal u is the exposure time. An output signal y is an average of image pixel values. Both values are processed in the inverse model, giving an estimated optimal

Input signal u, which is filtered via an LPF low-pass filter 602 with a rise time of 0.14 seconds to be used again as input signal u.

Fig. 7 shows a block diagram of a controlled system for operating a

Interior camera according to an embodiment. The controlled system implements the algorithm shown in FIG. 6. Using a mean target value and an image average (y), the exposure time is computed in a calculator 700 as the estimated optimal input signal (u) of the system. The mean target value is a predefined mean value which leads to good image quality. The image average is the calculated average of the image pixel values of a downsampled or downsampled image. The input signal (u) is filtered in the low-pass filter 602 with the rise time of 0.14 seconds. In an imaging device 702, the

Input signal of the system is mapped to camera control variables. The

Camera action variables are exposure time, gain and bit shift. 8 shows an illustration of an application case for an interior camera 104 according to an exemplary embodiment. In this case, in contrast, the head 110 is arranged close to the camera 104 or IR 112 at a near border of the head area 106 or a beginning of the head motion box HMB 106.

9 shows an illustration of an application case for an interior camera 104 according to an embodiment. The application case corresponds essentially to the application in Fig. 1. Here, in contrast, the head

110 far from the camera 104 or I R 112 at a remote boundary of the head portion 106 and one end of the head motion box HMB 106 is arranged. 10 shows an illustration of an application case for an indoor camera

104 according to an embodiment. The application case essentially corresponds to the application in FIG. 1. In this case, an object 1000 is arranged between the camera and the head 110. The object 1000 partially covers the head 110. The obscuring object 1000, for example, a hand or the steering wheel shadows parts of the face.

11 shows an illustration of an application case for an interior camera 104 according to an embodiment. The application corresponds essentially to the application in Fig. 1. Here, the head 110 is illuminated by extraneous light sources 1100. Ambient light, for example from the

Infrastructure or from front headlights oncoming vehicles and / or sunlight from different angles, such as frontal, side and rear causes a large dynamic range. The ambient light, different head positions to the position of the camera 104

or the IR 112 due to the head motion field 106 (HMB) and interfering objects requires a camera control with good image quality for computer vision.

FIG. 12 shows a flowchart of a method for operating a

Interior camera according to an embodiment. The method points a step 1200 of the regulation. Here, at least one camera parameter of the indoor camera is controlled by using at least one quality parameter of a previously acquired image of the indoor camera when a head of a target person is detected in the image. The camera parameter is set to a predefined value if no head is detected.

If an exemplary embodiment comprises a "and / or" link between a first feature and a second feature, then this is to be read so that the embodiment according to one embodiment, both the first feature and the second feature and according to another embodiment either only first feature or only the second feature.

Claims

claims

A method for operating an interior camera (104) for a vehicle (100), wherein in a step (1200) of the rule at least one

Camera parameter (122) of the indoor camera (104) is controlled using at least one quality parameter (118) of a previously acquired image of the indoor camera (104) when a head (110) of a target person (108) is detected in the image, wherein the camera parameter ( 122) is set to a predefined value if no head (110) is detected.

2. The method according to claim 1, wherein in step (1200) of the rule as a quality parameter (118), a contrast, a brightness and / or a brightness distribution of the image is used as the camera parameters (122) an exposure time and / or sensitivity of the

Indoor camera (104) to regulate.

3. Method according to one of the preceding claims, in which a contrast, a brightness and / or a brightness distribution of the image is used as quality parameter (118) in step (1200) of the sequence to produce a light intensity of a camera parameter (122)

Lighting device (112) of the interior camera (104) to regulate.

4. The method according to any one of the preceding claims, with a

A step of detecting the head (110) in a captured image of the indoor camera (104), wherein in particular the quality parameter (118) is related to a header (502) of the image.

5. The method according to any one of the preceding claims, with a

A step of adjusting in which a color depth of a raw image signal (210) of the indoor camera (104) is adjusted to a working image signal (212), the head (110) being in an image of the

Working image signal (212) is detected.

A method according to claim 5, wherein in the setting step, a color depth spectrum is extracted from the raw image signal (210) to obtain the working image signal (212).

A method according to any one of claims 5 to 6, wherein, in the setting step, raw color steps of the raw image signal (210) are assigned using a processing rule to working color levels of the working image signal (212) to obtain the working image signal (212).

An apparatus (102) arranged to execute steps (1200) of the method according to one of the preceding claims in at least one corresponding unit (120).

A computer program adapted to carry out the method according to any one of the preceding claims.

10. A machine-readable storage medium on which the computer program according to claim 9 is stored.