WO2017129306A1

WO2017129306A1 - Method and device for locating a head of a driver in a vehicle and head localization system

Info

Publication number: WO2017129306A1
Application number: PCT/EP2016/080883
Authority: WO
Inventors: Anke SCHWARZ; Zhuang LIN; Rainer Stiefelhagen
Original assignee: Robert Bosch Gmbh
Priority date: 2016-01-29
Filing date: 2016-12-14
Publication date: 2017-08-03
Also published as: DE102016201329A1

Abstract

The invention relates to a method for locating a head (108) of a driver (110) in a vehicle (100). First, a 3D image signal (112) is read in, which represents a three-dimensional image of the head (108) that has been detected by the at least one sensor (104) of the vehicle (100). Subsequently, the 3D image signal (112) is evaluated using a regression method in order to establish a localization value (114) for locating the head (108).

Description

Description title

Method and device for locating a head of a driver in a vehicle and head localization system

State of the art

The invention is based on a device or a method according to the preamble of the independent claims. The subject of the present invention is also a computer program.

Based on the Kopforientierung a driver can, for example, a

Lane change intent, attention or fatigue of the driver. A corresponding Kopfposenschätzung, for example, in a 2-D image-based method or using depth data for

Real-time capable Kopfposenschätzung be realized.

DISCLOSURE OF THE INVENTION Against this background, with the approach presented here, a method for locating a head of a driver in a vehicle, a device using this method, a head localization system, and finally a corresponding computer program according to FIGS

Main claims presented. The measures listed in the dependent claims advantageous refinements and improvements of the independent claim device are possible.

A method for locating a driver's head in a vehicle is presented, the method comprising the steps of: Reading a 3-D image signal representing a three-dimensional image of the head detected by at least one sensor of the vehicle; and

Evaluating the 3-D image signal using a regression method to produce a localization value for locating the head.

The sensor may in particular be a photonic mixer detector, PMD sensor for short. The sensor can for example be part of a time-of-flight camera, TOF camera for short. In this case, the sensor can be designed to illuminate a scene comprising the head by means of a light pulse and to measure for each pixel a transit time of the light from the sensor to the head and back, the transit time being directly proportional to the distance of the pixel from the sensor. The sensor can thus detect a specific distance to the head for each pixel. Here, the head can with a single

Depth of field can be completely detected by the sensor. Under a

Regression methods can be understood as a statistical analysis method for performing a regression analysis by which relationships between a dependent and one or more independent variables can be modeled. For example, a functional relationship determined by the regression method may be used to generate a

Predictive model for predicting a head pose of the head. For example, the regression method may be based on a linear regression in which a model can be specified by a linear combination of different variables. For example, the location value may be a location, i. H. a position, or a position, d. H. to represent an orientation of the head. Here, the localization value may include, for example, space coordinates or inclination angles of the head in an x, y, or z direction.

The approach described here is based on the finding that depth data from a recording of a driver's head produced by a 3-D sensor can be evaluated by means of a regression-based method in such a way that a pose of the head, in particular an orientation of the head, is fast and accurate robust can be determined. For example, the depth data can be generated by a TOF camera with its own light source. The advantage of such an evaluation is thus that it is particularly robust to varying light conditions, that is largely independent of light conditions in the vicinity of the head can be performed.

The approach presented here can significantly reduce errors in the determination of the head pose. At the same time, the efficiency of a depth-based head pose estimation can be significantly increased.

According to one embodiment, in the step of evaluating, the 3-D image signal may be evaluated to provide a location and, additionally or alternatively, a location of the head as the location value. A location can be understood as meaning a position of the head in the vehicle interior. Under a position of the head can be understood an orientation of the head in the vehicle interior, d. h., the head can change its position by displacement and its position by twisting. Thus, for example, the localization value may be cubic or planar coordinates to the tri-or

two-dimensional location or certain angles relative to

represent corresponding orientation axes for determining the position. By this embodiment, the head with high accuracy and

Reliability can be located.

According to another embodiment, in the step of evaluating, the 3-D image signal may be evaluated to generate a roll angle of the head as the localization value. Additionally or alternatively, in the step of evaluating, a pitch or yaw angle of the head may be generated as the location value. A roll angle may be understood to mean an angle representing rotation of the head about a longitudinal axis of the head. A nickel angle may be understood to mean an angle representing rotation of the head about a transverse axis of the head. A yaw angle may be understood to mean an angle representing rotation of the head about a vertical axis of the head. By this embodiment, the position of the head can be determined with high accuracy and reliability.

Furthermore, the method may comprise a step of determining, depending on the embodiment, a line of sight, a tiredness or a

Attention of the driver using the localization value can be determined. For example, the viewing direction, fatigue or attention may serve as an input to a controller for controlling the vehicle. By this embodiment, the gaze direction, the tiredness or the attention of the driver depending on the

Head pose to be determined.

In addition, in an optional provisioning step, a control signal for controlling the vehicle may be provided depending on the viewing direction, tiredness or attention. For example, the control signal may be used to control a driver assistance function of the vehicle.

Thereby, the vehicle can be controlled depending on the viewing direction, fatigue or attention, whereby the ride comfort and the traffic safety can be increased. According to a further embodiment, in the step of evaluating the

3-D image signal using at least one regression tree and, additionally or alternatively, at least one regression forest are evaluated. A regression tree can be understood as an ordered, directed tree which serves to display hierarchically successive regression rules. A regression forest can be understood as a set of interrelated regression trees. By this embodiment, the 3-D image signal can be evaluated particularly efficiently and precisely. It is also advantageous if, in the reading-in step, a 2-D image signal is read in which represents a two-dimensional image of the head detected by the sensor and, additionally or alternatively, at least one further sensor. Accordingly, in the step of evaluating, the 2-D image signal may be further evaluated to generate the localization value.

This can increase the accuracy of locating the head.

It is also advantageous if in the step of reading a previous

Localization value is read to locate the head. The earlier localization value may represent an older value than the localization value. This can be done in the step of evaluating using the previous localization value, at least one local binary feature will be extracted from the 3-D image signal. Using the local binary feature and at least one global linear regression rule, a deviation value representing a deviation between the previous localization value and the localization value can then be determined. Finally, the localization value may be a function of the previous one

Localization value and the deviation value are generated. By a local binary feature may be understood a binary feature that can be extracted from depth data representing a local area of the head, for example using a regression tree or forest. A global linear regression law may be understood as a rule for performing a linear regression that relates to an entire region of the head. Depending on the embodiment, the regression rule may include regression trees or forests. This embodiment enables precise localization of the head in real time.

According to a further embodiment, in the step of evaluating the 3-D image signal can be evaluated using quaternions for representing a rotation of the head about at least one axis of rotation.

As a result, the computational effort during evaluation can be reduced.

This method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit.

The approach presented here also provides a device which is designed to implement the steps of a variant of a method presented here

to implement, control or implement appropriate facilities. Also by this embodiment of the invention in the form of a device, the object underlying the invention can be solved quickly and efficiently.

For this purpose, the device can have at least one computing unit for processing signals or data, at least one memory unit for storing Signals or data, at least one interface to a sensor or an actuator for reading sensor signals from the sensor or for outputting data or control signals to the actuator and / or at least one

Communication interface for reading or outputting data embedded in a communication protocol. The arithmetic unit may be, for example, a signal processor, a microcontroller or the like, wherein the memory unit is a flash memory, an EPROM or a

magnetic storage unit can be. The communication interface can be designed to read or output data wirelessly and / or by line, wherein a communication interface that can read or output line-bound data, for example, electrically or optically read this data from a corresponding data transmission line or output in a corresponding data transmission line.

In the present case, a device can be understood as meaning an electrical device which processes sensor signals and outputs control and / or data signals in dependence thereon. The device may have an interface, which may be formed in hardware and / or software. In the case of a hardware-based embodiment, the interfaces can be part of a so-called system ASIC, for example, which contains a wide variety of functions of the device. However, it is also possible that the interfaces are their own integrated circuits or at least partially consist of discrete components. In a software training, the interfaces may be software modules that are present, for example, on a microcontroller in addition to other software modules.

In an advantageous embodiment, the device is used to control the vehicle. For this purpose, the device can access, for example, sensor signals such as steering angle, acceleration or environmental sensor signals. It is controlled by actuators such as steering or brake actuators or a

Engine control unit.

The approach presented here also creates a head localization system with the following features: a device according to a preceding embodiment; and at least one sensor for generating the 3-D image signal. Of advantage is also a computer program product or computer program with

Program code, which may be stored on a machine-readable medium or storage medium, such as a semiconductor memory, a hard disk memory or an optical memory, and for carrying out, implementing and / or controlling the steps of the method according to one of the above

described embodiments is used, in particular when the

Program product or program is executed on a computer or a device.

Embodiments of the invention are illustrated in the drawings and explained in more detail in the following description. It shows:

Fig. 1 is a schematic representation of a vehicle with a

Head localization system according to an embodiment; Fig. 2 is a schematic representation of a device according to a

Embodiment;

3 is a flowchart of a method according to a

Embodiment;

4 is a schematic representation of a sequence of a

Head pose estimation using a device according to an embodiment; Fig. 5 is a schematic representation of a sequence of a

Head pose estimation using a device according to an embodiment; and

6 is a flowchart of a method according to a

Embodiment. In the following description of favorable embodiments of the present invention are for the in the various figures

represented and similar elements acting the same or similar

Reference numeral used, wherein a repeated description of these elements is omitted.

1 shows a schematic representation of a vehicle 100 with a head localization system 102 according to one exemplary embodiment. The

Head localization system 102 includes a sensor 104, such as a TOF camera, and a device 106 connected to the sensor 104. The sensor 104 is configured to capture a three-dimensional image of a head 108 of a driver 110 of the vehicle 100 and to send a 3-D image signal 112 representing the three-dimensional image to the device 106. The device 106 is configured to evaluate the 3-D image signal 112 using a regression method to image the head 108 in FIG

Vehicle interior to locate. Here, the device 106 generates a location value 114 representing, for example, a head position or orientation.

The device 106 makes it possible to determine the head orientation as robustly as possible and in real time, even in the case of strong occlusions.

Current fatigue estimation systems are based, inter alia, on non-visual parameters. The systems warn on the basis of a steering angle based on or using lane keeping sensors

Driving behavior. These parameters may be significantly further in the signal chain than directly measurable physiological changes. Furthermore, the steering behavior can not always be clearly linked to fatigue. For example, when driving with trailer a similar

Steering angle signal can be generated as when driving with increased fatigue.

By means of the device 106, the head position and orientation can be determined stepwise in a regression-based method. The advantage is that this regression trees are applied to depth data and a deviation from the head position and orientation can be output directly by a global linear transformation. Thus, a direct determination of the head position and orientation is made possible. FIG. 2 shows a schematic representation of a device 106 according to a

Embodiment. The device 106 is, for example, a device described above with reference to FIG. The device 106 comprises a read-in unit 210 for reading in the 3-D image signal 112. The read-in unit 210 is connected to an evaluation unit 220, which is designed to receive the 3-D image signal 112 from the read-in unit 210 and using the regression method evaluate. As a result of the evaluation, the evaluation unit 220 outputs the localization value 114.

According to this embodiment, the evaluation unit 220 outputs the

Localization value 114 to an optional determination unit 230 configured to determine a driver's line of sight, fatigue, or attention using the localization value 114, and to generate an output signal 235 representing sight, fatigue, or attention. For example, the evaluation unit 220 generates a value as the localization value 114, which represents a location or a position of the head in the vehicle interior, wherein the position is defined, for example, by an angle value representing a roll, pitch or yaw angle of the head. Accordingly, the determination unit 230 determines the viewing direction, fatigue, or attention depending on the location or location of the head.

Optionally, the device 106 includes a provisioning unit 240 configured to receive the output signal 235 from the determination unit 230 and to provide, using the output signal 235, a control signal 245 for controlling the vehicle, for example to a vehicle

Interface to a corresponding control unit of the vehicle. The

Control signal 245 thus serves to control the vehicle depending on the driver's gaze, fatigue or attention. According to a further embodiment, the read-in unit 210 is designed to read in an optional 2-D image signal 250 in addition to the 3-D image signal 112 and to forward it to the evaluation unit 220. Depending on

Embodiment, the 2-D image signal 250 from either the sensor producing the 3-D image signal 112 or from another sensor of the

Vehicle generated. In a similar way, the

Evaluation unit 220 may be configured to further use the 2-D image signal 250 to generate the localization value 114. For example, in this case, the 3-D image signal 112 can be evaluated using the 2-D image signal 250.

FIG. 3 shows a flowchart of a method 300 according to FIG

Embodiment. The method 300 for efficient

For example, head orientation estimation of depth data may be in the

Connection with a preceding with reference to Figures 1 and 2

be executed or controlled device described. Here, the driver is continuously detected by a depth sensor. The sensor captures both 2-D data and depth data. From the data received will be

Head position and orientation determined. It then turns the line of sight, a level of drowsiness, the attention or certain

Security settings determined.

In Fig. 3, the driver is through a block 302, an input through a block 304, the sensor through a block 306, the head position and orientation by a block 308, a computing unit for processing the sensor data through a block 310, an output a block 312, the viewing direction through a block 314, the level of drowsiness through a block 316, the attention through a block 318 and a safety setting, such as an adjustment for an airbag, represented by a block 320.

4 shows a schematic representation of a sequence 400 of a

Kopfposenabschätzung using a device according to an embodiment. The head pose estimation can be performed, for example, using a device as described above with reference to FIGS. 1 to 3. Shown is a step-by-step approach to a head pose estimation from depth images. In this case, an update ΔΘ ^{1 of} a head orientation and position is calculated in each step t, hereinafter also referred to as the deviation value. The head orientation and position Θ ^{1 of} a step t is indicated by an arrow.

The head pose estimation is based on a step-by-step regression technique. This allows the head orientation and position to be estimated with very high efficiency. Here, in each step, a

Update of head orientation and position calculated. According to an exemplary embodiment, an extraction of local binary features based on depth data is performed for this purpose. Such feature extraction is performed using local regions that have been projected relative to the head orientation and position. By using the quaternion representation, the head orientation can be determined directly via a global linear regression.

5 shows a schematic representation of a sequence 500 of a

Kopfposenabschätzung using a device according to an embodiment. The sequence 500 shown in FIG. 5 is an overview of sub-steps of a preceding with reference to FIG. 4

described step t. The input data is a former

Localization value representing an earlier head pose Θ ^{1 "1.} In addition, a ground-truth head orientation gezeigt is shown A deviation value ΔΘ ¹ is determined in a step 510, which in turn is divided into two sub-steps 520 530. Local binary features Φ ^{1 are} calculated in the first sub-step 520. The local binary features are schematically indicated by circled points in Fig. 5. In the second sub-step 530, a global linear mapping W ^{1 is} carried out Updating the head pose Θ ¹ is used in step 540 using the

Deviation value ΔΘ ¹ determined.

Hereinafter, the sequence 500 shown in FIG. 5 will be explained again in detail.

The aim of the approach presented here is, according to one embodiment, a location and a rotation of the head on the basis of a single

To determine depth scans. For this a step-by-step approach is chosen each step provides an update to the correct head orientation and position. As shown in FIG. 5, in each step t, after the determination of a local binary feature Φ ^1, a global linear regression W is applied. First, a training will be described by which regression forests are learned to determine the local linear features Φ ¹ . In addition, the global linear projection W is trained. This is the test phase

described in which the approach presented here to an unknown

Depth scan is applied.

First, a supervised training step with labeled training data is performed. Each training pattern i consists of a depth map D, and labeled ground truth data Θ _; = {, -, θ, -} of the head position I _i = {x, y, z} and the head orientation shown in quaternions θ _ί =, ₂ , ₃ , ₄ }. Of the

The advantage of using quaternions is that they allow a rotation of a four-dimensional vector to be represented in a compact manner. In comparison to Euler angles, rotations can be calculated in a more efficient way. This can increase the efficiency in the stepwise determination of the head orientation.

The facial area is made using a commercial

Face detector detected in the preprocessing phase. During the

Training process, a regression forest for each step t for the determination of the local binary features Φ ¹ and the global linear projection W is learned.

The local binary features Φ ¹ are calculated from regression forests. To learn the regression forests, the depth data D is used instead of two-dimensional color information. The training requires two-dimensional ground-truth positions S ^ and two-dimensional positions of the previous step S ^"1. For each frame i, the ground truth positions 5 _; e R ^{kxl are} generated by dividing K local areas of a three-dimensional

Mean face relative to the ground Truth head orientation and position are projected. The projection of these local areas by using the determined head orientation from the previous step t-1 results in the two-dimensional positions of the previous step S ^{'~ l} e R ^kx2. For each local area k, a binary forest is trained independently of each other. The learned forests are applied to the k local areas of the earlier step S ^-1 to obtain the binary feature vector Φ '. The binary feature vector is 1 when the pattern reaches a leaf node.

Otherwise, the binary feature vector is 0.

It also learns a global linear regression W by which a

Deviation ΔΘ 'can be determined from a current head position and orientation. In the training phase, a suitable mapping W is learned using the previously determined binary vector Φ 'and the residual ΔΘ- ₅ with respect to the ground-truth head pose. The optimal global linear regression W is obtained by minimizing the following objective function:

The ground truth residuals are determined as follows:

where R _; is the θ from the quaternions _ί calculated rotation matrix represents and R ^'1 of 0 / ^"1 calculated.

After each step t, the head orientation and position of each frame i is updated as follows

With

ΔΘ;

AR ^ is the rotation matrix calculated from the quaternions Αθ '. R \ is the rotation matrix calculated from the quaternions 0 ¹ .

In the test phase the head position and the head orientation Θ _; , = {,, θ,} from an unknown distance image D, are determined. In each step a residual of the head position and orientation ΔΘ 'relative to the current head orientation is determined. For each step t, a local substep 520 and a global substep 530 are performed. In local substep 520, the local binary features Φ 'are trained using the trained

Regression forest determined. Subsequent to this substep, the binary features are represented by the learned global linear regression W ¹ with the deviation of the head position and orientation

Deviation value Θ '= {Δ /', Δ0} linked as described in equation (4).

After each step, the head pose is updated with = / (Θ ' ^~ ΑΘ') as described in Equation (3).

FIG. 6 shows a flowchart of a method 600 according to FIG

Embodiment. The method can be performed or controlled, for example, in connection with a device described above with reference to FIGS. 1 to 4. The steps described with reference to FIGS. 4 and 5 can be sub-steps of the method 600. The method 600 includes a step 610 in which the 3-D image signal is read. In a further step 620, the 3-D image signal is evaluated using a regression method to generate the localization value.

According to an embodiment, the 3-D image signal is generated in step 620 using at least one regression tree or one

Regression forest evaluated, such as in a manner as described above with reference to FIG. 5.

Steps 610, 620 may be performed continuously. If an exemplary embodiment comprises a "and / or" link between a first feature and a second feature, then this is to be read so that the embodiment according to one embodiment, both the first feature and the second feature and according to another embodiment either only first feature or only the second feature.

Claims

claims

A method (600) for locating a head (108) of a driver (110) in a vehicle (100), the method (600) comprising the steps of:

Reading (610) a 3-D image signal (112) representing a three-dimensional image of the head (108) detected by at least one sensor (104) of the vehicle (100); and

Evaluating (620) the 3-D image signal (112) using a regression method to obtain a localization value (114) for

Locate the head (108) to produce.

The method (600) according to claim 1, wherein in the step of evaluating (620) the 3-D image signal (112) is evaluated to determine a location and / or a location of the head (108) as the localization value (114). to create.

3. Method (600) according to one of the preceding claims, wherein in the step of evaluating (620) the 3-D image signal (112) is evaluated to determine a roll and / or pitch and / or yaw angle of the head (108 ) as the localization value (114).

A method (600) according to any preceding claim including a step of determining a driver's (110) gaze and / or fatigue and / or attention using the localization value (114).

The method (600) of claim 4, further comprising a step of providing a control signal (245) for controlling the vehicle (100) in FIG Dependence on the direction of vision and / or fatigue and / or attention.

Method (600) according to one of the preceding claims, wherein in the step of evaluating (620) the 3-D image signal (112) is evaluated using at least one regression tree and / or at least one regression forest.

Method (600) according to one of the preceding claims, wherein in the read-in step (610) a 2-D image signal (250) is read in which comprises a two-dimensional image of the sensor detected by the sensor (104) and / or at least one further sensor In the step of evaluating (620), the 2-D image signal (250) is further evaluated to generate the localization value (114).

A method (600) according to any of the preceding claims, wherein in the step of reading (610) a previous location value Θ ^{1 "is} read to locate the head (108) ^1, wherein the earlier localization value Θ ^{1" 1} an older value than the location value (114), wherein in the step of evaluating (620) using the earlier localization value Θ ^{1 "1,} at least one local binary feature Φ ^{1 is} extracted from the 3-D image signal (112) using the local binary feature Φ ¹ and at least one global linear regression law W ¹ , a deviation between the earlier localization value Θ ^{1 "1} and the

Localization value (114) representing deviation value ΔΘ ^{1 is} determined and the localization value (114) as a function of the earlier localization value Θ ^{1 "1} and the deviation value ΔΘ ^{1 is} generated.

Method (600) according to one of the preceding claims, wherein in the step of evaluating (620) the 3-D image signal (112) is evaluated using quaternions for representing a rotation of the head (108) about at least one axis of rotation.

10. Device (106) having units (210, 220, 230, 240) which are designed to execute and / or control the method (600) according to one of the preceding claims.

11. A head locating system (102) comprising: a device (106) according to claim 10; and at least one sensor (104) for generating the 3-D image signal (112).

12. Computer program adapted to carry out the method (600)

according to any one of claims 1 to 9 execute and / or to control.

13. Machine-readable storage medium on which the computer program

is stored according to claim 12.