DE102013021929A1 - Motor vehicle with cameras and image analysis devices - Google Patents

Abstract

The invention relates to a method for operating a motor vehicle (10), which has at least two cameras (16, 18) and a respective analysis device (20, 24) for camera-based determination of a respective property parameter (E1, E2) for each camera (16, 18). a person (12) filmed by the cameras (16, 18), wherein the analysis devices (20, 24) control at least one vehicle system (22, 26) by means of their respective property parameter (E1, E2). The object of the invention is to determine at least one of the property parameters (E1, E2) more reliably. For this purpose, a coupling device (28) controls at least one of the analysis devices (20, 24) a segmentation (38, 38 ') and / or a feature extraction (40, 40') and / or on the segmentation (38, 38 ') and or the feature extraction (40, 40 ') based determination of the respective property parameter (E1, E2) in dependence on analysis data (S1, S2, F1, F2, E1, E2) of the respective remaining analysis means (20, 24).

Description

The invention relates to a motor vehicle which has two cameras and, for each camera, a respective analysis device for camera-image-based determination of property parameters of a person filmed by the cameras. Such a property parameter may indicate, for example, the person's head position or the person's line of sight. At least one of the analysis devices transfers the determined property value to a vehicle system, which provides a predetermined function in the motor vehicle as a function of the property parameter.

This is from the DE 20 2009 018 513 U1 an adjusting drive for a motor vehicle, which comprises at least one biometric sensor, for example a camera, as well as an evaluation device which detects data of the biometric sensor and derives therefrom an authorization of the filmed person.

From the DE 10 2007 023 140 A1 For identification of a driver of a motor vehicle, a system is known, which determines the motion sequences of a filmed person as a function of image data of a camera and differentiates between the movements of an already known driver and a new driver. To determine the motion sequences, movements or motion sequences are extracted from image data by an image processing unit.

From the DE 10 2007 023 141 A1 a method for adjusting at least one comfort and / or safety system in a motor vehicle is known. Film data of a camera are used to determine surveying and / or position data of at least one recorded part of a person's body, and then at least one comfort and / or security system is set directly as a function of the surveying and / or position data.

From the EP 1 562 102 A2 is a function triggering by Blickauswahl known. Here, the driver is filmed and determined from the image data, the driver's line of sight. As soon as the driver looks at a symbol surface and presses a confirmation key, a function of the motor vehicle assigned to the symbol surface is activated.

From the DE 102 57 963 A1 is a system for determining a 3D position of car occupants known. The 3D position is performed by observing the vehicle occupants with at least two cameras, which do not necessarily have to be operated according to a stereo principle. The use of two cameras is necessary to obtain a spatial description of the head position that could not be captured by a single camera.

From the DE 10 2008 038 816 A1 For determining a condition of a driver, a method is known in which a three-dimensional model of the head of the driver is created by means of two cameras and thereby characteristic features of the head, such as nostrils and corners of the mouth on the driver's head, are detected and measured.

In summary, camera systems are known from the prior art that observe the driver or other vehicle occupants. Property parameters of the observed person for controlling vehicle systems, such as comfort systems, assistance systems or safety systems, are determined from the image data of the camera systems. Important here is the suitable choice of the detection or operating principle (eg color image or infrared) of the camera system used to robust and required for the control of the vehicle system property parameters of the filmed person. B. to be able to reliably determine under as many lighting situations.

The object of the invention is to reliably determine a property parameter which is to be used for controlling at least one vehicle system.

The object is solved by the subject matters of the independent claims. Advantageous developments of the invention will become apparent from the features of the dependent claims.

The invention is based on the idea of coupling analysis devices, which each extract a separate property parameter from image data from different cameras, in order thereby to support at least one of the analysis devices in their analysis by means of analysis data from another analysis device. For example, one can support an eye-tracking system, which determines a viewing direction as a property parameter, such that, for example, an analysis system which analyzes thermal image data of a thermal imaging camera transmits information on the position of the face to the analysis device of the eye-tracking system. Then it can be verified, if the eyes were recognized correctly.

Generally, the invention provides a motor vehicle, which has a first camera for filming a person and a first analysis device, ie z. B. an eye tracker, the first Analysis device is designed to determine from image data of the first camera a person describing the first property parameter. In addition, a second camera for filming the same person and a second analysis device is provided, so z. B. the thermal image analysis, wherein the second analysis device is adapted to determine from the image data of the second camera a person describing the second property parameter.

The two cameras can be cameras of the same functional principle or even the same camera. However, it is preferably provided that the first camera and the second camera capture their image data on the basis of different functional principles, wherein the first camera and the second camera preferably each have one of the following functional principles: PMD (Photonic Mixing Device), thermal image acquisition, infrared image acquisition, color image acquisition, Color intensity image capture (black and white), stereoscopy. In the latter case, the corresponding camera is a stereo camera.

The two analysis devices may be two analysis devices known per se from the prior art and provided for the control of the same or different vehicle systems. At least the first analysis device is designed to control at least one vehicle system by its first property parameter, that is, the at least one vehicle system is configured to provide a predetermined function in the motor vehicle depending on the first property parameter. The second analysis device can also output its property parameter to the at least one vehicle system or also to another vehicle system. In the case of at least one vehicle system, corresponding refinements of the invention are a driver assistance system and / or a safety system for avoiding or protecting traffic accidents and / or an air conditioning system and / or an infotainment system and / or a rear seat entertainment system. The two analysis devices can be designed to determine the same property parameter. However, it is preferably provided that the two analysis devices determine different property parameters. Of course, it may be provided that one or both of the analysis devices also determine a plurality of property parameters, that is to say, for example, both a viewing direction and a motion vector which describes a temporal change of the viewing direction.

In order to support at least the first analysis device when determining the first property parameter, according to the invention the first analysis device and the second analysis device are coupled via a coupling device. The coupling device is designed to operate the first analysis device as a function of analysis data that has been determined by the second analysis device on the basis of the image data of the second camera.

The invention has the advantage that analysis data which the first analysis device itself has determined on the basis of the image data of the first camera can now be checked in the first analysis device and / or can first be derived on the basis of the analysis data of the second analysis device. Likewise, the first property parameter can now be determined not only on the basis of the image data of the first camera, but also on the basis of the analysis data. In other words, the first property parameter is now additionally determined as a function of analysis data of the second analysis device by means of the coupling device.

According to one embodiment of the invention, it is provided that the coupling device also conversely uses analysis data, which the first analysis device has determined on the basis of the image data of the first camera, for operating the second analysis device.

Further developments of the invention relate to the property parameters determined by the two analysis devices. The first analysis device and / or the second analysis device can each be designed to determine at least one of the following property parameters: a head position, a head orientation (position of the head vertical axis in space and / or direction vector of a face of the head), a viewing direction, a hand position , a hand position, a finger position, a movement direction, a movement speed.

Additional developments arise as a function of which analysis data from the second analysis device uses or reverses the coupling device for operating the first analysis device, which analysis data of the first analysis device for operating the second analysis device. The first case described in the following (first analysis device operated by analysis data of the second analysis device) accordingly applies conversely also to the second case (second analysis device operated by analysis data of the first analysis device).

According to a development, the coupling device is designed to enable the first analysis device as a function of segmentation data of the operate second analysis device. The segmentation data describe a segmentation of the image data of the second camera. A segmentation is a division of an image described by the image data into image areas, wherein pixels within a respective image area fulfill a respective predetermined homogeneity criterion. By way of example, one image surface may have skin-colored pixels, and an adjacent image surface may have, for example, pixels of a specific hair color. This results in a segmentation of an image in the face and hair area. Thus, for example, a segmentation performed by the first analysis device can be made plausible, verified or supplemented by the segmentation data of the second analysis device.

According to another development, the coupling device is designed to operate the first analysis device as a function of feature data of the second analysis device. Feature data describe a mapping of pixels or image areas to body features of the person determined with the image data of the second camera by means of a feature extraction. For example, feature data may indicate that a particular image area represents hair or a face or a hand. The feature data of the second analysis device can hereby be based on segmentation data of the second analysis device. If the first analysis device is based, for example, on a color image evaluation and if shadow or colored illumination disturbs the recognition of skin surfaces, for example, the image analysis of a thermal imaging camera can be used as the second analysis device since the self-heat of uncovered skin is a reliable indication of skin areas. Thus, the second analysis device can recognize much more robust uncovered skin and transfer the feature "skin" for individual pixels or already segmented image surfaces to the coupling device, which can support the first analysis device in segmenting or classifying image surfaces by means of this feature data or even completely replace the classification can, since already contained in the feature data classification result of the second analysis device is sufficient. By means of the feature data, for example, it can also be made plausible in the first analysis device whether the recognition of eyes is correct, because they must be located within an image area which describes a face.

According to another development, the coupling device is designed to operate the first analysis device as a function of the at least one second property value. In other words, the final analysis result, namely the second property value, is used to control the first analyzer.

The refinements explained below relate to the manner in which the analysis data of the second analysis device can be used to operate the first analysis device.

The coupling device is in particular designed to validate and / or compare and / or replace analysis data of the first analysis device (ie data determined by the first analysis device based on the image data of the first camera) by means of the analysis data of the second analysis device and / or or to combine. By plausibility is meant that the analysis data of the first analysis device is compared with the analysis data of the second analysis device, and if it is matched by a confidence signal, it is signaled that the analysis data of the first analysis device has been correctly determined. When a difference is detected, however, it is signaled by the confidence signal that the analysis data has been incorrectly determined by the first analysis device or at least the risk of misrecognition exists. By matching, it is meant that the first analyzing means iteratively determines its analysis data, for example, until such analysis data coincide with the analysis data of the second analysis means at least up to a predetermined tolerance value. By supplementing it is meant that the first analysis device bases both its own analysis data and the analysis data of the second analysis device on the determination of the first property value. In other words, the first property value is then also based on analysis data which the first analysis device has not itself determined, but which have been determined by the second analysis device. By replacement is meant that the analysis data of the first analyzer is discarded and instead the analysis data of the second analyzer is used. This can be decided, for example, as a function of respective confidence values for the analysis data of the first analysis device and the second analysis device if the probability of an error detection of the analysis data of the first analysis device is greater than that for the second analysis device. Examples of such confidence values are known per se, for example, in the context of classification based on hidden Markov models or other statistical classifiers. By combining, it is meant that analysis data based on both the analysis data of the first analyzer and that of the second analyzer is detected. For example, a weighted sum of the Analysis data of both analysis facilities are calculated. The weighting can here again be based on the described confidence values.

The embodiments of the invention described so far relate only to two analysis devices. It is preferably provided that in the motor vehicle at least one further analysis device for determining at least one further property parameter describing the person from image data of at least one further camera is provided. The coupling device is then designed to operate the first analysis device as a function of the analysis data determined by the at least one further analysis device. In other words, the first analysis device thus uses analysis data of both the second and the at least one further analysis device for determining the first property parameter.

Generally speaking, the motor vehicle according to the invention thus carries out the following method based on the motor vehicle having at least two cameras and for each camera a respective analysis device for camera-image-based determination of a respective property parameter of a person filmed by the cameras. Each analysis device controls by means of their respective property parameter at least one vehicle system. According to the method, a coupling device controls in at least one of the analysis devices a segmentation and / or a feature extraction and / or the particular determination of the respective property parameter based on the segmentation and / or the feature extraction in dependence on analysis data of the respectively remaining analysis devices.

Each analysis device can be provided, for example, as a program module of a processor device of the motor vehicle. Also, the coupling device may be provided as such a program module or as a coupling circuit, for example, between two control devices.

The invention also includes further developments of the method according to the invention, which have features which have already been described in connection with the developments of the motor vehicle according to the invention. For this reason, the corresponding developments of the method according to the invention will not be described again here.

The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car.

In the following an embodiment of the invention is described. For this purpose, the single FIGURE (FIG.) Shows a schematic representation of an embodiment of the motor vehicle according to the invention.

The exemplary embodiment explained below is a preferred embodiment of the invention. In the exemplary embodiment, however, the described components of the embodiment each represent individual features of the invention, which are to be considered independently of each other, which also develop the invention independently of one another and thus also individually or in a different combination than the one shown as part of the invention. Furthermore, the described embodiment can also be supplemented by further features of the invention already described.

In the figure is schematically a motor vehicle 10 shown, which may be, for example, a motor vehicle, especially a passenger car. In the motor vehicle 10 can a person 12 for example, in a passenger compartment 14 are located. The car 10 can be a first camera 16 and a second camera 18 exhibit. Both a detection area 16 ' the camera 16 and a detection area 18 ' the camera 18 are on the person 12 directed.

The camera 16 generates video data V1, which is sent to a first analysis device 20 be transmitted. The camera 18 generates video data V2 from the person 12 , which to a second analysis device 24 be transmitted.

At the first camera 16 and the second camera 18 each may be, for example, a PMD camera, an infrared camera, a color camera, a black and white camera or a stereo camera. The cameras 16 . 18 can be of the same camera type or different camera types.

The first analysis device 20 determined from the video data V1 at least one property parameter E1, which of the first analysis device 20 to a first vehicle system 22 can be issued. The second analysis device 24 determined from the video data V2 at least a second property parameter E2, which they to a second vehicle system 26 can spend.

The first vehicle system 22 adjusts a function in the motor vehicle depending on the first property parameter E1 10 provided. The second vehicle system 26 adjusts a function in the motor vehicle depending on the property parameter E2 10 provided. In the vehicle systems 22 . 26 These each in particular each a driver assistance system and / or a security system. But it can also be through the vehicle system 22 and / or the vehicle system 26 in each case an infotainment system and / or a rear-seat infotainment system be provided.

The first analysis device 20 can be configured to, for example, a line of sight as the property parameter 30 the person 12 , a spatial orientation 32 a face 34 the person 12 or a head direction 36 For example, an alignment of a head vertical axis to determine from the video data V1. The second analysis device 24 can be designed as a property parameter E2 also one of the described property parameters 30 . 32 . 34 to investigate. The property parameters illustrated here in the figure are to be considered as examples only.

The first analysis device 20 can z. B. be designed to determine the property parameter E1, the video data V1 by means of a segmentation 38 to segment, then for example from segmented image areas by means of a feature extraction 40 Determine image characteristics and based on the image characteristics by means of a parameter determination 42 determine the property parameter E1. For example, through the segmentation 38 a face area 44 on the one hand and a hair surface 46 and a background area 48 on the other hand, be segmented into frames of the video data V1. The feature extraction 40 can for the face area 44 , the hair surface 46 and the background area 48 For example, determine that face area 44 the feature "face", the hair surface 46 the feature "hair" and the background area 48 has the feature "irrelevant". Algorithms for providing segmentation 48 and a feature extraction 40 are known per se from the prior art. The parameter determination 42 can, for example, an area ratio of the face area 44 and the hair surface 48 on the alignment vector 32 of the face 44 shut down.

The second analysis device 24 can also be a segmentation 38 ' , a feature extraction 40 ' and a parameter determination 42 ' exhibit. The choice of segmentation 38 ' , the feature extraction 40 ' and the parameter determination 42 ' depends on the used property parameter E2. Here again can be drawn from the state of the art.

The first analysis device 20 and the second analysis device 24 are via a coupling device 28 coupled together. The coupling device 28 may be designed to produce a segmentation result 82 the segmentation 38 ' for example, to the segmentation 38 transferred to. Likewise, it can be provided by the coupling device 28 a segmentation result S1 of the segmentation 38 to the segmentation 38 ' transferred to. Likewise, feature data F2 of the feature extraction 40 ' and feature data F1 of the feature extraction 40 via the coupling device 28 be transmitted. The property parameters E2, E1 can also be sent to the respective other analysis device 20 . 24 be transmitted.

The exchanged analysis data S1, S2, F1, F2, E1, E2 can then be used for plausibility, matching, supplementing, replacing and / or combining with the respectively received by the receiving analysis device 20 . 24 self-determined analysis data are processed. In addition to the exchange of segmentation data between the segmentations shown in the figure 38 . 38 ' may for example also be provided by the coupling device 28 the segmentation data 82 also to the feature extraction 40 or to the parameter extraction 42 , Similarly, the feature data F2 to the segmentation 38 or to the parameter extraction 42 be transmitted. Finally, the property parameter E2 can also be applied to the segmentation 38 and / or feature extraction 40 be transmitted. In the same way, the reverse can be provided by the coupling device 28 the analysis data S1, F1, E1 to each segmentation 38 ' , the feature extraction 40 ' and the parameter determination 42 ' transferred to.

The first and the second analysis device 20 . 24 and the coupling device 28 can each by a program module of a processor device of the motor vehicle 10 be realized.

The cameras 16 . 18 and the analyzers 20 . 24 may be known from the prior art, known to be provided independently of each other devices for controlling the vehicle systems 22 . 26 be. Essential to the invention, the coupling device 28 , At the camera 16 or the camera 18 can it be z. B. also act on a thermal imaging camera. In the future, there will be various camera systems in the vehicle that monitor the driver or passengers. In addition, comfort systems and especially the air conditioning will always adapt to the needs of the driver. Thus, air flows in the vehicle can be controlled and tempered. One useful information is the knowledge of the body or surface temperature distribution of the occupants, which is why the use of passive thermal imaging cameras is to be expected. Against this background, it is thus provided according to the invention that the data determined via the eye-tracking, PMD camera or thermal imaging camera, for example regarding the direction of the head, to compare with each other and so plausibilisieren. This can be of great importance, above all if this information is relevant for assistance or security systems.

Overall, it is thus shown by the example, such. B. in a gesture control and / or an eye tracking a comparison of analysis data from a thermal imaging camera and / or eye-tracking camera and / or PMD camera can be done to thereby obtain more robust and / or more reliable analysis results against environmental variations ,

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 202009018513 U1 [0002]
• DE 102007023140 A1 [0003]
• DE 102007023141 A1 [0004]
• EP 1562102 A2 [0005]
• DE 10257963 A1 [0006]
• DE 102008038816 A1 [0007]

Claims (10)

Motor vehicle ( 10 ) comprising: - a first camera ( 16 ) for filming a person ( 12 ) and a first analysis device ( 20 ), which is adapted from image data (V1) of the first camera ( 16 ) a person ( 12 ) to determine the descriptive first property parameter (E1); - a second camera ( 18 ) for filming the person ( 12 ) and a second analysis device ( 24 ), which is adapted from image data (V2) of the second camera ( 18 ) a person ( 12 ) to determine the descriptive second property parameter (E2); and at least one vehicle system ( 22 ), which is designed, depending on the first property parameter (E1), a predetermined function in the motor vehicle ( 10 ) to provide; characterized in that the first analysis device ( 20 ) and the second analysis device ( 24 ) via a coupling device ( 28 ), which is designed to enable the first analysis device ( 20 ) as a function of the second analysis device ( 24 ) based on the image data (V2) of the second camera ( 18 ) to operate analysis data (S2, F2, E2) determined. Motor vehicle ( 10 ) according to claim 1, wherein the first analysis device ( 20 ) and / or the second analysis device ( 24 ) is designed to determine at least one of the following as the respective property parameter (E1, E2): a head position, a head orientation ( 32 . 36 ), a line of sight ( 30 ), a hand position, a hand position, a finger position, a movement direction, a movement speed. Motor vehicle ( 10 ) according to one of the preceding claims, wherein the coupling device ( 28 ) is adapted to the first analysis device ( 20 ) as a function of segmentation data (S2) of the second analysis device ( 24 ), wherein the segmentation data (S2) is a segmentation ( 44 . 46 . 48 ) of the image data (V2) of the second camera ( 18 ). Motor vehicle ( 10 ) according to one of the preceding claims, wherein the coupling device ( 28 ) is adapted to the first analysis device ( 20 ) as a function of feature data (F2) of the second analysis device ( 24 ), wherein the feature data (F2) one to the image data (V2) of the second camera ( 18 ) by means of a feature extraction ( 40 ' ) determined assignment of image areas ( 44 . 46 . 48 ) to body characteristics of the person ( 12 ). Motor vehicle ( 10 ) according to one of the preceding claims, wherein the coupling device ( 28 ) is adapted to the first analysis device ( 20 ) depending on the second property parameter (E2). Motor vehicle ( 10 ) according to one of the preceding claims, wherein the coupling device ( 28 ) is designed by the first analysis device ( 20 ) based on the image data (V1) of the first camera ( 16 ) determined analysis data by means of the analysis data (S2, F2, E2) of the second analysis device ( 24 ) and / or to supplement and / or to replace and / or to combine. Motor vehicle ( 10 ) according to one of the preceding claims, wherein the first camera ( 16 ) and the second camera ( 18 ) capture their image data (V1, V2) on the basis of different operating principles, the first camera ( 16 ) and the second camera ( 18 ) each have one of the following functional principles: PMD, thermal imaging, infrared imaging, color imaging, color intensity imaging, stereoscopy. Motor vehicle ( 10 ) according to one of the preceding claims, wherein the at least one vehicle system ( 22 ) comprises a driver assistance system and / or a safety system and / or an air conditioning system and / or an infotainment system and / or a rear seat entertainment system. Motor vehicle ( 10 ) according to one of the preceding claims, wherein at least one further analysis device for determining at least one person ( 12 ) is provided from image data of at least one further camera and the coupling device ( 28 ) is adapted to the first analysis device ( 20 ) to operate as a function of analysis data determined by the at least one further analysis device. Method for operating a motor vehicle ( 10 ), which has at least two cameras ( 16 . 18 ) as well as for each camera ( 16 . 18 ) a respective analysis device ( 20 . 24 ) for camera-image-based determination of a respective property parameter (E1, E2) of one of the cameras ( 16 . 18 ) filmed person ( 12 ), the analysis facilities ( 20 . 24 ) by means of their respective property parameter (E1, E2) at least one vehicle system ( 22 . 26 ), characterized in that a coupling device ( 28 ) in at least one of the analysis devices ( 20 . 24 ) a segmentation ( 38 . 38 ' ) and / or feature extraction ( 40 . 40 ' ) and / or on the segmentation ( 38 . 38 ' ) and / or feature extraction ( 40 . 40 ' ) based determination of the respective property parameter (E1, E2) in Dependence on analysis data (S1, S2, F1, F2, E1, E2) of the other analysis devices ( 20 . 24 ) controls.

Images