WO2020239448A1

WO2020239448A1 - Determining luggage for a vehicle

Info

Publication number: WO2020239448A1
Application number: PCT/EP2020/063397
Authority: WO
Inventors: Torsten Schön
Original assignee: Audi Ag
Priority date: 2019-05-24
Filing date: 2020-05-14
Publication date: 2020-12-03
Also published as: DE102019207616A1

Abstract

The invention relates to a method for determining a piece of luggage transported in a vehicle, in which a camera directed at a luggage compartment of a vehicle captures and provides an image of a piece of luggage arranged in the luggage compartment of the vehicle and a sensor of a chassis of the vehicle captures a value of a physical variable of the chassis that depends on a mass of the piece of luggage and provides a sensor signal with the captured value, to a system for capturing a piece of luggage transported in a vehicle, and to a vehicle.

Description

Baggage determination for a vehicle

DESCRIPTION: The invention relates to a method for determining an item of luggage transported in a vehicle, in which a camera directed into a trunk of a vehicle captures and provides an image of an item of luggage arranged in the trunk of the vehicle and a sensor of a chassis of the vehicle provides a value of a detects a physical variable of the chassis that is dependent on a mass of the item of luggage and provides a sensor signal with the detected value. The invention also relates to a system for determining an item of luggage transported in a vehicle and to a vehicle. Vehicles, in particular passenger cars (cars), which are primarily used to transport occupants, i. H. People who serve are mostly also designed to transport items of luggage. Luggage items are usually arranged for transportation in a rear area of a vehicle, which, depending on the vehicle model, is referred to as a load or trunk. In the following, a trunk is understood to mean any rear area of a vehicle that can be loaded with an item of luggage.

In certain situations, transporting an item of luggage in a trunk of a vehicle can be dangerous. The potentially dangerous situations also include unloading the vehicle, i.e. removing the item of luggage from the trunk. In particular, there can be a risk of injury if a heavy and / or large item of luggage has to be removed from the vehicle. A heavy item of luggage is to be understood as meaning a piece of luggage with a large weight, with a proportionality between the weight and a mass of the luggage item. A large item of luggage is understood to mean an item of luggage with large external dimensions, the external dimensions defining a volume, ie the space required, of the luggage item. In the following, the terms mass and size are used with these meanings throughout.

To avoid dangerous unloading situations, DE 10 2016 207 961 A1 discloses a system and method for a vehicle which blocks opening of a loading flap of a trunk of the vehicle when a maximum inclination of the vehicle detected by a sensor of the vehicle is exceeded. Furthermore, a position and / or a mass of an object arranged in the trunk of the vehicle is monitored by sensors by means of a camera and / or a weight sensor. In this way, it is prevented that an object arranged in the trunk after opening the loading flap automatically slides out of the trunk as a result of the inclination.

Other baggage-related problems can arise when loading a vehicle. For example, items of luggage to be transported can exceed a transport capacity of the vehicle.

DE 10 2011 016 641 A1 discloses a system and method for supporting loading of a vehicle. In the method, a camera of a mobile phone captures an image in advance of one or more objects to be transported. Based on the captured image or the captured images, a control unit of the vehicle predicts a size and / or a mass of each object and compares the predicted sizes and / or masses with a transport capacity of the vehicle, whereby stackability of the objects can also be taken into account. The result of the comparison is displayed on the mobile phone. The above method is based on predicted quantities prior to loading. Alternatively, the loading of the vehicle can also be monitored and controlled during this.

DE 10 2015 003 934 A1 discloses a system and method for supporting loading of a vehicle. In the method, a camera and / or a weighing device of the vehicle detect a size and / or a mass of one or more objects in a rear area of a vehicle. A display unit in the rear area shows the recorded sizes and / or masses of the objects and can also display a packing plan for the objects. In addition, the system enables the detection of slipping of objects while the vehicle is in motion and the output of a corresponding warning.

However, the above systems and methods do not offer sufficient accuracy when determining items of luggage in order to provide further functions of the vehicle in connection with items of luggage being transported. A determination of an item of luggage within the meaning of the invention is to be understood as determining a size and / or a mass of the item of luggage.

The invention is therefore based on the object of proposing an improved method for determining an item of luggage to be transported in a vehicle, which method determines a size and / or a mass of the item of luggage arranged in a trunk of the vehicle with high accuracy. Further objects of the invention are to provide a system for determining an item of luggage to be transported in a vehicle and to provide a vehicle.

The present invention relates to a method for determining an item of luggage transported in a vehicle, in which a camera directed into a trunk of a vehicle captures and provides an image of an item of luggage arranged in the trunk of the vehicle and a sensor of a chassis of the vehicle provides a Detects the value of a physical variable of the chassis that is dependent on a mass of the item of luggage and provides the detected value as a sensor signal. The chassis can be designed as an active chassis with a corresponding sensor system. For example, shock absorbers of the chassis can include sensors which measure a spring deflection of the respective shock absorber. The spring travel is a measure of a force acting on the shock absorber in a damping direction. A piece of luggage arranged in the trunk compresses one or more shock absorbers of the chassis due to its mass.

In the method according to the invention, a control unit of the vehicle determines a size and a mass of the item of luggage from the image provided and from a time curve of the sensor signal provided while the vehicle is driving. In short, the method is based on a dynamic determination of the mass of the item of luggage. The dynamic determination increases the accuracy of the determined mass, since static sources of error, which can occur, for example, as a result of a certain parking situation of the vehicle, are excluded. Thanks to a large number of different driving situations of the vehicle while driving, a statistical error in the determined mass of the item of luggage is reduced. The control device can be designed as a dedicated control device. Alternatively, a control device that is already present in the vehicle can be used.

In a preferred embodiment, the size and the mass of the item of luggage are determined by a deep neural network (Deep Neural Network, DNN) executed by the control device, which receives the provided image and the timing of the provided sensor signal as input. In other words, the size and mass of the item of luggage are determined by means of artificial intelligence (Kl, English: Artificial Intelligence, AI). The neural network has a large number of hidden layers that enable processing of complex inputs such as image information and time curves, and can in particular use a YOLO (You Only Look Once) architecture for real-time acquisition of Objects. During the productive use of the neural network, there is no further learning of the deep neural network, ie a level of knowledge of the deep neural network remains constant during the productive use.

In some embodiments, a preprocessor executed by the control unit divides a driving time of the vehicle into a plurality of time intervals, transforms the time curve of the provided sensor signal by calculating a plurality of values which each characterize the values of the sensor signal in exactly one time interval, and forms one of the characterizing ones Table that includes values and characterizes the journey of the vehicle. In this way, the large amount of values transmitted by the sensor while the vehicle is in motion can be compressed. A characterizing value can be, for example, a minimum value, maximum value or mean value of the time interval or a frequency spectrum of the time interval calculated using a Fourier transformation. A table as a two-dimensional arrangement of the characterizing values is particularly suitable for pattern recognition through the deep neural network.

In an advantageous embodiment, the deep neural network is monitored during a plurality of journeys of the vehicle using a plurality of predetermined pieces of luggage, each of which has a known size and a known mass and is arranged at a plurality of different positions in the trunk trained in advance. The deep neural network is trained prior to productive use by means of a so-called supervised learning process until an error probability when detecting the predetermined items of luggage falls below a predetermined value. In this way, with a correspondingly high training effort, almost any accuracy of the baggage item detection can be achieved. Particularly preferably, a plurality of cameras captures images of a plurality of items of luggage arranged in the trunk of the vehicle and provides them and the control device determines the size and mass of each item of luggage. Thanks to the plurality of cameras, mutual concealment of several pieces of luggage is largely excluded or at least reduced, so that the risk of overlooking a piece of luggage or incorrectly detecting a piece of luggage is reduced.

A display unit of the vehicle can display the size and the mass of each item of luggage arranged in the trunk. In this way, the display unit offers an occupant of the vehicle a good overview of all items of luggage arranged in the trunk. The display unit is preferably a human-machine interface (HMI) of an infotainment system of the vehicle.

The control device advantageously links each detected piece of luggage with a planned flight of an occupant of the vehicle and, based on the determined size and the determined mass of each piece of luggage, determines whether the piece of luggage is permitted as hand luggage or checked baggage, and displays and provides the authorization in the form of an authorization symbol a booking button on the display unit in case a piece of luggage is not permitted for the planned flight. This is an example of a more extensive service in connection with items of baggage, which is made possible thanks to the high accuracy of the baggage detection. It goes without saying that the booking button leads in the usual way to an airline's booking portal, on which, for example, additional hand luggage and / or checked baggage can be booked while driving. Alternatively, on the basis of the notification, the occupant can make a piece of luggage that has been determined to be inadmissible permissible by repacking or unpacking. The invention also relates to a system for determining an item of luggage transported in a vehicle, which system comprises a camera which is directed into a trunk of a vehicle and is configured to capture and provide an image of an item of luggage arranged in the trunk, and a chassis with a A sensor which is configured to detect a value of a physical variable of the chassis that is dependent on a mass of the item of luggage and to provide the value as a sensor signal. The invention is therefore suitable for any vehicle with an active chassis and at least one camera arranged in the trunk and directed into it. Accordingly, the system according to the invention can be easily installed in a large number of vehicles.

According to the invention, the system comprises a control device with a deep neural network which is configured to determine a size and a mass of the item of luggage from the image provided and a time curve of the sensor signal provided while the vehicle is traveling. The deep neural network enables dynamic detection of the transported items of luggage with a high level of accuracy, which also allows further services in connection with items of luggage transported in the vehicle.

The system is particularly preferably configured to carry out a method according to the invention.

Another subject matter of the invention is a vehicle with a system according to the invention for determining an item of luggage transported in a trunk of the vehicle. The vehicle has an expanded range of functions with regard to items of luggage transported in the trunk, which is associated with greater satisfaction for the occupants of the vehicle.

A significant advantage of the method according to the invention consists in the high accuracy with which a size and / or mass of one in one Trunk of a vehicle transported baggage is determined. The high level of accuracy enables an extended range of functions with further services in connection with transported luggage.

The invention is shown schematically on the basis of embodiments in the drawings and is described further with reference to the drawings. It shows:

Figure 1 in a schematic representation, a perspective partially

Rear view of a vehicle according to an embodiment of the invention;

Figure 2 in a block diagram of a control unit of a system according to a

Embodiment of the invention for determining an item of luggage in a trunk of the vehicle shown in Figure 1;

Figure 3 is a schematic representation of a partial plan view of a

Dashboard of the vehicle shown in Figure 1;

FIG. 4 shows, in a schematic representation, an enlarged detailed view of a display unit of the dashboard shown in FIG.

The figures are described comprehensively, the same components are assigned the same reference symbols.

FIG. 1 shows, in a schematic illustration, a perspective partial rear view of a vehicle 10 according to an embodiment of the invention. The vehicle 10 comprises a trunk that can be closed with a trunk lid 21 and a system 70 (see also FIG. 2) for determining two pieces of luggage 41, 42 transported in a trunk 20 of the vehicle 10, here a suitcase or a handbag. Of course, the system 70 also allows exactly one to be determined Luggage item or more than two items of luggage in the trunk 20. The vehicle 10 also includes a dashboard 50, which is not visible here (see FIG. 3).

The system 70 comprises three cameras 22 which are arranged in the trunk 20 and directed into the trunk 20. The cameras 22 are configured to record and provide images 71 (see FIG. 2) of the items of luggage 41, 42 arranged in the trunk 20. Of course, if necessary, a lighting device (not shown) can be provided for illuminating the items of luggage 41, 42 in the trunk 20.

Furthermore, the system 70 includes a chassis 30 designed as an active chassis with a plurality of shock absorbers 31 and a corresponding plurality of wheels 32 each assigned to a shock absorber 31. Each shock absorber 31 includes a sensor 31a which is configured to measure a value of one of a mass 61d , 62d (see FIG. 4) of the item of luggage 41, 42 depending on the physical variable of the chassis 30 and provide the value as a sensor signal 72.

However, the invention is not limited to three cameras and shock absorber sensors, but can also easily be implemented with a different number of cameras or other sensors of the chassis 30. A control device 73 (not visible here) also belongs to the system 70 (see FIG. 2).

FIG. 2 shows in a block diagram the control device 73 of the system 70 for determining an item of luggage 41, 42 in the trunk 20 of the vehicle 10 shown in FIG. 1. The control device 73 comprises a deep neural network (DNN) 73b, which have a YOLO architecture and is configured to determine sizes 61a, 62a and masses 61d, 62d of items of luggage 41, 42 from the provided images 71 and time profiles 72a of the provided sensor signals 72 while the vehicle 10 is traveling. Furthermore, the control device 73 comprises a preprocessor 73a, which is configured to subdivide a travel time of the vehicle 10 into a plurality of time intervals, to transform the time courses 72a of the provided sensor signals 72 by calculating a plurality of values which each characterize the values of the sensor signals 72 in exactly one time interval and from the characterizing values to form a table 72b comprising the characterizing values and characterizing the journey of the vehicle 10.

FIG. 3 shows a schematic representation of a partial top view of the dashboard 50 of the vehicle 10 shown in FIG. 1. The dashboard 50 comprises in the usual manner a steering wheel 51, one arranged behind the steering wheel 51 and, for a driver of the vehicle 10, through the steering wheel 51 visible through instrument display unit 52, a center console 53 and a arranged in an upper area of the center console 53 and designed as a human-machine interface (HMI) formed display unit 54 of an infotainment system of the vehicle 10. This embodiment of the dashboard 50 serves only as Example and does not limit the invention in any way, i. H. the invention can easily be combined with different configurations of the dashboard 50 which include a display unit.

The system 70 is configured to perform the following procedure.

To determine the items of luggage 41, 42 transported in the vehicle 10, the three cameras 22 directed into the trunk 20 of the vehicle 10 capture images 71 of the items of luggage 41, 42 arranged in the trunk 20 of the vehicle 10 and provide them. Furthermore, the sensors 31a of the chassis 30 of the vehicle 10 record values of a physical variable of the chassis 30 that is dependent on the masses 61d, 62d of the items of luggage 41, 42, for example a spring deflection of each shock absorber 31, and provide the recorded values as sensor signals 72. The images 71 and the sensor signals 72 can be provided in a customary manner via a vehicle bus (CAN). The control unit 73 of the vehicle 10 determines a size 61a, 62a and a mass 61d, 62d of each piece of luggage 41, 42 from the image 71 provided and from the time curves 72a of the sensor signals 72 provided while the vehicle 10 is traveling. The control unit 73 leads to this the deep neural network 73b, which receives the provided images 71 and - after preprocessing - the time profiles 72a of the provided sensor signals 72 as input.

Before evaluating the time courses 72a by the deep neural network 73, a preprocessor 73a, which is also executed by the control unit 73, divides a driving time of the vehicle 10 into a plurality of time intervals and transforms the time courses 72a of the provided sensor signals 72 by calculating a plurality of Values which each characterize the values of the sensor signals 72 in exactly one time interval. Then the preprocessor 73a forms a table 72b comprising the characterizing values and characterizing the journey of the vehicle 10 from the characterizing values, which table 72b serves as input for the deep neural network 73b.

Before the first productive use of the system 70, the deep neural network 73b is trained in a known manner during a plurality of journeys of the vehicle 10 on the basis of a plurality of predetermined items of luggage 41, 42 by means of monitored learning. The predetermined items of luggage 41, 42 used for training each have a known size 61a, 62a and a known mass 61d, 62d and are each arranged in different combinations and at a plurality of different positions in the trunk 20. It is noted that the knowledge level of the deep neural network 73b remains constant after the supervised learning is completed. During the productive use of the system 70, the system 70 does not learn any further.

One or each of the display units 52, 54 of the vehicle 10 displays the determined size 61a, 62a and the determined mass 61d, 62d of each item of luggage 41, 42 arranged in the trunk 20. FIG. 4 shows, in a schematic representation, an enlarged detailed view of a display unit 52, 54 of the dashboard 50 shown in FIG. 3. The control device 73 here links each detected piece of luggage 41, 42 with a planned flight 63a of an occupant of the vehicle 10. The display unit 52, 54 correspondingly shows a display content 60 which in each case comprises a baggage item 61, 62 for each baggage item 41, 42 in the trunk 20 as well as a flight entry 63 matching the planned flight 63a.

Each item of luggage entry has the determined size 61a, 62a, a booked type of posting 61b, 62b, an admissibility symbol 61c, 62c and the determined mass 61d, 62d of the respective luggage item 41, 42. The flight entry 63 has a planned flight number 63a and a booking button 63b, which is provided by the control device 73 if a piece of luggage 41, 42 is not permitted for the planned flight.

On the basis of the determined size 61a, 62a and the determined mass 61d, 62d of each piece of baggage 41, 42, the control device 73 determines the admissibility of each piece of luggage 41, 42 as hand luggage 61b, 62b or checked baggage and shows the admissibility in the form of an admissibility symbol 61c, 62c. In the example shown, the piece of luggage 41 in the form of a suitcase is not permitted as hand luggage, which is why the flight entry 63 has the booking button 63b. By actuating the booking button 63b, an occupant of the vehicle 10 arrives at a booking portal of the corresponding airline, where he can make a rebooking while the vehicle 10 is driving. Alternatively, the occupant can unpack or repack the inadmissible baggage item 41 in order to establish its admissibility as hand baggage.

The link shown here with a planned flight shows only an example of a more extensive service that is made possible by the system 70 according to the invention. Of course, the system 70 is also suitable for other types of further services, which require a determination of transported baggage items 41, 42 with a high degree of accuracy.

REFERENCE CHARACTERISTICS LIST:

10 vehicle

20 trunk

21 trunk lid

22 camera

30 chassis

31 shock absorbers

31a sensor

32 wheel

41 piece of luggage

42 piece of luggage

50 dashboard

51 steering wheel

52 Instrument display unit

53 center console

54 Infotainment display unit

60 Display content

61 Baggage entry

61a luggage size

61b Type of baggage check-in 61c Approval symbol

61 d luggage mass

62 Baggage entry

62a luggage size

62b Type of baggage check-in 62c Approval symbol 62d Baggage mass

63 flight entry

63a flight number

63b booking button

70 system

71 pic

72 sensor signal 72a passage of time

72b table

73 control unit

73a preprocessor

73b Deep neural network

74 edition

Claims

PATENT CLAIMS:

1. A method for determining an item of luggage (41, 42) transported in a vehicle (10), in which

- A camera directed into a trunk (20) of a vehicle (10)

(22) captures and provides an image (71) of an item of luggage (41, 42) arranged in the trunk (20) of the vehicle (10);

a sensor (31a) of a chassis (30) of the vehicle (10) detects a value of a physical variable of the chassis (30) dependent on a mass (61d, 62d) of the item of luggage (41, 42) and the detected value as a sensor signal (72) provides;

a control unit (73) of the vehicle (10) from the image provided

(71) and from a time curve (72a) of the provided sensor signal

(72) determines a size (61a, 62a) and a mass (61d, 62d) of the item of luggage (41, 42) while the vehicle (10) is traveling.

2. The method according to claim 1, wherein the size (61a, 62a) and the mass (61d, 62d) of the item of luggage (41, 42) from a by the control device (73) executed deep neural network (73b), which provided image (71) and the time curve (72a) of the provided sensor signal (72) receives as input, is determined.

3. The method according to claim 1 or claim 2, wherein a preprocessor (73a) executed by the control unit (73) divides a travel time of the vehicle (10) into a plurality of time intervals

Time curve (72a) of the provided sensor signal (72) by calculating a plurality of values which each characterize the values of the sensor signal (72) in exactly one time interval, transformed and a table (comprising the characterizing values and characterizing the journey of the vehicle (10)) 72b).

4. The method according to any one of claims 1 to 3, wherein the deep neural network (73b) during a plurality of journeys of the vehicle (10) based on a plurality of predetermined Luggage items (41, 42), each of which has a known size (61a, 62a) and a known mass (61d, 62d) and which are arranged in a plurality of different positions in the trunk (20), trained in advance by means of monitored learning becomes.

5. The method according to any one of claims 1 to 4, wherein a plurality of cameras (22) captures and provides images (71) of a plurality of items of luggage (41, 42) arranged in the trunk (20) of the vehicle (10) the control unit (73) determines the size (61a, 62a) and the mass (61d, 62d) of each piece of luggage (41, 42).

6. The method according to any one of claims 1 to 5, in which a display unit (52, 54) of the vehicle (10) the size (61a, 62a) and the mass (61d, 62d) of each item of luggage arranged in the trunk (20) (41, 42) indicates.

7. The method according to claim 6, wherein the control device (73) links each detected piece of luggage (41, 42) with a planned flight (63a) of an occupant of the vehicle (10) and based on the determined size (61a, 62a) and the determined mass (61 d, 62d) of each piece of baggage (41, 42) determines the admissibility of the baggage (41, 42) as hand luggage (61b, 62b) or checked baggage and indicates the admissibility in the form of an admissibility symbol (61c, 62c) and providing a booking button (63b) on the display unit (52, 54) if a piece of luggage (41, 42) is not permitted for the planned flight (63a).

8. System (70) for determining an item of luggage (41, 42) transported in a vehicle (10), which a camera (22) which is directed and configured in a trunk (20) of a vehicle (10), an image (71) to detect and provide an item of luggage (41, 42) arranged in the trunk (20), a chassis (30) with a sensor (31a) which is configured to record a value of a to detect the physical variable of the chassis (30) dependent on a mass (61d, 62d) of the item of luggage (41, 42) and to provide the value as a sensor signal (72), and a control device (73) with a deep neural network (73b) , which is configured, a size (61a, 62a) and a mass (61d, 62d) of the item of luggage from the provided image (71) and a time curve (72a) of the provided sensor signal (72) while the vehicle (10) is traveling (41, 42) includes. 9. The system of claim 8, which is configured to carry out a method according to any one of claims 1 to 7.

10. Vehicle (10), with a system (70) according to one of claims 8 or 9 for determining an item of luggage (41, 42) transported in a trunk (20) of the vehicle (10).