WO2019201537A1

WO2019201537A1 - Method and controller for denoting a person

Info

Publication number: WO2019201537A1
Application number: PCT/EP2019/056872
Authority: WO
Inventors: Bastian Volpert; Markus Birk; Stefan Bemetz; Carolin HELLER; Roland Rehtanz; Steffen BIEL
Original assignee: Zf Friedrichshafen Ag
Priority date: 2018-04-19
Filing date: 2019-03-19
Publication date: 2019-10-24
Also published as: DE102018205965A1

Abstract

A method for denoting a person (2) who is in the surroundings (4) of a vehicle, wherein the method involves, as steps, capturing (K1) at least one image using at least one camera (22, 24, 26) arranged on the vehicle, wherein the at least one image has image information in respect of the person (2), recognising (K2) the person (2) on the basis of the image information of the at least one image, associating (S4) an area of an at least partial representation of the surroundings (4) of the vehicle with the recognised person (2) and denoting (S5) the area associated with the person (2), and a controller for performing the method.

Description

Method and control device for identifying a person

Technical area

The invention relates to detecting a person in the vicinity of a vehicle by means of camera-based image acquisition.

State of the art

For realizing driver assistance systems environment awareness is used on a vehicle to detect environmental situations. This may allow the driver or the vehicle to detect conditions in the vehicle environment, to recognize and respond accordingly to hazards.

From DE 102015209857 A1 an emergency braking system for a vehicle is known, which is based on a sensor-based environment detection. The sensor-based environment detection serves to detect and recognize a pedestrian to initiate emergency braking. Furthermore, from DE 102017120498 A1 a further sensor-supported environment detection of a vehicle for perceiving road conditions is known. To do this, sensor data from several sensors is combined to create an environment view.

Presentation of the invention

The present invention relates to a method of identifying a person in the vicinity of a vehicle.

The marking of the person in the environment of the vehicle, for example a pedestrian, can be done visually. Such a visual identification can be done in an image-based representation of the vehicle environment or in an abstract representation of the vehicle environment. Under the vehicle environment can also be understood an environment around the vehicle. An image-based representation of the vehicle environment may include, for example, at least one region or section of a be captured all around picture or panorama picture. For example, an abstract representation may merely be an abstract representation of the detected person with respect to a vehicle coordinate system without an image-based representation.

As a matter of principle, a visual identification of the person as such can take place by means of any abstract information suitable for visual marking. The marking may be signaling, marking and / or highlighting the person or his environment in a vehicle surroundings display. The person may be identified in such a way temporarily, cyclically or permanently. By characterizing the person who is located in the vicinity of the vehicle, the person can be displayed to a driver or any other operator or user of the vehicle in an abstract manner. Thus, the method of warning against the person in the vicinity of the vehicle can be used.

The vehicle may be a non-autonomous vehicle, that is, a manned vehicle, or an autonomous vehicle, that is, a driverless vehicle. In the case of an autonomous vehicle, the method may be applied so that the designated person is automatically taken into account as additional information in the autonomous operation of the vehicle. Such consideration may include an intervention in the vehicle drive, in the vehicle steering or in a work task performed by the vehicle. Thus, for example, emergency braking can be initiated if the person is an obstacle that can not be bypassed, or a deviation from a driving route is performed in order to drive around the person or if a person is reposting a work task, for example, if the person blocks a work site to be approached. Such a response to a person identified may also be performed by a driver of a vehicle himself.

A method step is a capture of at least one image with at least one camera arranged on the vehicle, wherein the at least one image has image information relating to the person. The image capture may be capturing or generating a digital image of an area of the vehicle's surroundings corresponding to a viewing angle of the camera as its detection area. The viewing angle can be up to approximately 180 degrees with a wide-angle lens or a fisheye lens. In principle, the camera can be any imaging camera sensor, which can be, for example, an RGB camera, an infrared camera or a 3D camera. These can each be a panoramic camera or an omnidirectional camera. The image captured by the camera may have as image information pixels with corresponding RGB information, heat information or depth information. For continuous detection of the person, each of the mentioned cameras can also be designed as a video camera.

Another method step is recognizing the person based on the image information of the at least one image.

Recognition of the person in the image information can be based on known image processing algorithms, for example this can be done by means of segmentation algorithms or matching algorithms. As a segmentation algorithm, semantic segmentation for person recognition may be applied in the image. As a matching algorithm, template matching can be used for this purpose.

Yet another method step is to assign a region of an at least partial representation of the surroundings of the vehicle to the recognized person and to identify the area assigned to the person.

The area in the environment representation can be defined by an outline, a contour or a silhouette of the person represented himself. The area in the environment representation may also be an area adjacent to the person depicted in the representation. Such an adjacent area may be adjacent to or spaced from the representation of the person. As such, the representation of the environment as such can basically be any real or abstract visualization of the vehicle environment, which is displayed on a visual display device, for example on a screen, as Still image or as a video, can be visualized. If it is an autonomous vehicle, the representation can also be further processed in an in-vehicle process without a visualization made visible to the user of the vehicle. The representation can also be visualized by a user of the autonomous vehicle and remote from the vehicle.

In the context of the invention, therefore, a person detected with a camera or a camera system can be visually highlighted, in addition to a mere visual reproduction thereof, by an additional visual identification of the person in order to visually warn the driver or user. An advantageous effect of such a warning system can be the reduction of personal accidents with the vehicle.

The invention is based on the finding that a person recognition on the basis of a real image, which is thus incumbent only on the driver or viewer, is disadvantageous, since it can lead to dangerous situations when operating a vehicle. A concept of the invention therefore consists in the fact that both the person recognition and a person identification are automatically provided on a vehicle, in order to immediately and automatically recognize perilous danger situations.

This is advantageous since the invention can thus be used to warn directly and essentially only automatically when a person is in the vehicle environment, but not when another object is in the vehicle environment. As a result, the emergence of a hazardous situation for persons can be minimized. This is especially true when a person abruptly moves toward the vehicle in the vehicle environment or when the vehicle quickly approaches a non-moving person.

An embodiment has, as a further method step, detection of at least one point cloud with at least one radar sensor arranged on the vehicle, wherein the at least one point cloud has at least one point with respect to the person. The radar sensor may be a one-dimensionally or punctually sensing sensor, a two-dimensional or fan-like sensing sensor or a three-dimensional or scanning sensor. The point cloud can be a corresponding dimension in the one-dimensional case may already consist of one point. With the radar sensor, a contour of the person can thus be detected discreetly with individual points, with a horizontally oriented fan-like radar sensor a transverse plane is detected as a contour of the person and with a vertically oriented fan-like radar sensor a sagittal plane or a frontal plane can be detected as a contour of the person. The person detection by means of a radar sensor is advantageous due to its spatially and temporally high resolution.

The embodiment has, as additional method steps, a determination of a distance of the recognized person from the vehicle based on the at least one point of the at least one point cloud and a marking of the area assigned to the person as a function of the determined distance. The distance determination may be based on the points detected on the person, for example in the center of gravity of these points, with respect to any point of the vehicle. Hereby, the distance between the person and the vehicle can be determined with a radar-based distance measurement. In a characterization of the person, the specific distance can be taken into account so that the marking is scaled as a function of the distance. This can be done by means of a distance-dependent coloring of the marking, for example based on a traffic light system. A person who is in a danger area around the vehicle, in the direction of travel or in the working radius can be colored accordingly. A distance-dependent identification and thus a fusion of radar data and image data is advantageous because it can thus be warned in front of people, which may represent a danger due to their relative location to the moving vehicle.

Based on the at least one point of the point cloud with respect to the person and their position relative to the vehicle in a vehicle coordinate system can be determined. For this purpose, the position and orientation of the radar sensor can be determined in the vehicle coordinate system or determined by means of calibration.

As an alternative or in addition to the radar-based distance definition or position determination, a distance to a central distance can be determined on the basis of the camera-based image acquisition. detected image processing or person whose position can be determined. For example, methods of stereophotogrammetry can be used for this purpose, it being possible to use at least two images of the same or different cameras from different image acquisition locations. In addition to image-based person recognition, a person can also be identified based on the radar-based point cloud detection. For this purpose, for example, motion patterns of individual points in temporally offset measured point clouds can be taken into account.

Alternatively or additionally, the embodiment may, as additional method steps, comprise determining a directional angle of the recognized person with respect to a vehicle-specific reference direction based on the at least one point of the at least one point cloud and marking the area associated with the person as a function of the determined direction angle. The vehicle-specific reference direction can be determined for example by the vehicle longitudinal axis. Such a consideration of a relative orientation of the recognized person with respect to the vehicle may be advantageous, since such a person can be identified and warned of her, for example, if she is in an angle or sector related to the direction of travel.

A further embodiment has, as a further method step, a visualization of the at least partial representation of the environment of the vehicle based on the at least one captured image. The visualization of the vehicle environment may be based on a single image of a camera or on a combination of several images of one or more cameras. Multiple images can be stitched together to form a bird's-eye view image, a panoramic image or an all-around image of the vehicle environment using stitching and projection. Such a pictorial representation has the advantage that the environment of the vehicle can be reproduced in real and a person identified in it can be placed in an intuitive spatial relation to the vehicle environment.

In a further embodiment, the at least partial representation of the surroundings of the vehicle is an image of the surroundings of the vehicle from a bird's-eye view. The environment of the vehicle or a part of the same can basically in any Perspective, for example, in a side view, in a perspective view or in a plan view are shown. An environment representation from a bird's-eye view can be advantageous, since with it the vehicle itself can be represented and thus spatial relations between the person identified in the representation and the vehicle can be recognized. The bird's eye view may be based on a central projection or a spherical projection of at least one image on a presentation plane. Such a "bowl" presentation has the advantage that almost the entire vehicle environment can be visualized in an image. Alternatively or additionally, a panoramic image can also be generated with a spherical projection.

In a further embodiment, the assigned area is formed based on a contour of the recognized person. The marking of the person can thus refer to a line-like or area-like marking based on the contour, that is to say a marking based on an outline of the person in its pictorial representation. The visible in a pictorial representation of the person and / or the vehicle environment contour can thus be marked abstract in any representation perspective and thus highlighted in the representation visually. For example, the contour or the area enclosed by it can be colored. The contour may correspond to the silhouette of the person or have a symbolic contour, which may differ from the actual silhouette.

In a further embodiment, the assigned area has a segment with respect to a foot point of the recognized person in the at least partial representation of the surroundings of the vehicle. The segment may be any geometric shape suitable for identifying the person. For example, the segment may be bar-shaped, circular or triangular. A corresponding segment can be arranged to identify the person in the environment representation so that it includes a projection point of the person on a presentation level. The segment may also be located adjacent or spaced from the projection point in the illustration. Such a segment-like representation may be advantageous because the person can not be covered as in a contour-based areal labeling, but can be surrounded by the segment. this has the effect that a real representation of the person and an abstract marking of the same can be combined in one presentation. For example, the person can be covered or deposited with a partially transparent colored area segment for this purpose. Full coverage of the person, on the other hand, has the beneficial effect of generalizing a detailed environment representation.

A further embodiment has, as a further method step, a marking of the area assigned to the person with a visual warning signal. The warning signal can be a visual warning signal which has at least one coloring of the assigned area and its visualization or display. The coloring can be done with a signal color, for example red, yellow or green. The warning signal can be visualized temporarily, ie flashing, cyclically, that is flashing, or permanently, ie permanently lit.

Dyeing of the area assigned to the person can be effected as a function of the distance, that is to say with a choice of color as a function of the distance between the vehicle and the person. The coloring can also be done depending on the direction angle of the person with respect to the vehicle-specific reference direction. By coloring the area assigned to the person with a signal color, for example red, it is thus advantageously possible to warn against persons who are in a vicinity of the vehicle, in a direction of travel or in a working radius of the vehicle. By contrast, the area associated with the person can be dyed with a different signal color, for example green, if there is no danger from this person, that is, if he is not in the vicinity of the vehicle, not in the direction of travel or outside the working radius of the vehicle , The close range can be a few meters. The direction of travel may be determined based on a route or movement trajectory of the vehicle.

A further embodiment has, as a further method step, determining a relative position of the person with respect to the vehicle based on at least one of the steps of detection, that is, based on the step of capturing an image with the camera and / or based on the step of detecting a point cloud with the radar sensor, on. The relative position can be a relative position of the person with respect to the vehicle and / or a relative direction or angle between a vehicle-specific reference direction and the person. The consideration of the relative position is advantageous since the identification of the person or of the area assigned to her can be carried out on the basis of a vehicle-specific coordinate system detached from a superordinate coordinate system. In this way, it is possible to make a self-sufficient warning in a moving vehicle and in the near real-time to a person in their environment.

The embodiment has, as an additional method step, a marking of the area assigned to the person as a function of the determined relative position. A person identification or a marking of a person assigned area can thus take place in dependence of the position of the person in a vehicle coordinate system. Hazardous areas can thus be defined in the coordinate system of the vehicle, with marking only taking place when the person is in such a danger area. An advantageous effect of this may be that, independently of a current absolute position of the vehicle, an area for a vehicle-mounted tool can be defined, in which a person can be identified.

A further embodiment has, as a further method step, an indication of the determined relative position in the representation of the environment of the vehicle. In addition to identifying the area associated with the person, in the illustration, a distance to a person, a distance between the vehicle and the person, a directional angle to the person can be displayed. In addition, these parameters can also be used as the basis for operating an autonomous vehicle without displaying it. This embodiment has the advantage that additional visual information about persons in the vicinity of the vehicle can be provided to a driver or user.

A further embodiment has, as a further method step, detection of a multiplicity of point clouds with at least one radar sensor arranged on the vehicle, the point clouds each having at least one point with respect to the person. The acquisition of the point clouds can be synchronized with several sensors. chron. Thus, a large area of the vehicle environment can be detected from a single location of the vehicle. This has the advantage that, for example, fan-like scanning radar sensors can be combined in such a way that a large number of persons can be detected synchronously in different vertical and / or horizontal detection planes. Alternatively or additionally, the detection of the point clouds with one or more sensors can be carried out continuously, that is to say temporally successively, at specific time intervals. Thus, points can be recorded with respect to the same person and thus the person tracked or tracked. This has the advantageous effect that in addition to the location of the person and their movement or kinematics can be determined. This in turn has the advantage that even a moving person can be marked continuously in the near real time.

The embodiment has, as additional method steps, determining a movement behavior of the person based on the detected point clouds, wherein the movement behavior comprises at least one of resting the person, moving the person, a direction of movement of the person, a movement speed of the person and a change of one these movement characteristics, and further identifying the area assigned to the person as a function of the determined movement behavior. The marking of the area assigned to the person in dependence of one or more of the named types can thus visualize the movement of the person, that is, reproduce it. The movement may be related to a location or movement of the vehicle, and a danger situation derived therefrom. For example, it is thus possible to be warned of a person moving towards the vehicle with a signal color, for example red, while a person moving away from the vehicle can not be identified or can be highlighted with a different signal color, for example green. The consideration of such a movement behavior when marking a person has the advantage that a dynamically changing environment can be monitored and can be explicitly warned in front of this moving people. A further embodiment has, as a further method step, an indication of the determined movement behavior in the representation of the surroundings of the vehicle. In addition to identifying the area assigned to the person, one or more movement behavior of the person can be displayed in the display. In addition, these motion parameters can also be used as the basis for operating an autonomous vehicle without displaying them. This embodiment has the advantage that additional visual information on the movement behavior of persons in the vicinity of the vehicle can be provided to a driver or user.

A further embodiment has, as further method steps, a classification of the recognized person as a person moving or as a non-moving person and a marking of the recognized person as a function of the classification. For example, a person staying in the vehicle environment can then be identified when it is in motion. An identification of the area assigned to the person can alternatively or additionally be carried out on each of the mentioned movement characteristics alone or, for example, on the basis of a corresponding truth table. Classification may also be classifying, and classifying may also be understood as classifying. Identifying a person as a function of a classification is advantageous, as it can only be used to warn persons of artificial intelligence when they pose a danger. A driver or user remains so constantly aware of hazards, as an unnecessary warning of non-existent hazards can be minimized. Furthermore, deliberate ignoring of a perceived lack of danger by the driver or user can be minimized.

A further embodiment has, as a further method step, outputting of a warning signal as a reaction to a recognized person. The issuing of the warning signal can be effected as a function of the classification according to the movement behavior, that is, if the person has been classified as moved or as not moved. As an alternative or in addition to the method step of marking the area associated with the person with a visual warning signal, such a warning signal can be separately visualized or output. This can basically any other Warning signal be visualized or issued to warn the driver or user in front of a person in the vehicle environment. For this purpose, any medium, such as a screen, a speaker or a display can be used. The warning signal may be a visual warning signal, for example a light signal from a beacon. The warning signal can also be an audible warning signal, for example a sound from a horn. Furthermore, the warning signal may also be a haptic warning signal, for example triggered by a vibration of a steering wheel or control lever of the vehicle. One or more such warning signals reinforce advantageously in addition to the identification of the person a warning of a dangerous situation.

A further embodiment has, as a further method step, intervention in the drive train of the vehicle as a reaction to a recognized person. If a person is marked, possibly as a function of one of the classifications mentioned, intervention can be made in the vehicle drive or drive train of the vehicle. An autonomously driving vehicle can thus be braked automatically or diverted around the person as an obstacle. Further, an emergency stop or emergency stop may be performed as a result of the designated person to avoid collision with the person. Especially for the operation of autonomous vehicles, such a response can be an improvement in the safety of people staying in the vicinity of the vehicle.

A further embodiment has, as a further method step, detection of an object which is not a person based on the image information of the at least one image, wherein the at least one image has image information relating to the object. In principle, each method step of the invention or one of its embodiments can also be applied to any object recognizable in an image which is not a person. In addition to warning people, it is also possible to warn against individually defined objects or, alternatively, against essentially all objects in the vehicle environment. This has the advantage that collisions with different objects and persons can be minimized, whereby a degree of automation of the vehicle can be increased. As an additional method step, the embodiment has an allocation of a region of the visualized, at least partial, representation of the environment of the vehicle to the object recognized in the image information and a marking of the region assigned to the object. In principle, each method step of the invention or one of its embodiments can furthermore also be applied to a recognition and identification of the object, which is not a person. In addition to identifying persons, individually defined objects or, alternatively, substantially all objects in the vehicle environment can also be identified. For this purpose, an object can also be assigned to one of the areas mentioned in connection with the marking of a person. This has the advantage that the vehicle environment can be monitored with regard to a large number of obstacles.

A further embodiment has, as further method steps, a classification as a recognized person or as a recognized object and a marking of the area assigned to the person and of the area assigned to the object as a function of the classification. Thus, for example, an identification step may take place in one of the mentioned types if it concerns one person and another labeling step takes place in another of the named types if the object is or is omitted. The classification of this embodiment may be combined with the classification based on the mentioned motion characteristics. Identifying a person as a function of one of these classifications is advantageous because it can further minimize unnecessary warning of non-existing hazards.

The present invention further relates to a control device which is adapted to carry out the method steps of the invention or the method steps of one or more of its embodiments. The control device may be a control device with which such, programmed method steps can be controlled or regulated on the vehicle and / or in a user environment of a user of an autonomous vehicle.

The present invention also relates to a vehicle having such a control device. The vehicle may be autonomous, semi-autonomous or non-autonomous Be a vehicle. The vehicle may be any vehicle designed to transport people and / or goods.

The vehicle can basically be any manned or unmanned transport vehicle, for example from the field of logistics, or any work vehicle. In one embodiment, the vehicle is an autonomous industrial truck. Such a material handling vehicle may for example be a forklift. When operating a forklift, the avoidance of accidents at work with people who are in the vicinity of the forklift, of particular importance, since the work safety in the environment of the forklift can be increased.

In a further embodiment, the at least one camera arranged on the vehicle is arranged at a rear region of the industrial truck. The arrangement of sensors in general and the camera or the radar sensor in particular in the rear area of an industrial truck, for example, the forklift, is advantageous because there are stationary vehicle parts in the rear area. An alternative or additional arrangement of a sensor on a front area, for example on a lift mast of a fork-lift truck, or a side area can also be provided. In these areas, however, moving parts of the vehicle, such as the fork of the forklift, which can make a fixed positioning difficult. In contrast, a lateral arrangement of at least one camera or a radar sensor can produce a corresponding lateral detection area.

Brief description of the drawings

FIG. 1 shows a flowchart with method steps of an exemplary embodiment of a method for identifying a person who is located in the environment of a vehicle.

Figure 2 shows an embodiment of a vehicle in a perspective

View. FIG. 3 shows exemplary embodiments of a marking of a person located in the surroundings of the vehicle for further explanation of the method according to FIG. 1 and of the vehicle according to FIG. 2.

FIG. 4 shows further exemplary embodiments of a marking in the

Person surrounding the vehicle for further explanation of the method according to FIG. 1 and the vehicle according to FIG. 2.

Detailed description of embodiments

In FIG. 1, method steps S1, S2, S3, S4, S5 are shown in a time sequence for carrying out the method for identifying a person 2 who is in the environment 4 of a forklift 10 as a vehicle 2 (see the following figures).

In a first step S1, a data acquisition with a plurality of cameras 22, 24, 26 and a plurality of radar sensors 32, 34, 36, which are arranged on the forklift 10. In a first sub-step K1, an image of the surroundings 4 of the forklift 10 is recorded with each of the cameras 22, 24, 26. At least one of these images from one of the cameras 22, 24, 26 has image information relating to the person 2. In a second substep R1, each of the radar sensors 32, 34, 36 detects a point cloud with respect to the environment 4 of the forklift 10, wherein in at least one of these point clouds a point with respect to the person 2 is present.

After step S1, in a second step S2, a person recognition is carried out based on the data acquired in step S1 with the cameras 22, 24, 26 or with the radar sensors 32, 34, 36. In a first sub-step K2, the person 2 is automatically detected by means of image processing based on image information of at least one image of the cameras 22, 24, 26. In a second substep R2, based on at least one point cloud of a radar sensor 32, 34, 36, a distance or a movement behavior of the person 2 identified in substep K2 is determined. For this purpose, the image information of at least one of the images captured with the cameras 22, 24, 26 and at least one of the point clouds, which are associated with the radar Sensors 32, 34, 36 have been detected, are transformed by means of a transformation in a common coordinate system, to establish a spatial relationship between the image information and the point clouds. Through such a linkage by means of transformation, image information which comprises at least one pixel or its value or values and one point of a point cloud are assigned to the same person.

In a further step S3, the environment 4 of the forklift 10 is visualized based on the images captured by the cameras 22, 24, 26. For this purpose, the images of the cameras 22, 24, 26 are combined by stitching to form an overall image of the environment 4 of the forklift 10, this image 5 shows the environment 4 from a bird's eye view. In the image 5 of the environment 4 of the forklift 10 is to be seen in the center, which includes the center of a central projection of Figure 5 underlying.

After step S2 or step S3, in a fourth step S4, an area of the image 5 of the environment 4 of the forklift 10 is assigned to a person 2 identified in the sub-step K2. The environment representation performed in step S3 and the area allocation performed in step S4 will be explained in more detail with reference to FIGS. 3 and 4.

After step S4, in a fifth step S5, the person 2 identified in sub-step K2 is signaled by identifying the area assigned to her in step S4. The personal identification carried out in step S5 will also be explained in more detail with reference to FIGS. 3 and 4.

FIG. 2 shows a forklift 10 as a vehicle with a control unit 50 for carrying out the method steps S1, S2, S3, S4, S5 according to FIG.

At a rear portion 1 1 of the forklift 10, three cameras 22, 24, 26 and three radar sensors 32, 34, 36 are arranged. The cameras 22, 24, 26 are arranged on a roof 12 of the forklift 10, with the first camera 22 as a rearview camera, the second camera 24 as a camera looking laterally to the left and the third camera 26 as a camera looking laterally to the right with corresponding viewing angles are set back, left and right. The cameras 22, 24, 26 are designed as wide-angle cameras, wherein their respective viewing angle is up to about 180 degrees. The first radar sensor 32 and the second radar sensor 34 are each arranged on a lateral spar 13 of an open cab 14 of the forklift 10. The driver's cab 14 is bounded at the top by the roof 12 and to the rear by the two bars 13.

The radar sensors 32, 34 are fan-like scanning radar sensors 32, 34, which form a scanning plane horizontally, obliquely or vertically at the level of their respective mounting location on the bars 13. In this embodiment, the radar sensors 32, 34 are mounted at approximately the same height on the bars 13, whereby they detect point clouds in a common scan plane, for example in a horizontal plane.

Another radar sensor 36 is disposed below the other radar sensors 32, 34 on the chassis 15 of the forklift 10 in the rear region 1 1. The further radar sensor 36 detects a point cloud in a scanning plane spanned by the other two radar sensors 32, 34. The Scakes are here parallel to each other, for example, horizontally aligned.

In the cameras 22, 24, 26 and the radar sensors 32, 34, 36 are each identical cameras or sensors.

In FIG. 3, persons 2 staying in the environment 4 of the forklift 10 are identified in a so-called top-view view. An area assigned to persons 2 is defined in each case by their illustrated contour 3. The contour 3 is filled to identify the person each with a signal color, for example, red. Thus, the person 2 is colored within its contour 3. The contour 3 of the person 2 is extracted from a background image visualized representation of the environment 4 of the forklift 1 1.

The image 5 of the environment 4 of the forklift 10, which shows the background 4 from a bird's-eye view including the forklift 10 as a background, is made of individual images of the cameras 22, 24, 26 generated by stitching and central projection. As a result, substantially the entire environment 4 of the forklift 1 1 is visualized. For the identification of the persons 2 in the environment 4 of the forklift 10, the control unit 50 is used, which is also designed as a central evaluation unit 40 for generating the image 5.

In FIG. 3, beam segments 6 are visualized as areas assigned to the person 2 as a further exemplary embodiment. Depending on the distance of a person 2 from the forklift 10, the beam segments 6 are dyed with different signal colors, for example red for close persons 2, yellow for persons 2 further away and green for persons 2 located further away. Depending on the distance between the forklift 10 and a person 2 or depending on a direction angle in the forklift 10 to a person 2 starting from the direction of travel or the longitudinal axis of the forklift 1 1 per person 2 are a plurality of respective beam segments 6 at a foot of each person 2 projected and so shown in Figure 5 of the environment of the forklift 1 1. The distances are calculated from the point clouds of the radar sensors 32, 34, 36 based on the points of the point clouds assigned to a person 2 by means of the evaluation unit 40.

The marking of a person 2 by means of one or more beam segments 6 is further shown on an object 9, which is not a person 2, by means of evaluation of the point clouds generated by the radar sensors 32, 34, 36 by the evaluation unit 40.

The embodiments shown in FIG. 4 of identifying a person 2 located in the environment 4 of the forklift 10 initially differ from the exemplary embodiments shown in FIG. 3 in that an image 5 of the environment 4 of the forklift 10 is made of an abstract background and of one of the Cameras 22, 24, 26 extracted by stitching of the evaluation unit 40 generated panoramic image persons 2 is formed. The embodiments may therefore be referred to as simplified. Another difference is that for identifying the persons 2 not their contour 3 is used and colored, but that each person 2 is assigned a circle segment 7, which is shown below her feet. The center of each circle segment refers to the position or representation of the forklift 10. The respective circle segment 7 is colored in dependence of the distance between a person 2 and the forklift 10, which is determined radar-based, by means of a signal color. The associated circle segment 7 of a close to the forklift 11 resident person 2 is dyed, for example, with red, a circle segment of a person further away is colored, for example, yellow and a circle segment of an even further away person 2 is dyed, for example, with green.

Correspondingly colored line segments 8 represent a standard for the distance-dependent coloration mentioned by way of example. The line segments 8 thus also define a region assigned to a person 2, which area is colored according to the distance.

The exemplary embodiments shown in FIGS. 3 and 4 explain how a person 2 in the environment 4 of a forklift 10 is indicated to his or her forklift 10 by means of abstract marking of an associated area of an environment representation with a signal color, and how he thus appears in front of the person 2 is warned.

reference numeral

person

contour

environment

Picture of the environment

bar segment

circular segment

line segment

object

10 forklifts

11 rear area

12 roof

13 spar

14 driver's cab

15 chassis

22 first camera

24 second camera

26 third camera

32 first radar sensor

34 second radar sensor

36 third radar sensor

40 evaluation unit

50 control unit

51 Data acquisition

K1 image capture

R1 point cloud detection

52 person recognition

K2 image-based person recognition

R2 point cloud based distance determination

53 Environment representation

54 Area assignment

55 person identification

Claims

claims

1 . Method for identifying a person (2) in the environment (4) of a vehicle, comprising the steps

Detecting (K1) at least one image with at least one camera (22, 24, 26) arranged on the vehicle, the at least one image having image information relating to the person (2),

Recognizing (K2) the person (2) based on the image information of the at least one image,

Assigning (S4) a region of an at least partial representation of the environment (4) of the vehicle to the recognized person (2) and

Flag (S5) the area assigned to the person (2).

2. The method according to claim 1, with

Detecting (R1) at least one point cloud with at least one radar sensor (32, 34, 36) arranged on the vehicle, wherein the at least one point cloud has at least one point with respect to the person and

Determining (R2) a distance of the recognized person (2) from the vehicle based on the at least one point of the at least one point cloud and

Marking (S5) the area assigned to the person (2) as a function of the determined distance.

3. The method according to claim 1 or 2, with

Visualizing (S3) the at least partial representation of the environment (4) of the vehicle based on the at least one captured image.

4. The method according to any one of the preceding claims, wherein

the at least partial representation of the environment (4) of the vehicle is an image (5) of the surroundings (4) of the vehicle from a bird's-eye view.

5. The method according to any one of the preceding claims, wherein the assigned area is formed based on a contour (3) of the recognized person (2).

6. The method according to any one of the preceding claims, wherein

the assigned area comprises a segment (6, 7, 8) with respect to a foot point of the recognized person (2) in the at least partial representation of the environment (4) of the vehicle.

7. The method according to any one of the preceding claims, with

Identifying (S5) the area associated with the person (2) with a visual warning signal.

8. The method according to any one of the preceding claims, with

Determining a relative position of the person (2) with respect to the vehicle based on at least one of the steps of detecting (K1, R1) and

Marking (S5) the area assigned to the person (2) as a function of the determined relative position.

9. The method according to claim 8, with

Displaying the determined relative position in the representation of the environment (4) of the vehicle.

10. The method according to any one of the preceding claims, with

Detecting (R1) a plurality of point clouds with at least one radar sensor (32, 34, 36) arranged on the vehicle, the point clouds each having at least one point with respect to the person,

Determining a movement behavior of the person (2) based on the detected point clouds, wherein the movement behavior comprises at least one of resting the person (2), moving the person (2), a direction of movement of the person (2), a movement speed of the person (2) and a change of one of these movement characteristics and

Marking (S5) the area assigned to the person (2) as a function of the determined movement behavior.

1 1. The method of claim 10, with

Display of the determined movement behavior in the representation of the environment (4) of the vehicle.

12. The method according to any one of the preceding claims, with

Performing a classification of the recognized person (2) as a moving one

Person or as a non-moving person and

Identification (S5) of the recognized person (2) depending on the classification.

13. The method according to any one of the preceding claims, with

Issuing a warning signal as a response to a recognized person (2).

14. The method according to any one of the preceding claims, with

Engaging the drive train of the vehicle as a response to a recognized person (2).

15. The method according to any one of the preceding claims, with

Recognizing an object (9) which is not a person (2) based on the image information of the at least one image, the at least one image having image information relating to the object (9),

Assigning (S4) a region of the visualized at least partial representation of the environment (4) of the vehicle to the object (9) recognized in the image information and identifying (S5) the region associated with the object (9).

16. The method according to claim 15,

marked by

Making a classification as a recognized person (2) or as a recognized object (9) and

Identifying (S5) the area associated with the person (2) and the area associated with the object (9) depending on the classification.

17. Control device which is adapted to perform the method steps according to one of the preceding claims.

18. Vehicle with a control device (50) according to claim 17.

19. Vehicle according to claim 18, wherein

the vehicle is an autonomous industrial truck.

20. Vehicle according to claim 18 or 19, wherein

the at least one camera (22, 24, 26) arranged on the vehicle is arranged on a rear region (11) of the industrial truck.