WO2019091725A1

WO2019091725A1 - Method and display device for guiding a working machine

Info

Publication number: WO2019091725A1
Application number: PCT/EP2018/078323
Authority: WO
Inventors: Marcus Hiemer; Mark Mohr; Mauro Cesar ZANELLA
Original assignee: Zf Friedrichshafen Ag
Priority date: 2017-11-10
Filing date: 2018-10-17
Publication date: 2019-05-16
Also published as: DE102017220005A1

Abstract

The invention relates to a method and a display device for guiding a working machine along a route, in which a movement of the working machine along said route is controlled, involving: acquiring image information using at least one sensor, and visualising the image information with the display device, wherein the image information is information relating to the surroundings of the working machine, and said working machine can be controlled on the basis of the image information.

Description

Method and display device for guiding a work machine

Known methods for guiding a work machine along a route include controlling a movement of the work machine along the route.

Guiding a work machine may be a user guided machine guidance. The operation of the work machine may also include or be understood as controlling or regulating the work machine by a user.

In principle, a work machine can be any vehicle that can be used for a work, for example an agricultural machine.

A route can also be defined as a path or a route along which the work machine is to be moved. The route may include a two- or three-dimensional trajectory with or without altitude information. A movement of the work machine can also be referred to as a drive with the work machine.

Known display devices for guiding a work machine along a route are configured for controlling a movement of the work machine along the route.

In Fig. 1, a view of such a known display device is shown schematically. The display device can show in perspective a route 101 to be traveled on a model of a work site 102. The route 101 to be traveled can also be referred to as an artificial route. Furthermore, the working machine 1 and optionally a working device 103, which can be mounted on the working machine 1, are shown symbolically. A terrain 104 traversed by the work machine 1 or work implement 103 according to the route 101 to be traveled can also be displayed.

From WO 2010/083949 A1 a tracking system is known with which a vehicle is moved along a predetermined lane. To drive off the lane and perform work, a driver of the vehicle is presented with the predetermined lane and the position of the vehicle. Solutions are provided for improving known methods and display devices for guiding a work machine.

Such a solution for a method for guiding a work machine consists in acquiring image information with at least one sensor, visualizing the image information with a display device, wherein the image information has information about the environment around the work machine and wherein the work machine is controllable with the image information. Image information can be image data.

Capturing image information may also be defined as detecting or capturing image information.

As image information, basically any visualizable information can be understood, which can be represented as a picture in a two-dimensional image coordinate system. Image information can basically be captured by photogrammetry, remote sensing or scanning. Image information may also come from an external source, such as a satellite image or a drone-captured image.

In principle, any sensor for capturing image information can be provided. As a sensor, a camera may be provided. The camera can produce a photographic image of the environment as image information. Alternatively or additionally, a temperature camera which generates a temperature image as image information or a 3D camera which generates a depth image as image information may be provided as the sensor. With such cameras, areal information about the environment can be captured pixel-like and simultaneously.

Visualizing image information can also be understood as displaying, displaying or displaying image information. At least one coordinate transformation, in particular a transformation from a global coordinate system into a local image coordinate system or from a camera system into an image system, may be necessary for the visualization.

The visualization can also be defined as displaying, displaying or visualizing.

A display device may be, for example, a display, a monitor or a screen. The display device may be portable, such as a tablet, or fixed to the be connected. The display device can be installed in the work machine, in particular in a guide post of the work machine.

Information about the environment around the work machine may include information about visible light. It is possible to visualize a picture of the environment around the working machine in the visible spectrum. An image can be a photo. Alternatively or additionally, an infrared image or a depth image of the surroundings of the working machine can be detected. In an infrared image, information about the environment may include infrared radiation. In a depth image, distances from the work machine to objects in the environment can be represented as pixel-like or voxel-like.

The environment around the work machine can also be defined as a scene or scenery. Furthermore, the surroundings may also be the periphery or the environment of the working machine.

The solutions provided are based on the knowledge that an automated, in particular a positionally accurate, guiding a work machine along a route to be processed can not be sufficient to optimally execute corresponding work tasks along the route.

A basic idea may be seen to provide real environmental information to a user visually in real time while working with a work machine along a work route to incorporate actual conditions of the environment into the work to be performed. Environment information can thus be used as a real-time representation of the environment around the work machine for controlling the same, wherein the rules can be user-managed or autonomous.

A further basic idea can be seen in that the movement of a work machine that can already be guided or regulated by means of satellite navigation can be adapted by adding real-time image acquisition of the work environment to the actual position of objects to be processed, as an additional functionality for regulating the movement behavior of the work machine. It can be deliberately deviated from a predetermined target route to improve the execution of the task object specific For example, information about the field, plants or crop during the work to be performed along a working route can be taken into account in real time with an agricultural machine in field work to be done along a route of the agricultural machine. A tractor, a combine harvester or a harvester can thus also be controlled and moved taking into account actual conditions, for example an existing crop or cereal cutting edge or the position of a swath.

The work machine can (partially) automatically drive along the working route as a given movement trajectory. The basis for the trajectory can be global coordinate data, for example GPS coordinates. Also, a route previously traveled by a human may be repeatedly traveled, and this may be referred to as a "playback function" for moving the work machine.

One embodiment consists in rules of movement of the work machine with the image information and / or with a working position of the work machine. The work machine can be regulated based on the image information in their lateral dynamics and / or longitudinal dynamics. The working machine can optionally be regulated in terms of its working position in its longitudinal dynamics and based on the image information in their lateral dynamics.

In principle, only one camera can be provided as the sensor. The image information may visualize a region of the environment detected or selected by the camera. To capture the widest possible area of the environment, the camera can have a wide-angle lens or a fisheye lens. This has the advantage that the viewing angle and thus the size of the detected environment can be increased.

Another embodiment is to provide a plurality of cameras on the work machine and capture the image information with the plurality of cameras, wherein the image information acquired by the plurality of cameras is at least partially redundant. The cameras can be arranged on the working machine. A camera can have a field of vision oriented in the direction of movement of the working machine. Another camera can be a backward in the direction of movement of the working machine Have field of view. Another camera may have a laterally to the left or right aligned in the direction of movement of the working machine field of view.

A field of view of a camera may have an angle of view. The image angle can be changed in particular with an objective. The field of view or the angle of view of the camera can also be called a field of view.

Another embodiment is that the information about the environment of the work machine is a panoramic image. Images taken by at least one camera may overlap, whereby the fields of view of the cameras may at least partially overlap. In corresponding overlapping areas, redundant image information may be present. By means of a stitching individual images of one or more cameras can be combined to the panoramic image. The panorama image may also be referred to as a surround view. When calculating a panoramic image, brightness differences or contrast differences can be compensated. When calculating a panorama image, distortions can also be reduced. It is also possible to remove stubborn objects from the pictures. For example, a mirror, a chimney or an exhaust of a work machine can be removed. With a panoramic image, the working machine can be controlled taking into account a complete 360 ° field of view.

A further embodiment consists in that a detection and / or visualization of the image information is provided in real time. Real time can be defined in particular as near real time. Capturing the image information in real time can provide a real-time image of the environment around the work machine. A real-time image can be continuously captured and visualized. The real-time image can be displayed by a display device in real time. In order to regulate the work machine, image information can thus be available in real time in order to continuously adapt the movement of the work machine to ambient conditions and thus to correct the movement in real time.

Another embodiment is that the image information can be visualized in different perspectives. Perspectives can also be referred to as views or real-time views. The image information can represent a region of the environment in the field of view of a camera in perspective. Alternatively or additionally, the image information can be viewed as an all-round view or in bird's-eye view ("Bird It is also possible to visualize a trajectory view around the work machine.The perspectives can optionally be visualized separately or side by side.An illustrated image information can also be visualized in changeable, changeable or changing perspectives.Various perspectives have the advantage of that a user can select the representation of the environment on a display device and thus can focus on specific areas of the environment to influence the movement behavior of the working machine.

The regulation of a movement of the work machine along the route can already be done exclusively based on the image information. A further embodiment is that a detection of a working position of the working machine, a visualization of a working route and a visualization of the working position is provided, wherein the working machine is adjustable with a comparison of the working position with the working route. Rules of the work machine based on comparing the work position with the work route may be optional or permanently supportive. The work machine can be temporarily or permanently self-adjustable with the detected work position.

The regulation of a movement of the working machine along the route can first be carried out in a first step as a function of a visual comparison of the working position with the working route. The visual deviation of a current working position from a given working route can be taken into account in order to move the working machine further on or back to the working route.

The regulation of the movement of the working machine along the route can then take place in a second step as a function of the visualized image information. It may be desirable to deviate from the work route followed in the first step in order to carry out the work tasks efficiently and precisely. A to be done for doing the task actually following track can be departed so deviating from the working route with the working machine or a work attachment attached thereto, the predetermined working route can be left.

The first control step may also be referred to as coarse control, in which the work machine is moved along a predetermined working route, and the second control step may also be referred to as fine control, in which the along the working route to be executed work task depending on the actual environmental conditions can be performed accurately.

The detection of the working position can be carried out by means of satellite navigation or satellite positioning. A satellite navigation system which can be used for this purpose can in particular use GPS, GLONASS and / or GALILEO satellites. A receiver for corresponding satellite signals may be provided on the agricultural machine and detect the working positions of the working machine. The accuracy of position data or working positions can be increased by using a reference station, for example, differential GPS can be used. Fast and high-frequency exposure of position information can thus be provided in real time.

In principle, a multiplicity of display devices can be provided for visualizing various information on the work machine. Another embodiment is that a visualization of the image information, the working route, and / or the working position is provided with a single display device. On a display device or display, the current working position of the machine can be visualized with respect to a given working route and display additional image information of the environment. The image information can be projected onto a terrain model around the working route and visualized on the display device. It is also possible to visualize the position of a work tool. It can be done a visual readjustment of the movement of the machine based on the image information. For example, such a cutter bar of a tractor can be precisely moved in a field.

Another embodiment is that controlling the movement of the work machine along the route has a departure from the working route. The rules of the movement of the working machine can not be understood as precisely as possible a precise departure of a given working route, but rather as a adapted to the actual environmental conditions targeted deviation from a given working route can be understood. The rules may in particular have a lateral deviation along the working route. This has the advantage that it can react flexibly to changing environmental conditions or to obstacles. A solution for a display device for guiding a work machine is that the display device is designed to visualize image information acquired by a sensor, the image information having information about the environment around the work machine and wherein the display information displayed on the display device for controlling the work machine is provided. Another solution is a work machine having such a display device.

The display device can also be designed to visualize a working route and a working position of the working machine detected by a sensor. The display device can visualize combined data having image information, the working position and / or the working route. Such combined data may also be referred to as fused data. With the display device, in addition to the representation shown in FIG. 1, in addition to the absolute or relative position of the working machine, information about the environment is also displayed as a real-time scene. A real-time view of the environment may be displayed on the display device as a full-frame with combined data or as a trajectory view as shown in FIG.

Embodiments are explained below with reference to the schematic figures 2 to 4 on.

Fig. 1 shows an indication of a route and a position of a work machine for guiding the same from the prior art.

2 shows a schematic flowchart of method steps for a method for guiding a work machine.

Fig. 3 shows a plan view of a working machine with four cameras for

Visualize the environment around the work machine.

4 shows a representation of components for visualizing image information and a working position of a work machine.

FIG. 2 shows method steps for guiding a work machine 1. A control S3 of the work machine 1 is performed based on a visualization S22 of image information 11. The image information 11 is generated in a preceding step of detecting S12 of the image information 11 and based on the rule S3. The control S3 of the work machine 1 optionally takes place on the basis of a visualization S23 of the work position 12. The work position 12 is generated in a preceding step of detecting S11 of the work position 12 and based on the rules S3.

The control S3 of the work machine 1 is optionally based on a visualization S21 of the work route 14. The work route 14 is generated in a previous step of detecting the work route 14 (not shown).

3 shows a tractor as an agricultural machine or work machine 1 in plan view with four cameras 2, 3, 4, 5, an electronic control unit 6 and a satellite navigation system 7.

The cameras 2, 3, 4, 5 are connected with connections 10 to the control unit 6, for example CAN connections, Ethernet connections, LVDS connections. The connections 10 transmit raw data acquired by the cameras 2, 3, 4, 5 to the control unit 6. The control unit 6 communicates via the connections 10 or electrical connections to the cameras 2, 3, 4, 5, wherein the image acquisition is controlled and parameters of the Image recording can be set.

The cameras 2, 3, 4 and 5 on the work machine 1 have four fields of view 20, 30, 40 and 50.

A first field of view 20 of the first camera 2 is aligned in the direction of travel of the working machine 1 to the rear. The first camera 2 records a first image as a function of the first field of view 20. A second field of view 30 of the second camera 3 is aligned laterally to the left in the direction of travel of the work machine 1. The second camera 3 records a second image as a function of the second field of view 30. A third field of view 40 of the third camera 4 is aligned in the direction of travel of the work machine 1 to the front. The third camera 4 takes on a third image as a function of the third field of view 40. A fourth field of view 50 of the fourth camera 5 is aligned laterally to the right in the direction of travel of the work machine 1. The fourth camera 5 records a fourth image as a function of the fourth field of view 50.

The cameras 2, 3, 4 and 5 are arranged and aligned such that the fields of view 20, 30, 40, 50 generate pairs overlap areas 60, 70, 80 and 90. Regions of the environment in the overlap areas 60, 70, 80 and 90 are thus recorded in two images each by two cameras.

An overlap area 70 results from an overlap of the third field of view 40 and the fourth field of view 50, which define the image capture with the third camera 4 and the fourth camera 5. An overlap area 60 results from an overlap of the fourth field of view 50 and the first field of view 20, which define the image capture with the fourth camera 5 and the first camera 2. An overlap area 90 results from an overlap of the first field of view 20 and the second field of view 30, which define the image capture with the first camera 2 and the second camera 3. An overlap area 80 results from an overlap of the second field of view 30 and the third field of view 40, which define the image capture with the second camera 3 and the third camera 4. Based on the individual images recorded in the fields of view 20, 30, 40, 50, a panoramic view is calculated.

FIG. 4 shows the components for visualizing image information and a working position of the working machine with a display device 8.

With the display device 8 image information 11, a working position 12 and / or a working route 14 of the work machine 1 are visually displayed.

Image information 11 is transmitted from the controller 6, which is connected to the cameras 2, 3, 4 and 5, to the display device 8. The control unit 6 has optional N inputs for N cameras.

The work machine 1 has a satellite navigation system 7. The satellite navigation system 7 has an antenna (not shown) which receives satellite signals, for example, a GPS receiver is disposed on the work machine 1. The satellite navigation system 7 additionally uses a terrestrial positioning system 9, for example a RTK (Real Time Kinematic) positioning system, to generate position data.

The working position 12 is generated by a satellite navigation system 7 and a related optional positioning system 9. The working position 12 is transmitted to the display device 8 via a line, for example CAN or Ethernet. Combined data 13 will be stored on a storage medium 15, for example on a USB stick or an SD card, to be later evaluated offline by the user or user of the work machine 1.

REFERENCE CHARACTERS

working machine

first camera

second camera

third camera

fourth camera

control unit

Satellite Navigation System

display

positioning system

links

image information

working position

Combined data

labor route

storage medium

Field of vision first camera

Field of vision second camera

Field of view third camera

Field of vision fourth camera

first overlap area

second overlap area

third overlap area

fourth overlap area

route

work site

implement

run over terrain

Capture working position

Capture image information

Visualize work route Visualize image information Visualize work position Rules work machine

Claims

claims

A method for guiding a work machine (1) along a route having rules (S3) of movement of the work machine (1) along the route, characterized by detecting (S12) image information (11) with at least one sensor, visualizing (S22 ) of the image information (11) with a display device (8), wherein the image information (11) information about the environment around the work machine (1) and wherein the work machine (1) with the image information (11) is controllable.

2. The method according to claim 1, characterized by rules (S3) of the movement of the working machine (1) with the image information (11) and / or with a working position (12) of the working machine (1).

3. The method of claim 1 or 2, characterized by providing a plurality of cameras (2, 3, 4, 5) on the work machine (1) and detecting (S12) the image information (11) with the plurality of cameras (2, 3 , 4, 5), wherein the image information (11) detected by the plurality of cameras (2, 3, 4, 5) is at least partially redundant.

4. The method according to any one of claims 1 to 3, characterized in that the information about the environment of the working machine (1) is a panoramic image.

5. The method according to any one of claims 1 to 4, characterized by detecting (S12) and visualizing (S22) of the image information (11) in real time.

6. The method according to any one of claims 1 to 5, characterized in that the image information (11) can be visualized in different perspectives.

7. The method according to any one of claims 1 to 6, characterized by detecting (S11) a working position (12) of the working machine (1), visualizing (S23) the working position (12) and visualizing (S21) a working route (14), wherein the working Machine (1) with comparing the working position (12) with the working route (14) is adjustable.

8. The method according to any one of claims 1 to 7, characterized by visualizing (S22, S21, S23) of the image information (11), the working route (14), and / or the working position (12) of the work machine (1) with a single display device (8th).

9. The method according to any one of claims 1 to 8, characterized in that the rules (S3) of the movement of the working machine (1) along the route has a deviation from the working route (14).

10. A display device (8) for guiding a work machine (1) along a route, which is designed for controlling a movement of the work machine (1) along the route, characterized in that the display device (8) is formed by a sensor visualize captured image information (11),

wherein the image information (11) has information about the surroundings around the work machine (1) and wherein the image information (11), which can be displayed on the display device (8), is provided for controlling the work machine (1).

11. Work machine (1), which has a display device (8) according to claim 10.