WO2015135786A1

WO2015135786A1 - Assistance system for a piece of agricultural machinery, and method for assisting an operator

Info

Publication number: WO2015135786A1
Application number: PCT/EP2015/054336
Authority: WO
Inventors: Rainer Resch; Heike Froeschle
Original assignee: Amazonen-Werke H. Dreyer Gmbh & Co. Kg
Priority date: 2014-03-11
Filing date: 2015-03-03
Publication date: 2015-09-17
Also published as: DE102014103195A1; EP3117261A1

Abstract

Disclosed is an assistance system (1) for a piece of agricultural machinery (4) to assist an operator in an agricultural work environment (21, 41), e.g. a field (41) and/or a cab (41) of the piece of machinery (4), characterized in that the assistance system (1) comprises a mobile display unit (3, 5) designed to display information (61-68) to the operator onto which the agricultural work environment (21, 41) is superimposed to create an augmented reality effect.

Description

Assistance system for an agricultural work machine and method for assisting an operator

The invention relates to an assistance system for an agricultural

A work machine having the features of the preamble of claim 1 and a method of assisting an operator to operate an agricultural work machine having the features of the preamble of claim 12.

Such an assistance system is known, for example, from EP 1 674 324 A1. In the viewing window of a combine harvester, a head-up display is shown in order to show the driver information such as crop parameters or operating parameters directly into the field of view. This makes it possible for the operator to read these parameters from the head-up display without averting the view from the field to be processed.

Furthermore, fixed screens are also known in the cab, on which diverse information of the machine can be displayed.

In such assistance systems, it may happen that the operator a large amount of information is displayed simultaneously and this

Operator prompting difficult. The object of the present invention is therefore an assistance system

to provide a simple user guidance is possible even with a variety of information to be displayed.

This task is used in an assistance system with the characteristics of

The preamble of claim 1 having the features of the characterizing part, according to which the assistance system comprises a mobile display unit adapted to display to the operator information superimposed on the agricultural working environment in the sense of an augmented reality. Because the assistance system comprises a mobile display unit, it is basically possible to display different information depending on the position of the display unit. Thus, the operator can hold the mobile display unit in a particular direction to the work environment to properly display the information desired thereto. As a result, it is no longer necessary to display all the information on the display unit at the same time or to switch it over using the keyboard.

By having the information overlaid with the agricultural

Working environment in the sense of an enriched reality are displayed, the information is automatically with the underlying agricultural

Work environment shown linked. As a result, the information can be more easily associated with the features of the work environment, thus reducing the operator's workload and increasing safety. This makes it easy even with a lot of information

User guidance possible.

The agricultural working machine can be a tractor, like

for example, a tractor, or a combine harvester. On the working machine a tillage implement, a seeder a fertilizer spreader, a sprayer or the like may be arranged. The work machine may include a cab for the operator.

The agricultural work environment may include a field and / or the cab of the work machine. It is also conceivable that the working environment includes an interior and / or exterior of the work machine, such as

For example, a motor, a boom and / or a roller. It is also conceivable that the agricultural working environment comprises a stable, a supply of feed and / or an animal. In other words, the agricultural work environment may include anything, including the operator in the

agricultural labor information needed.

The assistance system can be a data processing unit with a

Microprocessor, a memory, an image and / or signal processing unit. The data processing unit can be designed to be mobile Display unit and / or the content displayed on it (images of the working environment, the information and their positions, graphics, text, labels, notes, etc.) to calculate. The mobile display unit may include the data conditioning unit. It is also conceivable that the mobile

Display unit is connected via a wired or wireless data interface with the data processing unit. The wireless data interface can be used as a WLAN, Bluetooth or infrared interface, as well as CAN, ISOBUS or NFC or similar. be educated.

Mobile display unit may mean here that the display unit is movable relative to the work machine. Mobile display unit may also mean that the display unit has no fixed connection to the work machine and optionally via a cable or via a wireless data interface with the

Working machine is connected.

The information may be planning and / or management data, information on machine operation, help information for the operator (help desk) and / or training information.

Planning and management data can mean here that it is

Order data, work-related data, information about the used

Seeds or plants grown on the field,

Information about dew formation, the disease pressure and / or um

Environmental information such as dew formation, weather data, geographical data or the stock temperature oa. is.

Information about machine operation can be operating parameters (for example driving speed, direction, etc.), operating information and / or operating functions.

Auxiliary information may be information about the virtual

Operator support act. For example, it can be virtual

Manuals and / or a video chat with a service representative. It is also conceivable that the auxiliary information

Customer service information includes, for example, assistance with the

Error analysis or a catalog for spare parts procurement. It is also conceivable that the assistance system provides information about the training. For example, a teacher can selectively display information on the mobile display unit for the operator to convey the operation of the work machine. Enriched reality can mean here that the information displayed correlates to the agricultural working environment. Likewise

Enriched reality means that a camera image and / or 3D data correlated to the agricultural work environment along with it

Information is displayed. The information can with

rotationally invariant features of the agricultural work environment. Rotationally invariant features can be pattern patterns,

Color structures that are working width, aspect ratios, etc. in the working environment.

The mobile display unit may be a smart phone. As a result, the mobile display unit is always in the field of vision of the operator and is particularly easy to carry. The data glasses can be designed as a head-mounted display (HMD). The data glasses can be designed with a display for one eye or with two displays for both eyes of the operator. It is also conceivable that the data glasses for a single eye have multiple displays. The displays may be LCD, DMD or LCoS displays, for example. It is also conceivable that the data glasses have a projection unit, optionally a laser projector, with which the information is projected directly onto the retina of the operator.

The data glasses can have a projection unit and at least one partially transparent region, by means of which the working environment is directly visible and via which the information can be faded in with the projection unit.

Due to the partially transparent region of the data glasses, the operator can directly record the working environment. At the same time the information by means of the projection unit in the partially transparent area with the

Overlayed working environment. As a result, the visual impression of the enriched reality is particularly high-quality and the consideration for the operator is not very exhausting. Data glasses are known for example from the project "Google Glass".

The mobile display unit may be a tablet computer or a mobile phone. Since tablet computers or mobile phones are produced in large quantities, they are available at very low cost. The tablet computer or the

Mobile phone may be configured with a touch-sensitive screen for input.

The assistance system can be designed with a camera and / or with a 3D sensor for detecting the agricultural working environment. By means of the camera, the working environment can be detected and displayed as part of the enriched reality on the mobile display unit. The camera can be designed, for example, as a CCD or CMOS camera. By the

Capture the working environment with the 3D sensor, it is possible to

display a three-dimensional image of the working environment as an enriched reality via the mobile display unit. The assistance system may be adapted to the characteristics of the

recognize agricultural working environment in a camera image and / or in 3D data and to link the information with it. This makes it particularly easy to recognize the features of the agricultural work environment and to evaluate the display of the information. The recognition of the features can be done with an image and / or 3D data processing unit. The image and / or 3D data processing unit may comprise a digital filter. The image and / or 3D data processing unit can be integrated in the data processing unit described above.

The assistance system can be designed with a position detection system for determining the position of the mobile display unit in relation to the agricultural working environment. The location detection system can be a camera, a

Position sensor and / or a location system include. By doing that

Assistance system comprises a location detection system, it can be particularly easily determined the orientation of the assistance system relative to the agricultural work environment. Due to the known location, the

Information particularly easily linked to the work environment and about the mobile display unit. By using the camera, the position detection system is particularly simple. For the

Position detection system, the same camera can be used as the previously described camera for detecting the agricultural work environment. If the position detection system comprises a position sensor or a location system, the position of the assistance system can also be determined if the field of view of the camera is covered.

The assistance system can be a microphone, a headphone and / or a

Speaker include. This makes it possible that the operator with another person, such as a trainer or a

Service employee communicates online.

The assistance system can have at least one data interface with a database, a planning and / or management system, a

Control unit of the agricultural machine, a help desk and / or a training system connectable. That's it

Assistance system for many applications. The

Data interface can be designed as a radio interface and optionally include a GSM, WLAN, UMTS or LTE or other mobile data interface. With the data interface, a connection via a data network such as the Internet can be built. As a result, an interaction of the operator with remote applications, servers, people or the like is possible.

In addition, the invention with the claim 12, a method for assisting an operator in the operation of an agricultural

Working machine in an agricultural working environment ready, characterized in that the operator with a mobile display unit of an assistance system information overlaid with the agricultural

Working environment in the sense of an augmented reality. By having the information overlaid with the agricultural

Work environment are displayed in the context of an enriched reality, the information of the operator is displayed particularly clearly and linked to the working environment. Furthermore, the operator can direct the mobile display unit to the part of the work environment to which the

Operator would like to receive specific information. This is the result

Selection of relevant information particularly easy.

The method may have the features previously described in relation to claims 1-11, individually or in any combination. In the method, the working environment with a picture and / or

Signal processing unit are recognized and used to make positions for the

Information to be calculated. The information is then displayed in the mobile display unit so that it appears to the operator at the calculated positions. The method can be designed so that the information with the direct field of view of the operator or with a

Camera image are superimposed.

Further features and advantages of the invention are explained below with reference to the exemplary drawings. Showing:

1 shows a first embodiment of the assistance system with a data glasses in a perspective view. and

2 shows a second embodiment of the assistance system with a

Tablet computer in a perspective view.

1 shows a first embodiment of the assistance system 1 with a data glasses 3 in a perspective view. You can see the field of view of an operator through a mobile display unit, here as

Data glasses 3 is formed. The data glasses 3 is transparent in the areas 33 and gives a direct view of the scene 2 with the agricultural

Work environment 21 free. It can be seen that the operator is currently working the field 21 with an agricultural machine and a plow (not shown here), wherein the plow furrows 22 with the working width A to each other. Furthermore, it can be seen that the data glasses 3 are connected via the data interfaces 37, 71 to a control unit 7 of the working machine. This allows, for example, operating parameters of

Work machine to the data glasses 3 are transmitted. The data glasses 3 have the projection unit 31 and the partially transparent area 32. This is arranged here in the region of the right eye of the operator, but can also run in front of the left eye or both eyes. In region 32, the operator can see directly on the field 21 as through the transparent regions 33. At the same time, information 62-65 can be displayed in area 32 by means of production unit 31.

To capture the agricultural working environment 21, the data glasses 3 have a camera 34. In this way, the already picked plow furrows 22 can be detected and used, for example, the working width A can be determined automatically. In addition, the position of the data glasses 3 with respect to the working environment 21 is determined with the camera 34 and the position sensor 36. Together, they form the situation detection system. The position sensor 36 is here with

Acceleration sensors formed and thus detects movement of the

Data glasses 3 around all three spatial axes. This makes it possible to determine the position of the data glasses 3 with respect to the working environment 21 and to hide the information 62, 65 associated therewith in the partially transparent area 32 at the intended position.

Furthermore, it can be seen that the data glasses 3 are formed with a microphone 38 and a headphone 35. This makes it possible for the operator to contact another person and, for example, instructions for

Processing of the field 21 receives. Furthermore, the operator at a

Participate training for plowing the field 21, wherein the teacher via the camera 34 receives an image of the field 21 and the operator on the

Headphone 35 corresponding instructions for editing can notify.

In addition, the operator can talk to the teacher via the microphone 38. The data glasses 3 are equipped with a Bluetooth data interface 37. This is by radio with a corresponding Bluetooth data interface 71 of

Control unit 7 connected to the work machine. This makes it possible that digital data between the data glasses 3 and the control unit 7 are exchanged, such as operating parameters of the working machine. Furthermore, the control unit 7 is connected to the Internet via a UMTS mobile radio interface 72. This makes it possible for the assistance system 1 to communicate with a database, a server application and / or another person via the Internet. In addition, the data glasses 3 have a data processing unit 39, in which all data for the operation of the data glasses 3 are processed. The data include camera images of the camera 34, audio data of the microphone 38 and the headphone 35, position data of the position sensor 36, data streams from / to the

Data interface 37 and / or control data to the projector 31. The

Data processing unit 39 is formed with a microprocessor and a memory.

Overall, the assistance system 1 is used as follows: The operator puts on the data glasses 3 similar to a normal glasses and begins with the fieldwork. The plow furrows 22 are generated at a distance A on the field 21. The operator is displayed via the data glasses 3 (mobile display unit) the information 62-65 superimposed in the sense of an enriched reality with a direct view of the working environment 21.

The arrow 64 indicates the speed on the scale 63. For example, the target speed is reached when the arrow 64 is in the center of the scale 63. In addition, the operator receives a feedback, if the

Work machine is moving too fast or too slow.

Furthermore, the arrow 62 along the virtual Pflugfurche 65 indicates the still to be plowed furrow in advance. If the operator follows the virtual furrow 65, then the arrow 62 is located exactly on the virtual furrow 65. If the predicted travel path of the working machine deviates from this, the arrow 62 becomes corresponding to the virtual furrow 65 twisted so that the

Operator is pointed to a correction of the steering movement.

By the assistance system 1 in Fig. 1 is a particularly simple

Operator guidance possible, whereby even complex information can be displayed.

FIG. 2 shows a second embodiment of the assistance system 1 with a tablet computer 5. You can see the working machine 4 with the

Driver's cab 41. The cab 41 is here the agricultural

Working environment. It can be seen that the driver's cab 41 has the dashboard 42 with the screen 43, the display elements 44 and the switches 45. Furthermore, the steering wheel 46 can be seen, with which the operator can perform corresponding control movement of the working machine 4.

Furthermore, it can be seen that the operator is holding the tablet computer 5 with his or her hand 1 1 in order to receive instructions for operating the driver's cab 41 from a support employee.

It can be seen that the tablet computer 5 is a touch-sensitive

Screen 51 has. The camera image 61 of a detail of the dashboard 52 is shown on it. Furthermore, it can be seen that the

Operation information 66-68 with the camera image 61 are displayed overlaid.

The tablet computer has for this purpose on the back of a camera 53, which is adapted to capture a portion of the work environment 41 behind the tablet computer 5 as a camera image 61.

Furthermore, the tablet computer 5 comprises a computing unit with a computer program product executed thereon, with which the image information of the

Camera 53 are evaluated. It will now automatically the

Display elements 44 and the switches 45 detected in the image. This makes it possible that the display elements 44 and the switches 45 automatically the

Information 66, 67 are assigned. Moreover, it is possible to display the information text 68 depending on the camera image 61. In addition, the tablet computer 5 has a button 52 to one

Switch home screen. With the microphone 55 and the speaker 54 and the data interface 56 (WLAN and UMTS data interface), the operator can contact a training manager or service technician via

Internet record. Conceivable here is also a video chat. This is the result

Tablet computer 5 particularly flexible use.

The assistance system 1 is used as follows: The tablet computer 5 is directed with the camera 53 onto the dashboard 52. The assistance system 1 now recognizes by an automatically running image processing in the camera image 61, the display elements 44 and the switch 45. This information is now superimposed in terms of an enriched reality with the cab 41 displayed on the screen 51. In this case, corresponding instructions for operating the work machine 4 are displayed in the information text 68, the arrows 66 and 67 correlated with the display elements 44 and the switch 45 are displayed. Now moves the operator the tablet computer 5 relative to the dashboard 42, the arrows 66, 67 are automatically moved with the camera image 61. As a result, even complex information can be displayed ergonomically and particularly easily readable by the assistance system 1.

It is understood that the data glasses 3 in FIG. 1 can also be used for an application in the driver's cab 41 of the working machine 4 as described in FIG. 2. It is also conceivable that the tablet computer 5 is directed from the Fig. 2 on the scenery 2 of Fig. 1 and the operator information for editing the field 21 are displayed. Overall, the assistance system 1 according to the first and second embodiments are widely used. Conceivable here are planning and / or management tasks, provision of operating parameters,

Control and operating tasks for the working machine 4, inquiries to a help desk and / or training.

It is understood that in the embodiments described above mentioned features are not limited to these specific combinations and are also possible in any other combinations.

Claims

claims

1 .

Assistance system (1) for an agricultural working machine (4) for

Assisting an operator in an agricultural working environment (21, 41), for example a field (21) and / or a driver's cab (41) of the

Work machine (4), characterized in that the assistance system (1) comprises a mobile display unit (3, 5), which is adapted to the operator information (61 - 68) superimposed with the agricultural working environment (21, 41) in the sense of augmented reality.

Second

Assistance system (1) according to claim 1, wherein the mobile display unit is a data goggles (3).

Third

Assistance system (1) according to claim 2, wherein the data glasses (3) a

Projection unit (31) and at least one partially transparent region (32) through which the work environment (21) is directly visible and through which the information (61-65) with the projection unit (31) are fade.

4th

Assistance system (1) according to claim 1, wherein the mobile display unit is a tablet computer (5) or a mobile phone.

5th

Assistance system (1) according to at least one of the preceding claims, wherein the assistance system (1) with a camera (34, 53) and / or with a 3D sensor for detecting the agricultural working environment (21, 41) is formed.

6th Assistance system (1) according to at least one of the preceding claims, wherein the assistance system (1) is designed to provide features of

Agricultural work environment in a camera image (61) and / or in 3D data to recognize and to link the information (61 - 68) with it.

7th

Assistance system (1) according to at least one of the preceding claims, wherein the assistance system (1) with a position detection system (34, 36, 53) for determining the position of the mobile display unit (3, 5) relative to the

agricultural working environment (21, 41) is formed.

8th.

Assistance system (1) according to claim 7, wherein the position detection system comprises a camera (34, 53), a position sensor (36) and / or a positioning system (GPS system).

9. assistance system (1) according to at least one of the preceding claims, wherein the assistance system (1) comprises a microphone (38, 55), a headphone (35) and / or a speaker (54).

10th

Assistance system (1) according to at least one of the preceding claims, wherein the assistance system (1) via at least one data interface (37, 56, 71) with a database, a planning and / or management system, a control unit of the agricultural machine (4), a Helpdesk and / or a training system is connectable.

1 1. An assistance system (1) according to claim 10, wherein the data interface (37, 56) has a connection via a data network, e.g. the internet is buildable.

12th Method for assisting an operator in the operation of an agricultural machine (4) in an agricultural

Work environment (21, 41), characterized in that the operator with a mobile display unit (3, 5) of an assistance system (1) information superimposed with the agricultural environment (21, 41) in the sense of an augmented reality (augmented reality) are displayed.