DE102020213498A1 - Method for transferring a machine to a safe state using gesture control, a control device, and a machine and computer program - Google Patents

Abstract

The development of man-machine interfaces leads from the usual buttons, switches and controllers to well-known input aids on computers, such as computer mice, touch-sensitive screens, etc. While such man-machine interfaces for local applications, such as control panels or terminals or PCs are intended, operating concepts for autonomous cars have been developed which enable gesture control of the car by people outside the car. A method for transferring a machine 2 with at least one additional actuator 3 is designed as an attachment and/or as a tool in a security state by gesture control of a person 5 is proposed, with at least one gesture being recorded by person 5, with a gesture command being determined based on the recorded gesture, with a control command being output to machine 2 based on the gesture command, and with machine 2 through the control command is checked to transfer the additional actuators to the safe state.

Description

The invention relates to a method for transferring a machine into a safe state by gesture control of a person. The invention also relates to a control device for a machine and the machine with the control device and a corresponding computer program.

State of the art

The development of man-machine interfaces leads from the usual buttons, switches and controllers to well-known input aids on computers, such as computer mice, touch-sensitive screens, etc. While such man-machine interfaces for local applications, such as control panels or terminals or PCs are intended, operating concepts for autonomous cars have been developed, which enable gesture control of the car by people outside the car.

For example, the reference discloses DE 10 2018 213 600 A1 , which probably forms the closest prior art, a gesture recognition device for a vehicle, which has a sensor arrangement and a computing unit, the sensor arrangement being set up to capture sensor data from a vehicle environment and in particular a person located outside of one's own vehicle. The processing unit is set up to evaluate the sensor data and to recognize a gesture or a posture of the person outside the vehicle. The computing unit is also set up to control the vehicle based on the detected gesture or the detected body posture and the classification of the person detected.

Disclosure of Invention

The invention relates to a method for switching a machine to a safe state by gesture control with the features of claim 1. The invention also relates to a control device with the features of claim 9, a machine with the features of claim 10 and a computer program with the features of Claim 11. Preferred or advantageous embodiments of the invention result from the dependent claims, the following description and the attached figures.

The invention relates to a method for switching a machine, in particular a working device, into a safe state. The machine has at least one additional actuator. The additional actuator system is designed in particular in addition to and/or independently of a traction drive of the machine. The additional actuators can be designed as an attachment and/or as a tool.

For the machine, in particular for the additional actuators, the safety state is preferably designed as a state in which specifiable safety requirements are met.

The transfer to the safe state is implemented by a person's gesture control. In the most general embodiment of the invention, the person can be anyone, regardless of size, age, appearance, etc. A gesture of gesture control is understood to mean all of the person's bodily signals. Gestures are preferably used which are visualized by arm positions or arm movements. Examples of such gestures are defined in practice by trade associations, such as the BG Wood and Metal according to the DGUV information. Another well-known convention of gesture communication is aircraft marshalling, which is used for communication between air traffic controllers and an aircraft, for example at an airport. However, any gesture can be used for gesture control.

In a method step, at least one gesture is recorded by the person. The gesture is preferably recorded with a sensor device, as will be described below.

A gesture command is determined on the basis of the recorded gesture. The recorded physical gesture is thus assigned to a gesture command. The determination preferably takes place with an evaluation device, as will be described below.

Based on the gesture command, a control command for controlling the machine is issued to the machine. The control command is preferably output via a control device, as will be described below.

It is envisaged that the machine will be controlled by the control command to transfer the additional actuators to the safe state. The machine is thus also transferred to a safe state. Control can be understood as regulation and/or control.

One consideration of the invention is that gesture control is also possible, in addition to controlling or stopping driving movement of a vehicle as an alternative or in addition to converting additional actuators, in particular attachments and/or tools, into a safe state. This opens up a new field of activity for gesture control. In particular, the gesture control is carried out by a person who is arranged at a distance from the machine. The distance is preferably greater than 5 m, in particular greater than 10 m and in particular greater than 20 m.

The machine is preferably designed as a stationary machine, with the additional actuators also being able to implement the basic function of the machine. For example, the machine can be in the form of a conveyor belt, a work machine, a manufacturing machine, a transport machine, etc. It is also conceivable that the machine is implemented as a robot, in particular as an articulated robot or as a Cartesian robot.

In an alternative embodiment of the invention, the machine can be designed as a moving machine, in particular as an excavator, tractor, construction machine, or autonomous vehicle with the additional actuators. In this embodiment it can be provided that the gesture control for the transfer of the additional actuators into the safety state takes place independently of the traction and/or the drive of the machine. However, a coupling is preferably provided such that on the one hand the traction and/or the drive of the machine is stopped and on the other hand the additional actuator system is transferred to the safety state.

In particular, the machine can be designed as a fully autonomous machine, specifically without a driver and/or operator, or as a partially autonomous machine, specifically with a driver and/or operator. In the latter case, it is possible for the method for transferring the machine, in particular the additional actuators, into the safety state by gesture control to be designed as an override via the driver/operator.

• Die Zusatzaktorik, insbesondere das Werkzeug und/oder das Anbaugerät, wird in eine Parkposition und/oder Sicherungsposition und/oder Ablageposition gefahren. So kann beispielsweise für die Ausbildung des Werkzeugs als Schaufel von einem Schaufelbagger als Maschine die Schaufel nach oben gefahren werden. Beispielsweise bei einer Planierraupe als Maschine kann das Schiebeschild nach oben gefahren werden. Es kann vorgesehen sein, dass ein Ausleger der Maschine eingefahren wird und/oder auf die Maschine abgelegt wird. Es kann auch vorgesehen sein, dass die Zusatzaktorik ein oder mehrere Stützeinrichtungen zum seitlichen Stützen der Maschine aufweist und die Stützen in die Stützposition als Sicherungsposition gefahren werden.

In a preferred embodiment of the invention, the safety status for the additional actuators can have any selection of the following actions:

• The additional actuators, in particular the tool and/or the attachment, are moved into a parked position and/or secured position and/or storage position. For example, to form the tool as a shovel, the shovel can be moved upwards by a shovel excavator as a machine. For example, with a bulldozer as a machine, the sliding blade can be moved up. It can be provided that a boom of the machine is retracted and/or placed on the machine. Provision can also be made for the additional actuator system to have one or more support devices for laterally supporting the machine and for the supports to be moved into the support position as a securing position.

The energy supply to the additional actuators is interrupted and the additional actuators are de-energized, in particular de-energized. This safety state corresponds to a well-known emergency stop.

The material feed to the additional actuators is interrupted. If the additional actuator is provided for material transport and/or for material processing, the material feed is stopped in particular. For example, in the case of a road construction machine, the supply of road material, such as tar, can be stopped.

The additional actuators are transferred from a working state to a release state. It can be provided here that the release state means an increase in the distance to a work location.

The additional actuator is opened, in particular a non-positive state of the additional actuator is canceled. For example, a shovel or other tool is opened by an excavator or other machine.

The additional actuators are secured, in particular secured in a form-fitting manner. For example, machine parts of the additional actuators are secured in a form-fitting manner, for example by retracting safety bolts on presses, cutting devices, etc.

The additional actuators are braked in particular by friction in order to immobilize them.

In a preferred development of the invention, it is provided that the machine is controlled by a further gesture, which is interpreted as a gesture command, in order to transfer the machine and/or the additional actuators back into a working state. It should thus be possible not only to transfer the machine, in particular the additional actuators, to the safe state via gesture control, but also to cancel the safe state and return the machine, in particular the additional actuators, to the working state. The gestures for transferring to the safe state and for transferring to the working state are preferably designed differently.

• Alter der Person: Es kann vorgesehen sein, dass nur Personen mit einem vorgebbaren Alter, insbesondere keine Kinder, autorisiert sind.

In an advantageous development, before the machine is checked by the gesture control, it is checked whether the person performing the gesture is authorized. The authorization can be determined, for example, based on a set of rules. Criteria can be applied as follows:

• Age of the person: It can be provided that only persons of a predeterminable age, in particular no children, are authorized.

Position of the person: It can be provided that only persons in a predeterminable position are authorized. For example, it is conceivable that only persons who are located within a site, for example a company site, are recognized as authorized persons.

Identification features of the person: It can be provided that only persons with definable identification features are authorized. Such identification marks could be, for example, special clothing such as a signal vest, a helmet, special headgear or a visually recognizable ID card.

• Rückkanalantwort: Es kann vorgesehen sein, dass auf eine bestimmte Geste die Maschine und/oder die Zusatzachse mit einem Rücksignal antwortet, so dass beispielsweise die Funktionsfähigkeit getestet werden kann. Das Rücksignal kann als eine Betätigung der Zusatzaktorik ausgebildet sein, es kann jedoch auch ein Leuchtsignal oder ähnliches sein.

Furthermore, it can be provided that any selection of the following actions can be controlled by the gesture control:

• Feedback channel response: Provision can be made for the machine and/or the additional axis to respond to a specific gesture with a feedback signal, so that the functionality can be tested, for example. The return signal can be designed as an actuation of the additional actuators, but it can also be a light signal or the like.

Control of the machine: The machine can be controlled by additional gestures. For example, it is possible to navigate the machine as far as it can be moved.

In a preferred development of the invention it is provided that the gestures, gesture commands and the control commands and their functional relationship can be learned. This means that the method, the control device and the machine are not limited to the gestures that are available on delivery, but you can learn your own or individual gestures. For example, learning can be implemented using machine learning or artificial intelligence.

Another subject of the invention relates to a control device for a multi-rail, in particular as described above. In particular, the control device is designed in terms of programming and/or circuitry to carry out the method described above.

The control device is suitable and/or designed to control the machine with the at least one additional actuator. The control device has a sensor device for recording at least one gesture. The sensor device can have a video camera, a thermal camera, a lidar, a radar or other sensors for recording the gestures.

Furthermore, the control device has an evaluation device for determining a gesture command on the basis of the sensor signal from the sensor device. The control device has a control device for outputting a control command or the control command based on the gesture command for controlling the machine. The evaluation device and/or the control device can be designed as software modules and/or hardware modules. In particular, these are designed as a digital data processing device, such as a computer, a cloud service, etc.

In the context of the invention, it is proposed that the control command be designed to convert the additional actuators into the previously described security state.

The control device optionally has an authorization module, the authorization module being suitable and/or designed to authorize the person generating the gesture. For this purpose, the authorization module has a set of rules, for example, as was previously described. It is intended that only gestures by authorized persons will lead to a control command.

The control device optionally has a learning module for learning new gestures, gesture commands and/or control commands.

Another subject matter of the invention relates to a machine with at least one additional actuator system, as has been described above. The machine has the control device as previously described.

Another subject matter of the invention relates to a computer program with program code means, the computer program being designed to execute the method as described above when it is executed on a digital data processing device and/or on the control device as described above.

• 1 eine schematische Darstellung einer Kontrollvorrichtung für eine Maschine als ein Ausführungsbeispiel für das erfindungsgemäße Verfahren;

Further features, advantages and effects of the invention result from the description of preferred exemplary embodiments of the invention and the attached figures. These show:

• 1 a schematic representation of a control device for a machine as an embodiment of the method according to the invention;

the 1 shows a machine 1 with an additional actuator system 2 in a highly schematized representation. As an alternative to this, the machine 1 is implemented as a mobile machine, such as a tractor, an autonomous vehicle, etc. In particular, the machine is designed as an autonomous, mobile work machine, such as an autonomous bulldozer. The machine can be implemented as a construction and/or work machine. It can also be designed as a stone crusher, a scrap press.

The additional actuator 2 is designed, for example, as a tool such as an excavator shovel, a clearing blade from a tractor or a bulldozer, a roller mechanism from stone crushing machines or choppers. Alternatively, the additional actuator 2 can be designed as an attachment, such as a boom of a stationary or mobile crane as a machine or as a support device of a vehicle. A retrofit solution on existing machines is also conceivable.

The machine 1 has a control device 4 , the control device 4 being designed to switch the machine 1 to a safe state via gesture control by a person 5 .

For this purpose, the control device 4 has a sensor device 6 for recording at least one gesture from the person 5 . The sensor device 6 can in particular have an imaging sensor, such as a camera. For example, the sensor can be designed as a 360° camera. Several individual cameras can also be used, which are then evaluated accordingly. Alternatively or additionally, the sensor device 6 can have a thermal camera, a lidar, radar or other sensors for recording the gestures. In particular, the sensor device 6 serves to record the gestures of the person 5 at a distance. The measuring distance is, for example, greater than 5 m or 10 m.

The control device 4 has an evaluation device 7 for determining a gesture command on the basis of a sensor signal from the sensor device 6. The sensor signal can be in the form of an analog and/or digital signal. The evaluation device 7 has a recognition algorithm for predefined gestures. This detection algorithm can be based on digital image processing, artificial intelligence, machine learning.

The gesture command is forwarded to a control device 8 for the output of a control command based on the gesture command. The control command is then transferred to the machine 2 or the additional actuator system 3 via an interface 9 .

Optionally, the control device 4 has a learning module 10 for learning and/or training other or additional gestures. This takes into account that other gestures may be used as the basis for the gesture command in other countries. The supplementing of the further gestures can take place, for example, via Kl elements or methods for machine learning, but also via methods of classic image processing. For example, the control commands based on the gesture commands can also be adapted via the learning module 10 .

Optionally, the control device 4 has an authorization module 11, the authorization module 11 being suitable and/or designed to authorize the person generating the gesture. For example, person recognition is carried out in advance of the gesture control, for example using face recognition or colour/shape markings on vests or helmets. It can thus be ensured that the security state can only be generated by a person 5 known and/or authorized by the control device 4 .

The evaluation device 7, the control device 8 and, optionally, the learning module 10 and/or the authorization module 11 can, for example, be designed together as a digital data processing device. As an alternative to this, the devices or modules can be formed as software components in the cloud. A computer program runs on the digital data processing device to implement a method for converting the machine 2 with the additional actuator system 3 into a safe state by gesture control of the person 5.

The control command runs via the interface 9 to the machine 2 and then to the additional actuator 3 or from the interface 9 directly to the additional actuator 3.

The control command triggers the safety status in additional actuator 3. Provision can be made for only the safety state in the additional actuator system 3 to be triggered in order to bring the machine 2 into the safety state. However, it can alternatively be provided that further, upstream or downstream or parallel actions are implemented in the machine 2 in order to implement the security state of the machine 2 .

In the safety state in the additional actuator 1, it can be provided that an emergency stop/emergency stop is carried out, which indicates that movements (travelling, pivoting, lifting movements) of the additional actuator 3 are set. The security status can be additionally or optionally defined in that the additional actuator 3 stops the current activity and/or carries out work steps that reduce or prevent the possible risk to the environment. Measures can also be taken to protect yourself, for example to prevent possible damage to yourself. In addition to stopping, the safety status can also be used to carry out or stop other or further work steps, such as opening tools as additional actuators 3 or maintenance flaps as additional actuators 3, swinging back booms as additional actuators 3, parking position of grippers as additional actuators 3 or securing states in the additional actuator system 3 by retracting retaining bolts, locking brakes and/or extending supports. A further, alternative embodiment provides that the safety state can also be designed as a standard position of the additional actuator 3, for example the tool, in particular a shovel in a shovel excavator, can be set to the “up” state, i.e. moved up.

The machine 2 optionally has a parallel emergency stop device 12 , which can implement an emergency stop in the machine 2 and/or in the additional actuator system 3 in parallel and thus redundantly to the control device 4 .

It can be provided that the control device 4 not only serves to activate the safety state, but also puts the machine 2 or the additional actuator system 3 back into a working state via gesture control.

• - Normaler Stopp
• - Notstopp
• - visuell bemerkt machen (siehst Du mich?) mit Aufforderung zur visuellen Bestätigung (ich sehe Dich!). Beispielsweise kann dies als einen Rückkanal ausgebildet sein, wobei die Maschine 2 eine Signaleinrichtung erhält, die Signale bestätigen kann, zum Beispiel über Statuslichter, Piktogramme, akustische Signale oder dergleichen. Durch eine Geste wird über das Gestenkommando, den Kommandobefehl die Signaleinrichtung angesteuert, ein optisches oder akustisches Ausgangssignal auszugeben.
• - Positionierungsanforderungen: nach links drehen, nach rechts drehen, rückwärts fahren, vorwärts fahren, aufwärts fahren, abwärts fahren. (Optional ergänzend wird von der Maschine 2 geprüft, ob den Anforderungen stattgegeben werden kann.)

Other gestures can also be used in addition to the activation of the safety state. In particular, a larger, automatically recognizable gesture set can be used, which forms a rudimentary command set from person 5 to machine 2, such as:

• - Normal stop
• - emergency stop
• - get noticed visually (can you see me?) with a request for visual confirmation (I see you!). For example, this can be in the form of a return channel, with the machine 2 receiving a signaling device that can confirm signals, for example via status lights, pictograms, acoustic signals or the like. The signaling device is controlled by a gesture via the gesture command, the command command, to output an optical or acoustic output signal.
• - Positioning requirements: turn left, turn right, drive backward, drive forward, drive up, drive down. (Optionally, machine 2 also checks whether the requirements can be met.)

Furthermore, the gesture, the gesture command and/or the control command can also act on several machines/manufacturing systems in the network.

• - Die Person 5 benötigt kein Eingabegerät.
• - Keine Beschränkung des Befehlssatzes. Neben der Aktivierung des Sicherungszustands können weitere Befehle umgesetzt werden.
• - Die Kontrollvorrichtung kann zusätzlich zu einer bestehenden Notauseinrichtung 12 eingesetzt werden.
• - Die Kontrollvorrichtung 4 ist nachrüstbar, da diese keine tiefgreifende Integration in die Maschine 2 benötigt. In einfachen Fällen kann die Kontrollvorrichtung 4 an einen bestehenden Notausschalter angeschlossen werden bzw. ist die Erzeugung des Notaussignals weniger aufwendig.
• - Die Kontrollvorrichtung 4 ist vorteilhaft für autonome Maschinen 2, insbesondere Baumaschinen. Herkömmliche Notaussysteme befinden sich typischerweise an den Maschinen selbst, wo der Bediener es schnell auslösen kann. Bei autonomen Maschinen müsste erst eine Person zu der Maschine eilen bzw. eine entsprechende Fernbedienung muss zur Hand sein. Durch die Kontrollvorrichtung 4 ist dies nicht notwendig.
• - Durch das Lernmodul 10 kann die Kontrollvorrichtung 4 International verwendet werden und unterschiedliche Gesten lernen.

The optional benefits of Controller 4 are:

• - Person 5 does not require an input device.
• - No instruction set limitation. In addition to activating the backup state, other commands can be implemented.
• - The control device can be used in addition to an existing emergency stop device 12.
• - The control device 4 can be retrofitted, since it does not require extensive integration into the machine 2. In simple cases, the control device 4 can be connected to an existing emergency stop switch, or the generation of the emergency stop signal is less complex.
• - The control device 4 is advantageous for autonomous machines 2, in particular construction machines. Traditional emergency stop systems are typically located on the machines themselves, where the operator can activate them quickly. With autonomous machines, a person would first have to rush to the machine or have a corresponding remote control at hand. This is not necessary due to the control device 4 .
• - Through the learning module 10, the control device 4 can be used internationally and learn different gestures.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102018213600 A1 [0003]

Claims (11)

Method for transferring a machine (2) with at least one additional actuator (3) designed as an attachment and/or as a tool to a safe state by gesture control of a person (5), wherein at least one gesture is recorded by the person (5), a gesture command is determined on the basis of the recorded gesture, a control command being issued to the machine (2) on the basis of the gesture command, wherein the machine (2) is controlled by the control command to convert the additional actuators into the safe state. procedure after claim 1 , characterized in that the machine (2) is designed as a stationary machine (2). procedure after claim 1 , characterized in that the machine (2) is designed as a moving machine (2). Method according to one of the preceding claims, characterized in that the transfer to the safety state for the additional actuators (3) comprises at least one, some or all of the following actions: - Transferring the additional actuators (3) to a parking position and/or safety position and/or storage position ; - Terminating an energy supply to the additional actuators (3); - Ending a material feed to the additional actuators (3); - Transferring the additional actuators (3) from a working state to a release state; - Opening the additional actuators (3); - Secure the additional actuators (3); - Stalling of the additional actuators (3). Method according to one of the preceding claims, characterized in that the machine (2) is controlled by a further gesture command to convert the machine (2) and/or the additional actuator system (3) back into a working state. Method according to one of the preceding claims, characterized in that before the machine (2) is checked by the gesture control, it is checked whether the person (5) performing the gesture is authorized. Method according to one of the preceding claims , characterized in that at least one, some or all of the following actions can be controlled by further gesture commands: - feedback channel response - navigation of the machine (2) Method according to one of the preceding claims, characterized in that gesture commands are learned on the basis of gestures by the person (5). Control device (4) for a machine (2), the machine (2) having at least one additional actuator (3) designed as an attachment and/or as a tool, with a sensor device (6) for recording at least one gesture from a person ( 5) and for outputting a sensor signal, with an evaluation device (7) for determining a gesture command based on the sensor signal, with a control device (8) for outputting a control command based on the gesture command for controlling the machine, characterized in that the control command is designed to convert the additional actuators (3) into a secure state. Machine (2) with at least one additional actuator (3) designed as an attachment and/or tool, characterized by a control device (4) according to one of the preceding claims. Computer program with program code means to implement the method according to one of Claims 1 until 8th to be carried out when the computer program is executed on a digital data processing device and/or on the control device (4).

Images