WO2023110993A1

WO2023110993A1 - Creating a work instruction

Info

Publication number: WO2023110993A1
Application number: PCT/EP2022/085809
Authority: WO
Inventors: Gero Jacob Corman; Miriam HERMANN; Timm SCHULZ-ISENBECK; Lukas Lehmann; Martin Kumke; Niklas Brandt; Nils Kessler; Christian Kiefer; Yannick Wehmann
Original assignee: Volkswagen Aktiengesellschaft
Priority date: 2021-12-15
Filing date: 2022-12-14
Publication date: 2023-06-22

Abstract

The invention relates to an apparatus (10) for creating a work instruction of an expert (32) for an activity, and to a corresponding system (18), method and computer program. It is provided that, in order to perform intuitive programming, the creator (32) of a new application (46) wears a camera (22) or other sensor (22) that can sense the environment and the action of the application creator (32). To create a new application (46), the creator (32) can then perform activities (36, 38, 40, 42) as they should appear in the subsequent application (46). For example, the creator can perform assembly work, carry out work on a machine (36), rectify faults (38), or similar. All of these gestures or other actions are recognised by the camera or sensor system (22) and translated into an augmented reality program (46). As a result, the creator (32) of a new application performs the activities to be programmed quite normally and thus intuitively creates a new program (46).

Description

Creating a work manual

The invention relates to a device for creating work instructions for an expert for an activity and a corresponding system, method and computer program.

Augmented reality applications, i.e. computer-assisted supplementation of reality with virtual elements, are ubiquitous. Augmented reality technology expands human perception in real time through texts, videos, images or three-dimensional animations. Creating an application for an augmented reality is complex and not very intuitive. In particular, a new application is mainly programmed on a computer.

US Pat. No. 8,836,768 B1 relates to users of wearable glasses with a pair of two-dimensional cameras that optically capture information for user gestures performed with a user object in an interaction zone in response to viewing displayed images with which the user can interact. The eyewear system intelligently signals, captures, and processes optical information to quickly determine a sparse set of locations suitable for identifying user gestures. The displayed images can be generated by the eyewear system and presented with a virtual display on the eyewear, or can be generated and/or viewed without the eyewear. In some embodiments, the user may see local views directly, but supplemented with images showing web-provided tags that identify viewed objects and/or provide information. Eyewear-mounted systems can communicate wirelessly with cloud servers and with non-eyewear-mounted systems that the user can carry in their pocket or purse. The disadvantage here is that a person skilled in the art does not receive any suggestion from the publication as to how augmented reality software can be created,

The publication US 2020 004 335 4 A1 relates to a desktop system for intuitive guidance in an augmented reality long-distance video communication environment. The table system includes augmented reality goggles worn by a field worker, equipped with a video camera that receives actual image information on site and an augmented shows instructions on a transparent display; a tabletop that receives the actual image information from the augmented reality glasses and displays the actual image information on a touch screen, captures hand motion information and instructions that are displayed by a remote expert in an upper area of the touch screen, and transmits the hand motion information and instructions to the augmented -Reality glasses transmit. The disadvantage here is that a person skilled in the art does not receive any suggestion from this publication either as to how augmented reality software can be created. It only teaches how information can be transferred from an expert to augmented reality glasses.

The invention is now based on the object of specifying an improved possibility of creating work instructions, in particular in the form of augmented reality software. An intuitive possibility should preferably be specified in order to enable rapid and error-free development of the software.

This object is achieved by a device for creating work instructions for an expert for an activity, having: an input interface for receiving expert data including operating data and/or sensor data with information on the behavior of the expert during the activity; an analysis unit for determining an action of the expert based on the expert data and determining a work instruction for the work instruction; an output interface for transmitting the work instructions to a storage unit, the analysis unit being designed to determine work instructions for the activity of the expert from one or more work instructions.

The above object is further achieved by a system for creating work instructions for an expert for an activity, comprising: a device as defined above; a sensor for acquiring expert data; a storage unit for storing the work instruction, and preferably an AR display device.

Furthermore, the above object is achieved by a method for creating work instructions for an expert for an activity, comprising the steps: receiving expert data, including operating data and/or sensor data with information about the behavior of the expert during the activity;

determining an action of the expert based on the expert data;

determining a work instruction for the work instruction based on the expert data and a work instruction for the activity of the expert from one or more work instructions and

Transmitting the work instruction to a storage unit.

Finally, the above object is achieved by a computer program with program code means to carry out all the steps of a method as defined above when the computer program is executed on a computer or a corresponding computing unit.

A front-end interface can provide a cost-effective device that can preferably be used with existing systems, modules, and units. In particular, it is conceivable to directly record an operator input from an expert on a system or machine or to use an existing sensor infrastructure, such as a surveillance camera of the system. An input interface can be wired and/or wireless and preferably support one or more communication protocols. An action by the expert can be determined quickly and reliably and a work instruction can be determined by an analysis unit. As a result, the expert and his movements can be recorded efficiently and work instructions can be provided on the basis of the movements. The work instructions preferably include an augmented reality program. It goes without saying that work instructions can also be created in text form, with the movements and activities of the expert being particularly preferably recorded. For example, turning a screw can be detected from a movement and this can be recorded in writing in the work instructions. Alternatively, a corresponding overlay can also take place in an augmented reality environment. An output interface enables a cost-effective device that can preferably be used with existing storage systems, in particular cloud storage. By using an input interface for receiving, a technically simple and cost-effective device can be created which, in particular, does not have to be designed to record the information itself. The device can thus be integrated into already existing systems. It goes without saying that the device can also be used purely in terms of software can be created. In particular, it is conceivable to create the device in the form of an app on a smartphone.

In a preferred embodiment of the invention, it is provided that the analysis unit is designed to translate the behavior of the expert during the activity into an augmented reality program and the work instructions include an augmented reality application containing the augmented reality program. As a result, an intuitive possibility can be specified for creating an augmented reality program. In particular, a work instruction can be created that can be grasped quickly and can be understood intuitively. An expert, preferably without programming knowledge, is enabled to pass on his expert knowledge in an understandable way.

In a further preferred embodiment of the invention, it is provided that the analysis unit is designed to interpret an action by the expert and, based on the interpretation of the action by the expert, to determine the action. Determining the action means that the analysis unit records, based on the expert data, what the expert performs in his job. For example, it can be determined from a turning movement of the hand and the position of the hand that an expert is operating a rotary knob, and the extent of the turning movement can also be detected. For example, the action can be determined from this, such as setting a speed using a rotary control, the speed being increased by X units. For example, it can also be recognized when a screw is screwed or the like. In particular, it can be determined from the position of the screw and stored construction plans what kind of screw it is.

In a further preferred embodiment of the invention, it is provided that the analysis unit is designed to determine the action of the expert using artificial intelligence. As a result, a determination of the action of the expert can be made more precise and improved. In particular, an improved determination can take place with each use of the artificial intelligence. A reliable and improving device can be provided.

In a further preferred embodiment of the invention, it is provided that the input interface is designed to receive expert data, including camera data from a camera that records an action by the expert; lidar data of a lidar detecting an action of the expert; Glove data, a sensor glove that performs an action of expert detects and/or receive suit data of a sensor suit detecting an action of the expert. An action by the expert can be reliably detected by the aforementioned sensors and devices. In addition, expert data recorded in this way allow the expert action to be reliably determined. It goes without saying that the expert data can be supplemented by operator inputs on a machine that the expert operates. For example, the expert data may include source code when the expert reprograms a machine.

Another preferred embodiment of the invention provides that the analysis unit is designed to recognize one or more of the following activities of the expert: assembly activities of the expert, carrying out work on a machine, eliminating errors and/or setting up a machine. Based on the pre-categorization of the activities of the expert, an improved recognition and determination of individual expert actions can take place.

In a further preferred embodiment of the invention, it is provided that the input interface is designed to receive expert data, including recognized gestures of the expert; the analysis unit is designed to determine the gestures and to determine a representation corresponding to the gestures and the output interface is designed to transmit a control command to a VR display device, in particular VR glasses of the expert, the control command an output of the determined display on the VR display device. As a result, the expert can be given an intuitive opportunity to improve a work instruction or to specify it more precisely. Instructions for use can be created in the form of an augmented reality program, which allows the expert to use the advantages of augmented reality without the expert having to have any prior knowledge in the field of augmented reality programming.

Unless stated otherwise in the individual case, the various embodiments of the invention mentioned in this application can advantageously be combined with one another.

Machine learning or machine learning is a generic term for the "artificial" or machine generation of knowledge from experience: An artificial system learns from examples and can generalize them after the learning phase has ended. To do this, machine learning algorithms build a statistical model based on training data. This means that the examples are not simply learned by heart, but that patterns and regularities are recognized in the learning data. That's how it can be System also assess unknown data, a so-called learning transfer, or fail to learn unknown data, for example through overfitting.

Extended reality or augmented reality is the computer-aided extension of the perception of reality. This information can appeal to all human sensory modalities. Often, however, augmented reality is only understood as the visual representation of information, i.e. the supplementing of images or videos with computer-generated additional information or virtual objects by means of overlays/overlays.

An augmented reality headset, AR headset for short, also called AR or augmented reality glasses, is a type of semi-transparent display arranged on the head and preferably in front of the eyes of a user, which gives the user an insight into the augmented Reality (AR) should provide. It goes without saying that virtual reality glasses can also be regarded as AR glasses if a video recording of the environment is shown on the displays of the VR glasses and supplemented by AR elements.

Work instructions are to be understood in particular as instructions on how work is to be carried out. The work instructions can contain tips and additional information for the work to be carried out. A work instruction can be designed in text form, in audio form, and video form and/or as an augmented reality application.

The invention is explained below in exemplary embodiments with reference to the associated drawings. Show it:

Figure 1 is a schematic representation of a device for creating a

work instruction of an expert for an activity;

Figure 2 is a schematic representation of a system for creating a

work instruction of an expert for an activity;

FIG. 3 shows a schematic representation of an expert in an activity;

Figure 4 shows a schematic representation of various expert activities and the

Creation of a work manual for the expert activities and FIG. 5 shows a schematic representation of the steps of a method according to the invention.

FIG. 1 schematically shows a device 10 for creating work instructions for an expert for an activity. The device 10 comprises an input interface 12, an analysis unit 14 and an output interface 16.

The input interface 12 is designed to receive expert data, including operating data and/or sensor data, with information about the behavior of the expert during the activity. The expert data can include, for example, positions and movements of the expert, actions recognized by the expert and/or operator inputs by the expert and can come, for example, from a sensor suit, sensor glove, AR input device, a smartphone, a radar, laser or lidar sensor, a camera and/or or come from smart glasses. Furthermore, the input interface 12 can be designed for wireless communication or be connected to a proprietary transmission network, for example a wired transmission network.

The analysis unit 14 is designed to process and analyze the expert data. The analysis unit 14 determines an action of the expert based on the expert data and determines a work instruction for the work instruction. The work instructions can therefore include several work instructions. The analysis unit 14 preferably translates the work instructions into work instructions in the form of an augmented reality program. Furthermore, the analysis unit 14 can interpret an action by the expert and determine the action based thereon.

For example, tightening of a screw or operation of a button can be determined from a recorded and analyzed hand movement of the expert. It goes without saying that the work instructions can be supplemented with this information. The action of the expert is preferably determined by means of artificial intelligence.

Furthermore, the analysis unit 14 can recognize various activities of the expert, such as assembly activities of the expert, carrying out work on a machine, eliminating errors and/or setting up a machine. It goes without saying that the expert can also enter his activity beforehand by input or speech recognition. The output interface 16 is designed to transmit the control command to a vehicle unit of the motor vehicle, it being possible for the vehicle unit to include in particular a display device and/or a driver assistance system. The output interface 16 can be designed analogously to the input interface 12 for communication. It goes without saying that the input interface 12 and the output interface 16 can also be designed in combination as a communication interface for sending and receiving.

Figure 2 shows a schematic representation of a system 18 according to the invention for creating work instructions for an expert for an activity with a device 10, units for recording expert data, a storage unit 28 and an augmented reality display device 30.

The units for acquiring expert data can in particular include an operating unit 20, an acquisition unit 22, in particular with a radar and/or lidar sensor and/or a camera, a sensor suit 24 and/or a sensor glove 26. It goes without saying that other units, such as a smartphone, are also conceivable. For reasons of clarity, no other units are shown in the drawing.

In the system 18 shown, the input interface 12 can be configured in particular to receive expert data including gestures recognized by the expert. The analysis unit 14 can determine the gestures of the expert and determine a representation corresponding to the gestures. The output interface 16 can transmit a control command to an AR display device or augmented reality display device 30, in particular to the expert's AR glasses. The control command causes the determined representation to be output on the AR display device. This allows the expert to supplement and improve the work instructions. In particular, the expert can influence how the activity is included in the work instructions directly when carrying out his activity.

Storage unit 28 may include non-volatile data storage, such as hard disk storage, flash memory, or the like. Furthermore, the storage unit 28 can also include cloud storage. The work instructions are stored in the storage unit 28 . It goes without saying that the analysis unit 14 in particular can also include a buffer memory for processing the expert data. In Figure 3, an expert 32 or an expert 32 is shown in an activity. For reasons of readability, the masculine form was chosen in the text, nevertheless the information refers to members of any gender.

The expert 32 is in a hall and carries out an activity on a system 34 . Here, the expert 32 is recorded by one or more recording units 22, with the recording units 22 generating expert data.

Furthermore, the expert 32 wears AR glasses and can directly view and/or change the work instruction that he generates during his activity. It goes without saying that the expert 32 can also carry out his activity without AR glasses.

In the example shown, the expert 32 can set up a machine in the system 34 and use the operating unit 20 of the system 34 for this purpose, for example. In addition to the movements and activities of the expert 32, his operating inputs can then also be recorded and stored in the work instructions.

Various expert activities and the creation of work instructions for the expert activities are shown schematically in FIG.

The expert activity can include, for example, carrying out work on a machine 36, in particular an assembly activity or setting up a machine, correcting errors 38, training an apprentice 40 or another activity 42. Other activities 42 can in particular include maintenance of a system 34, filling of a system 34 with resources or the like.

The activity of the expert 32 can be recorded by one or more recording units 22 . Additionally or alternatively, the expert 32 can also use a sensor suit 24, a sensor glove 26 and/or an AR input device or VR input device. This generates expert data with information about the behavior of the expert 32 during the activity.

Based on a translation 44 of the expert data, for example, as described above, using the analysis unit 14, a work instruction, in particular in the form of an augmented reality program 46, can then be created. FIG. 5 shows the steps of a method according to the invention for creating work instructions for an expert 32 for an activity.

In a first step S1, expert data including operating data and/or sensor data with information about a behavior of the expert 32 during the activity is received.

Then, in a second step S2, an action by the expert 32 is determined based on the expert data.

In a third step S3, a work instruction for the work instruction is determined based on the expert data and a work instruction for the activity of the expert 32 from one or more work instructions.

Finally, in a fourth step S4, the work instructions are transmitted to a storage unit 28.

The invention has been described in detail. One skilled in the art will recognize that at least one of the following embodiments can be provided or at least one of the following advantages can be achieved with the disclosed teachings.

In order to carry out intuitive programming, the creator, ie the expert, of a new application wears a camera or another sensor which can record the environment and the action of the application creator. To create a new application, the creator can then carry out actions as they should appear in the later application. He can carry out assembly work, carry out work on a machine, rectify errors or the like. All of these gestures or other actions are recognized by the camera or sensor system and translated into an augmented reality program. As a result, the creator of a new application carries out the activities to be programmed quite normally and thus intuitively creates a new program. reference list

contraption

input interface

analysis unit

output interface

system

operating unit

registration unit

sensor suit

sensor glove

storage unit

Augmented Reality (AR) display device

expert

Attachment

Working on a machine

Fix bugs

training an apprentice

Other activities

translation

Augmented reality program procedural steps

Claims

Device (10) for creating work instructions for an expert (32) for an activity, with:

- An input interface (12) for receiving expert data comprising operating data and/or sensor data with information about a behavior of the expert (32) during the activity;

- An analysis unit (14) for determining an action of the expert (32) based on the expert data and determining a work instruction for the work instruction;

- An output interface (16) for transmitting the work instructions to a storage unit (28), characterized in that the analysis unit (14) is designed to determine work instructions for the activity of the expert (32) from one or more work instructions. Device (10) according to claim 1, wherein the analysis unit (14) is designed to translate the behavior of the expert (32) during the activity into an augmented reality program (46) and the work instructions an augmented reality application, including the augmented reality program (46). Device (10) according to one of the preceding claims, wherein the analysis unit (14) is adapted to interpret an action of the expert (32) and based on the interpretation of the action of the expert (32) to determine the action. Device (10) according to one of the preceding claims, wherein the analysis unit (14) is designed to determine the action of the expert (32) using artificial intelligence. Device (10) according to any one of the preceding claims, wherein the input interface (12) is designed to receive expert data, comprising camera data from a camera which detects an action by the expert (32); lidar data of a lidar detecting an action of the expert (32); glove data, a sensor glove (26) detecting an action of the expert (32) and/or receiving suit data of a sensor suit (24) detecting an action of the expert (32). Device (10) according to one of the preceding claims, wherein the analysis unit (14) is designed to recognize one or more of the following activities of the expert (32); Expert assembly activities, carrying out work on a machine (36), troubleshooting (38) and/or setting up a machine. Device (10) according to one of the preceding claims, wherein the input interface (12) is adapted to receive expert data comprising recognized gestures of the expert (12); the analysis unit (14) is designed to determine the gestures and to determine a representation corresponding to the gestures; and the output interface (16) is designed to transmit a control command to an AR display device (30), in particular VR glasses of the expert (32), the control command being an output of the determined representation on the AR display device (30) causes. System (18) for creating work instructions for an expert (32) for an activity, with:

- A device (10) according to any one of the preceding claims;

- a sensor for acquiring expert data;

- a storage unit (28) for storing the work instructions and

- preferably a VA display device (30). Method for creating work instructions for an expert (32) for an activity, comprising the steps:

- Receiving (S1) of expert data comprising operating data and/or sensor data with information about a behavior of the expert (32) during the activity;

- Determining (S2) an action of the expert (32) based on the expert data;

- Determining (S3) a work instruction for the work instruction based on the expert data and determining a work instruction for the activity of the expert (32) from one or more work instructions and

- Transmission (S4) of the work instructions to a storage unit (28). 14 Computer program with program code means to carry out all the steps of a method according to the preceding claim 9 when the computer program is executed on a computer or a corresponding computing unit.