DE102022133569A1 - Method for generating data groups for use in training with augmented reality or with mixed reality, measuring device for use in the method and computer software product - Google Patents

Abstract

A method for generating data groups for use in training with augmented reality or with mixed reality includes using a measuring device (1), which records its own position and/or orientation in three-dimensional space, in addition to a device for generating image data.

Description

The invention relates to a method for generating (in particular by an authoring system) data groups for use in training with augmented reality or with mixed reality, which involves training for work on an object, in particular for assembly work and/or disassembly work on an object Motor vehicle (in the construction and/or maintenance and/or repair of such).

Augmented Reality takes a user through their real world and a virtual world at the same time. Several senses of the user can be addressed. For example, glasses are put on for the eyes, which enable the user to see the real world at the same time and superimpose computer-generated images. Authoring systems, especially for training, make it possible to provide additional data (additional information) in addition to the images that are displayed. When training for a montage, the user is to be shown a video and/or an image and/or text and/or a three-dimensional model when viewing a training object, depending on the location he is viewing, with the video, the image, the text or the model instructs him what to do at this point during assembly. The relevant (virtually available) data is typically assigned to a reference point (or a reference area). (The following specification of a reference point always means that an area containing the reference point can also be meant).

As part of the generation of augmented reality images, the DE 10 2009 056 013 A1 that the position of the user is determined using the camera carried by him. Also in the method and system for enabling the generation of augmented reality content according to the EP 2 560 145 A2 there is a so-called tracker in a user device. In the system according to the EP 3 567 341 A1 it is also about measuring distances. The aforementioned publications disclose the use of measuring devices, in particular for the position when generating the augmented reality images for the user.

A completely different question is that of making the preparations for augmented reality training at all. Here, too, the measurement of positions can be helpful. So far, tablet computers have been used to photograph and/or film a model of the object to be processed. From the company AR-Experts GmbH, Erlenweg 16, 71277 Rutesheim, it is known to use a tablet computer provided with LIDAR technology. "LIDAR" stands for "light detection and ranging" or "laser imaging, detection, and ranging"), it is a scanning of the environment using laser beams for optical distance measurement and the like. Using LIDAR technology, distance measurements can also be taken at the same time for a sample object that is to be photographed or filmed by the tablet computer. The corresponding data (image data) are linked internally with the data for the distance measurement. The so-called GIRI app is presented at https://www.ar-giri.com, with the help of which work instructions can be created in augmented reality.

The disadvantage of the technology used is that since the distance is measured by the same device that takes the photographs, the resolution regarding the distance is a few centimeters, which is not sufficient, especially for desired screwing processes with small distances between the screw holes. The LIDAR technology also has the disadvantage that shoring positions are partially covered or in gaps ("shading"). If the assembly consists of blind shoring in a cavity, the shoring positions are even completely covered ("concealment"). In the case of complex surfaces, the shoring position must be determined precisely. All of these are challenges that cannot be solved sufficiently well using LIDAR technology.

It is the object of the invention to indicate a way in which a method for generating data groups for use in training with augmented reality or with mixed reality can be improved.

The object is achieved by a method with the features according to patent claim 1, by a measuring device with the features according to patent claim 6 and by a computer software product with the features according to patent claim 11.

• - Bereitstellen eines Messgeräts, das ausgelegt ist, seine eigene Position und/oder Orientierung (im dreidimensionalen Raum) zu erfassen und entsprechende Messdaten auszugeben (bei Position und Orientierung sind dies 6D-Messdaten);
• - Halten des Messgeräts an oder auf einen Bezugspunkt eines Musterobjekts;
• - Erzeugen von Bilddaten zu dem Musterobjekt (vorzugsweise derart, dass das Messgerät ebenfalls zumindest partiell abgebildet ist; Ausnahme: Messung in Hohlräumen, wo komplette Verdeckung existiert);
• - Messen der Position und/oder Orientierung des Messgeräts an oder auf dem Bezugspunkt des Musterobjekts und Ausgeben der Messdaten;
• - Empfangen der Messdaten;
• - Verknüpfen der Messdaten mit den Bilddaten zur Schaffung einer Datengruppe.

The method according to the invention includes:

• - Providing a measuring device that is designed to record its own position and/or orientation (in three-dimensional space) and to output corresponding measurement data (in the case of position and orientation, this is 6D measurement data);
• - stopping the measuring device at or on a reference point of a sample object;
• - Generation of image data for the sample object (preferably such that the measuring device is also at least partially imaged; exception: measurement in cavities where there is complete occlusion);
• - Measuring the position and/or orientation of the measuring device at or on the reference point of the sample object and outputting the measurement data;
• - receiving the measurement data;
• - Linking the measurement data with the image data to create a data group.

The invention separates the function of the measuring device from the function of the device for generating image data and thus enables a more precise measurement than with technologies in which the distance measurement (e.g. by LIDAR) is carried out at the user device (tablet computer).

The invention has the advantage that the measuring device can be brought to shadowed or covered installation positions and precisely indicates the place where action is to be taken on complex surfaces (example: indication of a slot for an electrical fuse).

According to a preferred embodiment of the invention, the measuring device has a device for classifying the reference point. More preferably, this is an input device. The measuring device can already indicate which type of reference point is meant.

Examples of such reference points are a target assembly position for a component to be assembled, a screw point, a cable duct or a clip point.

According to a further preferred embodiment, information is additionally assigned to the reference point in the linked data (i.e. the data group), i.e. (virtually available) data, in particular as a video and/or as an image and/or as text and/or as a 3D Model.

According to a further preferred embodiment, the image data is obtained using a portable device, in particular a tablet computer or a smartphone. This has the advantage of quick handling. The portable device is inherently distinct from the meter. Correspondingly, the portable device receives the measurement data and combines them with the image data.

The measuring device according to the invention comprises a base body and a pointer. (The pointer may also be referred to as a "tip" that protrudes from the body). The measuring device is designed to determine a position determined by the pointer and/or its own orientation in three-dimensional space and to output corresponding measurement data, in particular to transmit them wirelessly (the latter via radio, ^Bluetooth® , WLAN, . . . ).

Due to the shape of the measuring device with base body and pointer, an assistant can specify or determine a point ("the reference point") very precisely.

In a preferred embodiment, the measurement data are output when an input device is actuated. This prevents data overload from occurring on the part of the device generating the image data. Electrical energy is also saved in the measuring device, which would otherwise have to be provided by a particularly large accumulator. A smaller accumulator is sufficient here. If necessary, electrical energy can even be generated by actuating the input device, which energy can be used for the measurement and for sending out the data.

The measuring device can have a number of input devices, with the aid of which information about an object determined by the pointer is additionally output. These can be input devices that simultaneously cause the measurement data to be output and also output a code number, for example 01: screw location, I0: cable duct, and the like. The input devices can be freely assignable and possibly programmable, but preferably the input devices simply output a code (such as the above-mentioned code numbers "0I, I0 etc."), and their interpretation is carried out by the device that captures the image data (tablet computers and the like).

According to a further preferred embodiment of the measuring device according to the invention, the position and/or orientation is repeatedly measured and output when an input device is actuated for a longer period of time. In this way, entire contours can be scanned and their shape transmitted. An example of this is the location of a seal, such as on a tailgate and the like.

It can be helpful if false information is deleted again. Naturally, this can take place on the device that generates the image data and to which the measurement data are transmitted. Alternatively or additionally, it is provided that the measuring device has an input device for revoking a previous input. In this way, the assistant who guides the measuring device to the corresponding points on the sample object can react more quickly if he has made a mistake.

The computer software product according to the invention (stored as usual on a data storage cher device) is intended for a data processing device, in particular for a tablet computer or a smartphone, and the computer software product is used to implement a computer program; after uploading the computer software product to a working memory and putting it into operation, the computer program is made to run. In order to implement an authoring system, the computer program links image data (which may have been recorded by the data processing device itself) with measurement data received externally from the data processing device. (The image data is subjected in particular to image recognition, where preferably the tip of the above-mentioned measuring device ("pointer") or another structure on another measuring device is recorded. The corresponding location and, if necessary, the orientation of this tip or the other feature are then determined assigned measurement data.)

For applications or application situations that can arise in the process and which are not explicitly described here, it can be provided that according to the process, an error message and / or a request for user feedback is issued and / or a default setting and / or a predetermined initial state is set.

The motor vehicle is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

As a further solution, the invention also comprises a computer-readable storage medium, comprising instructions which, when executed by a computer or a computer network, cause this to execute an embodiment of the method according to the invention. The storage medium can e.g. at least partially as a non-volatile data store (e.g. as a flash memory and/or as an SSD - solid state drive) and/or at least partially as a volatile data store (e.g. as a RAM - random access memory). . However, the storage medium can also be operated, for example, as a so-called app store server on the Internet. A processor circuit with at least one microprocessor can be provided by the computer or computer network. The instructions may be provided in binary or assembler and/or programming language (e.g., C) source code.

The invention also includes the combinations of features of the described embodiments. The invention also includes implementations that each have a combination of the features of several of the described embodiments, unless the embodiments were described as mutually exclusive.

• 1a eine Draufsicht auf das erfindungsgemäße Messgerät;
• 1b eine Seitenansicht des erfindungsgemäßen Messgeräts;
• 2 eine Momentaufnahme der Nutzung des Messgeräts gemäß einer Ausführungsform des erfindungsgemäßen Verfahrens; und
• 3 die Schritte gemäß einer Ausführungsform des erfindungsgemäßen Verfahrens.

Exemplary embodiments of the invention are described below. For this shows:

• 1a a top view of the measuring device according to the invention;
• 1b a side view of the measuring device according to the invention;
• 2 a snapshot of the use of the measuring device according to an embodiment of the method according to the invention; and
• 3 the steps according to an embodiment of the method according to the invention.

The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that are to be considered independently of one another and that each also develop the invention independently of one another. Therefore, the disclosure is also intended to encompass combinations of the features of the embodiments other than those illustrated. Furthermore, the described embodiments can also be supplemented by further features of the invention that have already been described.

In the figures, the same reference symbols designate elements with the same function.

a in 1a The measuring device shown (also referred to as "tagger" or "tracker") is denoted by 1 as a whole. The measuring device 1 comprises a base body 10 and a tip 12 as a pointer, which 1a has approximately half the length of the body, but can also be a little shorter or much longer. The tip 12 may have a length of between 0.25 and 3 times that of the body 10 . As in 1b (Side view) to see, and shown there symbolically, the measuring device 1 includes a sensor 102 for detecting the position and orientation of the measuring device.

The measuring device 1 also includes means 104 for the wireless transmission of measurement data.

The sensor 102 is coupled to this device 104 in a manner not shown in the figures. In the present case, the measuring device 1 has four keys 106a, 106b, 106c and 106d, the function of which can be freely assigned. Depending on the key pressed, a different code is transmitted. For button 106a, the value "00" is transmitted ("0" in decimal). system), for key 106b the value "0I" is transmitted ("1" in decimal system), for the key 106c the value "I0"("2" in decimal system) is transmitted, and for the key 106d the value " II” (“3” in the decimal system). The buttons can be assigned in such a way that a measured value is transmitted in any case, or the measured value can be transmitted by the actuation of the input device 110 on the back of the device 1 (seen in 1b) be dependent. The button 106a can be pressed when specifying the location for a screw (see also the illustrated symbol). Button 106b can be pressed when cables are to be inserted into a cable duct. Button 106c can be pressed when a clip position is to be displayed. The button 106d can be pressed when the location for mounting a component is to be shown.

A previous actuation can be revoked by pressing the additional button 108, so that the previous information is canceled at the receiving end.

The 2 FIG. 12 now represents a kind of snapshot as it could be created when performed according to an embodiment of the method for creating a data group.

It is about generating data groups for use in training with augmented reality (or possibly with mixed reality) for assembly work during the construction and/or maintenance and/or repair of a motor vehicle. The motor vehicle is denoted by 100 as a whole. A first person 200 (in the hand shown here) holds the measuring device 1 described above at a predetermined point. A photo of the situation is taken by a second person 300 using a tablet computer 2 . The display 20 on the tablet computer 2 shows the image of the motor vehicle at a first location 100a-img and at other locations such as 100b-img and the image of the measuring device 1 as 1 img.

The tablet computer now subjects the image recorded in this way to image data recognition and thus recognizes the image 1img of the measuring device 1 in the image, in particular also of its tip. The person 200 then presses the input device 106a, 106b, 106c or 106d, possibly together with the input device 110, and transmits the measurement data to the tablet computer. The tablet computer links this measurement data with the image data.

• Das Verfahren beginnt mit dem Schritt S00, dem Bereitstellen eines Musterobjekts (Kraftfahrzeug für Schulungszwecke, gegebenenfalls nur teilweise aufgebaut), eines Tabletcomputers (wie dem Tabletcomputer 2 in 2) und des Messgeräts 1 aus 1a/b. Im Schritt S10 wird das Messgerät 1 an oder auf den Bezugspunkt des Musterobjekts (des Kraftfahrzeugs) gehalten. Im darauffolgenden Schritt S12 hält die Person 300 den Tabletcomputer ebenfalls in Richtung des Kraftfahrzeugs. Im Schritt S14 wird an dem Messgerät eine der Tasten 106a, 106b, 106c, 106d, gegebenenfalls mit der Eingabevorrichtung 110, gedrückt. Im optionalen Schritt S16 bleibt die Taste über eine Zeitdauer Δt gedrückt; dies ermöglicht es, neben einer einzelnen Position auch eine Kontur zu bestimmen. Während der Zeitdauer Δt werden die Messdaten mehrfach (etwa in zeitlichen Abständen von wenigen Millisekunden, insgesamt möglicherweise 20 bis 100 Messdaten während Δt) gedrückt. Im Schritt S18 erfolgt der Empfang der Messdaten durch den Tabletcomputer 2. Im optionalen Schritt S20 wird geprüft, ob die Messdaten tatsächlich zu den Bilddaten passen. Diese Prüfung kann beinhalten, dass die Taste 108 gedrückt wird und die Daten widerrufen werden. Es kann auch beinhalten, dass die Person 300 am Tablet die Daten händisch löscht. In der Regel sind die Daten in Ordnung, und es wird zum Schritt S22 übergegangen. Im Schritt S22 können Zusatzdaten mit dem Punkt verknüpft werden, etwa ein Video, in dem die Person 200 zeigt, wie eine Schraube in ein vorher gezeigtes Schraubgewinde einzuschrauben ist oder wie Kabel in einen Kabelkanal zu führen sind. Nachfolgend ist das Verfahren entweder abgeschlossen und geht zum Ende S100 über oder es wird zum Schritt S10 nochmals zurückgekehrt.

An embodiment of the method according to the invention is also in 3 illustrates:

• The method begins with step S00, the provision of a sample object (motor vehicle for training purposes, possibly only partially assembled), a tablet computer (such as tablet computer 2 in 2 ) and meter 1 1a / b. In step S10, the measuring device 1 is held at or on the reference point of the sample object (the motor vehicle). In the subsequent step S12, the person 300 also holds the tablet computer in the direction of the motor vehicle. In step S14, one of the keys 106a, 106b, 106c, 106d on the measuring device is pressed, if necessary with the input device 110. In optional step S16, the button remains pressed for a period of time Δt; this makes it possible to determine not only a single position but also a contour. During the period Δt, the measurement data are pressed several times (e.g. at intervals of a few milliseconds, a total of possibly 20 to 100 measurement data during Δt). In step S18, the tablet computer 2 receives the measurement data. In the optional step S20, it is checked whether the measurement data actually match the image data. This check may include pressing button 108 and revoking the data. It can also include the person 300 manually deleting the data on the tablet. As a rule, the data is correct, and a transition is made to step S22. In step S22, additional data can be linked to the point, for example a video in which the person 200 shows how to screw a screw into a previously shown screw thread or how to route cables into a cable duct. The method is then either completed and proceeds to the end S100, or a return is made to step S10 again.

On in the 3 In a manner not shown, the optional step S20 returns to one of the previous steps, for example S10, for example S12, for example S14, if the data received in step S18 were not correct.

Overall, the examples show how a hardware-supported authoring process for AR training can be provided using an active tracker.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102009056013 A1 [0003]
• EP 2560145 A2 [0003]
• EP 3567341 A1 [0003]

Claims (11)

Method for generating data groups for use in training with augmented reality or with mixed reality, which is training for working on an object, in particular for assembly work and/or disassembly work on a motor vehicle (100), the method comprising: - Providing a measuring device (1) which is designed to detect its own position and/or orientation and to output corresponding measurement data; - Holding the measuring device (1) at or on a reference point of a sample object (100); - Generating image data for the sample object (100); - Measuring the position and/or orientation of the measuring device (1) at or on the reference point of the sample object (10) and outputting the measurement data; - receiving the measurement data; - Linking the measurement data with the image data to create a data group. procedure after claim 1 , characterized in that the measuring device has a device, preferably an input device (106a, 106b, 106c, 106d) in order to classify the reference point, that corresponding information is transmitted, received and linked to the image data and the measurement data in the data group becomes. procedure after claim 1 or 2 , characterized in that the reference point is a target assembly position for a component to be assembled and/or a screw point and/or a cable duct and/or a clip point. Procedure according to one of Claims 1 until 3 , characterized in that in the data group the reference point is additionally assigned data, in particular as a video and/or as an image and/or as text and/or as a 3D model. Procedure according to one of Claims 1 until 4 , in which the image data are obtained by means of a portable device, in particular a tablet computer (2) or smartphone, and wherein the portable device (2) receives the measurement data and links them to the image data. Measuring device (1) with a base body (10) and a pointer (12), the measuring device being designed to determine a position determined by the pointer and/or its own orientation in three-dimensional space and to output corresponding measurement data, in particular to transmit them wirelessly. Measuring device (1) after claim 6 , characterized in that the measurement data are output when an input device (110; 106a, 106b, 106c, 106d) is actuated. Measuring device (1) after claim 7 with a plurality of input devices (106a, 106b, 106c, 106d), information about an object determined by the pointer (12) being additionally output. Measuring device (1) after claim 7 or 8th , characterized in that when an input device (110) is actuated for a longer period of time, the position and/or orientation is repeatedly measured and output. Measuring device (1) according to one of Claims 7 until 9 with input means (108) for revoking a previous input. Computer software product for a data processing device, in particular for a tablet computer and/or a smartphone, the computer software product being designed to implement a computer program that links image data with measurement data externally received from the data processing device to implement an authoring system.

Images