DE102019220019A1 - PROCESS FOR ENRICHMENT WITH VIRTUAL EXPANSION MOTIVES - Google Patents

Abstract

The invention relates to a computer-implemented method for enriching a target image with virtual expansion motifs, which can be displayed to a user in the context of an augmented reality application in a predetermined spatial relationship to an object corresponding to the target image. The method comprises steps that are carried out within and with the aid of a layout and typesetting program for desktop publishing, in particular the creation of an expansion motif level for the target image, the arrangement of one or more expansion motifs on the expansion motif level, the creation of a Set of extension data for each of the extension motifs. Furthermore, the method comprises steps that are carried out within and with the help of an editor program for augmented reality applications, in particular simulating the display and / or the behavior of the augmentation motifs in the context of augmented reality application and / or modifying or expanding one or more of the sets of extension data.

Description

FIELD OF THE INVENTION

The present invention relates to a method for enriching a target image with virtual expansion motifs in the context of an augmented reality application that can be carried out by an end user on a portable terminal.

BACKGROUND OF THE INVENTION

Augmented reality applications are applications that can be executed on portable end devices. These are used to reproduce reality, for example by means of a camera contained in the end device, and to add virtual objects (expansion motifs), such as videos, links, more detailed information on the target image, etc. The end user of the application then has the option of interacting with the extension motifs in various ways, for example by clicking on them. In general, such an application recognizes a predetermined target image in reality and displays expansion motifs linked to the target image on the basis of links in a database.

To create objects in augmented reality, editor programs are known that allow target images from the real world, which are captured with a camera and viewed on the screen of a portable terminal, to add virtual, e.g. three-dimensional, objects with which the user of the terminal can interact.

These editor programs differ in their operation from layout and typesetting programs, such as "InDesign" or "Illustrator" from Adobe. These programs are used by those skilled in the field of media design for creating a layout in printed matter and are used, for example, to position graphic elements on a target image and to edit the overall design of a printed product. Such programs are well known to those skilled in the art and are experienced in their use.

Since the creator of a printed matter, for example a target image, is familiar with the use of a layout and typesetting program, but is usually not familiar with or used to using a program for creating objects in augmented reality, a skilled person would have to be incorporate into another program if he wishes to use augmented reality.

For this reason, it is desirable to provide a way of augmenting target objects with augmented reality objects in which the creator of a printed matter does not need to learn new skills relating to another program. It is therefore the object of the present invention to provide such a possibility.

SUMMARY OF THE INVENTION

The object according to the invention is achieved by a method having the features of claim 1. The dependent claims contain further aspects and preferred embodiments of the invention.

• - Innerhalb und mit Hilfe eines Layout- und Satzprogramms für das Desktop-Publishing:
  • ◯ Öffnen oder Erstellen des Zielbilds;
  • ◯ Erstellen einer Erweiterungsmotiv-Ebene für das Zielbild;
  • ◯ Anordnen von einem oder mehreren Erweiterungsmotiven auf der Erweiterungsmoti v-Ebene;
  • ◯ Erstellen eines Satzes von Erweiterungsdaten für jedes der Erweiterungsmotive, wobei die Erweiterungsdaten Informationen über die Anzeige des jeweiligen Erweiterungsmotivs im Rahmen der erweiterte-Realität-Anwendung enthalten;
  • ◯ Verknüpfen der Sätze von Erweiterungsdaten mit dem Zielbild;
  • ◯ Speichern der Sätze von Erweiterungsdaten und der Verknüpfung mit dem Zielbild;
• - Innerhalb und mit Hilfe eines Editorprogramms für erweiterte-Realität-Anwendungen, welches von dem Layout- und Satzprogramm verschieden ist:
  • ◯ Öffnen des Zielbilds und Auslesen der Verknüpfung der Sätze von Erweiterungsdaten mit dem Zielbild;
  • ◯ Simulieren der Anzeige und/oder des Verhaltens der Erweiterungsmotive im Rahmen der erweiterte-Realität-Anwendung in dem vorbestimmten räumlichen Verhältnis zu einem dem Zielbild entsprechenden Objekt und/oder
  • ◯ Modifizieren oder Erweitern von einem oder mehreren der Sätze von Erweiterungsdaten.

A first aspect of the present invention is a computer-implemented method for enriching a target image with virtual expansion motifs that can be displayed to a user in the context of an augmented reality application in a predetermined spatial relationship to an object corresponding to the target image, comprising the following steps :

• - Within and with the help of a layout and typesetting program for desktop publishing:
  • ◯ Open or create the target image;
  • ◯ Create an extension motif layer for the target image;
  • ◯ Arranging one or more expansion motifs on the expansion motif level;
  • ◯ creating a set of extension data for each of the extension motifs, the extension data containing information about the display of the respective extension motif in the context of the augmented reality application;
  • ◯ associating the sets of extension data with the target image;
  • ◯ storing the sets of extension data and linking to the target image;
• - Within and with the help of an editor program for augmented reality applications, which is different from the layout and typesetting program:
  • ◯ opening the target image and reading out the link between the sets of extension data and the target image;
  • ◯ Simulating the display and / or the behavior of the expansion motifs in the context of the augmented reality application in the predetermined spatial relationship to an object corresponding to the target image and / or
  • ◯ Modifying or extending one or more of the sets of extension data.

This makes it possible for the media specialist, who is well acquainted with the use of layout and typesetting programs, to add expansion objects to a target image without having to acquire further knowledge in relation to a further program. As many steps as possible are implemented in a program expansion of the program already known to the person skilled in the art. Only the steps that cannot be implemented in it, for example adding a third dimension to the expansion motifs, are carried out in the separate program. Since the layout and typesetting program is designed for only two-dimensional work, the third dimension must be added in another program. However, the invention minimizes the work in this other program so that the knowledge that the person skilled in the art additionally needs to acquire remains minimal.

According to a second aspect, in the first aspect the expansion data contain information about the spatial relationship in which the expansion motif is to be displayed in the context of the augmented reality application to an object corresponding to the target image. This can be used to define where an object of augmented reality should be arranged. This is already determined in the process of creation in the layout and typesetting program, so that the specialist, even if he has only superficial knowledge of the editor program for augmented reality, can reliably control these spatial links.

According to a third aspect, in the second aspect the spatial relationship is derived from the arrangement of the expansion motif in the expansion motif level in relation to the target image. A new level is defined for the extension motifs, in which the motifs are created. The anchoring in the plane then takes place relative to already existing planes, so that the objects in the augmented reality are finally displayed in the correct place relative to the actually existing printed product.

According to a fourth aspect, in the second or third aspect, the information relates to the spatial relationship in two dimensions (x, y).

According to a fifth aspect, in the fourth aspect, when one of the sets of expansion data is modified or expanded, information about the spatial relationship in which the expansion motif is to be displayed in relation to an object corresponding to the target image in the context of the augmented reality application is added to the expansion data a third dimension (z) added. This enables the person skilled in the art to carry out most of the activities in the layout and typesetting program known to him, while the last step, the addition of the third dimension, is carried out in the separate program. This step is not possible in the layout and typesetting program; therefore it is moved to the other program.

According to a sixth aspect, in one of the preceding aspects, when the sets of expansion data are stored and the linkage with the target image, the sets of expansion data are stored in a database in such a way that the sets of expansion data are linked to data about the target image. This enables the end user to call up the augmented reality objects when the target image is scanned by the portable terminal. The inserted extension motifs are accessed in the database using the target image.

According to a 7th aspect, in one of the preceding aspects, when the sets of extension data and the linkage with the target image are stored, information about the target image together with information about the arrangement of the extension motifs on the extension motif level is stored in a file, the format of which differs from that Layout and typesetting program for desktop publishing can be read. This enables a completed layout with augmented reality to be further processed in the layout and typesetting program by a person skilled in the art at a later point in time. This preserves the flexibility and offers the possibility, if the situation changes, of adding new information not only in the editor program for augmented reality applications, but also in the layout and typesetting program. There is also the possibility of developing a completely new design on the basis of an existing design.

According to an 8th aspect, in the 7th aspect, a tag is created for an extension motif arranged on the extension motif level, which tag is linked to at least a part of the set of extension data for the extension motif, and when the sets of extension data and the Linking with the target image, the tag and the extension data associated with it are saved together with in the file, the format of which can be read by the layout and typesetting program for desktop publishing. Such tags are provided by layout and typesetting programs such as InDesign to store various types of information. This feature can be used advantageously to relate the extension data to the respective extension motifs of the set of extension data and to enable this relationship to be passed on and stored.

According to a ninth aspect, in one of the preceding aspects, the user can interact with a displayed expansion motif within the framework of the augmented reality application, preferably in real time.

According to a 10th aspect, in the 9th aspect, a set of extension data for an extension motif contains information relating to the type of interaction and the information within and with the aid of the layout and typesetting program for desktop publishing and / or the Editor program for augmented reality applications can be added or modified to the set of augmentation data. With this aspect of the invention, the person skilled in the art can already define in the layout and typesetting program which type of interaction with an extension motif is possible by the end user. This does not necessarily require the editor program, which is why it can be integrated into the layout and typesetting program.

According to an 11th aspect, in one of the preceding aspects, the extension motifs are selected from a library of extension motifs which is made available to the user by the layout and typesetting program for desktop publishing.

According to a 12th aspect, in the 11th aspect, the library of expansion motifs can be filled within and with the aid of the layout and typesetting program for desktop publishing and / or within and with the aid of the editor program for augmented reality applications. Similar to the aforementioned aspects, this ensures that the person skilled in the art can carry out the largest possible part of his work within the layout and typesetting program that is well known to him.

According to a 13th aspect, in one of the preceding aspects, an extension motif has a type which defines attributes, the values of which can be determined by the set of extension data for the extension motif. This enables different expansion motifs to serve different purposes. For example, video objects, links, product information objects, contact information objects or other objects that require different attributes can be integrated. These attributes are necessarily dependent on the type of object - for example a video link for video objects, contact information for contact information objects or product data for a product information object.

A 14th aspect of the present invention is a system of computer programs comprising instructions which configure a computer system in such a way that it carries out the method according to one of the 1st to 13th aspects.

Figure list

• 1 shows a program interface of a layout and typesetting program.
• 2 shows a program interface of a layout and typesetting program with an additional dialog for login.
• 3 shows a program interface of a layout and typesetting program with a dialog for selecting a campaign, a project and a goal.
• 4th shows a program interface of a layout and typesetting program with an input dialog for extension data and an element library.
• 5 shows a program interface of a layout and typesetting program in which extension motifs have been added to a finished target image.
• 6th shows the program surface of an editor program for augmented reality applications, which is executed on a portable terminal.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

1 shows schematically a program window 1 a layout and typesetting program, such as Adobe InDesign or Illustrator, with a title bar 2 , a toolbar 3 and a representation of an object to be set 4th . In the figure, the object to be set contains a text area 5 filled with text and an advertisement 6th that contains an advertisement for a product. This is an exemplary embodiment; the program window and the object to be set can be constructed entirely differently without departing from the scope of the invention.

The layout / the sentence is created in the layout and typesetting program, or an already created layout or created sentence is opened in the layout and typesetting program, whereby the layout / the sentence contains all the necessary levels for elements such as logos, images, punch, etc. . contains. From this layout / set (hereinafter: “document”) a target image for augmented reality applications is created, which is enriched in the layout and typesetting program with virtual expansion motifs. Since conventional layout and typesetting programs do not offer such an enrichment function, a button is used, for example 7th in the toolbar 3 a program extension ("plugin") is switched on, which is intended to enable the execution of the inventive To support procedural steps. The conventional layout / typesetting program is used in a known manner to create the target image. In addition to the known functions of the program, the plug-in enables the addition of a further "level" in the layout / typesetting program and the addition of expansion motifs to this additional level. The extension motifs are then linked with tags and / or other information. The extension motifs are provided by the plugin. The plug-in also uses the function of the conventional layout / typesetting program, which makes it possible to add so-called "tags" to graphic elements in order to save extension data for the extension motifs in these tags.

When the program extension is started, a window opens that asks the user to specify a server, for example a server provided by a manufacturer of the plug-in, by URL or IP address, to enter a valid user name and the associated password, and a button 9 to register with this ( 2 ).

3 shows a window 10 for the preparation of the Augmented Reality Document that will appear after successful registration. For example, the user can select a campaign, a project and a goal using the drop-down menus 11 selected or via the button 12th can be newly created. Different options are available depending on which user has logged on. The previously created elements of the document are combined on a locked document level, and a new level is created for expansion motifs. The dialog can then be opened by pressing the button 13th getting closed. The document is then saved in a file, the name of which is given a suffix, which indicates that it was created with the help of the program extension.

4th shows an element library 14th with expansion motifs available for augmented reality 15th which is opened and presented to the user on the screen. The available expansion motifs 15th differ according to the logged in user. The element library 14th showing the expansion motifs 15th contains, can with the help of the layout and typesetting program or with the help of an editor program for augmented reality with new expansion motifs 15th created and / or expanded. A window will also open 16 , the input fields 17th contains. The input fields are intended to allow the entry of extension data that is linked to extension motifs.

The expansion motifs 15th can be accessed by the user from the element library 14th be dragged into the newly created level and positioned there, as in 5 shown. From the position at which the extension motifs are inserted, the spatial relationship of the extension motif relative to the target image can be recognized, and this information is stored in the extension data of the extension motif. The spatial relationship of an expansion motif to the target image can be specified, for example, in two-dimensional coordinates (x, y).

For each placed extension motif 15th the user can preferably add additional extension data in one of the input fields 17th of the window 16 Add. Such additional extension data can on the one hand contain information about the representation of the extension motif in the augmented reality application, such as position, color, shadow, etc., and on the other hand can contain information about the interaction of an end user with the extension motif, such as a result of the Clicking on an extension motif, playing a specific video when the end user interacts, etc.

Each extension motif preferably has a type which defines attributes. The values of these attributes can be determined by the set of extension data. For example, an extension motif of the type “video” can have an attribute “URL”, which is defined in the extension data. An end user, who scans the target image by means of an augmented reality application, is shown the expansion motif in the spatial relationship to the target image, as it was determined, for example, with the help of the layout and statutes program. If the end user clicks on the extension motif, for example, the linked video can be displayed to him using the defined URL.

The extension data used can be called up by the user, for example via a button, a separate tab or via another window, so that the user always has an overview of the extension data assigned.

Are all expansion motifs 15th linked with extension data, the user can save the created target image with its extension motifs via a button and upload it to the database, in which these are saved.

The target image and the associated expansion motifs are stored in the database, together with the expansion data, so that when a corresponding target image is recognized by the terminal, the associated expansion motifs can be loaded with the associated expansion data.

In addition, the target image with the extension data can be saved in a file in a format that can be read by the layout or typesetting program. This ensures that the created target image and the extension motifs defined for it can be further edited by the user after saving.

A so-called tag can be created for each extension motif, which contains at least part of the set of extension data for the extension motif 15th contains. This tag is also saved in the file when the extension motifs and extension data are saved, the format of which can be read by the layout or typesetting program.

As in 7th shown, the user can switch to an editor program for augmented reality applications such as "Snooperator" after successful uploading. In this application, the user has the option of checking the target image and its enhancements, for example by simulating augmented reality in the editor program, modifying or further processing the target image and its enhancements, and publishing them.

The simulation of the display and / or the behavior of the extension motifs in the context of the augmented reality application in the predetermined spatial relationship to an object corresponding to the target image allows the user to display the extension motifs as they are displayed to an end user who has the corresponding Scans target image. The user can also test what happens in the augmented reality program when the end user interacts with the extension motif, e.g. clicks on the extension motif. Such a simulation is typically not possible in the layout and statutes program.

In the modification or further processing step, the user has the option of changing or moving the extension motifs or adding new extension motifs. He can also add extension data that cannot be created in the layout and statutes program. For example, in the editor program for augmented reality applications, the expansion data of the expansion motifs 15th Information about the spatial relationship in a third dimension (depth or z-axis) can be added. This enables the expansion motifs to be displayed to the end user in three dimensions in the augmented reality program. The depth is preferably only created in the editor program for augmented reality applications because it typically cannot be represented in the layout and statutes program because it only works two-dimensionally.

When all the data is stored in the database, an end user will be given the target image 4th in reality scans using an application that supports augmented reality on his end device 18th in addition to the target image 4th the expansion motifs displayed as in 8th you can see. For this purpose, images that are recognized by the camera of the terminal are compared with the target images in the database, and when a target image in the database is recognized, the linked expansion motifs including expansion data are provided. On his device 18th the end user can interact with these expansion motifs - preferably in real time.

Claims (14)

Computer-implemented method for enriching a target image with virtual expansion motifs, which can be displayed to a user in the context of an augmented reality application in a predetermined spatial relationship to an object corresponding to the target image, comprising the following steps: - Within and with the help of a layout and typesetting program for desktop publishing: ◯ Open or create the target image; ◯ Create an extension motif layer for the target image; ◯ Arranging one or more expansion motifs on the expansion motif level; ◯ creating a set of expansion data for each of the expansion motifs, the expansion data containing information about the display of the respective expansion motif in the context of the augmented reality application; ◯ associating the sets of extension data with the target image; ◯ storing the sets of extension data and linking to the target image; - Within and with the help of an editor program for augmented reality applications, which is different from the layout and typesetting program: ◯ opening the target image and reading out the link between the sets of extension data and the target image; ◯ Simulating the display and / or the behavior of the expansion motifs in the context of the augmented reality application in the predetermined spatial relationship to an object corresponding to the target image and / or ◯ Modifying or extending one or more of the sets of extension data. Procedure according to Claim 1 , wherein the expansion data contain information about the spatial relationship in which the expansion motif in the context of the augmented reality application to is to be displayed to an object corresponding to the target image. Procedure according to Claim 2 , the spatial relationship being derived from the arrangement of the expansion motif in the expansion motif plane in relation to the target image. Method according to one of the Claims 2 or 3 , the information relating to the spatial relationship in two dimensions (x, y). Procedure according to Claim 4 When modifying or expanding one of the sets of expansion data, information about the spatial relationship in which the expansion motif is to be displayed in the context of the augmented reality application to an object corresponding to the target image is added to the expansion data in a third dimension (z) . Method according to one of the preceding claims, wherein when the sets of expansion data are stored and the linkage with the target image, the sets of expansion data are stored in a database in such a way that the sets of expansion data are linked with data about the target image. Method according to one of the preceding claims, wherein when the sets of extension data and the link with the target image are stored, information about the target image together with information about the arrangement of the extension motifs on the extension motif level are stored in a file, the format of which depends on the layout and typesetting program for desktop publishing. Procedure according to Claim 7 , wherein a tag is created for an expansion motif arranged on the expansion motif level, which tag is linked to at least part of the set of expansion data for the expansion motif, and the tag and the extension data associated with it are saved together with in the file, the format of which can be read by the layout and typesetting program for desktop publishing. Method according to one of the preceding claims, wherein the user can interact with a displayed expansion motif within the scope of the augmented reality application, preferably in real time. Procedure according to Claim 9 , wherein a set of expansion data for an expansion motif contains information relating to the type of interaction, and the information within and with the aid of the layout and typesetting program for desktop publishing and / or the editor program for augmented reality applications Set of extension data can be added or modified. Method according to one of the preceding claims, wherein the extension motifs are selected from a library of extension motifs which is made available to the user by the layout and typesetting program for desktop publishing. Procedure according to Claim 11 The library of expansion motifs can be filled within and with the aid of the layout and typesetting program for desktop publishing and within and with the aid of the editor program for augmented reality applications. Method according to one of the preceding claims, wherein an extension motif has a type which defines attributes, the values of which can be determined by the set of extension data for the extension motif. A system of computer programs comprising instructions that configure a computer system to perform the method according to any of the Claims 1 to 13th executes.

Images