DE102022003091A1 - System for generating information or interaction elements - Google Patents

Abstract

The invention relates to a system (S) for generating information or interaction elements (2) on a display device or a display and operating device (5) in a vehicle (1). The system (S) is set up to: - read in a default element; - load a stored GAN model (M) corresponding to the default element; - read in default values of an input space (E) for the GAN model; - using the GAN Model (M) and the predetermined values of the input space (E) to generate information or interaction elements (2); and - display and/or save the generated information or interaction elements (2).

Description

The invention relates to a system for generating information or interaction elements on a display device or a display and operating device.

Information is typically displayed on display devices, in particular on display and operating devices such as touch screens, by using appropriate information elements, for example text, pictograms, colors, background images and/or colors or the like. In addition, at least some of these elements can be selected as interaction elements in the event of an additional operating option, be it as a touchscreen, via buttons attached to the edge of the display or the like. In principle, but especially in vehicles where information has to be recorded quickly and without being distracted from traffic for a long time, such an information and interaction element now plays a crucial role in terms of its comprehensibility. It can be advantageous if the information and interaction element has an appearance that is as realistic as possible and optimized for the respective user. Images that look as real as possible are usually easier to capture, at least intuitively and from the corner of your eye, than content that appears synthetic.

From the DE 10 2018 121 808 A1 is known a system in which synthetic data is refined using auxiliary inputs. The system there uses a so-called Generative Adversarial Network (GAN) and essentially serves to provide visual scenarios that appear as realistic as possible in order to be able to train autonomous vehicle applications, for example.

The object of the present invention is to provide a system for improved generation of information or interaction elements on a display device or a display and operating device in a vehicle.

According to the invention, this object is achieved by a system with the features in claim 1. Advantageous refinements and further developments of the system according to the invention result from the dependent claims.

The system according to the invention is used to generate information or interaction elements on a display device or a display and operating device in a vehicle. It is set up to read in a default element and to load a stored generative adversarial network corresponding to the default element, hereinafter referred to as a GAN model. Furthermore, it is set up to read in predetermined values of an input space of the so-called latent space for the GAN model. The information or interaction elements are then generated using the GAN model and the specified values of this input space. The information or interaction elements generated can then be displayed and/or saved via the system according to the invention.

The advantage of the system is that a user of the vehicle can adapt the information and interaction elements very individually to his or her own needs by selecting the default elements and, by using the GAN model, receives a realistic representation of the information or interaction elements, which make them very easy to understand for him. Thus, by using GAN-based models to generate corresponding information and interaction elements, which also represent design components in the vehicle, they can be individually adapted for the person using the vehicle with a high-quality appearance. This makes a new digital design experience possible in which individual adaptation can be carried out without using pre-installed designs, meaning that not only the possibility of adaptation within the given framework, but actual individual adaptation is possible.

According to a very advantageous embodiment of the system according to the invention, it can further be provided that the GAN model corresponding to the default element is pre-trained with a class of objects corresponding to the default element. The GAN model can therefore be pre-trained accordingly in order to be able to make the desired adjustments very efficiently.

According to a further very advantageous embodiment of the system according to the invention, this is characterized by a central main display and/or a combination instrument in the vehicle. This central main display and/or the instrument cluster can include at least one touch-sensitive operating area in order not only to display information but also to be able to interact with the user.

According to a very advantageous embodiment of the system according to the invention, the input space can be designed as a multi-dimensional vector, with a reading device being set up to read in individual values of the vector via gestures of the user. Such gestures can be used to display the multidimensional vector of the input be adjusted accordingly in its entirety or at least in some of its individual values. The reading device for this can, for example, include the touch-sensitive area, so that the gestures are, for example, swiping gestures on this touch-sensitive area. The touch-sensitive area can be a touchpad or can be combined with the display area accordingly, so that, for example, a touchscreen is used as a display and operating area in one, for example in the central area of a vehicle, in the area of the so-called head unit as the central main display.

The reading device can additionally or alternatively comprise at least one interior camera of the vehicle. The gestures for reading in individual values of the vector do not then have to be limited to a touchpad or touchscreen. Rather, it is known from the field of vehicles that gestures that are made without reference to a touch-sensitive control surface can also be recognized as control gestures. Such gestures are typically recorded, recognized and evaluated accordingly via one or more interior cameras in the interior of the vehicle. For example, a pointing gesture can be recognized and interpreted as such and can therefore be used to read in the individual values of the vector.

A further very favorable embodiment of the system according to the invention can also provide that when using a multi-dimensional vector as an input space or latent space, further individual values of the vector are specified in a fixed manner and/or depending on the default element. For example, a specification can be made in advance for certain values of the vector or such a specification can be made depending on the default value. Since the vector is dimensioned differently depending on the GAN model and application, necessary adjustments to the dimensions of the vector to the respective GAN model can be easily implemented through this fixed specification of certain individual values of the vector or an adjustment of several individual values at the same time.

According to a further very favorable embodiment, the central main display and/or the instrument cluster are set up to store the generated information or interaction elements. This can be done, for example, in a memory module within a controller, preferably linked to a user profile. This allows the system to reuse the information or interaction elements without any additional effort, preferably user-related, so that when the vehicle is used by a registered user, their individual design can always be used. It can then be developed further depending on the user.

According to a further very favorable embodiment of the invention, the display area of the central main display and / or the instrument cluster, for example the display of the head unit and the display or displays of an instrument cluster, referred to below as an IC cluster, are set up to provide a selection menu Display selection of the default element, input space and/or GAN model. Such a display shows the user which options are available, for example for the default element, the input space and/or the GAN model, and he is given the opportunity to influence this accordingly and make one or two selections in the menu to influence the system for generating the information or interaction elements according to its own wishes.

According to a very advantageous embodiment of the invention, the system is set up to enable its users to share the stored information or interaction elements with other users via a network. If the user finds one or more of the elements created to be particularly successful, particularly worth seeing or the like, he can share these information or interaction elements with other users via a corresponding network so that they can see the elements and use them themselves if necessary.

According to a very advantageous embodiment of the system according to the invention, the integration into the vehicle can be set up in such a way that the GAN model corresponding to the default element and in particular pre-trained is received by a server external to the vehicle. Without the effort of having to maintain different GAN models in the vehicle itself, the corresponding GAN model can be downloaded from a server external to the vehicle, for example a cloud server such as the vehicle manufacturer's backend server, and then used on the system in the vehicle.

An alternative to this may be a type of hybridized system in which one subsystem is integrated into the vehicle and one subsystem is external to the vehicle. The vehicle-external subsystem can then again be designed on a vehicle-external server and should include the at least one GAN model, so that a large number of complex GAN models do not have to be kept in the vehicle itself. Instead of downloading and Using the GAN model in the vehicle, as in the example described above, it is now possible for the vehicle-integrated subsystem to simply collect data from the vehicle-external subsystem, such as the default element and/or values of the input space, and transmit it to the other subsystem, so that the use of the GAN model can then take place within the vehicle-external subsystem. Since this is typically designed as a server external to the vehicle, it is possible to provide a correspondingly high level of computing power and computing capacity easily and cost-effectively.

Further advantageous refinements of the system according to the invention also result from the exemplary embodiment, which is described in more detail below with reference to the figures.

• 1 eine schematische Darstellung eines Generative Adversarial Networks;
• 2 eine schematische Darstellung eines GAN-basierten Systems zur Erzeugung von Informations- oder Interaktionselementen; und
• 3 eine schematische Darstellung des Zusammenspeils aus dem GAN-Modell und einer Interaktion eines Nutzers.

Show:

• 1 a schematic representation of a Generative Adversarial Network;
• 2 a schematic representation of a GAN-based system for generating information or interaction elements; and
• 3 a schematic representation of the combination of the GAN model and a user interaction.

Before the system S presented here (cf. 2 ) is described in more detail, the underlying deep learning architecture is first explained for a better understanding. 1 represents a Generative Adversarial Network (GAN) in simplified form and is intended to explain the principle of this architecture. The goal of a GAN is to generate new content (e.g. images) that is similar to a real training data set. In order to generate this content and decide whether the artificial data set has essential properties of the training data, two main components are essentially required. These two networks based on deep learning are, on the one hand, a generator G and, on the other hand, a discriminator D. The generator G has the task of generating new content. This new content is in 1 shown dashed with F. For example, noise N together with an output O of a so-called input space E or latent input space can serve as input. This input space E can, as indicated by the circles, be a multi-dimensional vector whose individual values indicated by the circles deliver the corresponding output O. Which individual values of the vector influence which properties of the generated output O can only be determined exploratively/heuristically.

The newly generated data F then goes as input into the discriminator D. This network is trained with the help of real input data sets R and the generated data F. A probability P is calculated, which indicates whether the data F corresponds to a real or artificial input (real or fake). The goal is to generate the data F in such a way that the discriminator D considers the generated data F to be real. The information as to whether the data F is viewed as real or not real goes back into the neural networks as feedback FB, so that, for example, the generator G learns to generate more real data F. How the networks are trained and the problems associated with it should not be explained in this short description. However, this is known to the relevant expert. Ultimately, the training should lead to a GAN model M, which can generate and output a real-looking data set F based on an input space E in the form of a vector, as shown on the far right 1 is shown.

Based on 2 and 3 The GAN-based system S will be explained below, which is completely or at least partially on-board in a vehicle 1 (cf. 3 ) should be used. It is intended to be used to create individual information or interaction elements 2, which are in the 2 and 3 are indicated by way of example as icons and provided with the reference number 2. You could also call them user interface designs. In summary, the system S should contain at least the following elements in whole or in part: A pre-trained GAN model M#n is created locally in a backend 3 (cf. 3 ) of the vehicle 1 installed or, if necessary, downloaded into it from external to the vehicle 1. The GAN model M#n itself, which expects a latent input space as input space E, generates 4 new information or interaction elements 2, which are also referred to as user interface elements, depending on the selection made by a person using the system S. These allow the person (user) 4 to create highly individual designs of the information or interaction elements 2. This can, for example, include the entire display of a display or display and operating device 5, which is also referred to as user interface 5. This user interface 5 can, as in 3 is indicated, for example include a central main display 5.1 and/or an instrument cluster 5.2, which are also referred to as a head unit or IC cluster. This could e.g. B. an adjustment of the speed display together with color tone and gradients as well as fonts, etc. can be done. But other elements can also be individually selected and/or created, e.g. fonts, icons for navigation, for media, for mobile phones, etc. The information or interaction elements generated 2 can be displayed and then saved and shared with other users.

2 shows the person 4 using the vehicle 1. They can drive the vehicle 1, for example. It represents user 4. His interaction with the GAN models M#1, M2#, ... M#n and the input space E is indicated by the dotted arrows. In the first step, the user selects 4 information or interaction elements 2 that he would like to change or adapt individually. The example shown here involves information or interaction elements 2 in the form of icons, which represent navigation, the telephone, media and radio in the user interface 5. Accordingly, the GAN model M corresponding to the information or interaction elements 2, here the GAN model M#2, is selected. The selected GAN model M#2 is loaded into the system S. It now expects input from user 4. This input takes place via inputs in the input space E. What these inputs look like is described in more detail below.

The GAN model M#2 itself, which is appropriately selected here, is a pre-trained model which is preferably executed locally in the vehicle 1. Appropriate design elements, icons, etc. (depending on the model) are used as training data. With the help of the GAN models M, the user 4 can now have realistic design elements generated by the GAN (Generated Design Element(s)) generated as information or interaction elements 2. The user 4 selects individual values within the permitted input space E. These values go in real time as input into the system S or the GAN model M loaded into the system S. Based on the values, the GAN model M generates new design elements (Generated Design Element(s)). The user 4 can change the input space E interactively. The interaction of the user 4 with the input space E then generates an output of data F in real time, here in the form of the icons mentioned as information or interaction elements 2. The user 4 can then use the generated information or interaction elements 2 for the display can save them, assign them to his profile and share the information or interaction elements 2 accordingly in order to show them to other users or make them available to them.

3 shows the interaction between the GAN model M and the interaction with the user 4. The schematic representation is divided into the backend 3 already mentioned above and a frontend 6, which together form the system S. In backend 3, a user 4 -in 3 represented by his hand - selected, design element-dependent GAN model M, here again the GAN model M#2, loaded. The interaction with the frontend 6 is indicated by the arrows and is intended to represent the input or output of the GAN model M. The front end 6 is divided into two components 5.1 and 5. 2 of the user interface or the display or display and operating device 5. These components can, for example, be the central main display 5.1 and the instrument cluster 5.2 in front of a steering wheel 7 of the vehicle 1.

The input space E or latent input space described above is a multidimensional vector whose individual values lead to a corresponding output O. So that the user does not have to define this multidimensional vector manually with numbers, the values are set via an interactive interface LSN (Latent Space Navigator), e.g. on the central main display 5.1. For example, in the case of a touch-sensitive display surface, the X/Y coordinates of the LSN are adopted as individual values for the vector of the input space E and communicated to the GAN model M.

Since the vector is dimensioned differently depending on the GAN model M and the application, certain individual values of the vector can be specified or several individual values can be adjusted at the same time. Using a SUB submenu, users can select 4 different input space solutions or switch between different design elements and the corresponding GAN models M. The resulting information or interaction elements 2 generated by the selected GAN model in the form of design elements, icons, etc. are used in the instrument cluster 5.2 and/or in the central main display 5.1. The design changes of the information or interaction elements 2 take place in real time; depending on the setting of the submenu SUB, as well as the position of the LSN and the resulting (individual) values of the input space E.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102018121808 A1 [0003]

Claims (13)

System (S) for generating information or interaction elements (2) on a display device or a display and operating device (5) in a vehicle (1), which is set up for this purpose: - read in a default element; - to load a stored GAN model (M) corresponding to the default element; - read in specified values of an input space (E) for the GAN model; - to generate information or interaction elements (2) using the GAN model (M) and the specified values of the input space (E); and - display and/or save the generated information or interaction elements (2). System (S) after Claim 1 , characterized in that the GAN model (M) corresponding to the default element is pre-trained with a class of objects corresponding to the default element. System (S) after Claim 1 or 2 characterized by a central main display (5.1) and/or a combination instrument (5.2) as a display device or a display and operating device (5). System (S) after Claim 3 , characterized in that the central main display (5.1) and/or the instrument cluster (5.2) comprise at least one touch-sensitive operating area. System (S) according to one of the Claims 1 until 4 , characterized in that the input space (E) is designed as a multi-dimensional vector, with a reading device being set up to read individual values of the vector via gestures of a user. System (S) after Claim 4 or 5 , characterized in that the reading device includes the touch-sensitive operating area. System(S) after Claim 5 or 6 , characterized in that the reading device comprises at least one interior camera of the vehicle (1). System (S) after Claim 5 , 6 or 7 , characterized in that further individual values of the vector are fixed and/or specified depending on the default element. System (S) according to one of the Claims 3 until 8th , characterized in that the central main display (5.1) and/or the instrument cluster (5.2) are set up to store the information or interaction elements (2), in particular in a user profile. System (S) according to one of the Claims 3 until 9 , characterized in that a display area of the central main display (5.1) and/or the instrument cluster (5.2) is set up to provide a selection menu (SUB) for selecting the default element, the input space (E) and/or the GAN model (M). to display. System (S) according to one of the Claims 1 until 10 , characterized in that it is set up to share the stored information or interaction elements (2) with other users via a network. System (S) according to one of the Claims 1 until 11 characterized by the integration into the vehicle (1), the system (S) being set up to receive the pre-trained GAN model (M) corresponding to the default element from a server external to the vehicle. System (S) according to one of the Claims 1 until 11 characterized by a vehicle-integrated subsystem and a vehicle-external subsystem, which is designed on a vehicle-external server and which includes the at least one GAN model (M).

Images