DE102020121565A1 - Device and method for interacting with a graphical user interface and for testing an application - Google Patents

Abstract

The invention relates to a device (1) and a method for interacting with a graphical user interface and for testing an application. A voice command is provided from which a text element is identified. A lexical content is determined for an interactive element (22, 23, 24) of the user interface. A semantic degree of similarity of the text element to the lexical content is determined and the interactive element (22, 23, 24) is then interacted with depending on the degree of similarity.

Description

The invention relates to a method for interacting with a graphical user interface, the graphical user interface having an interaction element.

This is to be understood in such a way that the user interface has exactly one or more than one interaction element. As a rule, the user interface will have a large number of interaction elements. An interaction element is an interactive element with a graphical representation, that is, an element that can be displayed and interacted with to trigger an action. The interaction element can therefore in particular be what is known as a “widget”. For example, an interaction element can be clickable. An action is triggered when the interaction element is clicked. There are interaction elements that require input, drop-down menus, and many other types of interaction elements.

Provision can be made for the interaction with the user interface to control functionalities of an application that uses the user interface as an interface. In particular, provision can be made for a program flow of the application to be controlled by triggering an action by way of an interaction with the interaction element. An interaction with an interaction element can in particular trigger an action that transfers the state of the application and/or the user interface to a different state.

An interaction can also be referred to as an action taken.

An application can be characterized in particular by the fact that it is a software application or a computer program that performs a function that is useful for the user.

Graphical user interfaces, often also referred to as graphical user interfaces, are ubiquitous. Interaction often occurs by the user clicking on graphically displayed interactive elements with a computer mouse or his fingers and making entries using a keyboard. This is cumbersome.

There is therefore a need for more ergonomic interaction options.

In recent years, voice controls have been developed that can improve operation. Language assistants usually go a different way here and do not interact with a graphical user interface, but trigger functionalities of the application in a different way.

Direct voice control of graphical user interfaces is only implemented very rudimentarily these days. It is known to remotely control graphical user interfaces by using voice commands. The voice commands only replace manual operation by uttering commands such as "move the cursor to" or "click on the button labeled <Name>", where <Name> is specifically specified. Flexible voice control is therefore not possible since the user must speak the correct terms and navigate through the graphical user interface in the same way as with manual interaction.

The invention also relates to a method for testing an application using a user interface as an interface.

An interaction is facilitated in such an environment since the operating system with which the application is executed generally provides a software interface for interacting with the graphical user interface. Interactions that are normally performed manually can therefore be computer-aided via the software interface.

Application testing is of great practical importance. In addition to manually executed tests, automated test procedures are becoming more and more important because they can be carried out with less effort and create a high level of comparability.

It is known to use crawlers for this purpose, which essentially trigger randomized interactions with the graphical user interface and successively explore a large number of states of the user interface. Such an approach is slow and can only explore a fraction of the state space.

Alternative methods have recently been developed in which a complete test run for a test application is first programmed. This test run is then applied to an alternative application. With such an approach, a larger area of the state space of the application to be tested can be explored in the same amount of time compared to a randomized crawler, but the method is very complex because a complete test run has to be programmed for each individual test case.

Against this background, the invention is based on the object of creating a method for interacting with a graphical user interface and for testing an application that facilitates navigation through the graphical user interface. Furthermore, the aim is to reduce the time required to develop a test protocol for an application and to be able to test different applications with the same test protocol.

To the extent that variants of the invention are described below, they can be combined with one another as desired, provided that a combination is not ruled out for technical reasons.

The invention proposes the features of claim 1 in order to solve the aforementioned problem. In particular, it is therefore proposed according to the invention in a method for interacting with a graphical user interface of the type described at the outset to solve the stated task that a voice command is provided from which a text element is identified, that lexical content is determined for the interaction element, that a semantic Degree of similarity of the text element is determined to the lexical content and that is interacted with depending on the degree of similarity with the interaction element. This is to be understood in such a way that, in addition to the one identified text element, other text elements can also be identified. In addition to the one voice error, other voice commands can also be given. Both will even be the norm.

This greatly simplifies navigation through the graphical user interface. This enables voice control that is not tied to terms specified by the user interface. Due to the determination of a semantic degree of similarity between text elements of the voice command and interaction elements, syntactically completely dissimilar but semantically similar terms can be used. For example, it is possible to use the voice command "Sign me in" on a graphical user interface whose corresponding button is labeled "Login". This enables natural language control without having to use the terminology of the graphical user interface to be controlled.

This is useful not only for people who cannot recognize the terms due to poor eyesight, but also for sighted people since the interaction elements can be outside the displayed area and therefore not visible to the user at all times.

The voice command can consist of a single word, but it will usually be formed by a sequence of text elements. A text element can consist of a single word or be formed by a sequence of words. The voice command is preferably formed from words in a natural language.

The voice command is preferably formed according to the rules of a grammar. This is particularly preferably a formal grammar.

The grammar preferably divides text elements into different classes. Some possible classes are described in more detail below by way of example. The identified text element preferably belongs to a class whose text elements designate interaction elements.

Provision can be made for the provided voice command to be broken down into a sequence of text elements with the aid of a computer. In this case, the sequence can also consist of just one text element, even if the sequence regularly includes a plurality of text elements. The breakdown can take place, for example, using a text parser. This is preferably done according to the rules of the aforementioned grammar.

It can be provided, for example, that the grammar is used to identify whether a text element consists of a single word or a sequence of several words. For this purpose, for example, signal words or punctuation marks can be provided which cause several words to be combined into one text element.

A list is preferably created which contains those text elements that belong to a class whose text elements designate interaction elements in the order in which they occur. Any existing macros are resolved beforehand.

The voice command can preferably be formed according to the rules of the aforementioned grammar, otherwise freely. However, it is advantageous if additional information about an application that has a similar functionality to the application whose user interface is to be interacted with is used to set up the voice command. For this purpose, description documents of such an application or also results from testing such an application can be used. What is important here is that it can be a different application with a user interface that differs from the user interface that is the subject of the method according to the invention. This is a great advantage of the invention once created Voice command can be applied to a variety of different applications.

Provision can be made for the spoken command to be provided by converting a spoken input into a sequence of text elements. In principle, any voice recording device and any voice recognition method can be used for this purpose that appear suitable to the person skilled in the art. Alternatively or additionally, it can be provided that the voice command is provided by providing a sequence of text elements. For example, this can be done by entering a key or by reading the content of a text file. In this regard, too, the sequence of text elements can only have a single text element, although it preferably has more than one text element.

In principle, any method known from computer linguistics for determining a semantic similarity can be used to determine the degree of semantic similarity of the text element to the lexical content. For example, the cosine similarity of the two text elements being compared can be calculated, normalized to the length of the strings being compared. Cosine similarity returns a numerical value in an interval [-1, 1], where values close to 1 indicate a high degree of semantic similarity.

The basis for determining the degree of semantic similarity can be a model space for a set of words, with each word in the set of words being assigned an element in the model space. A metric exists in the model space, with which a distance between the elements can be determined, with the distance describing a semantic distance between the elements.

For example, the model space can be an n-dimensional vector space with n>=2. The elements can be vectors in the vector space and the metric can be given by an angle between vectors of the vector space, for example.

In order to assign elements of the model space or vectors to the words, relationships between words can be determined starting from a text corpus. Words with high semantic similarity are then assigned to elements of the model space that are close to each other. A well-known model is, for example, the word2vec model, which is trained with a neural network.

To carry out the method, the text elements to be compared with one another are then compared with the model space, so that the text elements are assigned elements from the model space. The metric can then be used to determine the semantic distance between the text elements.

If the semantic similarity of a first text element to a second text element is to be determined, one or both of which consist of several words in the model space, it can be provided that the semantic similarities are calculated in pairs and arranged in a matrix. It can then be provided that a scalar describing the semantic similarity is assigned to the matrix entries determined in this way. For this purpose, it can be provided that a maximum value is determined for each row or for each column and the values determined in this way are added to one another. For normalization, the sum can then be divided by the number of added values. If cosine similarity is used for each entry in the matrix, the scalar determined in this way is again between -1 and 1.

The basis for determining a semantic degree of similarity can therefore be a model of a set of words, with the model depicting relationships between the words. The relationships can consist, for example, in a joint occurrence in a context, for example in a sentence or some other unit of meaning. A large number of methods in this regard are known to the person skilled in the art.

Such relationships between words are not taken into account in purely syntactic comparisons of text elements.

In order to be able to assign suitable lexical content to the interaction element, it can be provided that the lexical content for the interaction element is determined by identifying lexical content arranged on the interaction element.

This approach fails if no lexical content is placed on the interaction element.

However, a description characterizing the interaction element will usually be arranged in its vicinity. A graphically represented descriptive element will thus regularly be spatially assigned to the interaction element.

It can therefore be advantageous if, in addition to or instead of identifying a lexical content arranged on the interaction element, the lexical content for the interaction element is determined by distances from surrounding the description elements with a lexical content are determined and the lexical content of the closest description element is selected. For this purpose, it can be provided, for example, that Euclidean distances from the interaction element to its neighboring description elements are determined, with the description element with the smallest distance then being selected. To determine the distance between two elements, for example, the Euclidean distance from corners of the elements, for example the top left corner of the elements, can be determined. It can be provided here that a description element is preferably selected which overlaps with the interaction element.

It is preferably first checked whether lexical content is arranged on the interaction element. If this test fails, the immediate environment can then be examined for lexical content as described.

Provision can be made for only those elements with which an interaction is not possible to apply as descriptive elements. However, this is not mandatory since an adjacent interaction element can also have a common description. It can therefore also be provided that all graphically represented adjacent elements with lexical content can be considered as descriptive elements.

In a further advantageous embodiment of the method, it can be provided that the lexical content of the interaction element or of the description element is determined by being taken directly from a text field of a data structure describing a current state of the user interface. Such a data structure is often available via a graphical user interface software interface. In this case, the data structure can have information about the interaction element or the description element in the form of data fields. In this case, the text field can in particular be a data field of the data structure assigned to the interaction element or the description element.

If no such data field is available, the lexical content of the interaction element or the description element can also be determined by applying a text recognition and/or image recognition method to the graphical representation of the interaction element or the description element. All common methods can be used for this. For example, OCR (optical character recognition) methods can be used for text recognition.

It can happen that the text field of the data structure or the text determined by the text recognition contains a character such as an icon that is not covered by the grammar. Provision can be made for a method to be used which assigns lexical content to such a character or also to a graphic element determined by image recognition. For this purpose, for example, a list can be stored that contains an assignment of lexical content to the characters and/or graphic elements. In such a case, the method would therefore essentially consist of accessing the corresponding list element.

Provision can be made for the interaction element and/or the description element and/or information about these elements to be identified by means of a data structure describing a current state of the user interface.

A text determined for an interaction element or a description element can still contain unwanted, meaningless Unicode characters, line breaks or the like. It can therefore be provided that, in order to determine the lexical content, a recognized text is first cleaned by removing a selection of characters or replacing them with spaces. For example, it can be provided that all special characters are replaced by spaces and all stop words are removed.

Provision can be made for the recognized and preferably already cleaned character string to be broken down into a list of words which then form the determined lexical content.

Navigation on the user interface can be made considerably easier if it is provided that the voice command includes a designation for a macro, the macro in turn including a sequence of text elements. The sequence of text elements for the macro is preferably stored in a text file. This has the advantage that the voice command can be designed much more simply and contain more abstract instructions, while details can be stored in the macros. When processing the voice command, the macros are preferably resolved directly, in that the sequence of text elements formed by the macro replaces the designation of the macro in the voice command. The macro itself can have macros, which are then resolved recursively.

It can be provided that in the event that an interaction requires an input, such an input is not included in the voice command solved macros, the procedure is halted until input is given. Provision can be made here for a graphically or acoustically perceptible indication to be given to the user of the method. It can be provided that the input takes place via voice control or via the usual input paths required by the user interface.

Provision can be made for such an interaction element, if it is outside a current display of the graphical user interface, to only be interacted with after it has been displayed within the current display by scrolling the display. In addition, it can be provided that an interaction element is always displayed before an interaction with it.

In an advantageous embodiment of the method it can be provided that the interaction element is identified by an interactive element of a current state of the graphical user interface being selected. Functionalities known from conventional crawlers, which can access a data structure describing the graphical user interface, can be used for the identification, for example.

In order to exclude existing but not visible interactive elements from an interaction, it can be provided that only such an interactive element is selected that has a finite extent. It can also be expedient to select only such an interactive element that is not completely or only partially covered by other displayed elements.

In order to avoid incorrect interactions, in a further advantageous embodiment of the method it can be provided that the interaction element is only interacted with in the event that the ascertained degree of similarity exceeds a threshold value. For example, experiments have shown that when using cosine similarity, a threshold with the value 0 is particularly suitable. A threshold value that is set too high should also be avoided in order to avoid that good candidates are ultimately not selected.

If the threshold value is not reached or no similarity is determined in any other way and if such a threshold value is also not reached when determining further degrees of similarity between text elements of the voice command and lexical contents of interaction elements or if a similarity is determined in another way, it can be provided that an interaction is selected at random. Interactions that have already taken place previously in the course of executing the method or the voice command are preferably excluded here. Such configurations of the method make it possible to ensure that even larger deviations in the voice command and functionality of the application to which the method is applied do not represent an obstacle.

In a further advantageous embodiment of the method, it can be provided that degrees of similarity of text elements of the voice command to lexical content determined for interaction elements of the graphical user interface are determined and that the interaction element is interacted with first for which the determined degree of similarity is highest. What is decisive in this embodiment of the method is that more than one degree of similarity is determined before the interaction in question. For this purpose, degrees of similarity can be determined from a text element to several lexical contents, from several text elements to one lexical contents and from several text elements to several lexical contents.

In this description of the invention, “several” means “at least two”.

The determination of several degrees of similarity and the selection of a best pair can lead to a considerable flexibility in navigating a user interface using voice commands.

This becomes particularly evident when it is provided that the order of the text elements of the voice command is not taken into account. Provision can thus be made for a number of text elements for which a degree of similarity is determined, in particular all text elements in the list which contains the text elements which belong to a class which describes the interaction elements, to be treated equally. This also makes it possible to successfully execute those voice commands which specify interactions in a specific order, even if the application to which the method is applied requires a different order of interactions or even does not provide for some of the interactions contained in the voice command at all.

If the order is not to be completely ignored, but also not necessarily to be adhered to, this can be done by introducing a weighting that lowers the degree of similarity the more the later the text element occurs in the voice command. Alternatively or additionally it can be provided that with the same degrees of similarity or minor differences, the order of the text elements takes precedence.

It can be provided that at least one action is assigned to the text element of the voice command and that the at least one action assigned to the text element is compared with an action assigned to the interaction element. The degree of similarity of the text element to the interaction element is preferably determined as a function of the result of the comparison. The degree of similarity is particularly preferably only determined if the actions are compatible. If an interaction element only allows a click as an interaction and an input is linked to the text element, then the actions are not compatible and it would be superfluous to determine a degree of similarity. Provision can also preferably be made for the degree of similarity to be weighted even in the case of mutually compatible actions. An example where a lowering can be useful is given below in the description of the in 1 described procedure.

For better control of the execution of a voice command, it can be provided that it is stored whether and/or how often the interaction element was interacted with during the execution of the method. Alternatively or additionally, it can be provided that a state model of the user interface is created and/or updated from interactions with the user interface and from the states of the user interface reached as a result of the interactions. The information collected can be used to further influence the method in an advantageous manner, as the following exemplary embodiment also shows.

Provision can thus be made for the degree of similarity to be reduced if it is recognized that the interaction element has already been interacted with during the execution of the method. A weighting factor is preferably provided for this purpose, which is multiplied by the previously determined degree of similarity or subtracted from it. The degree of similarity is preferably further reduced the more frequently an interaction element has been interacted with. Such refinements of the method have the advantage, among other things, that the voice command can be processed faster and more reliably, since in particular it is avoided that an action that does not lead to the desired result is triggered several times or even repeatedly.

It can also be provided that in the case of states of the user interface that are reached for the first time in the course of the execution of the method, those text elements of the voice command that require or enable an input as an interaction are executed first. It can be provided that this only takes place if a threshold value for the ascertained degree of similarity is exceeded.

Alternatively, it can also be provided that the ascertained degree of similarity for such text elements is increased.

The state of the user interface and/or the application can be saved after the method or the voice command has been executed. This can be useful, for example, for evaluation or testing purposes.

To achieve the above object, the invention also proposes using a method for interacting with a graphical user interface to execute a function of an application that uses the user interface as an interface, the method according to the invention, in particular as described above or according to a a method for interacting with a graphical user interface directed protection claims, is formed. The voice command is preferably provided by the user, for example by way of a voice input.

In order to achieve the aforementioned object, the invention also proposes the features of the independent claim directed to a method for testing an application. In particular, it is therefore proposed according to the invention in a method for testing an application using a user interface as an interface to solve the stated task that at least one voice command is specified, that a method for interaction with a graphical user interface is carried out for each of the at least one voice command, with this Method according to the invention, in particular as described above or according to a protection claim directed to a method for interaction with a graphical user interface, and that a state model of the user interface is generated and/or updated and stored so that it can be called up, the state model showing the interactions carried out and the states of the user interface reached as a result.

Several or a large number of voice commands are preferably specified. In this case, the voice commands preferably test different functions of the application.

The status model can then be used to evaluate the test result the. For example, it can be determined which functions the tested application has.

The method steps mentioned above are each preferably carried out with the aid of a computer. All method steps are preferably carried out with the aid of a computer. However, it is preferably provided that at least the voice input takes place by a user speaking a voice command. Alternatively, this can also be read out from a file using a computer.

In order to achieve the aforementioned object, the invention also proposes the features of the subordinate claim directed to a technical device. In particular, it is therefore proposed according to the invention for a technical device to solve the stated object that the technical device is set up to use a method for interacting with a graphical user interface and/or a method for testing a user interface, for example the aforementioned user interface, as an interface Execute application, the respective method being designed according to the invention, in particular as described above or according to one of the protection claims directed to a corresponding method.

In order to achieve the aforementioned object, the invention also proposes an arrangement with a technical device and with a technical facility on which an application using the user interface as an interface is stored ready for execution and access. The technical device is designed according to the invention, in particular as described above or below. Furthermore, at least one interface is formed, which enables data to be transmitted from the technical device to the technical facility. The at least one interface preferably also enables data to be transmitted from the technical device to the technical device.

The interfaces can in particular be hardware interfaces or software interfaces.

The technical device and/or the technical facility is preferably an electronic device such as a computer or a smartphone. They can be the same or separate devices.

Input means for inputting a voice command are preferably configured on the technical device and/or on the technical facility.

The technical device preferably has at least one data memory and one data processing unit. A computer program can be stored in the data memory, which is executed to carry out the method described above.

The technical device preferably has a display on which elements of the graphical user interface to be displayed can be displayed.

A preferably optical sensor, such as a video camera, can be designed with which what is shown on the display can be recorded. The sensor can be connected to the technical device in order to transmit the recorded data to it.

The technical device and the technical device can use common resources such as a shared data memory or a shared processor.

The invention will now be described in more detail using a few exemplary embodiments, but is not limited to these few exemplary embodiments. Further exemplary embodiments result from the combination of the features of individual or multiple claims with one another and/or with one or more features of the exemplary embodiments.

• 1 ein Flussdiagramm eines Ausführungsbeispiels eines erfindungsgemäßen Verfahrens zur Interaktion mit einer graphischen Benutzeroberfläche,
• 2 ein Flussdiagramm eines Ausführungsbeispiels eines Verfahrens zum Testen einer Applikation, die eine Benutzeroberfläche aufweist,
• 3 eine Anzeige einer graphischen Benutzeroberfläche, die sich in einem konkreten Zustand befindet.
• 4 bis 6 jeweils ein Ausführungsbeispiel einer erfindungsgemäßen Anordnung.

It shows:

• 1 a flow chart of an embodiment of a method according to the invention for interaction with a graphical user interface,
• 2 a flowchart of an embodiment of a method for testing an application that has a user interface,
• 3 a graphical user interface display that is in a particular state.
• 4 until 6 each an embodiment of an arrangement according to the invention.

In the following description of the invention, elements that have the same function are given the same reference numbers, even if the design or shape differs.

1 shows an embodiment of a method 100 according to the invention.

In method step 101, the method is initialized by providing a voice command that is to be applied to an application with a graphical user interface. The application can be an online shop, for example. For example, the voice command can be “Sign me in”.

• Für Melde mich an Mache
• Klicke Anmelden
• Und Schreibe Max Mustermann In Benutzername
• Und Schreibe 123456 In Passwort

A macro can be stored for the designation "Log me in", which reads as follows, for example:

• Sign me up for Mache
• Click Sign In
• And write John Doe in username
• And write 123456 in password

The voice command can be provided by voice input. To do this, the user speaks the words "Sign me in".

Using a microphone and a voice recognition process, the voice input is converted into a voice command, i.e. a sequence of words.

In step 102, the voice command is then broken down into its text elements using a grammar and a parser, with the text elements being divided into classes.

The text element "Sign me up" belongs to a first class of words. The first class describes a macro. The parser recognizes this from a list of labels, which includes the text element "Sign me in". To execute the voice command "Sign me in", the macro "Sign me in" is then executed. As a result, the name of the macro is therefore replaced by its content.

The text elements "For", "Do" and "And" belong to a second class. The second class describes words that control the voice command. The words "For" and "Do" indicate that the intervening text element is a label of a macro. For example, the word "And" means that in addition to the interaction before the "And", there is another interaction that is described after the "And".

The text element "click" and the word combination "write" and "in" belong to a third class. The third class relates to the nature of the interaction. For example, the word "click" can be interpreted as meaning that the interaction element "login" should be clicked. This can be a short or a long click. The text element consisting of the word combination "Write" and "In" means that text element between the words "Write" and "In" should be entered as text in the input belonging to the interaction element after the word "Write".

The text elements "login", "username" and "password" belong to a fourth class. The fourth class describes words denoting elements of interaction. As will be explained in more detail below, it is not necessary for the application to use exactly these terms. It is sufficient that there is a semantic similarity.

A list is then created listing the text elements of the fourth class in the order of their occurrence. Associated with each element of the list are all the actions compatible with the type of interaction taken from the text elements of the third class. Actions that are possible for compatible interaction elements are assigned here. To take the previous example, the action "click" and "long-click" is assigned to the word "click" if the graphical user interface allows such actions.

In step 103 the current state of the graphical user interface is analyzed. As a rule, the operating system on which the application is running provides an interface that can be used to analyze the current state of the graphical user interface. For example, Android or websites on desktop PCs offer the possibility of accessing a structure that hierarchically lists all graphical elements of the current state of the graphical user interface and provides some basic information about these elements such as their type, location, horizontal extent and vertical level . It is also possible to see whether the elements currently appear on the displayed display or whether they first have to be reached by scrolling the display.

In particular, interactive elements and the interaction options with them, such as clicking, entering text or selecting from a drop-down menu, as well as a possible designation directly linked to the interaction element and graphically displayed on it, can therefore be derived from the structure. A possibility is also provided to interact with the interaction elements by using a peripheral device or by a programmed instruction replacing it.

Non-interactive elements such as purely descriptive elements can also be derived for which properties such as location, horizontal extent and vertical level can also be stored. The descriptive elements can in particular have a text-based or an image-based description (such as an icon).

In the exemplary embodiment described here, in step 103 the interaction elements and the description elements of the current state of the graphical user interface are specifically determined, together with all available information about these elements that is required for the further process.

In step 104, the respective designation of the current interaction elements is determined. This is easy if a designation is directly linked to the interaction element, since this can then be taken from the information available on the interaction element. That's how it is in, for example 3 in the case of the interaction element provided with the reference numeral 24, the designation "login" is displayed graphically on the interaction element 24.

Frequently, however, a graphically represented designation is not directly linked to the interaction element, but there is a non-interactive description element in the vicinity of the interaction element. This occurs in 3 for example on the interaction element 22, respectively. 23 to which the descriptive element 25 ("Username") resp. 26 ("password") is assigned.

In order to assign the correct descriptive element to these interaction elements, provision can be made for determining the distance between the surrounding descriptive elements and the respective interaction element, with the distances between the respective upper left corners being determined, for example. Then, for example, that descriptive element that has the smallest distance to the interaction element can be assigned to the interaction element. Should there be overlapping elements, it can be arranged that an overlapping element with the smallest distance is selected, otherwise a non-overlapping element with a smallest distance is selected.

The lexical content of the descriptive elements is then extracted. This can be linked directly to the description element in the hierarchical structure. If it is not, for example because an image file is stored, the lexical content can be extracted, for example, using a text recognition method (e.g. using OCR software) or, in the case of icons, using a method that assigns lexical content to the icon.

As already described above, the recognized text can still be cleaned up and broken down into words for the final determination of the lexical content.

As a result, in step 104 a lexical content is determined for the interaction elements, which designates them.

In step 105, a semantic degree of similarity between the words in the list created in step 102 and the lexical content determined in step 104 is then determined. The dashed lines indicate that data determined from steps 102 and 104 are processed in step 105. Possible methods for determination have already been mentioned above. For example, if the list has N words and there are M interaction elements with associated lexical content, then a total of up to N*M degrees of similarity are determined.

In order to reduce the number of calculations required, it can be provided that only those degrees of similarity are determined for which the actions are compatible with one another. This can be done by checking whether the actions assigned to a text element in the list contain an action that can be carried out with the interaction element. If such an action is not possible, there is no need for a similarity calculation of the semantic content since an interaction with the interaction element is ruled out.

In step 106 the calculated degree of similarity is modified by means of a weighting. If the degree of similarity is expressed by a real number, the weighting can be multiplicative. Alternatively, subtraction with a number can also be provided.

Firstly, it can be provided that certain actions reduce the calculated degree of similarity. For example, if a word in the list contains an optional data element, a click action can be assigned to it, whereby the degree of similarity is reduced. In this way, for example, drop-down menus can be handled adequately, which only reveal an editable input when they are clicked.

Secondly, it can be provided that the calculated degree of similarity is reduced if it is established that the interaction element has already been interacted with when the method and the voice command are being carried out. It it can be provided that the degree of similarity is reduced further and further as the number of interactions increases. In this way it can be avoided that undesired interaction chains are repeated and it can be achieved that new interaction paths are reached. In order to make this possible, it can be provided that the states reached during the execution of the voice command and the interactions carried out are stored.

In step 107 a suitable candidate for the next interaction is then selected. For this purpose, that interaction element is selected for which—after possibly being lowered in step 106—the highest degree of similarity has been determined.

If the highest ascertained degree of similarity is less than a threshold value, the exemplary embodiment described here provides for any interaction element to be selected, with only one being selected that has not yet been interacted with.

In step 108 the interaction element is then interacted with. If an association exists, applying an action associated with the corresponding text element from the list. If an entry is required, then what is to be entered has already been linked to the corresponding action in step 102 beforehand. If the same effective degrees of similarity are assigned to several actions, one of the two is selected.

In step 109 a termination criterion is then checked. For example, it can be checked whether a maximum time has been reached and/or whether a desired target state has been reached. For example, the process can be ended if the registration has been successfully completed in the example discussed here.

Provision can also be made for the selected text element to be deleted from the list after an interaction. In this case, the method can also be terminated when the list no longer contains any further elements.

If the termination criterion is met, the method is terminated in step 110. In particular when the method is used to test an application, it can be useful if the states reached during the execution of the voice command and the interactions carried out are saved.

If the termination criterion is not met, the method continues in step 103.

2 FIG. 2 shows an embodiment of a method 200 for testing an application that has a graphical user interface.

To carry out the method, a large number of voice commands are first provided in step 201 . This can be done by verbal input or by providing a text file containing the voice commands.

In step 202 it is then checked whether all voice commands have already been processed or whether a voice command has remained unprocessed.

If a voice command is still unprocessed, in step 203 method 100, which is described in 1 is shown executed. The result of the method 100 is added to an initially empty state model that already exists after a first iteration, the state model including the interactions carried out for the respective voice command and the states of the graphical user interface achieved as a result.

The method then continues in step 202 with the remaining voice commands.

If no more voice commands need to be executed, the method 200 is ended in step 204 in that the state model that has been created is stored so that it can be called up for further analysis.

3 Figure 12 shows a graphical user interface display 14 which is in a current state. In the state shown, a total of three interactive interaction elements 22, 23 and 24 and two non-interactive description elements 25 and 26 are displayed. In order to log in, it is first necessary to enter the user name and password in fields 22 and 23 before clicking button 24 registration can take place. The lexical content “login” is arranged on the interaction element 24 . In the case of the two input fields 22 and 23, the lexical content “username” and “password” assigned to them is located in the respective nearest description fields 25 and 26 “A login occurs through interaction with the user interface. Despite the different order of the text elements provided in the voice command and despite considerable syntactic deviations from the text fields, the voice command can be executed successfully and the user is successfully logged on.

The methods 100, 200 described above can be carried out, for example, with the arrangements 12 or with the technical devices 1 be carried out in accordance with the 4 until 6 are formed and are described below. For this purpose, the method steps are carried out automatically using the technical device 1, which can be a server or a smartphone, for example.

The arrangement 12 comprises the technical device 1 and the technical device 2.

In 4 and 5 device 1 has resources that are separate from device 2. In 6 an example is shown in which the technical device 1 and the technical facility 2 use common resources such as the processor 15 and the data memory 16.

For example, the technical device 2 can be an electronic device 13 such as a computer or a mobile device, preferably a smartphone. The electronic device 13 can also form the technical device 1 at the same time as in 6 . However, the technical device 1 can, as in 4 or 5 can also be a separate device such as a server.

The technical facility 2 and/or the technical device 1 have a display 14 on which the current status of graphic elements of the graphic user interface can be displayed.

A software application using the user interface as an interface is stored on the technical device 2 ready for execution and access.

A computer program is stored on the technical device 1 ready to be executed and accessed.

The technical device 2 has a data input interface 6 via which data can be received.

The technical device 2 can receive data from the data output interface 3 of the technical device 1 via the data input interface 6 . For example, the received data may include specific actions to be performed by interaction elements of the user interface.

The technical device 1 has a data input interface 9 via which data can be received. As in 4 For this purpose, the data input interface 9 can be connected to a data output interface 17 of the technical device 2 via a wireless and/or a wired data line 10 such as an Internet connection.

In 4 and 6 lexical content of elements of the user interface is determined via a data structure or other memory content stored in the form of data. In 5 it is provided that the graphical representation of the user interface currently displayed on the display 14 is recorded by means of a sensor 8 such as a video camera and the recordings are transmitted to the technical device 1 .

The technical device 1 and the technical device 2 each have at least one data memory 16 and one processor 15 . The latter are in 4 and 5 not shown explicitly for reasons of clarity.

In 4 and 5 are the interfaces 3, 6, 9 and 17 as hardware interfaces and in 6 designed as software interfaces.

In summary, the invention relates to a device 1 and a method for interacting with a graphical user interface and for testing an application. A voice command is provided from which a text element is identified. A lexical content is determined for an interactive element 22, 23, 24 of the user interface. A semantic degree of similarity of the text element to the lexical content is determined and the interactive element 22, 23, 24 is then interacted with depending on the degree of similarity.

reference list

1

Technical device

2

Technical facility

3

Data output interface from 1

6

Data input interface from 2

8th

sensor

9

Data input interface from 1

10

data line

12

test arrangement

13

electronic device

14

Display of a graphical user interface

15

processor

16

data storage

17

Data output interface of 2

22

interaction element

23

another interaction element

24

another interaction element

25

description element

26

another descriptive element

Claims (17)

A method for interacting with a graphical user interface, the graphical user interface having an interaction element, characterized in that a voice command is provided from which a text element is identified, that lexical content for the interaction element is determined with computer assistance, that a semantic degree of similarity of the text element to the lexical content is determined with the aid of a computer and that the interaction element is interacted with depending on the degree of similarity. Method according to the preceding claim, characterized in that the lexical content for the interaction element is determined by identifying a lexical content arranged on the interaction element and/or by determining distances from surrounding description elements with lexical content and the lexical content of the closest description element is selected. Method according to one of the preceding claims, characterized in that the lexical content of the interaction element or the description element is determined by taking it directly from a text field of a data structure describing a current state of the user interface and/or by clicking on a graphic representation of the interaction element or the description element a method of text recognition and/or image recognition is used, in particular in the case of an initially identified graphic element and/or an initially identified character that is not covered by the grammar, a method is used that gives the graphic element and/or the character a assigns lexical content. Method according to one of the preceding claims, characterized in that the voice command is provided by a spoken input being converted into a sequence of text elements and/or that the voice command is provided by a sequence of text elements, in particular by a keyboard input or by a text file , provided. Method according to one of the preceding claims, characterized in that the voice command is broken down into a sequence of text elements with the aid of a computer, preferably by means of a parser. Method according to one of the preceding claims, characterized in that the voice command includes a designation for a macro, the macro for its part including a sequence of text elements preferably stored in a text file. Method according to any one of the preceding claims, characterized in that the interaction element is identified by selecting an interactive element of a current state of the graphical user interface, in particular wherein the interactive element has a finite extent and/or is unobscured. Method according to one of the preceding claims, characterized in that the interaction element is only interacted with if the ascertained degree of similarity exceeds a threshold value. Method according to one of the preceding claims, characterized in that semantic degrees of similarity of one or more than one text element of the voice command to one or more than one lexical content, which is determined for one or more than one corresponding interaction element, are determined and that with that interaction element interacted first for which the determined degree of similarity is highest. Method according to one of the preceding claims, characterized in that at least one action is assigned to the text element of the voice command and that the at least one action assigned to the text element is compared with an action assigned to the interaction element, in particular wherein the determination of the degree of similarity of the text element to the interaction element in Dependence on the result of the comparison takes place. Method according to one of the preceding claims, characterized in that it is stored whether and/or how often the interaction element has been interacted with during the execution of the method and/or that from interactions with the user interface and from the states of the user interface achieved as a result of the interactions a state model of the user interface is created and/or updated. Method according to one of the preceding claims, characterized in that the degree of similarity is preferably reduced by means of a weighting factor if it is recognized that the interaction element has already been interacted with previously during the execution of the method. Method according to one of the preceding claims, characterized in that in the case of states of the user interface which are reached for the first time during the execution of the method, those text elements of the voice command which require or enable an input as an interaction are executed first. Use of the method according to one of the preceding claims for executing a function of an application using the user interface as an interface, in particular with the user providing the voice command. Method for testing an application using a user interface as an interface, characterized in that at least one voice command is specified, that the method according to one of the preceding claims is carried out for each of the at least one voice command and that a state model of the user interface is generated and/or updated and is stored in a retrievable manner, the state model comprising the interactions carried out and the states of the user interface achieved as a result. Technical device (1), characterized in that the technical device (1) is set up, the method for interacting with a graphical user interface according to one of Claims 1 until 13 and/or the method for testing an application using an application or the user interface as an interface claim 15 to be carried out, in particular with input means being designed for inputting a voice command. Arrangement (12) with a technical device (1) according to the preceding claim and with a technical device (2) on which an application using the user interface as an interface is stored ready for execution and access, with at least one interface being formed that transmits of data from the technical device (1) to the technical facility (2).

Images