KR20220040997A - Electronic apparatus and control method thereof - Google Patents

Abstract

Disclosed is a control method of an electronic device. The control method comprises: a step of identifying a characteristic of at least one user interface (UI) element included in a screen of the electronic device; a step of obtaining a database comprising an obtained natural language utterance based on the at least one characteristic of an identified UI element; a step of identifying whether or not the utterance of the received speech input matches the natural language utterance included in the obtained database, when the speech input is received; and a step of automatically accessing at least one UI element, when the utterance of the speech input is identified as matching the natural language utterance. Therefore, the present invention is capable of improving a user experience.

Description

Electronic apparatus and control method thereof

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device providing a UI and a control method thereof. The present disclosure claims priority based on Indian Provisional Application No. 202041041137, filed on September 23, 2020, the contents of which are incorporated herein by reference.

In general, electronic devices (eg smart TVs, smartphones, etc.) are becoming more powerful and sophisticated with the advent of new applications (eg streaming video applications such as Netflix, YouTube, etc.). Instead of accessing the channel list of the video application, the user can browse through several applications. Each application has a layout and characteristics/functions. Despite these advances, the use of electronic devices with touch screens, keyboards, remote controls or mice is still unusual. The dominant access method continues to interact with electronic devices with limited input range and inadequate support for text input and fine-grained navigation or selection.

Some of the existing methods/electronics provide several viable ways to bridge the gap between interface complexity and remote control capabilities. Speech recognition technology is one of them. Intelligent voice agents/features such as Alexa, Siri and Google Assistant are integrated into electronic devices, allowing users to manage an increasingly sophisticated suite of applications accessible from their electronic devices via voice input. For example, a phrase/command/voice input of "Jarvis, good morning" causes the user to start a morning routine. Upon receiving voice input, the intelligent voice agent/assistant turns on lights, provides weather-related information, gives news briefings and activates coffee machines. The user has to memorize specific phrases/commands to control the electronic device. The user has to memorize a specific phrase/command to control the electronic device, and it is a bit unnatural for the specific phrase/command to converse with the home appliance/electronic device/IoT device (e.g., tomorrow [time] from [location] to [name] ] and scheduling a meeting), which is rather difficult for non-technical users.

The existing method (2)/electronic device 10 provides a method for reducing the gap between interface complexity and remote control function as shown in FIG. 1 . For example, the user 1 may access a UI element (eg, video) of a video application by uttering a specific number displayed on the UI. The following scenario should be considered in which the user 1 wants to access the third video displayed on the screen of the electronic device 10 . To play/access the video, the user 1 must utter a specific voice command, such as “play video 3”. However, in order to access the UI elements displayed on the screen of the electronic device 10, the user 1 must You must be close to the device 10 to read the name or number of UI elements, but this is not always feasible. Also, uttering certain voice commands is not a natural way to access/call UI elements. ) cannot access sub-functions or sub-pages of this video application that are not displayed on the screen of the electronic device 10 .

As shown in FIG. 1 , another existing method 3/electronic device 10 provides a method for reducing the gap between interface complexity and remote control function. For example, user 1 can utter "show name" to display an overlay with a UI element name, or "show name" to display an overlay with a UI element name to display a UI element in a social media application (e.g.: You can access the search bar, home icon, like button, share button, etc.). If the UI element does not have a name or if the user 1 prefers to use numbers instead, the user 1 says "show numbers" to display a numeric tag for each UI element on the screen of the electronic device 10 can do. However, the user 1 can only access UI elements displayed on the screen of the electronic device 10 using this method 3 . The user 1 cannot access a sub-function or sub-page of this social media application that is not displayed on the screen of the electronic device 10 . In addition, in order to access the UI element displayed on the screen of the electronic device 10 , the user 1 must be close to the electronic device 10 to read the name or number of the UI element, which is not always executable.

As another example, consider the scenario where the user 1 wants to access a search page as shown in FIG. 1 . Some existing methods (4)/electronic devices (10) generally provide a limited set of control and search guidelines and do not support deeper recommendation or search queries. Table 1 shows an example of a limited control set.

Accordingly, it is necessary to provide a useful alternative for automatically accessing UI elements of electronic devices using voice input/voice-based interaction.

The present disclosure is in accordance with the above-described necessity, and according to an embodiment of the present disclosure, when a user's voice input matches a predicted Natural Language (NL) utterance of at least one identified UI element, the display is displayed on the screen of the electronic device. At least one identified UI element (eg, executable UI element, non-executable UI element, text UI element, and non-text UI element) may be automatically accessible. As a result, users can utilize natural language to access the various UI components identified and enhance the user experience. The natural language utterance may be predicted by acquiring a database/knowledge graph based on characteristics (eg, relative position, performance, function, type, shape, etc.) of at least one identified UI element. The user's voice input includes utterances for a characteristic of at least one identified UI element appearing in the database.

According to an embodiment of the present disclosure, a knowledge graph may be obtained by determining a degree of similarity between UI elements in at least one UI element among other UI elements. The degree of similarity between each UI element may be determined by the position of each UI element, the relative position of each UI element among other UI elements, the performance of each UI element, the function of each UI element, and the shape of each UI element displayed on the screen of the electronic device. can

According to an embodiment of the present disclosure, each UI element of at least one UI element may be clustered based on the degree of similarity. Furthermore, the electronic device may determine a text representation of a UI element other than text from a visible characteristic (eg, a heart-shaped icon) and a relative position of each UI element among other UI elements. In addition, the electronic device provides predefined information (eg, performance (press, copy), verb (eg, tap, punch, click, copy and duplicate)), type (eg, button, text) between at least one of each UI element. ) and determine a predefined screen order that transitions through executable UI elements.

According to an embodiment of the present disclosure, a natural language transformation for a single-step intent and/or a multi-step intent is acquired by performing semantic transformation on the acquired knowledge graph. In addition, the electronic device identifies an action and an action sequence for a single-step intent and/or a multi-step intent by using the acquired knowledge graph, and maps the acquired natural language variant to the identified action and the identified action sequence to obtain at least one A natural language model is dynamically acquired to predict a natural language utterance corresponding to the identified UI element. As a result, the user can improve the user experience by accessing a sub-function or sub-page of an application/UI element that is not displayed on the screen of the electronic device.

A voice-based interaction method of an electronic device according to an embodiment of the present invention for achieving the above object, the step of identifying at least one user interface (UI) element displayed on a screen of the electronic device, at least one identification determining a characteristic of the identified UI element, obtaining a database based on at least one characteristic of the at least one identified UI element, wherein the database is a natural language (NL) utterance corresponding to the at least one identified UI element comprising, wherein the natural language utterance is predicted based on at least one characteristic of the at least one identified UI element, and receiving a voice input from a user of the electronic device, the voice input being displayed on a screen of the electronic device. determining whether an utterance of the received voice input matches at least one natural language utterance in an acquired database, the utterance comprising an utterance indicative of at least one characteristic of the displayed at least one identified UI elements, and to the electronic device; and automatically accessing at least one of the at least one UI elements when it is determined that the utterances of the voice input received from the user match the predicted natural language utterances of the at least one identified UI elements. .

According to an embodiment, the at least one UI element may include at least one of an executable UI element, a non-executable UI element, a text UI element, and a non-text UI element.

According to an embodiment, acquiring the database further includes acquiring the knowledge graph and storing the acquired knowledge graph in the database. The knowledge graph is obtained by the position of each UI element, the relative position of each UI element between other UI elements, the function of each UI element, the type of each UI element, and the shape of each UI element displayed on the screen of the electronic device. In addition, the method is based on the position of each UI element, the relative position of each UI element among other UI elements, the function of each UI element, the function of each UI element, and the appearance of each UI element displayed on the screen of the electronic device. and determining a degree of similarity between each UI element of at least one UI element among other UI elements. The method also includes clustering each UI element of the at least one UI element based on the degree of similarity. The method also includes determining the textual representation of the non-text UI element from the visible characteristic and the relative position of each UI element among other UI elements. Also, the method includes mapping each UI element of the at least one UI element and predefined information corresponding to a function of each UI element. The method also includes determining a predefined screen sequence transitioned through the executable UI element.

Predicting the natural language utterance of the at least one identified UI element may include performing semantic transformation on the acquired knowledge graph to obtain natural language variants for at least one of a single-step intent and a multi-step intent, the acquired knowledge identifying at least one of an action and an action sequence for the at least one single-step intent and the multi-step intent using the graph, mapping the obtained natural language variants with the at least one identified action and the at least one identified action sequence; The method may include dynamically acquiring a natural language model for predicting natural language utterance of at least one identified UI element.

According to an embodiment, the performing of the semantic translation includes receiving the acquired knowledge graph, and expressing each UI element of the received knowledge graph as a domain, a verb, a synonym, a slot, a slot type, and a texture on the screen of the electronic device. categorizing into at least one of a given slot, performance, and relative position, and obtaining natural language variants based on the categorization.

According to an embodiment, the step of identifying at least one of the action and the action sequence for the at least one single-step intent and the multi-step intent is based on the step of receiving an acquired knowledge graph and the performance of each UI element. The method may further include determining action routines.

According to an embodiment, the step of dynamically obtaining the natural language model comprises: predicting natural language utterance of at least one identified UI element by mapping the obtained natural language variants and the at least one identified action. It may include dynamically acquiring the model. the at least one identified action sequence clustering similar natural language variants, assigning a dynamic intent to similar natural language variants, the at least one identified action and the at least one identified action sequence and the dynamic intent The method may further include: associating , dynamically acquiring the natural language model based on the clustered natural language transformation, the dynamic intent, and the action routines, and storing the dynamically acquired natural language model in a database. there is.

According to an embodiment, the determining whether the utterances of the received voice input match the predicted natural language utterances of the at least one identified UI elements include: a text representation/text UI element displayed on the screen 140 . Determining a text representation score by reading screen information in the form of determining a variance score associating variants, determining a relevance score by comparing reference objects mentioned in an utterance of the received speech input with nearby element information, and an utterance of the received speech input for a dynamic language generator determining a matching score as a final score that matches with , and combining the determined score with a relevance score. The match score determines the target element (ie the UI element) on which the action will be executed. For similar scores, the user is presented with an option to choose among the conflicting elements.

According to an embodiment, the at least one characteristic of the at least one identified UI element includes a position of each UI element, a relative position of each UI element among other UI elements, a function of each UI element, and a function of each UI element. It may include the performance, the type of each element, and the appearance of each UI element displayed on the screen of the electronic device.

The voice-based interaction electronic device provided in the embodiments of the present disclosure includes a memory and an interaction engine operatively connected to the processor. The interaction engine identifies at least one user interface (UI) element displayed on the screen of the electronic device, identifies a characteristic of the at least one identified UI element, and determines the at least one characteristic of the at least one identified UI element. obtain a database based on: the database includes natural language (NL) utterances corresponding to the at least one identified UI elements, the natural language utterances being predicted based on at least one characteristic of the at least one identified UI elements; , receive a voice input from a user of the electronic device, wherein the voice input includes an utterance indicating at least one characteristic of at least one identified UI element displayed on a screen of the electronic device, wherein the utterance of the received voice input is Determine whether utterances of the speech input received from the user match at least one natural language utterance in the obtained database, and if it is determined that utterances of the at least one identified UI elements match the predicted natural language utterances of the at least one identified UI elements, one of the at least one UI elements characterized by automatically accessing at least one UI element.

In an embodiment, priority is determined between conflicting commands based on specific priority rules supplied to the electronic device (eg, Go to Home.) Here, home is an application home ( home) or system home.

1 is a view for explaining the related technology.
2A and 2B are diagrams illustrating a configuration of an electronic device according to an exemplary embodiment.
3 is a flowchart illustrating a method of controlling an electronic device according to an exemplary embodiment.
4 is a flowchart illustrating a method of dynamically building a model according to an embodiment.
5 is a flowchart illustrating a method of dynamically building a model according to an embodiment.
6 is a diagram for describing a score system according to an embodiment.
7 is a diagram for describing an example scenario of dynamically building a model according to an embodiment.
8 is a diagram for explaining a difference between a related technology and a proposed method according to an embodiment.
9A to 9C are diagrams for explaining functions related to a natural language synthesizer according to an embodiment.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a component). and does not exclude the presence of additional features.

The expression "at least one of A and/or B" is to be understood as indicating either "A" or "B" or "A and B".

As used herein, expressions such as "first," "second," "first," or "second," can modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component); When referring to "connected to", it should be understood that an element may be directly connected to another element or may be connected through another element (eg, a third element).

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this application, terms such as "comprises" or "consisting of" are intended to designate that a feature, number, step, operation, element, component, or a combination thereof described in the specification exists, but at least one other feature It should be understood that it does not preclude the possibility of the presence or addition of numbers, steps, acts, elements, components, or combinations thereof.

In the present disclosure, a “module” or “unit” performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of “modules” or a plurality of “units” are integrated into at least one module and implemented with at least one processor (not shown) except for “modules” or “units” that need to be implemented with specific hardware. can be

The embodiments described below and various features and advantages thereof will be described in more detail along with the detailed description of the invention below with reference to the non-limiting embodiments shown in the accompanying drawings. Detailed descriptions of well-known components and process technologies are omitted so as not to obscure the gist of the embodiments below. In addition, various embodiments described below are not necessarily mutually exclusive, and some embodiments may be combined with at least one other embodiment to form a new embodiment. The term "or" as used hereinafter may mean non-exclusive or exclusive, unless the context clearly dictates otherwise. The examples used below merely aid in understanding how the embodiments below may be practiced, and are used to enable those skilled in the art to practice the embodiments below. Accordingly, it should be understood that the examples are not intended to be limiting on the embodiments below.

As is customary in the art, embodiments may be described and illustrated with respect to a described function or block performing the functions. These blocks have the meanings of managers, units, modules, hardware components, etc., and include logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, and built-in hardware. Physically implemented by analog and/or digital circuits, such as circuitry, and the like, and may be selectively driven by firmware. The circuit is embodied, for example, on at least one semiconductor chip or on a substrate support such as a printed circuit board. Circuits constituting the block may be implemented by dedicated hardware, or a processor (eg, at least one programmed microprocessor and related circuits), or a combination of dedicated hardware for performing some functions of the block and other functions of the block It can be performed by a processor for performing the Each block of the embodiment is physically separable into two or more interactive and separate blocks without departing from the scope of rights described in the present disclosure. Similarly, blocks of the embodiment can be physically combined into more complex blocks without departing from the scope of the rights described in this document.

Accordingly, embodiments to be described below describe a method for voice-based interaction of an electronic device. The method identifies at least one UI element displayed on a screen of an electronic device, identifies a characteristic of the identified at least one UI element, and obtains a database based on the identified characteristic of the at least one UI element. The database includes natural language utterances of the identified at least one UI elements, wherein the natural language utterance is predicted based on at least one characteristic of the identified at least one UI elements.

Also, the method may receive a voice input from a user of the electronic device, and the voice input may indicate characteristics of at least one UI element stored in a database. Also, determine whether the utterances of the received voice input match the predicted natural language utterances of the identified at least one UI elements, and determine whether utterances of the voice input received from the user match the predicted natural language utterances of the identified at least one UI elements When it is identified as doing so, the method may include automatically accessing (or accessing or executing) a UI element among at least one UI element.

Accordingly, embodiments described below provide an electronic device for voice-based interaction. The electronic device includes an interaction engine coupled with a processor and memory. The interaction engine identifies at least one UI element displayed on the screen of the electronic device. Further, the interaction engine determines a characteristic of the identified at least one UI element. In addition, the interaction engine predicts natural language utterance of the identified at least one UI element by acquiring a database based on the characteristic of the identified at least one UI element. Further, the interaction engine receives a voice input from a user of the electronic device, wherein the voice input includes an utterance indicating a characteristic of the identified at least one UI element presented in the database. Further, the interaction engine determines whether the utterance of the received voice input matches the natural language utterance of the identified at least one UI element. Further, the interaction engine automatically accesses (or executes) at least one of the UI elements when the utterances of the voice input received from the user match the predicted natural language utterances of the identified at least one UI elements.

Unlike the conventional method, the proposed method executes the identified at least one UI element displayed on the screen (eg, an actionable UI element, Automatically make accessible non-actionable UI elements, text UI elements, and non-textual UI elements. Thereby, the user may utilize the natural language to access various UI components identified, and thus the user's experience may be improved. The natural language utterance is predicted by obtaining a database/knowledge graph based on characteristics (eg, relative location, performance, function, type, shape, etc.) of the identified at least one UI element. The user's voice input may include an utterance of a characteristic of the identified at least one UI element existing in the database.

Unlike the conventional method, the proposed method determines a degree of similarity between at least one UI element among other UI elements and each UI element, clusters each UI element among at least one UI element based on the similarity degree, and includes a visible feature determine the text representation of non-text UI elements and the relative position of each UI element among other UI elements, and obtain predefined information between each UI element among at least one UI element and the performance of each corresponding UI element. A knowledge graph may be obtained by mapping and determining a predefined screen sequence converted through an executable UI element. The degree of similarity between at least one UI element and each UI element among other UI elements is the position of each UI element, the relative position of each UI element among other UI elements, the function of each UI element, the performance of each UI element, and the It is determined based on the appearance (or shape) of each UI element.

Unlike the conventional method, the proposed method performs semantic translation on the obtained knowledge graph to obtain natural language transformations for single-step intent and/or multi-step intent. Moreover, the electronic device uses the obtained knowledge graph to identify actions and action sequences for single-step intents and/or multi-step intents, and maps natural language variants to the identified actions and identified action sequences to model the natural language model. predicts the identified at least one UI element by dynamically acquiring As a result, the user can access a sub-function or sub-page of the displayed application/UI element that enhances the user's experience without being displayed on the screen of the electronic device.

Referring to FIGS. 2A to 9C in more detail, like reference numerals are indicated in the embodiments to consistently indicate corresponding features in the drawings.

2A is a block diagram illustrating a configuration of an electronic device according to an exemplary embodiment.

The electronic device 100 may be, for example, a TV, a smartphone, a tablet, a laptop, an Internet of Things (IoT) device, smart glasses, or a smart watch, but is not limited thereto. According to an embodiment, the electronic device 100 includes a memory 110 and a processor 120 .

The memory 110 stores the identified at least one UI element and the characteristic of the database/knowledge graph, and the characteristic of the identified at least one UI element. In addition, the memory 110 stores instructions executed by the processor 120 . Memory 110 may include non-volatile storage elements. Examples of such non-volatile storage elements include magnetic hard disks, optical disks, floppy disks, flash memories, or electrically programmable memory (EPROM) or electrically erasable and programmable memory (EEPROM). may include Memory 110 may also be a non-transitory storage medium in some examples. Here, the word "non-transitory" denotes that the storage medium does not contain a carrier wave or a propagated signal. However, the term "non-transitory" should not be construed as meaning that memory is not removable. In one example, the memory 110 may store information of a larger capacity. In one example, the non-transitory storage medium may store data (eg, random access memory (RAM) or cache) that may change over time. The memory 110 may be an internal storage of the electronic device 100 , or may be an external storage of the electronic device 100 , cloud storage, or other types of external storage.

The processor 120 may perform instructions stored in the memory 110 and perform various processes. The processor 120 may include at least one processor, and may be a general-purpose processor such as a central processing unit (CPU) or an application processor (AP), a graphics-only processing unit such as a graphics processing unit (GPU), and visual processing. It may be an AI-only processor such as a device (VPU) and/or a neural processing unit (NPU).

According to an embodiment, the processor 120 identifies at least one characteristic of at least one User Interface (UI) element included in the screen 140 of the electronic device 100, and selects at least one of the identified UI elements. A database may be obtained based on the characteristics. Here, the database may include natural language utterances obtained based on at least one characteristic of the identified UI element. Here, the at least one characteristic may include at least one of a relative position of each UI element, a function of each UI element, a performance of each UI element, a type of each element, and a shape of each UI element. In this case, the database may be generated in real time based on the current screen 140 of the electronic device 100 .

Thereafter, when the user's voice input is received, the processor 120 identifies whether the utterance of the received voice input matches the natural language utterance included in the database, and identifies that the utterance of the voice input matches the natural language utterance , at least one UI element can be accessed automatically. Here, the at least one UI element may include at least one of an executable UI element, a non-executable UI element, a text UI element, or a non-text UI element.

Meanwhile, the processor 120 may identify whether the utterance of the received voice input matches the natural language utterance included in the database through a plurality of matching steps. Hereinafter, it has been described that text UI elements, synonyms (or synonyms) of text UI elements, and non-text UI elements are sequentially compared, but it is not necessarily limited to this order.

According to an example, the processor 120 may first compare the utterance of the voice input with the natural language utterance corresponding to the text UI element included in the database. Here, the text UI element may be an executable UI element. For example, when the natural language utterance corresponding to the text UI element is identified in the first comparison result database corresponding to the utterance of the voice input, the processor 120 may determine (eg, when the natural language utterance having a similarity greater than or equal to a threshold value is identified) , can automatically access the identified text UI element.

According to an example, when a specific text UI element is identified as a result of the first comparison, the second comparison may not be performed, but the present invention is not limited thereto. A comparison may be made.

Also, the processor 120 may second compare the speech input utterance with the natural language utterance corresponding to a synonym of the text UI element included in the database according to the first comparison result. For example, when a natural language utterance corresponding to a synonym (or a synonym) of a text UI element is identified in the second comparison result database, the processor 120 identifies (eg, a natural language utterance having a similarity greater than or equal to a threshold value) ), you can automatically access that text UI element. In this case, the processor 120 may identify the text UI element based on the first comparison result as well as the second comparison result.

According to an example, when a specific text UI element is identified as a result of the second comparison, the third comparison may not be performed, but the present invention is not limited thereto. A comparison may be made.

Also, the processor 120 may perform a third comparison of the speech input utterance with the natural language utterance corresponding to the shape of the non-text UI element included in the database according to the second comparison result. Here, the non-text UI element may be an executable UI element. For example, when the natural language utterance corresponding to the shape of the non-text UI element is identified in the third comparison result database (eg, natural language utterance having a similarity greater than or equal to a threshold value is identified), the processor 120 determines the identified ratio You can automatically access text UI elements. In this case, the processor 120 may identify the non-text UI element based on at least one of the first comparison result and the second comparison result as well as the third comparison result.

In addition, the processor 120 may obtain a learning result of the neural network model stored in the database, and identify whether or not matching is performed with at least one of the first comparison result, the second comparison result, or the third comparison result and the learning result. Here, the neural network model is a model trained to output a matching result by being learned by various learning data, which will be described in detail later.

Also, the processor 120 may obtain a matching score for at least one UI element included in the screen. Here, the matching score may be obtained based on the degree of similarity by comparing the utterance of the voice input with the natural language utterance corresponding to the UI element included in the database.

In this case, when the matching score of each of the plurality of UI elements included on the screen is equal to or greater than the first threshold value, and the difference between the matching scores of each of the plurality of UI elements is within the second threshold value, the processor 120 selects one of the plurality of UI elements. A guide UI for selecting any one may be provided. Here, the first threshold value may be preset as a numerical value for identifying the same/similarity of the natural language utterance corresponding to the UI element included in the database for the utterance of the voice input, or may be set by the user. In addition, the second threshold value may be preset as a value for identifying the degree to which the matching degree of each of the plurality of UI elements is considered to be the same/similar, or may be set by the user.

In addition, when the matching score of one UI element is equal to or greater than the first threshold value and greater than the second threshold value than the other UI elements, the processor 120 may access one UI element, that is, execute the corresponding UI element. However, even in this case, the processor 120 may perform a guide UI requesting the user's execution confirmation for the corresponding UI element.

Also, when the matching scores of all UI elements included in the screen are less than the first threshold value, the processor 120 may provide a guide UI providing corresponding information. For example, the processor 120 may provide information indicating that a UI element corresponding to the user's voice input cannot be identified. However, even in this case, the processor 120 may provide a UI element having the highest matching score as a recommended UI element.

In addition, the processor 120 determines the position of each UI element, the relative position of each UI element with respect to other UI elements, the function of each UI element, the capability of each UI element, the type of each UI element, and each A knowledge graph may be obtained by identifying at least one of the shapes of UI elements, and the obtained knowledge graph may be stored in a database.

In addition, the processor 120 determines the degree of similarity between each UI element based on the position of each UI element, the relative position of each UI element, the function of each UI element, the performance of each UI element, and the similarity of at least one of the shape of each UI element. It is possible to identify and cluster each UI element based on the identified similarity to obtain a knowledge graph, and store the acquired knowledge graph in a database.

In addition, the processor 120 may determine a textual representation of the non-text UI element from the visible characteristics and relative position of each UI element.

In addition, the processor 120 may obtain a knowledge graph by mapping each UI element and predefined information corresponding to the performance of each UI element, and store the acquired knowledge graph in a database. In this case, the processor 120 may obtain a knowledge graph by determining a predefined screen sequence transferred through an operable UI element.

Also, the processor 120 may obtain a knowledge graph by determining a predefined screen sequence transferred through an operable UI element.

In addition, the processor 120 performs semantic translation on the knowledge graph to obtain natural language variations for at least one of the single-step intent and the multi-step intent, and uses the knowledge graph to perform the single-step intent and a natural language model for predicting natural language utterances of the identified UI elements by identifying at least one action and at least one action sequence for at least one of the multi-step intents and mapping the obtained natural language variants with the identified actions and action sequences. It can be created dynamically.

In addition, the processor 120 categorizes each UI element of the knowledge graph into at least one of a domain, a verb, a synonym, a slot, a slot type, a slot expressed as a texture, a performance, and a relative position on the screen 140, Based on it, a natural language transformation can be obtained.

In addition, the processor 120 includes information on the types of UI elements included on the screen 140 and performance corresponding thereto, verbs related to performance, performance on action information, a graph of an element sequence operable on the screen, and a corresponding action sequence , a predefined table set including predefined actions and action sequences may be determined, and an action routine may be determined based on the performance of each UI element. In this case, a unique identity may be assigned to a predefined action and an action sequence.

Further, the processor 120 clusters similar natural language variants, assigns a dynamic intent to the similar natural language variants, associates the at least one identified action and the at least one identified action sequence with the dynamic intent, and A natural language model may be dynamically generated based on transformations, dynamic intentions, and action routines, and the natural language model may be stored in a database.

In addition, the processor 120 determines a text expression score by reading screen information included on the screen 140 , extracts synonyms for verbs and nouns from text expressions and text UI elements included in the screen information, and scores a synonym The relevance score can be determined by assigning , determining a variance score correlating the natural language variants used for weighted learning to the dynamic language generator 170E, and comparing the proximity element information with reference objects included in the utterance of the speech input. there is.

It may also determine a matching score for dynamic language generator 170E as a final score that matches an utterance of the voice input, and combine the determined matching score with a relevance score.

2B is a block diagram illustrating the electronic device 100 for automatically accessing a UI element displayed on the screen 140 of the electronic device 100 based on the user's voice interaction with the electronic device 100, according to an embodiment. It is also

In an embodiment, the electronic device 100 includes a memory 110 , a processor 120 , a communication unit 130 , a display 140 , a sensor 150 , an application controller 160 , and an interaction engine 170 . can do. A detailed description of the configuration overlapping with the configuration shown in FIG. 2A among the configurations shown in FIG. 2B will be omitted.

The communicator 130 includes a standard-based electronic circuit that enables wired or wireless communication. The communication unit 130 is configured so that internal hardware components and external devices communicate internally through at least one network.

The at least one sensor 150 or 150a-n is, for example, an ambient light sensor, a 3-axis accelerometer, an altimeter, an optical heart rate sensor. , oxygen saturation (SpO2) monitor, bioimpedance sensor, proximity sensor, compass, Electrocardiogram (ECG) sensor) , Global Positioning System (GPS), gyroscope, gesture sensor, Ultraviolet (UV) sensor, magnetometer, electrodermal activity sensor, skin It may include, but is not limited to, a temperature sensor (skin temperature sensor) and the like.

The application controller 160 is configured to control at least one application 160a - 160n of the electronic device 100 . Examples of the application include, but are not limited to, a web application, a video player application, a camera application, a business application, a learning application, a health application, a lifestyle application, an entertainment application, a utility application, a travel application, and the like.

In one embodiment, interaction engine 170 includes logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronics may be implemented with processing circuits such as passive electronic components, active electronic components, optical components, hardwired circuits, etc., and selectively driven by firmware there is. The circuits may be implemented with, for example, at least one semiconductor (semiconductors).

In one embodiment, the interaction engine 170 includes a context orchestrator 170a, a visual semantic NL estimator 170b, an action sequence planner 170c, a natural language synthesizer. (NL synthesizer) 170d, dynamic language generator 170e (eg, voice assistant), and artificial intelligence (AI) engine 170f.

In an embodiment, the context orchestrator 170a identifies at least one UI element displayed on the screen 140 of the electronic device 100 using the AI engine 170f. At least one UI element (eg, click button, home icon, text bar, etc.) may include an executable UI element, a non-executable UI element, a text UI element and a non- Contains text UI elements. Further, the context orchestrator 170a determines the characteristic(s) of the at least one UI element identified using the AI engine 170f. The identified characteristics of the at least one UI element include the position of each UI element, the relative position of each UI element among other UI elements, the function of each UI element, the performance of each UI element, the type of each UI element, and also the electronic device. The appearance of each UI element displayed on the screen 140 of 100 is included.

Further, the context orchestrator 170a obtains a database/knowledge graph based on the characteristic(s) of the at least one UI element identified using the AI engine 170f. In addition, the context orchestrator 170a determines the position of each UI element, the relative position of each UI element among other UI elements, the function of each UI element, the performance of each UI element, the type of each UI element, and also the electronic device ( 100) determines the appearance of each UI element displayed on the screen 140 . In addition, the context orchestrator 170a determines the position of each UI element, the relative position of each UI element among other UI elements, the function of each UI element, the performance of each UI element, the type of each UI element, and also the electronic device ( 100), a degree of similarity between each UI element among at least one UI element among other UI elements is determined based on the appearance of each UI element displayed on the screen 140 of FIG. Also, the context orchestrator 170a clusters each UI element among at least one UI element based on the similarity. In addition, the context orchestrator 170a determines, from the visible characteristics, the text representation among non-text UI elements and the relative position of each UI element among other UI elements. In addition, the context orchestrator 170a maps predefined information between each UI element among at least one UI element and the performance of each corresponding UI element. In addition, the context orchestrator 170a determines a predefined screen sequence switched through the executable UI element. In addition, the context orchestrator 170a stores the acquired knowledge graph in the database.

In an embodiment, the visual semantic natural language predictor 170b performs semantic translation on the obtained knowledge graph to obtain natural language transformations for single-step intent and/or multi-step intent. In addition, the visual semantic natural language predictor 170b receives the knowledge graph obtained from the context orchestrator 170a. In addition, the visual semantic natural language predictor 170b calculates each UI element of the received knowledge graph as a domain, a verb, a synonym, a slot, a slot type, a text display slot, a performance, and a relative position on the screen 140 of the electronic device 100 . categorize as In addition, the visual semantic natural language predictor 170b acquires natural language transformations according to categorization.

In one embodiment, the action sequence planner 170c uses the acquired knowledge graph to identify actions and action sequences for single-step intents and/or multi-step intents. In addition, the action sequence planner 170c receives the knowledge graph obtained from the context orchestrator 170a. In addition, the action sequence planner 170c determines an action routine based on the performance of each UI element.

In an embodiment, the natural language synthesizer 170d predicts natural language utterance of the identified at least one UI element by dynamically acquiring the natural language model and mapping the obtained natural language transformation to the identified action and the identified action sequence. In addition, natural language synthesizer 170d clusters similar natural language variants. In addition, natural language synthesizer 170d imposes dynamic intent for similar natural language variants. In addition, natural language synthesizer 170d associates the dynamic intent with the identified activity and the identified sequence of activities. In addition, the natural language synthesizer 170d dynamically acquires a natural language model based on the clustered natural language transformation, dynamic intent, and action routine. In addition, the natural language synthesizer 170d stores the dynamically acquired natural language model in the database.

In an embodiment, the dynamic language acquirer 170e receives an input from the natural language synthesizer 170d. In addition, the dynamic language acquirer 170e receives a voice input from the user of the electronic device 100 , wherein the voice input includes an utterance indicating a characteristic of the identified at least one UI element presented in the database. Further, the dynamic language acquirer 170e determines whether an utterance of the received voice input matches a predicted natural language utterance of the identified at least one UI element. Further, the dynamic language acquirer 170e automatically accesses at least one of the UI elements if the utterances of the voice input received from the user match the predicted natural language utterances of the identified at least one UI elements.

As described above, at least one or more modules/components may be implemented through the AI engine 170f. Functions related to the AI engine 170f may be performed through the memory 110 and the processor 120 . One or more processors may control processing of input information according to a predefined operation rule or AI engine 170f stored in non-volatile memory or volatile memory. A predefined action rule or artificial intelligence model is provided through training or learning.

Here, being provided through learning means that a predefined operation rule or AI engine 170f of a desired characteristic is created by applying a learning process to a plurality of learning data. Learning may be performed in the electronic device 100 itself performing AI according to the present embodiment, or may be implemented through a separate server/system.

The AI engine 170f may be configured with a plurality of neural network layers. Each layer has a plurality of weights, and a layer operation is performed through calculation of the previous layer and an operation between the plurality of weights. Examples of neural networks include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), and GAN (GAN). Generative Adversarial Networks) and Deep Q-Networks, which are not limited to the above examples.

The learning process is a method of training, allowing, and controlling a predetermined target device (eg, a robot, an IoT device) to make a decision or make a prediction on its own by using a plurality of learning data. . Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the above example.

Although various hardware components of the electronic device 100 are illustrated in FIGS. 2A and 2B , it should be understood that other embodiments are not limited thereto. In another embodiment, the electronic device 100 may include fewer or more components. In addition, labels or names of components are used only for description and do not limit the scope of the invention. At least one component may be coupled together to perform the same or substantially similar function as the voice-based interaction.

3 is a flowchart 300 illustrating a method of automatically accessing a UI element displayed on the screen 140 of the electronic device 100 based on a voice interaction between the user and the electronic device 100 according to an exemplary embodiment. . Operations 302-312 may be performed by the electronic device 100 .

According to the method illustrated in FIG. 3 , at least one UI element displayed on the screen 140 of the electronic device 100 is identified in 302 .

Also, at 304 , a characteristic of the identified at least one UI element is determined.

Further, at 306 , a database is obtained based on a characteristic of the at least one identified UI element, the database comprising a natural language utterance of the at least one identified UI element, wherein the natural language utterance is at least one of the identified at least one UI element. predicted based on the characteristics of

Also, in 308 , a voice input of the user of the electronic device 100 is received, and the voice input includes an utterance of a characteristic of the identified at least one UI element appearing in the database.

Also, at 310 , it is determined whether the utterance of the received voice input matches the predicted natural language utterance of the identified at least one UI element.

In addition, if it is determined in 312 that the utterance of the received voice input from the user matches the predicted natural language utterance of the identified at least one UI element, the UI element among the at least one UI element is automatically accessed.

Each of the actions, acts, blocks, steps, etc. of the flowchart 300 may be performed in the order depicted, in a different order, or concurrently. Also, in some embodiments, some actions, operations, blocks, steps, etc. may be omitted, added, modified, or skipped without departing from the scope of the present invention.

4A and 4B are diagrams illustrating a model for predicting natural language utterance of a UI element related to a video application displayed on the screen 140 of the electronic device 100 and simulating an action related to the displayed UI element dynamically, according to an embodiment. This is an example of a flowchart explaining how to (dynamically) build.

According to FIG. 4A , in 401 , it is assumed that the user of the electronic device 100 is looking at the screen 140 and wants to check an unread notification (eg, a bell-shaped notification icon) of a video application.

In 402 (platform 402), screen reading APIs (eg, accessibility 402a, optical character recognition (OCR) 402b, and image classification 402c) read screen information and corresponding characteristics, and this screen information These may be supplied to the context orchestrator 170a to generate a knowledge graph.

In 403 , the context orchestrator 170a analyzes screen information such as a view hierarchy, content of UI elements (eg, textual representation and non-textual representation), and positions of UI elements. Context orchestrator 170a also obtains inferences such as the relative position of textual representation and UI element performance. In addition, the context orchestrator 170a stores and learns information such as a screen sequence related to a specific application (eg, a video application).

In an embodiment, the context orchestrator 170a includes a relationship predictor 170aa, a text presenter 170ab, a UI element performance identifier 170ac, and a screen sequence detector 170ad. The relationship predictor 170aa groups UI elements (eg, a bell-shaped icon, a search icon, etc.), determines their relative positions, and expresses them as textual expressions. Table 2 shows examples of input-output of the relationship prediction unit 170aa.

Input Output Name: Connect Device,
Position : 50, 1180,
Type: Button Relation: Left of bell-shaped Button
Group : {Textbox, Button} Name: Notification,
Position :150, 1180,
Type: Button Relation: Right to connect device button
Group : {Textbox, Button}

The text presenter 170ab converts all non-textual representations (non-textual content/UI elements) into textual representations (textual content/UI elements) and obtains inferences between all non-textual representations. Table 3 shows an input-output example of the text presenter 170ab.

Input Output Name: Home, Position: 50, 1180,
Type: Button, Image Buffer<> Text representation: Shape: Home Name : Notification, Position : 150, 1180, Type : Button, Image Buffer<> Text representation: Shape: Bell

The UI element performance identifier 170ac identifies a predefined performance of a UI element (eg, an operable UI element) and stores the performance of the UI element in the context orchestrator 170a. Table 4 shows an example of the input-output of the UI element capability identifier 170ac.

Input Output Name: Home, Position: 50, 1180,
Type: Button Capability : Click, Focus

Screen sequence detector 170ad determines view screen transitions that include additional actionable UI elements recorded as part of context orchestrator 170a. Table 5 shows an example of input-output of the picture sequence detector 170ad.

Input Output Application: YouTube/video application Home Screen => Account => Data Home Screen => Connect a device

In addition, the context orchestrator 170a obtains a knowledge graph 404 using the functions of the relationship predictor 170aa, the text presenter 170ab, the UI element performance identifier 170ac, and the screen sequence detector 170ad. . Table 6 shows an example of a notification (UI element) of the acquired knowledge graph 404 .

UI element knowledge graph
information Notification Loc: (1020, 10) Name: Notifications Content shape: Bell Content-type: Icon Type: Button Capability: Press, Focus

Referring to FIG. 4B , at 405 , the visual semantic natural language predictor 170b receives the knowledge graph obtained from the context orchestrator 170a.

In addition, the visual semantic natural language predictor 170b calculates each UI element of the received knowledge graph as a domain, a verb, a synonym, a slot, a slot type, a textually expressed slot, a performance, and a relative position of the screen of the electronic device 100 . (140). Also, the visual semantic natural language predictor 170b obtains a natural language transformation based on the classification. Table 7 shows examples of natural language classification and natural language transformations related to notifications (UI elements) of the visual semantic natural language predictor 170b.

Domain Video Application YouTube NL categorization
(notification) Relation : { Left of Search, Right of Connect} Verb: Touch, Tap, Click Slot : { Bell <Shape>, Notification <name> } Type : { Button, Icon, Image } NL variation Click on the bell-shaped button Tap the icon left to search icon

At 406 , the action sequence planner 170c receives the obtained knowledge graph from the context orchestrator 170a . In addition, the action sequence planner 170c determines an action routine/action sequence graph for single-step and multi-step commands/intents based on the performance of each UI element. In addition, the action sequence planner 170c determines a predefined set of action routines/sequences related to the performance list found in the user interface. Operation routines/sequences may be static or dynamic. Static action sequences are tabulated because the sequence does not change. A dynamic action sequence is built during execution based on the screen sequence of the knowledge graph. Table 8 shows an example of an action routine related to a UI element of the action sequence planner 170c.

UI element type Capability Action Routines Button Click Subroutine: Click
Parameters: UI element position text box Click Subroutine: Write
Parameters: UI Element, Content text box Focus Subroutine: Focus
Parameters: UI Element list box Scroll up Subroutine: Scroll Up
Parameters: UI Element Multi-step Ex
switch account Click (Account) > Click (Switch)
Account)

In addition, the action sequence planner 170c determines the action sequence identity based on the actionable UI element graph of the screen stored in the knowledge of the context orchestrator 170a. The screen's actionable UI element graph can be a predefined input or can learn from understanding the user's screen navigation actions (both touch and voice-based actions).

At 407 - 408 , natural language synthesizer 170d receives inputs from visual semantic natural language predictor 170b and action sequence planner 170c . The natural language synthesizer 170d clusters similar natural language variants, and the natural language variants are determined by the visual semantic natural language predictor 170b. The natural language synthesizer 170d then assigns dynamic intents for similar natural language variants, and associates the dynamic intents with the identified actions and the identified action sequences, and the identified actions and the identified action sequences are combined with the action sequence planner 170c. is determined by Table 9 shows examples of natural language transformations and action sequences similar to dynamic intentions in the natural language synthesizer 170d.

Dynamic intent ID-1 NL variations Click on the bell-shaped button Click search button Tap plus image action sequence AS1

At 409 , dynamic language acquirer 170e receives input from natural language synthesizer 170d and dynamically acquires a natural language model based on clustered natural language transformations, dynamic intent, and action routines. Also, the dynamic language acquirer 170e stores the dynamically acquired natural language model in the database. At 410 , the dynamic language acquirer 170e receives a voice input (eg, click a bell button). The dynamic language acquirer 170e determines whether an utterance of the received voice input matches a predicted natural language utterance of the at least one identified UI element. In 411-412 , when the utterance of the received voice input from the user matches the predicted natural language utterance of the at least one identified UI element, the dynamic language acquirer 170e is configured to display the screen 140 of the electronic device 100 . Automatically access UI elements (such as notifications) of the video app displayed in . As a result, users can utilize natural language to access identified UI elements that enhance the user experience. The proposed method simplifies user interaction so that the user naturally operates the button displayed on the screen of the electronic device 100 even if the user does not know the real name of the button displayed on the screen of the electronic device 100 . The proposed method shows the quality and relationship of non-textual components in a textual way.

5 is a diagram in which the electronic device 100 predicts natural language utterance of a UI element related to a social media application displayed on the screen 140 and dynamically builds a model for simulating an action related to the displayed UI element, according to an embodiment. This is an example of a scenario.

In 501 , a scenario is assumed in which the user of the electronic device 100 browses through a social media application (eg, F-share) and expresses a liking for a picture displayed on the screen 140 very naturally.

The technology flow is the same as described in FIG. 4 . At 502 , the context orchestrator 170a obtains a knowledge graph using the functions of the relationship predictor 170aa , the textual presenter 170ab , the UI element performance identifier 170ac , and the screen sequence detector 170ad . Table 10 shows an example regarding the notification (UI element) of the acquired knowledge graph.

UI element knowledge graph Notification Loc: (700, 10) Name: Like Content shape: Heart Content-type: icon Type: Button Capability: Press, Focus

Then, the visual semantic natural language predictor 170b receives the obtained knowledge graph from the context orchestrator 170a. In addition, the visual semantic natural language predictor 170b calculates each UI element of the received knowledge graph as a domain, a verb, a synonym, a slot, a slot type, a textually expressed slot, a performance, and a relative position of the screen of the electronic device 100 . (140). In addition, the visual semantic natural language predictor 170b obtains a natural language transformation based on the classification. Table 11 shows examples of natural language classification and natural language transformations related to notifications (UI elements) of the visual semantic natural language predictor 170b.

Domain Image app F-share NL categorization
(notification) Relation : { Bottom to photo, Above
Android} Verb: Touch, Tap, Click … Slot: { Heart, Like, Love} Type: { Button, Icon, Image } NL variation Click on the heart-shaped icon I like the photo

Then, the action sequence planner 170c receives the obtained knowledge graph from the context orchestrator 170a. In addition, the action sequence planner 170c determines an action routine/action sequence graph for single-step and multi-step commands/intents based on the performance of each UI element. Further, the action sequence planner 170c determines a predefined set of action routines/sequences related to the performance list obtained in the user interface. Operation routines/sequences may be static or dynamic. Static action sequences are tabulated because the sequence does not change. The dynamic action sequence is built during execution based on the screen sequence in the knowledge graph (while the user controls the screen, the dynamic action sequence is acquired in real time. This is different from the traditional access required by the developer to design the action sequence explicitly. different).

In addition, the action sequence planner 170c determines the action sequence identity based on the actionable UI element graph of the screen stored in the knowledge of the context orchestrator 170a. The screen's actionable UI element graph can be a predefined input or can learn from understanding the user's screen navigation actions (both touch and voice-based actions).

In addition, the natural language synthesizer 170d receives inputs from the visual semantic natural language predictor 170b and the action sequence planner 170c. The natural language synthesizer 170d collects similar natural language variants, and the corresponding natural language variants are determined by the visual semantic natural language predictor 170b. In addition, natural language synthesizer 170 assigns dynamic intents to similar natural language variants, associates the dynamic intents with identified actions and identified action sequences, which identified actions and identified action sequences are combined with the action sequence planner ( 170c). Table 12 shows examples of similar natural language variants and dynamic intent and action sequences in natural language synthesizer 170d.

Dynamic intent ID-2 NL variations Click on the like icon I like the photo Click on the love button action sequence AS2

The dynamic language generator 170e receives input from the natural language synthesizer 170d and dynamically generates a natural language model based on the collected natural language variants, dynamic intent, and action routines. In addition, the dynamic language generator 170e stores the dynamically generated natural language model in the database. The dynamic language generator 170e receives a voice input (eg likes a picture). The dynamic language generator 170e determines whether an utterance of the received voice input matches predicted natural language utterances of at least one identified UI element. When the utterance of the received voice input matches the predicted natural language utterances of the identified at least one UI element, the dynamic language generator 170e displays the UI element of the social media application displayed on the screen 140 of the electronic device 100 ( e.g. a heart-shaped icon). As a result, the user may utilize natural language to access the identified UI element(s), which enhances the user's experience.

6 illustrates a score mechanism for automatically accessing a UI element displayed on the screen 140 of the electronic device 100 based on a voice interaction with the electronic device 100, according to an embodiment.

The text expression score 601 is determined by reading screen information displayed on the screen 140 of the electronic device 100 , and the screen information is in the form of a text expression/text UI element. Context orchestrator 170a also includes text representation score 601 . While extracting synonyms of verbs and nouns from text representation/text UI element(s), a synonym score 602 is assigned. The greater the distance between the synonyms and the main expression (textual representation), the lower the synonym score 602 . The visual semantic natural language predictor 170b includes a synonym score 602 and receives an input from a text representation score 601 . The variant score 603 is determined by associating the generated natural language variants with the dynamic language generator 170e that may be used to perform weight learning. The relevance score 604 is determined by comparing adjacent element information with reference objects mentioned in the utterance of the received voice input. The matching score 605 is a final score obtained by matching the utterance of the received voice input with the dynamic language generator 170e and combining it with the relevance score 604 .

In one embodiment, the interaction engine 170 extracts the best matching content from the screen information. If multiple matches exist, the corresponding score is used for clear reconciliation.

7 is a score in order for the electronic device 100 to predict natural language utterances of a UI element of a social media application displayed on the screen 140 and dynamically build models for an action simulation associated with the displayed UI element, according to an embodiment. This is an example scenario using the mechanism.

At 701 , assume a scenario in which the user of the electronic device 100 is browsing a social media application (eg, F-share) and expresses a very natural way of liking a picture displayed on the screen 140 . In this scenario, different scores are searched for by the interaction engine 170 , and the interaction engine 170 selects the best match among the found screen matches. There are two heart-shaped icons named element 1 and element 2 on the screen 140 . In this scenario, it can be discussed which one is best for the user.

The context orchestrator 170a generates a knowledge graph using the functions of the relevance measurer 170aa, the text presenter 170ab, the UI element function determiner 170ac, and the screen sequence detector 170ad. Table 13 shows an example of a notification (UI element) of the generated knowledge graph.

UI element knowledge graph Notification Loc: (700, 10) Name: Like Content shape: Heart (098) “textual
representation score (601)” Content-type: icon Type: Button Capability: Press, Focus

In addition, the visual semantic natural language predictor 170b receives the knowledge graph generated from the context orchestrator 170a. In addition, the visual semantic natural language predictor 170b compares each UI element of the input knowledge graph with a domain, a verb, a synonym, a slot, a slot type, a text-expressed slot, a function, and a relative position on the screen of the electronic device 100 ( 140). In addition, the visual semantic natural language predictor 170b generates natural language variants based on the classification. Table 14 shows examples of natural language classification and natural language transformations for notifications (UI elements) of the visual semantic natural language predictor 170b.

domain image application F-share RELATED: {below photo, above Android} natural language
classification
(notice) Verbs: touch (0.7), tap (0.9), click (1.0)
"Thesaurus Score (602)" Slots: {Heart(1.0), Like(0.8), Very Like(0.8)}
"Thesaurus Score (602)" Type: {button, icon, image} natural language transformation Click the heart icon (0.98) "Variation score (603)" Likes the picture (0.8) "Variation score (603)"

Also, the action sequence planner 170c receives the knowledge graph generated from the context orchestrator 170a. In addition, the action sequence planner 170c determines an action routine/action sequence graph for single-step and multi-step commands/intents based on the function of each UI element. In addition, the action sequence planner 170c determines a preset set of action routines/sequences associated with the list of functions found in the user interface. Action routines/sequences can be static or dynamic. A static action sequence is created as a table since the sequence does not change. The dynamic action sequence is configured on-fly based on the screen sequence of the knowledge graph.

In addition, the natural language synthesizer 170d receives inputs from the visual semantic natural language predictor 170b and the action sequence planner 170c. The natural language synthesizer 170d clusters similar natural language variants, and the corresponding natural language variants are determined by the visual semantic natural language predictor 170b. In addition, natural language synthesizer 170 assigns a dynamic intent to similar natural language variants, associates the dynamic intent with an identified action and an identified action sequence, and the identified action and the identified action sequence are combined with the action sequence planner ( 170c). Table 15 shows examples of similar natural language variants and dynamic intent and action sequences in natural language synthesizer 170d.

Dynamic intent ID-2 NL variations Click on the like icon I like the photo Click on the love button action sequence AS2

In addition, the dynamic language generator 170e receives input from the natural language synthesizer 170d and dynamically generates a natural language model based on the clustered natural language variants, dynamic intent, and action routines. In addition, the dynamic language generator 170e stores the dynamically generated natural language model in the database. The dynamic language generator 170e receives a voice input (I like the photo), and the dynamic language generator 170e determines whether an utterance of the received voice input matches predicted natural language utterances of at least one identified UI element. do. Table 16 shows an example of candidate estimation in the dynamic language generator 170e.

Candidate estimation Element-1 Heart Loc (700, 10) (08) Element-2 Heart Loc (980, 1200) (08)

In addition, the dynamic language generator 170e determines the relevance score 604 by comparing the adjacent element information with reference objects mentioned in the utterance of the received voice input. (i.e. I like the photo(0.9)).

Also, the dynamic language generator 170e determines the matching score 605 as the final score. Table-17 shows an example of a match score 605 in the dynamic language generator 170e.

final output Element-1 Heart Loc (700, 10) (09) (winner) Element-2 Heart Loc (980, 1200) (08)

When the utterance of the received voice input matches the predicted natural language utterances of the identified UI element, the dynamic language generator 170e displays a UI element (eg, heart) of the social media application displayed on the screen 140 of the electronic device 100 . automatically access the icon (element-1). In this scenario, the user mentions "Like" and "Photo", and the interaction engine 170 was supposed to actuate the element-1 heart icon based on Table 17.

At 702 , assume a scenario in which the dynamic language generator 170e receives a voice input (eg, clicking a heart icon). The dynamic language generator 170e determines whether the utterance of the received voice input matches predicted natural language utterances of the identification UI element. Table 18 shows an example of candidate estimation in dynamic language generator 170e.

Candidate estimation Element-1 Heart Loc (700, 10) (08) Element-2 Heart Loc (980, 1200) (08)

In addition, the dynamic language generator 170e determines the relevance score 604 by comparing the adjacent element information with reference objects mentioned in the utterance of the received voice input. (ie click the heart icon (0)). Also, the dynamic language generator 170e determines the matching score 605 as the final score. Table 19 shows an example of a match score 605 in the dynamic language generator 170e.

final output Element-1 Heart Loc (700, 10) (08) Element-2 Heart Loc (980, 1200) (08)

For multiple matches with the same score in this scenario, it is necessary to start a disambiguation flow to allow the user to select one of them. Since the utterance has an indication to the target element without any supportive information, the relevance score 604 is zero. Therefore, this process will introduce ambiguity to the user interface.

8 shows a comparison of existing methods with a limited set of control and search instructions and a proposed method of predicting natural language utterances of UI elements for multi-step intent, according to an embodiment of the present disclosure.

801 to 803 indicate existing methods, where a scenario in which a user wants to save/bookmark a page of a search application displayed on the screen of the existing electronic device 10 is assumed.

In this scenario, the existing method does not access the sub-function or sub-page not displayed on the screen of the electronic device 10 of this navigation application, so the user has to manually perform the step-by-step process. Using existing methods, the user can only access UI elements displayed on the screen of the electronic device 10 .

804 to 806 indicate the method proposed in the present disclosure, wherein the user writes to the screen of the electronic device of the displayed application/UI element by utilizing natural language (ie, by storing the bookmark as a voice input to the electronic device 100). Sub-functions or sub-pages that are not displayed may be accessed, which may improve the user's experience.

9A to 9C illustrate functions related to the natural language synthesizer 170d of the electronic device 100 for automatically accessing a UI element displayed on the screen 140, according to an exemplary embodiment.

The natural language synthesizer 170d includes a natural language generation task 901 , a dynamic intent label 902 , a dynamic language model 903 , and a dynamic pseudo capsule. ) 904 . The natural language generation task 901 includes the following modules, and a semantic feature builder accumulates all screen information and corresponding grammar tags. A command encoder uses grammar tags to obtain variants with slot place holders. A command decoder uses real-time screen information, ie, tag values and their substitutes, for the output variants of the command decoder. The command decoder obtains the final utterance variants. The dynamic intent label 902 includes the following modules, and the unsupervised similarity classifier is a module for determining similar utterances and classifying them into buckets. A dynamic label generator assigns a dynamic ID to a group of similar utterances within a bucket. The dynamic language model 903 includes the following modules, and the slot expansion module replaces the variants with real slot values to obtain a complete utterance. A language modeler forms a language model in real time using the obtained transforms. In the dynamic pseudo capsule 904 , the intent and action determination module receives input from the dynamic intent label 902 and the action sequence planner 170c and writes a prediction based on the received inputs to the memory 110 (eg, Bixby). stored in a pseudo dynamic capsule).

In one embodiment, the interaction engine 170 assists in generating single/multiple intentional utterances based on single-step or multi-step. The interaction engine 170 determines named entities along with domain and classification information from the screen-text and uses this information to generate open domain natural language utterances. The acquired natural language utterances are classified and learned by a dynamic language model, and it is further used to improve the accuracy of Audio Speech Recognition (ASR) and Natural Language based End Point Detection (NLEPD) of VA (Voice Assistance) (eg, Bixby in this case). will help In addition, the interaction engine 170 dynamically substitutes natural language intent and action pairs into the voice-assisted VA NLU system for intent prediction, action planning, and execution. Context information obtained from structures such as natural language (NL), natural language classification intent (NL Categorical Intent), and dynamic language model (Dynamic LM) may be refreshed for every change on the screen 140 . Multi-modal information is fused to clarify the selection of on-screen content through voice, and to associate and distinguish names for similar control types on different applications.

The interaction engine 170 classifies natural language understanding (NL) into intents, associating the obtained natural language understanding with operational routines, and eliminating the need to explicitly learn the user's voice interaction for any application. In addition, the interaction engine 170 associating a sequence of screen actions helps to establish a single/multiple intent natural language for a single action or a multi-action action. Using the obtained classified natural language to assist ASR in more appropriate speech to text (STT) recognition, and to determine EPD (End Point Detection) for unlearned utterances, NLEPD (Natural A dynamic language model to assist Language based End Point Detection) is obtained.

In addition, the interaction engine 170 uses the better named object recognition in the speech recognition module, for example, the Bixby ASR module, and uses the dynamically acquired language model to contextually correct the EPD. In addition, since the interaction engine 170 uses dynamic NL Capsule development from screen context information, explicit natural language learning is not required. Also, the interaction engine 170 may disambiguate screen control contents using a dynamic action sequence using a multi-modal fusion and screen transition sequence together with screen context information.

Embodiments of the present disclosure may be performed by using at least one hardware device and performing network management functions to control components.

Also, the methods according to the above-described various embodiments may be implemented only by upgrading software or hardware of an existing electronic device.

In addition, the various embodiments described above may be performed through an embedded server provided in the electronic device or an external server of the electronic device.

Meanwhile, according to a temporary example, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage media readable by a machine (eg, a computer). A device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, the electronic device A) according to the disclosed embodiments. When the instruction is executed by the processor, the processor may perform a function corresponding to the instruction by using other components directly or under the control of the processor. Instructions may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish that data is semi-permanently or temporarily stored in the storage medium.

Also, according to an embodiment, the method according to the various embodiments described above may be included in a computer program product and provided. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

In addition, each of the components (eg, a module or a program) according to the various embodiments described above may be composed of a single or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be omitted. Components may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity, so that functions performed by each corresponding component prior to integration may be performed identically or similarly. According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallelly, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added. can

In the above, preferred embodiments have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and those of ordinary skill in the art pertaining to the disclosure without departing from the gist of the claims. Various modifications are possible by the present disclosure, of course, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

100: electronic device 110: memory
120: processor

Claims (20)

In the control method of the electronic device 100,
identifying at least one user interface (UI) element included in the screen 140 of the electronic device 100 ;
identifying at least one characteristic of the identified UI element;
obtaining a database including natural language utterances obtained based on at least one characteristic of the identified UI element;
when a voice input is received, identifying whether an utterance of the received voice input matches the natural language utterance included in the acquired database; and
automatically accessing the at least one UI element when the utterance of the voice input is identified as matching the natural language utterance;
At least one characteristic of the identified UI element is:
The position of each UI element, the relative position of each UI element with respect to other UI elements, the function of each UI element, the capability of each UI element, the type of each UI element, or the A control method comprising at least one of an appearance. According to claim 1,
The at least one UI element,
A control method comprising at least one of an executable UI element, a non-executable UI element, a text UI element, or a non-text UI element. According to claim 1,
The step of identifying whether it matches the natural language utterance comprises:
first comparing the utterances of the voice input with natural language utterances corresponding to text UI elements included in the database;
second comparing the utterances of the voice input with natural language utterances corresponding to synonyms of text UI elements included in the database according to the first comparison result;
A third comparison step of comparing the utterance of the voice input with a natural language utterance corresponding to a shape of a non-text UI element included in the database according to the second comparison result;
and identifying whether the match is made based on at least one of the first comparison result, the second comparison result, and the third comparison result. According to claim 1,
The step of identifying whether it matches the natural language utterance comprises:
obtaining a learning result of a neural network model stored in the database; and
and identifying whether the match is made based on at least one of the first comparison result, the second comparison result, or the third comparison result and the learning result. The method of claim 1,
The step of identifying whether matching with the natural language utterance comprises:
Including; obtaining a matching score for at least one UI element included in the screen;
The step of automatically accessing the at least one UI element comprises:
obtaining a matching score for at least one UI element included in the screen;
Providing a guide UI for selecting any one of the plurality of UI elements when the matching score of each of the plurality of UI elements is equal to or greater than the first threshold value, and the difference between the matching scores of each of the plurality of UI elements is within the second threshold value step; and
When the matching score of one UI element is equal to or greater than the first threshold value and greater than the second threshold value than the other UI elements, executing the one UI element or requesting a user's execution confirmation for the one UI element Providing a guide UI; including, a control method. 6. The method of claim 5,
The step of automatically accessing the at least one UI element comprises:
When the matching score of all UI elements included in the screen is less than the first threshold value, at least one of a guide GUI notifying the corresponding information or a guide UI including at least one recommended UI element is provided. According to claim 1,
Acquiring the database includes:
identifying a degree of similarity between at least one of a degree of similarity between each UI element based on the position of each UI element, a relative position of each UI element, a function of each UI element, a performance of each UI element, and a shape of each UI element;
obtaining a knowledge graph by clustering each UI element based on the identified similarity; and
Storing the acquired knowledge graph in the database; method comprising. According to claim 1,
Acquiring the database includes:
Determining a textual representation of a non-text UI element from the visual characteristics and relative position of each UI element; According to claim 1,
Acquiring the database includes:
obtaining a knowledge graph by mapping each UI element and predefined information corresponding to the performance of each UI element; and
Storing the acquired knowledge graph in the database; Containing, a control method. According to claim 1,
Acquiring the database includes:
obtaining a knowledge graph by determining a predefined screen sequence transferred through an operable UI element on the screen; and
Storing the acquired knowledge graph in the database; Containing, a control method. According to claim 1,
performing semantic translation on the knowledge graph to obtain natural language variations for at least one of a single-step intent and a multi-step intent;
identifying at least one action and at least one action sequence for at least one of the single-step intent and the multi-step intent using the obtained knowledge graph; and
dynamically generating a natural language model for predicting natural language utterance of the identified UI element by mapping the obtained natural language variant with the identified action and the identified action sequence. 12. The method of claim 11,
The step of performing the semantic translation comprises:
categorizing each UI element of the knowledge graph into at least one of a domain, a verb, a synonym, a slot, a slot type, a slot expressed as a texture, a performance, and a relative position on the screen 140 ; and
Obtaining the natural language variant based on the categorization; including a control method. 12. The method of claim 11,
The step of identifying the action and the action sequence comprises:
Information on the types of UI elements included on the screen 140 and performance corresponding thereto, performance-related verbs, performance on action information, an element sequence graph operable on the screen and corresponding action sequences, predefined actions and determining a predefined table set including an action sequence; and
Including; determining an action routine based on the performance of each UI element;
A control method, wherein a unique identity is assigned to the predefined actions and action sequences. 12. The method of claim 11,
The step of dynamically generating the natural language model comprises:
clustering the similar natural language variants;
assigning dynamic intent to the similar natural language variants;
associating the at least one identified action and the at least one identified sequence of actions with the dynamic intent;
dynamically generating the natural language model based on the clustered natural language variant, the dynamic intent, and the action routines; and
Storing the dynamically generated natural language model in the database; including a control method. According to claim 1,
The step of identifying whether the natural language utterances match,
determining a text expression score by reading screen information included on the screen (140);
extracting synonyms for verbs and nouns from text expressions and text UI elements included in the screen information and assigning a synonym score;
determining a variance score correlating the natural language variants used for weighted learning for a dynamic language generator (170E);
determining a relevance score by comparing reference objects included in the utterance of the received speech input and proximity element information; and
determining a match score for the dynamic language generator (170E) as a final score that matches an utterance of the received speech input, and combining the determined match score with the relevance score. According to claim 1,
The at least one characteristic is
A control method comprising at least one of a relative position of each UI element, a function of each UI element, a performance of each UI element, a type of each element, and a shape of each UI element. According to claim 1,
The database is
A control method that is generated in real time based on the current screen 140 of the electronic device 100 . In the electronic device 100,
memory 110; and
and a processor 120 connected to the memory 110 to control the electronic device 100;
The processor 120,
Identifies at least one UI (User Interface) element included in the screen 140 of the electronic device 100,
identify at least one characteristic of the identified UI element;
acquiring a database including natural language utterances obtained based on the at least one characteristic of the identified UI element;
When a voice input is received, it is identified whether the utterance of the received voice input matches the natural language utterance included in the acquired database,
automatically accessing the at least one UI element when the utterance of the voice input is identified as matching the natural language utterance;
At least one characteristic of the identified UI element is:
The position of each UI element, the relative position of each UI element with respect to other UI elements, the function of each UI element, the capability of each UI element, the type of each UI element, or the appearance of each UI element ) comprising at least one of, an electronic device. 19. The method of claim 18,
The processor is
First comparing the utterance of the voice input with natural language utterances corresponding to text UI elements included in the database,
comparing the utterance of the voice input with a natural language utterance corresponding to a synonym of a text UI element included in the database according to the first comparison result;
a third comparison of the speech input utterance with a natural language utterance corresponding to a shape of a non-text UI element included in the database according to the second comparison result;
Obtaining the learning result of the neural network model stored in the database,
The control method of identifying whether the match is based on at least one of the first comparison result, the second comparison result, or the third comparison result and the learning result. 19. The method of claim 18,
The processor is
obtaining a matching score for at least one UI element included in the screen;
When the matching score of each of the plurality of UI elements is equal to or greater than a first threshold value, and the difference between the matching scores of each of the plurality of UI elements is within a second threshold value, a guide UI for selecting any one of the plurality of UI elements is provided, and ,
When the matching score of one UI element is equal to or greater than the first threshold value and greater than the second threshold value than the other UI elements, executing the one UI element or requesting a user's execution confirmation for the one UI element An electronic device that provides a guide UI.

Images