KR20220077455A - Electronic apparatus and control method thereof - Google Patents

Abstract

An electronic device is disclosed. The electronic device responds to at least one UI element related to the UI screen by inputting information and environment information about a memory, a display, and a user interface (UI) screen in which information on the learned first neural network model is stored to the first neural network model and a processor for controlling a display to obtain usage probability information to be used, and to provide guide information related to a UI screen based on the obtained usage probability information, wherein the first neural network model includes: UI screen information, a user action on the UI screen It may be learned to output the usage probability information corresponding to the UI element based on the information and the environment information.

Description

Electronic apparatus and its control method { Electronic apparatus and control method thereof }

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device providing an application UI and a control method thereof.

BACKGROUND ART With the development of electronic technology, various types of electronic devices are being developed and distributed. In particular, a user terminal such as a smart phone has been continuously developed in recent years to satisfy user needs.

Smartphones provide various functions through downloadable applications. These applications are provided in the form of an icon interface on the screen of the smart phone and are provided in a form that can change the display position through a user input.

SUMMARY OF THE INVENTION The present disclosure has been made in accordance with the above-mentioned necessity, and an object of the present disclosure is to provide an electronic device that provides an adaptive UI screen according to a user's usage pattern, and a method for controlling the same.

In order to achieve the above object, an electronic device according to an embodiment of the present disclosure stores information on a memory, a display, and a user interface (UI) screen in which information on a learned first neural network model is stored, and environment information, as described above. Control the display to obtain usage probability information corresponding to at least one UI element related to the UI screen by input to the first neural network model, and to provide guide information related to the UI screen based on the obtained usage probability information , wherein the first neural network model may be trained to output usage probability information corresponding to a UI element based on UI screen information, user action information on the UI screen, and environment information.

In addition, the processor acquires a first UI screen in which at least one UI element included in the UI screen is changed and a second UI screen different from the first UI screen, and obtains information on the first UI screen, the second UI screen 2 inputting information on the UI screen and the environment information into the first neural network model to obtain first usage probability information corresponding to the first UI screen and second usage probability information corresponding to the second UI screen, The display may be controlled to provide guide information related to the UI screen based on the obtained first and second usage probability information.

In addition, the first UI screen is a UI screen in which at least one of an arrangement position, type, color, and size of at least one UI element included in the UI screen is changed, and the second UI screen is the first UI screen At least one of an arrangement position, a type, a color, and a size of at least one UI element included in the UI screen may be a different UI screen.

In addition, the memory further stores information on the learned second neural network model, and the processor includes: information on the first UI screen, the first usage probability information, information on the second UI screen, and the The display may be controlled to obtain recommendation information related to a UI element by inputting second usage probability information into the second neural network model, and to provide guide information related to the UI screen based on the obtained recommendation information.

Also, the first neural network model may be a deep learning model, and the second neural network model may be a reinforcement learning model.

In addition, the second neural network model may be learned in a direction to maximize prediction of a change in the usage probability of the UI element by changing the properties of the UI element based on the UI screen and environment information.

In addition, the processor is configured to obtain the first UI screen and the second UI screen by inputting the UI screen and related information into a third neural network model, or by changing at least one UI element included in the UI screen. A first UI screen and the second UI screen may be acquired.

Here, the guide information related to the UI screen may include guide information related to a change of at least one of an arrangement position of a UI element, a type of a UI element, a color of a UI element, and a size of the UI element.

In addition, the obtained usage probability information may include a UI element and probability information corresponding to a user input type for the UI element.

In addition, the environment information includes at least one of weather information, time information, location information, and sensor information, and the user action information includes at least one of a user action type, a user action period, and information on a UI element corresponding to the user action. one, and the information on the UI element may include at least one of a UI element type, a UI element color, and a UI element size.

On the other hand, in the control method of the electronic device for storing information on the learned first neural network model according to an embodiment of the present disclosure, information on a user interface (UI) screen and environment information are input to the first neural network model. to obtain usage probability information corresponding to at least one UI element related to the UI screen, and providing guide information related to the UI screen based on the obtained usage probability information. Here, the first neural network model may be trained to output usage probability information corresponding to a UI element based on UI screen information, user action information on the UI screen, and environment information.

In addition, the method further includes obtaining a first UI screen in which at least one UI element included in the UI screen is changed and a second UI screen different from the first UI screen, wherein the obtaining of the usage probability information includes: The information on the first UI screen, the information on the second UI screen, and the environment information are input to the first neural network model, and the first usage probability information corresponding to the first UI screen and the second UI screen are displayed. Acquiring corresponding second usage probability information and providing the guide information may include providing guide information related to the UI screen based on the obtained first and second usage probability information.

In addition, the first UI screen is a UI screen in which at least one of an arrangement position, type, color, and size of at least one UI element included in the UI screen is changed, and the second UI screen is the first UI screen At least one of an arrangement position, a type, a color, and a size of at least one UI element included in the UI screen may be a different UI screen.

Also, the electronic device further stores information on a second neural network model, and includes information on the first UI screen, the first usage probability information, information on the second UI screen, and the second usage probability information. to the second neural network model to obtain recommendation information related to a UI element, wherein the providing of the guide information includes providing guide information related to the UI screen based on the obtained recommendation information. can do.

Here, the first neural network model may be a deep learning model, and the second neural network model may be a reinforcement learning model.

In addition, the second neural network model may be learned in a direction to maximize prediction of a change in the usage probability of the UI element by changing the properties of the UI element based on the UI screen and environment information.

In addition, the step of obtaining the first UI screen and the second UI screen may include inputting the UI screen and related information into a third neural network model to obtain the first UI screen and the second UI screen, or the UI The first UI screen and the second UI screen may be acquired by changing at least one UI element included in the screen.

Here, the guide information related to the UI screen may include guide information related to a change of at least one of an arrangement position of a UI element, a type of a UI element, a color of a UI element, and a size of the UI element.

In addition, the obtained usage probability information may include a UI element and probability information corresponding to a user input type for the UI element.

In addition, the environment information includes at least one of weather information, time information, location information, and sensor information, and the user action information includes at least one of a user action type, a user action period, or information on a UI element corresponding to the user action. one, and the information on the UI element may include at least one of a UI element type, a UI element color, and a UI element size.

According to the above-described various embodiments, it is possible to provide an adaptive UI screen according to a user's usage pattern, so that the user's convenience is improved.

1 is a view for explaining a method of providing a UI of an electronic device according to an embodiment of the present disclosure.
2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.
3 is a diagram for explaining a method of learning a first neural network model according to an embodiment.
4 is a diagram for describing information obtained through a learned first neural network model according to an embodiment.
5A and 5B are diagrams for explaining information obtained through a learned second neural network model according to an embodiment.
6 is a diagram for explaining an example of UI guide information according to an embodiment.
7 is a diagram for sequentially explaining a method of providing guide information using a first neural network model and a second neural network model according to an embodiment.
8 is a diagram illustrating an implementation example of an electronic device according to an embodiment of the present disclosure.
9 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

Terms used in the embodiments of the present disclosure are selected as currently widely used general terms as possible while considering the functions in the present disclosure, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. . In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

The expression "at least one of A and/or B" is to be understood as indicating either "A" or "B" or "A and B".

As used herein, expressions such as "first," "second," "first," or "second," can modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component); When referring to "connected to", it should be understood that an element may be directly connected to another element or may be connected through another element (eg, a third element).

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and are intended to indicate that one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In the present disclosure, a “module” or “unit” performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of “modules” or a plurality of “units” are integrated into at least one module and implemented with at least one processor (not shown) except for “modules” or “units” that need to be implemented with specific hardware. can be

Hereinafter, an embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings.

1 is a view for explaining a method of providing a UI of an electronic device according to an embodiment of the present disclosure.

According to an embodiment, the electronic device 100 includes a smart phone, a tablet PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, an e-book terminal, a digital broadcasting terminal, a navigation device, an MP3 player, and other mobile devices. or a non-mobile computing device, but is not limited thereto. As another example, the electronic device 100 may be a home appliance having a display, such as a refrigerator. Also, the electronic device 100 may be a wearable terminal such as a watch, glasses, a hair band, and a ring having a communication function and a data processing function. Also, since the electronic device 100 has a built-in touch screen, it may be implemented to execute a program using a finger or a pen (eg, a stylus pen).

The electronic device 100 may download and install an application from a server (not shown) that provides the application. In this case, the user may execute an application on the electronic device 100 and input a user account to log in to the server (not shown) through the input user account, and the electronic device 100 may be based on the logged-in user account. to communicate with a server (not shown). Here, the application may be a type of software directly used by the user on the OS.

Meanwhile, an application previously installed in the electronic device 100 or downloaded from a server (not shown) may be provided on the screen in the form of an application UI (hereinafter, App UI), for example, an icon interface. In the case of such App UI, it is common for the user to directly set the location and size according to the user's intention.

Hereinafter, various embodiments of recommending various settings of the App UI based on the history of use of the App UI according to various embodiments of the present disclosure will be described.

2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

The electronic device 100 according to FIG. 2 includes a memory 110 , a display 120 , and a processor 130 .

The memory 110 may store data necessary for various embodiments of the present disclosure. The memory 110 may be implemented in the form of a memory embedded in the electronic device 100 or may be implemented in the form of a memory capable of communicating with the electronic device 100 (or detachable) depending on the purpose of data storage. For example, data for driving the electronic device 100 is stored in a memory embedded in the electronic device 100 , and data for an extended function of the electronic device 100 can communicate with the electronic device 100 . can be stored in memory. Meanwhile, in the case of a memory embedded in the electronic device 100 , a volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory ( Examples: one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (such as NAND flash or NOR flash, etc.) ), a hard drive, or a solid state drive (SSD), etc. In addition, in the case of a memory capable of communicating with the electronic device 100 , a memory card (eg, a compact flash (CF)) , SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), external memory that can be connected to the USB port (e.g. For example, it may be implemented in a form such as a USB memory).

According to an example, the memory 110 may store at least one instruction for controlling the electronic device 100 or a computer program including the instructions.

According to another example, the memory 110 may store information about a neural network model including a plurality of layers. Here, storing information about the neural network model means various information related to the operation of the neural network model, for example, information on a plurality of layers included in the neural network model, parameters used in each of the plurality of layers (for example, filter coefficients) , bias, etc.) may be stored. For example, the memory 110 may store information about the first neural network model learned to predict and output usage probability information (or usage probability information) corresponding to a UI element according to an embodiment. In addition, the memory 110 may store information about the second neural network model learned to predict and obtain recommendation information related to UI elements. However, when the processor 130 is implemented as hardware dedicated to the neural network model, information about the neural network model may be stored in the processor 130 internal memory.

According to an embodiment, the memory 110 may be implemented as a single memory that stores data generated in various operations according to the present disclosure. However, according to another embodiment, the memory 110 may be implemented to include a plurality of memories each storing different types of data or each storing data generated in different steps.

The display 120 may be implemented as a display including a self-luminous device or a display including a non-light-emitting device and a backlight. For example, LCD (Liquid Crystal Display), OLED (Organic Light Emitting Diodes) display, LED (Light Emitting Diodes), micro LED (micro LED), Mini LED, PDP (Plasma Display Panel), QD (Quantum dot) display , Quantum dot light-emitting diodes (QLEDs) may be implemented in various types of displays. The display 230 may include a driving circuit, a backlight unit, and the like, which may be implemented in the form of an a-si TFT, a low temperature poly silicon (LTPS) TFT, or an organic TFT (OTFT). On the other hand, the display 120 is implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a three-dimensional display (3D display), a display in which a plurality of display modules are physically connected, etc. can be

The processor 130 is electrically connected to the memory 110 and the display 120 to control the overall operation of the electronic device 100 . The processor 130 may include one or a plurality of processors. Specifically, the processor 130 may perform the operation of the electronic device 100 according to various embodiments of the present disclosure by executing at least one instruction stored in the memory 110 .

According to an embodiment, the processor 130 may include a digital signal processor (DSP), a microprocessor (microprocessor), a graphics processing unit (GPU), an artificial intelligence (AI) processor, a neural network (NPU) for processing a digital image signal. Processing Unit), TCON (Time controller), but is not limited thereto, Central processing unit (CPU), MCU (Micro Controller Unit), MPU (micro processing unit), controller (controller), application processor (application processor (AP)), communication processor (communication processor (CP)), may include one or more of an ARM processor, or may be defined by the term. In addition, the processor 130 may be implemented as a system on chip (SoC) or large scale integration (LSI) with a built-in processing algorithm, or as an application specific integrated circuit (ASIC) or field programmable gate array (FPGA) type.

In addition, the processor 130 for executing the neural network model according to an embodiment is a general-purpose processor such as a CPU, an AP, a digital signal processor (DSP), etc., a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an NPU. It can be implemented through a combination of artificial intelligence dedicated processor and software. The processor 130 may control to process input data according to a predefined operation rule or a neural network model stored in the memory 110 . Alternatively, when the processor 130 is a dedicated processor (or artificial intelligence-only processor), it may be designed with a hardware structure specialized for processing a specific neural network model. For example, hardware specialized for processing a specific neural network model may be designed as a hardware chip such as an ASIC or FPGA. When the processor 130 is implemented as a dedicated processor, it may be implemented to include a memory for implementing an embodiment of the present disclosure, or may be implemented to include a memory processing function for using an external memory.

According to an embodiment, the processor 130 may obtain usage probability information corresponding to at least one UI element related to the UI screen by inputting information and environment information about the UI screen into the first neural network model. have. Here, the usage probability information may include a UI element and prediction probability information corresponding to a user action type (or user input type) for the UI element. Here, the UI element may be the aforementioned App UI, but is not limited thereto, and various UI elements selectable by the user on the UI screen provided by the electronic device 100, for example, a widget UI, a sub provided by the App UI Various OS UIs such as menu UI and notification UI may be included. In addition, the user action type includes not only various touch input types such as touch, swipe, drag, flick, etc., but also direction information (eg, left swip) in the case of a touch input type having direction information. It may also include.

Subsequently, the processor 130 may control the display 120 to provide guide information related to the UI screen based on the obtained usage probability information. Here, the guide information related to the UI screen may include guide information related to a change of at least one of an arrangement position of a UI element, a type of a UI element, a color of a UI element, and a size of the UI element. For example, the guide information may include information for recommending at least one of a location change, a display type change, a color change, or a size change of a specific app UI in a specific time period.

According to an example, the UI screen may be various types of UI screens provided through the display 120 of the electronic device 100 . Here, the information on the UI screen may include information on at least one of an arrangement position of a UI element included in the UI screen, a type of a UI element, a color of the UI element, and a size of the UI element. Here, the arrangement position of the UI element may include, for example, x and y coordinates on the screen, and the type of the UI element may include an application name corresponding to the UI element, an application category, and the like. The processor 130 may input a UI screen image to the first neural network model or input information related to the UI screen to the first neural network model.

According to an example, the environment information may include at least one of weather information, time information, location information, and sensor information. However, weather information, time information, location information, or sensor information is not necessarily required for learning, and at least a portion may be omitted and other information may be added as necessary. Here, the time information may include various information related to time, such as time, day of the week, holidays, etc., and the weather information may include various information related to weather such as temperature, dust, ozone index, precipitation, wind, and humidity, and a sensor The information may include various information that can be collected by a sensor, such as brightness information, proximity information, and location information. According to another example, device information corresponding to the electronic device 100 may be used for learning. The device information may include various information such as a function, a model, a type, a manufacturer, and a location.

Here, the first neural network model may be a model trained to predict and output usage probability information corresponding to a UI element based on the UI screen, user action information on the UI screen, and environment information. Learning may be performed using the UI screen and environment information as input data, and user action information on the UI screen as output data. The user action information for the UI screen may include at least one of a user action type (action type (touch, scroll, etc.)) for the UI screen, a user action duration, or information on a UI element corresponding to the user action. have. Here, the information of the UI element may include at least one of a type of a UI element, a color of the UI element, and a size of the UI element.

According to an embodiment, the processor 130 obtains a first UI screen in which a feature of at least one UI element included in the UI screen is changed and a second UI screen different from the first UI screen, and the first UI Information on the screen, information on the second UI screen, and environment information are input to the first neural network model to obtain first usage probability information corresponding to the first UI screen and second usage probability information corresponding to the second UI screen can do. Subsequently, the processor 130 may provide guide information related to the UI screen based on the obtained first and second usage probability information. Here, the first UI screen may be a UI screen in which a property of at least one UI element included in the current UI screen is changed. Also, the second UI screen may be a UI screen having different properties of at least one UI element included in the first UI screen. Here, the properties of the UI element may include at least one of an arrangement position, type, color, and size of the UI element. Also, the types of UI elements may include various display shapes such as buttons, texts, and icons.

According to an embodiment, the processor 130 inputs the information on the first UI screen, the first usage probability information, the information on the second UI screen, and the second usage probability information into the second neural network model to be related to the UI element. Recommendation information may be acquired, and guide information related to the UI screen may be provided based on the acquired recommendation information.

According to an example, the processor 130 acquires the first UI screen and the second UI screen by changing at least one UI element included in the UI screen based on the number of properties of the UI element, or the processor 130 is displayed on the UI screen. The first UI screen and the second UI screen may be obtained by randomly changing at least one included UI element. According to another example, the processor 130 may acquire the first UI screen and the second UI screen based on the recommendation information related to the UI element output from the second neural network model. For example, based on a change value capable of increasing or decreasing a user use probability based on a property of a UI element changeable on the UI screen, a first UI screen and a second UI screen in which the properties of the UI element are changed are obtained can do.

Meanwhile, the processor 130 may input UI screen information and environment information into the first neural network model (and/or the second neural network model) according to various events. As an example, the periodically collected environment information and UI screen may be input to the first neural network model (and/or the second neural network model. As another example, when user input to a specific UI screen is periodically repeated, the Environment information and UI screen may be input to the first neural network model As another example, UI screen information and environmental information may be input to the first neural network model (and/or the second neural network model) according to a specific user input .

In addition, the time point at which the guide information is provided may be the same as or within a critical time as the time point at which information is obtained from the first neural network model (and/or the second neural network model), but may be different. According to an example, in the latter case, the processor 130 may provide guide information periodically or according to a user command or at a specific time. Here, the specific time point may be before a critical time point from a time point when a change of the UI element is recommended.

However, according to another embodiment, it may be possible for the processor 130 to automatically change and provide the UI screen based on output information of the first neural network model (and/or the second neural network model) without providing guide information. .

Meanwhile, the above-described first neural network model and second neural network model may be stored in the memory 110 , but may also be received from an external device (eg, an external server). In addition, the first neural network model and the second neural network model may be updated by data received from an external device.

Meanwhile, each of the above-described first neural network model and second neural network model may include a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between the operation result of a previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the neural network model. For example, a plurality of weights may be updated so that a loss value or a cost value obtained from the neural network model during the learning process is reduced or minimized. The artificial neural network may include a deep neural network (DNN), for example, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Reinforcement Learning Network (RLN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, but is not limited to the above example. According to an example, the first neural network model may be implemented as a deep learning model, and the second neural network model may be implemented as a reinforcement learning model.

Meanwhile, the output part of the first neural network model may be implemented to enable softmax processing. Here, softmax is a function that normalizes all input values to values between 0 and 1 and makes the sum of output values always equal to 1, and it is possible to output probability values for each class. In addition, output parts of the first neural network model and the second neural network model may be implemented to enable Argmax processing. Argmax is a function that selects the most probable one among multiple labels. Here, the probability value for each class can be output by selecting the ratio having the largest value among the probability values. That is, when the output part of each of the first neural network model and the second neural network model has been subjected to Argmax processing, only one category information having the highest probability value can be output.

According to an embodiment, when the user's usage information related to the UI element is obtained, the processor 130 obtains usage pattern information based on the obtained usage information, and inputs the obtained usage pattern information to the first neural network model. may be For example, the processor 130 may acquire usage pattern information using a K-means algorithm. Here, the K-means algorithm refers to an unsupervised learning algorithm that collects and categorizes similar data. Here, unsuperised learning is an algorithm that is applied when there is only an input and no output, and can find regularities between the inputs. According to an example, 1. classify data close to the central value, 2. change the central value to the central value of the classified data, 3. the central value Steps 1 and 2 can be repeated until this no longer changes.

According to an embodiment, the processor 130 may obtain usage information of a recent critical time period, and may acquire a recent usage pattern based on the obtained usage information. For example, the processor 130 may acquire usage information of the last two weeks section on a daily basis, and acquire a recent usage pattern based on the acquired usage information. That is, as of today, the latest usage pattern is obtained based on the usage information of the last two weeks, and when tomorrow comes, the latest usage pattern is obtained based on the usage information of the last two weeks as of tomorrow, and the latest You can update usage patterns. However, for the convenience of explanation, the daily unit and the two-week period are given as examples, and are not limited thereto, of course.

3 is a diagram for explaining a method of learning a first neural network model according to an embodiment.

According to an embodiment, the first neural network model may be trained based on a pair of input training data and output training data, or may be trained based on input training data. Here, the learning of the neural network model means that the basic neural network model (for example, a neural network model including arbitrary random parameters) is learned using a plurality of training data by a learning algorithm, thereby obtaining a desired characteristic (or purpose). It means that a predefined action rule or neural network model set to be executed is created. Such learning may be performed through the electronic device 100, but is not limited thereto, and may be performed through a separate server and/or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. However, this is an example of supervised learning, and it goes without saying that the neural network model can be trained based on unsupervised learning in which the neural network model is trained by inputting only input data without using the recommended administrative information as output data.

For example, as shown in FIG. 3 , the first neural network model 310 inputs and outputs a UI screen 321 , environment information (or environment data) 322 , and user action information (or user input data) 323 , respectively. It can be learned using training data pairs. For example, as shown in FIG. 3, input data including the UI screen 321 and environment information 322, and output data of the user action information 323 may be used as input/output training data pairs. . However, this is an example of supervised learning, and it is possible to train a neural network model based on unsupervised learning by inputting only input data without using the user action information 323 as output data. Of course. Here, the user action information may include at least one of a user action type (action type (touch, scroll, etc.)) for the UI screen, a user action duration, or information on a UI element corresponding to the user action. . Here, the information of the UI element may include at least one of a type of a UI element, a color of the UI element, and a size of the UI element. However, in FIG. 3 , it is illustrated that the user action information includes a user action type and a UI element type (eg, an application name) for convenience of explanation.

4A and 4B are diagrams for explaining information obtained through a learned first neural network model according to an embodiment.

According to an embodiment, as shown in FIG. 4A , the processor 130 applies the UI screen 421 (or screen-related information) and the current environment information 422 of the current electronic device 100 to the learned first neural network model ( 310) to obtain usage probability information 430 for at least one UI element. Here, the usage probability information may include at least one UI element and probability information corresponding to a user input type (or/and input coordinates) for the UI element. For example, as shown in FIG. 4B , the first neural network model 310 may output probability information corresponding to the user input type (or/and input coordinates) based on the characteristics of the UI screen and the characteristics of the environment information. have. For example, the output part of the first neural network model 310 is implemented to enable softmax processing to output probability values for each class (eg, user input type (or/and input coordinates) corresponding to UI elements) can do. However, when the output part of the first neural network model 310 is Argmax-processed, the probability value for each class may be output by selecting the ratio having the largest value among the probability values. Here, each class is illustrated as including the name of the UI element and the user action type, but it may be implemented in other forms that can identify the UI element, such as coordinates of the UI element and the user action type.

5A and 5B are diagrams for explaining information obtained through a learned second neural network model according to an embodiment.

According to an embodiment, the processor 130 may display a plurality of UI screens 521 , 522 , 523 , 524 changed from the current UI screen (eg, 421 of FIG. 4 ) of the electronic device 100 (or screen-related information). ) into the learned second neural network model 510 to obtain UI element change recommendation information. Here, the UI element change recommendation information may include a UI screen with the highest use probability among the plurality of UI screens 521 , 522 , 523 , and 524 or information related to a UI element included in the UI screen. For example, as shown in FIG. 5A , change information of a specific UI screen 521 or a UI element related to the UI screen may be acquired as recommendation information.

According to an example, the second neural network model may be implemented as a reinforcement learning model. Reinforcement learning is a method in which an algorithm finds a combination that can produce an optimal result through various trials. , is an algorithm that learns in the direction of maximizing rewards through actions. The agent learns in such a way that it becomes more frequent to act that increases the reward and less often to act that reduces the reward. For example, an agent called the UI usage probability prediction model (the first neural network model described above) recognizes the current state (UI screen and environment information) in the environment and changes the property x of the UI element by a. Through an action, it is possible to learn in the direction of maximizing the prediction (reward) of the change in the performance probability of the UI element.

6 is a diagram for explaining an example of UI guide information according to an embodiment.

According to an embodiment, the processor 130 may provide guide information related to the UI screen based on information output from the first neural network model or information output from the second neural network model.

As an example, the processor 130 is information for guiding a change in the arrangement position of UI elements based on information obtained by inputting specific environment information (eg, time information: 9:00 pm) and UI screen information into the first neural network model. 610 may be provided.

As another example, the processor 130 is information for guiding a change in the arrangement position of UI elements based on information obtained by inputting specific environment information (eg, time information: 6:00 am) and UI screen information to the first neural network model 620 may be provided.

According to an example, the processor 130 may provide guide information according to a preset event. For example, guide information may be provided periodically or according to a user command or at a specific time. Here, the specific time point may be before a critical time point from a time point when a change of the UI element is recommended.

7 is a diagram for sequentially explaining a method of providing guide information using a first neural network model and a second neural network model according to an embodiment.

According to an embodiment, the processor 130 acquires a plurality of UI screens 721 to 724 by changing a property of at least one UI element included in the UI screen 710 , and obtains the obtained UI screens 721 to 721 . 724 ) and environment information may be input to the first neural network model 310 to obtain a plurality of usage probability information 731 to 734 related to UI elements included in each UI screens 721 to 724 . Subsequently, the processor 130 may obtain UI change recommendation information by inputting the obtained plurality of usage probability information 731 to 734 into the second neural network model 510 . For example, the UI change recommendation information may be a UI screen having a UI property with a high probability of use. Here, the second neural network model 510 may continuously learn property change information and user usage probability information of UI elements included in the UI screens 721 to 724 . For example, a frequently used case may be learned to have a relatively high probability, and a less frequently used case may be trained to have a low probability.

Since this has been described with reference to FIGS. 5A and 5B , a detailed description thereof will be omitted.

Subsequently, the processor 130 may provide information 740 for guiding a change of a UI element based on the obtained UI change recommendation information.

8 is a diagram illustrating an implementation example of an electronic device according to an embodiment of the present disclosure.

According to FIG. 8 , the electronic device 100 ′ includes a memory 110 , a display 120 , a processor 130 , a speaker 140 , a communication interface 150 , a user interface 160 , an input/output interface 170 and A position sensor 180 is included. A detailed description of the configuration shown in FIG. 8 overlapping the configuration shown in FIG. 2 will be omitted.

The memory 110 stores various modules for driving the electronic device 100 . For example, the memory 110 may store software including a base module, a sensing module, a communication module, a presentation module, a web browser module, and a service module. In this case, the base module is a basic module that processes signals transmitted from each hardware included in the electronic device 100 and transmits them to a higher layer module. The sensing module is a module for collecting information from various sensors and analyzing and managing the collected information, and may include a face recognition module, a voice recognition module, a motion recognition module, an NFC recognition module, and the like. The presentation module is a module for configuring a display screen, and may include a multimedia module for playing and outputting multimedia content, and a UI rendering module for performing UI and graphic processing. The communication module is a module for performing communication with the outside. The web browser module refers to a module that accesses a web server by performing web browsing. The service module is a module including various applications for providing various services.

Meanwhile, the processor 130 may perform a graphic processing function (a video processing function). For example, the processor 130 may generate a screen including various objects such as icons, images, and texts using a calculator (not shown) and a rendering unit (not shown). Here, the calculating unit (not shown) may calculate attribute values such as coordinate values, shape, size, color, etc. of each object to be displayed according to the layout of the screen based on the received control command. In addition, the rendering unit (not shown) may generate screens of various layouts including objects based on the attribute values calculated by the calculation unit (not shown).

Meanwhile, the processor 130 may process audio data. Specifically, the processor 130 may perform various processes such as decoding, amplification, and noise filtering on audio data.

The speaker 140 may be a component that outputs various types of audio data processed by the input/output interface 170 as well as various notification sounds or voice messages. The processor 130 may control the speaker 140 to output information corresponding to the UI screen or various notifications in the form of audio according to various embodiments of the present disclosure. For example, the processor 130 may control the speaker 140 to output at least one of a selected recipe or a modified recipe in a voice form.

The communication interface 150 may communicate with a network device (not shown).

According to an embodiment, the communication interface 150 may include a wireless communication module, for example, a Wi-Fi module, a Bluetooth module, and the like. However, the present invention is not limited thereto, and the communication interface 150 includes Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), LTE-A (LTE) in addition to the above-described communication methods. Advanced), 4G (4th Generation), 5G (5th Generation), etc., various wireless communication standards, infrared communication (IrDA, Infrared Data Association) technology, etc., may perform communication.

The user interface 160 may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or may be implemented as a touch screen capable of performing the above-described display function and manipulation input function together. Here, the button may be various types of buttons such as a mechanical button, a touch pad, a wheel, etc. formed in an arbitrary area such as the front, side, or rear of the exterior of the main body of the electronic device 100 .

Input/output interface 170 is HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt (Thunderbolt), VGA (Video Graphics Array) port, The interface may be any one of an RGB port, a D-subminiature (D-SUB), and a digital visual interface (DVI). The input/output interface 170 may input/output at least one of audio and video signals.

The position sensor 180 may sense the position of the electronic device 100 . The location sensor 180 may be implemented as a global positioning system (GPS) receiver, but is not limited as long as location information of the electronic device 100 can be obtained.

Meanwhile, the electronic device 100 ′ may further include a microphone (not shown). A microphone is a component for receiving a user's voice or other sound and converting it into audio data. For example, a user command input on the UI screen according to various embodiments of the present disclosure may be input in the form of a voice command.

9 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

According to the control method of the electronic device for storing information on the learned first neural network model illustrated in FIG. 9 , information on a user interface (UI) screen and environment information are input to the first neural network model to at least relate to the UI screen. Use probability information corresponding to one UI element is acquired (S910). Here, the first neural network model may be a model trained to output usage probability information corresponding to a UI element based on UI screen information, user action information on the UI screen, and environment information.

Next, guide information related to the UI screen is provided based on the obtained usage probability information (S920).

In addition, the control method may acquire a first UI screen in which at least one UI element included in the UI screen is changed and a second UI screen different from the first UI screen. In this case, in step S910, information on the first UI screen, information on the second UI screen, and environment information are input to the first neural network model, and the first usage probability information corresponding to the first UI screen and the second UI screen are displayed. Corresponding second usage probability information may be acquired. Also, in operation S920, guide information related to the UI screen may be provided based on the obtained first and second usage probability information.

Here, the first UI screen is a UI screen in which at least one of the arrangement position, type, color, and size of at least one UI element included in the UI screen is changed, and the second UI screen is at least one included in the first UI screen. At least one of the arrangement position, type, color, and size of one UI element may be a different UI screen.

In addition, the electronic device further stores information on the second neural network model, and the control method includes information on the first UI screen, first usage probability information, information on the second UI screen, and second usage probability information. The method may further include inputting the second neural network model to obtain recommendation information related to the UI element. In step S920, guide information related to the UI screen may be provided based on the obtained recommendation information.

In this case, the first neural network model may be a deep learning model, and the second neural network model may be a reinforcement learning model. Here, the second neural network model may be learned in a direction to maximize prediction of a change in the usage probability of the UI element by changing the properties of the UI element based on the UI screen and environment information.

In addition, the step of obtaining the first UI screen and the second UI screen may include inputting the UI screen and related information into the third neural network model to obtain the first UI screen and the second UI screen, or at least one included in the UI screen. The first UI screen and the second UI screen may be acquired by changing the UI element of .

Also, the guide information related to the UI screen may include guide information related to a change of at least one of an arrangement position of a UI element, a type of a UI element, a color of a UI element, and a size of a UI element. Here, the obtained usage probability information may include a UI element and probability information corresponding to a user input type for the UI element.

In this case, the environment information includes at least one of weather information, time information, location information, and sensor information, and the user action information includes at least one of a user action type, a user action period, or information on a UI element corresponding to the user action. may include Here, the information of the UI element may include at least one of a type of a UI element, a color of the UI element, and a size of the UI element.

According to the above-described various embodiments, it is possible to provide an adaptive UI screen according to a user's usage pattern, so that the user's convenience is improved.

Meanwhile, the above-described methods according to various embodiments of the present disclosure may be implemented in the form of an application that can be installed in an existing electronic device.

In addition, the above-described methods according to various embodiments of the present disclosure may be implemented only by software upgrade or hardware upgrade of an existing electronic device.

In addition, various embodiments of the present disclosure described above may be performed through an embedded server provided in the electronic device or at least one external server of the electronic device.

Meanwhile, according to an embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage media readable by a machine (eg, a computer). A device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, the electronic device A) according to the disclosed embodiments. When executed by the processor, the processor may perform a function corresponding to the instruction by using other components directly or under the control of the processor The instruction may include code generated or executed by a compiler or an interpreter. A device-readable storage medium may be provided in the form of a non-transitory storage medium, where the 'non-transitory storage medium' is a tangible device and a signal (eg, electromagnetic waves), and this term does not distinguish between a case in which data is stored semi-permanently in a storage medium and a case in which data is temporarily stored. It may contain buffers.

According to an embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store ^TM ) or on two user devices (eg It can be distributed online (eg download or upload), directly between smartphones (eg smartphones). In the case of online distribution, at least a portion of the computer program product (eg, a downloadable app) is stored at least in a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or a relay server's memory. It may be temporarily stored or temporarily created.

In addition, each of the components (eg, a module or a program) according to the above-described various embodiments may be composed of a single or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be omitted. Components may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallelly, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added. can

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is generally used in the technical field belonging to the present disclosure without departing from the gist of the present disclosure as claimed in the claims. Various modifications may be made by those having the knowledge of

100: electronic device 110: memory
120: display 130: processor

Claims (20)

a memory in which information about the learned first neural network model is stored;
display; and
Obtaining usage probability information corresponding to at least one UI element related to the UI screen by inputting information and environment information about the UI (User Interface) screen into the first neural network model,
A processor for controlling the display to provide guide information related to the UI screen based on the obtained usage probability information;
The first neural network model is
An electronic device learned to output usage probability information corresponding to a UI element based on UI screen information, user action information on the UI screen, and environment information. According to claim 1,
The processor is
Obtaining a first UI screen in which at least one UI element included in the UI screen is changed and a second UI screen different from the first UI screen,
Information on the first UI screen, information on the second UI screen, and the environment information are input to the first neural network model, and the first usage probability information corresponding to the first UI screen and the second UI screen are displayed. Obtaining corresponding second usage probability information,
and controlling the display to provide guide information related to the UI screen based on the obtained first and second usage probability information. 3. The method of claim 2,
The first UI screen,
a UI screen in which at least one of the arrangement position, type, color, and size of at least one UI element included in the UI screen is changed;
The second UI screen,
The electronic device of claim 1, wherein at least one of an arrangement position, type, color, and size of at least one UI element included in the first UI screen is a different UI screen. 3. The method of claim 2,
The memory further stores information about the learned second neural network model,
The processor is
inputting the information on the first UI screen, the first usage probability information, the information on the second UI screen, and the second usage probability information into the second neural network model to obtain recommendation information related to UI elements,
and controlling the display to provide guide information related to the UI screen based on the obtained recommendation information. 5. The method of claim 4,
The first neural network model is a deep learning model,
The second neural network model is a reinforcement learning model, the electronic device. 6. The method of claim 5,
The second neural network model is
An electronic device that is learned in a direction to maximize prediction of a change in usage probability of a UI element by changing a property of the UI element based on the UI screen and environment information. 3. The method of claim 2,
The processor is
acquiring the first UI screen and the second UI screen by inputting the UI screen and related information into a third neural network model;
and acquiring the first UI screen and the second UI screen by changing at least one UI element included in the UI screen. According to claim 1,
Guide information related to the UI screen,
An electronic device comprising guide information related to a change of at least one of a layout position of a UI element, a type of a UI element, a color of the UI element, and a size of the UI element. According to claim 1,
The obtained usage probability information is,
An electronic device comprising a UI element and probability information corresponding to a user input type for the UI element. According to claim 1,
The environment information is
At least one of weather information, time information, location information, or sensor information,
The user action information is
It includes at least one of information of a user action type, a user action period, and a UI element corresponding to the user action,
The information of the UI element,
An electronic device comprising at least one of a type of a UI element, a color of the UI element, and a size of the UI element. In the control method of an electronic device for storing information about the learned first neural network model,
obtaining usage probability information corresponding to at least one UI element related to the UI screen by inputting information on a user interface (UI) screen and environment information into the first neural network model; and
Including; providing guide information related to the UI screen based on the obtained usage probability information;
The first neural network model is
A control method learned to output usage probability information corresponding to a UI element based on UI screen information, user action information on the UI screen, and environment information. 12. The method of claim 11,
Obtaining a first UI screen in which at least one UI element included in the UI screen is changed and a second UI screen different from the first UI screen; further comprising,
The step of obtaining the usage probability information includes:
Information on the first UI screen, information on the second UI screen, and the environment information are input to the first neural network model, and the first usage probability information corresponding to the first UI screen and the second UI screen are displayed. Obtaining corresponding second usage probability information,
The step of providing the guide information includes:
A control method for providing guide information related to the UI screen based on the obtained first and second usage probability information. 13. The method of claim 12,
The first UI screen,
a UI screen in which at least one of the arrangement position, type, color, and size of at least one UI element included in the UI screen is changed;
The second UI screen,
The control method, wherein at least one of the arrangement position, type, color, and size of at least one UI element included in the first UI screen is a different UI screen. 13. The method of claim 12,
The electronic device further stores information about the second neural network model,
obtaining recommendation information related to a UI element by inputting the information on the first UI screen, the first usage probability information, the information on the second UI screen, and the second usage probability information into the second neural network model further including;
The step of providing the guide information includes:
A control method for providing guide information related to the UI screen based on the obtained recommendation information. 15. The method of claim 14,
The first neural network model is a deep learning model,
The second neural network model is a reinforcement learning (Reinforcement learning) model, a control method. 16. The method of claim 15,
The second neural network model is
A control method that is learned in the direction of maximizing change prediction of the use probability of a UI element by changing the properties of the UI element based on the UI screen and environment information. 13. The method of claim 12,
Acquiring the first UI screen and the second UI screen comprises:
acquiring the first UI screen and the second UI screen by inputting the UI screen and related information into a third neural network model;
The control method of acquiring the first UI screen and the second UI screen by changing at least one UI element included in the UI screen. 12. The method of claim 11,
Guide information related to the UI screen,
A control method comprising guide information related to a change of at least one of a layout position of a UI element, a type of a UI element, a color of the UI element, and a size of the UI element. 12. The method of claim 11,
The obtained usage probability information is,
A control method comprising a UI element and probability information corresponding to a user input type for the UI element. 12. The method of claim 11,
The environment information is
At least one of weather information, time information, location information, or sensor information,
The user action information is
It includes at least one of information on a user action type, a user action period, or a UI element corresponding to the user action,
The information of the UI element,
A control method comprising at least one of a type of a UI element, a color of the UI element, and a size of the UI element.

Images