WO2020022723A1

WO2020022723A1 - Electronic device and control method therefor

Info

Publication number: WO2020022723A1
Application number: PCT/KR2019/009035
Authority: WO
Inventors: 최원종; 김수필; 함진아
Original assignee: 삼성전자주식회사
Priority date: 2018-07-20
Filing date: 2019-07-22
Publication date: 2020-01-30

Abstract

An electronic device and a control method therefor are disclosed. A control method for an electronic device, according to the present disclosure, comprises the steps of: acquiring information about an application being executed and storing same in a memory, when a user command is inputted during execution of the application; and inputting a user question related to the application in a learned artificial intelligence model so as to output an answer to the acquired user question, when the user question related to the application is inputted, wherein the answer to the user question can be determined on the basis of the information about the application being executed.

Description

Electronic device and its control method

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method thereof capable of providing an optimized answer to a user question based on a personalized database.

Artificial Intelligence (AI) system is a computer system that implements human-level intelligence, and unlike conventional rule-based smart systems, it is a system that machines learn, judge, and become smart. As the artificial intelligence system is used, the recognition rate is improved and the user taste can be understood more accurately, and the existing rule-based smart system is gradually replaced by the deep learning-based artificial intelligence system.

Artificial intelligence technology consists of elementary technologies that utilize machine learning (deep learning) and machine learning.

Machine learning is an algorithm technology that classifies / learns the characteristics of input data by itself, and element technology is a technology that uses machine learning algorithms such as deep learning, and it includes linguistic understanding, visual understanding, reasoning / prediction, knowledge expression, motion control, etc. It consists of technical fields.

The various fields in which artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing and applying / processing human language / characters, including natural language processing, machine translation, dialogue system, question answering, speech recognition / synthesis, and the like. Visual understanding is a technology that recognizes and processes objects as human vision, and includes object recognition, object tracking, image retrieval, human recognition, scene understanding, spatial understanding, and image enhancement. Inference prediction is a technique of judging information, logically inferring, and predicting information, and includes knowledge / probability based inference, optimization prediction, preference based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data generation / classification) and knowledge management (data utilization). Motion control is a technique for controlling autonomous driving of a vehicle and movement of a robot, and includes motion control (navigation, collision, driving), operation control (action control), and the like.

Artificial intelligence technologies, on the other hand, typically operate on an external server. However, if the data used in artificial intelligence technology is data related to personal information, problems related to security may occur.

The present disclosure has been made to solve the above-described problem, and relates to an electronic device and a control method thereof capable of providing an optimized answer to a user question based on a personalized database.

According to an embodiment of the present disclosure, a method of controlling an electronic device may include: when a user command is input while an application is running, through the type of the application, content information related to the running application, and the application; Acquiring and storing at least one of the used functions in a memory and inputting a user question related to the application to a trained artificial intelligence model and outputting an answer to the obtained user question; And answering the user question may be determined based on at least one of a type of the application, content information related to the running application, and a function used through the application.

The storing may include obtaining and storing at least one of a type of the application, content information related to the running application, and a function used through the application.

In this case, the storing may include extracting and storing the core information of the text when the running application includes the text.

In this case, the storing may include storing history information including information on the performed function and the user interaction when a user interaction for performing a function on the running application is detected. .

The outputting may include outputting at least one of providing a search result for the user question, performing an action on the user question, and performing an application function on the user question.

In this case, the outputting may include recommending an application for playing the searched content based on the information related to the application when the user question is a question related to content search.

At this time, the outputting step, if the user question is input, transmitting the user question to an external server, receiving an answer candidate for the user question from the external server and the received response candidate and the data And outputting an answer to the user question based on the information stored in the base.

The storing may include storing at least one of domain information of the running application, a type of the application, content information related to the running application, and a function used through the application, wherein the artificial intelligence model is stored. May obtain an answer to the user question based on at least one of the stored domain information and the type of the application, content information related to the running application, and a function used through the application.

In this case, the user question may be at least one of user voice or text.

On the other hand, the electronic device according to another embodiment of the present disclosure for achieving the above object, if a user command is input while the application is running, the type of the application, the content information associated with the running application and the application; Acquire and store at least one of the functions used in the memory, and if a user question related to the application is input, input a user question related to the application to a learned artificial intelligence model to answer the obtained user question. Output and answer the above user questions

In this case, the processor may acquire and store at least one of a type of the application, content information related to the running application, and a function used through the application, in the memory.

In this case, when the running application includes text, the processor may extract key information of the text and store the core information in the memory.

In this case, when a user interaction for performing a function on the running application is detected, the processor may store history information including information about the performed function and the user interaction in the memory.

In this case, the processor may output at least one of providing a search result for the user question, performing an action on the user question, and performing an application function on the user question.

In this case, when the user question is a question related to content search, the processor may recommend an application for playing the searched content based on information related to the application.

In this case, the electronic device further includes a communication unit, and when the user question is input, the processor transmits the user question to an external server through the communication unit, and receives a candidate for answering the user question from the external server. Received through the communication unit, based on the received response candidate and the information stored in the memory may output the answer to the user question.

In this case, the processor may store at least one of domain information of the running application, the type of the application, content information related to the running application, and a function used through the application in the memory, and the artificial intelligence The model may obtain an answer to the user question based on at least one of the stored domain information and the type of the application, content information related to the running application, and a function used through the application.

In this case, the user question may be at least one of user voice or text.

According to various embodiments of the present disclosure as described above, the electronic device may protect the privacy of the user while using an artificial intelligence model and may output a user-specific answer.

1 is an exemplary diagram for briefly describing an operation of an electronic device according to the present disclosure.

2 is a block diagram schematically illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

3 is a block diagram illustrating in detail an electronic device according to an embodiment of the present disclosure.

4 is a block diagram illustrating a conversation system according to an exemplary embodiment of the present disclosure.

5A and 5B are exemplary diagrams for describing data stored in a knowledge base according to an embodiment of the present disclosure.

6 is an exemplary diagram for describing a method of classifying domains (or categories) of data stored in a knowledge base according to an embodiment of the present disclosure.

7 is an exemplary view for explaining a domain management method according to an embodiment of the present disclosure.

8 is an exemplary diagram for describing a method of obtaining data stored in a knowledge base according to an embodiment of the present disclosure.

9 is an exemplary diagram for describing a method of outputting a question about a user's voice based on an application operation by an electronic device according to an embodiment of the present disclosure.

10A and 10B are exemplary diagrams for describing a method of recommending a user's application operation according to another exemplary embodiment of the present disclosure.

11A and 11B are exemplary views for describing various embodiments according to the present disclosure.

12 is an exemplary diagram for describing an operation of an electronic device and a server according to an embodiment of the present disclosure.

13 is a system diagram for describing an operation of an electronic device and a server according to an exemplary embodiment of the present disclosure.

14 is a system diagram for describing an operation of an electronic device and a server according to another exemplary embodiment of the present disclosure.

15 is a flowchart illustrating a control method of an electronic device according to an embodiment of the present disclosure.

Hereinafter, various embodiments of the present disclosure will be described with reference to the accompanying drawings. However, this is not intended to limit the techniques described in this document to specific embodiments, but should be understood to cover various modifications, equivalents, and / or alternatives to the embodiments of this document. . In connection with the description of the drawings, similar reference numerals may be used for similar components.

In this document, expressions such as "have," "may have," "include," or "may include" include the presence of a corresponding feature (e.g., numerical, functional, operational, or component such as a component). Does not exclude the presence of additional features.

In this document, expressions such as "A or B," "at least one of A or / and B," or "one or more of A or / and B" may include all possible combinations of items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" includes (1) at least one A, (2) at least one B, Or (3) both of cases including at least one A and at least one B.

As used herein, the expressions “first,” “second,” “first,” or “second,” and the like may modify various components in any order and / or in importance. It is used to distinguish it from other components and does not limit the components.

One component (such as a first component) is "(functionally or communicatively) coupled with / to" to another component (such as a second component) or " When referred to as "connected to," it is to be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is said to be "directly connected" or "directly connected" to another component (e.g., a second component), the component and the It may be understood that no other component (eg, a third component) exists between the other components.

The expression "configured to" used in this document is, for example, "having the capacity to," "suitable for," " , "" Designed to, "" adapted to, "" made to, "or" capable of. " The term "configured to" may not necessarily mean only "specifically designed to" in hardware. Instead, in some situations, the expression “device configured to” may mean that the device “can” along with other devices or components. For example, the phrase “subprocessor configured (or set up) to perform A, B, and C” may execute a dedicated processor (eg, an embedded processor) or one or more software programs stored on a memory device to perform the operation. By doing so, it may mean a general-purpose processor (for example, a CPU or an application processor) capable of performing the corresponding operations.

An electronic device according to various embodiments of the present disclosure may be, for example, a smartphone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a PDA, a PMP. (portable multimediaplayer), an MP3 player, a medical device, a camera, or a wearable device. Wearable devices may be accessory (e.g. watches, rings, bracelets, anklets, necklaces, eyeglasses, contact lenses, or head-mounted-devices (HMDs), textiles or clothing integral (e.g. electronic clothing), And may include at least one of a body-attachable (eg, skin pad or tattoo), or bio implantable circuit, In certain embodiments, an electronic device may comprise, for example, a television, a digital video disk (DVD) player, Audio, Refrigerator, Air Conditioner, Cleaner, Oven, Microwave, Washing Machine, Air Purifier, Set Top Box, Home Automation Control Panel, Security Control Panel, Media Box (e.g. Samsung HomeSyncTM, Apple TVTM, or Google TVTM), Game Console (Eg, XboxTM, PlayStationTM), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

In another embodiment, the electronic device may include a variety of medical devices (e.g., various portable medical measuring devices such as blood glucose meters, heart rate monitors, blood pressure meters, or body temperature meters), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), Computed tomography (CT), cameras or ultrasounds), navigation devices, global navigation satellite systems (GNSS), event data recorders (EDR), flight data recorders (FDRs), automotive infotainment devices, ship electronic equipment (E.g. ship navigation systems, gyro compasses, etc.), avionics, security devices, vehicle head units, industrial or home robots, drones, ATMs in financial institutions, point of sale (POS) points in stores or Internet of Things devices (eg, light bulbs, various sensors, sprinkler devices, fire alarms, thermostats, street lights, toasters, exercise equipment, hot water tanks, heaters, boilers, etc.). .

In this document, the term user may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) that uses an electronic device.

Hereinafter, with reference to the drawings will be described in more detail with respect to the present disclosure.

The electronic device 100 stores a user's application use, content use, web content use, etc. in the knowledge base 460, and when a user's question is input, the electronic device 100 outputs a response to a user question based on the knowledge base 460. Can be.

Specifically, the electronic device 100 determines a domain of data to be stored based on the user's application use, content use, web content use, and the like, and when the domain is determined, the user's application use, content use, and web content in the determined domain. You can store data about usage.

For example, the electronic device 100 stores the operation order of the user's application in the knowledge base 460, and when the user executes the application, determines the domain associated with the executed application, and stores the stored order in the determined domain. The application may be controlled according to the operation order of the stored application based on the data. Or, if a movie is reserved through a movie application, the electronic device 100 determines a domain (eg, a movie domain) related to the movie application, and transmits a reservation confirmation to another user based on data stored in the domain. I can recommend that. Alternatively, the electronic device 100 may store an action of a user who executes content through a specific application in the knowledge base 460, and when a user command to play related content is input, the electronic device 100 determines a domain related to the related content, An application for reproducing related content may be obtained based on data stored in a domain, and the related content may be reproduced through the obtained application. Alternatively, when a user command for bookmarking specific content is input, the electronic device 100 may determine a domain related to the bookmarked content and bookmark the content related to the bookmarked content based on the data stored in the determined domain. .

That is, the electronic device 100 obtains various data based on various usage histories of the electronic device 100, stores the data in the knowledge base 460 for each domain, and if a user question is input, the question stored in the knowledge base 460. Based on this, you can print the answer.

Meanwhile, the knowledge base 460 refers to a database that stores a data set consisting of subject data, object data, and predicate data according to a suitable schema. That is, the knowledge base 460 may express the association type between the data and the data in a generalized form (for example, in the form of a table of a relational database) and store the set of association types between the expressed data and the data. have. The subject data refers to data representing an object to be expressed, the association data refers to data representing a relationship between the subject and the object, and the object data refers to data representing the content or value of the association relationship. The set of subject data, connection relation data, and target data may be referred to as triple data.

The knowledge base 460 may be constructed by storing the triple data acquired by the electronic device 100 through a web page or various applications in a table form of a relational database.

According to an embodiment, the electronic device 100 may obtain triple data by performing morphological analysis and parsing on the text identified on the web page. For example, if the text "Seoul population is currently 97.7 million" is displayed on the web page, the electronic device 100 may stem and analyze the text to search for "Seoul (subject data) and population (association relation)." Data), and 9,770,000 (object data) 'triple data can be obtained. The knowledge base 460 may be constructed by storing triple data acquired by the electronic device 100 in a table form of a relational database as shown in Table 1.

주어 데이터Subject data	연관 관계 데이터Affinity data	목적 데이터Purpose data
서울Seoul	인구population	977만977 million

According to another embodiment, the electronic device 100 may obtain triple data including an application name (given data), an performed function (association data), and a content name (purpose data) based on the use of the application. . For example, when a movie reservation is executed in a movie application, the electronic device 100 obtains triple data consisting of a movie application name (subject data), a movie reservation (association relation data), and a movie title (purpose data). can do. The knowledge base 460 may be constructed by storing the triple data acquired by the electronic device 100 in the form of a table of relational data as shown in Table 1 below.

주어 데이터Subject data	연관 관계 데이터Affinity data	목적 데이터Purpose data
영화 어플리케이션 명칭Movie Application Name	영화 예매Movie ticket	영화 제목Movie title

Meanwhile, as an embodiment of the present disclosure, the electronic device 100 may output an answer to a user's voice based on triple data stored in the knowledge base 460. Specifically, when a user's voice including a question is input, the electronic device 100 recognizes and analyzes the user's voice to determine the intention of the question, and the intention of the identified question is based on any of the triple data stored in the knowledge base 460. It may be determined whether it corresponds to the association data. For example, when a user's voice of 'Saturday OOO movie is good?' Is input, the electronic device 100 determines that the user's intention is 'movie reservation', and the connection relationship data stored in the knowledge base 460 is stored. Identify data that is 'movie booking'. In addition, the electronic device 100 may identify the subject data in which the 'movie booking' of the triple data is association data using the table of relational data stored in the knowledge base 460. The electronic device 100 may identify the subject data (movie application) having the highest frequency of use among the identified subject data, and may output an answer for recommending a movie reservation to the identified movie application.

2 is a block diagram schematically illustrating a configuration of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 2, the electronic device 100 may include a memory 110 and a processor 120.

The memory 110 may store instructions or data related to at least one other component of the electronic device 100. In particular, the memory 110 may be implemented as a nonvolatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 110 is accessed by the processor 120, and may read / write / modify / delete / update data, etc. by the processor 120. In the present disclosure, the term memory refers to a memory 110, a ROM (not shown), a RAM (not shown), or a memory card (not shown) mounted in the electronic device 100 (eg, micro SD). Card, memory stick). In addition, the memory 110 may store programs and data for configuring various screens to be displayed on the display area of the display unit 150.

In addition, the memory 110 may store a dialogue system that provides a response to a user input (especially a user's voice). In this case, as shown in FIG. 4, the dialogue system includes an automatic speech recognition unit (ASR) 410, a natural language understanding unit (NLU) 420, and a dialogue manager (DM). 430, a natural language generator (NLG) 440, a text-to-speech (TTS) 450, and a knowledge database 460.

The automatic voice recognition unit 410 may perform voice recognition on a user voice input through a microphone or the like. The natural language understanding unit 420 may determine the intent of the user's voice based on the speech recognition result. The conversation manager 430 may obtain information about a response to the user's voice based on the natural language understanding result and the data stored in the knowledge base 460. For example, the conversation manager 430 may obtain information for generating a response. As described above, the obtained information may be obtained from the intent and knowledge of the user's voice identified through the natural language understanding unit 420. It may be determined based on the data stored in the base 460. The natural language generator 440 may obtain the natural language as a response to the user's voice based on the information obtained through the conversation manager 430. The TTS 450 may convert the obtained natural language into a voice, thereby enabling the conversation system to provide a response to the user's voice as a voice, thereby allowing the user to communicate with the electronic device 100. .

In particular, the natural language generator 440 according to an embodiment of the present disclosure inputs the information obtained through the conversation manager 430 and the knowledge base 460 as an input value of the artificial intelligence model, as a response to the user's voice. Natural language can be obtained.

Knowledge base 460 may store data for personalized responses. In this case, the data stored in the knowledge base 460 may vary. According to an embodiment, the knowledge base 460 may store at least one of a type of an application used by the electronic device 100, content information related to the application, and a function used through the application. As another example, the knowledge base 460 may store key information of text included in an application. As another example, when a user interaction for performing a function on an application being used is detected, the knowledge base 460 may store information about the performed function and the used user interaction. As another example, the knowledge base 460 may store category information about a running application and knowledge information about a running application. As another example, the knowledge base 460 may store information related to continuous use of an application, store information related to operation of an application, or use information about a specific application when using content of the same type when using a specific application. And store information by matching information about a specific type of content. In this case, the information on the application may be at least one of a type of the application, content information related to the running application, and a function (payment function, search function, advance reservation function, etc.) used through the application.

The memory 110 may also store an artificial intelligence agent for operating the conversation system. In detail, the electronic device 100 may use an artificial intelligence agent to generate natural language in response to a user voice. At this time, the artificial intelligence agent is a dedicated program for providing an AI (Artificial Intelligence) based service (for example, a voice recognition service, a secretary service, a translation service, a search service, etc.), and an existing general purpose processor (for example, CPU) or a separate AI dedicated processor (eg, GPU, etc.).

Specifically, when a user voice is input, the artificial intelligence agent may operate. The AI agent may obtain a response by inputting the user question into the learned AI learning model.

Of course, the AI agent may operate when a user voice (particularly, a trigger voice for executing an AI function) is input or a predetermined button (for example, a button for executing an AI assistant function) is selected. . Alternatively, the AI agent may be in a previously executed state before a user voice is input or a preset button is selected. In this case, after the user voice is input or a predetermined button is input, the artificial intelligence agent of the electronic device 100 may obtain an answer to the user question. Also, the AI agent may be in a standby state before a user voice is input or a preset button is selected. Here, the standby state is a state in which a predefined user input is received to control the start of the operation of the artificial intelligence agent. When a user voice is input or a preset button is selected while the AI agent is in a standby state, the electronic device 100 may operate the AI agent and acquire natural language in response to the user voice.

In addition, according to an embodiment of the present disclosure, the memory 110 may store an AI model trained to generate (or obtain) an answer to a user question. The artificial intelligence model learned in the present disclosure may be constructed in consideration of application fields of the recognition model or computer performance of the device. For example, the artificial intelligence model may be trained to obtain natural language using information obtained from the conversation manager 430 and the knowledge database 460 as input data. In order to generate natural natural language, the learned AI model may be, for example, a model based on a neural network. The AI model can be designed to simulate a human brain structure on a computer and can include a plurality of weighted network nodes that simulate neurons in a human neural network. The plurality of network nodes may form a connection relationship so that neurons simulate the synaptic activity of neurons that send and receive signals through synapses. In addition, the document summary model may include, for example, a neural network model or a deep learning model developed from the neural network model. In the deep learning model, a plurality of network nodes may be located at different depths (or layers) and exchange data according to a convolutional connection relationship. Examples of the learned AI model may include, but are not limited to, a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN).

In addition, in the above-described embodiment, the AI model is described as being stored in the electronic device 100. However, this is only an example, and the AI model may be stored in another electronic device. For example, the artificial intelligence model may be stored on at least one external server. The electronic device 100 receives a user voice and transmits it to an external server storing the artificial intelligence model, and inputs the user voice received from the electronic device 100 as an input value to output the result. Of course it can.

The processor 120 may be electrically connected to the memory 110 to control overall operations and functions of the electronic device 100.

In detail, the processor 120 may execute an application stored in the memory 110. When a user command is input while the application is running, the processor 120 may obtain and store at least one of a type of the application being executed, content information related to the application being executed, and a function used by the application in the memory 110. have. Specifically, when a user command is input while the application is running, the processor 120 acquires at least one of a type of the application being executed, content information related to the application being executed, and a function used through the application and the knowledge base 460. ) Can be stored. When a user question related to an application is input to the learned artificial intelligence model as an input value, the processor 120 uses the type of the running application stored in the knowledge base 460, content information related to the running application, and the application. The user may output an answer to a user question determined based on at least one of the functions.

In this case, the answer output by the processor 120 may be at least one of an answer providing a search result for a user question, an answer for performing an action on the user question, and an answer for performing a function of an application for the user question. .

Meanwhile, if the user question is a question related to content search, the processor 120 may recommend an application for playing the searched content based on information related to the application.

Meanwhile, the processor 120 may obtain an answer to the user question based on at least one of the stored domain information and the type of the executed application, content information related to the executed application, and a function used through the application.

On the other hand, if a user question is input, the processor 120 may transmit the user question to an external server, and receive a candidate to answer the user question input from the external server. The processor 120 may output an answer to a user question based on the received answer candidate and the information stored in the knowledge base 460. Specifically, the processor 120 calculates the similarity between the received answer candidate and the data stored in the knowledge base 460 through an artificial intelligence model, and has the highest similarity with the data of the knowledge base 460 among the received answer candidates. Answer candidates can be output.

The user question in the present disclosure is described as a case of a user voice, but is not limited thereto. That is, the user question may be input in the form of text.

Meanwhile, functions related to artificial intelligence according to the present disclosure are operated through the processor 120 and the memory 110. The processor 130 may be composed of one or a plurality of processors. At this time, one or a plurality of processors are general purpose processors such as CPU, AP, GPU. It may be a graphics dedicated processor such as a VPU or the like, or an AI dedicated processor such as an NPU.

The one or more processors control to process the input data according to a predefined operating rule or artificial intelligence model stored in the memory 110. The predefined action rule or artificial intelligence model is characterized by being made through learning.

In this case, to be made through learning means that a predetermined operating rule or artificial intelligence model of a desired characteristic is created by applying a learning algorithm to a plurality of learning data. Such learning may be made in the device itself in which the artificial intelligence according to the present disclosure is performed, or may be made through a separate server / system.

The artificial intelligence model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and the layer is calculated through the calculation result of the previous layer and the calculation of the plurality of weights. Examples of neural networks include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), and deep There are Deep Q-Networks, and the neural network in the present disclosure is not limited to the above examples except where specified.

The learning algorithm is a method of training a predetermined target device (eg, a robot) using a plurality of learning data so that the predetermined target device can make a decision or make a prediction by itself. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, where the learning algorithm in the present disclosure is specified. It is not limited to the above-described example except for.

As shown in FIG. 3, the electronic device 100 may further include a communication unit 130, an input unit 140, a display 150, and an audio output unit 160 in addition to the memory 110 and the processor 120. have. However, the present invention is not limited to the above-described configuration, and some configurations may be added or omitted as necessary.

The communication unit 130 is a component for performing communication with an external device. In an embodiment, the electronic device 100 may receive a plurality of answer candidates for a user's voice from an external server through the communication unit 130.

On the other hand, the communication unit 130 may be connected to the external device to communicate through a third device (for example, a repeater, hub, access point, server or gateway, etc.). The wireless communication may be, for example, LTE, LTE Advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global network (GSM). Cellular communication using at least one of the System for Mobile Communications, and the like. According to an embodiment, the wireless communication may include, for example, wireless fidelity (WiFi), Bluetooth, Bluetooth low power (BLE), Zigbee, near field communication (NFC), magnetic secure transmission, and radio. At least one of a frequency (RF) or a body area network (BAN). Wired communication may include, for example, at least one of a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard232 (RS-232), power line communication, or a plain old telephone service (POTS). have. The network in which wireless or wired communication is performed may include at least one of a telecommunication network, for example, a computer network (eg, LAN or WAN), the Internet, or a telephone network.

The input unit 140 is a component for receiving a user command. In this case, the input unit 140 may include a camera 141, a microphone 142, and a touch panel 143.

The camera 141 is a component for acquiring image data around the electronic device 100. The camera 141 may capture a still image and a video. For example, the camera 141 may include one or more image sensors (eg, a front sensor or a rear sensor), a lens, an image signal processor (ISP), or a flash (eg, an LED or an xenon lamp, etc.). The microphone 142 is a component for acquiring sound around the electronic device 100. The microphone 142 may receive electrical sound signals to generate electrical voice information. The microphone 142 may use various noise removing algorithms for removing noise generated in the process of receiving an external sound signal. Image information or voice information input through the camera 141 or the microphone 142 may be input as an input value of an artificial intelligence model.

The touch panel 143 is a component that can receive various user inputs. The touch panel 143 may receive data by user manipulation. The touch panel 143 may be configured in combination with a display which will be described later.

The input unit 140 may have various configurations for receiving various data in addition to the camera 141, the microphone 142, and the touch panel 143 described above.

The display 150 is a component for outputting various images. The display 150 for providing various images may be implemented as various types of display panels. For example, display panels can include Liquid Crystal Display (LCD), Organic Light Emitting Diodes (OLED), Active-Matrix Organic Light-Emitting Diode (AM-OLED), Liquid Crystal on Silicon (LcoS), or Digital Light Processing (DLP). It can be implemented in various display technologies such as. In addition, the display 150 may be coupled to at least one of a front region, a side region, and a rear region of the electronic device 100 in the form of a flexible display.

The audio output unit 160 is configured to output not only various audio data on which various processing tasks such as decoding, amplification, and noise filtering are performed, but also various notification sounds or voice messages. The audio processor is a component that performs processing on audio data. The audio processor may perform various processing such as decoding, amplification, noise filtering, and the like on the audio data. The audio data processed by the audio processor 150 may be output to the audio output unit 160. In particular, the audio output unit may be implemented as a speaker, but this is only an example, and may be implemented as an output terminal capable of outputting audio data.

As described above, the processor 120 controls the overall operation of the electronic device 100. The processor 120 may include a RAM 121, a ROM 122, a main CPU 123, a graphics processor 124, first to n interface 125-1 to 125-n, and a bus 126. have. In this case, the RAM 121, the ROM 122, the main CPU 123, the graphics processor 124, and the first through n interfaces 125-1 through 125-n may be connected to each other through the bus 126. .

The ROM 122 stores a command set for system booting. When the turn on command is input and the power is supplied, the main CPU 123 copies the O / S stored in the memory to the RAM 121 according to the command stored in the ROM 122, and executes the O / S to boot the system. Let's do it. When booting is completed, the main CPU 123 copies various application programs stored in the memory to the RAM 121 and executes the application programs copied to the RAM 121 to perform various operations.

In detail, the main CPU 123 accesses the first memory 110 or the second memory 120 to perform booting using an operating system stored in the first memory 110 or the second memory 120. do. The main CPU 123 performs various operations using various programs, contents, data, etc. stored in the first memory 110 or the second memory 120.

The first to n interfaces 125-1 to 125-n are connected to the various components described above. One of the interfaces may be a network interface connected to an external device through a network.

Hereinafter, various embodiments according to the present disclosure will be described with reference to FIGS. 5A through 11B.

In detail, as illustrated in FIG. 5A, when a preset user voice is input, the electronic device 100 transmits data related to an application or web content that is executed at a time when a preset user command is input to the knowledge base 460. Can be stored. For example, when a user command such as “bixby! Remember!” 510 is input, the electronic device 100 may store data related to an application or web content that is running at a point in time when a preset user voice is input. 460). In this case, the time when the preset user voice is input may mean a preset time before and after the time when the user voice is input.

Meanwhile, as illustrated in FIG. 5B, when an operation of pushing the button 520 provided in the electronic device 100 is detected, the electronic device 100 may execute an application or web running at a time when a preset user command is input. Data related to the content may be stored in the knowledge base 460. In this case, a method of pushing the button 520 provided in the electronic device 100 may vary. For example, the electronic device 100 may be pushed by pushing the button 520 for a preset time (for example, 2 seconds) and pushing the button 520 for a preset number of times (for example, 3 times). ) May store data related to an application or web content running at the time when the corresponding motion is detected in the knowledge base 460.

However, the present invention is not limited to the embodiments of FIGS. 5A and 5B, and operations for storing data in the knowledge base 460 may vary. For example, when the electronic device 100 satisfies a specific condition among executed applications or web content, the electronic device 100 may store data related to the corresponding application or web content in the knowledge base 460. In this case, the specific condition may be, for example, a condition of using an application or web content a predetermined number of times. Alternatively, the specific condition may be a condition of using an application or web content for a predetermined time. Alternatively, the specific condition may be a use condition of all applications or web content detected by the electronic device 100. Alternatively, the specific condition may be a condition related to a touch input for bookmarking specific web content.

According to the above-described embodiment, when data related to an application or web content for storing in the knowledge base 460 is determined, the electronic device 100 may determine a domain (or category) of the determined data.

In detail, the electronic device 100 may determine a domain based on the content of the running application or the web content.

For example, as shown in FIG. 6, when the electronic device 100 executes a news article in a browser application, the electronic device 100 may determine a domain based on the content of the executed news article. The domain determined at this time may be, for example, an IT domain.

Alternatively, when the electronic device 100 executes a game application, the electronic device 100 may determine a domain based on the game application to be executed. The domain determined at this time may be, for example, a sports domain.

Alternatively, when the electronic device 100 executes an application related to the payment details, the electronic device 100 may determine a domain based on the payment details. For example, if there are many restaurants that go frequently in the payment details, the determined domain may be a restaurant domain.

Meanwhile, the domain may be stored in the knowledge base 460. In this case, when a new domain needs to be added, the electronic device 100 may add a new domain corresponding to data related to an application or web content. In this case, the added domain may be one of domains stored in an external server. Of course, the domain can be added or deleted by user command.

Meanwhile, when there is no data added to one of the domains in the knowledge base 460 for a preset time, the electronic device 100 may delete the corresponding domain. That is, the electronic device 100 may save the memory by deleting the unused domain.

Meanwhile, the electronic device 100 may edit domains in the order in which data is stored. That is, the electronic device 100 may determine the priority of domains according to the size or number of data stored in each domain. For example, as illustrated in FIG. 6, when the size or number of data stored in the domain is in the order of a restaurant domain, a movie domain, an IT domain, and a sports domain, the electronic device 100 may serve as a restaurant domain, a movie domain, an IT domain, or a sport. Priority can be given in order of domain. When the user's voice is input, the electronic device 100 may output an answer for the user's voice according to the priority.

The electronic device 100 may store various data in the knowledge base 460 according to various embodiments of FIGS. 5A through 6. In this case, when the data stored in the knowledge base 460 increases, the electronic device 100 may remove some of the stored data to secure a memory space. For example, the electronic device 100 may first delete the stored data. Alternatively, the electronic device 100 may first delete data having a low number of uses.

Meanwhile, the electronic device 100 may not only remove some of the data but also transmit the data to an external server. That is, the electronic device 100 may secure the memory space of the electronic device 100 by transmitting old data (or unused data) to an external server such as a personal cloud server.

When the application is executed, the electronic device 100 may obtain the name of the application to be executed, the content name to be executed in the application, and the function to be executed in the application to be stored in the knowledge base 460.

In detail, the electronic device 100 may textify and tokenize data identified in the application or the web content. Tokenization refers to the task of classifying text into words. When the tokenization is completed, the electronic device 100 performs Part Of Speech Tagging (POS). POS refers to the task of identifying and tagging parts of speech in text. The electronic device 100 may parse the text and remove the preset stop word. After removing the negative word, the electronic device 100 may determine the headword. Heading determination refers to the process of grouping the refraction forms of a word so that it can be analyzed as a single item identified in the auxiliary theorem or dictionary form of the word. Through the above-described process, the electronic device 100 may obtain an application name, a content name, and a function name and store it in the knowledge base 460.

For example, it may be assumed that the electronic device 100 is executing a news article on a web page. In this case, the electronic device 100 may obtain an original sentence (eg, Samsung research has AI Center.) Of the news. The electronic device may separate raw sentences through tokenization, POS, and the like. For example, the original sentence may be classified as "(Samsung research) -noun phrase, (has) -verb, (AI Center) -noun phrase". The electronic device 100 may generate a tuple from the separated sentence, generalize it, and store the tuple structure in the knowledge base 460. For example, if the generated tuple is "Samsung research, AI Center, have", the electronic device 100 is "Samsung research, AI Center, have", "Samsung research, AI Center, contain", "Samsung research, Data such as AI Center, include "may be stored in the knowledge base 460.

Meanwhile, as described above, data stored in the knowledge base 460 may be generated in a specific situation. For example, the data stored in the knowledge base 460 may be stored when the user command is not input to the electronic device, when the electronic device 100 is being charged, or at dawn (for example, between 0 am and 6 am). It may be generated when any one of the conditions, such as). That is, the data stored in the knowledge base 460 may be generated in a situation where the user does not use the electronic device 100 to efficiently use the electronic device 100.

Hereinafter, a method of outputting a response to a user question when a user question is input while data is stored in the knowledge base 460 according to the embodiments of FIGS. 5A to 8 will be described.

When the user question is input, the electronic device 100 may determine the intention of the user question. In detail, the electronic device 100 may determine the intention of the user's voice through the natural language understanding unit 420. The electronic device 100 may determine a domain suitable for the intention of the user voice based on the intention of the user voice and the domain stored in the knowledge base 460. In detail, the electronic device 100 may determine the domains most similar to the user intentions by determining similarities between the identified user intentions and the plurality of domains stored in the knowledge base 460. The electronic device 100 may output a result value for the user intention based on the data included in the determined domain and the intention for the user voice. In detail, the electronic device 100 may determine similarity between the identified user intention and the plurality of data included in the determined domain, and may determine the data most similar to the user intention. The electronic device 100 may output an answer to a user question based on the determined data.

9 through 11B are exemplary diagrams for describing various embodiments according to the present disclosure.

As described above, the electronic device 100 may store data obtained based on the use of various applications or web contents in the knowledge base 460. That is, the electronic device 100 may determine data stored in the domain and the knowledge base 460 based on the application or the web content. For example, when the electronic device 100 executes a news article of a web browser, the electronic device 100 analyzes the news article to determine the domain as the IT domain, and selects "Samsung Electronics, OO Won, Operating Profit". The same data may be obtained and stored in the knowledge base 460. As another example, when the electronic device 100 executes the movie booking application, the electronic device 100 determines the domain as the movie domain based on the movie application, and the data such as "movie name, movie application name, advance reservation". Can be stored. Alternatively, the electronic device 100 may store data such as "a movie theater, a movie theater location, a movie theater name" based on a movie application. Alternatively, the electronic device 100 may store data such as "movie ticket, quantity of movie tickets" based on the movie application. As another example, when the electronic device 100 executes an application related to payment details, the electronic device 100 may analyze the payment details and store "restaurant name, price, function (payment, cancellation, etc.)".

When data is stored in the knowledge base 460, when a user voice of "Saturday OOO movie is good?" Is input, the electronic device 100 may output an appropriate answer based on the user voice. For example, the electronic device 100 confirms that the user does not have a schedule after 2 pm on the basis of the information obtained from the calendar application, and based on the information stored in the knowledge base 460, the user mainly uses the OOO application. After confirming that the movie reservation is made, it is possible to print out an answer such as "2:30 pm at the OOO cinema. In this case, the electronic device 100 may obtain information related to a seat preferred by the user, a quantity of movie tickets to be reserved, and a location of a movie theater through a movie application, and reserve a movie theater and a seat of a proper location to the user.

As shown in FIG. 10A, the electronic device 100 reserves a user command (for example, "bixby memorizes") to store a series of operations for booking a movie through a movie application and sharing the movie reservation content to another user. ! ") Can be entered. In detail, the electronic device 100 may capture a reservation screen after booking a movie through a movie application, and store a series of operations shared to other users through the chat application in the knowledge base 460. Thereafter, as illustrated in FIG. 10B, when the booking screen is executed through the movie application, the electronic device 100 may recommend sharing the booking screen to other users through the chat application. That is, the electronic device 100 may not only store data related to one application or web content in the knowledge base 460, but also store one data related to a plurality of applications or web content in the knowledge base 460. Can be stored. That is, the electronic device 100 may not only store data included in the application or the web content itself, but may also store a plurality of user commands (for example, movie reservation, photo sharing, etc.) input to the application or the web content. If some of the plurality of user commands stored in 460 are stored, the user command may be recommended.

As illustrated in FIG. 11A, when a user command of “OOO play” is input, the electronic device 100 may search for data related to OOO content in the knowledge base 460. At this time, when the user watches up to 6 episodes of the OOO content and the record viewed through the specific application is stored in the knowledge base 460, the electronic device 100 viewed up to 6 episodes. Do you want to play it? "

On the other hand, as shown in Figure 11b, when the user command "show the daughter picture" is input, the electronic device 100 outputs a response such as "I have a picture taken yesterday. Send to the wife through the OO application?" In other words, the electronic device 100 may not only answer the user's question (daughter photo search) but also recommend additional actions not instructed by the user, in this case, the knowledge base 460 may include the user's daughter. There may be data associated with a series of actions that a picture is retrieved and sent to the wipe through an OO application.

Meanwhile, as described above, the intention of the user corresponding to the user voice and the generation of the natural language corresponding to the result data of the user intention may be performed by the electronic device 100, but is not limited thereto. That is, as shown in FIG. 12, it is a matter of course that the intention of the user corresponding to the user voice and the generation of the natural language corresponding to the result data of the user intention may be performed by the server 200. That is, when a user voice is input, the electronic device 100 transmits the user voice to the server 200, and the server 200 identifies the user's intention corresponding to the received user voice and results data on the user's intention. Of course, the natural language corresponding to the control unit may be generated and transmitted to the electronic device 100.

In the above-described embodiment, a method of outputting an answer based on the knowledge base 460 by the electronic device 100 has been described, but is not limited thereto. That is, the electronic device 100 receives a plurality of answer candidates corresponding to the user's voice from the server 200, and responds to the user's voice based on the plurality of answer candidates and the data stored in the knowledge base 460. You can of course output.

First, the electronic device 100 may receive a user voice in operation S1410. When the user voice is input, the electronic device 100 may perform voice recognition on the input user voice (S1420). In detail, the voice recognition may be performed through the automatic voice recognition unit 410.

The electronic device 100 may determine the intention of the user's voice based on the voice recognition result in operation S1430. The electronic device 100 may transmit the determined intention of the user's voice to the external server 200 (S1440).

The server 200 may obtain a plurality of answer candidates for the intention of the user's voice based on the intention of the user's voice (S1450). In this case, the server 200 may obtain a plurality of answer candidates based on a database included in the server 200. In detail, the server 200 may input the intention of the user's voice and a plurality of data included in the database into the artificial intelligence model to obtain data similar to the intention of the user's voice as the answer candidate. For example, if the intention of the user's voice is a movie recommendation, the server 200 may obtain an answer candidate such as an answer candidate related to a movie ranking currently being released and an answer candidate related to a movie ranking by genre.

The server 200 may transmit the obtained plurality of answer candidates to the electronic device 100 (S1460). The electronic device 100 may determine a final answer based on the similarity between the data stored in the knowledge base 460 and the received plurality of answer candidates in operation S1470. Specifically, the electronic device 100 inputs the data stored in the knowledge base 460 and the received plurality of answer candidate data to the artificial intelligence model to determine a domain most similar to the intention of the user's voice, and is included in the determined domain. Of the data, data similar to the intention of the user's voice may be determined as the final answer. For example, when the electronic device 100 is an answer candidate related to a movie ranking currently being released from a server and an answer candidate related to a movie ranking by genre, the electronic device 100 selects a movie domain among a plurality of domains from the received response candidate. Determine and recommend a movie similar to the intention of the user question, among the data contained in the movie domain. For example, when the movie domain of the knowledge base 460 includes information for reserving a movie through a movie application and information related to an action movie, the electronic device 100 may reserve an action movie among the currently open movies. You can print out the answer. Alternatively, when there is no action movie among the currently open movies, the electronic device 100 may output an answer for booking the movie with the highest number of tickets currently being opened. Alternatively, when there is no action movie among the currently open movies, the electronic device 100 may output an answer recommending to download or watch the movie with the highest number of views in the action movie genre. That is, the electronic device 100 may recommend an answer to a user question according to the similarity between the data included in the determined domain and the plurality of answer candidates.

The electronic device 100 may output a natural language response to the user's voice using the natural language generator 440 and the TTS 450 (S1480).

Meanwhile, although the electronic device 100 performs voice recognition in the embodiment of FIG. 13, the present invention is not limited thereto. For example, the electronic device 100 may include only an artificial intelligence model for calculating a similarity between the knowledge base 460 and the plurality of answer candidates and the data of the knowledge base 460. In this case, the electronic device 100 transmits a user's voice to the server 200 so that the server 200 processes voice recognition, conversation management, natural language generation, and the like. Of course, only the candidate can be determined.

As shown in FIG. 14, the server 200 may include both a speech recognition system and a knowledge base 460. In this case, the server 200 may be configured as a general external server, but may be configured as a personal cloud server.

First, the electronic device 100 may receive a user voice in operation S1510. The electronic device 100 may transmit the received user voice to the server 200 in operation S1520. The server 200 may perform voice recognition on the received user voice (S1530). In detail, the speech recognition may be performed through the automatic speech recognition unit.

The server 200 may determine the intention of the user's voice based on the speech recognition result (S1540), and obtain a plurality of answer candidates for the intention of the user's voice based on the intention of the user's voice (S1550). In detail, the server 200 may obtain a plurality of answer candidates based on a database included in the server 200. For example, the server 200 may input the intention of the user's voice and a plurality of data included in the database into the artificial intelligence model to obtain data similar to the intention of the user's voice as the answer candidate.

The server 200 may determine a final answer based on the similarity between the data stored in the knowledge base and the plurality of answer candidates received (S1560). In detail, the server 200 may input data stored in the knowledge base and received plurality of answer candidate data into an artificial intelligence model to determine data similar to the intention of the user's voice as the final answer.

The server 200 may obtain a natural language response to the user's voice using the natural language generator and the TTS (S1570), and transmit the obtained natural language response to the electronic device 100 (S1580). The electronic device 100 may output the received natural language response in operation S1590.

Meanwhile, in the present disclosure, the user voice is input and the natural language response to the input user voice is output, but the present invention is not limited thereto. That is, the user's question inputted to the electronic device 100 and the outputted answer may be in the form of text instead of voice.

The electronic device 100 may execute an application in operation S1610. In more detail, when a user command for executing an application is input, the electronic device 100 may execute an application corresponding to the user command. In this case, the electronic device 100 may execute not only an application but also a web browser and web content.

When the user command is input, the electronic device 100 may obtain information about the running application and store it in the memory in operation S1620. In detail, the electronic device 100 may store information about an application being executed in the knowledge base 460. The user command may be a voice command (for example, remember Bixby!) Or a command for pushing a button provided in the electronic device 100 in a specific method. The information on the running application may be information related to a running application name, a content name played by the application, and a function name executed by the application. Alternatively, the information about the application may be information related to the content (for example, news article) displayed by the application. Alternatively, the information related to the application may be information related to the operation of the application.

When a user question related to an application is input to the learned artificial intelligence model as an input value, the electronic device 100 may output an answer to the user question determined based on information about the stored application. In detail, the electronic device 100 may calculate a similarity between the user question and the data stored in the knowledge base 460 using an artificial intelligence model, and output an answer related to the data having the highest similarity as an answer to the user question. have. Meanwhile, the electronic device 100 may transmit a user voice to the server 200, and the server 200 may transmit a plurality of answer candidates for the user voice to the electronic device 100. In this case, the electronic device 100 may calculate the similarity between the plurality of answer candidates and the data stored in the knowledge base 460 using an artificial intelligence model, and output the answer candidate with the highest similarity as the answer to the user question. have.

On the other hand, the term "part" or "module" as used in the present disclosure includes a unit composed of hardware, software, or firmware, and for example, may be used interchangeably with terms such as logic, logic block, component, or circuit. Can be. The "unit" or "module" may be an integrally formed part or a minimum unit or part of performing one or more functions. For example, the module may be configured as an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure may be implemented in software that includes instructions stored in a machine-readable storage media. As a device capable of calling and operating according to the called command, the device may include an electronic device according to the disclosed embodiments (eg, the electronic device 100.) When the command is executed by the processor, the processor directly, Alternatively, other components may be used to perform functions corresponding to the instructions under the control of the processor, and the instructions may include code generated or executed by a compiler or an interpreter. It may be provided in the form of a non-transitory storage medium, where 'non-transitory' means that the storage medium does not contain a signal. Only it means that the material (tangible) data is not case that the permanently or temporarily stored in the storage medium.

According to one embodiment, a method according to various embodiments disclosed in the present disclosure may be provided included in a computer program product. The computer program product may be traded between the seller and the buyer as a product. The computer program product may be distributed online in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)) or through an application store (eg Play StoreTM). In the case of an online distribution, at least a portion of the computer program product may be stored at least temporarily or temporarily created in a storage medium such as a server of a manufacturer, a server of an application store, or a relay server.

Each component (eg, a module or a program) according to various embodiments may be composed of a singular or plural entity, and some of the above-described subcomponents may be omitted, or other subcomponents may be omitted. It may be further included in various embodiments. Alternatively or additionally, some components (eg, modules or programs) may be integrated into one entity to perform the same or similar functions performed by each corresponding component prior to integration. In accordance with various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or at least some of the operations may be executed in a different order, omitted, or other operations may be added. Can be.

Claims

In the control method of an electronic device,

If a user command is input while the application is running, acquiring and storing at least one of a type of the application, content information related to the running application, and a function used by the application in a memory; And

If a user question related to the application is input, outputting an answer to the obtained user question by inputting a user question related to the application to a learned artificial intelligence model; Including,

The answer to the user question is determined based on at least one of the type of the application, the content information associated with the running application, and the function used through the application.
The method of claim 1,

The storing step,

If the running application includes text, extracting and storing key information of the text; Control method comprising a.
The method of claim 1,

The storing step,

If user interaction for performing a function on the running application is detected, storing history information including information on the performed function and the user interaction; Control method comprising a.
The method of claim 1,

The outputting step,

And providing at least one of providing a search result for the user question, performing an action on the user question, and performing an application function on the user question.
The method of claim 1,

The outputting step,

If the user question is a question related to content search, recommending an application for playing the searched content based on information related to the application; Control method comprising a.
The method of claim 1,

The outputting step,

If the user question is input, transmitting the user question to an external server;

Receiving candidates for answering the user question from the external server; And

Outputting an answer to the user question based on the received answer candidate and the information stored in the memory; Control method comprising a.
The method of claim 1,

The storing step,

Storing at least one of domain information on the running application, a type of the application, content information related to the running application, and a function used through the application;

The artificial intelligence model,

And obtaining an answer to the user question based on at least one of the stored domain information and the type of the application, content information related to the running application, and a function used through the application.
The method of claim 1,

And the user question is at least one of user voice or text.
In an electronic device,

Memory; And

A processor to execute an application stored in the memory; Including,

The processor,

When a user command is input while an application is running, at least one of a type of the application, content information related to the running application, and a function used through the application is acquired and stored in the memory, and a user question related to the application is obtained. If is input, and outputs the answer to the user question obtained by inputting a user question related to the application to the learned artificial intelligence model,

The answer to the user question is determined based on at least one of a type of the application, content information related to the running application, and a function used through the application.
The method of claim 9,

The processor,

And when the running application includes text, extracts key information of the text and stores the essential information in the memory.
The method of claim 9,

The processor,

And when a user interaction for performing a function on the running application is detected, the electronic device stores history information including information on the performed function and the user interaction in the memory.
The method of claim 9,

The processor,

And outputting at least one of providing a search result for the user question, performing an action on the user question, and performing an application function on the user question.
The method of claim 9,

The processor,

And if the user question is a question related to content search, recommending an application for playing the searched content based on the information related to the application.
The method of claim 9,

Communication unit; More,

The processor,

When the user question is input, the user question is transmitted to an external server through the communication unit, a response candidate for the user question is received from the external server through the communication unit, and the received response candidate and stored in the memory An electronic device that outputs an answer to the user question based on the information.
The method of claim 9,

The processor,

Storing at least one of domain information of the running application, the type of the application, content information related to the running application, and a function used through the application in the memory;

The artificial intelligence model,

And obtaining an answer to the user question based on at least one of the stored domain information and the type of the application, content information related to the running application, and a function used through the application.