KR102596841B1 - Electronic device and method for providing one or more items responding to speech of user - Google Patents

Abstract

According to various embodiments, electronic devices and systems may search for one or more items in response to a user's voice command related to searching for an item. When multiple items are searched, the user can perform various activities related to the multiple items. Each of the plurality of items may include a plurality of objects. Electronic devices and systems may identify, from the activity, an object that a user uses to identify at least one item (eg, a user's preferred item) among a plurality of items. By matching the identified object and the characteristics of the identified object, electronic devices and systems can determine the user's preferences related to retrieval of the item. The determined preference can be used for sorting multiple items.

Description

Electronic device and method for providing one or more items in response to a user's utterance {ELECTRONIC DEVICE AND METHOD FOR PROVIDING ONE OR MORE ITEMS RESPONDING TO SPEECH OF USER}

Various embodiments described below relate to a device that provides one or more items to a user in response to the user's utterance and a system including the device.

When a user uses an internet shopping service, various products may be output as items. Users can sort output products using criteria provided by the operator (e.g., price range, color, product type, seller, low price, high price, registration date, most product reviews). The operator of the service can decide what layout to deliver information related to the item to the user. In the example of the Internet shopping service described above, the user can identify the prices and ratings of products according to the layout determined by the operator.

When a user sorts various items output from a service provided through an electronic device, the criteria for sorting the items may be limited by the operator of the service. Accordingly, the user may require a solution to change the criteria for sorting items to suit the user's intention.

The technical problem to be achieved in this document is not limited to the technical problem mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

According to various embodiments, a display, at least one communication circuit, a microphone, at least one speaker, at least one processor operably coupled to the display, the communication circuit, the microphone, and the speaker, and electrically coupled to the processor. At least one memory, wherein the memory is configured to store an application program including a user interface, wherein the memory, when executed, causes the processor to display the user interface to include one or more objects. display on a display, receive a first user input for selecting one of the objects, and transmit, through the communication circuit, first information related to the selected object to an external server, and through the communication circuit , receiving second information about one or more properties of the selected object from the external server, displaying it on the user interface, and receiving a second user input for selecting at least one property among the properties, An electronic device may be provided that stores instructions for transmitting third information related to the selected attribute to the external server through the communication circuit.

According to various embodiments, it includes a communication interface, a processor operatively connected to the communication interface, and at least one memory electrically connected to the processor, wherein the memory, when executed, causes the processor to display information on a display. Receiving, via the communication interface, a request for first information regarding one or more attributes related to an object selected from among the objects, from an electronic device displaying a user interface including one or more objects, and In response to a request for first information, transmit the first information to the electronic device through the communication interface, and transmit, from the electronic device through the communication interface, the first information to at least one attribute selected from the one or more attributes. A system may be provided that receives a request for related second information and stores instructions for transmitting the second information to the electronic device through the communication interface according to the request for the second information.

According to various embodiments, it includes a memory for storing a voice signal obtained from a user, a display for outputting a user interface related to the user, and a processor, wherein the processor, in response to the voice signal, displays within the user interface. (in) displaying a plurality of items based on a first order, wherein each of the plurality of items includes at least one visual element, and the user interface is displayed together with the plurality of items, and 1 Contains at least one executable object for changing the order, in a designated operating mode of the user interface, in response to an input of the user selecting the at least one executable object, indicated by the selected object displaying the plurality of items within the user interface based on a second order, and in response to an input of the user selecting one of the at least one visual element in the designated operation mode, An electronic device may be provided that displays the plurality of items within the user interface based on a first order and a third order that is distinct from the second order.

An electronic device and a method performed by the electronic device according to various embodiments may sort a plurality of items based on the characteristics of a visual element selected by a user.

The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 is a block diagram showing an integrated intelligence system according to an embodiment.
Figure 2 is a diagram showing how relationship information between concepts and actions is stored in a database, according to an embodiment.
FIG. 3 is a diagram illustrating a user terminal displaying a screen for processing voice input received through an intelligent app, according to one embodiment.
4 is a block diagram of an electronic device in a network environment, according to various embodiments.
FIG. 5 is a diagram for explaining the structure of an electronic device and system according to various embodiments.
FIG. 6 is a diagram conceptually illustrating hardware components or software components used by systems according to various embodiments to manage preferences.
FIG. 7 is a flowchart illustrating an operation in which an electronic device or system aligns a plurality of items provided to a user using a preference object, according to various embodiments.
8A to 8C are exemplary diagrams for explaining a user interface provided to a user by an electronic device according to various embodiments.
9A to 9C are exemplary diagrams illustrating an operation of an electronic device requesting a user to select at least one of a plurality of attributes or features of an object selected by the user, according to various embodiments.
FIGS. 10A and 10B are exemplary diagrams illustrating an operation in which an electronic device changes the order in which a plurality of items are arranged within a display using a preference object created based on a user input, according to various embodiments.
FIG. 11 is an example diagram illustrating the structure of a preference object managed by an electronic device or system according to various embodiments.
FIG. 12 is a signal flow diagram to explain interaction between electronic devices and systems according to various embodiments.
FIG. 13 is a diagram illustrating an operation of a system identifying preferences from a user according to various embodiments.
FIGS. 14A to 14C are diagrams illustrating an operation of an electronic device changing the order of a plurality of items based on preferences obtained from a user, according to various embodiments.
FIG. 15 is a diagram illustrating an operation in which a system connected to a plurality of content providing devices shares a preference object related to one of the plurality of content providing devices according to various embodiments.
FIG. 16 is an exemplary diagram illustrating an operation in which a system according to various embodiments shares preferences among a plurality of content providing devices.
FIGS. 17A and 17B are exemplary diagrams illustrating an operation in which an electronic device shares preferences between a plurality of applications related to each of a plurality of content providing devices, according to various embodiments.
FIGS. 18A and 18B are exemplary diagrams for explaining an operation of an electronic device outputting a preference object to a user according to various embodiments.
FIG. 19 is a diagram illustrating an example of a user interface provided by an electronic device to a user to identify a preference object, according to various embodiments.
Figure 20 is a flowchart for explaining the operation of an electronic device according to various embodiments.
Figure 21 is a flowchart for explaining the operation of a system according to various embodiments.
FIG. 22 is a flowchart illustrating an operation in which a system according to various embodiments acquires a score related to the attributes of an object identified by a user of an electronic device.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosed form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. The same reference numerals in each drawing indicate the same members.

1 is a block diagram showing an integrated intelligence system according to an embodiment.

Referring to FIG. 1, the integrated intelligence system 10 of one embodiment may include a user terminal 100, an intelligent server 200, and a service server 300.

The user terminal 100 of one embodiment may be a terminal device (or electronic device) capable of connecting to the Internet, for example, a mobile phone, a smartphone, a personal digital assistant (PDA), a laptop computer, a TV, a white appliance, It could be a wearable device, HMD, or smart speaker.

According to the illustrated embodiment, the user terminal 100 may include a communication interface 110, a microphone 120, a speaker 130, a display 140, a memory 150, or a processor 160. The components listed above may be operatively or electrically connected to each other.

The communication interface 110 of one embodiment may be configured to connect to an external device to transmit and receive data. The microphone 120 in one embodiment may receive sound (eg, a user's speech) and convert it into an electrical signal. The speaker 130 in one embodiment may output an electrical signal as sound (eg, voice). Display 140 in one embodiment may be configured to display images or video. The display 140 in one embodiment may also display a graphic user interface (GUI) of an app (or application program) being executed.

The memory 150 in one embodiment may store a client module 151, a software development kit (SDK) 153, and a plurality of apps 155. The client module 151 and SDK 153 may form a framework (or solution program) for performing general functions. Additionally, the client module 151 or SDK 153 may configure a framework for processing voice input.

In one embodiment of the memory 150, the plurality of apps 155 may be programs for performing designated functions. According to one embodiment, the plurality of apps 155 may include a first app 155_1 and a second app 155_3. According to one embodiment, each of the plurality of apps 155 may include a plurality of operations to perform a designated function. For example, the apps may include an alarm app, a messaging app, and/or a schedule app. According to one embodiment, the plurality of apps 155 are executed by the processor 160 to sequentially execute at least some of the plurality of operations.

The processor 160 in one embodiment may control the overall operation of the user terminal 100. For example, the processor 160 may be electrically connected to the communication interface 110, the microphone 120, the speaker 130, and the display 140 to perform a designated operation.

The processor 160 of one embodiment may also execute a program stored in the memory 150 to perform a designated function. For example, the processor 160 may execute at least one of the client module 151 or the SDK 153 and perform the following operations for processing voice input. The processor 160 may control the operation of the plurality of apps 155 through the SDK 153, for example. The following operations described as operations of the client module 151 or SDK 153 may be operations performed by the processor 160.

The client module 151 in one embodiment may receive voice input. For example, the client module 151 may receive a voice signal corresponding to a user utterance detected through the microphone 120. The client module 151 may transmit the received voice input to the intelligent server 200. The client module 151 may transmit status information of the user terminal 100 to the intelligent server 200 along with the received voice input. The status information may be, for example, execution status information of an app.

The client module 151 of one embodiment may receive a result corresponding to the received voice input. For example, if the intelligent server 200 can calculate a result corresponding to the received voice input, the client module 151 may receive a result corresponding to the received voice input. The client module 151 may display the received result on the display 140.

The client module 151 of one embodiment may receive a plan corresponding to the received voice input. The client module 151 may display the results of executing multiple operations of the app according to the plan on the display 140. For example, the client module 151 may sequentially display execution results of a plurality of operations on a display. For another example, the user terminal 100 may display only some results of executing a plurality of operations (eg, the result of the last operation) on the display.

According to one embodiment, the client module 151 may receive a request from the intelligent server 200 to obtain information necessary to calculate a result corresponding to the voice input. According to one embodiment, the client module 151 may transmit the necessary information to the intelligent server 200 in response to the request.

The client module 151 in one embodiment may transmit information as a result of executing a plurality of operations according to the plan to the intelligent server 200. The intelligent server 200 can use the result information to confirm that the received voice input has been processed correctly.

The client module 151 in one embodiment may include a voice recognition module. According to one embodiment, the client module 151 can recognize voice input that performs a limited function through the voice recognition module. For example, the client module 151 may run an intelligent app for processing voice input to perform an organic action through a designated input (e.g., wake up!).

The intelligent server 200 in one embodiment may receive information related to the user's voice input from the user terminal 100 through a communication network. According to one embodiment, the intelligent server 200 may change data related to the received voice input into text data. According to one embodiment, the intelligent server 200 may generate a plan for performing a task corresponding to the user's voice input based on the text data.

According to one embodiment, the plan may be generated by an artificial intelligence (AI) system. Artificial intelligence systems may be rule-based systems or neural network-based systems (e.g., feedforward neural network (FNN), recurrent neural network (RNN)). ))) It could be. Alternatively, it may be a combination of the above or a different artificial intelligence system. According to one embodiment, a plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, an artificial intelligence system can select at least one plan from a plurality of predefined plans.

The intelligent server 200 of one embodiment may transmit a result according to the generated plan to the user terminal 100 or transmit the generated plan to the user terminal 100. According to one embodiment, the user terminal 100 may display results according to the plan on the display. According to one embodiment, the user terminal 100 may display the results of executing an operation according to the plan on the display.

The intelligent server 200 of one embodiment includes a front end 210, a natural language platform 220, a capsule DB 230, an execution engine 240, It may include an end user interface (250), a management platform (260), a big data platform (270), or an analytic platform (280).

The front end 210 of one embodiment may receive voice input from the user terminal 100. The front end 210 may transmit a response corresponding to the voice input.

According to one embodiment, the natural language platform 220 includes an automatic speech recognition module (ASR module) 221, a natural language understanding module (NLU module) 223, and a planner module (223). It may include a planner module (225), a natural language generator module (NLG module) (227), or a text to speech module (TTS module) (229).

The automatic voice recognition module 221 of one embodiment may convert voice input received from the user terminal 100 into text data. The natural language understanding module 223 in one embodiment may determine the user's intention using text data of voice input. For example, the natural language understanding module 223 can determine the user's intention by performing syntactic analysis or semantic analysis. The natural language understanding module 223 in one embodiment determines the meaning of a word extracted from a voice input using linguistic features (e.g., grammatical elements) of a morpheme or phrase, and matches the meaning of the identified word to the user's intention. You can determine your intention.

The planner module 225 in one embodiment may generate a plan using the intent and parameters determined by the natural language understanding module 223. According to one embodiment, the planner module 225 may determine a plurality of domains required to perform the task based on the determined intention. The planner module 225 may determine a plurality of operations included in each of the plurality of domains determined based on the intention. According to one embodiment, the planner module 225 may determine parameters required to execute the determined plurality of operations or result values output by executing the plurality of operations. The parameters and the result values may be defined as concepts of a specified type (or class). Accordingly, the plan may include a plurality of operations and a plurality of concepts determined by the user's intention. The planner module 225 may determine the relationship between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module 225 may determine the execution order of a plurality of operations determined based on the user's intention based on a plurality of concepts. In other words, the planner module 225 may determine the execution order of the plurality of operations based on the parameters required for execution of the plurality of operations and the results output by executing the plurality of operations. Accordingly, the planner module 225 may generate a plan that includes association information (eg, ontology) between a plurality of operations and a plurality of concepts. The planner module 225 can create a plan using information stored in the capsule database 230, which stores a set of relationships between concepts and operations.

The natural language generation module 227 of one embodiment may change specified information into text form. The information changed to the text form may be in the form of natural language speech. The text-to-speech conversion module 229 in one embodiment can change information in text form into information in voice form.

According to one embodiment, some or all of the functions of the natural language platform 220 may be implemented in the user terminal 100.

The capsule database 230 may store information about the relationship between a plurality of concepts and operations corresponding to a plurality of domains. A capsule according to one embodiment may include a plurality of action objects (action objects or action information) and concept objects (concept objects or concept information) included in the plan. According to one embodiment, the capsule database 230 may store a plurality of capsules in the form of CAN (concept action network). According to one embodiment, a plurality of capsules may be stored in a function registry included in the capsule database 230.

The capsule database 230 may include a strategy registry in which strategy information necessary for determining a plan corresponding to a voice input is stored. The strategy information may include standard information for determining one plan when there are multiple plans corresponding to voice input. According to one embodiment, the capsule database 230 may include a follow up registry in which information on follow-up actions is stored to suggest follow-up actions to the user in a specified situation. The follow-up action may include, for example, follow-up speech. According to one embodiment, the capsule database 230 may include a layout registry that stores layout information of information output through the user terminal 100. According to one embodiment, the capsule database 230 may include a vocabulary registry where vocabulary information included in capsule information is stored. According to one embodiment, the capsule database 230 may include a dialogue registry in which information about dialogue (or interaction) with a user is stored. The capsule database 230 can update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating operation objects or concept objects. The developer tool may include a vocabulary editor for updating the vocabulary. The developer tool may include a strategy editor that creates and registers a strategy for determining the plan. The developer tool may include a dialogue editor that creates a dialogue with the user. The developer tool may include a follow up editor that can edit follow-up utterances to activate follow-up goals and provide hints. The subsequent goal may be determined based on currently set goals, user preferences, or environmental conditions. In one embodiment, the capsule database 230 may also be implemented within the user terminal 100.

The execution engine 240 of one embodiment may calculate a result using the generated plan. The end user interface 250 may transmit the calculated result to the user terminal 100. Accordingly, the user terminal 100 may receive the result and provide the received result to the user. The management platform 260 of one embodiment can manage information used in the intelligent server 200. The big data platform 270 in one embodiment may collect user data. The analysis platform 280 of one embodiment may manage quality of service (QoS) of the intelligent server 200. For example, the analytics platform 280 can manage the components and processing speed (or efficiency) of the intelligent server 200.

The service server 300 in one embodiment may provide a designated service (eg, food ordering or hotel reservation) to the user terminal 100. According to one embodiment, the service server 300 may be a server operated by a third party. The service server 300 in one embodiment may provide the intelligent server 200 with information for creating a plan corresponding to the received voice input. The provided information may be stored in the capsule database 230. Additionally, the service server 300 may provide result information according to the plan to the intelligent server 200.

In the integrated intelligence system 10 described above, the user terminal 100 can provide various intelligent services to the user in response to user input. The user input may include, for example, input through a physical button, touch input, or voice input.

In one embodiment, the user terminal 100 may provide a voice recognition service through an internally stored intelligent app (or voice recognition app). In this case, for example, the user terminal 100 may recognize a user utterance or voice input received through the microphone and provide a service corresponding to the recognized voice input to the user. .

In one embodiment, the user terminal 100 may perform a designated operation alone or together with the intelligent server and/or service server based on the received voice input. For example, the user terminal 100 may run an app corresponding to a received voice input and perform a designated operation through the executed app.

In one embodiment, when the user terminal 100 provides a service together with the intelligent server 200 and/or the service server, the user terminal detects a user utterance using the microphone 120, and A signal (or voice data) corresponding to the detected user utterance may be generated. The user terminal may transmit the voice data to the intelligent server 200 using the communication interface 110.

In response to a voice input received from the user terminal 100, the intelligent server 200 according to one embodiment provides a plan for performing a task corresponding to the voice input, or an operation according to the plan. can produce results. The plan may include, for example, a plurality of operations for performing a task corresponding to a user's voice input, and a plurality of concepts related to the plurality of operations. The concept may define parameters input to the execution of the plurality of operations or result values output by the execution of the plurality of operations. The plan may include association information between a plurality of operations and a plurality of concepts.

The user terminal 100 in one embodiment may receive the response using the communication interface 110. The user terminal 100 uses the speaker 130 to output a voice signal generated inside the user terminal 100 to the outside, or uses the display 140 to output an image generated inside the user terminal 100 to the outside. It can be output as .

Figure 2 is a diagram showing how relationship information between concepts and operations is stored in a database, according to various embodiments.

The capsule database (e.g., capsule database 230) of the intelligent server 200 may store capsules in the form of a CAN (concept action network) 231. The capsule database may store operations for processing tasks corresponding to the user's voice input, and parameters necessary for the operations in the form of a CAN (concept action network) 231.

The capsule database may store a plurality of capsules (capsule(A) 230-1, capsule(B) 230-4) corresponding to each of a plurality of domains (eg, applications). According to one embodiment, one capsule (eg, capsule(A) 230-1) may correspond to one domain (eg, location (geo), application). Additionally, one capsule may be associated with at least one service provider (eg, CP 1 (230-2) or CP 2 (230-3)) to perform functions for a domain related to the capsule. According to one embodiment, one capsule may include at least one operation 232 and at least one concept 233 for performing a designated function.

The natural language platform 220 may create a plan for performing a task corresponding to the received voice input using capsules stored in the capsule database. For example, the planner module 225 of the natural language platform can create a plan using capsules stored in the capsule database. For example, plan 234 is created using the operations 4011, 4013 and concepts 4012, 4014 of capsule A 230-1 and the operations 4041 and concepts 4042 of capsule B 404. can be created.

Figure 3 is a diagram illustrating a screen in which a user terminal processes voice input received through an intelligent app according to various embodiments.

The user terminal 100 can run an intelligent app to process user input through the intelligent server 200.

According to one embodiment, on screen 310, when the user terminal 100 recognizes a designated voice input (e.g., wake up!) or receives an input through a hardware key (e.g., a dedicated hardware key), the user terminal 100 processes the voice input. You can run intelligent apps for For example, the user terminal 100 may run an intelligent app while executing a schedule app. According to one embodiment, the user terminal 100 may display an object (eg, an icon) 311 corresponding to an intelligent app on the display 140. According to one embodiment, the user terminal 100 may receive voice input through a user's utterance. For example, the user terminal 100 may receive a voice input saying “Tell me this week’s schedule!” According to one embodiment, the user terminal 100 may display a user interface (UI) 313 (e.g., input window) of an intelligent app displaying text data of a received voice input on the display.

According to one embodiment, on screen 320, the user terminal 100 may display a result corresponding to the received voice input on the display. For example, the user terminal 100 may receive a plan corresponding to the received user input and display 'this week's schedule' on the display according to the plan. FIG. 4 shows a network environment according to various embodiments. This is a block diagram of the electronic device 401 in 400. Referring to FIG. 4, in the network environment 400, the electronic device 401 communicates with the electronic device 402 through a first network 498 (e.g., a short-range wireless communication network) or a second network 499. It is possible to communicate with the electronic device 404 or the server 408 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 401 may communicate with the electronic device 404 through the server 408. According to one embodiment, the electronic device 401 includes a processor 420, a memory 430, an input device 450, an audio output device 455, a display device 460, an audio module 470, and a sensor module ( 476), interface 477, haptic module 479, camera module 480, power management module 488, battery 489, communication module 490, subscriber identification module 496, or antenna module 497. ) may include. In some embodiments, at least one of these components (eg, the display device 460 or the camera module 480) may be omitted, or one or more other components may be added to the electronic device 401. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, the sensor module 476 (e.g., a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented while being embedded in the display device 460 (e.g., a display).

Processor 420, for example, executes software (e.g., program 440) to operate at least one other component (e.g., hardware or software component) of electronic device 401 connected to processor 420. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 420 stores instructions or data received from another component (e.g., sensor module 476 or communication module 490) in volatile memory 432. The commands or data stored in the volatile memory 432 can be processed, and the resulting data can be stored in the non-volatile memory 434. According to one embodiment, the processor 420 includes a main processor 421 (e.g., a central processing unit or an application processor), and an auxiliary processor 423 (e.g., a graphics processing unit, an image signal processor) that can operate independently or together with the main processor 421. , sensor hub processor, or communication processor). Additionally or alternatively, the auxiliary processor 423 may be set to use less power than the main processor 421 or to specialize in a designated function. The auxiliary processor 423 may be implemented separately from the main processor 421 or as part of it.

The auxiliary processor 423 may, for example, act on behalf of the main processor 421 while the main processor 421 is in an inactive (e.g., sleep) state, or while the main processor 421 is in an active (e.g., application execution) state. ), together with the main processor 421, at least one of the components of the electronic device 401 (e.g., the display device 460, the sensor module 476, or the communication module 490) At least some of the functions or states related to can be controlled. According to one embodiment, coprocessor 423 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 480 or communication module 490). there is.

The memory 430 may store various data used by at least one component (eg, the processor 420 or the sensor module 476) of the electronic device 401. Data may include, for example, input data or output data for software (e.g., program 440) and instructions related thereto. Memory 430 may include volatile memory 432 or non-volatile memory 434.

The program 440 may be stored as software in the memory 430 and may include, for example, an operating system 442, middleware 444, or application 446.

The input device 450 may receive commands or data to be used in a component of the electronic device 401 (e.g., the processor 420) from outside the electronic device 401 (e.g., a user). The input device 450 may include, for example, a microphone, mouse, keyboard, or digital pen (eg, stylus pen).

The sound output device 455 may output sound signals to the outside of the electronic device 401. The sound output device 455 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display device 460 can visually provide information to the outside of the electronic device 401 (eg, a user). The display device 460 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display device 460 may include touch circuitry configured to detect a touch, or a sensor circuit configured to measure the intensity of force generated by the touch (e.g., a pressure sensor). there is.

The audio module 470 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 470 acquires sound through the input device 450, the sound output device 455, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 401). Sound may be output through an electronic device 402 (e.g., speaker or headphone).

The sensor module 476 detects the operating state (e.g., power or temperature) of the electronic device 401 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 476 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 477 may support one or more designated protocols that can be used to connect the electronic device 401 directly or wirelessly with an external electronic device (e.g., the electronic device 402). According to one embodiment, the interface 477 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 478 may include a connector through which the electronic device 401 can be physically connected to an external electronic device (eg, the electronic device 402). According to one embodiment, the connection terminal 478 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 479 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 479 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 480 can capture still images and moving images. According to one embodiment, the camera module 480 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 488 can manage power supplied to the electronic device 401. According to one embodiment, the power management module 388 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 489 may supply power to at least one component of the electronic device 401. According to one embodiment, the battery 489 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 490 provides a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 401 and an external electronic device (e.g., the electronic device 402, the electronic device 404, or the server 408). It can support establishment and communication through established communication channels. Communication module 490 operates independently of processor 420 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 490 is a wireless communication module 492 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 494 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 498 (e.g., a short-range communication network such as Bluetooth, WiFi direct, or IrDA (infrared data association)) or a second network 499 (e.g., a cellular network, the Internet, or It can communicate with external electronic devices through a computer network (e.g., a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 492 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 496 within a communication network such as the first network 498 or the second network 499. The electronic device 401 can be confirmed and authenticated.

The antenna module 497 may transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module may include one antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 497 may include a plurality of antennas. In this case, at least one antenna suitable for the communication method used in the communication network, such as the first network 498 or the second network 499, is connected to the plurality of antennas by, for example, the communication module 490. can be selected Signals or power may be transmitted or received between the communication module 490 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, RFIC) in addition to the radiator may be additionally formed as part of the antenna module 497.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 401 and the external electronic device 404 through the server 408 connected to the second network 499. Each of the electronic devices 402 and 404 may be the same or different type of device from the electronic device 401. According to one embodiment, all or part of the operations performed in the electronic device 401 may be executed in one or more of the external electronic devices 402, 404, or 408. For example, when the electronic device 401 must perform a function or service automatically or in response to a request from a user or another device, the electronic device 401 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 401. The electronic device 401 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology. This can be used.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. In this document, “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “A. Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” When mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 436 or external memory 438) that can be read by a machine (e.g., electronic device 401). It may be implemented as software (e.g., program 440) including these. For example, a processor (e.g., processor 420) of a device (e.g., electronic device 401) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. Device-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is stored semi-permanently in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smartphones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single entity or a plurality of entities. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

FIG. 5 is a diagram for explaining the structure of an electronic device and system according to various embodiments. Electronic devices and systems according to various embodiments may provide utterance-related services to a user in response to a voice signal containing the user's utterance. Electronic devices and systems can provide personalized services to each of a plurality of users.

Users can access the system using electronic devices such as smart phones, smart pads, personal digital assistance (PDA), laptops, and desktops. Referring to FIG. 5 , a first user 510 and a first electronic device 520 corresponding to the first user 510 are shown. The first electronic device 520 may include a display 521, a microphone 522, and a speaker 523 to provide a user interface (UI) to the user. According to some embodiments, a touch panel for receiving touch input may be disposed on the display 521. The first electronic device 520 may correspond to the user terminal 100 of FIG. 1 . Each of the display 521, microphone 522, processor 524, memory 525, and communication circuit 526 of the first electronic device 520 is the display 140, microphone 120, and processor of FIG. 1. It may correspond to each of 160, memory 150, and communication interface 110.

The first electronic device 520 may include at least one processor 524. The first electronic device 520 may include a memory 525 that stores at least one instruction related to a user interface. The processor 524 includes a means (e.g., Integrated Circuit (IC), Very Large Scale Integration (VLSI), Arithmetic Logic Unit (ALU), or Field Programmable Gate Array (FPGA) for executing functions corresponding to the instructions. It may contain more than one. Additionally, the processor 524 may include a memory (e.g., cache memory) that at least temporarily stores instructions stored in the memory 525 and data obtained by executing functions corresponding to the instructions. . The processor 524 may execute at least one instruction stored in the memory 525 to create the user interface or perform a function related to a user's input corresponding to the generated user interface.

Through the user interface, interaction may occur between the first user 510 and the electronic device according to various embodiments. The interaction may be performed using the display 521, microphone 522, speaker 523, and other input means that may be included in or connected to the first electronic device 520 (e.g., a joy stick, of the electronic device 520). It can be generated by a physical button coupled to the housing, VR device). From the perspective of the first electronic device 520, the interaction includes output of an image signal through the display 521, output of an audio signal through the speaker 523, and output of a touch signal by the touch sensor on the display 521. It may include the input of a voice signal through the input and microphone 522.

The voice signal input through the microphone 522 may include the speech of the first user 510. The remark may include one or more words included in the native language of the first user 510. When the voice signal includes a plurality of words, the order of the plurality of words may correspond to the order in which they are used in a conversation between the first user 510 and another person. The remark may be a remark based on the natural language of the first user 510.

In response to receiving a voice signal through the microphone 522, the first electronic device 520 may execute a function corresponding to the voice signal. The function may include providing a speech response system service to the first user 510 in response to a command based on the natural language of the first user 510 included in the voice signal. According to some embodiments, the first electronic device 520 may independently execute a function corresponding to a voice signal.

According to various embodiments, a system 530 is provided that is connected to a plurality of electronic devices including a first electronic device 520 and recognizes a user's speech corresponding to each of the plurality of electronic devices. It can be. Recognizing a user's speech means generating a digital electrical signal that converts the user's speech included in the voice signal into a form that can be analyzed by the electronic device or system 530 (e.g., text format). . The system 530 may recognize a voice signal obtained from the first user 510 and generate a text signal corresponding to the voice signal. The text signal can be used to provide various services related to the speech response system to the first user 510.

Referring to FIG. 5 , the first electronic device 520 may be connected to the system 530 using a communicator (eg, communication module, communication interface, communication circuitry) 526. System 530 may correspond to intelligent server 225 of FIG. 1 . The communicator 526 may be connected to a wireless network (e.g., a network based on at least one of Bluetooth, Near Field Communication (NFC), Wireless Fidelity (WiFi), and Long Term Evolution (LTE)) or a wired network (e.g., Ethernet, One component (e.g., communication chip, antenna, LAN port, optical port) for accessing a network based on at least one of LAN (Local Area Network), WAN (Wide Area Network), and xDSL (Digital Subscriber Line) It may include more. The display 521, microphone 522, speaker 523, processor 524, memory 525, and communicator 526 included in the first electronic device 520 communicate with each other using a communication bus. Can be operationally linked.

In response to receiving a voice signal through the microphone 522, the first electronic device 520 may transmit the received voice signal to the system 530. For example, the voice signal may be included in one or more packets included in the wireless signal. The wireless signal may be transmitted toward system 530 via communicator 526. System 530 may include a communication interface 531 for communicating with communicators included in a plurality of electronic devices, such as communicator 526 .

System 530 may include at least one processor 532. In response to the voice signal received through the communication interface 531, the processor 532 has a function of identifying a command based on the natural language of the first user 510 from the voice signal, and a service of a speech response system in response to the command. At least one of a function that executes and a function that provides a result of executing a service through the user interface of the first electronic device 520 may be executed. System 530 may include memory 536 that stores at least one instruction for executing at least one of the above functions. The processor 532 may execute at least one of the above functions using instructions stored in the memory 546.

Referring to FIG. 5, at least some of the data stored in the memory 546 may be related to a plurality of databases. The processor 532 may manage data stored in the memory 546 based on a plurality of databases.

In order to perform the function of identifying a command based on the natural language of the first user 510 from the voice signal, the processor 532 of the system 530 includes at least one speech recognition database ( speech recognition database) (536) can be used. The text signal generated in response to the voice signal may be related to the natural language included in the voice signal (eg, a speech input by the first user 510 toward the microphone 522).

The speech recognition database 536 includes voice signals collected from a plurality of users using a speech response system, a result of identifying natural language included in the voice signal (i.e., text signal), and conversion between the voice signal and the text signal. It may contain the necessary information. As information necessary for conversion between the voice signal and the text signal, the speech recognition database 536 may include information related to an acoustic model and a language model.

Acoustic models and language models refer to models that recognize speech signals based on GMM (Gaussian Mixture Model), DNN (Deep Neural Network), or BLSTM (Bidirectional Long Short Term Memory). The acoustic model is used to recognize voice signals in units of phonemes based on features extracted from the voice signals. The speech signal recognition speech response system can estimate the words represented by the speech signal based on the results of recognizing the speech signal acquired by the acoustic model in units of phonemes. Language models are used to obtain probability information based on connection relationships between words. The language model provides probability information about the next word that will be linked to the word entered into the language model. For example, when the word “this” is input to the language model, the language model may provide information about the probability that “is” or “was” will be connected to “this”. According to one embodiment, the speech signal recognition speech response system may select the connection relationship between words with the highest probability based on probability information provided by a language model and output the selection result as a speech recognition result.

The acoustic model and language model may be constructed using a neural network. A neural network is a recognition model implemented in software or hardware that imitates the computational power of a biological speech response system using a large number of artificial neurons (or nodes). A neural network can perform human cognitive functions or learning processes through artificial neurons. A neural network may include multiple layers. For example, a neural network may include an input layer, one or more hidden layers, and an output layer. The input layer can receive input data for training of the neural network and transmit it to the hidden layer, and the output layer can generate output data of the neural network based on signals received from nodes of the hidden layer.

One or more hidden layers may be located between the input layer and the output layer, and may convert the input data passed through the input layer into a value that is easy to predict. Nodes included in the input layer and one or more hidden layers may be connected to each other through connection lines with connection weights, and nodes included in the hidden layer and output layer may also be connected to each other through connection lines with connection weights.

The input layer, one or more hidden layers, and the output layer may include multiple nodes. A hidden layer may be a convolutional filter or a fully connected layer in a convolutional neural network (CNN), or may be various types of filters or layers grouped based on special functions or characteristics. A recurrent neural network (RNN), in which the output value of the hidden layer is re-input to the hidden layer at the current time, can be used for acoustic models and language models.

Among neural networks, a neural network that includes multiple hidden layers is called a deep neural network. Training a deep neural network is called deep learning. Among the nodes of a neural network, nodes included in the hidden layer are called hidden nodes. The speech recognition database 536 may include information related to an acoustic model, a language model, and a neural network related to each model (eg, the connection weight, properties of nodes included in the neural network).

In response to receiving the speech signal obtained from the first user 510 through the communication interface 531, the processor 532, based on the acoustic model and language model generated from the speech recognition database 536, A text signal corresponding to the received voice signal can be generated. The text signal may include a plurality of words included in the first user 510's speech. Based on the type or order of a plurality of words included in the text signal, the processor 532 may identify a command based on the natural language of the first user 510 included in the voice signal.

To execute the service of the speech response system corresponding to the identified command, the processor 532 of the system 530 may use the CAN database 535 associated with the Concept Action Network (CAN). The processor 532 identifies concepts and actions corresponding to the command using the CAN database 535, and the operation of connecting the identified concepts and actions is based on the operation related to the concept action network 231 of FIG. 2. It can be done.

In response to the plan corresponding to the command being created by connecting the concept and action, the processor 532 may perform an operation based on the plan created using the execution engine 240 of FIG. 1. . The operation may include outputting the recognized text message to the first user 510 through the first electronic device 520. The operation may be an operation to control parameters of the first electronic device 520 (for example, the volume of the speaker 523, the brightness of the display 521, and whether the first electronic device 520 is locked). The operation may be an operation of executing an application (eg, a photo application, a weather application, or other third-party application) stored in the first electronic device 520. The operation may be an operation of providing content to the first user 510.

System 530 may be connected to one or more content providing devices operated by a content providing entity through a wireless network or a wired network. Referring to FIG. 5 , a first content providing device 540 to a third content providing device connected to the system 530 are shown. The first content providing device 540 may receive a request related to providing content from the system 530 through the communicator 541.

For example, when the first user 510 inputs a statement containing a command requesting content related to the first content providing device 540 into the microphone 522, the system 530 The command can be identified by recognizing the voice signal received from 520). In response to an identified command, system 530 may connect concepts and actions to create a plan corresponding to the command. While executing the created plan, the system 530 may request the first content providing device 540 to provide content.

The content providing device may be connected to a communicator and include a processor that executes a function corresponding to a request received from the system. Referring to FIG. 5, in response to receiving the request through the communicator 541, the processor 542 of the first content providing device 540 sends a message to the user from among the data stored in the first content providing device 540. You can identify the data to be provided. The first content providing device 540 may include a memory 543 for storing instructions related to content search and a content database 544 that manages the content. Processor 542 may transmit content including identified data from content database 544 to system 530 .

Content provided by the content providing device may include a result of searching or selecting an item among a plurality of items in response to a command identified by the user. An item is data provided to a user and may be data corresponding to an object that the user wishes to obtain from a content provider. Items can correspond to rows or instances in the database. For example, when a user uses a hotel reservation service, the item may correspond to a hotel. As another example, when a user uses an internet shopping service, the item may correspond to a product. For example, when the first content providing device 540 provides content related to a restaurant, the content database 544 may store a plurality of items corresponding to each of a plurality of restaurants.

An item may contain one or more objects. Objects can correspond to fields or columns in a database. For example, when a user uses a hotel reservation service, the item may include objects such as a hotel location and a hotel phone number. As another example, when a user uses an internet shopping service, the item may include objects such as product price and product seller. When the first content providing device 540 provides content related to a restaurant, the item includes objects such as the location of the restaurant, the type of restaurant (Korean, Western, Chinese), the restaurant's menu, and the user's evaluation of the restaurant. can do.

The object may be a standard for determining the order of items in content provided to the system 530. If the command of the first user 510 identified from the voice signal is a command related to the first content providing device 540 (e.g., “Search for a restaurant nearby”), the system 530 A request to provide search results for adjacent restaurants to 510 may be transmitted to the first content providing device 540 . In this case, the content received by the system 530 may include a plurality of items corresponding to each of the restaurants adjacent to the first user 510. The order of a plurality of items included in the content may be related to the object (ie, the location of the restaurant) included in the command. For example, an item corresponding to a restaurant located closest to the first user 510 may be placed at the highest level of the content.

In response to execution of the service of the speech response system corresponding to the identified command, the processor 532 may provide the result of executing the service or the execution process of the service through the user interface of the first electronic device 520. . The processor 532 may transmit the result of executing the service or the execution process of the service to the first electronic device 520. For example, the processor 532 may transmit content received from the first content providing device 540 to the first electronic device 520. The first electronic device 520 may output the received content on the display 521 according to the layout of the user interface.

Since the results provided by at least one of the plurality of content providing devices are difficult to satisfy all users, the user may use the electronic device or system 530 to indicate the user's intention, preference, or purpose in order to obtain one or more desired items. You can enter . System 530 may transmit a request to search, select, or sort items based on the user's input intention, preference, or purpose to at least one of the plurality of content providing devices. Preferences can be expressed as the user's intent, preference, or purpose. Hereinafter, operations related to embodiments will be described based on preferences, but the various embodiments are not limited thereto.

In response to a command related to searching for an item input from the user, the system 530 may transmit a request for searching, selecting, or sorting the item using preferences to at least one of the plurality of content providing devices. Preferences are data personalized to the user. Preferences may be related to criteria by which at least one of a plurality of items is selected by a user. Processor 532 of system 530 may manage the preferences using a preference database 534.

The preference may be related to an object of which the user relatively more concerned with when the user selects at least one of a plurality of items. Among the plurality of objects included in the item, the object that the user considers relatively more important is called the preference object. A plurality of items provided to the user may be sorted based on the preference object. Users can more easily identify their preferred item from a plurality of sorted items.

If the command of the first user 510 identified from the voice signal is a command related to the first content providing device 540 (e.g., “Search for a restaurant nearby”), the system 530 determines the location of the restaurant. can be identified as a preference object. A preference object can be used to align a plurality of items identified in response to the command. The preference object may be used to sort a plurality of items searched in response to another command input from the first user 510 after the above command.

FIG. 6 is a diagram conceptually illustrating hardware components or software components used by systems according to various embodiments to manage preferences. Below, with reference to FIG. 6, the operation of the system providing content to the user will be described. The hardware component or software component shown in FIG. 6 corresponds to the processor 524 of the plurality of electronic devices (e.g., the first electronic device 520) and the processor 532 of the system 530 of FIG. 5. , may correspond to at least one of an application, a thread, and a process running on at least one of the processors 524 and 530.

Referring to FIG. 6, the UI generator 610 may generate a user interface (UI) and change the user interface in response to a user's input related to the generated user interface. For example, the processor 524 of the first electronic device 520 of FIG. 5 may change the user interface in response to the input of the first user 510 by executing instructions related to the UI generator 610.

The UI generator 610 may include at least one of a preference dashboard controller 611, a result display generator 612, and a preference selector 613.

The preference selector 613 may add an interface to initiate a function for changing preferences on a user interface provided to the user. To allow a user to add or change preferences, preference selector 613 can change the operating mode of the user interface. The operation mode may include a preference change mode related to a state of adding or changing a preference and an item display mode related to a state of displaying an item on a user interface based on a preference. The operation of changing the interface related to the preference selector 613 and the operation mode of the speech response system will be described later.

The preference dashboard controller 611 may create a user interface for outputting information (eg, preference object) related to personalized preferences to the user in response to the operation mode being changed to preference change mode. The user interface created by the preference dashboard controller 611 may receive input for changing preferences from the user. In response to receiving input for changing preferences from the user, the preference dashboard controller 611 may change information related to preferences (eg, preference objects).

Information related to preferences output to the user may be generated by the user or may be generated based on tracked user activities. Hereinafter, preferences created by the user inputting one or more preference objects are referred to as user preferences. Hereinafter, preferences generated based on user activities tracked by an electronic device or system are referred to as system preferences.

The preference dashboard controller 611 can display on the user interface which preferences have been applied to the layout provided to the user. For example, the preference dashboard controller 611 can emphasize (emphasize) a preference object among a plurality of objects output on the user interface. Alternatively, the preference dashboard controller 611 may output a list of preference objects in a designated area of the user interface.

The result screen generator 612 may adjust the layout of the screen based on the preference object determined by the preference dashboard controller 611. The result screen generator 612 may generate a user interface including the results of arranging the item and the objects included in the item according to the adjusted layout.

The preference controller 620 may be connected to a Concept Action Network (CAN) 650. The CAN 650 may be related to the CAN database 535 of FIG. 5. The preference controller 620 can manage objects related to the user's preferences. The preference controller 620 includes a database related to the user's preferences (in the example of FIG. 6, the speech response system preference database 660, the user preference database 670, and the user interaction log DB 680). ) or controlling updating the database. The preference controller 620 may analyze user logs or perform learning using user logs.

The preference controller 620 may include a user preference controller 621 that performs functions related to user preferences created by directly inputting object preferences from the user. The function performed by the user preference controller 621 may include at least one of a function of adding a user preference to the user preference database 670 and a function of deleting a user preference stored in the user preference database 670. The user preference controller 621 is connected to the CAN 650 and can identify objects inside the capsule structure. The user preference controller 621 may indicate that at least one of the plurality of objects corresponds to a preference object by adding a tag to at least one of the plurality of objects.

For example, when a user selects a specific object in the preference change mode, the user preference controller 621 may display the specific object as a preference object using a tag or flag related to the selected specific object. In response to selection of a specific object, the user preference controller 621 may identify a plurality of attributes of the specific object.

Preference controller 620 may include a preference feature extractor 626 that identifies characteristics of the object's attributes. In response to selection of a specific object, when the user preference controller 621 identifies a plurality of attributes of the specific object, the preference feature extractor 626 may identify features of the identified attributes. The characteristics may be related to the properties of the object preferred by the user.

For example, when a user selects a rating, preference feature extractor 626 determines the characteristics of the rating that the user prefers (e.g., whether the user wants it to match the selected rating, or what range of ratings is above or below the rating). preference) can be identified. For example, when a user selects an image, the user's preferred image characteristics can be identified among the atmosphere of the image, the objects included in the image, and the color of the image.

The preference feature extractor 626 may request the user to select which feature he or she prefers among the plurality of identified features. The result of the user selecting a feature may be included in the user preference by the user preference controller 621. Additionally, the results may be stored in the user preference database 670.

Preference controller 620 may include a system preference generator 622 for generating system preferences. System preference generator 622 may include user log tracker 623 and preference exchanger 624.

The user log tracker 623 uses log data indicating user activity related to the user interface (for example, the user selects an item provided through the user interface) to identify a preference object among a plurality of objects. can do. User log tracker 623 may identify the log data from user interaction log database 680. The user interaction log database 680 may store signals input by the user into the electronic device in chronological order in response to the user interface. According to some embodiments, user log tracker 623 may request the user to select a preference object from among candidates for the preference object.

Preference objects identified by the user log tracker 623 may be stored in the user preference database 670. Preference objects stored in the user preference database 670 may be managed by the user preference controller 621.

The preference exchanger 624 may control the preference object identified using a specific content providing device so that it is used to sort items provided by another content providing device. For example, the preference exchanger 624 may use the object's capsule or inheritance relationship to change a preference object corresponding to a specific content providing device so that it can be utilized in an operation related to another content providing device. The changed preference object can be used when another content providing device searches for an item. When a user uses another content providing device, a changed preference object may be provided to the user.

The preference controller 620 may include a preference sorter 625 that identifies the priority of the plurality of preferences, corresponding to the plurality of preferences stored in the system preference database 660 and the user preference database 670. . The priority of the identified preference can be used to search for items by the content providing device. The priority of the identified preferences can be used to determine the layout of items to be presented to the user.

The preference sorter 625 can identify the priority using scores corresponding to each preference. To determine a score corresponding to each preference, the preference sorter 625 may use the user preference learner 630. The user preference learner 630 may identify the user's activities related to preference objects on the user interface from log data stored in the user interaction log database 680.

Based on the user's activities related to the identified preference objects, user preference learner 630 can be trained to determine the priority of preference objects. The user preference learner 630 may use the preference learning model 640 related to learning the priorities of preference objects. The preference learning model 640 can correspond to a neural network that supports deep learning. The preference learning model 640 may be personalized for each of a plurality of users by the user preference learner 630.

The preference learning model 640 can be used to classify or expand preference objects. For example, preference sorter 625 may combine preference objects and other information based on preference learning model 640. Other information bound to the preference object may include, for example, context information related to the user's activities (e.g., time information, location information, electronic device information corresponding to the electronic device used by the user, financial information, biological information, motion information, purchase information). A preference object combined with other information can be used to learn the preference learning model 640, so that the preference learning model 640 is personalized for each of a plurality of users.

Capsules implemented by the capsule developer may be stored in the CAN 650. A capsule developer may correspond to an operator of a system (e.g., system 530 in FIG. 5) or an operator of a content provision device. Capsule developers can store candidates that can be used as preference objects in CAN 650.

The preference feature extractor 626 extracts preference objects frequently used by the user based on the user's usage history (e.g., a user interaction log included in the user interaction log database 680). It can be provided to the user. The preference object provided by the preference feature extractor 626 to the user may be output within the UI (e.g., preference dashboard) of the electronic device (e.g., FIGS. 18A to 18B and FIG. 19).

Preference objects enabled or disabled by the user may be stored in the user preference database 670 by the system. When there are multiple preference objects activated by the user, the preference sorter 625 may determine the priority of each of the free objects. The model created through learning can be stored in the preference learning model 640. When the electronic device outputs a UI in response to a user's utterance, a list of items to which preference objects are applied according to priority may be output within the UI.

FIG. 7 is a flowchart illustrating an operation in which an electronic device or system aligns a plurality of items provided to a user using a preference object, according to various embodiments.

In operation 710, an electronic device according to various embodiments may provide a user interface including a plurality of items to a user. The user interface may be generated in response to a voice signal including a user's utterance. In some embodiments, the electronic device may receive the user's speech through a microphone. In response to receiving the user's utterance, in operation 710, the electronic device may provide the user with a user interface including a plurality of items. A user can input a voice signal into an electronic device included in the speech response system.

For example, referring to FIG. 5 , a first user 510 may input a voice signal into the microphone 522 of the first electronic device 520. The voice signal input to the first electronic device 420 may be recognized by the system 530 connected to the first electronic device 420.

The system may request provision of content from at least one of a plurality of content providing devices connected to the system in response to the recognized voice signal. In response to the request, the at least one content providing device may transmit content including a plurality of items to the system. The system may generate information related to a user interface to be provided to the user through an electronic device from the received content or a plurality of items included in the content. The system may transmit the generated information to an electronic device. The electronic device may output a user interface corresponding to the received information to the user.

A user can control a user interface output from an electronic device and perform operations related to a plurality of items. The user interface may include a list in which a plurality of items are arranged in a specified order. A user can scroll the list to identify a plurality of items and select at least one of the plurality of items in the list. The user interface may include an interface for removing selected items from a list (e.g., a delete button), an interface for sorting selected items into separate lists (e.g., lists related to items of interest) (e.g., a like button), and It may include at least one of an interface (eg, a navigation button) for outputting related more detailed information. In response to a user performing an action to select an interface (e.g., a gesture of touching a navigation button), the electronic device or system may execute a function related to the selected interface.

In addition to the interfaces illustrated above, the user interface may include an interface (eg, a preference selection mode button) for controlling preferences related to the user and objects included in the item. The interface can switch the operating mode of the user interface. The operation mode switched by the interface may include a preference change mode related to adding, changing, and deleting preferences, and an item display mode that changes the order or layout of a plurality of items in the user interface based on the preference.

In operation 720, the electronic device may identify whether the operation mode of the user interface has been changed to the preference change mode by the user. If the user selects an interface for controlling preferences included in the user interface, and the operation mode is not the preference change mode, the electronic device may change the operation mode to the preference change mode. In response to changing to the preference change mode, the electronic device may provide the user with an interface that allows the user to select an object related to a plurality of items.

According to various embodiments, depending on whether the operation mode is a preference change mode, the operation performed by the electronic device in response to the selection of an object within the user interface may vary. If the operation mode is not a preference change mode, the electronic device may perform an operation related to the item corresponding to the selected object (for example, an operation of outputting detailed information of the item) in response to the selection of an object by the user. there is. When the operation mode is a preference change mode, the electronic device may perform an operation to determine the selected object as a preference object in response to the selection of an object by the user.

In operation 730, in response to selection of an object by the user in the preference change mode, the electronic device may identify the preference object selected by the user. As described above, the electronic device may determine the object selected by the user within the user interface as the preference object. The result of the electronic device determining the preference object may be transmitted to the system. The system stores the results in at least one of a user interaction log database (e.g., user interaction log database 680 in FIG. 6) and a user preference database (e.g., user preference database 670 in FIG. 6). You can save it.

At operation 740, in response to identifying the preference object selected by the user, the system may identify the user's preferred characteristics in the identified preference object. The system can identify the user's preferred attribute among the object's attributes. For example, if the object selected by the user has a score of item made by public, the attribute of the object may be a value corresponding to the object selected by the user (e.g., 4.5). there is. In the example above, the system may determine from the properties of the identified object that the user prefers an item with a rating of 4.5. In the above example, the characteristic that the system identifies is the value of the object. Depending on the type of object, features can be type or text. In response to the user identifying preferred features, the system may associate tags associated with the identified features to the preference object.

At operation 750, the system may determine the score of the preference object selected by the user. When there are a plurality of preference objects related to a plurality of items, the score can be used to determine the priority of the plurality of preference objects. In response to the identification of the preference object identified in operation 730 and the preference object previously stored in the system, the system may determine the correlation or importance between the plurality of identified preference objects. Determining the correlation or importance between the plurality of identified preference objects may be performed by the system's preference aligner (e.g., preference aligner 625 in FIG. 6).

According to some embodiments, when determining correlation or importance between a plurality of identified preference objects, the system may utilize specified rules. For example, the system assigns the highest score to the preference object most recently specified by the user and the lowest score to the preference object most recently specified by the user, based on rules related to the temporal order in which the preference objects are identified. can be assigned. Based on rules related to the frequency with which preference objects are used, the system assigns relatively high scores to preference objects that are frequently used in sorting multiple items and assigns relatively high scores to preference objects that are used relatively infrequently in sorting multiple items. A relatively low score can be assigned.

When a plurality of scores are assigned to one preference object according to a plurality of rules, a weight corresponding to each of the plurality of rules may be applied to the plurality of scores, and the combination of the plurality of scores to which the weights are applied may be applied to the preference object. It may be determined by the final score. The system may associate the order of the final score of each of the plurality of preference objects and the priority of the plurality of preference objects.

According to some embodiments, when determining correlation or importance between a plurality of identified preference objects, the system may utilize a neural network or deep learning. For example, referring to Figure 6, a user preference learner 630 and a preference learning model 640 may be related to the neural network or deep learning. In response to identification of a preference object selected by a user, the system may learn a correlation between the identified preference object and a previously selected preference object from user interaction log data associated with the identified preference object.

For example, in response to a user selecting a first object as a preference object, the system can learn how the selected first object and previously stored preference objects were used. The system can expand information related to preferences using context information, location information, device information, and financial information, as well as user interaction log data. Information related to extended preferences may be related to training and classification information by deep learning, and may be used to create a personalized model corresponding to the user.

In operation 760, the electronic device or system may change the user interface provided to the user based on the determined score. The system may determine the order of the plurality of items within the user interface and the layout of one or more objects associated with each of the plurality of items on the user interface based on the determined score. Information related to the order and layout of the determined plurality of items may be transmitted to the electronic device.

The electronic device may change the order of a plurality of items or change the position of an object within the user interface according to the transmitted information. The order of the plurality of items may be determined based on the identified preference object and the identified characteristic. The position of the object may be changed based on the preference object selected by the user and the scores of each of the plurality of preference objects.

8A to 8C are exemplary diagrams for explaining a user interface provided to a user by an electronic device according to various embodiments.

Referring to FIGS. 8A to 8C, the user interface may be provided to the user through the electronic device 810 included in the speech response system. The electronic device 810 may correspond to the first electronic device 520 of FIG. 5 . The user interface may include a voice signal input from the user and one or more items searched in response to remarks included in the voice signal. 8A to 8C, it is assumed that the speech response system searches for a plurality of restaurants in response to a statement related to restaurant search input from the user (for example, “Hey Bixby, tell me a restaurant nearby”). Each of the plurality of items provided to the user in the home may correspond to the plurality of searched restaurants.

FIG. 8A is a diagram illustrating an example of outputting results of searching a plurality of restaurants through the electronic device 810. Referring to FIG. 8A , in the user interface displayed on the display 820 of the electronic device 810, the name of the content provider used to search for a plurality of restaurants may be displayed at least in part. A plurality of restaurants may be displayed in at least a portion of the user interface (eg, a portion excluding a portion where the name of the content provider is displayed) based on the first order.

Objects related to each of the plurality of restaurants may be displayed on at least a portion of the user interface. Referring to FIG. 8A, objects related to each of a plurality of restaurants include the name of the restaurant, an image related to the restaurant, the rating of the restaurant, the address of the restaurant, the type of restaurant (cafe, Korean, pub, Japanese, Chinese, Western), The distance between the user and the restaurant calculated from the address of the restaurant, the number of views for the restaurant, and the number of reviews for the restaurant may be displayed on the user interface. The objects may be placed on the user interface based on the layout of the objects within the first order or item (restaurant). The layout of objects in each of the plurality of items may match each other. Referring to FIG. 8A, the arrangement of objects corresponding to restaurant A and the arrangement of objects corresponding to ribs B may coincide with each other.

The user may select some of the plurality of restaurants or identify the plurality of restaurants in a first order by performing a gesture on at least a portion of the user interface where the plurality of restaurants are displayed. An operation performed by the electronic device 810 or system in response to a gesture may vary depending on the current operating mode of the user interface. The operation mode may include an item display mode and a preference change mode. In response to a gesture on the user interface performed by the user, the electronic device 810 may perform one of a plurality of operations according to the current operation mode among the plurality of operation modes of the user interface.

When the current operation mode is the item display mode, which is a mode for displaying items according to preset preferences, in response to the selection of an object by the user on the user interface, the electronic device 810 provides detailed information on the item related to the selected object. Can be printed. In the example of FIG. 8A , in the item display mode, when the user touches the restaurant name (object) of Restaurant A, the electronic device 810 may output detailed information about Restaurant A on the user interface. Detailed information about Restaurant A may include not only the objects displayed in FIG. 8A but also all objects related to Restaurant A stored in the content provider.

When the current operation mode is a preference change mode, which is a mode for controlling preferences, in response to the selection of an object by the user on the user interface, the electronic device 810 adds, deletes, or changes the preference object based on the selected object. can be performed. Switching between the item display mode and the preference change mode may be performed by an interface associated with switching the operation mode included in at least a portion of the user interface.

For example, the user may touch the menu button 830 on the display 820, touch the submenu 840 output on the display 820 in response to the touch of the menu button 830, and operate You can change the mode. In response to touching the submenu 840, the electronic device 810 may toggle the operation mode between the item display mode and the preference change mode. According to some embodiments, in addition to the menu button 830 or the submenu 840, the user may input a voice signal containing a statement related to a change in preference into the electronic device 810 or open the housing of the electronic device 810. You can change the operation mode by pressing the button exposed to the outside.

FIG. 8B is a diagram illustrating an example of a user interface in which the operation mode is changed to preference change mode according to a user's command. When the current operation mode is the preference change mode, in response to the selection of an object by the user on the user interface, the electronic device 810 may determine the selected object as the preference object. The preference object may include information related to the attributes or characteristics of the selected object. Information related to the attributes or characteristics of the selected object may be identified by the user's selection or analysis of the user's log data (eg, log data tracking by the user log tracker 623 in FIG. 6).

In the example of FIG. 8B , when the user touches the image object 851 related to restaurant A, the electronic device or system may determine a preference object based on the touched image object 851. The preference object may include information related to the characteristics of the image object 851. An operation in which an electronic device or system according to various embodiments identifies characteristics of the image object 851 will be described later.

When the user touches the rating object 852 related to B ribs, the electronic device or system may determine the rating object 852 as a preference object. The preference object may include information related to the value (4.4) of the rating object 852 selected by the user.

When the user touches the name object 853 of C beer, the electronic device or system may determine the name object 853 as a preference object. The preference object may include information related to the string (C beer) of the name object 853 touched by the user.

When the user touches the distance object 854 of the D rib, the electronic device or system may determine the distance object 854 as a preference object. The preference object may include information related to the value (2.5 km) of the distance object 854.

In preference changing mode, the user interface may include an interface to escape from the preference adjusting mode. For example, the user may touch the preference change completion button 841 to change the operation mode from the preference change mode to the remaining mode (eg, item display mode). In response to a change in the operation mode from the preference changing mode to the remaining mode (in response to transition of the operation mode from the preference adjusting mode to other mode), the first order in which the plurality of items are arranged on the user interface is the first order. It can be changed to a second order that is distinct from . The second order may be identified based on a preference object specified by the user.

When the image object 851 is determined to be a preference object, the electronic device or system compares the characteristics of the image object included in each of the plurality of items and the characteristics of the image object 851 touched by the user, You can decide the order.

In response to determining that the rating object 852 is a preference object, the electronic device or system may compare the value of the rating object included in each of the plurality of items with the value (4.4) of the rating object 852. The second order may be determined based on the result of comparing the value of the rating object included in each of the plurality of items with the value (4.4) of the rating object 852.

In response to the name object 853 being determined to be a preference object, the second order may be changed based on the similarity of the name object 853 to the string (C beer) or whether the string is included.

In response to determining that the distance object 854 is a preference object, the second order is whether the value of the distance object included in each of the plurality of items is included in the interval related to the value (2.5 km) of the distance object 854. It may change depending on.

If there exists a preset preference object which is generated before entering the preference adjusting mode, the electronic device 810 highlights the object corresponding to the preset preference object. can do. For example, when the preset preference object corresponds to a rating object with a value of 4.4, the electronic device 810 selects a rating object 852 corresponding to the preset preference object from among a plurality of rating objects displayed on the user interface. can be emphasized. Emphasizing the rating object 852 may include changes in the color of text or images included in the rating object 852, appending of shapes or images related to the rating object 852, and blinking of the rating object 852. It may include at least one application of animation.

The user's selection of a preference object may be performed on a list of multiple items, as shown in FIG. 8B, or may also be performed on a user interface that outputs detailed information on one of the multiple items. For example, in the item display mode, when the user selects one of the plurality of items (B ribs) shown in FIG. 8A, the electronic device 810 displays detailed information related to the selected item (i.e., B ribs and All related objects) can be output. While detailed information related to the selected item is displayed, the user can change the operation mode to preference change mode by touching the submenu 840.

FIG. 8C is an example diagram for explaining an operation in which a preference object is selected from a user interface that outputs detailed information on one (B rib) of a plurality of items shown in FIG. 8A. Referring to FIG. 8C, when the user selects one of the plurality of items (B ribs), the electronic device 810 displays the item selected by the user and the list instead of outputting a list based on the first order of the plurality of items. All related objects can be output. For example, an interface that includes all objects related to an item selected by the user may obscure at least a portion of the list based on the first order of the plurality of items.

Referring to FIG. 8C, in the preference change mode, at least one of all objects related to the item selected by the user may be selected by the user. The electronic device may determine at least one object selected by the user as a preference object. Referring to FIG. 8C, a view count object 855, a parking information object 856, and a reviewer ID object 857 may be output as objects related to the item selected by the user. The electronic device may determine at least one of the objects output in response to the user's selection as a preference object.

If the object determined as the preference object is an object that is not output on the list of a plurality of items, the electronic device may change the user interface or layout so that the object determined as the preference object is output on the list of the plurality of items. For example, referring to FIGS. 8B to 8C, the parking information object 856 and the reviewer ID object 857 are objects that are not displayed on the list of a plurality of items. When the user selects one of the parking information object 856 and the reviewer ID object 857, the electronic device may determine the selected object as a preference object, and display the determined preference object on a list of a plurality of items. You can change the layout of the list.

In the preference change mode, when the object selected by the user includes a plurality of properties or features, the electronic device or system may request the user to select at least one of the plurality of properties or features. The electronic device or system provides a user interface for user selection based on the object's properties (type, value, string, GPS coordinates, image) and characteristics (type, value, string, GPS coordinates, image of the selected object). It can be added to the table.

9A to 9C are exemplary diagrams illustrating an operation of an electronic device requesting a user to select at least one of a plurality of attributes or features of an object selected by the user, according to various embodiments. Hereinafter, in the user interface of the preference change mode of FIG. 8B, an operation in which the electronic device or system requests the user to select at least one of a plurality of attributes or characteristics of the object selected by the user will be described.

Referring to FIG. 9A , when a user touches the rating object 852 related to B ribs, the electronic device or system can identify the attributes and characteristics of the rating object 852. The attributes of the rating object 852 are integer data between 0 and 5, and the characteristics of the rating object 852 may correspond to the value (4.4) of the rating object 852. Preference objects may be associated with attributes and characteristics identified from rating object 852.

The electronic device or system may request the user to select at least one of a plurality of attributes or characteristics identified from the rating object 852. For example, the electronic device may ask the user whether it prefers to match the value 4.4 of the rating object 852, or whether it prefers a value greater than or equal to the value 4.4 of the rating object 852. . Referring to FIG. 9A, on the user interface displayed on the display 920 of the electronic device 910, an interface 910 requesting selection of at least one of a plurality of features related to the rating object 852 may be output. there is. The interface 910 may be output at a location adjacent to the rating object 852 selected by the user.

The preference object may be determined based on features selected by the user in the interface 910. For example, if the user selects 'only 4.4', the electronic device may determine that the user prefers the value (4.4) of the rating object 852. A preference object may be created corresponding to the value (4.4) of the rating object 852. When the user touches the preference change completion button 841 and the operation mode changes from the preference change mode to the item display mode, the electronic device displays a plurality of items depending on whether it matches the value (4.4) of the rating object 852. You can change their order. For example, at least one item with a rating matching the value (4.4) of the rating object 852 may have a higher priority than the remaining items.

As another example, if the user selects '<= 4.4', the electronic device may determine that the user prefers ratings from 0 to 4.4. The preference object may include information about the interval (rating from 0 to 4.4) related to the rating object 852. When the user touches the preference change completion button 841 and the operation mode changes from the preference change mode to the item display mode, the electronic device displays a plurality of values based on the section (rating from 0 to 4.4) corresponding to the preference object. You can change the order of items. For example, a plurality of items related to the section (referring to FIG. 9A, B ribs, C beer, D ribs) may have a higher priority than the remaining items (Restaurant A).

As another example, if the user selects '>= 4.4', the electronic device may determine that the user prefers a rating from 4.4 to 5. The preference object may include information about the section (rating from 4.4 to 5) related to the rating object 852. When the user touches the preference change completion button 841 and the operation mode changes from the preference change mode to the item display mode, the electronic device orders the plurality of items based on the section (rating from 4.4 to 5). You can change it. For example, a plurality of items related to the section (referring to FIG. 9A, restaurant A and ribs B) may have a higher priority than the remaining items.

Referring to FIG. 9B, when the user touches the image object 920 related to B ribs, the electronic device can identify the properties and characteristics of the image object 920. The image object 920 is image or video data related to the B ribs and may include one or more subjects related to the B ribs. The electronic device may identify characteristics of the image object 920 (eg, a plurality of subjects included in the image object 920, a location where the image object 920 was photographed). The electronic device may request the user to select a preferred subject or feature from among a plurality of subjects included in the image object 920 or a plurality of features of the image object 920.

Referring to FIG. 9B , the request may be output on the display 920 of the electronic device 910 in the form of an interface 930. The interface 930 may include a list related to the type or hashtag of the subject included in the image object 920. The order of the plurality of features of the image object 920 within the interface 930 may be changed depending on accuracy.

The preference object may be determined based on any one of the characteristics of the image object 920 selected through the interface 930. For example, when the user selects 'Korean food', the preference object may include the image object 920 and information related to the feature ('Korean food') selected by the user. In response to the user touching the preference change completion button 841, the speech response system changes the operation mode, giving relatively high priority to the item containing the image object related to the feature selected by the user ('Korean food'). Can be assigned. The order of a plurality of items in the user interface may be changed based on the assigned priority.

Referring to FIG. 9C , when a user touches the address object 940 of restaurant A, the electronic device can identify the properties and characteristics of the address object 940. The address object 940 may include data related to the GPS coordinates or address of Restaurant A. Based on the characteristics of the address object 940, the electronic device may request the user to select the user's preferred characteristics in the address object 940.

Referring to FIG. 9C , the request may be output on the display 920 of the electronic device 910 in the form of an interface 950. Interface 950 may include a map corresponding to address object 940. When a user selects a preferred region on the interface 950, the preference object may include information related to the selected region.

When the user selects buildings A and C on the interface 950 and then touches the preference change completion button 841, the electronic device selects the items present in buildings A and C (A restaurant, C beer, D ribs) can be assigned a relatively high priority. The order of the plurality of items may be changed based on the assigned priority. For example, items that exist in Buildings A and Building C (Restaurant A, Beer C, Galbi D) may be placed on the user interface with priority over the remaining items.

FIGS. 10A and 10B are exemplary diagrams illustrating an operation in which an electronic device changes the order in which a plurality of items are arranged within the display 820 using a preference object created based on a user input, according to various embodiments. am. Hereinafter, an operation in which an electronic device or system changes the order of a plurality of items based on an object selected by the user in the user interface of the preference change mode of FIG. 8B will be described.

Referring to FIG. 10A , in the preference change mode, in response to selection of the distance object 1010 of restaurant A by the user, the electronic device may determine the distance object 1010 as the preference object. Since the distance object 1010 is a value representing the distance between the user and the item (Restaurant A), the preference object may include information related to the distance between the user and the item.

The user may touch the distance object 1010 and then touch the preference change completion button 841. In response to touching the preference change completion button 841, the operation mode of the user interface may be changed from the preference change mode to the item display mode.

Referring to FIG. 10B, after the operation mode is changed from the preference change mode to the item display mode, the order of the plurality of items may be changed based on the preference object. Since the preference object contains information related to the distance between the user and the item, items that are close to the user may have a relatively high priority. The user interface may output a preference object related to the order of a plurality of items to at least a portion 1020 of the display 820. According to various embodiments, changing the order of a plurality of items based on the preference object may be performed even before changing to the item display mode (eg, simultaneously with touching the distance object 1010).

FIG. 11 is an example diagram illustrating the structure of a preference object 1110 managed by an electronic device or system according to various embodiments.

The preference object 1110 may be information in which the object 1120 and the feature 1130 corresponding to the object 1120 are matched. Object 1120 may be selected by the user based on the operations described in FIGS. 8B to 10A. Alternatively, the object 1120 may be identified based on the results of tracking log data.

Object 1120 may have values in various formats depending on its properties. Referring to FIG. 11, object 1120 is an object included in items related to a restaurant and may represent the style of the restaurant. In this case, the object 1120 may have one of a plurality of values related to the style of the restaurant (e.g., “southeastAsian”, “American dining”, “korea traditional”). The object selected by the user is a specific When it has a value, the characteristic 1130 of the preference object 1110 can be determined by the specific value.

The electronic device or system may request the user to select a value preferred by the user among a plurality of values that the object selected by the user may have, referring to the operations described in FIGS. 9A to 9C. In this case, the feature 1130 of the preference object 1110 may be determined by a value selected by the user. The preference object 1110 may be stored in a database (eg, the user preference database 670 or the system preference database 660 in FIG. 6) that manages the preference object 1110 in the system.

FIG. 12 is a signal flow diagram illustrating interaction between electronic devices and systems according to various embodiments. According to some embodiments, electronic devices and systems may provide the user 1210 with a service corresponding to the user's 1210 speech 1231. Electronic devices such as the electronic device 810, the system 530, and the content providing device 1220 may be connected to each other through a wireless network or a wired network. System 530 may correspond to system 530 of FIG. 5 . The content providing device 1220 may correspond to any one of the plurality of content providing devices shown in FIG. 5 . The electronic device 810 may correspond to one of the plurality of electronic devices in FIG. 5 .

Referring to FIG. 12 , the user 1210 may input an utterance 1231 into the electronic device 810 . The utterance 1231 may include a command for executing at least some of the functions of the electronic device 810 or the system. For example, utterance 1231 may include a wake-up command. The wake-up command may be a command that switches the state of the electronic device 810 from an inactive state to an active state. The inactive state may indicate a state in which at least one of the functions of the electronic device 810 or the components of the electronic device 810 is inactivated. The wake-up command may indicate the initiation of interaction between the user and the electronic device 810. The wake-up command may be a voice input used to activate the voice recognition function of the electronic device 810 and the system 530.

The wake-up command may be configured with at least one designated or specified keyword, such as “hey bixby.” The wake-up command may be a voice input that requires identification of whether it corresponds to the at least one keyword. The wake-up command may be a voice input that does not require natural language processing or may require a limited level of natural language processing.

Utterance 1231 may further include a voice command subsequent to the wake-up command. The voice command may be a command requesting the provision of a plurality of items from the electronic device 810 or the system 430, such as “find hotels with 3 stars.” The voice command may be a command based on natural language used by the user 1210. The voice command included in the utterance 1231 may be recognized by the electronic device 810 or the system 530. Referring to FIG. 12 , the electronic device 810 may transmit a voice signal 1232 corresponding to an utterance 1231 to the system 530.

System 530 may include components for recognizing speech signals 1232 (e.g., speech recognition database 536 of FIG. 5). System 530 can identify a text signal that corresponds to voice signal 1232. System 530 can identify, from the identified text signal, the spoken command following the wake-up command. System 530 may determine a plan of action to be performed in response to the identified voice command. For example, system 530 may determine the plan of action based on CAN database 535 of FIG. 5.

When the voice command is a command requesting provision of a plurality of items from the speech response system, the system 530 communicates with the content providing device 1220 to perform a search for the plurality of items to identify the plurality of items. You can. Referring to FIG. 12, the system 530 may transmit a request signal 1233 requesting provision of a plurality of items corresponding to a voice command to the content providing device 1220. If there is a preference related to a plurality of items and created before input of the utterance 1231, the request signal 1233 may include information related to the preference.

The content providing device 1220 may search a database in which N items are stored (e.g., the content database 544 in FIG. 5) and identify n (N >= n) items corresponding to the voice command. . When the request signal 1233 includes information related to a preference, the content providing device 1220 can identify n items that satisfy the preference. The content providing device 1220 may transmit a response signal 1234 containing information related to the identified n items to the system 530. The response signal 1234 is information related to n items and may include one or more objects corresponding to each of the n items. The response signal may include the priorities of n items. When the request signal 1233 includes information related to a preference, the priorities of n items included in the response signal may be set corresponding to the preference.

In response to receiving response signal 1234, system 530 may identify n items included in response signal 1234. In response to the identification of the n items included in response signal 1234, system 530 generates user interface signal 1235, which is information related to a user interface to be presented to user 1210 via electronic device 810. can do. The user interface signal 1235 may include information related to the identified n items (eg, an object corresponding to each of the n items, a priority of the n items). The user interface signal 1235 may include layout information related to the arrangement of objects in each of the n items.

In response to receiving the user interface signal 1235, the electronic device 810 may output a user interface corresponding to the user interface signal 1235 to the user 1210. On the user interface, n items may be arranged according to a first order. Objects corresponding to each of the n items may be arranged according to the layout information within an area corresponding to each of the n items on the user interface.

Referring to FIG. 12, the user 1210 may perform various inputs 1236 on the user interface. For example, in the item display mode, the user 1210 can sort n items or select at least one of the n items. Information related to various inputs 1236 performed by the user may be included in the log data signal 1237 and transmitted from the electronic device 810 to the system 530.

System 530 may include a database (e.g., user interaction log database 680 of FIG. 6) that stores information included in log data signal 1237. System 530 may identify preference objects from log data signal 1237, for example, using speech response system preference generator 622 of FIG. 6. From various inputs 1236 performed by the user, the system 530 can identify objects that the user attaches relatively more importance to. The object identified by system 530 may be determined to be a preference object.

The user 1210 can change the operation mode of the user interface from the item display mode to the preference change mode. The operating mode may be changed, for example, in response to a touch on the submenu 840 of FIG. 8A. In the preference change mode, the user 1210 may perform a preference input 1238 to directly select a preference object from among objects output on the user interface.

Referring to FIG. 12 , in response to the preference input 1238, the electronic device 810 may transmit a preference control signal 1239 related to the configuration of the preference object to the system 530. The preference control signal 1239 may include information (eg, object properties, object characteristics) related to the object selected by the user in the preference change mode. After the user selects a specific object as a preference object, in order to generate a preference control signal 1239, the electronic device 810 outputs the interface 910 of FIG. 9A to the user to identify characteristics related to the selected specific object. You can.

In response to receiving preference control signal 1239, system 530 may store the identified preference object in a database (e.g., preference database 534 in FIG. 5, user preference database 670 in FIG. 6). there is. When the number of preference objects stored in the database is 2 or more, the correlation between preference objects or the priority of preference objects may be determined. For example, the preference sorter 625 of FIG. 6 can be used to determine the correlation between preference objects or the priority of preference objects.

FIG. 13 is a diagram illustrating an operation of a system identifying preferences from a user according to various embodiments. The operations of FIG. 13 may be performed by system 530 of FIG. 5 or hardware or software components of FIG. 6. While a user interface including a plurality of items is delivered to the user through an electronic device (eg, the first electronic device 520 of FIG. 5), an operation of identifying the preference of FIG. 13 may be performed. The plurality of items may be items identified by the system in response to a user's voice command.

A user may perform an operation to change the order in which a plurality of items are arranged in the user interface. The operation of changing the order in which the plurality of items is arranged includes excluding at least one of the plurality of items from the user interface, and inserting another item that is distinct from the plurality of items between the plurality of items arranged in the order. It may include an operation of adding to and an operation of sorting the plurality of items based on at least one object included in the plurality of items.

The system may collect interactions between the user and the user interface related to a plurality of items included in the user interface. The system may collect interactions related to specific items selected by the user. Interactions collected by the system may include the user's actions of selecting a specific item, skipping it, or browsing detailed information about a specific item.

13, information related to interactions between users and user interfaces may be stored in the system's user interaction log database 680. Information stored in the user interaction log database 680 includes (1) information 1330 related to an item selected or removed by the user among a plurality of items included in the user interface, (2) user and user interface It may include information 1340 related to the item remaining after various interactions, and (3) a voice signal 1350, which is a voice signal input from the user and includes a voice command related to the object.

The system may use the user interaction log database 680 or the preference object extractor 1360 to identify actions in which a user changes the order of a plurality of items. The preference object extractor 1360 can identify objects related to the operation of changing the order. The preference object extractor 1360 may identify at least part of the information included in the user's voice command from the voice signal 1350. For example, when a user inputs a voice command such as “find hotels with 2 stars” into the system, the preference object extractor 1360 extracts specific objects (in the example of the voice command, distinguished by the number of stars) from the voice command. A value (in the example of the voice command above, 2 stars) related to the hotel rating object can be identified.

Systems according to various embodiments may use the preference object extractor 1360 to identify preference objects from information stored in the user interaction log database 680. The preference object identifier 1320 may correspond to a processor included in an electronic device or system or a thread running on the processor. The preference object extractor 1360 may identify the frequency with which a specific object is used to sort a plurality of items and the probability that a specific object is selected. The preference object extractor 1360 may identify characteristics of a specific object based on the identified frequency or probability.

The identified specific object and the characteristics associated with the specific object can be used to create a preference object. When a plurality of features of a specific object are identified based on the identified frequency or probability, the system may request the user to select at least one of the identified plurality of features. The system requesting the user to select at least one of the identified plurality of features may be performed, for example, based on the operations of FIGS. 9A-9C.

The user can change the operation mode of the user interface to preference change mode. The system may use the preference object identifier 1320 to identify the object 1310 selected by the user in the preference change mode. The preference object identifier 1320 may correspond to a processor included in an electronic device or system or a thread running on the processor.

In response to the identification of object 1310, preference object identifier 1320 may identify one or more characteristics associated with the identified object 1310. When a plurality of features are identified as the object 1310, the preference object identifier 1320 may request the user to select at least one of the identified plurality of features. Requesting the user to select at least one of the plurality of identified features may be performed, for example, based on the operations of FIGS. 9A-9C. The preference object identifier 1320 may generate information related to the preference object by associating the object 1310 collected from the user and the characteristics related to the object 1310.

Information related to the preference object generated from the preference object identifier 1320 and the preference object extractor 1360 may be stored in the user preference database 670. Information related to the stored preference object may be used to determine or change the order of a plurality of items provided to the user.

FIGS. 14A to 14C are diagrams illustrating an operation of an electronic device changing the order of a plurality of items based on preferences obtained from a user, according to various embodiments. Referring to FIGS. 14A to 14C , a user interface including a plurality of items may be provided to the user through the display 820 of the electronic device 810.

It is assumed that the user inputs a statement (e.g., “Hey Bixby, show me pants under 50,000 won”) containing a voice command to search for one or more items into the electronic device 810. The utterance may include a wake-up command (“Hey Bixby”) associated with the electronic device 810 or the speech response system. At the same time as the wake-up command is identified, the electronic device 810 recognizes the voice signal input after the wake-up command as an external electronic device (e.g., the system of FIG. 5 (for example, 530)).

Referring to FIG. 14A, an example of a user interface output on the display 820 in response to the voice command is shown. In response to the utterance including the voice command, the electronic device 810 may display a text message 1410 as a visual object corresponding to the result of recognizing the voice command on the user interface. The text message 1410 may be feedback regarding recognition of the utterance. If the text message 1410 does not match the voice command included in the user's statement, the user may perform an action to re-enter the voice command. An operation for re-entering a voice command may include, for example, touching a button (not shown) that activates the microphone of the electronic device 810.

The system may identify a plurality of items or provide search results to the user in response to the voice command. The system may determine conditions for searching for an item based on the object included in the voice command. In the example user's statement above (“Hey Bixby, show me pants under 50,000 won”), the system sets a search condition (have a value of 50,000 won or less) related to the type of item (pants) and object (price object). can be identified. The system may request a content providing device related to the item (for example, a content providing device of a clothing shopping business) to search for an item corresponding to the search condition.

Referring to FIG. 14A, a list of a plurality of searched items may be displayed in a partial area 1420 of the user interface on the display 820. When a list of multiple items is provided to the user through the display 820, a plurality of objects related to each of the multiple items are visual elements (e.g., text elements, image elements, video elements). It can be output on the display 820 in format.

When a list of a plurality of items is output through the display 820, visual elements related to each of the plurality of objects may be arranged on the display 820 based on a layout generated by an electronic device or system. Referring to FIG. 14A, for each item (pants), visual elements corresponding to a photo object, price object, name object, and evaluation object may be included in the layout. The layout may indicate the location of visual elements corresponding to each object based on XML (eXtensible Markup Language) format. The layout may represent some objects to be output on the display 820 among all objects related to the item (in the case of FIG. 14A, a photo object, a price object, a name object, and an evaluation object among all objects related to the item (pants)). .

The layout includes (1) a content providing device that provides a result of identifying a plurality of items, (2) a system that stores preferences related to a plurality of items, and (3) a display 820 based on the state of the display 820. ) may be determined based on at least one of the electronic devices 810 that determine the area in which the visual element will be placed. For example, while transmitting the result of identifying a plurality of items to the system, the content providing device may transmit a layout which is generated according to intention of a content provider to the system. Based on the preference, the system can change the layout transmitted from the content providing device and transmit it to the electronic device 810. The system can change the layout so that visual elements corresponding to preference objects are included in the layout or are emphasized. The electronic device 810 may provide information related to the size and resolution of the display 820 and the area 820 set to display a plurality of items on the display 820 (e.g., the size of the area 820, the area 820 The layout changed by the system can be further changed based on the form of .

Ultimately, the layout includes at least one of not only a content providing device that provides services related to the search of a plurality of items, but also a device (system) that recognizes voice commands and a device (electronic device 810) that directly interacts with the user. It can be created or changed by one. The order of a plurality of items displayed to the user may also be changed not only by the content providing device but also by devices (systems and electronic devices) that manage the user's preferences.

For example, the user may change the operation mode of the user interface to the preference change mode by touching the menu button 1430 in the area 820 where a plurality of items are displayed. In the preference change mode, in response to selection of a visual element by a user, the system may identify at least one of an object, an attribute of the object, and a characteristic of the object associated with the selected visual element. Characteristics of an object may be identified by user input through the interfaces 910, 930, and 950 of FIGS. 9A to 9C, or log data related to the object.

According to some embodiments, the system may identify preference objects from activities performed by the user in operating modes other than the preference change mode. For example, from the user's statement (“Show me pants under 50,000 won”), the system can identify the preference object (price object) and characteristics (have a value of 50,000 won or less) for a specific item (pants). there is. The identified preference object can be used to sort a list of multiple items currently provided to the user.

Figure 14b shows an example of a result in which the order of a plurality of items is changed in response to a user's input or a preference identified from the user's log data. When a user enters a price object as a preference object, based on the user's input through the interfaces 910, 930, and 950 of FIGS. 9A to 9C, or the user's activity shown in log data related to the price object, the system selects the selected The characteristics of the price object can be identified. For example, the system can use the above features to identify features in which users prefer price objects with relatively low values.

Referring to FIG. 14B, in response to the user inputting a price object as a preference object, a list of a plurality of items may be sorted according to characteristics related to the price object. Since users prefer price objects with relatively low values, relatively high priority may be assigned to items related to price objects with relatively low values. In the list of a plurality of items displayed on the display 820, the order of the items may be determined or changed in accordance with the priority. Referring to FIG. 14b, the lowest priced item (C jeans) may be placed first in the list of multiple items. That is, multiple items can be sorted in ascending order of price.

The identified preference object is stored in a specific database of the system (for example, the preference database 534 in FIG. 5) and can be used when performing a new operation according to a voice command newly input from the user. Figure 14c shows an example of a user interface output to the user in response to a voice command (“Show me your pants”) newly input from the user after Figures 14a and 14b. The user interface may include a text message 840, which is a visual element that feeds back the result of recognizing a voice command, and an interface 850 that feeds back the result of performing an operation corresponding to the voice command.

The operation corresponding to the voice command is an operation to search for a specific item (pants), and the voice command may not include additional search conditions other than the specific item. The system may identify previously stored preference objects associated with items included in the voice command. For example, the system identifies preference objects created from voice commands previously entered by the user (“Show me pants under 50,000 won”) and preference objects selected by the user in preference change mode (price objects with relatively small values). can do. Referring to FIG. 14C, based on at least one identified preference object, a plurality of items (pants) having a price of 50,000 won or less may be displayed on the interface 850 of the display 820 in ascending order of price. there is.

As described above, a preference object can influence multiple search operations related to an item. When a plurality of content providing devices provide search results related to the item, a preference object related to one of the plurality of content providing devices may be used by the remaining content providing devices.

FIG. 15 is a diagram illustrating an operation in which a system connected to a plurality of content providing devices shares a preference object related to one of the plurality of content providing devices according to various embodiments. The operation of FIG. 15 may be performed by a system (eg, system 530 of FIG. 5 ) connected to a plurality of content providing devices and at least one electronic device.

At operation 1510, the system may receive a speech signal containing the user's utterance. The voice signal may be obtained from the user's electronic device (eg, the first electronic device 520 in FIG. 1 or the electronic device 810 in FIGS. 8A to 8C). The voice signal may include a voice command based on the user's natural language. The voice command may be related to the action of retrieving at least one item.

At operation 1520, the system may identify the first content providing device corresponding to the received voice signal. When the voice command is related to an operation of searching for at least one item, the first content providing device may be a device that searches for the item. The voice command may include an identifier of the first content provision device (eg, “Find a restaurant in the first content provision service”). That is, the system can identify the first content providing device corresponding to the type or identifier of the item included in the received voice signal.

In response to identification of the first content providing device, at operation 1530, the system may determine whether a preference corresponding to the first content providing device exists. The preference corresponding to the first content providing device may be determined from the user's activities on a plurality of items searched for on the first content providing device before receiving the voice signal. Preferences corresponding to the first content providing device determined from the activity may be stored in a specific database of the system (e.g., preference database 534 in FIG. 5). A preference may contain information related to one or more preference objects. If a preference corresponding to the first content providing device exists, in operation 1540, the system may identify the preference.

If there is no preference corresponding to the first content providing device, in operation 1550, the system may identify a second content providing device that inherits the same object as the first content providing device. Inheriting a specific object means that different content providing devices commonly use the format or data structure of the specific object. The object that is the target of inheritance can be used in both the first content providing device and the second content providing device in the form of a capsule.

In response to identifying the second content providing device, at operation 1560, the system may identify preferences of the identified second content providing device. By identifying the preferences of the second content providing device, the system can identify an object related to the preferences of the second content providing device among a plurality of objects included in the item related to the first content providing device.

In response to identifying the preferences of the first content providing device or the preferences of the second content providing device, at operation 1570, the system may request retrieval of an item from the first content providing device according to the identified preferences. The search may be performed based on the CAN database 535 of FIG. 5. The search may include information related to preference objects included in the identified preferences. When the preference of the second content providing device is identified, a preference object commonly used by the first content providing device and the second content providing device can be used to search for an item. The first content providing device may search for one or more items in response to the request. One or more retrieved items may be transmitted to the system.

In response to one or more items sent to the system, at operation 1580, the system may provide a user interface for outputting the one or more items to a user. The user interface may include visual elements corresponding to the one or more items and at least one object included in each of the one or more items according to a layout based on the identified preferences.

FIG. 16 is an exemplary diagram illustrating an operation in which a system according to various embodiments shares preferences among a plurality of content providing devices.

A preference object may be created or managed as a capsule unit in which multiple objects are combined. The content providing device may create a capsule containing all objects related to the stored item. The speech response system can identify an object that the user considers relatively important among the objects included in the capsule as a preference object.

When managing preference objects on a capsule basis, systems according to various embodiments can share preferences between different capsules used in different content providing devices. Referring to FIG. 16, an example operation of sharing preferences between different content providing devices (a first content providing device and a second content providing device) is shown. Referring to FIG. 16, a plurality of objects (CuisineStyle, ReviewRating, and ReviewCount) related to the item of the first content providing device may be grouped into a first capsule 1610, and a plurality of objects related to the item of the second content providing device Objects (CuisineStyle, and Storeinfo) may be grouped into a second capsule 1620.

Systems according to various embodiments may include objects shared by different content providing devices using a capsule library. Within a capsule or between multiple capsules, multiple objects may have a hierarchical structure. Within a capsule, at least one object and capsule included in the capsule may have a hierarchical structure. Additionally, a plurality of capsules may have a layered structure. In this case, an object defined in a higher-level capsule can be inherited by a lower-level capsule. High-level capsules can be identified from a capsule library, which is a set of information related to the definition of an object or capsule. A capsule of a content providing device that uses an object defined in a higher-level capsule may be a lower-level capsule than the higher-level capsule. When a plurality of content providing devices commonly use an object defined in a specific capsule, the plurality of content providing devices may share the commonly used object.

Sharing of capsule libraries and objects may be performed by the system (e.g., preference exchanger 624 in FIG. 6). Referring to FIG. 16, a library capsule 1630 is shown containing objects (CuisineStyle, ReviewRating) related to restaurant items included in the capsule library. Each of the first content providing device and the second content providing device may inherit at least some of the objects included in the library capsule 1630 to create the first capsule 1610 and the second capsule 1620.

Referring to FIG. 16, at least one (eg, ReviewCount) of the plurality of objects included in the first capsule 1610 may be defined by the first content providing device. At least one of the plurality of objects (CuisineStyle, ReviewRating) included in the first capsule 1610 may be defined by an object included in the library capsule 1630.

Referring to FIG. 16, the second capsule 1620 used in the second content providing device may be created using the definition of some of the objects (CuisineStyle) included in the library capsule 1630. That is, the second capsule 1620 may include some objects (CuisineStyle) among the objects included in the library capsule 1630.

Since the first capsule 1610 and the second capsule 1620 are defined using one library capsule 1630, at least one object can be shared between the first content providing device and the second content providing device. For example, the object (CuisineStyle) commonly used in the first capsule 1610 and the second capsule 1620 is defined based on the object (CuisineStyle) included in the library capsule 1630, thereby providing a value of the same type. You can have it. Preferences or preference objects related to the object (CuisineStyle) may also be shared between the first content providing device and the second content providing device.

Sharing of preferences or preference objects may be performed by an electronic device or process that manages preference objects in an utterance response system (e.g., system 530 in FIG. 5, based on preference exchanger 624 in FIG. 6). there is. For example, the user searches for a plurality of items using the first content providing device, and the speech response system sends a specific value (“chinesecusine”) related to a specific object (CuisineStyle) included in the first capsule 1610 to the user. Assume that it is identified by the preference of . In this case, the preference object may include information matching the specific object and value (eg, “CusinStyle.chinesecusine”).

When a preference object is defined identically in a plurality of capsules, the preference object may be shared by a plurality of capsules and a plurality of content providing devices corresponding to each of the plurality of capsules. Since the first capsule 1610 and the second capsule 1620 are defined using one library capsule 1630, an object (CuisineStyle) commonly used in the first capsule 1610 and the second capsule 1620 Related preference objects may be shared by each of the first content providing device and the second content providing device. A preference object (e.g., a preference object including the "CusinStyle.chinesecusine") determined from the user's activity on a plurality of items searched in the first content providing device searches and sorts the items using the second content providing device. can be used for

Preference sharing between a plurality of capsules may be performed by the capsule database 230 of the intelligent server 200 of FIG. 1. The capsule database 230 may be included as at least part of an electronic device that manages preferences (eg, system 530 of FIG. 5). The capsule database 230 provides content using a preference object related to one of a plurality of content providing devices (e.g., a first content providing device) and an object corresponding to the preference object among the remaining content providing devices. It can be transmitted to the device.

For example, a preference object (eg, a preference object including “CusinStyle.chinesecusine”) determined from the user's activities on a plurality of items searched for in the first content providing device may be transmitted to the capsule database. Additional information (eg, the user's delivery food preference) related to the preference object may be transmitted to the capsule database together. The capsule database can identify that the object (CusinStyle) related to the preference object is an object included in the library capsule 1630. The capsule database may identify the first capsule 1610 and the second capsule 1620 that inherit the library capsule 1630. The capsule database can use the preference object to search for an item of the second content providing device corresponding to the identified second capsule 1620.

FIGS. 17A and 17B are exemplary diagrams illustrating an operation in which an electronic device shares preferences between a plurality of applications related to each of a plurality of content providing devices, according to various embodiments.

Referring to FIG. 17A , the user may input a first voice command related to searching for a specific item (restaurant) by the first content providing device into the electronic device 810 of the speech response system. The first voice command may include a command to use the first content providing device to search for an item. The electronic device 810 may output the result of recognizing the first voice command on the display 820 in the form of a text message 1710.

In response to the input of the first voice command, the electronic device may output the results of searching a plurality of items from the first content providing device on the display 820. A list of multiple items may be output to at least a partial area 1720 of the display 820. The order of the plurality of items output to the area 1720 may be a first order based on a preference object created in the past. In the area 1720, visual elements corresponding to at least one object related to a plurality of items may be arranged according to the first layout. Referring to FIG. 17A, a visual element 1730 corresponding to the name object of the item and a visual element 1740 corresponding to the image object of the item may be included in a partial area 1720 where a plurality of items are output.

In response to output of the plurality of items, the user may perform various activities related to the plurality of items. The activity may include sorting a plurality of items in ascending or descending order of a specific object (eg, price object), and selecting or removing at least one of the plurality of items. The activity may include inputting a preference object into the electronic device. The operation of the user entering a preference object may be performed, for example, based on the operations described in FIGS. 9A to 10B.

A user can switch the operation mode of a user interface that outputs a plurality of items between a preference change mode and an item display mode. In the item display mode, in response to the user's touch on any one of the visual elements 1730 and 1740, the electronic device 810 displays one of the visual elements 1730 and 1740 touched on the display 820. Detailed information on the corresponding item can be output.

In the preference change mode, in response to the user's touch on any one of the visual elements 1730 and 1740, the electronic device 810 changes any one of the visual elements 1730 and 1740 touched on the display 820. You can identify preference objects related to . Referring to FIG. 17A, the user may touch at least a portion of the visual element 1730 corresponding to the name object. For example, the user may select only the “Chinese” portion of the visual element 1730 using a drag gesture. In this case, the electronic device or system may determine that the user is interested in items whose names include “Chinese food.” In response to the above determination, the electronic device or system may match the name object and “Chinese food” to create a preference object.

As another example, the user may touch at least a portion of the visual element 1740 corresponding to the image object. The electronic device or system creates an image object similar to the image object selected by the user (or containing features of the image object selected by the user) based on the characteristics of the touched image object (for example, the type of food subject included in the image). You may decide that you are interested in an item containing an image object. The electronic device or system can generate a preference object by matching the image object and the features.

The created preference object can be personalized to the user. The preference object can be used not only for searching for items using the first content providing device, but also for searching for items using the second content providing device. Referring to FIG. 17B, the user may input a second voice command related to the search for a specific item (restaurant) by the second content providing device into the electronic device 810 of the speech response system. The electronic device 810 may output the result of recognizing the second voice command on the display 820 in the form of a text message 1750.

The second voice command may be input after the first voice command is input. In this case, the activity performed by the user in response to the first voice command and a preference object created based on the activity may be used to search for an item corresponding to the input of the second voice command. For example, if the user selects the visual element 1730 and inputs into the electronic device 810 that he or she is interested in an item with a name including “Chinese food,” the system may respond to the input of the second voice command. The second content providing device may be requested to search for an item whose name includes “Chinese food.” The request may be performed based on the operation related to sharing of the preference object described in FIGS. 15 and 16.

Referring to Figure 17b, as the preference object related to the first voice command is shared, even though the second voice command does not include a command to search for an item with a name including "Jungsik", "Jungsik" An item with a name including may have a relatively high priority among a plurality of items provided by the second content providing device.

When a plurality of items provided by the second content providing device are output in at least a partial area 1760 of the display 820, an item (name including “Chinese food”) having a relatively high priority among the plurality of items items) can be output first.

FIGS. 18A and 18B are exemplary diagrams for explaining an operation of an electronic device outputting a preference object to a user according to various embodiments.

Referring to FIG. 18A , when a user inputs a voice command to search for at least one item into the electronic device 810, the electronic device recognizes the voice command and sends the result of the voice command to the electronic device 810 in the form of a text message 1810. ) can be output on the display 820. Hereinafter, it is assumed that the user enters a voice command (“Find a hotel in San Jose under $400 for Thanksgiving weekend”) containing a search condition related to a hotel item.

Based on the recognized voice command, the system may communicate with at least one content providing device to retrieve the user's voice command and one or more hotel items. The action of searching for the item may be identified based on preference objects created by the user's past activities. The operation of searching for the item may be performed based on search conditions included in the voice command.

Search conditions may be created based on a format similar to the preference object by matching a specific object and the properties of the specific object. For example, from the voice command, a search condition can be created by matching a location object and a specific location (San Jose). Additionally, from the voice command, search conditions can be created by matching a price object and a specific price range ($400 or less). Additionally, from the voice command, a search condition can be created by matching a period object and a specific time range (Thanksgiving weekend). Search conditions entered from voice commands can be used to create preference objects.

Ultimately, the content providing device can search for the preference object and one or more items that satisfy the one or more search conditions. Referring to FIG. 18A , the system may respond to a voice command through the electronic device 810 and provide the user with a result of identifying one or more items. The result of identifying one or more items or performing a voice command may be provided to the user through a partial area 1820 on the display 820.

The electronic device 710 or system may output search conditions or preference objects used to search for items to the user. Referring to FIG. 18A, a preference object used to search for an item may be provided to the user through a partial area 1830 on the display 820. When the user touches the partial area 1830, the electronic device 810 may output detailed information about the preference object on the display 820 as shown in FIG. 18B. Referring to FIG. 18B, a preference object created from a user's voice command may be output in a partial area 1850 of the display 820, and a preference object created before input of the voice command may be displayed in a portion of the display 820. It may be output to area 1840.

A user may change one or more preference objects that are displayed in the example user interface of FIG. 18B. For example, the user can change the properties of a preference object related to whether Wi-Fi is supported among the plurality of preference objects shown in FIG. 18B. A user may touch an interface related to star ranking on the example user interface of FIG. 18B to change properties of a preference object related to the star ranking.

FIG. 19 is a diagram illustrating an example of user interfaces (UIs) provided by an electronic device to a user to identify a preference object according to various embodiments.

Referring to FIG. 19, the electronic device may output UI 1910 to the user. The UI 1910 may include a list of items previously provided to the user and visual elements for identifying the object from the user. For example, referring to FIG. 19, the electronic device displays visual elements related to hotel-related objects (star rating, amenities, room type) within the UI 1910. can be printed out. The user may select a visual element related to the object for which a preference is to be input from among the visual elements corresponding to each of the objects output within the UI 1910.

For example, when a user touches an area 1911 within a visual element associated with an amenity object, the electronic device displays a UI 1920 for identifying the user's preferred attribute among one or more attributes included in the amenity object. can be output. One or more properties included in the amenity object may be identified from an electronic device or a system connected to the electronic device (e.g., system 530 of FIG. 5). Referring to FIG. 19 , the electronic device may output visual elements corresponding to each of the attributes included in the amenity (non-smoking, whether pets are allowed, whether or not breakfast is included) within the UI 1920. The electronic device may output a visual element (eg, a like button) within the UI 1920 for receiving a user's selection related to at least one of a plurality of properties included in the amenity object.

Referring to FIG. 19, within the UI 1920, the user can touch visual elements 1921 and 1922 corresponding to non-smoking status and pet accommodation status, respectively. In response to touching the visual elements 1921 and 1922, the electronic device may determine that the user has a relative preference for non-smoking and pet-friendly hotels. In response to touching the visual elements 1921 and 1922, the electronic device may provide feedback indicating that properties related to the selected visual elements 1921 and 1922 are included in the preference (e.g., UI 1930 ). The electronic device may transmit attributes selected by the user (eg, non-smoking status and pet accommodation availability) among a plurality of attributes included in the amenity object to a system connected to the electronic device.

The electronic device may output a visual element 1923 related to termination or transition of the UI 1920 within the UI 1929. When the user touches the visual element 1923 or a visual element 1924 that is floated within or over the UI 1923 by the operating system, the electronic device terminates displaying the UI 1920 and You can return to the UI (for example, UI 1910) output before the output of 1920. Since the preference has been identified from the user based on the UI 1920, the electronic device can change the display of the UI to return to based on the result of identifying the preference.

Referring to FIG. 19, a UI 1930 that is output after identifying a user's preference based on the UI 1920 is shown. The electronic device may output a visual element 1931 corresponding to the attributes selected by the user within the UI 1930. After output of the UI 1930, when the user searches for a hotel, the electronic device and the system connected to the electronic device display the object (amenity object) selected through the UI 1920 and the attributes selected by the user (non-smoking status and You can search for hotels based on pet availability. A combination of objects and attributes (e.g., preference objects) identified through the UI 1920 is, for example, based on what is described in FIGS. 15 and 16, between a plurality of content providing services related to the search of a hotel. can be shared in

//

FIG. 20 is a flowchart 2000 for explaining the operation of an electronic device according to various embodiments. For example, the electronic device of FIG. 20 may correspond to the first electronic device 520 of FIG. 5 .

Referring to FIG. 20, in operation 2010, an electronic device according to various embodiments may display a user interface including one or more objects on the display. The UI displayed on the display can be identified from an application program stored in the memory of the electronic device. The application program may correspond to a voice-based assistance program.

Referring to FIG. 20, in operation 2020, an electronic device according to various embodiments may receive a first user input for selecting one object from among the objects included in the UI. For example, the first user input may be related to the user's input described in FIGS. 8A-8C. For example, the first user input may relate to an input for changing the operating mode to a preference change mode or selecting one or more objects from a plurality of objects within the preference change mode.

Referring to FIG. 20, in operation 2030, an electronic device according to various embodiments may transmit first information related to the selected object to an external server through a communication circuit. The first information may include at least one of the name of the object and properties related to the object. The external server of FIG. 20 may correspond to system 530 of FIG. 5 .

Referring to FIG. 20, in operation 2040, an electronic device according to various embodiments may, through a communication circuit, receive one or more properties of an object selected from an external server (e.g., a system connected to the electronic device). You can receive second information about. The second information may include a plurality of attributes identified from the system and associated with the object.

Referring to FIG. 20, in operation 2050, an electronic device according to various embodiments may display received second information on a UI. For example, the electronic device may output at least one of the interfaces 910, 930, and 950 for selecting at least one of the attributes included in the second information on the display, as shown in FIGS. 9A to 9C.

Referring to FIG. 20, in operation 2060, an electronic device according to various embodiments may receive a second user input for selecting at least one attribute from among attributes displayed on a UI. For example, the user may select at least one of a plurality of attributes related to the object selected by the first user input based on the interfaces 910, 930, and 950 of FIGS. 9A to 9C.

Referring to FIG. 20, in operation 2070, an electronic device according to various embodiments may transmit third information related to the selected attribute to an external server through a communication circuit. The external server to transmit the third information may correspond to the external server of operation 2030. The third information may include parameters for identifying the attribute selected by the user.

Referring to FIG. 20, in operation 2080, an electronic device according to various embodiments may receive fourth information related to third information from an external server. The fourth information may be related to information (eg, preference object) matching the object and attribute identified in each of the first and second user inputs.

Referring to FIG. 20, in operation 2090, an electronic device according to various embodiments may reconstruct one or more objects based at least in part on fourth information and display them on the user interface. For example, while performing operation 2010, the electronic device may change at least one of the layout and order of objects included in the UI output on the display based on the fourth information. The fourth information may include a score based at least in part on the properties of one or more objects included in the UI. When searching for a plurality of items, the electronic device can be used to sort the searched plurality of items, starting from the preference object with the highest score among the plurality of preference objects, based on one or more scores included in the fourth information.

Figure 21 is a flowchart 2100 for explaining the operation of a system according to various embodiments. The system may correspond to the electronic device of FIG. 20 and a device (eg, an external server) connected to a wireless or wired network.

Referring to FIG. 21, in operation 2110, a system according to various embodiments may receive a request for first information about one or more properties related to an object from an electronic device connected to the system. The electronic device connected to the system may display a UI including one or more objects on the display, for example, based on operation 2010 of FIG. 20 . The request may be related to one or more objects selected by the user of the electronic device from among objects displayed on the UI. The request may be received through a communication interface included in the system (eg, communication interface 531 in FIG. 5). The request may be related to the first information of operation 2030 of FIG. 20.

In response to a request for first information received from an electronic device, in operation 2120, a system according to various embodiments may transmit first information to the electronic device through the communication interface. The first information may include one or more attributes related to one or more objects selected by the user of the electronic device. The first information may be related to the second information in operation 2040 of FIG. 20. The electronic device receiving the first information of operation 2120 allows the user to use an interface (e.g., interfaces 910, 930, and 950 of FIGS. 9A to 9C) for selecting one or more attributes included in the first information. It can be printed to .

Referring to FIG. 21, in operation 2130, a system according to various embodiments receives, from an electronic device, second information related to at least one attribute selected by the user from among one or more attributes included in the first information. Requests can be received. The system may receive a request for second information through a communication interface. The request for second information may be generated by the electronic device, for example, based on operations 2060 to 2070 of FIG. 20 . The request for second information may include information to identify one or more attributes selected by the user of the electronic device.

In response to a request for second information received from the electronic device, in operation 2140, the system according to various embodiments may transmit the second information to the electronic device through a communication interface. The operation of the system acquiring the second information will be described in more detail with reference to FIG. 22. The electronic device may receive the second information, for example, based on operation 2080 of FIG. 20.

FIG. 22 is a flowchart 2200 illustrating an operation in which a system according to various embodiments acquires a score related to an attribute of an object identified by a user of an electronic device. The system of FIG. 22 may correspond to the system of FIG. 21.

Referring to FIG. 22, in operation 2130, a system according to various embodiments requests, from an electronic device, second information related to at least one attribute selected by the user among one or more attributes included in the first information. can receive. Operation 2130 of FIG. 22 may correspond to operation 2130 of FIG. 21. The request for second information may include information for identifying the attribute selected by the user and the object corresponding to the selected attribute.

Referring to FIG. 22, in operation 2210, a system according to various embodiments may generate a score based at least in part on an attribute already specified and stored in the memory and the selected at least one attribute. The score may be related to determining whether to use the attribute selected by the user and information matching the object corresponding to the selected attribute (eg, preference object) to sort the list of items to be provided to the user. For example, when sorting a plurality of items searched in response to a user's request, an electronic device or system according to various embodiments may, based on the scores of each of a plurality of preference objects related to the plurality of items, You can select a preference object to use for sorting multiple items.

Referring to FIG. 22, in operation 2220, a system according to various embodiments may generate second information including a score. Referring to FIG. 22, in operation 2140, a system according to various embodiments may transmit second information to an electronic device. The generated score may be transmitted to the electronic device as at least part of the second information. Operation 2140 of FIG. 22 may correspond to operation 2140 of FIG. 21.

According to various embodiments, electronic devices and systems may search for one or more items in response to a user's voice command related to searching for an item. When a plurality of items are searched, a plurality of items arranged according to the first order may be provided to the user through the user interface of the electronic device. By controlling the user interface, a user can perform various activities related to a plurality of items. Each of the plurality of items may include a plurality of objects. Electronic devices and systems may identify, from the activity, an object that a user uses to identify at least one item (eg, a user's preferred item) among a plurality of items. By matching the identified object and the characteristics of the identified object, electronic devices and systems can determine the user's preferences related to retrieval of the item. The determined preference can be used to sort a plurality of items. For example, a preference can be used to change the order in which a plurality of items are arranged on a user interface from a first order to a second order. For example, when a user enters a new voice command, the preference can be used to retrieve an item in response to the new voice command.

Methods according to embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

When implemented as software, a computer-readable storage medium that stores one or more programs (software modules) may be provided. One or more programs stored in a computer-readable storage medium are configured to be executable by one or more processors in an electronic device (configured for execution). One or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure.

These programs (software modules, software) include random access memory, non-volatile memory including flash memory, read only memory (ROM), and electrically erasable programmable ROM. (EEPROM: electrically erasable programmable read only memory), magnetic disc storage device, compact disc-ROM (CD-ROM), digital versatile discs (DVDs), or other forms of disk storage. It can be stored in an optical storage device or magnetic cassette. Alternatively, it may be stored in a memory consisting of a combination of some or all of these. Additionally, multiple configuration memories may be included.

In addition, the program may be operated through a communication network such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof. It may be stored on an attachable storage device that is accessible. This storage device can be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to the device performing an embodiment of the present disclosure.

In the specific embodiments of the present disclosure described above, elements included in the disclosure are expressed in singular or plural numbers depending on the specific embodiment presented. However, singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components, and even components expressed in plural may be composed of singular or singular. Even expressed components may be composed of plural elements.

Meanwhile, in the detailed description of the present disclosure, specific embodiments have been described, but of course, various modifications are possible without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be determined not only by the scope of the patent claims described later, but also by the scope of this patent claim and equivalents.

Claims (20)

In electronic devices,
display;
at least one communication circuit;
mike;
at least one speaker;
at least one processor operably coupled to the display, the communication circuitry, the microphone, and the speaker; and
At least one memory electrically connected to the processor,
The memory is configured to store an application program including a user interface,
When executed, the memory causes the processor to:
Displaying the user interface on the display to include one or more objects corresponding to each of a plurality of items, and receiving, in a preference change mode, a first user input for selecting one object from among the objects,
Transmitting first information related to the selected object to an external server through the communication circuit,
receive second information about one or more attributes of the selected object from the external server through the communication circuit and display it on the user interface;
Receiving a second user input selecting at least one attribute from among the attributes,
Transmitting, through the communication circuit, third information related to the selected attribute to the external server for each input that selects the object during the preference change mode,
Receiving fourth information associated with the third information from the external server through the communication circuit,
Reconstructing a plurality of objects belonging to the item based at least in part on the fourth information, which is a preference object identified in each of the first user input and the second user input, and displaying them on the user interface
An electronic device that stores instructions. delete According to paragraph 1,
The fourth information above is,
An electronic device comprising a score based at least in part on attributes of the one or more objects included in the user interface. According to paragraph 1,
The application program is,
An electronic device containing a voice-based assistance program. According to paragraph 1,
The instructions are such that the processor,
arranging the one or more objects arranged based on a first layout within the user interface within the user interface based on a second layout related to the received fourth information
Electronic devices. According to paragraph 1,
The instructions are such that the processor,
Within the user interface, output visual elements for reconstruction of the objects,
The first user input is received after receipt of a user input related to the visual element.
Electronic devices. According to paragraph 1,
The instructions are such that the processor,
In response to selection of an object related to an image among the objects by the first user input, transmitting the first information including a request for obtaining properties of the image to the external server,
In response to receiving the second information related to the first information including a request to obtain properties of the image, displaying on the user interface one or more properties related to the image included in the second information. Electronic devices. In the system,
communication interface;
a processor operatively connected to the communication interface; and
At least one memory electrically connected to the processor,
The memory, when executed, allows the processor to:
Displaying a user interface including one or more objects on a display, and requesting, from an electronic device in a preference change mode, first information about one or more properties related to an object selected from among the objects, the communication Through the interface, receive,
In response to a request for the first information, transmitting the first information to the electronic device through the communication interface,
Receiving, from the electronic device, through the communication interface, a request for second information related to at least one attribute selected from the one or more attributes,
In response to a request for the second information, transmit the second information to the electronic device through the communication interface,
The second information is,
Containing information for reconstructing a plurality of objects included in the user interface displayed on the electronic device,
A system for storing instructions. According to clause 8,
The instructions are such that the processor,
Receiving a request for the second information,
Generating a score based at least in part on an attribute already specified and stored in the memory and the selected at least one attribute,
A system for transmitting the score as at least part of the second information to the electronic device. delete According to clause 8,
The first information is,
A system comprising information for configuring a user interface for selecting, from a user of the electronic device, at least one of the one or more attributes. According to clause 8,
The instructions are such that the processor,
A system for transmitting the first information including one or more properties related to the image to the electronic device in response to receiving a request for the first information regarding properties of an object including an image among the objects. In electronic devices,
mike,
Memory;
display; and
A processor operatively coupled to the microphone, the memory, and the display.
Including,
The processor,
Receiving the user's utterance through the microphone,
In response to the utterance, a plurality of items are displayed based on a first order within a user interface output within the display, each of the plurality of items including at least one visual element, and the user an interface displayed with the plurality of items and comprising at least one executable object for changing the first order;
In a preference change mode of the user interface, in response to the user's input selecting the at least one executable object, change the plurality of items within the user interface based on a second order indicated by the selected object. mark,
In the preference change mode, in response to the user's input for selecting one of the at least one visual element, the plurality of visual elements are displayed based on a third order that is distinct from the first order and the second order. display items within the user interface,
In the preference change mode, the visual element is,
Corresponds to one of the objects related to each of the plurality of items,
The third order is,
An electronic device that re-sorts the plurality of items based on an object corresponding to the selected visual element. According to clause 13,
The processor,
In response to the user's input for selecting a visual element corresponding to an image object of one of the plurality of items among the visual elements, a list containing a plurality of features identified in the image object is displayed on the user interface. An electronic device that produces output. According to clause 14,
The processor,
In response to the user's input selecting at least one of the plurality of features, identify one or more items including an image object related to the feature among the plurality of items,
An electronic device that determines the third order such that one or more items including an image object related to the feature have a relatively high priority among the plurality of items. According to clause 13,
The processor,
In response to the user's input for selecting one of the at least one visual element, identifying an object corresponding to the selected visual element and a characteristic of the selected visual element,
An electronic device that requests a second electronic device connected to the electronic device to transmit information related to the third order used to change the order of the plurality of items based on the identified object and characteristic. According to clause 16,
The second electronic device,
An electronic device that generates the third order based on a preference object including information matching the identified object and characteristics. According to clause 17,
The second electronic device,
An electronic device that generates information for changing the arrangement of the visual element in the user interface, based on the preference object. delete According to paragraph 1,
The preference change mode is,
While the preference change mode is set, based on an input of the user selecting an object, the at least one processor adds or changes the user's preference for the one or more objects included in the user interface. An electronic device that is configured to do one thing.

Images