KR20190013390A - Electronic device and method for providing search result thereof - Google Patents

Abstract

According to various embodiments of the present document, the present invention relates to a method for providing a search result of an electronic device and an electronic device for the method. The method for providing a search result of an electronic device comprises: an operation of displaying a screen which includes an object; an operation of sensing a user input of selecting the object; an operation of generating a captured image by capturing the screen which includes the object in response to the user input; an operation of transmitting at least a part of the captured image to an external apparatus; an operation of receiving a search result, which is obtained based on object information corresponding to the object and additional information related to the object from the external apparatus; and an operation of displaying the search result.

Description

ELECTRONIC DEVICE AND METHOD FOR PROVIDING SEARCH RESULT THEREOF

This document relates to an electronic device providing search results and a method for providing search results.

In particular, this document relates to a method of providing search results associated with objects contained in a screen of an electronic device.

This document also relates to an artificial intelligence (AI) system that simulates the functions of the human brain using cognitive learning algorithms, and its applications.

As the communication technology and the user interface of the electronic device evolve, the user can easily receive necessary information through the electronic device without restriction of the place and time.

When an electronic device provides a screen containing an object, the user may want to retrieve associated information associated with the provided object.

To do this, the user may separately store an object, perform an image search using the stored object as a keyword, or perform a text search by directly inputting text related to the object.

In recent years, artificial intelligence systems that implement human-level intelligence have been used in various fields. Artificial intelligence system is a system that the machine learns, judges, and becomes smarter, unlike the existing rule-based smart system. Artificial intelligence systems are becoming more and more recognizable as users use them, and existing rule-based smart systems are increasingly being replaced by deep-run-based artificial intelligence systems.

Artificial intelligence technology consists of elemental technologies that utilize machine learning (eg, deep learning) and machine learning.

Machine learning is an algorithm technology that classifies / learns the characteristics of input data by itself. Element technology is a technology that simulates functions such as recognition and judgment of human brain using machine learning algorithms such as deep learning. Understanding, reasoning / prediction, knowledge representation, and motion control.

The various fields in which artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing, applying, and processing human language / characters, including natural language processing, machine translation, dialogue system, query response, speech recognition / synthesis, and the like. Visual understanding is a technology for recognizing and processing objects as human vision, including object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, and image enhancement. Inference prediction is a technique for judging and logically inferring and predicting information, including knowledge / probability based reasoning, optimization prediction, preference base planning, and recommendation. Knowledge representation is a technology for automating human experience information into knowledge data, including knowledge building (data generation / classification) and knowledge management (data utilization). The motion control is a technique for controlling the autonomous travel of the vehicle and the motion of the robot, and includes motion control (navigation, collision, traveling), operation control (behavior control), and the like.

Conventionally, there has been an inconvenience that a user has to go through various steps in order to retrieve association information related to an object included in a screen. In particular, if the electronic device does not support image retrieval, the user had to manually enter text about the object. In addition, when association information related to an object is searched in various fields, a user often inconveniences to perform a search again based on the search result.

According to an embodiment of the present disclosure, an electronic device search result providing method includes: displaying an application execution screen; Receiving a user input while displaying the application execution screen; Capturing the application execution screen according to the user input to generate an image; And displaying a search result performed using first information on a first region corresponding to the user input on the generated image and second information on a second region different from the first region on the generated image And the first information and the second information may be obtained by a learned model.

The first region may be obtained from the generated image based on the touch coordinates corresponding to the input of the user.

The method may further include displaying the generated image in place of the application execution screen.

And, the above method can be performed by execution of an artificial intelligence agent.

The first information may be information obtained by recognizing the first area of the generated image.

The second information may be information obtained by recognizing the second area of the generated image.

Also, the user input may be an input to the application execution screen.

The method may further include transmitting user feedback based on the search result to an external device.

Transmitting at least a portion of the generated image to an external search server; And receiving the search result from the external search server.

The operation of displaying the search result may be an operation of displaying the search result while the application execution screen is displayed.

According to one embodiment of the present disclosure, an electronic device comprises: a display; A user input section; A communication unit; A processor electrically connected to the display, the user input and the communication unit; And a memory for storing one or more computer programs executed by the processor, wherein the one or more computer programs are operable to: display an application execution screen via the display; When receiving a user input while displaying the application execution screen, generating an image by capturing the application execution screen according to the user input; And displaying a search result performed using first information on a first region corresponding to the user input on the generated image and second information on a second region different from the first region on the generated image Operation.

The first region may be obtained from the generated image based on the touch coordinates corresponding to the input of the user.

The one or more computer programs may further include an operation of displaying the generated image in place of the application execution screen.

And, the one or more computer programs can be executed by execution of an artificial intelligence agent.

The first information may be information obtained by recognizing the first area of the generated image.

The second information may be information obtained by recognizing the second area of the generated image.

Also, the user input may be an input to the application execution screen.

The one or more computer programs may further include an operation of transmitting user feedback according to the display of the search result to an external device.

In addition, the operation of displaying the search result may be an operation of displaying the search result while the application execution screen is displayed.

Meanwhile, a method of acquiring information using a learned model of an electronic device, according to an embodiment of the present disclosure, includes: displaying an application execution screen; Capturing the application execution screen to generate an image; Acquiring first information on a first region on the generated image through a learned first model using the generated image as input data; And acquiring second information on a second region different from the first region on the generated image through the learned second model using the acquired first information and the generated image as input data can do.

According to this document, the user can easily search the search results associated with the objects contained in the screen.

In addition, a user-customized search result that reflects the user's search intention can be provided by providing the search result using the context information acquired based on the selection of the object.

Accordingly, the number of retrieval steps and the number of times of re-searching the user are reduced, thereby improving the satisfaction and convenience of the user using the electronic device.

1 shows a use diagram of an electronic device providing search results according to various embodiments.
Figure 2 shows block diagrams of a network system including an electronic device, in accordance with various embodiments.
Figures 3-5 illustrate a flow diagram of a network system including an electronic device according to various embodiments.
FIG. 6 is a diagram illustrating a process of providing search results associated with objects according to various embodiments.
Figure 7 is a diagram that provides search results associated with an object using perimeter information according to various embodiments.
Figure 8 shows use diagrams of an electronic device for selecting objects according to various embodiments.
9A and 9B are diagrams that provide search results associated with objects in accordance with various embodiments.
Figure 10 is a diagram showing searching for search results associated with objects according to various embodiments.
11 is a diagram that provides search results associated with objects in accordance with various embodiments.
12 is a diagram for providing search results associated with an object using photographing information according to various embodiments.
Figures 13 to 16C are diagrams that provide search results associated with objects in accordance with various embodiments.
Fig. 17 is a diagram showing an example in which an electronic device and an external device according to various embodiments cooperate to learn and recognize data.
18 is a block diagram showing a configuration of an electronic device according to various embodiments.
19A and 19B are block diagrams showing a learning unit and a recognition unit according to various embodiments.
20 is a flowchart for explaining a control method of an electronic device according to various embodiments.
21 is a flow diagram of an electronic device using a recognition model according to various embodiments.
22 to 25 are flowcharts of a network system using a recognition model according to various embodiments.
26 and 27 are flow diagrams of an electronic device using a recognition model, in accordance with various embodiments.
28 and 29 are flowcharts of a system using a recognition model according to various embodiments.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. It should be understood, however, that this invention is not intended to be limited to the particular embodiments described herein but includes various modifications, equivalents, and / or alternatives of the embodiments of this document . In connection with the description of the drawings, like reference numerals may be used for similar components.

 In this document, the expressions "having," " having, "" comprising," or &Quot;, and does not exclude the presence of additional features.

In this document, the expressions "A or B," "at least one of A or / and B," or "one or more of A and / or B," etc. may include all possible combinations of the listed items . For example, "A or B," "at least one of A and B," or "at least one of A or B" includes (1) at least one A, (2) Or (3) at least one A and at least one B all together.

As used herein, the terms "first," "second," "first," or "second," and the like may denote various components, regardless of their order and / or importance, But is used to distinguish it from other components and does not limit the components.

(Or functionally or communicatively) coupled with / to "another component (eg, a second component), or a component (eg, a second component) Quot; connected to ", it is to be understood that any such element may be directly connected to the other element or may be connected through another element (e.g., a third element). On the other hand, when it is mentioned that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component) It can be understood that there is no other component (e.g., a third component) between other components.

As used herein, the phrase " configured to " (or set) to be "configured according to circumstances may include, for example, having the capacity to, To be designed to, "" adapted to, "" made to, "or" capable of ". The term " configured to (or set up) "may not necessarily mean" specifically designed to "in hardware. Instead, in some situations, the expression "configured to" may mean that the device can "do " with other devices or components. For example, a subprocessor configured to (or configured to) execute the phrases "A, B, and C" may be implemented as a processor dedicated to performing the operation (e.g., an embedded processor), or one or more software programs To a generic-purpose processor (e.g., a CPU or an application processor) that can perform the corresponding operations.

Electronic devices in accordance with various embodiments of the present document may be used in various applications such as, for example, smart phones, tablet PCs, mobile phones, videophones, electronic book readers, desktop PCs, laptop PCs, netbook computers, workstations, a portable multimedia player, an MP3 player, a medical device, a camera, or a wearable device. Wearable devices may be of the type of accessories (eg, watches, rings, bracelets, braces, necklaces, glasses, contact lenses or head-mounted-devices (HMD) (E.g., a skin pad or tattoo), or a bio-implantable circuit. In some embodiments, the electronic device may be, for example, a television, a digital video disk (Eg Samsung HomeSyncTM, Apple TVTM, or Google TVTM), game consoles, home appliances, air conditioners, air conditioners, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air purifiers, set top boxes, home automation control panels, (E.g., Xbox (TM), PlayStation (TM)), an electronic dictionary, an electronic key, a camcorder, or an electronic photo frame.

In an alternative embodiment, the electronic device may be any of a variety of medical devices (e.g., various portable medical measurement devices such as a blood glucose meter, a heart rate meter, a blood pressure meter, or a body temperature meter), magnetic resonance angiography (MRA) A navigation system, a global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), an automobile infotainment device, a marine electronic equipment (For example, marine navigation systems, gyro compasses, etc.), avionics, security devices, head units for vehicles, industrial or domestic robots, drones, ATMs at financial institutions, of at least one of the following types of devices: a light bulb, a fire detector, a fire alarm, a thermostat, a streetlight, a toaster, a fitness device, a hot water tank, a heater, a boiler, .

In this document, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

1 shows a use diagram of an electronic device providing search results according to various embodiments.

First, as shown in FIG. 1 (a), the electronic device A can display a screen including an object 111 (for example, a building). The object may be, for example, one object among a plurality of objects included in the content 110. In this case, a situation may arise where the user U of the electronic device A wants to additionally know information related to the object 111. [ For example, the user U may want to know the name or use of the building, which is the object 111. [ In addition, the screen including the object 111 may be an execution screen of the application.

1 (b), the user U can select the object 111. In this case, For example, the user U may long press one point of the display area where the object 111 is displayed. Alternatively, the user U may multi-touch or strongly touch an object using a finger or an electronic pen to select the object 111, draw the periphery of the object 111, You can drag it diagonally. Alternatively, the user U may touch the object 111 after pressing (or pressing) a button provided on the electronic device A (e.g., a button for executing an artificial intelligence function). Alternatively, the user may select the object 111 using a predefined action. Specific examples thereof will be described below through various embodiments.

The electronic device A may sense a user input that selects a point. In response to the user input, the electronic device A may capture a screen including the object 111 to generate a captured image. The captured image may be stored, for example, in the memory of electronic device A. [

At this time, the electronic device A may display the generated captured image instead of the screen including the object 111. [

Next, the electronic device A can detect an object region (or a first region) associated with the object 111 in the captured image based on the touch coordinates corresponding to the user input. At this time, the detected object area may be highlighted or displayed as a pop-up screen. Here, highlighted display may mean that the selected object area is displayed to be distinguished from other areas. For example, highlighting and displaying may include display of different shades, different contrasts, or complementary colors, or displaying the boundary of the object area in a dotted or solid line, or displaying an indicator indicating the object area, and the like can do. Specific examples in which the object region is highlighted will be described below through various embodiments.

Next, the electronic device A can acquire information (or first information) about the object included in the object area based on the detected object area, and acquires the object area of the captured image from the surrounding area (Or second information) about the object on the basis of the first area and the second area. At this time, the electronic device A can acquire information on the object and context information through the learned model. This will be described later in detail.

The electronic device A can obtain the search result associated with the object 111 using the information about the object 111 and the context information 121 obtained according to the selection of the object 111. [

At this time, the context information may be information acquired according to the selection of the object, and may include surrounding information about the peripheral region of the object on the screen including the object 111 at the time when the user selects the object 111. [ The peripheral information of the object 111 may be another object (for example, text or image) located close to the selected object, or may be an object included in the document displayed as the current display among the entire document including the object 111 It may be some text.

As another example, if the screen including the object 111 selected by the user is one image, the peripheral information acquired according to the selection of the object 111 may include positional information Or time information, information about other objects than the selected one of the images, or additional information about other objects, and the like.

Alternatively, when the user selects the object 111, the context information may include shooting information shot by a camera provided in the electronic device A. [ Here, "When the user selects the object 111" means that the user may select the object 111, or may select the object 111 within a predetermined time (for example, 1 to 10 seconds) .

On the other hand, the electronic device A can acquire the search result associated with the object 111 by using the context information as well as the use history information of the electronic device A of the user 1. Alternatively, the electronic device A may obtain a search result associated with the object 111 using the profile of the user of the electronic device A. [

In Fig. 1 (c), the electronic device A can display a search result 131 associated with the acquired object on the screen. As a method of displaying the search result on the screen, the search result may be displayed in a part of the display area where the object is not displayed so as not to overlap with the selected object. In this case, the display position of the object can be changed so that the object and the search result do not overlap. For example, if the search result is displayed in the lower area of the display, the display position of the object is changed so that the object is displayed in the upper area of the display. If the search result is displayed in the upper area of the display, Can be changed.

In addition, the search result can be displayed on a screen separate from the screen on which the object is displayed. For example, a separate pop-up window is displayed on the screen on which the object is displayed, and the search result can be displayed through the window. In this case, the display position of the pop-up window is determined according to the position of the selected object, and the size of the pop-up window is determined by the size of the object on the screen and the size of the screen excluding the object on the entire screen. Even when the search result is displayed on a screen separate from the screen on which the object is displayed, the pop-up window can be displayed so as not to overlap with the object.

Of course, a separate screen displaying the search result may be displayed instead of the screen on which the object is displayed. In this case, an image of the selected object and a search result associated with the object may be displayed on a separate screen displaying the search result. In this case, the background part of the separate screen can be displayed transparently so that the screen including the object can be seen.

Search results associated with an object may include, for example, detailed information about the object, advertising information associated with the object, purchase information of the object, other object information associated with the object, and the like. Specifically, detailed information about an object can be collected through a web document search in a newspaper article through a search engine, a document uploaded to the SNS, a document disclosed in a website, and the like. Further, the advertisement information related to the object can be collected at a website providing advertisement of goods and services related to the object, or at a website of a company or a company providing a service for manufacturing goods. Purchase information related to an object can also be collected through a search of goods and services on a web site that sells goods and services through a search engine. Further, other object information associated with the object may be collected through a search corresponding to the object in the search engine and a search using another query having a high degree of similarity and relevance.

In addition, the search results associated with the object may be comprised of various content (e.g., text, images, videos, etc.) and UI elements (e.g., icons, hyperlinks, etc.). For example, if the selected object is a "building ", the search results associated with the object may include textual content and image content for guiding the selected object, and control actions (e.g., Etc.), and may include a hyperlink for accessing a web page including additional information (e.g., location information, tour information) for the selected object.

Further, the search result associated with the object may be changed by user input. For example, when the selected object is "hotel ", a screen (for example, a pop-up screen) including" detailed information about the hotel " When a user input (e.g., a drag input) is received while the "detailed information on the hotel" is being provided, the screen including the "detailed information about the hotel" is removed and a new screen Can be provided.

Further, the search result associated with the object may be provided not only as visual information, but also as a combination of at least one of auditory, tactile, and olfactory information. Specific examples of search results associated with an object are described below through various embodiments.

According to various embodiments, the electronic device A uses the object area and the surrounding information of the object as input data of the learned object recognition model to generate the object information (for example, the object name, the position of the object, Another object, an object name, a subject of a document including the object, etc.) and additional information related to the object, and obtain the search result associated with the object using the object information reflecting the additional information.

Alternatively, the electronic device A may acquire general object information (e.g., an object name, an object ID, an object type, an object attribute, etc.) using the object area as input data of the learned object recognition model, The search results associated with the object can be obtained using general object information and separately collected context information (e.g., location information of the object, other objects around the object, the subject of the document containing the object, etc.). The object recognition model learned in this document can be constructed considering the application field of the recognition model or the computer performance of the device. For example, the learned object recognition model can be set to estimate the object information that reflects the context by using the object region and the surrounding information of the object as input data. The learned object recognition model may be, for example, a model based on a neural network. The object recognition model may include a plurality of weighted network nodes that may be designed to simulate human brain structures on a computer and simulate the neurons of a human neural network. The plurality of network nodes may each establish a connection relationship so that the neurons simulate synaptic activity of the neurons sending and receiving signals via synapses. The object recognition model may also include, for example, a neural network model or a deep learning model developed in a neural network model. In the deep learning model, a plurality of network nodes are located at different depths (or layers) and can exchange data according to a convolution connection relationship. Examples of the object recognition model include, but are not limited to, Deep Neural Network (DNN), Recurrent Neural Network (RNN), and Bidirectional Recurrent Deep Neural Network (BRDNN).

In addition, the electronic device A may use an artificial intelligence agent to retrieve information related to the object selected by the user as described above. At this time, the artificial intelligence agent is a dedicated program for providing an artificial intelligence (AI) -based service (for example, speech recognition service, secretarial service, translation service, search service, CPU) or a separate AI dedicated processor (e.g., GPU, etc.). In particular, an artificial intelligence agent can control various modules to be described later.

Specifically, when the object 111 is selected on the screen by a predetermined user input (for example, a long press or the like) or a button provided on the electronic device A (for example, a button for executing the AI agent The artificial intelligence agent may be operated when the object 111 is selected after being pressed. The artificial intelligent agent captures a screen including the object 111 to generate a captured image and detects an object area associated with the object 111 in the captured image based on the touch coordinates corresponding to the user input , The search result associated with the object 111 can be obtained and provided using the detected object area and the context information 121 (for example, the surrounding information of the object and the like) obtained according to the selection of the object 111 have.

Of course, if a specific icon is touched on the screen or a button provided on the electronic device A (e.g., a button for executing the AI agent) is pressed, the AI agent may operate. In this case, the artificial intelligent agent captures the currently displayed screen to generate a captured image, and then, based on the user's input for selecting the input object, (111) associated with the object (111), acquires a search result associated with the object (111) using the context information (121) acquired according to the detected object region and the selection of the object (111) Can be provided.

The artificial intelligence agent may be a previously executed state in which a preset user input to the object 111 is sensed or a button provided in the electronic device A is selected. In this case, after predetermined user input is detected for the object 111 or a button provided on the electronic device A is selected, the AI agent of the electronic device A captures a screen, Information can be retrieved and provided. For example, if an AI agent is executed by an AI-only processor, a predetermined user input to the object 111 is detected or a button provided in the electronic device A is selected by the general purpose processor The function of the electronic device A can be executed by the AI dedicated processor after a predetermined user input to the object 111 is detected or a button provided in the electronic device A is selected .

In addition, the artificial intelligence agent may be in a standby state previously selected when a preset user input to the object 111 is detected or a button provided in the electronic device A is selected. Here, the standby state is a state in which a predefined user input is received to control the start of operation of the AI agent. If a predetermined user input is detected for the object 111 while the artificial intelligence agent is in the standby state or a button provided for the electronic device A is selected, the electronic device A operates the artificial intelligence agent, It is possible to retrieve and provide related information of the object 111 after capturing a screen using an intelligent agent.

In addition, the artificial intelligence agent may be in a state in which a preset user input to the object 111 is sensed or a button previously provided in the electronic device A is selected. When the predetermined user input to the object 111 is detected or the button provided in the electronic device A is selected in a state where the AI agent is terminated, the electronic device A executes the AI agent, It is possible to retrieve and provide related information of the object 111 after capturing a screen using an intelligent agent.

Meanwhile, the artificial intelligence agent can control various devices or modules to be described later. This will be described in detail later.

Further, specific examples of acquiring search results associated with an object using the learned object recognition model between the electronic device A and the server will be described below through various embodiments.

2A shows a block diagram of an electronic device A according to various embodiments. The electronic device A includes a display 110, a memory 120, a user input 130, a communication unit 150 and a processor 140, as shown in Fig. The configurations shown in FIG. 2 are illustrations for implementing embodiments of the present disclosure, and appropriate hardware / software configurations of a level obvious to those skilled in the art may be further included in the electronic device A.

The display 110 may provide various screens. In particular, the display 110 may provide a screen that includes objects. In addition, the display 110 may further provide a screen (e.g., a pop-up screen) containing search results related to the object on the screen including the object. Or the display 110 may separately provide a screen including search results related to the object.

Memory 120 may store instructions or data related to at least one other component of electronic device A. [ In particular, the memory 120 may be implemented as a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 120 is accessed by the processor 140 and read / write / modify / delete / update of data by the processor 140 and the like can be performed. The term memory in this disclosure includes memory 120, a memory card (not shown) (e. G., Micro SD (not shown)) mounted in ROM (not shown), RAM Card, memory stick). The memory 120 may store programs and data for configuring various screens to be displayed in a display area of the display unit 110. [

In addition, the memory 120 may store an artificial intelligence agent for providing information related to the selected object, and may store the recognition model (object recognition model, peripheral information recognition model, face recognition model, etc.) of the present disclosure.

The memory 120 also includes a search action detection module 141, a screen capture / coordinate acquisition module 142, a user UI module 143, an object region detection / classification module 144, A search result acquisition module 145, and a search result acquisition module 146.

The user input unit 130 may receive various user inputs and transmit them to the processor 140. In particular, the user input 130 may include a touch sensor, a (digital) pen sensor, a pressure sensor, or a key. The touch sensor can use, for example, at least one of an electrostatic type, a pressure sensitive type, an infrared type, and an ultrasonic type. The (digital) pen sensor may be, for example, part of a touch panel or may include a separate recognition sheet. The key may include, for example, a physical button, an optical key, or a keypad.

In particular, the user input unit 130 may acquire an input signal according to a user input for touching an object after a predetermined user touch (for example, a long press touch) or a specific button is pressed to select an object. The user input unit 130 may transmit an input signal to the processor 140.

The communication unit 150 can perform communication with an external electronic device or a server. At this time, the communication unit 150 may transmit a captured image to an external server or transmit information on the object region and context information (e.g., peripheral information of the object). In addition, the communication unit 150 may receive information about the captured image or the object region and a search result for the object in response to the context information.

The processor 140 may be electrically coupled to the display 110, the memory 120 and the user input 130 to control the overall operation and functionality of the electronic device A. [ In particular, the processor 140 includes a search action detection module 141, a screen capture / coordinate acquisition module 142, a user UI module 143, an object region detection / classification module 144, a face detection module 145, And retrieve information related to the object selected by the user using the result acquisition module 146 or the like.

In detail, the processor 140 may use the user input unit 130 to obtain an input signal according to a user input for selecting an object on the screen displayed on the display 110. In response to the input signal, the processor 140 may capture a screen including the object to generate a captured image, and store the generated captured image in the memory 130. In addition, the processor 140 acquires a search result associated with the object using the object region corresponding to the object included in the captured image and the context information obtained according to the selection of the object, So that the display 110 can be controlled.

In addition, the processor 140 may control the communication unit 150 to transmit at least a part of the captured image to an external server. At this time, the processor 140 may control the communication unit 150 to transmit information (for example, coordinate information of the object region, etc.) together with the selected object region. The processor 140 receives the search result from the external server through the communication unit 150 based on the object information about the object included in the captured image and the additional information related to the object, So that the display 110 can be controlled. The description of the processor 140 will be described later in more detail in FIG. 2B.

Figure 2B shows block diagrams of a network system including an electronic device A, according to various embodiments.

The network system may include an electronic device A, a recommendation device B, an object recognition device C, a user characteristic recognition device D and a data collection device E.

At this time, the recommendation apparatus B, the object recognition apparatus C, the user character recognition apparatus D and the data collection apparatus E are distinguished from each other for convenience of explanation. The recommendation apparatus B, the object recognition apparatus C, At least some of the modules included in the user characteristic recognition device D and the data collection device E may be combined into one or more servers (for example, a cloud composed of a plurality of devices). At least some of the modules included in each of the recommendation device B, the object recognition device C, the user characteristic recognition device D and the data collection device E may be provided in the electronic device A. [

Here, the module includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit.

2B, the electronic device A may include a display 110, a memory 120, a user input 130, a communication unit 150, a camera 160 and a processor 140. The display 110, the memory 120, the user input unit 130, and the communication unit 150 have been described in detail with reference to FIG. 2A, and a detailed description thereof will be omitted. At least some of the processors 140 may operate with a plurality of modules that are distinguished according to a particular purpose or function. The software for the processor 140 to operate with a plurality of modules may be stored in the memory 120. [ In this case, the memory 120 may store a plurality of instructions for the processor 140 to operate with each of the plurality of modules.

The processor 140 includes a search action detection module 141, a screen capture / coordinate acquisition module 142, a user UI module 143, an object region detection / classification module 144, a face detection module 145, Module 146, as shown in FIG.

The search action detection module 141 obtains an input signal according to a user input (e.g., long touch, multi-touch, pen action, etc.) for selecting an object included in the screen displayed on the display 110 through the user input unit 130 can do.

When the search action detection module 141 obtains the input signal, the screen capture / coordinate acquisition module 142 may capture the screen to generate a captured image and collect coordinates corresponding to a point on the display 110 . For example, the screen capture / coordinate acquisition module 142 may capture a screen displayed on the display 110 to generate a captured image and collect coordinates corresponding to one point of the display 110 selected by the user.

As another example, when an input signal is obtained through a physical button or a touch sensor on the bezel as the user input unit 130, the search action detection module 141 generates a captured image by capturing a screen displayed on the display 110, The coordinates corresponding to one point of the display 110 selected by the user may be collected according to the input signal additionally obtained through the user input unit 130. [

The object region detection / classification module 144 may detect an object region associated with the object in the captured image, and store the object region in the memory 120, based on the captured image and the selected selected coordinates. At this time, the detected object area may include, for example, an image corresponding to the object area in the captured image. In addition, the object region detection / classification module 144 may determine the category (or object class) of the object corresponding to the detected object region. For example, the object region detection / classification module 144 may determine the category of the detected object region as one of face, goods, clothes or food.

In various embodiments, the object region detection / classification module 144 may optionally be provided in one of the object recognition devices C. [ In this case, the object region detection / classification module 201 of the object recognition apparatus C may correspond to the object region detection / classification module 144 of the electronic device A. [

The object region detection / classification module 144 may be divided into a plurality of modules. In this case, the object region detection module is configured to operate in the electronic device A, and the object region classification module can be configured to operate in an external device such as an object recognition device C and the like.

The face detection module 145 controls the camera 160 provided in the electronic device A to obtain an image including the user's face using the electronic device A, to detect the user's face from the image, (E.g., a facial image, a facial image, and the like) for the facial image of the user. For example, the face detection module 145 may acquire facial information as context information obtained according to the selection of the object and store the facial information in the memory 120.

The search result acquisition module 146 may transmit the detected object region and / or context information (e.g., peripheral information or facial information) to at least one external device through the communication unit 150. [ For example, the search result acquisition module 147 transmits the object region of the captured image detected through the object region detection / classification model 144 to the object recognition device C, Facial information on the facial face can be transmitted to the user character recognition device (D).

The search result acquiring module 146 may acquire the search result associated with the object as a result of transmitting the object area and the context information to the external device. For example, the search result acquisition module 146 may obtain search results associated with an object from the recommendation device B. [

The user UI module 143 may control the display 110 to provide the search results associated with the object obtained in the search result acquisition module 146 to the user.

The object recognition apparatus C may include an object region detection / classification module 201, a detailed object recognition module 202, and a context recognition module 203.

When receiving the captured image and the coordinates selected by the user from the electronic device A, the object region detection / classification module 201 detects the object region selected by the user in the captured image based on the captured image and the selected coordinates, The object category (or object class) of the object area. For example, the object region detection / classification module 201 may determine the category of the detected object region as one of face, goods, clothes or food.

The object region detection / classification module 201 may be configured separately, and the object region detection module may be configured to operate in the electronic device A, and the object region classification module may be configured to operate in the object recognition device C have. In this case, the object area detection module of the electronic device detects the object area selected by the user in the captured image, and then transmits the information on the object area to the object recognition device (C), and the object area classification module The object category of the object area can be determined using the information about the object area received from the object area A.

The object recognition apparatus C may have a plurality of detailed object recognition modules corresponding to each of a plurality of object categories.

In this case, when the object category is determined, the detailed object recognition module 202 corresponding to the object category can recognize the detected object area in more detail. For example, the detailed object recognition module 202 may recognize an object region and obtain object information related to the object region. Specifically, when the object category related to the object region detected by the object region detection / classification module 201 is a face, the detailed object recognition module 202 can recognize the face of a person as object information.

In particular, the object recognition apparatus C may estimate the object information related to the object region using the detected object region as input data of the learned object recognition model. The learned object recognition model can be, for example, a neural network or a deep learning based recognition model. In addition, the object recognition apparatus C may estimate the object information related to the object region by analyzing the detected object region through a rule-based object recognition model.

In various embodiments, the object region detection / classification module 201 may be optionally provided in one of the electronic device A and the object recognition device C. In this case, the object region detection / classification module 201 of the object recognition apparatus C may correspond to the object region detection / classification module 144 of the electronic device A. [ For example, the object region detection / classification module 201 may be provided in the electronic device A, and the detailed object recognition module 202 may be provided in the object recognition device C. Alternatively, both the object region detection / classification module 201 and the detailed object recognition module 202 may be provided in the electronic device A or the object recognition device C.

In addition, the object recognition apparatus C may include a peripheral information recognition module 203 for recognizing peripheral information related to the object.

The peripheral information recognition module 203 can acquire peripheral information related to the object. For example, the peripheral information recognition module 203 may acquire, as peripheral information, information related to an object among peripheral information located around the object. For example, when the selected object 111 is a word or an image included in a document displayed on the screen, the peripheral information obtained based on the selection of the object 111 is information extracted from the screen on which the object is displayed, Or information that is most useful to distinguish the object 111 from other similar objects may be determined by the rectangle information associated with the object.

The peripheral information recognition module 203 can determine (or estimate, determine) peripheral information by using the peripheral information recognition model, which is a learned model. For example, the object recognition apparatus C can determine the peripheral information related to the object by using the image of the peripheral region of the object of the screen on which the object is displayed as the input data of the learned peripheral information recognition model. The learned peripheral information recognition model may be, for example, a neural network or a deep learning based recognition model.

In addition, the peripheral information recognition module 203 can determine (or estimate, determine) peripheral information using a rule-based peripheral information recognition model. For example, the object recognition apparatus C can analyze the image of the peripheral region of the object on the screen on which the object is displayed through predetermined rules to determine the peripheral information related to the object.

In one embodiment, a word having the highest degree of association between the object and the surrounding information can be obtained as the surrounding information through the surrounding information recognition model. At this time, the peripheral information recognition model may be a learned model using the object and surrounding information of the object as learning data. For example, the word with the highest degree of relevance between the object and the surrounding information is the surrounding information having the highest probability based on the probability that the object and the surrounding information are displayed on one screen or the probability that the surrounding information is searched together with the object Can be determined.

For example, if the object selected by the user is an image of the 'Eiffel Tower' or the text of the 'Eiffel Tower', the perimeter information recognition module 203 applies the perimeter information of the object to the perimeter information recognition model, ',' History 'and' architectural information 'can be determined as the object with the highest degree of relevance and the object named' Eiffel Tower '.

When the object selected by the user is the image of the 'famous singer' or the text of the 'famous singer', the peripheral information recognition module 230 applies the peripheral information of the object to the peripheral information recognition model, The album and the song title can be determined as the object related to the famous singer and the surrounding information having the highest degree of relevance.

In another embodiment, information that may be most usefully used to distinguish the object 111 from other similar objects may be obtained through a peripheral information recognition model. One example of information that can be most usefully used to distinguish the object 111 from other similar objects is a word having a positive / negative attribute for the object. The peripheral information recognition model may be a learned model using words having positive / negative attributes among information related to objects and objects as learning data. A word with a positive / negative attribute is information about a feature whose object can be distinguished from other objects. In this case, the peripheral information recognition model may be the same recognition model as the peripheral recognition model for estimating the association between the object and the surrounding information, or may be a different recognition model.

For example, if the object selected by the user is an image of an 'automobile' or a text of an 'automobile', the peripheral information recognition module 203 determines that the object selected by the user is' It is possible to determine the words 'fuel economy', 'sales volume', and 'price', which are related to the object and have positive / negative attributes, as peripheral information about the object.

When the object selected by the user is the image of the 'household appliances' or the text of the 'household appliances', the peripheral information recognition module 203 applies the peripheral information of the object to the peripheral information recognition model, Quot ;, " mounted function ", or " price ", which are related to the object, i.e., the object having the positive / negative attribute, can be determined as the surrounding information about the object.

Another example of information that can be most usefully used to distinguish the object 111 from other similar objects is additional identification information of the object. The additional identification information of the object can be obtained as peripheral information that can be distinguished from other objects by inputting information about the object using the learned peripheral information recognition model. At this time, a plurality of additional identification information can be obtained, and based on the probability, information that is most useful among a plurality of additional identification information is selected. In this case, a plurality of peripheral information is acquired from the image of the peripheral region of the object among the images displayed on the object, and the peripheral information having a high probability based on the probability among the acquired peripheral information is selected as the additional identification information.

If the object selected by the user is a text corresponding to a 'person's face' image or a 'person's name', the peripheral information recognition module 230 applies the peripheral information of the object to the peripheral information recognition model, Occupation ',' sex 'and' age 'as additional identification information of the object.

If the object selected by the user is an image of 'car' or 'product', the peripheral information recognition module 203 applies' peripheral ',' model name ', and' 'Specification' can be determined as additional identification information of the object.

In the case where the object selected by the user is an image of 'building' or 'place', the peripheral information recognition module 203 applies the peripheral information of the object to the peripheral information recognition model as 'building name or place name' Name or country name " and " other landmark " may be determined as additional identification information of the object.

In various embodiments, the perimeter information recognition model may be learned such that the perimeter information of the object is determined in consideration of the type of object. At this time, a peripheral information recognition model can be learned to determine information that is most useful for distinguishing objects that are similar to objects.

In addition, the surrounding information recognition model can be instructed so that the type of the surrounding information of the object is determined in consideration of the type of the object so that the detailed information suitable for the object type can be retrieved. In addition, the degree of association between the object and the surrounding information may be learned through the non-background learning based on the frequency of the surrounding information included in the document or the image including the object. In addition, the surrounding information recognition model can be learned through reinforcement learning using user feedback on the search result.

The user character recognition device D can receive facial information from the electronic device A. [ For example, the user characteristic recognition device D may acquire facial information about the user's face detected by the face detection module 146 of the electronic device A from the electronic device A through the communication unit 150 of the electronic device A, Lt; / RTI >

The user characteristic recognizing device D can determine the characteristic of the user from the received facial information and store the characteristic information of the user. The user's feature information may include, for example, the user's age, gender or facial expression.

In particular, the user character recognition apparatus D may estimate the user characteristic information related to the face information by using the face information as input data of the learned face recognition model. The learned face recognition model may be, for example, a neural network or a deep learning based recognition model.

The additional data collecting device E can collect, for example, the user profile of the electronic device A, the usage history information of the electronic device, or the user's feedback from the electronic device A

The user profile may include at least one of the user's name, gender, ID, preference category and biometric information (e.g., key, weight, medical history, etc.) as user information previously registered in the electronic device A by the user have. The use history information may include information such as a user's object search history, a feedback history of a search result, or a product purchase history, for example. User feedback may include, for example, at least one of user feedback on search results associated with an object or user feedback on object information. Specifically, the user's feedback on the search results associated with the object may include, for example, whether to additionally select search results associated with the object, whether the user has access to the website associated with the object, Whether or not the user has purchased the goods related to the object, etc. The user feedback on the object information includes, for example, information on whether or not the user confirms the object information, user's correction information on the object information, Object reselection information of the object.

The recommendation apparatus B can provide search results associated with an object using at least one of object information, context information, and context recognition information. For example, the recommendation apparatus B can use at least one of the object information, the context information, and the context recognition information received by the object recognition apparatus C, and the user character recognition apparatus D and / The feature information and / or the additional data of the user received in step (E) may be used together.

More specifically, the recommendation apparatus B can request a separate search engine to search using the received object information, context information, and context recognition information. The recommendation apparatus B can provide the retrieved result received from the search engine to the electronic device A by applying the priority order. For example, the recommendation apparatus B may apply the priority to the search result using the feature information of the user and / or the additional data and provide it to the electronic device A. [ That is, the context information may be used to obtain a search result associated with an object, and may be used to prioritize search results by applying a priority to search results associated with the object.

The search engine is software operating on a server used / maintained / managed by a search service provider. The search engine receives object information and / or context information from an external device (for example, recommendation device B) as a search word, , And can retrieve data related to a query word among the data collected and stored in advance using the received query word, and provide the retrieved data to an external device.

As the recommendation device B provides search results using at least one of context information of the object, feature information of the user, and additional data as well as object information, a search result matching the user's intention can be provided.

The camera 160 can take a picture. In particular, the camera 160 may acquire an image including the user's face by photographing the face of the user using the electronic device A, in order to obtain the face image of the user as the context information.

3 shows a flow diagram of a network system including an electronic device A according to various embodiments.

First, in FIG. 3, the user of electronic device A may select an object included in the screen (301). For example, a user may select a work object by performing a predefined action or an action (e.g., long touch, multi-touch, pen action, etc.) that the user has defined.

The screen capture / coordinate acquisition module 142 of the electronic device A captures a screen including the object in response to a user input and displays coordinates corresponding to a point selected by the user on the display 110 (303). ≪ / RTI >

The electronic device A may send the captured image captured and the collected coordinates to the object recognition device C (305). At this time, the object recognition device C may be provided in the electronic device A or may be provided in a separate server communicating with the electronic device A.

The object region detection / classification module 201 of the object recognition apparatus C can detect an object region corresponding to the selected coordinates in the captured image and classify the object region based on the received captured image and the selected coordinates ( 307). At this time, the category of the object may be one of, for example, a person, a character, a commodity, a place, a plant or a food, and may be defined differently depending on an application service provided by the electronic device A.

In various embodiments, the object region detection / classification module 144 of the electronic device A detects an object region corresponding to the selected coordinates in the captured image based on the received captured image and the selected coordinates, can do. In this case, the electronic device A may transmit the object area and the object category to the object recognition device C.

In various embodiments, the object area detection module of the electronic device A may detect the object area and the electronic device A may transmit the detected object area to the object recognition device C. In this case, the object region classification module of the object recognition apparatus C may determine the object category of the object region.

The object region detection / classification module 201 can transfer the object region to a detailed object recognition module among a plurality of different detailed object recognition modules in consideration of the classified object category. For example, the detailed object recognition module may include a face object recognition module, a product object recognition module, a character object recognition module, a place object recognition module, and a food object recognition module.

The detailed object recognition module 202 of the object recognition apparatus C may acquire object information by recognizing the object region more finely than the category of the object classified by the object region detection / classification module 201 (309). For example, the face object recognition module can obtain the name of a person as object information. Alternatively, the merchandise object recognition module can recognize the ID of the merchandise as object information.

Meanwhile, the detailed object recognition module 202 may perform more detailed classification and acquire object information based on the classified detailed category. For example, the merchandise object recognition module may classify merchandise categories into subcategories such as fashion, bag, shoes, hat, packaged goods, or fresh food. Then, the detailed object recognition module 202 can recognize the object within the detailed category to acquire the object information.

When the object information is obtained, the object recognition apparatus C may transmit the acquired object information to the context analysis / recommendation apparatus G (311). The context analysis / recommendation device G may correspond to at least one of the recommendation device B of FIG. 2 and the context recognition module of the object recognition device C described above.

Also, the context analysis / recommendation device G may acquire the context information acquired according to the selection of the object (313). For example, the context analysis / recommendation apparatus G may include context information obtained according to the selection of the object transmitted from the electronic device A, as peripheral information located around the object on the screen including the object, It is possible to acquire at least one of the photographing information photographed by the provided camera.

The context analysis / recommendation apparatus G may acquire the recommendation information as a search result associated with the object (315) based on the object information and the context information acquired when the user selects the object. At this time, the context analysis / recommendation device G can obtain optimal recommendation information by analyzing the object information and the context information.

For example, the object information of the object selected by the user is the name of the celebrity, and the context information may be the text information located in the vicinity of the object selected by the user. In this case, the context analysis / recommendation device G may analyze the text information and acquire the concert ticket information as a search result associated with the object if the analyzed context aware information is content related to the concert.

As another example, if the object information is information related to a product, and the usage history information of the electronic device A is history information in which the user continuously selects a product and performs price comparison, the context analysis / And obtain the price comparison information as a search result associated with the search result.

As another example, if the object information is information related to the goods and the use history information of the electronic device A includes the user's preference information or interest information as purchase history information, the context analysis / Information matching the taste of the purchased product (for example, the style of clothes) can be obtained as a search result associated with the object.

As another example, the context information may be shot information shot by a camera provided in the electronic device A. [ In this case, the context analysis / recommendation device G may be an object that considers the age, the appearance, the sex, or the mood of the user in the case of information indicating the age, appearance, sex, facial expression or mood of the user, Can obtain search results associated with < RTI ID = 0.0 >

As another example, the object information may be product related information, and the context information may be text information located around the object. At this time, if the price information is already included in the text information, the context analysis / recommendation device G determines that the user is more interested in the detailed information such as the specification of the product than the price information on the product, Related specification information can be obtained.

Next, the context analysis / recommendation device G may transmit the search result associated with the object to the electronic device A as recommendation information (317). At this time, the recommendation information may include a customized search result related to an object matching the user's selection intention. The electronic device A may display the search result associated with the object in the display area as the received recommendation information (319).

4 is a flow diagram of a network system including an electronic device A according to various embodiments.

4, operations 401 to 405 for transferring the captured image and the collected coordinates to the object recognition apparatus C according to the selection of the user's object correspond to the operations 301 to 305 in Fig. 3 And redundant explanations are omitted.

On the other hand, the electronic device A may transmit the captured image and the collected coordinates as well as the context information to the object recognition device C. At this time, the context information may be, for example, peripheral information (e.g., text information or image information) located around the object selected by the user.

The object recognition apparatus C can detect and classify the object region (409). In addition, the object recognition apparatus C can recognize the object region in detail considering the classified object category (411). At this time, the object recognition apparatus C can improve the recognition result by recognizing the object region using the context information received in operation 407 (413).

The object recognition apparatus C may acquire object information as an improved recognition result and transmit the acquired object information to the context analysis / recommendation apparatus G (415).

When the context analysis / recommendation device G transmits the search result associated with the object to the electronic device A based on the received object information, the electronic device A can display the search result in the display area. Operations 417 to 421 corresponding to these operations correspond to operations 315 to 319 of FIG. 3, and redundant explanations are omitted.

5 is a flow diagram of a network system including an electronic device A according to various embodiments.

5, the object recognition apparatus C may transmit the recognition result (e.g., object information) of the object region corresponding to the object selected by the user to the electronic apparatus A (501).

In response, the electronic device A may send 505 the user feedback on the recognition result to the object recognition device C (503).

For example, the electronic device A can acquire the recognition result of the object area received by the object recognition device C and display it on the display area. The user can provide user feedback on the displayed recognition result. For example, the user can input the response feedback to the pop-up asking whether to confirm the recognition result.

When the object recognition apparatus C recognizes the object using the learned object recognition model, the object recognition model that recognizes the object using the received response feedback may be updated (505). In this case, the more the user feedback based on the recognition result is, the more the recognition function of the object recognition model can be continuously improved.

In various embodiments, even if there is no user feedback, the object recognition apparatus C can use the captured image or the object area transmitted by the electronic device A for object recognition to generate an object recognition model in a map / It can also be learned.

The object recognition apparatus C may transmit the recognition result (e.g., object information), which is an output value of the object recognition model, to the context analysis / recommendation apparatus G (507).

In addition, the electronic device A may transmit at least one of the context information, the user profile, and the usage history information to the data collection device E (509-513).

The usage history information may include usage history information on the search result, for example, as usage history information of the electronic device (A). The usage history information on the search result may include, for example, at least one of a purchase history of purchasing a product based on the search result, a search result evaluation history of the user, and a search history of searching for a search result or a connection link . Further, the data collection device E can collect user characteristic information on the search result. The user characteristic information at this time may be information analyzed based on the information photographed by the camera after providing the search result, and may include information indicating the user's facial expression or mood of the search result.

The collected data (e.g., context information, user profile, usage history information, or user characteristic information) collected by the data collection device E may be transmitted 515 to the context analysis / recommendation device G. FIG. The data collection device E may send the data to the context analysis / recommendation device G periodically (e.g., hourly or daily), or may send the context analysis / (G).

The context analysis / recommendation apparatus G may receive the object information received from the object recognition apparatus C at operation 517 and the collected data (e.g., context information, user profile, user characteristic information, History of use, etc.), a search result associated with an object corresponding to the user's intention can be obtained.

At this time, the context / recommendation apparatus G can request the search engine to search the object based on the object information received from the object recognition apparatus C and the collected data received from the data collection apparatus E. [ The context / recommendation device G may provide the retrieved result received from the search engine to the electronic device A by applying a priority order. For example, the context / recommendation device G acquires the search result using the object information through the search engine and searches the search result based on the use history information (for example, purchase history information, search history information, To the electronic device A by applying the priority to the electronic device A. In other words, the context / recommendation device G can provide or recommend to the electronic device A by applying a high priority to search results with history of use. As another example, the use of the user's electronic device A When the history includes usage histories (e.g., purchase histories, search histories, recognition result satisfaction, etc.) for recognition results similar to the recognition results of the object regions, the context analysis / recommendation apparatus G uses It is possible to determine the user's main field of interest with respect to the result and to provide or recommend the customized information conforming to the user's intention to the electronic device A based on the determination result.

Next, the context analysis / recommendation device G transmits 519 the search result associated with the acquired object to the electronic device A, and the electronic device A can display the received search result in the display area (521).

FIG. 6 is a diagram illustrating a process of providing search results associated with objects according to various embodiments.

6A, the electronic device A may sense a user input for selecting an object included in the screen using a touch action or a pen action. In response to the user input, electronic device A may generate a captured image 610 by capturing a screen including the object, as shown in FIG. 6B. At this time, the electronic device A can perform screen capturing through a background thread.

6 (c), the electronic device A may detect the object area 611 based on the touch coordinates corresponding to the user's selection in the captured image 610. [ Then, the electronic device A recognizes the object area 611 and classifies it into one object category. Then, the electronic device A can transmit the object area to the detailed object recognition device H related to the category classified for detailed object recognition. The detailed object recognition device H may correspond to, for example, the detailed object recognition module 202 of FIG.

Further, the electronic device A can transmit the photographing information photographed by the electronic device A to the user characteristic recognizing device D. [ The user characteristic recognizing device D can recognize the user characteristic based on the received photographing information. For example, the user character recognition device D can recognize the age, sex, facial expression, etc. of the user.

The detailed object recognition apparatus H can acquire object information (for example, an object name, an object identifier, and an object level of detail) by recognizing an object corresponding to the object region. Then, the detailed object recognition device H can transmit the object information to the detailed information acquisition device I.

The detailed information obtaining apparatus I can obtain detailed information on the object as the primary search result related to the object information based on the received object information. Detailed information about the object may include general information of the object or purchase information. The general information may include, for example, information such as the name, age, name, sex, or career of the person, if the object is a person. In addition, the purchase information may include goods related to the person (for example, a concert ticket, a book, a record, a VOD, or the like) or information capable of purchasing it.

The detailed object recognition apparatus H may acquire detailed information on the object information from the detailed information acquisition apparatus I and transmit the detailed information to the context analysis / recommendation apparatus G. [ Alternatively, the detailed information obtaining apparatus I may send detailed information on the object information directly to the context analysis / recommendation apparatus G. [

The context analysis / recommendation device G includes object information or detailed information received at the detailed object recognition device H, user property information received at the user property recognition device D, user profile received at the electronic device A, The recommendation information may be acquired as a secondary search result using at least one of the context information. For example, the context analysis / recommendation device G may include peripheral information (e.g., text information, image information) located around the object selected by the user, user characteristic information recognized based on the shooting information (e.g., ) And the use history data 621 of the electronic device A, based on the user's preference and interest information analyzed.

At this time, the context / recommendation device G includes object information and detailed information received from the detailed object recognition device H, user property information received from the user property recognition device D, user profile information received from the electronic device A, , And may request the search engine to search based on at least one of the context information. The context / recommendation device G may provide the user with a preferred search result for the retrieved result received from the search engine to the electronic device A. [ For example, the context / recommendation device G may obtain search results using object information and detailed information through a search engine, and may include user characteristics information (e.g., user gender, age, mood, (E.g., a user preference field, a user search history, a user purchase history, and the like) and context information, and provides the recommendation result to the electronic device A or recommends the recommendation result. Alternatively, the context / recommendation device G may determine or prioritize the search results based on at least one of the user characteristic information, the user profile, and the context information to provide or recommend to the electronic device A.

Next, the context analysis / recommendation device G may transmit the search result associated with the object, which is the obtained recommendation information, to the electronic device A. [

The electronic device A can provide the user with a search result 631 associated with the received object through the screen.

Figure 7 is a diagram that provides search results associated with an object using perimeter information according to various embodiments.

In Figure 7, electronic device A may provide search results 703 associated with an object based on the selected object 701 and perimeter information (e.g., text or images, etc.) 702 around the object. For example, a newspaper article including an image and text may be displayed on the electronic device A. In this case, when the user selects a specific artist's face 701 included in the image as an object, the object recognition apparatus C analyzes the selected face in the image and acquires the name of the specific artist corresponding to the face as object information .

As a result of analyzing the text of the newspaper article containing the image, the object recognition apparatus C can acquire the keyword " year-end concert " 702 among the entire contents of the newspaper article as the context recognition information related to the specific artist . In this case, the context analysis / recommendation apparatus G can use each of the 'specific artist name' and 'year-end concert' received from the object recognition apparatus C as a query. For example, the context analysis / recommendation device G may request the search engine to search for relevant information using the queries. Then, the search result based on the query can be received from the search engine. The object recognition apparatus C may selectively provide the received search result using characteristic information of the user, additional data, or the like.

In order to derive the context recognition information related to the object, the object recognition apparatus C may infer the entire contents of the document including the object or a word having the highest degree of association with the object, can do. In addition, the object recognition apparatus C can deduce a place or a time at which the image including the object is estimated to be photographed. In this case, the object recognition apparatus C can deduce the context recognition information related to the object by using the peripheral information of the object as input data of the learned peripheral information recognition model. The learned peripheral information recognition model may be, for example, a neural network or a deep learning based recognition model.

Figure 8 shows use diagrams of an electronic device for selecting objects according to various embodiments.

For example, as shown in FIG. 8A, a user can touch (e.g., a long touch or a double touch) a point 801 of a display area in which an object is displayed in order to select an object. Alternatively, as shown in FIG. 8B, the user can draw a portion 803 of the display area in which the object is displayed using the electronic pen.

Alternatively, as shown in FIG. 8 (c-1), the electronic device A may display a UI (e.g., icon) 805 that provides search results associated with objects on the screen. For example, when the user withdraws the electronic pen from electronic device A, electronic device A displays UI 805 that provides search results associated with the object, based on events generated as a result of the withdrawal of the electronic pen can do. In another example, in response to a user input that is dragged to the center of the display area (e.g., an edge area), the electronic device A may display a UI 805 that provides search results associated with the object.

In this situation, when the user selects the UI 805, the electronic device A captures a screen including the object, which is being displayed in the display area before displaying the UI, and, as shown in (c-1) The electronic device A can display the captured image as a result of capture of the screen. When the captured image is displayed, for example, the user can select an object by drawing one area 807 of the captured image with the electronic pen.

9A and 9B are diagrams that provide search results associated with objects in accordance with various embodiments.

9A, when the user selects the building 901 as an object, the electronic device A displays detailed information 902 about the building as a search result associated with the object in a part of the display area can do. In this case, when the building is a building that can be accommodated, the electronic device A may display a UI element 903 capable of making a room reservation or the like as a search result.

As another example, in (b) of FIG. 9A, if the user selects celebrity 905, electronic device A may display profile information 906 for celebrities as part of the search in a portion of the display area. At this time, the electronic device A may display a UI element 907 that may provide other search results associated with the selected object.

For example, when the object is a celebrity, the electronic device A may display a UI element capable of selecting a search category such as a user profile, broadcast progress information, movie appearance information, album information, or performance information as a search result. In this case, when a user input for selecting broadcast progress information is received, the electronic device A can display a search result related to the broadcast progress information on a part of the display area.

The UI elements for selecting the category information may be displayed in a sequential order according to the priority of the search categories. For example, if there are a plurality of search categories matching the user's intention as a search result associated with the object, the UI elements that can select respective category information according to the priority most suitable for the user's intention can be sequentially listed.

As another example, in FIG. 9A (c), the user may select text 908 as an object. For example, if the text 908 is a movie title, then the electronic device A may retrieve at least one of the movie related content, rating, thumbnail image, teaser image and movie reserve UI element as search results 909 associated with the text 908 Can be displayed on a part of the display area.

As another example, in (d) of Fig. 9B, the user can select an object 911 by dragging diagonally using the electronic pen. In this case, the electronic device A can highlight and display the object area 912 corresponding to the diagonal drag. Then, the electronic device A can display detailed information 913 about the object as a search result associated with the object 911 in a part of the display area. For example, the electronic device A can be regarded as a commodity, and the commodity price, the seller, the similar commodity, etc. related to the object can be displayed.

As another example, as shown in (e) of FIG. 9B, the user can draw along the boundary of the object 914 using the electronic pen. In this case, the electronic device A can highlight and display the segmented object region 915 corresponding to the drawing input. Then, the electronic device A can display detailed information 916 on the object as part of the display area as search results associated with the object.

Figure 10 is a diagram showing searching for search results associated with objects according to various embodiments.

As shown in (d) of FIG. 9B, when the user selects an object, the electronic device A, together with the object selected as the transition effect, searches for a search result associated with the object A message 1001 (e.g., searching) may be displayed on the screen. Alternatively, as shown in FIG. 9B (e), when the user selects an object, the electronic device A, together with the selected object, as shown in FIG. 10B, indicates that the search result associated with the object is being searched The message 1002 can be displayed on the screen.

In this way, when displaying the object area and the search status message corresponding to the user and the selected object, the user can be provided with an effect of shortening the time until the search result associated with the object is searched.

11 is a diagram that provides search results associated with objects in accordance with various embodiments.

For example, in FIG. 11A, a user may perform a user input for capturing a screen 1102 including an object 1101. 11B, the electronic device A generates a captured image 1103 and performs text recognition (e.g., an optical character reader (OCR)) on the generated captured image 1103, Or image recognition may be performed to detect text 1104 and object region 1105.

Then, the electronic device A can provide search results associated with the object using the detected object region and text.

For example, in FIG. 11A, if the screen 1102 is a web page including the object 1101, the electronic device A analyzes the captured web page based on the text recognition and the image recognition, And display the product information 1106, which is a search result associated with the object, in the display area by using the text 1104 and the text 1104 as the context information. The item information 1106 may include, for example, a UI element (not shown) capable of purchasing a product associated with the object. In this case, in response to the selection of the UI element, the electronic device A may display the goods purchase screen in the display area.

12 is a diagram for providing search results associated with an object using photographing information according to various embodiments.

In Fig. 12A, when the user selects the object 1201, the electronic device A can photograph a user who selects an object using a camera (e.g., a front camera).

The electronic device A can acquire the characteristic information of the user by using the photographing information generated according to the photographing by the user. For example, the electronic device A can recognize the sex, age, facial expression, mood, etc. of the user based on the photographing information. Next, the electronic device A may obtain search results associated with the object using the object region corresponding to the selected object and the user's feature information. Then, the search result 1203 related to the object can be displayed in the display area as shown in FIG. 12 (b).

Accordingly, the search result considering the characteristics and the state of the user is provided, and the user satisfaction using the electronic device A can be increased.

Figure 13 is a diagram that provides search results associated with objects in accordance with various embodiments.

In Fig. 13, when electronic device A provides a search result associated with an object according to this document, it may be provided with a search result on a pop-up screen.

In this case, as shown in Fig. 13A, the pop-up screen 1301 gradually appears from one side of the display area, and may be displayed in a part of the predefined display area as shown in Fig. 13B . At this time, the pop-up screen 1301 may be displayed in the display area so as not to overlap the objects.

In this case, the display position of the pop-up screen 1301 is determined according to the position of the selected object, and the size of the pop-up screen may be determined according to the size of the object on the screen and the size of the screen excluding the object on the entire screen. Alternatively, the background portion of the pop-up screen may be displayed transparently to provide a view of the screen containing the object.

In a method (not shown) of displaying a search result on a screen, a search result may be displayed in a partial area of a screen including an object so as not to overlap with the selected object. In this case, the display position of the object can be changed so that the object and the search result do not overlap. For example, if the search result is displayed in the lower area of the screen, the object is displayed in the upper area of the screen. If the search result is displayed in the upper area of the screen, the object is displayed in the lower area of the screen So that the display position of the object can be changed.

14 is a diagram that provides a plurality of search results associated with an object according to various embodiments.

In Fig. 14, the electronic device A can acquire search results associated with objects according to the document by search category. For example, if the object selected by the user is a hotel building, the search category may include "hotel information", "hotel visitor", "hotel location", and the like. In this case, the electronic device A may sequentially provide search results corresponding to the search category based on the user's preference in accordance with the user's intention.

For example, in FIG. 14 (a), electronic device A may display top ranking search results (e.g., hotel information) 1401 as search results associated with an object. At this time, when a user input (for example, a swipe input) is received, the electronic device A can display a search result (e.g., a hotel visitor) 1402 of the subordinate as shown in Fig. 14B.

Subsequently, when the user input is received, the electronic device A can display the next ranking search result (e.g., hotel location) 1403, as shown in Fig. 14C. Then, when additional user input is received, the electronic device A can display an input window 1404 in which information (for example, a hotel use inquiry) can be transmitted, as shown in (d) of FIG. In this case, when the user creates an inquiry in the input window and selects the send button, the electronic device A can transmit the input inquiry to an external device (for example, an administrator mail server) related to the object.

15 is a diagram that provides search results associated with objects in accordance with various embodiments.

In Fig. 15A, the electronic device A can display the search result 1501 related to the object acquired in accordance with the document in the display area. At this time, if the search result 1501 related to the object is a result related to one category among a plurality of search categories, identification information 1502 to 1505 of each of a plurality of other search categories may be displayed in a part of the display area have.

At this time, in response to the user input 1503 for selecting one piece of identification information among the plurality of pieces of identification information 1502 to 1505, the electronic device A, as shown in FIG. 15B, (1511) can be displayed in the display area. At this time, the identification information 1502, 1504, 1505, and 1506 of other categories except for the selected category may be displayed in a part of the display area.

16A-16C are diagrams that provide search results associated with an object selected by a user during execution of various applications, in accordance with various embodiments.

(E.g., a long press touch, a double tap touch, a horizontal drag touch, a strong pressure touch, an area where an object is displayed with a closed curve), which selects an object included in an execution screen of an application while the application is running , The electronic device A can capture the execution screen of the currently displayed application, identify the selected object, and retrieve the related information. Or when a user inputs a specific icon included in the execution screen of the application or a specific button included in the electronic device A and selects an object is detected while the application is being executed, Capture the run screen, identify the selected object, and retrieve relevant information.

At this time, the electronic device A may provide a screen (for example, a pop-up screen) including a search result for the object selected in the upward direction from below on the execution screen of the application.

The provided search result may be acquired for each of a plurality of search categories, and may include information on user characteristics (for example, user age, sex, mood, etc.) and user profile (e.g., user search history, Information, etc.), the priority for a plurality of search categories can be determined.

Then, the search result provides detailed information for a search category having a high priority, and can provide a menu for another search category. For example, if the selected object is a "hotel ", detailed information is provided for the" hotel late ", which is a high priority search category, and the detailed information such as "hotel reservation "," For a search category, a search result may be provided by a menu (or an icon).

Then, the electronic device A can update the user profile by generating feedback information based on user selection and user emotion states for a plurality of search categories.

In one embodiment of the present invention, while the web browser application is running, the electronic device A may receive user input to select an object displayed on the execution screen of the web browser application according to the user input. At this time, the user input includes a long press touch input for touching the displayed object over a predetermined time, a double tap touch input for touching the displayed object a plurality of times, a drag input for horizontally dragging an area including the displayed object, A touch input for touching a displayed object with a predetermined pressure or more, and a touch input for drawing an area including a displayed object into a closed curve. 16A, the electronic device A can display the search button 1610 on the execution screen of the web browser application. After the search button 1610 is selected by the user And receive user input selecting the object. In addition, the electronic device A may receive user input to select an object after a specific button (e.g., a button for executing an AI agent) is selected.

When the user input as described above is received, the electronic device A can capture an execution screen of the displayed web browsing application to generate a captured image.

Then, the electronic device A can obtain a search result for the selected object based on the object information about the object selected by the user among the captured images. At this time, the electronic device A can move the pop-up screen 1620 including the search result of the selected object in the upward direction from the downward direction as shown in (b) and (c) of FIG. The electronic device A may include a pop-up screen 1620 on the execution screen of the web browsing application, as shown in (c) of FIG. 16. That is, the electronic device A can remove the execution screen of the web browsing application on the display 110. [

In particular, the pop-up screen 1620 may include a plurality of categories (e.g., purchase information category, related news category, late category, etc.). At this time, detailed information is provided for the purchase information category having a high priority determined based on the user profile information, and only the related icon is provided for the related news category and the late category determined to have a low priority.

The electronic device A can update the user feedback information by reflecting user selection results for a plurality of categories.

In addition, the electronic device A may use the user's voice together to obtain search results for the selected object. For example, when the user's voice "Where can I buy" is entered during the selection of the object, the electronic device A determines the "purchase information category" as the search category for the selected object, You can provide search results based on categories.

In another embodiment of the present invention, while the gallery application is being executed, the electronic device A, as shown in FIG. 16B (a), selects a user input Lt; / RTI > At this time, since the user input is the same as described with reference to FIG. 16A, a duplicate description will be omitted.

When the user input is received, the electronic device A can capture an execution screen of the displayed gallery application to generate a captured image.

Then, the electronic device A can obtain a search result for the selected object based on the object information about the object selected by the user among the captured images. In this case, if the selected object includes text, the electronic device A can recognize the text included in the object using OCR, and use the recognized text as object information. Alternatively, the electronic device A can use metadata (e.g., photographing position information, photographing time information, etc.) of an image included on the execution screen of the gallery application as context information. For example, when the selected object is the Eiffel Tower, the electronic device A can recognize that the object selected by the user is the Eiffel Tower by using the location information included in the metadata of the image as "France, Paris ".

The electronic device A moves the pop-up screen 1640 including the search result of the selected object from the down direction to the upper direction and displays the pop-up screen 1640 in the lower area of the execution screen as shown in (b) Can be provided.

In particular, a plurality of categories (for example, detailed information category, peripheral information category, related image category, etc.) may be included on the pop-up screen 1640. [ At this time, the detailed information category having a high priority determined based on the user profile information is provided with detailed information, and only the icon of the related peripheral information category and the related image category having a low priority is provided.

The electronic device A can update the user feedback information by reflecting user selection results for a plurality of categories.

In another embodiment of the present invention, while the camera application is running, the electronic device A may receive user input to select an object displayed on the run screen of the gallery application according to user input. At this time, since the user input is the same as described with reference to FIG. 16A, a duplicate description will be omitted.

Further, the electronic device A can display the AR function button 1650 on the execution screen of the camera application as shown in FIG. 16C (a), and when the AR function button 1650 is selected by the user After that, as shown in FIG. 16 (b), a user input for selecting an object can be received.

When the user input is received, the electronic device A can capture an execution screen of the displayed camera application to generate a captured image. However, the electronic device A can use an image including an object among captured images without capturing a screen separately.

Then, the electronic device A can obtain a search result for the selected object based on the object information about the object selected by the user among the captured images. In this case, if the selected object includes text, the electronic device A can recognize the text included in the object using OCR, and use the recognized text as object information. Alternatively, the electronic device A can use metadata of the shot image (e.g., shooting location information, shooting time information, and the like) as the context information.

The electronic device A moves the pop-up screen 1660 including the search result of the selected object from the down direction to the upper direction and displays the pop-up screen 1660 in the lower area of the execution screen as shown in (c) Can be provided.

In particular, a plurality of categories (for example, detailed information category, peripheral information category, related image category, etc.) may be included on the pop-up screen 1640. [ At this time, the detailed information category having a high priority determined based on the user profile information is provided with detailed information, and only the icon of the related peripheral information category and the related image category having a low priority is provided.

The electronic device A can update the user feedback information by reflecting user selection results for a plurality of categories.

Figure 17 is a flow diagram that provides search results associated with an object in conjunction with an external device in accordance with various embodiments.

First, the electronic device A can display a screen including an object (1701).

In this case, when a user input for selecting an object is sensed (1702-Y), the electronic device A may capture a screen including the object to generate a captured image (1703). At this time, the captured image can be displayed in the display area instead of the screen including the object. In addition, the object may be detected from the captured image based on the touch coordinates corresponding to the user's input.

Then, the electronic device A can transmit at least a part of the captured image to the external device (1704). At this time, the electronic device A may transmit information (for example, coordinates of the object area) about the object area corresponding to the selected object together.

Then, the electronic device A may receive the retrieved search result from the external device based on the object information corresponding to the object included in the captured image and the additional information related to the object (1705).

At this time, the object information is information obtained by recognizing the object region corresponding to the object of at least a part of the captured image, and the additional information may be information obtained by recognizing the peripheral region of the object region among at least a part of the captured image. Particularly, the object information is information obtained by applying the object region to the learned recognition model set to estimate the object information, and the additional information is information obtained by applying the learned information to the learned recognition model set to estimate the additional information through the surrounding region .

The retrieved retrieval result may be the retrieval result based on the retrieval category obtained by applying the object information and the additional information to the learned recognition model set to estimate the retrieval category. At this time, the searched search result includes at least one of the history information of the electronic device A, the user profile of the user of the electronic device A, and the shooting information shot by the camera provided in the electronic device A, As shown in FIG.

Next, the electronic device A can display the obtained search result in the display area (1706). At this time, the electronic device A can display one side of the display area so that the search result does not overlap with the object. Further, the electronic device A can display the object information together with the search result. Further, the electronic device A can transmit user feedback according to the display of the search result or the object information to the external device.

18 is a block diagram showing a configuration of an electronic device according to various embodiments.

18, the electronic device A includes an image acquisition unit 1810, an image processing unit 1820, a display unit 1830, a communication unit 1840, a memory 1850, an audio processing unit 1860, An output 1870, a user input 1880, and a processor 1890. On the other hand, the configuration of the electronic device A shown in Fig. 18 is merely an example, and is not necessarily limited to the above-described block diagrams. Therefore, it goes without saying that part of the configuration of the electronic device A shown in Fig. 18 may be omitted, modified or added depending on the kind of the electronic device A or the purpose of the electronic device A.

The image acquisition unit 1810 may acquire image data through various sources. For example, the image acquiring unit 1810 may be implemented by a camera to acquire photographing information by photographing an external environment. The image data obtained through the image acquiring unit 1810 may be processed in the image processing unit 1820. [

The image processing unit 1820 may process the image data received by the image obtaining unit 1810. The image processor 1820 may perform various image processes such as decoding, scaling, noise filtering, frame rate conversion, or resolution conversion on the image data.

The display unit 1830 may display the image data processed by the image processing unit 1820 on a display area (or a display). The display area may refer to at least a portion of the display portion 1830 exposed on one side of the housing of the electronic device A. [

At least a portion of the display portion 1830 may be coupled to at least one of a front region and a side region and a rear region of the electronic device A in the form of a flexible display. The flexible display can be characterized by being able to bend, bend or speak without damage through a thin, flexible substrate such as paper.

The display unit 1830 may be implemented as a touch screen having a layer structure in combination with the touch panel 1881. The touch screen not only has a display function but also a function of detecting a touch input position and a touch area as well as a touch input pressure and has a function of detecting not only a real touch but also a proximity touch .

The communication unit 1840 can perform communication with various types of external devices according to various types of communication methods. The communication unit 1840 may include at least one of a Wi-Fi chip 1841, a Bluetooth chip 1842, a wireless communication chip 1843, and an NFC chip 1844. The processor 1890 can communicate with an external server or various external devices using the communication unit 1840. [

The memory 1850 can store various programs and data necessary for the operation of the electronic device A. [ The memory 1850 may be implemented as a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). Memory 1850 is accessed by processor 1890 and read / write / modify / delete / update of data by processor 1890 can be performed. The term memory in this document refers to a memory 1850, a ROM (not shown), a RAM (not shown) or a memory card (not shown) (e.g. micro SD Card, memory stick).

In the memory 1850, a program and data for configuring various screens to be displayed in the display area of the display unit 1830 may be stored. The memory 1850 may also store various recognition models of the document.

The audio processing unit 1860 is a component that performs processing on audio data. In the audio processing unit 1860, various processes such as decoding and amplification of audio data, noise filtering, and the like may be performed. The audio data processed in the audio processing unit 1860 may be output to the audio output unit 1870.

The audio output unit 1870 outputs various kinds of audio data as well as various kinds of notification sounds and voice messages that are subjected to various processing operations such as decoding, amplification, and noise filtering by the audio processing unit 1860. In particular, the audio output unit 1870 may be implemented as a speaker, but it may be implemented as an output terminal capable of outputting audio data, which is only an example.

The user input 1880 can receive various user inputs and communicate them to the processor 1890. The user input portion 1880 may include, for example, a touch panel 1881, a (digital) pen sensor 1882 or a key 1883. As the touch panel 1881, for example, at least one of an electrostatic type, a pressure sensitive type, an infrared type, and an ultrasonic type can be used. Further, the touch panel 1881 may further include a control circuit. The touch panel 1881 may further include a tactile layer to provide a tactile response to the user. (Digital) pen sensor 1882 may be part of, for example, a touch panel or may include a separate recognition sheet. Key 1883 may include, for example, a physical button, an optical key, or a keypad.

The processor 1890 (or the control unit) can control the overall operation of the electronic device A by using various programs stored in the memory 1750. [

The processor 1890 may comprise a RAM 1891, a ROM 1892, a graphics processor 1893, a main CPU 1894, first through n interfaces 1895-1 through 1895-n, have. At this time, the RAM 1891, the ROM 1892, the graphics processor 1893, the main CPU 1894, the first through n interfaces 1895-1 through 1895-n, etc. may be connected to each other via a bus 1896 .

19A, the processor 1900 may include at least one of a learning unit 1910 and a recognition unit 1920. [ The processor 1900 of FIG. 19A may correspond to the processor 1890 of the electronic device A of FIG. 18 or a processor of a data learning server (not shown).

The learning unit 1910 can generate or learn a recognition model having a criterion for a predetermined situation judgment. The learning unit 1910 can generate a recognition model having a determination criterion using the collected learning data.

For example, the learning unit 1910 may generate, learn, or update an object recognition model having a criterion for determining an object included in an image using an image including the object as learning data.

As another example, the learning unit 1910 may generate a peripheral information recognition model having a criterion for determining various additional information around an object included in an image, using peripheral information included in a screen including the object as learning data, Or updated.

As another example, the learning unit 1910 can generate, learn, or update a face recognition model having a criterion for determining a face of a user included in an image using an image photographed by a camera as learning data.

The recognition unit 1920 can use the predetermined data as the input data of the learned recognition model to estimate the recognition target included in the predetermined data.

For example, the recognition unit 1920 acquires (or estimates, estimates) object information on an object included in the object region using the object region (or image) including the object as input data of the recognition model, can do.

As another example, the recognition unit 1920 may apply at least one of the object information and the context information to the learned recognition model to estimate (or determine, or deduce) a search category to provide a search result. At this time, a plurality of search results may be obtained according to the priority order.

As another example, the recognition unit 1920 may apply the context information (e.g., the peripheral information of the object) to the learned recognition model to generate context recognition information (e.g., additional information related to the object, etc.) Can be estimated.

At least a portion of the learning unit 1910 and at least a portion of the recognition unit 1920 may be implemented in a software module or in the form of at least one hardware chip and mounted in an electronic device. For example, at least one of the learning unit 1910 and the recognition unit 1920 may be fabricated in the form of a dedicated hardware chip for artificial intelligence (AI), or a conventional general purpose processor (e.g., a CPU or an application processor or a graphics-only processor (e.g., a GPU), and may be mounted on the above-described various electronic devices or object recognition devices. At this time, the dedicated hardware chip for artificial intelligence is a dedicated processor specialized for probability calculation, and it has a higher parallel processing performance than conventional general purpose processors, so that it can quickly process computation tasks in artificial intelligence such as machine learning. When the learning unit 1910 and the recognition unit 1920 are implemented by a software module (or a program module including an instruction), the software module may be a computer-readable and non- transitory computer readable media. In this case, the software module may be provided by an operating system (OS) or provided by a predetermined application. Alternatively, some of the software modules may be provided by an Operating System (OS), and some of the software modules may be provided by a predetermined application.

 In this case, the learning unit 1210 and the recognition unit 1220 may be mounted on one electronic device or on separate electronic devices, respectively. For example, one of the learning unit 1210 and the recognition unit 1220 may be included in the electronic device A0, and the other may be included in an external server. The learning unit 1910 and the recognition unit 1920 may provide the model information constructed by the learning unit 1910 to the recognition unit 1920 through wired or wireless communication, Data may be provided to the learning unit 1910 as additional learning data.

19B is a block diagram of a learning unit 1910 and a recognition unit 1920 according to various embodiments.

Referring to FIG. 19B, a learning unit 1910 according to some embodiments may include a learning data obtaining unit 1910-1 and a model learning unit 1910-4. The learning unit 1910 may further include at least one of the learning data preprocessing unit 1910-2, the learning data selecting unit 1910-3, and the model evaluating unit 1910-5.

The learning data acquisition unit 1910-1 can acquire learning data necessary for a recognition model for reasoning an object to be recognized. In the embodiment of the present document, the learning data obtaining unit 1910-1 may obtain at least one of the entire image including the object, the image corresponding to the object region, the object information, and the context information as learning data. The learning data may be data collected or tested by the learning unit 1910 or the manufacturer of the learning unit 1910.

The model learning unit 1910-4 can use learning data to learn such that the recognition model has a determination criterion as to how to determine a predetermined recognition target. For example, the model learning unit 1910-4 can learn the recognition model through supervised learning using at least some of the learning data as a criterion. Alternatively, the model learning unit 1910-4 may learn, for example, self-learning by using learning data without any guidance, so that the model learning unit 1910-1 may perform learning by using unsupervised learning, . Also, the model learning unit 1210-4 can learn the recognition model through reinforcement learning using, for example, feedback as to whether the result of the situation judgment based on learning is correct. Also, the model learning unit 1910-4 can learn the recognition model using, for example, a learning algorithm including an error back-propagation method or a gradient descent

Also, the model learning unit 1910-4 may learn a selection criterion as to which learning data should be used to estimate the recognition target using the input data.

The model learning unit 1910-4 can determine a recognition model having a large relation between the input learning data and the basic learning data as a recognition model for learning when there are a plurality of recognition models previously constructed. In this case, the basic learning data may be pre-classified according to the data type, and the recognition model may be pre-built for each data type. For example, the basic learning data may be pre-classified by various criteria such as an area where the learning data is generated, a time at which the learning data is generated, a size of the learning data, a genre of the learning data, a creator of the learning data, .

Once the recognition model is learned, the model learning unit 1910-4 can store the learned recognition model. In this case, the model learning unit 1910-4 can store the learned recognition model in the memory 1750 of the electronic device A. [ Alternatively, the model learning unit 1910-4 may store the learned recognition model in the memory of the server connected to the electronic device A via a wired or wireless network.

The learning unit 1910 includes a learning data preprocessing unit 1910-2 and a learning data selection unit 1910-3 in order to improve the analysis result of the recognition model or to save resources or time required for generation of the recognition model .

The learning data preprocessing section 1910-2 can pre-process the acquired data so that the acquired data can be used for learning for situation determination. The learning data preprocessing unit 1910-2 can process the acquired data into a predetermined format so that the model learning unit 1910-4 can use the data acquired for learning for the situation determination.

The learning data selection unit 1910-3 can select data acquired by the learning data acquisition unit 1910-1 or data necessary for learning from data preprocessed by the learning data preprocessor 1910-2. The selected learning data may be provided to the model learning unit 1910-4. The learning data selection unit 1910-3 can select learning data necessary for learning from the acquired or preprocessed data according to a predetermined selection criterion. The learning data selection unit 1910-3 may also select learning data according to a predetermined selection criterion by learning by the model learning unit 1910-4.

The learning unit 1910 may further include a model evaluation unit 1910-5 to improve the analysis result of the data recognition model.

The model evaluating unit 1910-5 inputs the evaluation data to the recognition model and allows the model learning unit 1910-4 to learn again if the analysis result output from the evaluation data does not satisfy the predetermined criterion . In this case, the evaluation data may be predefined data for evaluating the recognition model.

For example, the model evaluating unit 1910-5 satisfies a predetermined criterion when the number or ratio of the evaluation data whose analysis result is not correct out of the analysis results of the learned recognition model for the evaluation data exceeds a preset threshold value It can be evaluated as not doing.

On the other hand, when there are a plurality of learned recognition models, the model evaluation unit 1910-5 evaluates whether each of the learned recognition models satisfies a predetermined criterion, and determines a model satisfying a predetermined criterion as a final recognition model . In this case, when there are a plurality of models satisfying the predetermined criterion, the model evaluating unit 1910-5 can determine any one or a predetermined number of models preset in descending order of the evaluation score as a final recognition model.

Referring to FIG. 19B, the recognition unit 1920 according to some embodiments may include a recognition data obtaining unit 1920-1 and a recognition result providing unit 1920-4.

The recognition unit 1920 may further include at least one of the recognition data preprocessing unit 1920-2, the recognition data selection unit 1920-3, and the model updating unit 1920-5.

The recognition data acquisition unit 1920-1 can acquire data necessary for the situation determination. The recognition result providing unit 1920-4 can determine the situation by applying the data acquired by the recognition data obtaining unit 1920-1 to the learned recognition model as an input value. The recognition result providing unit 1920-4 can provide the analysis result according to the data analysis purpose. The recognition result providing unit 1920-4 can apply the data selected by the recognition data preprocessing unit 1920-2 or the recognition data selecting unit 1920-3, which will be described later, to the recognition model as an input value, have. The results of the analysis can be determined by the recognition model.

In one embodiment, the recognition result providing unit 1920-4 may acquire object information corresponding to the object region by applying a learned recognition model to the object region including the object acquired by the recognition data obtaining unit 1920-1 Or estimated).

In another embodiment, the recognition result providing unit 1920-4 applies at least one of the object region, the object information, and the context information acquired by the recognition data obtaining unit 1920-1 to the learned recognition model to provide a search result (Or estimate) a search category to do

The recognizing unit 1920 recognizes the recognition data preprocessing unit 1920-2 and the recognition data selecting unit 1920-3 in order to improve the analysis result of the recognition model or to save resources or time for providing the analysis result .

The recognition data preprocessing section 1920-2 can preprocess acquired data so that the acquired data can be used for the situation determination. The recognition data preprocessing section 1920-2 can process the acquired data into a predefined format so that the recognition result providing section 1920-4 can use the data obtained for the situation determination.

The recognition data selection unit 1920-3 can select the data acquired by the recognition data acquisition unit 1920-1 or the data necessary for the situation determination among the data preprocessed by the recognition data preprocessor 1920-2. The selected data may be provided to the recognition result provider 1920-4. The recognition data selection unit 1920-3 can select some or all of the obtained or preprocessed data according to a predetermined selection criterion for the situation judgment. The recognition data selection unit 1920-3 can also select data according to a predetermined selection criterion by learning by the model learning unit 1910-4.

The model updating unit 1920-5 can control the recognition model to be updated based on the evaluation of the analysis result provided by the recognition result providing unit 1920-4. For example, the model updating unit 1920-5 provides the model learning unit 1910-4 with the analysis result provided by the recognition result providing unit 1920-4 so that the model learning unit 1910-4 And may request additional learning or updating of the recognition model.

20 is a diagram showing an example of learning and recognizing data by interlocking with the electronic device A and the server S according to an embodiment.

Referring to FIG. 20, the server S can learn a criterion for determining the situation, and the electronic device A can determine the situation based on the learning result by the server S.

In this case, the model learning unit 1910-4 of the server S can perform the function of the learning unit 1910 shown in Fig. 1919a. The model learning unit 1910-4 of the server S learns which object image, object information or context information to use to determine a predetermined situation, and a criterion on how to determine the situation using the data .

The recognition result delivering unit 1920-4 of the electronic device A applies the data selected by the recognition data selecting unit 1920-3 to the recognition model generated by the server S, Can be determined. Alternatively, the recognition result providing unit 1920-4 of the electronic device A may receive the recognition model generated by the server S from the server S and determine the situation using the received recognition model . In this case, the recognition result delivering unit 1920-4 of the electronic device A applies the object image selected by the recognition data selecting unit 1920-3 to the recognition model received from the server S, The corresponding object information can be determined. Alternatively, the recognition result providing unit 1920-4 may determine a search category in which the search result is to be obtained by using at least one of the context information and the context recognition information.

21 is a flow diagram of an electronic device using a recognition model according to an embodiment.

Referring to FIG. 21, the electronic device A may display a screen including an object (2101).

When the user input for selecting an object is sensed (2102-Y), the electronic device A may capture a screen including the object to generate a captured image (2103).

Next, the electronic device A may acquire a search result associated with the object using at least one of object information corresponding to the object area included in the captured image and additional information about the object corresponding to the peripheral area (2104 ).

At this time, the result of applying at least one of the object region and the surrounding information to the recognition model may be at least one of, for example, object information, additional information about the object, and a search category. The electronic device A may obtain search results related to the object using at least one of object information, additional information about the object, and a search category. For example, the electronic device A may send object information, additional information, and a search category to the recommendation device B and obtain search results related to the object in response thereto.

Once the search results are obtained, the electronic device A may display 2105 the search results associated with the acquired objects in the display area.

22 to 25 are flowcharts of a network system using a recognition model according to another embodiment.

22 to 25, the network system using the recognition model includes a first component 2201, 2301, 2401, 2501, a second component 2202, 2302, 2402, 2502, and a third component 2203, 2303 , 2403).

Here, the first components 2201, 2301, 2401, and 2501 may be the electronic device A, and the second components 2202, 2302, 2402, and 2502 may be the server S that stores the recognition model. Alternatively, the first components 2201, 2301, 2401, and 2501 may be general purpose processors, and the second components 2202, 2302, 2402, and 2502 may be dedicated artificial intelligence processors. Alternatively, the first component 2201, 2301, 2401, 2501 can be at least one application and the second component 2202, 2302, 2402, 2502 can be an operating system have. That is, the second components 2202, 2302, 2402, and 2502 may be more integrated, dedicated, delay less, Or a component having a large number of resources, can be a component that can process many operations required to create, update, or apply the data recognition model more quickly and effectively than the first component 2201, 2301, 2401, 2501.

In this case, an interface for transmitting / receiving data between the first component 2201, 2301, 2401, 2501 and the second component 2202, 2302, 2402, 2502 may be defined.

For example, an application program interface (API) having an argument value (or an intermediate value or a transfer value) of learning data to be applied to the recognition model may be defined. An API is a set of subroutines or functions that can be called for any processing of any protocol (e.g., a protocol defined in the electronic device A) to another protocol (e.g., a protocol defined in the server S) Can be defined. That is, it is possible to provide an environment through which an operation of another protocol can be performed through any one of the protocols through the API.

On the other hand, the third components 2203, 2303, and 2403 are based on data received from at least one of the first component 2201, 2301, 2401, 2501 and the second component 2202, 2302, 2402, To obtain search results associated with the object. The third components 2203, 2303, and 2403 may correspond to the recommendation device B of FIG. 2B, for example. At this time, the data received by the third components 2203, 2303, and 2403 may be, for example, at least one of an object region, context information, object information, and a search category. Also, according to one embodiment, the third components 2203, 2303, and 2403 may be implemented as a single device with the second components 2202, 2302, 2402, and 2502.

In one embodiment, in FIG. 22, first, the first component 2201 may display a screen including an object (2211).

In this case, when a user input for selecting an object is received, the first component 2201 may capture a screen including the object to generate a captured image (2212).

Next, the first component 2201 acquires (2213) an object region corresponding to the object included in the captured image, and transmits the obtained object region to the second component 2202 (2214).

The second component 2202 can obtain the object information by inputting the received object region as a recognition model (2215).

The second component 2202 may send the object information to the third component 2203 (2216). The third component 2203 may obtain the search result associated with the object using the object information (2217). At this time, the third component 2203 may obtain search results by additionally using additional data (e.g., user characteristic information, user profile, user feedback, etc.) in addition to the object information.

When the third component 2203 sends (2218) the search result associated with the object to the first component 2201, the first component 2201 can display the search result associated with the received object in the display area (2219). The third component 2203 may transmit the search result associated with the object directly to the first component 2201 or through the second component 2202.

In another embodiment, in FIG. 23, first, the first component 2301 may display a screen including an object (2311).

In this case, when a user input for selecting an object is received, the first component 2301 may capture a screen including the object to generate a captured image (2312).

The first component 2301 may transmit the captured image to the second component 2302 (2313). At this time, the first component 2301 may transmit information about the object region corresponding to the selected object together with the captured image.

The second component 2302 may separate the received captured image into an object region and a surrounding region (2314). At this time, the second component 2302 can separate the object region and the surrounding region based on the information on the received object region. In another embodiment, the first component 2301 may separate the captured image into an object region and a surrounding region and transmit the captured image to a second component 2302. [

The second component 2302 can input the object information and the additional information about the object by inputting the separated object region and the surrounding region as recognition models (2315). At this time, the second component 2302 can obtain the object information by inputting the object region into the object recognition model, and input the peripheral region to the peripheral information recognition model to obtain additional information about the object. In addition, the second component 2302 can determine the priorities of the search category and the search category based on the object information and the additional information about the object.

Alternatively, the second component 2302 acquires object information by inputting the object region as a recognition model, and acquires additional information through an application resource analysis of a portion corresponding to the peripheral region of the screen including the object . In this case, the second component 2302 can determine the priority of the search category and the search category based on the object information and the additional information about the object.

Application resource analysis refers to parsing the type, content (eg, image or text content) and screen location of a UI (user interface) element displayed on the execution screen when the application is executed. The application resource analysis of the portion corresponding to the peripheral region uses the position on the screen of the UI element on the application execution screen at the time of generating the captured image to grasp the type and content of the UI element displayed at the position corresponding to the peripheral region .

The second component 2302 can acquire additional information about the object by grasping the type and content of the UI element. In this case, the first component 2301 may acquire additional information about the object by grasping the type and content of the UI element, and may transmit the additional information to the second component 2302.

The second component 2302 may send 2316 the object information and the additional information as a result of applying the recognition model of operation 2315 to the third component 2303. At this time, the second component 2302 may transmit information on the search category together with the third component 2303.

The third component 2303 may obtain the search result associated with the object using the received object information and the additional information (2317). At this time, the third component 2303 may use the search category together to obtain the search result. In addition, the third component 2303 may additionally use additional data (e.g., user characteristic information, user profile, user feedback, etc.) in addition to object information and additional information to obtain search results. At this time, the additional data may be transmitted from the first component 2301 or another element or stored in the third component 2303.

When the third component 2303 sends (2318) the search result associated with the object to the first component 2301, the first component 2301 can display the search result associated with the received object in the display area (2319).

In another embodiment, in FIG. 24, first, the first component 2401 may display a screen including an object (2411).

In this case, when a user input for selecting an object is received, the first component 2401 may capture a screen including the object to generate a captured image (2412).

The first component 2401 can separate the generated captured image into an object area and a surrounding area (step 2413). In this case, the first component 2401 may determine an object region based on the user touch point, and may separate the object region and the surrounding region based on the information on the determined object region. To the second component 2402 (step 2415).

The second component 2402 may acquire object information and additional information about the object by inputting the object region and the surrounding information into the recognition model (2416). At this time, the second component 2402 can obtain the object information by inputting the object region into the object recognition model, and input the peripheral region into the peripheral information recognition model to obtain additional information about the object. In addition, the second component 2302 can determine the priority of the search category and the search category based on the object information and the additional information.

The second component 2402 may transmit the acquired object information and the additional information to the third component 2403 (2417). At this time, the second component 2402 may transmit information on the search category together with the third component 2403.

The third component 2403 may obtain the search result associated with the object using the received object information and the additional information (2418). At this time, the third component 2403 can obtain the search result associated with the object by applying the object information and the additional information to the recognition model as input data. At this time, the third component 2403 may use the search category together to obtain the search result. In addition, the third component 2403 can obtain search results using additional data (e.g., user characteristic information, user profile, user feedback, etc.) in addition to object information and additional information. At this time, the additional data may be transmitted from the first component 2401 or another element or may be stored in the third component 2403 in advance.

When the third component 2403 sends (2419) the search result associated with the object to the first component 2401, the first component 2401 can display the search result associated with the received object in the display area (2420).

In another embodiment, first, in FIG. 25, the first component 2501 may display a screen including an object (2511).

In this case, when a user input for selecting an object is received, the first component 2501 may capture a screen including the object to generate a captured image (2512).

The first component 2501 may transmit the captured image to the second component 2502 (2513). At this time, the first component 2501 may transmit information about the object region corresponding to the selected object together with the captured image.

The second component 2502 may separate the received captured image into an object region and a surrounding region (2514). At this time, the second component 2502 can separate the object region and the surrounding region based on the information on the received object region.

The second component 2502 can input the object information and the additional information about the object by inputting the separated object region and the surrounding region as recognition models (2515). At this time, the second component 2502 can acquire the object information by inputting the object region into the object recognition model, and input the peripheral region into the peripheral information recognition model to obtain additional information about the object. In addition, the second component 2502 can determine the priority of the search category and the search category based on the object information and the additional information about the object.

The second component 2502 may obtain the search results associated with the object using the acquired object information and the additional information (2516). At this time, the second component 2502 may apply the object information and the additional information to the recognition model as input data to obtain the search result associated with the object. At this time, the second component 2502 may use the search category together to obtain the search result. Also, the second component 2502 may obtain search results using additional data (e.g., user feature information, user profiles, user feedback, etc.) in addition to object information and additional information. At this time, the additional data may be transmitted from the first component 2501 or another element or may be stored in the second component 2502 in advance.

When the second component 2502 sends 2517 the search result associated with the object to the first component 2501, the first component 2501 can display the search result associated with the received object in the display area (2518).

25 and 26 are flowcharts for explaining an embodiment in which an electronic device according to another embodiment provides a search result for a first region selected by a user using a recognition model.

In Fig. 25, the electronic device A can display an application execution screen (2610). At this time, at least one object may be included in the application execution screen.

Electronic device A may receive user input 2620 while displaying an application run screen. At this time, the user input may be an input to the application execution screen for selecting the middle object as the application execution screen. In particular, when a user input is received, the electronic device A may execute an AI agent to perform a search for a first area containing the selected object.

The electronic device A may capture an application execution screen according to a user input to generate an image (2630). At this time, the electronic device A can display the generated image in place of the application execution screen.

The electronic device A may display the search result performed using the first information for the first area corresponding to the user input on the generated image and the second information for the second area different from the first area S2640). In this case, the first area may be an area obtained from the image generated based on the touch coordinates corresponding to the user input. The second area may be an area excluding the first area of the generated image.

Also, the first information for the first area and the second information for the second area can be obtained by the learned model. That is, the generated image may be input to the learned first model to obtain the first information, and the first information and the generated image may be input to the learned second model to obtain the second information. At this time, the learned model can be stored in the electronic device (A) or an external server.

In one embodiment, the operation of obtaining the first information and the second information and retrieving the information of the first area may be performed by an external search server. That is, the electronic device A can transmit at least a part of the generated image to the external search server and receive the search result from the external search server.

Further, the electronic device A can display the search result on the application execution screen, and can transmit the user feedback according to the display of the search result to an external device that stores the learned model.

In Fig. 26, the electronic device A may display an application execution screen (2710). At this time, at least one object may be included in the application execution screen.

Electronic device A may capture an application run screen to generate an image (2720). At this time, when a user input for the first area including the object is received from the application execution screen, the electronic device A can capture an application execution screen to generate an image. At this time, the electronic device A can display the generated image on the application execution screen.

The electronic device A may obtain 2730 the first information for the first region through the learned first model using the generated image as input data. At this time, the first model may be stored in the electronic device A, but this may be stored in an external server only in an embodiment.

The electronic device A may acquire 2740 the second information for the second region through the learned second model using the first information and the generated image as input data. At this time, the first model may be stored in the electronic device A, but this may be stored in an external server only in an embodiment.

When the first model and the second model are stored in the external server, the electronic device A transmits the generated image to the external server, and the first information and the image obtained by inputting the image from the external server into the first model, 1 information to the second model and receive the second information.

As described above, by acquiring the first information on the first area in which the user input is sensed as well as the second information on the second area around the first area, it is possible to more accurately search the information on the first area do.

28 and 29 are flowcharts of a system using a recognition model according to various embodiments.

In Fig. 28, electronic device A may display an application execution screen (2810). At this time, at least one object may be included in the application execution screen.

Electronic device A may capture an application run screen to generate an image (2820). Specifically, when a user command for selecting an object is received while the application execution screen is being displayed, the electronic device A can capture an application execution screen to generate an image.

The electronic device A may transmit the generated image to the server S (2830). At this time, the server S stores the learned model and can perform a search function based on the retrieved information.

The server S may acquire the first information for the first region through the learned first model using the generated image as input data (2840). In this case, the first area may be an area including an object detected based on the touch coordinate point on which the user input is received.

The server S may acquire the second information for the second region through the learned second model using the first information and the generated image as input data (2850). In this case, the second area may be a region excluding the first area of the generated image.

The server S may search information related to the first area based on the first information and the second information (S2860). That is, the server S can retrieve information about the first area including the object based on the information about the object and the additional information around the object.

The server S may send 2870 the search results associated with the first domain to the electronic device A and the electronic device A may provide 2880 the received search results.

28, the electronic device A captures an application execution screen and provides the generated image to the server S. The server S inputs the generated image to the learned recognition model The first information and the second information can be obtained, and the information can be retrieved based on the first information and the second information.

Meanwhile, in the above-described embodiment, one server S has acquired the first information and the second information through the learned model and retrieved the information related to the first region, but this is merely an embodiment , And a plurality of servers can divide and carry out the above-described operations. That is, the first server obtains the first information and the second information through the learned model, and the second server retrieves information related to the first region based on the first information and the second information obtained from the first server can do.

In Fig. 28, the electronic device A can display an application execution screen (2910). At this time, at least one object may be included in the application execution screen.

Electronic device A may capture an application run screen to generate an image (2920). Specifically, when a user command for selecting an object is received while the application execution screen is being displayed, the electronic device A can capture an application execution screen to generate an image.

The electronic device A may obtain 2930 the first information for the first region through the learned first model using the generated image as input data.

The electronic device A may transmit the generated image and the first information to the server S (2940).

The server S may acquire the second information on the second region through the learned second model using the first information and the generated image as input data (2950).

The server S may retrieve information related to the first area based on the first information and the second information (2960).

The server S may transmit 2970 the search results associated with the first domain to electronic device A and the electronic device A may provide 2980 the received search results.

That is, in the above-described embodiment, the operation of acquiring the first information through the first model for recognizing the object is performed by the electronic device A, and the second information is obtained through the second model for estimating the context information The server S can perform the processing. That is, the object recognition operation that can perform information even with a small throughput can be performed by the electronic device A, and the context estimation operation requiring a large amount of processing can be performed by the server S

28 and 29, one server S has acquired the first information or the second information through the learned model and retrieved the information related to the first region. However, Only a plurality of servers can perform the operations described above. That is, the first server obtains the first information and the second information through the learned model, and the second server retrieves information related to the first region based on the first information and the second information obtained from the first server can do.

As used herein, the term "module " includes units comprised of hardware, software, or firmware and may be used interchangeably with terms such as, for example, logic, logic blocks, components, or circuits. A module may be an integrally constructed component or a minimum unit or part thereof that performs one or more functions. For example, the module may be configured as an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented in software, including instructions stored on machine-readable storage media, such as a machine (e.g., a computer) (E.g., electronic device A) in accordance with the disclosed embodiments, when the instructions are executed by a processor, the processor may directly or indirectly, Or other components under the control of the processor. The instructions may include code generated or executed by a compiler or an interpreter. Non-transitory storage medium, where 'non-transient' means that the storage medium does not contain a signal, It does not distinguish that data is stored semi-permanently or temporarily on the storage medium.

According to a temporal example, the method according to various embodiments disclosed herein may be provided in a computer program product. A computer program product can be traded between a seller and a buyer as a product. A computer program product may be distributed in the form of a machine readable storage medium (eg, compact disc read only memory (CD-ROM)) or distributed online through an application store (eg PlayStore ™). In the case of on-line distribution, at least a portion of the computer program product may be temporarily stored, or temporarily created, on a storage medium such as a manufacturer's server, a server of an application store, or a memory of a relay server.

Each of the components (e.g., modules or programs) according to various embodiments may be comprised of a single entity or a plurality of entities, and some of the subcomponents described above may be omitted, or other subcomponents May be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity to perform the same or similar functions performed by each respective component prior to integration. Operations performed by a module, program, or other component, in accordance with various embodiments, may be performed sequentially, in parallel, repetitively, or heuristically, or at least some operations may be performed in a different order, .

A: Electronic device C: Object recognition device
D: User characteristic recognition device B: Recommendation device
E: Data collection device S: Server

Claims (20)

A method for providing search results of an electronic device,
Displaying an application execution screen;
Receiving a user input while displaying the application execution screen;
Capturing the application execution screen according to the user input to generate an image; And
Displaying the search result performed using the first information on the first region corresponding to the user input on the generated image and the second information on the second region different from the first region on the generated image
/ RTI >
Wherein the first information and the second information are obtained by a learned model,
How to provide search results.
The method according to claim 1,
Wherein the first region comprises:
Based on the touch coordinates corresponding to the user input,
How to provide search results.
The method according to claim 1,
The method comprises:
An operation of displaying the generated image in place of the application execution screen
Further comprising:
The method according to claim 1,
The method comprises:
What is being done by the execution of an AI agent
How to provide search results.
The method according to claim 1,
Wherein the first information comprises:
And the information obtained by recognizing the first area of the generated image
How to provide search results.
6. The method of claim 5,
Wherein the second information comprises:
The second region of the generated image is recognized as information obtained by recognizing the second region
How to provide search results.
The method according to claim 1,
Wherein the user input comprises:
The input to the application execution screen
How to provide search results.
The method according to claim 1,
The method comprises:
Transmitting the user feedback according to the display of the search result to an external device
Further comprising:
The method according to claim 1,
Transmitting at least a portion of the generated image to an external search server; And
Receiving the search result from the external search server
Further comprising:
The method according to claim 1,
The operation of displaying the search result includes:
An operation for displaying the search result while the application execution screen is displayed
How to provide search results.

In an electronic device,
display;
A user input section;
A communication unit;
A processor electrically connected to the display, the user input and the communication unit; And
And a memory for storing one or more computer programs executed by the processor,
The one or more computer programs,
Displaying an application execution screen through the display;
When receiving a user input while displaying the application execution screen, generating an image by capturing the application execution screen according to the user input; And
Displaying the search result performed using the first information on the first region corresponding to the user input on the generated image and the second information on the second region different from the first region on the generated image And an electronic device.
12. The method of claim 11,
Wherein the first region comprises:
Based on the touch coordinates corresponding to the user input,
Electronic device.
12. The method of claim 11,
The one or more computer programs,
An operation of displaying the generated image in place of the application execution screen
Further comprising:
12. The method of claim 11,
The one or more computer programs,
What is being done by the execution of an AI agent
Electronic device.
12. The method of claim 11,
Wherein the first information comprises:
And the information obtained by recognizing the first area of the generated image
Electronic device.
16. The method of claim 15,
Wherein the second information comprises:
The second region of the generated image is recognized as information obtained by recognizing the second region
Electronic device.
12. The method of claim 11,
Wherein the user input comprises:
The input to the application execution screen
Electronic device.
12. The method of claim 11,
The one or more computer programs,
Transmitting the user feedback according to the display of the search result to an external device
Further comprising:
12. The method of claim 11,
The operation of displaying the search result includes:
An operation for displaying the search result while the application execution screen is displayed
Electronic device.
A method for acquiring information using a learned model of an electronic device,
Displaying an application execution screen;
Capturing the application execution screen to generate an image;
Acquiring first information on a first region on the generated image through a learned first model using the generated image as input data; And
And acquiring second information on a second region different from the first region on the generated image, using the acquired first information and the learned second model using the generated image as input data Information acquisition method.

Images