KR20220135383A - Electronic device and method of operation thereof - Google Patents

Abstract

An electronic device comprising a camera, a display, a memory, and a processor operatively coupled with the camera, the display, and the memory is disclosed. The memory may store one or more instructions that, when executed, cause the processor to perform: acquiring an image containing an icon; recognizing the icon from the image; retrieving an icon corresponding to the recognized icon in the database; extracting information related to the searched icon from the database; and based on the extracted information, displaying a user interface including link information for installing an application associated with the recognized icon on the display. In addition to this, various embodiments identified through the specification are possible. Based on an icon displayed on the outside of an electronic device, it is possible to move to an installation site of an application related to the corresponding icon.

Description

ELECTRONIC DEVICE AND METHOD OF OPERATION THEREOF

Embodiments disclosed in this document relate to an electronic device and a method of operating the electronic device.

The electronic device may install or download an application distributed by an application developer through an application store. A user of the electronic device may access an application store in the electronic device, search for an application to be installed from the application store, and install the corresponding application in the electronic device.

In order to install an application installed in an external electronic device, the electronic device has a problem of having to search for and install the corresponding application in the application store.

Accordingly, various embodiments disclosed in this document are intended to provide an electronic device capable of moving to an installation site of an application related to the corresponding icon based on an icon displayed on the screen of the external electronic device, and an operating method thereof.

In order to share link information for installing an application installed in the electronic device, the electronic device is inconvenient to search in an application store to obtain link information and then to share it.

Accordingly, various embodiments disclosed in this document include an electronic device capable of sharing link information for moving to an installation site of an application related to the corresponding icon based on the icon displayed on the screen of the electronic device with an external electronic device, and the We want to provide a method of operation.

An electronic device according to an embodiment disclosed in this document includes a camera, a display, a memory, and a processor operatively connected to the camera, the display, and the memory, wherein, when the memory is executed, the processor , obtaining an image including an icon, recognizing the icon from the image, retrieving an icon corresponding to the recognized icon in a database, extracting information associated with the retrieved icon from the database, and One or more instructions for displaying on the display a user interface including link information for installing an application related to the recognized icon based on the extracted information may be stored.

In addition, the method according to an embodiment disclosed in this document acquires an image including an icon, recognizes the icon from the image, searches for an icon corresponding to the recognized icon in a database, Information related to the found icon may be extracted from the database, and a user interface including link information for installing an application related to the recognized icon based on the extracted information may be displayed on the display of the electronic device.

According to various embodiments of the present disclosure, based on an icon displayed on the outside of the electronic device (eg, a screen of an external electronic device or a surface of an external object), an electronic device capable of moving to an installation site of an application related to the corresponding icon and an operating method thereof.

Also, according to various embodiments disclosed in this document, an electronic device capable of sharing link information for moving to an installation site of an application related to the corresponding icon based on an icon displayed on the screen of the electronic device with an external electronic device, and an operation thereof method can be provided.

In addition, various effects directly or indirectly identified through this document may be provided.

1 is a block diagram of an electronic device in a network environment, according to various embodiments of the present disclosure;
2 is a block diagram illustrating a program according to various embodiments.
3 is a block diagram of an electronic device according to an exemplary embodiment.
4 is a flowchart illustrating an operation of an electronic device according to an exemplary embodiment.
5 is a diagram for explaining an operation of an electronic device according to an exemplary embodiment.
6 is a flowchart illustrating an operation of an electronic device according to an exemplary embodiment.
7 is a diagram for describing an operation of an electronic device according to an exemplary embodiment.
8 is a flowchart illustrating an operation of an electronic device according to an exemplary embodiment.
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100 , an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with at least one of the electronic device 104 and the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input module 150 , a sound output module 155 , a display module 160 , an audio module 170 , and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or an antenna module 197 . In some embodiments, at least one of these components (eg, the connection terminal 178 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) are integrated into one component (eg, display module 160 ). can be

The processor 120, for example, executes software (eg, a program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 120 converts commands or data received from other components (eg, the sensor module 176 or the communication module 190 ) to the volatile memory 132 . may be stored in , process commands or data stored in the volatile memory 132 , and store the result data in the non-volatile memory 134 . According to an embodiment, the processor 120 is the main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit (eg, a graphic processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 101 includes the main processor 121 and the sub-processor 123 , the sub-processor 123 uses less power than the main processor 121 or is set to be specialized for a specified function. can The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The secondary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or when the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the coprocessor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190 ). have. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176 ) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output a sound signal to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display module 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display module 160 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 160 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input module 150 , or an external electronic device (eg, a sound output module 155 ) connected directly or wirelessly with the electronic device 101 . The electronic device 102) (eg, a speaker or headphones) may output a sound.

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more specified protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a local area network (LAN) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various techniques for securing performance in a high-frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 192 may support various requirements defined in the electronic device 101 , an external electronic device (eg, the electronic device 104 ), or a network system (eg, the second network 199 ). According to an embodiment, the wireless communication module 192 may include a peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency for realizing URLLC ( Example: Downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be selected. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 197 .

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or a part of operations executed in the electronic device 101 may be executed in one or more external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of things (IoT) device. The server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199 . The electronic device 101 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2 is a block diagram 200 illustrating a program 140 in accordance with various embodiments. According to an embodiment, the program 140 executes an operating system 142 , middleware 144 , or an application 146 executable in the operating system 142 for controlling one or more resources of the electronic device 101 . may include Operating system 142 may include, for example, Android™, iOS™, Windows™, Symbian™, Tizen™, or Bada™. At least some of the programs 140 are, for example, preloaded into the electronic device 101 at the time of manufacture, or an external electronic device (eg, the electronic device 102 or 104 ), or a server (eg, the electronic device 102 or 104 ) when used by a user ( 108)), or may be updated.

The operating system 142 may control management (eg, allocation or retrieval) of one or more system resources (eg, a process, memory, or power) of the electronic device 101 . The operating system 142 may additionally or alternatively include other hardware devices of the electronic device 101 , for example, the input module 150 , the sound output module 155 , the display module 160 , and the audio module 170 . , sensor module 176 , interface 177 , haptic module 179 , camera module 180 , power management module 188 , battery 189 , communication module 190 , subscriber identification module 196 , or It may include one or more driver programs for driving the antenna module 197 .

The middleware 144 may provide various functions to the application 146 so that functions or information provided from one or more resources of the electronic device 101 may be used by the application 146 . The middleware 144 includes, for example, an application manager 201 , a window manager 203 , a multimedia manager 205 , a resource manager 207 , a power manager 209 , a database manager 211 , and a package manager 213 . ), a connectivity manager 215 , a notification manager 217 , a location manager 219 , a graphics manager 221 , a security manager 223 , a call manager 225 , or a voice recognition manager 227 . can

The application manager 201 may manage the life cycle of the application 146 , for example. The window manager 203 may manage one or more GUI resources used in the screen, for example. The multimedia manager 205, for example, identifies one or more formats required for playback of media files, and encodes or decodes a corresponding media file among the media files using a codec suitable for the selected format. can be done The resource manager 207 may manage the space of the source code of the application 146 or the memory of the memory 130 , for example. The power manager 209 may, for example, manage the capacity, temperature, or power of the battery 189 , and determine or provide related information required for the operation of the electronic device 101 by using the corresponding information. . According to an embodiment, the power manager 209 may interwork with a basic input/output system (BIOS) (not shown) of the electronic device 101 .

The database manager 211 may create, retrieve, or change a database to be used by the application 146 , for example. The package manager 213 may manage installation or update of an application distributed in the form of a package file, for example. The connectivity manager 215 may manage, for example, a wireless connection or a direct connection between the electronic device 101 and an external electronic device. The notification manager 217 may provide, for example, a function for notifying the user of the occurrence of a specified event (eg, an incoming call, a message, or an alarm). The location manager 219 may manage location information of the electronic device 101 , for example. The graphic manager 221 may manage, for example, one or more graphic effects to be provided to a user or a user interface related thereto.

Security manager 223 may provide, for example, system security or user authentication. The telephony manager 225 may manage, for example, a voice call function or a video call function provided by the electronic device 101 . The voice recognition manager 227, for example, transmits the user's voice data to the server 108, and based at least in part on the voice data, a command corresponding to a function to be performed in the electronic device 101; Alternatively, the converted text data may be received from the server 108 based at least in part on the voice data. According to an embodiment, the middleware 244 may dynamically delete some existing components or add new components. According to an embodiment, at least a portion of the middleware 144 may be included as a part of the operating system 142 or implemented as software separate from the operating system 142 .

Application 146 includes, for example, home 251 , dialer 253 , SMS/MMS 255 , instant message (IM) 257 , browser 259 , camera 261 , alarm 263 . , contacts 265, voice recognition 267, email 269, calendar 271, media player 273, album 275, watch 277, health 279 (such as exercise or blood sugar) measuring biometric information), or environmental information 281 (eg, measuring atmospheric pressure, humidity, or temperature information). According to an embodiment, the application 146 may further include an information exchange application (not shown) capable of supporting information exchange between the electronic device 101 and an external electronic device. The information exchange application may include, for example, a notification relay application configured to transmit specified information (eg, call, message, or alarm) to an external electronic device, or a device management application configured to manage the external electronic device. have. The notification relay application, for example, transmits notification information corresponding to a specified event (eg, mail reception) generated in another application (eg, the email application 269 ) of the electronic device 101 to the external electronic device. can Additionally or alternatively, the notification relay application may receive notification information from the external electronic device and provide it to the user of the electronic device 101 .

The device management application is, for example, a power source (eg, turn-on or turn-off) of an external electronic device that communicates with the electronic device 101 or some components thereof (eg, a display module or a camera module of the external electronic device). ) or a function (eg brightness, resolution, or focus). The device management application may additionally or alternatively support installation, deletion, or update of an application operating in an external electronic device.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but it should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C," each of which may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may simply be used to distinguish an element from other elements in question, and may refer elements to other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively”. When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logic block, component, or circuit. can be used as A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). may be implemented as software (eg, the program 140) including For example, the processor (eg, the processor 120 ) of the device (eg, the electronic device 101 ) may call at least one of the one or more instructions stored from the storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided in a computer program product (computer program product). Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store ^™ ) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly, online between smartphones (eg: smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. have. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. , or one or more other operations may be added.

Hereinafter, a configuration and operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 3 .

3 is a block diagram 300 of an electronic device according to an embodiment.

Referring to FIG. 3 , the electronic device 301 (eg, the electronic device 101 of FIG. 1 ) includes a camera 310 (eg, the camera module 180 of FIG. 1 ) and a display 320 (eg, FIG. 1 ). of the display module 160 ), a memory 330 (eg, the memory 130 of FIG. 1 ), and a processor 340 (eg, the processor 120 of FIG. 1 ). Although not shown in FIG. 3 , the electronic device 301 may further include at least one of the components of the electronic device 101 shown in FIG. 1 . For example, the electronic device 301 may further include a communication circuit (not shown) (eg, the communication module 190 of FIG. 1 ).

According to an embodiment, the camera 310 may capture still images and moving images. According to an embodiment, the camera 310 may photograph a screen displayed on the display of the external electronic device. For example, the screen may include a home screen. According to an embodiment, the screen captured by the camera 310 may include at least one icon corresponding to an application. According to various embodiments, the camera 310 may photograph an external object (eg, paper or cloth) having a surface on which an icon is printed. According to various embodiments, the camera 310 may photograph an external object formed in the shape of an icon.

According to an embodiment, the display 320 may visually provide information to the outside (eg, a user) of the electronic device 201 . According to an embodiment, the display 320 may display a user interface (UI). According to an embodiment, the display 320 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch. For example, the display 320 may receive a touch input corresponding to the user interface.

According to an embodiment, the memory 330 may store one or more instructions executed by the processor 340 . According to an embodiment, the memory 330 may store a database for installing an application based on an icon. According to an embodiment, the database may include an icon, an application title (or name) of an application associated with the icon, and an installation link. According to an embodiment, the icon may be an application icon. For example, the installation link of the application is link information (eg, URL) that moves the processor 340 to a website (or webpage) where the application can be installed (or downloaded). (uniform resource locator)). According to an embodiment, when the processor 340 moves to the website or webpage, the processor 340 changes the screen output on the display 320 to a screen corresponding to the website or webpage and outputs it. can mean According to an embodiment, the icon, the application title, and the installation link stored in the database may be connected and stored for each application.

According to various embodiments, the database may be stored in an external server communicating with the electronic device 301 .

According to an embodiment, the processor 340 may control the operation of the electronic device 301 by executing instructions stored in the memory 330 . According to an embodiment, the processor 340 may build a database stored in the memory 330 or an external server. For example, the processor 340 may collect (or crawl) the application title, icon, and installation link of at least one application from an application store (eg, Play Store™). For example, the installation link of the application may include link information (eg, a webpage) that moves to a website (or webpage) on which the searched application is installed as the user searches for an application to be installed on the electronic device 301 from the application store. : URL). According to an embodiment, the processor 340 may create a database by connecting the collected data for each at least one application. For example, the database may include an application title of the first application, an icon of the first application, and an installation link of the first application in a connected state with respect to the first application.

According to an embodiment, the processor 340 may acquire an image including an icon by using the camera 310 . For example, the processor 340 may acquire an image of a screen (eg, a home screen or a menu screen) of an external electronic device on which an icon captured by the camera 310 is displayed. As another example, the processor 340 may acquire an image of an external object having a surface on which an icon photographed by the camera 310 is printed. As another example, the processor 340 may acquire an image of an external object in the form of an icon captured by the camera 310 . According to an embodiment, the processor 340 may recognize an icon from an image acquired using the camera 310 . According to an embodiment, the processor 340 may perform operations described below to recognize an icon from an image.

According to an embodiment, the processor 340 may normalize the acquired image. For example, the icon included in the image captured by the camera 310 may not match the size of the icon of the application stored in the database. As another example, the icon included in the image captured by the camera 310 may not be a front image like an icon of an application stored in a database. According to an embodiment, the processor 340 adjusts at least one of a size, a rotation angle, and a degree of inclination of the image captured by the camera 310 so that the icon included in the image corresponds to the icon stored in the database. can be normalized. For example, the processor 340 may adjust the size of the image so that the icon included in the image has the same size as that of the icon stored in the database. As another example, the processor 340 may adjust the rotation angle or the degree of inclination of the image so that the icon included in the image becomes a front image like the icon stored in the database.

According to an embodiment, the processor 340 may separate the icon and the text area of the normalized image through segmentation. According to an embodiment, segmentation means dividing an image into a set of pixels, and regions of the image corresponding to the pixels classified into the same set may have similar properties (or characteristics). According to an embodiment, the processor 340 may perform segmentation based on a deep-learning algorithm. According to an embodiment, the processor 340 may classify icons and texts determined to be related to each other into one area. For example, the processor 340 may determine that the icon and the text positioned at the bottom of the icon are related to each other. According to an embodiment, that the icon and the text are related to each other may mean that they are related to the same application.

According to an embodiment, the processor 340 may recognize the icon from the separated icon and text area or recognize the icon and text associated with the icon. For example, when text is not included in the separated icon and text area, the processor 340 may recognize only the icon from the corresponding area.

According to an embodiment, the processor 340 may search the database for an icon corresponding to the recognized icon. According to an embodiment, the database may be stored in the memory 330 or stored in an external server communicating with the electronic device 301 . According to an embodiment, the processor 340 may search the database for an icon of the same image as the recognized icon. According to an embodiment, as the same icon is retrieved from the database, the processor 340 may extract information related to the retrieved icon from the database. According to an embodiment, the information related to the found icon may mean information (eg, an application title and an application installation link) stored in connection with the found icon in the database.

According to various embodiments, when the processor 340 recognizes the icon and the text associated with the icon together, the processor 340 determines the identity of the recognized text and the icon retrieved from the database by connecting (associating or linking) with the stored application title. can be identified. According to an embodiment, the processor 340 may extract information related to the icon retrieved from the database based on identity identification. For example, the processor 340 may determine that the recognized icon and the icon included in the database correspond to each other as the recognized text and the application title stored in connection with the icon retrieved from the database are identified as the same. According to an embodiment, the processor 340 may extract information related to the icon included in the database corresponding to the recognized icon from the database. According to an embodiment, the processor 340 may supplement icon recognition and icon-based search by further performing text search.

According to an embodiment, the processor 340 may display a user interface including link information for installing an application associated with an icon recognized based on the extracted information on the display 320 . For example, the link information may be a URL. According to an embodiment, the extracted information may include an application title, an icon, and an installation link of at least one application. According to an embodiment, the installation link may include link information (eg, URL) that causes the processor 340 to move to a website (or web page) for installing an application related to the icon. According to an embodiment, the processor 340 may match the icon extracted from the database and the installation link with each other and display it on the display 320 . For example, the processor 340 may match and display a UI element (eg, a button) to which an operation for outputting a screen corresponding to icon and link information is mapped.

According to an embodiment, as the processor 340 recognizes a plurality of icons from an image, a plurality of icons corresponding to the plurality of icons may be searched from the database, and information associated with the plurality of searched icons may be extracted from the database. Accordingly, the extracted information may include application titles, icons, and installation links of a plurality of applications. According to an embodiment, the processor 340 may display a user interface including link information for each icon on the display. For example, the processor 340 may match a plurality of icons and a plurality of installation links extracted from the database to display them on the display 320 .

According to an embodiment, the processor 340 may display an installation screen of an application corresponding to link information on the display 320 in response to receiving a user input for selecting a user interface. According to an embodiment, the user input for selecting the user interface may include an input for selecting an installation link displayed on the display 320 . According to an embodiment, the processor 340 may display an installation screen of an application corresponding to link information (eg, URL) on the display 320 based on the selected installation link.

In the above-described embodiment, a case in which the processor 340 recognizes an icon from an image acquired by using the camera 310 is exemplified, but the present invention is not limited thereto, and the processor 340 uses a communication circuit (not shown). Recognizing an icon from an image received from an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108 of FIG. 1) connected to the electronic device 301 using the display 320 An icon may be recognized from a designated area based on a user input on the screen displayed in .

According to various embodiments, the processor 340 is an external electronic device (eg, the electronic device 102 of FIG. 1 , the electronic device 104 , or a server connected to the electronic device 301 using a communication circuit (not shown)). 108) may receive an image including the icon. For example, the processor 340 downloads an image including an icon from a web site (or web page) navigated through web browsing, or includes the icon through a web site (or web page). By receiving the shared image, an image including an icon may be received from an external electronic device that provides a web site (or web page). According to an embodiment, the processor 340 may recognize an icon from an image received from an external electronic device. According to an embodiment, a method for recognizing an icon from an image obtained by using the camera 310 may be applied to a method for the processor 340 to recognize an icon from an image received from an external electronic device.

According to various embodiments, the processor 340 may designate an area including the icon on the screen displayed on the display 320 based on a user input. According to an embodiment, the user input may include a touch input using a stylus pen or at least a part of the user's body. According to an embodiment, the stylus pen may be an electronic pen capable of interworking with the electronic device 301 . For example, the processor 340 may detect an input of a user touching the display 320 using a stylus pen or a finger through the display 320 . According to an embodiment, the processor 340 may designate an area corresponding to the user input on the screen displayed on the display 320 . According to an embodiment, the designated area may be an area including an icon displayed on the display 320 .

For example, the processor 340 may acquire a screenshot based on a user input or acquire an image corresponding to a screen displayed on the display 320 through screen capture. In this case, the processor 340 may designate the entire area of the screen displayed on the display 320 .

According to an embodiment, the size of the icon included in the designated area may be different from that of the icon stored in the database. According to an embodiment, the processor 340 may adjust the size of the image corresponding to the designated area so that the icon included in the designated area has the same size as the icon stored in the database. According to an embodiment, the icon included in the designated area may be a front image like an icon stored in a database, unlike an icon included in an image captured by the camera 310 . According to an embodiment, the processor 340 may omit the operation of adjusting the rotation angle or the degree of inclination of the image corresponding to the designated area.

According to an embodiment, the processor 340 may recognize an icon from a designated area. According to an embodiment, the processor 340 may recognize an icon from an image corresponding to a designated area. According to an embodiment, the image corresponding to the designated area may include an image in which the processor 340 has adjusted the size of the icon included in the designated area to be the same as the size of the icon stored in the database.

According to an embodiment, the processor 340 may recognize the icon and text from the designated area as not only the icon but also the text is included in the designated area. According to an embodiment, the processor 340 searches a database for an icon corresponding to the recognized icon and text based on the recognized icon and text, extracts information related to the searched icon from the database, and recognizes based on the extracted information The above-described method may be equally applied to a method of displaying a user interface including link information for installing an application associated with an icon associated with the icon.

According to an embodiment, the processor 340 may share link information upon receiving an input for selecting a user interface. For example, in response to receiving an input for selecting a user interface, the processor 340 uses a communication circuit (not shown) to transmit/receive information to and from an external electronic device through an external electronic device through an application supporting network. to transmit link information. For example, the processor 340 may share link information with an external electronic device (or a user of the external electronic device) through a social network service (SNS) application, a cloud application, or a message application. have.

According to an embodiment, the electronic device 301 recognizes an icon from an image acquired by using the camera 310 and provides a user interface including an installation link for outputting an installation screen of an application related to the recognized icon. And, by outputting an installation screen of an application associated with the recognized icon based on a user input through the user interface, it is possible to reduce the hassle of a user having to search for and install an application to be installed from the application store. According to an embodiment, the electronic device 301 may simplify a procedure in which a user installs an application corresponding to an icon displayed on a home screen of an external electronic device on his/her electronic device 301 .

According to an embodiment, the electronic device 301 recognizes an icon from a designated area based on a user input on the screen displayed on the display 320 and includes an installation link for outputting an installation screen of an application related to the recognized icon. by providing a user interface that provides a user interface that provides a user interface that provides a user interface to the user, and shares an installation link with an external electronic device based on a user input through the user interface, thereby allowing a user of the electronic device to search for an application to be installed in the application store, and copy the installation link of the searched application. Thus, it is possible to reduce the hassle of sharing with an external electronic device. According to an embodiment, the electronic device 301 may simplify a procedure in which a user shares an application corresponding to an icon displayed on the home screen of the electronic device 301 with a user of the external electronic device.

The shape of the electronic device 301 according to various embodiments is not limited to the portable communication device (eg, a smartphone) exemplarily illustrated in FIGS. 5 and 7 . For example, the electronic device 301 may include an augmented reality (AR) device (eg, AR glasses). According to an embodiment, the electronic device 301 in the form of an AR device recognizes an icon from an image of a real environment acquired using a camera or an image of a virtual environment output through the display 320, and corresponds to the recognized icon. may provide an installation screen of an application to be used, or share an installation link of an application corresponding to a recognized icon with an external electronic device.

Hereinafter, an operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 4 .

4 is a flowchart 400 of an electronic device according to an embodiment. Operations of the electronic device described below may be performed by the electronic device 301 of FIG. 3 or the processor 340 of the electronic device 301 .

In operation 401, the electronic device may acquire an image including an icon. According to an embodiment, the electronic device may acquire an image including an icon by using a camera (eg, the camera 310 of FIG. 3 ). According to an embodiment, the electronic device may acquire an image obtained by photographing a screen displayed on the display of the external electronic device. According to an embodiment, the screen may include at least one icon corresponding to the application. For example, the screen may include a home screen.

In operation 403, the electronic device may recognize an icon from the image. According to an embodiment, the electronic device may normalize an image acquired using a camera. According to an embodiment, the electronic device may normalize the image so that an icon included in the image corresponds to an icon stored in a database by adjusting at least one of a size, a rotation angle, and a degree of inclination of the image. For example, the electronic device may adjust the size of the image so that the icon included in the image has the same size as that of the icon stored in the database. As another example, the electronic device may adjust the rotation angle or the degree of inclination of the image so that the icon included in the image becomes a front image like the icon stored in the database.

According to an embodiment, the electronic device may separate an icon and a text area of a normalized image through segmentation. According to an embodiment, segmentation means dividing an image into a set of pixels, and regions of the image corresponding to the pixels classified into the same set may have similar properties (or characteristics). According to an embodiment, the electronic device may classify icons and texts determined to be related to each other into one area. For example, the electronic device may determine that the icon and the text positioned at the bottom of the icon are related to each other. According to an embodiment, that the icon and the text are related to each other may mean that they are related to the same application.

According to an embodiment, the electronic device may recognize the icon from the separated icon and text area or recognize the icon and text associated with the icon. For example, when text is not included in the separated icon and text area, the electronic device may recognize only the icon from the corresponding area.

In operation 405, the electronic device may search for an icon in the database. According to an embodiment, the electronic device may search the database for an icon corresponding to the recognized icon. According to an embodiment, the database may be built by the electronic device and stored in a memory (eg, the memory 330 of FIG. 3 ) or stored in an external server communicating with the electronic device. For example, the electronic device may collect (or crawl) the application title, icon, and installation link of at least one application from an application store (eg, Play Store™). For example, the installation link of the application is link information (eg, URL) that allows the user to move to a website (or web page) on which the searched application is installed as the user searches for an application to be installed on the electronic device from the application store. may include. According to an embodiment, the electronic device may create a database by connecting the collected data for each at least one application. According to an embodiment, the electronic device may search the database for an icon of the same image as the recognized icon.

In operation 407, the electronic device may extract information related to the searched icon from the database. According to an embodiment, as the same icon is retrieved from the database, the electronic device may extract information related to the retrieved icon from the database. According to an embodiment, the information related to the found icon may include an icon found in the database and information stored in connection with the icon found in the database (eg, an application title, and an application installation link).

According to various embodiments, when the electronic device recognizes the icon and the text associated with the icon together, the electronic device may identify the sameness of the recognized text and the icon retrieved from the database to be connected (associated or linked) with the stored application title. . According to an embodiment, as the recognized text and the application title stored in connection with the icon retrieved from the database are identified as being the same, the electronic device may extract information related to the icon retrieved from the database.

In operation 409, the electronic device may display a user interface including link information for installing an application related to an icon. According to an embodiment, the electronic device may display a user interface including link information for installing an application associated with an icon recognized based on information extracted from the database on a display (eg, the display 320 of FIG. 3 ). can According to an embodiment, the information extracted from the database may include an application title, an icon, and an installation link of at least one application. According to an embodiment, the installation link may include link information (eg, URL) that causes the processor 340 to move to a website (or web page) for installing an application related to the icon. According to an embodiment, the electronic device may match the icon extracted from the database and the installation link with each other and display it on the display. For example, the electronic device may match and display a UI element (eg, a button) to which an operation of outputting a screen corresponding to icon and link information is mapped.

In operation 411, the electronic device may display an installation screen of an application corresponding to link information. According to an embodiment, in response to receiving an input for selecting a user interface, the electronic device may display an installation screen of an application corresponding to link information on the display. For example, the user input for selecting the user interface may include an input for selecting a UI element to which an operation of outputting a screen corresponding to link information displayed on a display is mapped. As an input for selecting a UI element is received, the electronic device may display an installation screen of an application corresponding to link information (eg, URL) on the display.

Hereinafter, an operation of an electronic device and a user interface provided by the electronic device according to an exemplary embodiment will be described with reference to FIG. 5 .

5 is a diagram 500 for explaining an operation of an electronic device. Operations of the electronic device 501 (eg, the electronic device 301 of FIG. 3 ) described below may be performed by a processor (eg, the processor 340 of FIG. 3 ) of the electronic device 501 .

The first screen S51 may be a screen of the external electronic device 502 communicating with the electronic device 501 . For example, the first screen S51 may be a home screen of the external electronic device 502 . According to an embodiment, the first screen S51 may include at least one icon. According to an embodiment, the at least one icon may be an icon corresponding to an application.

Referring to the second screen S52 , the electronic device 501 views the screen (eg, the first screen S51 ) of the external electronic device 502 using a camera (eg, the camera 310 of FIG. 3 ). can be filmed According to an embodiment, the electronic device 501 may acquire an image including a screen (eg, the first screen S51 ) of the external electronic device 502 . According to an embodiment, the electronic device 501 may recognize at least one icon from the image. A method for the electronic device 501 to recognize an icon from an image acquired using a camera according to an exemplary embodiment has been described above with reference to FIGS. 3 and 4 , and thus a redundant description will be simplified or omitted.

Referring to the third screen S53 , the electronic device 501 displays a user interface including link information for installing an application related to an icon recognized from an image on a display (eg, the display 320 of FIG. 3 ). can do. According to an embodiment, the electronic device 501 may search the database for an icon of the same image as an icon recognized from the image. According to an embodiment, the electronic device 501 may extract information related to the searched icon from the database, and display a user interface based on the extracted information. According to an embodiment, the extracted information may include an application title, an icon, and an installation link of at least one application respectively associated with at least one icon recognized from the image. According to an embodiment, the electronic device 501 may match an application title, an icon, and an installation link with each other and display it on the display. For example, the electronic device 501 includes a first icon 531 indicating that the first application is the first application and first link information (eg, a website or a web page on which to install (or download) the first application). : URL) can be displayed by matching the first installation link. For example, the electronic device 501 may match and display the first icon 531 and the first button 532 to which an operation of outputting a screen corresponding to the first link information is mapped.

According to an embodiment, in response to receiving a user input for selecting the first button 532 , the electronic device 501 may display the fourth screen S54 on the display. According to an embodiment, the fourth screen S54 may be an installation screen of the first application.

According to an embodiment, the electronic device 501 captures a screen (first screen S51 ) of the external electronic device 502 using a camera, and the screen (first screen S51 ) of the external electronic device 502 . )), it is possible to recognize the icon displayed in ), and provide link information for moving to a website or webpage on which an application related to the recognized icon can be installed. According to an embodiment, the electronic device 501 may output an installation screen of an application corresponding to link information without a process of searching for an application in an application store based on a user input.

Hereinafter, an operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 6 .

6 is a flowchart 600 illustrating an operation of an electronic device according to an exemplary embodiment. Operations of the electronic device described below may be performed by the electronic device 301 of FIG. 3 or the processor 340 of the electronic device 301 .

In operation 601, the electronic device may designate an area including the icon. According to an embodiment, the electronic device may designate an area including an icon on a screen displayed on a display (eg, the display 320 of FIG. 3 ) based on a user input. According to an embodiment, the user input may include a touch input using a stylus pen or at least a part of the user's body. For example, the electronic device may detect an input of a user touching the display using a stylus pen or a finger through the display. According to an embodiment, the electronic device may designate a region corresponding to a user input on a screen displayed on the display, and the designated region may include at least one icon displayed on the display.

According to various embodiments, the electronic device may designate the entire area of the screen displayed on the display based on a user input for obtaining a screenshot or performing a screen capture.

In operation 603, the electronic device may recognize an icon from the designated area. According to an embodiment, since operation 603 may correspond to operation 403 of FIG. 4 , a duplicate description will be omitted or simplified. According to an embodiment, the electronic device may normalize an image corresponding to a designated area. According to an embodiment, when the electronic device normalizes an image, at least one of a size, an angle, and a degree of inclination of the image is adjusted so that the size and shape of the icon included in the image correspond to the size and shape of the icon stored in the database. may include doing According to an embodiment, the icon stored in the database may be a front image, and the size of the icon stored in the database may have a specific size. For example, the icon included in the designated area may be a front image as a part of an image obtained by capturing a screen output on the display of the electronic device as it is. Accordingly, the electronic device may omit the operation of adjusting the rotation angle or the degree of inclination of the image corresponding to the designated area. For example, the icon included in the designated area may have a different size from the icon stored in the database according to the icon size setting of the electronic device. Accordingly, the electronic device may adjust the size of the image corresponding to the designated area so that the size of the icon included in the designated area is the same as the size of the icon stored in the database.

According to an embodiment, the electronic device may recognize the icon from the normalized image. According to an embodiment, the electronic device may separate an icon and a text area of a normalized image through segmentation. According to an embodiment, the electronic device may recognize the icon from the separated icon and text area or recognize the icon and text associated with the icon.

In operation 605, the electronic device may search for an icon in the database. According to an embodiment, the electronic device may search the database for an icon corresponding to the recognized icon. According to an embodiment, since operation 605 may correspond to operation 405 of FIG. 4 , a duplicate description will be omitted or simplified. According to an embodiment, the database may be built by the electronic device and stored in a memory (eg, the memory 330 of FIG. 3 ) or stored in an external server communicating with the electronic device. For example, the electronic device connects the application title, icon, and installation link of at least one application collected (or crawled) from an application store (eg, Play Store™) for each application and connects to the database. can create According to an embodiment, the installation link may include link information (eg, URL) that allows the electronic device to move to a website (or web page) for installing an application. According to an embodiment, the electronic device may search the database for an icon of the same image as the icon recognized from the specified area.

In operation 607, the electronic device may extract information related to the searched icon from the database. According to an embodiment, since operation 607 may correspond to operation 407 of FIG. 4 , a duplicate description will be omitted or simplified. According to an embodiment, the information related to the found icon may include an icon found in the database and information stored in connection with the icon found in the database (eg, an application title, and an application installation link).

In operation 609, the electronic device may display a user interface including link information for installing an application related to the icon. According to an embodiment, the electronic device may display a user interface including link information for installing an application associated with an icon recognized based on information extracted from the database on the display. According to an embodiment, since operation 609 may correspond to operation 409 of FIG. 4 , a duplicate description will be omitted or simplified. According to an embodiment, the electronic device may match the icon extracted from the database and the installation link with each other and display it on the display. According to an embodiment, the installation link may include link information (eg, URL) that allows the electronic device to move to a website (or webpage) for installing an application related to the icon. For example, the electronic device may match and display a UI element (eg, a button) to which an operation of sharing icon and link information is mapped.

In operation 611, the electronic device may share link information. According to an embodiment, the electronic device may share link information in response to receiving an input for selecting a user interface. For example, the input for selecting a user interface may include an input for selecting a UI element to which an operation of sharing link information is mapped. According to an embodiment, in response to receiving an input for selecting a user interface, the electronic device may transmit link information to the external electronic device through an application that supports transmitting and receiving information to and from the external electronic device through a network. For example, an application supporting transmission and reception of information with an external electronic device may include a social network service (SNS) application, a cloud application, or a message application. According to an embodiment, the electronic device may share link information with an external electronic device (or a user of the external electronic device) by transmitting the link information to the external electronic device.

Hereinafter, an operation and a user interface of an electronic device according to an exemplary embodiment will be described with reference to FIG. 7 .

7 is a diagram 700 for explaining an operation of an electronic device. Operations of the electronic device 701 (eg, the electronic device 301 of FIG. 3 ) described below may be performed by a processor (eg, the processor 340 of FIG. 3 ) of the electronic device 701 .

According to an embodiment, the first screen S71 may be a home screen of the electronic device 701 . According to an embodiment, the first screen S71 may include at least one icon. According to an embodiment, the at least one icon may be an icon corresponding to an application.

Referring to the second screen S72 , the electronic device 701 displays an area including the icon on the screen (the first screen S71 ) displayed on the display (eg, the display 320 of FIG. 3 ) based on the user input. can be specified. For example, the electronic device 701 may use a stylus pen interlocking with the electronic device 701 or at least a part of the user's body (eg, a finger) to draw a line corresponding to the border of the area including the icon. Upon receiving , the first area 721 may be designated. As another example, the electronic device 701 uses a stylus pen interlocking with the electronic device 701 or at least a part of the user's body (eg, a finger) to determine a rectangular area including the icon at a diagonal from one of four points. As a user input for dragging in a direction is received, the second area 722 may be designated. According to an embodiment, the electronic device 701 may recognize at least one icon from a designated area (eg, the first area 721 and the second area 722 ). A method for the electronic device 701 to recognize an icon from a designated area based on a user input has been described above with reference to FIGS. 3 and 6 according to an exemplary embodiment, and thus a redundant description will be simplified or omitted.

Referring to the third screen S73 , the electronic device 701 may display a user interface including link information for installing an application related to an icon recognized from a designated area on the display. According to an embodiment, the electronic device 701 may search the database for an icon of the same image as the icon recognized from the specified area. According to an embodiment, the electronic device 701 may extract information related to the searched icon from the database, and display a user interface based on the extracted information. According to an embodiment, the extracted information may include an application title, an icon, and an installation link of at least one application respectively associated with at least one icon recognized from the image. According to an embodiment, the electronic device 701 may match an application title, an icon, and an installation link with each other and display it on the display. For example, the electronic device 701 includes a first icon 731 indicating that it is a first application and first link information (eg, a website or a webpage for downloading) on which the first application can be installed (or downloaded). : URL) can be displayed by matching the first installation link. For example, the electronic device 701 may match and display the first icon 731 and the first button 732 to which the operation of sharing the first link information is mapped.

According to an embodiment, in response to receiving a user input for selecting the first button 732 , the electronic device 701 may display the fourth screen S74 on the display. According to an embodiment, the fourth screen S74 may be a screen of the second application. According to an embodiment, the second application may be an application that supports the electronic device 701 to transmit and receive information to and from an external electronic device through a network. For example, the second application may be a messenger application. According to an embodiment, in response to receiving a user input for selecting the first button 732 , the electronic device 701 transmits the first link information to the second application, and transmits the first link information to the external electronic device through the second application. 1 Link information can be transmitted. According to an embodiment, the fourth screen S74 may indicate a screen on which the first link information has been transmitted to the external electronic device.

Hereinafter, an operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 8 .

8 is a flowchart 800 of an electronic device according to an embodiment. Operations of the electronic device described below may be performed by the electronic device 301 of FIG. 3 or the processor 340 of the electronic device 301 .

In operation 801, the electronic device may acquire an image including an icon. According to an embodiment, the electronic device may acquire an image including an icon by using a camera (eg, the camera 310 of FIG. 3 ). For example, an image including an icon obtained using a camera may include an image of a screen (eg, a home screen or a menu screen) of an external electronic device on which the icon is displayed, an image of an external object on which an icon is printed, or an external image in the form of an icon. It may include an object image.

According to another embodiment, the electronic device is an external electronic device (eg, the electronic device 102 and the electronic device 104 of FIG. 1 ) connected to the electronic device using a communication circuit (eg, the communication module 190 of FIG. 1 ). , or may receive an image including the icon from the server 108). According to an embodiment, the image including the icon may be an image provided by an external electronic device connected to the electronic device through a network (eg, the first network 198 or the second network 199). For example, the image including the icon received using the communication circuit may be a screen (eg, a home screen, or a menu) of an external electronic device (an external electronic device or other external electronic device connected to the electronic device using the communication circuit). screen) image, an image of an external object printed with an icon, an image of an external object in the form of an icon, or a virtual image including an icon.

According to another embodiment, the electronic device designates a region including an icon on a screen displayed on a display (eg, the display 320 of FIG. 3 ) based on a user input, and an image corresponding to the designated region including the icon can be obtained. For example, the user input may include a touch input using a stylus pen or at least a part of the user's body. As another example, the user input may include an input for obtaining a screenshot or performing a screen capture. In this case, the electronic device may designate the entire area of the screen displayed on the display. According to an embodiment, the designated area may include at least one icon displayed on the display.

In operation 803, the electronic device may recognize an icon from the image. According to an embodiment, the electronic device may normalize the image. According to an embodiment, the electronic device may normalize the image so that an icon included in the image corresponds to an icon stored in a database by adjusting at least one of a size, a rotation angle, and a degree of inclination of the image. For example, the electronic device may adjust the size of the image so that the icon included in the image has the same size as that of the icon stored in the database. As another example, the electronic device may adjust the rotation angle or the degree of inclination of the image so that the icon included in the image becomes a front image like the icon stored in the database. According to an embodiment, the electronic device omits the operation of adjusting the rotation angle or the degree of inclination of the image with respect to the image corresponding to the specified area of the screen displayed on the display of the electronic device obtained based on the user input, and omits the size of the image. Only the operation to adjust the .

According to an embodiment, the electronic device may separate an icon and a text area of a normalized image through segmentation. According to an embodiment, the electronic device may classify icons and texts determined to be related to each other into one area. For example, the electronic device may determine that the icon and the text positioned at the bottom of the icon are related to each other. According to an embodiment, the electronic device may recognize the icon from the separated icon and text area or recognize the icon and text associated with the icon. According to an embodiment, the text associated with the icon may include an application title. According to an embodiment, the text associated with the icon may be an application title of an application associated with the icon.

In operation 805, the electronic device may search for an icon in the database. According to an embodiment, the electronic device may search the database for an icon corresponding to the recognized icon. According to an embodiment, the database may be built by the electronic device and stored in a memory (eg, the memory 330 of FIG. 3 ) or stored in an external server communicating with the electronic device. According to an embodiment, the electronic device collects data including an application title, an icon, and an installation link of an application of at least one application from an application store (eg, Play Store™), and collects the collected data. At least one application can be connected to create a database. According to an embodiment, the installation link of the application may include link information (eg: URL) may be included. According to an embodiment, the electronic device may search the database for an icon of the same image as the recognized icon.

In operation 807, the electronic device may extract information related to the searched icon from the database. According to an embodiment, as an icon of the same image as the recognized icon is retrieved from the database, the electronic device may extract information related to the retrieved icon from the database. According to an embodiment, the information related to the found icon may include an icon found in the database and information stored in connection with the icon found in the database (eg, an application title, and an application installation link).

According to various embodiments, when the electronic device recognizes both the icon and the text associated with the icon, the electronic device may identify the identity of the recognized text and the application title stored in the database in association with the icon found in the database. The electronic device may extract information related to the searched icon from the database based on identification between the recognized text and the application title stored in the database.

In operation 809, the electronic device may display a user interface including link information for installing an application related to the icon. According to an embodiment, the electronic device may display a user interface including link information for installing an application associated with an icon recognized based on information extracted from the database on the display. According to an embodiment, the electronic device recognizes a plurality of icons from an image, searches for a plurality of icons corresponding to the plurality of recognized icons in a database, extracts information related to the plurality of icons from the database, and extracts the extracted information. A user interface including link information for a plurality of icons recognized based on the information may be displayed on the display. According to an embodiment, the information extracted from the database may include an application title, an icon, and an installation link of at least one application. According to an embodiment, the electronic device may match the icon extracted from the database and the installation link with each other and display it on the display.

For example, the electronic device may display a user interface in which UI elements (eg, buttons) to which an operation of outputting a screen corresponding to icon and link information are mapped are matched with each other. According to an embodiment, in response to receiving an input for selecting a user interface, the electronic device may display an installation screen of an application corresponding to link information on the display.

As another example, the electronic device may display a user interface in which UI elements (eg, buttons) to which an operation of sharing icon and link information are mapped are matched. According to an embodiment, the electronic device may share link information in response to receiving an input for selecting a user interface. According to an embodiment, in response to receiving an input for selecting a user interface, the electronic device may transmit link information to the external electronic device through an application that supports transmitting and receiving information to and from the external electronic device through a communication circuit. For example, an application supporting transmission and reception of information with an external electronic device may include a social network service (SNS) application, a cloud application, or a message application.

According to an embodiment disclosed in this document, an electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 301 of FIG. 3 , the electronic device 501 of FIG. 5 , or the electronic device of FIG. 7 ( 701)) is a camera (eg, the camera module 180 of FIG. 1, or the camera 310 of FIG. 3), a display (eg, the display module 160 of FIG. 1, or the display 320 of FIG. 3); memory (eg, memory 130 of FIG. 1 , or memory 330 of FIG. 3 ), and a processor operatively coupled with the camera, the display, and the memory (eg, processor 120 of FIG. 1 , or FIG. 3, the processor 340), wherein the memory, when executed, causes the processor to obtain an image comprising an icon, recognize the icon from the image, and retrieve the recognized icon from the database. Searching for an icon corresponding to , extracting information related to the found icon from the database, and displaying a user interface including link information for installing an application related to the recognized icon based on the extracted information on the display It may store one or more instructions to be displayed.

According to an embodiment disclosed in this document, the instructions may cause the processor to acquire the image including the icon by using the camera.

According to an embodiment disclosed in this document, the electronic device includes a communication circuit (eg, the communication module 190 of FIG. 1 ), and the instructions are, The image including the icon may be received from an external electronic device (eg, the electronic device 102 , the electronic device 104 of FIG. 1 , or the server 108 of FIG. 1 ) connected thereto.

According to an embodiment disclosed in this document, the instructions, the processor designates an area including the icon on the screen displayed on the display based on a user input, and corresponds to the designated area including the icon It is possible to acquire the image that becomes

According to an embodiment disclosed in this document, the instructions, the processor normalizes the image, separates an icon and a text area of the normalized image through segmentation, and the separated icon and text It is possible to recognize the icon from a region, or to recognize the icon and text associated with the icon.

According to an embodiment disclosed in this document, the instructions include, by the processor, an icon in which the icon included in the image is stored in the database by adjusting at least one of a size, a rotation angle, and a degree of inclination of the image. The image may be normalized to correspond to .

According to an embodiment disclosed in this document, the text associated with the icon may include an application title.

According to an embodiment disclosed in this document, the instructions, the processor searches for an icon corresponding to the recognized icon in the database, and is associated with the recognized text and the searched icon, the application stored in the database The identity of the title may be identified, and information associated with the searched icon may be extracted from the database based on the identity identification.

According to an embodiment disclosed in this document, the instructions, the processor recognizes a plurality of icons from the image, searches the database for a plurality of icons corresponding to the recognized plurality of icons, and the database information associated with the plurality of searched icons may be extracted from the information, and a user interface including link information for each recognized plurality of icons may be displayed on the display based on the extracted information.

According to an embodiment disclosed in this document, the instructions, the processor collects data including an application title, an icon, and an installation link of at least one application from an application store, and collects the collected data The database may be created by connecting data for each of the at least one application.

According to an embodiment disclosed in this document, the instructions may cause the processor to display an installation screen of the application corresponding to the link information on the display in response to receiving an input for selecting the user interface. .

According to an embodiment disclosed in this document, the instructions may cause the processor to share the link information in response to receiving an input for selecting the user interface.

According to an embodiment disclosed in this document, the electronic device includes a communication circuit (eg, the communication module 190 of FIG. 1 ), and the instructions receive, by the processor, an input for selecting the user interface. Accordingly, through an application in which the processor transmits and receives information to and from an external electronic device (eg, the electronic device 102 , the electronic device 104 of FIG. 1 , or the server 108 of FIG. 1 ) through the communication circuit, the external electronic device The device may be configured to transmit the link information.

According to an embodiment disclosed in this document, an electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 301 of FIG. 3 , the electronic device 501 of FIG. 5 , or the electronic device of FIG. 7 ( 701))) obtains an image including an icon, recognizes the icon from the image, searches for an icon corresponding to the recognized icon in a database, and obtains an icon corresponding to the recognized icon from the database. A user interface including link information for extracting related information and installing an application related to the recognized icon based on the extracted information is displayed on the display of the electronic device (eg, the display module 160 of FIG. 1 , or may be displayed on the display 320 of FIG. 3 .

According to an embodiment disclosed in this document, the image including the icon may be acquired using a camera of the electronic device (eg, the camera module 180 of FIG. 1 or the camera 310 of FIG. 3 ). can

According to an embodiment disclosed in this document, an area including the icon may be designated on the screen displayed on the display based on a user input, and the image corresponding to the designated area including the icon may be acquired. .

According to an embodiment disclosed in this document, the image is normalized, the icon and text area of the normalized image are separated through segmentation, and the icon is recognized from the separated icon and text area, The icon and text associated with the icon may be recognized.

According to an embodiment disclosed in this document, an icon corresponding to the recognized icon is retrieved from the database, the recognized text is associated with the retrieved icon, and the identity of an application title stored in the database is identified, and the Information associated with the searched icon may be extracted from the database based on identity identification.

According to an embodiment disclosed in this document, upon receiving an input for selecting the user interface, an installation screen of the application corresponding to the link information may be displayed on the display.

According to an embodiment disclosed in this document, the link information may be shared when an input for selecting the user interface is received.

Claims (20)

In an electronic device,
camera;
display;
Memory; and
a processor operatively coupled to the camera, the display and the memory;
The memory, when executed, causes the processor to:
acquiring an image including an icon,
Recognizing the icon from the image,
retrieving an icon corresponding to the recognized icon in the database,
extracting information related to the searched icon from the database;
An electronic device for storing one or more instructions for displaying on the display a user interface including link information for installing an application related to the recognized icon based on the extracted information. The method according to claim 1,
The instructions, the processor,
to acquire the image including the icon by using the camera. The method according to claim 1,
The electronic device includes a communication circuit,
The instructions, the processor,
and receiving the image including the icon from an external electronic device connected to the electronic device using the communication circuit. The method according to claim 1,
The instructions, the processor,
Designating an area including the icon on the screen displayed on the display based on a user input,
to acquire the image corresponding to the designated area including the icon. The method according to claim 1,
The instructions, the processor,
normalize the image,
Separating the icon and text area of the normalized image through segmentation,
Recognizing the icon from the separated icon and text area or recognizing the icon and text associated with the icon. 6. The method of claim 5,
The instructions, the processor,
The electronic device normalizes the image so that the icon included in the image corresponds to the icon stored in the database by adjusting at least one of a size, a rotation angle, and a degree of inclination of the image. 6. The method of claim 5,
The text associated with the icon includes an application title. 8. The method of claim 7,
The instructions, the processor,
retrieving an icon corresponding to the recognized icon in the database,
identifying the identity of the application title stored in the database in association with the recognized text and the retrieved icon;
and extract information associated with the found icon from the database based on the identity identification. The method according to claim 1,
The instructions, the processor,
Recognizing a plurality of icons from the image,
retrieving a plurality of icons corresponding to the recognized plurality of icons in the database;
extracting information related to the plurality of searched icons from the database;
and displaying a user interface including link information for each of the plurality of recognized icons on the display based on the extracted information. The method according to claim 1,
The instructions, the processor,
collect data including an application title, an icon, and an installation link of at least one application from an application store;
The electronic device is configured to connect the collected data for each of the at least one application to generate the database. The method according to claim 1,
The instructions, the processor,
and displaying an installation screen of the application corresponding to the link information on the display in response to receiving an input for selecting the user interface. The method according to claim 1,
The instructions, the processor,
and share the link information in response to receiving an input for selecting the user interface. 13. The method of claim 12,
The electronic device includes a communication circuit,
The instructions, the processor,
In response to receiving an input for selecting the user interface, the processor transmits the link information to the external electronic device through an application that transmits and receives information to and from the external electronic device through the communication circuit. A method of operating an electronic device, comprising:
acquiring an image including an icon,
Recognizing the icon from the image,
retrieving an icon corresponding to the recognized icon in the database,
extracting information related to the searched icon from the database;
Displaying a user interface including link information for installing an application related to the recognized icon based on the extracted information on the display of the electronic device. 15. The method of claim 14,
A method of acquiring the image including the icon by using a camera of the electronic device. 15. The method of claim 14,
Designating an area including the icon on the screen displayed on the display based on a user input,
A method of acquiring the image corresponding to the designated area including the icon. 15. The method of claim 14,
normalize the image,
Separating the icon and text area of the normalized image through segmentation,
recognizing the icon from the separated icon and text area, or recognizing the icon and text associated with the icon. 18. The method of claim 17,
retrieving an icon corresponding to the recognized icon in the database,
identifying the identity of the application title stored in the database in association with the recognized text and the retrieved icon;
extracting information associated with the retrieved icon from the database based on the identity identification. 15. The method of claim 14,
Displaying an installation screen of the application corresponding to the link information on the display in response to receiving an input for selecting the user interface. 15. The method of claim 14,
and sharing the link information upon receiving an input to select the user interface.

Images