KR20230052655A - Electronic device and operating method thereof - Google Patents

Abstract

Disclosed is an electronic device comprising a microphone, a display, a communication circuit, a memory, and a processor operatively coupled with the microphone, the display, the communication circuit, and the memory. The memory stores one or more instructions which, upon execution, causes the processor to: obtain an image of a screen output on the display; obtain attribute information of at least one element constituting the screen; identify elements on which a user action is possible based on the attribute information; obtain location information and size information of the identified element; determine an area of the identified element on the image based on the location information and the size information; extract text information of the identified element from the image of the determined area; and perform control operations corresponding to user utterances received through the microphone based on the attribute information, the text information, and the location information of the identified element. In addition, various embodiments identified through the specification are possible. The present invention can reduce data throughput.

Description

Electronic device and its operating method {ELECTRONIC DEVICE AND OPERATING METHOD THEREOF}

Embodiments disclosed in this document relate to an electronic device that performs control based on voice recognition and an operating method thereof.

An electronic device uses voice recognition technology to replace a user's physical input with a voice input, and accordingly, the user can conveniently control the electronic device without having to move.

For example, an electronic device may receive a voice input from a user and provide various content services based on the user's intention or perform specific functions within the electronic device.

The electronic device may control a screen output on a display of the electronic device through voice recognition. To this end, the electronic device may use a screen layout extraction method provided at the system level or information on functions provided by applications (eg, function names). For example, information on functions provided by applications may require additional modification according to application expansion or function change. For example, the complexity of extracting the layout may increase because the implemented structure or method may differ depending on the purpose of the subject implementing the layout, and information extracted when an electronic device uses a screen layout extraction method provided at the system level. The accuracy of may be low, and it may also be difficult to connect the extracted information to the control operation.

Various embodiments disclosed in this document are an electronic device capable of performing a consistent control operation using the extracted text information by extracting text information in the same method (or condition) for all accessible screens, and the electronic device thereof. We want to provide an operating method.

An electronic device according to an embodiment disclosed in this document includes a microphone, a display, a communication circuit, a memory, and a processor operatively connected to the microphone, the display, the communication circuit, and the memory, When the memory is executed, the processor obtains an image of the screen output on the display, acquires attribute information of at least one element constituting the screen, and stores the attribute information in the attribute information. identifying an element capable of a user action based on the above, obtaining location information and size information of the identified element, and determining an area of the identified element on the image based on the location information and size information; , Extracting text information of the identified element from the image of the determined region, and performing a control operation corresponding to user speech received through the microphone based on attribute information, text information, and location information of the identified element. It may store one or more instructions to do so.

In addition, a method of operating an electronic device according to an embodiment disclosed in this document acquires an image of a screen output on a display, and obtains attribute information of at least one element constituting the screen. and identifies an element capable of a user action based on the attribute information, obtains location information and size information of the identified element, and identifies the identified element on the image based on the location information and size information. Determines an area of an element, extracts text information of the identified element from an image of the determined area, and responds to user speech received through a microphone based on attribute information, text information, and location information of the identified element. control operation can be performed.

According to various embodiments disclosed in this document, an electronic device capable of performing a consistent control operation using the extracted text information by extracting text information in the same method (or condition) for all accessible screens, and Its operating method can be provided.

According to various embodiments disclosed in this document, data throughput may be reduced by obtaining text information only for elements capable of user actions.

According to various embodiments disclosed in this document, even when the language of the text displayed on the screen does not match the user's language, the screen may be controlled in response to the user's utterance in the user's language.

In addition to this, various effects identified directly or indirectly through this document may be provided.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
2 is a block diagram illustrating an integrated intelligence system according to an embodiment.
3 is a diagram illustrating a form in which relationship information between concepts and actions is stored in a database according to an embodiment.
4 is a diagram illustrating a user terminal displaying a screen for processing a voice input received through an intelligent app, according to an embodiment.
5 is a block diagram of an electronic device according to an exemplary embodiment.
6 is a diagram illustrating software modules of an electronic device according to an exemplary embodiment.
7 is a diagram for explaining an operation of an electronic device according to an exemplary embodiment.
8 is a diagram for explaining an operation of an electronic device according to an exemplary embodiment.
9 is a diagram for explaining an operation of an electronic device according to an exemplary embodiment.
10 is a diagram for describing an operation of an electronic device according to an exemplary embodiment.
11 is a diagram for explaining an operation of an electronic device according to an exemplary embodiment.
12 is a diagram for explaining an operation of an electronic device according to an exemplary embodiment.
13 is a diagram for explaining an operation of an electronic device according to an exemplary embodiment.
14 is a flowchart illustrating an operation of an electronic device according to an exemplary embodiment.
15 is a flowchart illustrating an operation of an electronic device according to an exemplary embodiment.
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar elements.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 180 or communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor set to detect a touch or a pressure sensor set to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 is a peak data rate for eMBB realization (eg, 20 Gbps or more), a loss coverage for mMTC realization (eg, 164 dB or less), or a U-plane latency for URLLC realization (eg, Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2 is a block diagram illustrating an integrated intelligence system according to an embodiment.

Referring to FIG. 2 , an integrated intelligent system according to an embodiment may include a user terminal 201, an intelligent server 300, and a service server 400.

The user terminal 201 (eg, the electronic device 101 of FIG. 1 ) of an embodiment may be a terminal device (or electronic device) capable of connecting to the Internet, and may be, for example, a mobile phone, a smart phone, or a personal digital assistant (PDA). digital assistant), a notebook computer, a television (TV), a white appliance, a wearable device, a head mounted device (HMD), or a smart speaker.

According to the illustrated embodiment, the user terminal 201 may include a communication interface 290, a microphone 270, a speaker 255, a display 260, a memory 230, and/or a processor 220. there is. The components listed above may be operatively or electrically connected to each other.

The communication interface 290 (eg, the communication module 190 of FIG. 1 ) may be connected to an external device to transmit/receive data. The microphone 270 (eg, the audio module 170 of FIG. 1 ) may receive a sound (eg, a user's speech) and convert it into an electrical signal. The speaker 255 (eg, the sound output module 155 of FIG. 1 ) may output an electrical signal as sound (eg, voice). Display 260 (eg, display module 160 of FIG. 1 ) may be configured to display images or video. The display 260 according to an embodiment may also display a graphical user interface (GUI) of an app (or application program) being executed.

The memory 230 (eg, the memory 130 of FIG. 1 ) of an embodiment may store a client module 231 , a software development kit (SDK) 233 , and a plurality of applications. The client module 231 and the SDK 233 may constitute a framework (or solution program) for performing general functions. Also, the client module 231 or the SDK 233 may configure a framework for processing voice input.

The plurality of applications (eg, 355a and 355b) may be programs for performing designated functions. According to an embodiment, the plurality of applications may include a first app 235a and/or a second app 235b. According to one embodiment, each of the plurality of applications may include a plurality of operations for performing a designated function. For example, the applications may include an alarm app, a message app, and/or a schedule app. According to an embodiment, a plurality of applications may be executed by the processor 220 to sequentially execute at least some of the plurality of operations.

The processor 220 according to an embodiment may control overall operations of the user terminal 201 . For example, the processor 220 may be electrically connected to the communication interface 290, the microphone 270, the speaker 255, and the display 260 to perform a designated operation. For example, the processor 220 may include at least one processor.

The processor 220 of one embodiment may also execute a program stored in the memory 230 to perform a designated function. For example, the processor 220 may execute at least one of the client module 231 and the SDK 233 to perform the following operation for processing a voice input. The processor 220 may control operations of a plurality of applications through the SDK 233, for example. The following operations described as operations of the client module 231 or the SDK 233 may be operations performed by the execution of the processor 220 .

The client module 231 according to an embodiment may receive voice input. For example, the client module 231 may receive a voice signal corresponding to a user's speech detected through the microphone 270 . The client module 231 may transmit the received voice input (eg, voice signal) to the intelligent server 300 . The client module 231 may transmit state information of the user terminal 201 to the intelligent server 300 together with the received voice input. The state information may be, for example, execution state information of an app.

The client module 231 of one embodiment may receive a result corresponding to the received voice input from the intelligent server 300 . For example, the client module 231 may receive a result corresponding to the received voice input when the intelligent server 300 can calculate a result corresponding to the received voice input. The client module 231 may display the received result on the display 260 .

The client module 231 according to an embodiment may receive a plan corresponding to the received voice input. The client module 231 may display on the display 260 a result of executing a plurality of operations of the app according to the plan. The client module 231 may sequentially display, for example, execution results of a plurality of operations on a display. For another example, the user terminal 201 may display only a partial result of executing a plurality of operations (eg, a result of the last operation) on the display.

According to one embodiment, the client module 231 may receive a request for obtaining information necessary for calculating a result corresponding to a voice input from the intelligent server 300 . According to one embodiment, the client module 231 may transmit the necessary information to the intelligent server 300 in response to the request.

The client module 231 of one embodiment may transmit information as a result of executing a plurality of operations according to a plan to the intelligent server 300 . The intelligent server 300 can confirm that the received voice input has been properly processed using the result information.

The client module 231 according to an embodiment may include a voice recognition module. According to an embodiment, the client module 231 may recognize a voice input that performs a limited function through the voice recognition module. For example, the client module 231 may execute an intelligent app for processing a voice input by performing an organic operation in response to a designated voice input (eg, wake up!).

The intelligent server 300 of an embodiment transmits information related to the user's voice input from the user terminal 201 through a network 299 (eg, the first network 198 and/or the second network 199 of FIG. 1). can receive According to one embodiment, the intelligent server 300 may change data related to the received voice input into text data. According to an embodiment, the intelligent server 300 may generate at least one plan for performing a task corresponding to a user voice input based on the text data.

According to one embodiment, the plan may be generated by an artificial intelligent (AI) system. The artificial intelligence system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN)), and/or a recurrent neural network. network (RNN))). Alternatively, it may be a combination of the foregoing or other artificially intelligent systems. According to one embodiment, a plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the artificial intelligence system may select at least one plan from among a plurality of predefined plans.

An embodiment of the intelligent server 300 may transmit a result according to the generated plan to the user terminal 201, or transmit the generated plan to the user terminal 201. According to an embodiment, the user terminal 201 may display a result according to the plan on a display. According to an embodiment, the user terminal 201 may display a result of executing an operation according to a plan on a display.

The intelligent server 300 of one embodiment includes a front end 310, a natural language platform 320, a capsule database 330, an execution engine 340, It may include an end user interface 350 , a management platform 360 , a big data platform 370 , or an analytic platform 380 .

The front end 310 according to an embodiment may receive a voice input received by the user terminal 201 from the user terminal 201 . The front end 310 may transmit a response corresponding to the voice input to the user terminal 201 .

According to one embodiment, the natural language platform 320 includes an automatic speech recognition module (ASR module) 321, a natural language understanding module (NLU module) 323, a planner module ( planner module 325, natural language generator module (NLG module) 327, and/or text to speech module (TTS module) 329.

The automatic voice recognition module 321 according to an embodiment may convert voice input received from the user terminal 201 into text data. The natural language understanding module 323 according to an embodiment may determine the user's intention using text data of voice input. For example, the natural language understanding module 323 may determine the user's intention by performing syntactic analysis and/or semantic analysis. The natural language understanding module 323 of an embodiment identifies the meaning of a word extracted from a voice input using linguistic features (eg, grammatical elements) of a morpheme or phrase, and matches the meaning of the identified word to the intention of the user. intention can be determined.

The planner module 325 according to an embodiment may generate a plan using the intent and parameters determined by the natural language understanding module 323 . According to an embodiment, the planner module 325 may determine a plurality of domains required to perform a task based on the determined intent. The planner module 325 may determine a plurality of operations included in each of the determined plurality of domains based on the intention. According to an embodiment, the planner module 325 may determine parameters necessary for executing the determined plurality of operations or result values output by execution of the plurality of operations. The parameter and the resulting value may be defined as a concept of a designated format (or class). Accordingly, the plan may include a plurality of actions and/or a plurality of concepts determined by the user's intention. The planner module 325 may determine relationships between the plurality of operations and the plurality of concepts in stages (or hierarchically). For example, the planner module 325 may determine an execution order of a plurality of operations determined based on a user's intention based on a plurality of concepts. In other words, the planner module 325 may determine an execution order of the plurality of operations based on parameters required for execution of the plurality of operations and results output by the execution of the plurality of operations. Accordingly, the planner module 325 may generate a plan including information related to a plurality of operations and a plurality of concepts (eg, an ontology). The planner module 325 may generate a plan using information stored in the capsule database 330 in which a set of relationships between concepts and operations is stored.

The natural language generation module 327 according to an embodiment may change designated information into a text form. The information changed to the text form may be in the form of natural language speech. The text-to-speech conversion module 329 according to an embodiment may change text-type information into voice-type information.

According to one embodiment, some or all of the functions of the natural language platform 320 may be implemented in the user terminal 201 as well. For example, the user terminal 201 may include an automatic speech recognition module and/or a natural language understanding module. After the user terminal 201 recognizes the user's voice command, it may transmit text information corresponding to the recognized voice command to the intelligent server 300 . For example, the user terminal 201 may include a text-to-speech module. The user terminal 201 may receive text information from the intelligent server 300 and output the received text information as voice.

The capsule database 330 may store information about relationships between a plurality of concepts and operations corresponding to a plurality of domains. A capsule according to an embodiment may include a plurality of action objects (or action information) and/or concept objects (or concept information) included in a plan. According to an embodiment, the capsule database 330 may store a plurality of capsules in the form of a concept action network (CAN). According to an embodiment, a plurality of capsules may be stored in a function registry included in the capsule database 330.

The capsule database 330 may include a strategy registry in which strategy information necessary for determining a plan corresponding to a voice input is stored. The strategy information may include reference information for determining one plan when there are a plurality of plans corresponding to the voice input. According to an embodiment, the capsule database 330 may include a follow-up registry in which information on a follow-up action for suggesting a follow-up action to a user in a specified situation is stored. The follow-up action may include, for example, a follow-up utterance. According to an embodiment, the capsule database 330 may include a layout registry for storing layout information of information output through the user terminal 201 . According to an embodiment, the capsule database 330 may include a vocabulary registry in which vocabulary information included in capsule information is stored. According to an embodiment, the capsule database 330 may include a dialog registry in which dialog (or interaction) information with a user is stored. The capsule database 330 may update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating action objects or concept objects. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a strategy editor for creating and registering a strategy for determining a plan. The developer tool may include a dialog editor to create a dialog with the user. The developer tool may include a follow up editor that can activate follow up goals and edit follow up utterances that provide hints. The subsequent goal may be determined based on a currently set goal, a user's preference, or environmental conditions. In one embodiment, the capsule database 330 may also be implemented in the user terminal 201 .

The execution engine 340 of one embodiment may calculate a result using the generated plan. The end user interface 350 may transmit the calculated result to the user terminal 201 . Accordingly, the user terminal 201 may receive the result and provide the received result to the user. The management platform 360 of one embodiment may manage information used in the intelligent server 300 . The big data platform 370 according to an embodiment may collect user data. The analysis platform 380 of an embodiment may manage quality of service (QoS) of the intelligent server 300 . For example, the analysis platform 380 may manage the components and processing speed (or efficiency) of the intelligent server 300 .

The service server 400 according to an embodiment may provide a designated service (eg, food order or hotel reservation) to the user terminal 201 . According to one embodiment, the service server 400 may be a server operated by a third party. The service server 400 of one embodiment may provide information for generating a plan corresponding to the received voice input to the intelligent server 300 . The provided information may be stored in the capsule database 330. In addition, the service server 400 may provide result information according to the plan to the intelligent server 300. The service server 400 may communicate with the intelligent server 300 and/or the user terminal 201 through the network 299 . The service server 400 may communicate with the intelligent server 300 through a separate connection. Although the service server 400 is shown as one server in FIG. 2, the embodiments of this document are not limited thereto. At least one of the services 401, 402, and 403 of the service server 400 may be implemented as a separate server.

In the integrated intelligent system described above, the user terminal 201 may provide various intelligent services to the user in response to user input. The user input may include, for example, an input through a physical button, a touch input, or a voice input.

In one embodiment, the user terminal 201 may provide a voice recognition service through an internally stored intelligent app (or voice recognition app). In this case, for example, the user terminal 201 may recognize a user's utterance or voice input received through the microphone, and provide a service corresponding to the recognized voice input to the user. .

In one embodiment, the user terminal 201 may perform a designated operation alone or together with the intelligent server and/or service server based on the received voice input. For example, the user terminal 201 may execute an app corresponding to the received voice input and perform a designated operation through the executed app.

In one embodiment, when the user terminal 201 provides a service together with the intelligent server 300 and/or the service server, the user terminal detects user speech using the microphone 270, and the A signal (or voice data) corresponding to the detected user speech may be generated. The user terminal may transmit the voice data to the intelligent server 300 through the communication interface 290 .

Intelligent server 300 according to an embodiment, as a response to the voice input received from the user terminal 201, a plan for performing a task corresponding to the voice input, or an operation performed according to the plan can produce results. The plan may include, for example, a plurality of operations for performing a task corresponding to a user's voice input and/or a plurality of concepts related to the plurality of operations. The concept may define parameters input to the execution of the plurality of operations or result values output by the execution of the plurality of operations. The plan may include association information between a plurality of actions and/or a plurality of concepts.

The user terminal 201 of one embodiment may receive the response using the communication interface 290 . The user terminal 201 outputs a voice signal generated inside the user terminal 201 to the outside using the speaker 255, or transmits an image generated inside the user terminal 201 to the outside using the display 260. can be output as

3 is a diagram illustrating a form in which relationship information between a concept and an operation is stored in a database according to an exemplary embodiment.

The capsule database (eg, the capsule database 330) of the intelligent server 300 may store capsules in the form of a concept action network (CAN). The capsule database may store an operation for processing a task corresponding to a user's voice input and parameters necessary for the operation in the form of a concept action network (CAN).

The capsule database may store a plurality of capsules (Capsule A 331 and Capsule B 334) corresponding to each of a plurality of domains (eg, applications). According to an embodiment, one capsule (eg, capsule A 331) may correspond to one domain (eg, location (geo), application). In addition, one capsule includes at least one service provider capsule (eg, CP 1 (332), CP 2 (333), CP3 (335), and/or CP4 (336)) for performing functions for domains related to the capsule. )) can correspond. According to an embodiment, one capsule may include at least one operation 330a and at least one concept 330b to perform a designated function.

The natural language platform 320 may create a plan for performing a task corresponding to a received voice input using a capsule stored in the capsule database 330 . For example, the planner module 325 of the natural language platform may generate a plan using capsules stored in a capsule database. For example, plan 337 is created using operations 331a, 332a and concepts 331b, 332b of capsule A 3310 and operations 334a and concept 334b of capsule B 334. can do.

4 is a diagram illustrating a screen in which a user terminal processes a voice input received through an intelligent app according to an embodiment.

The user terminal 201 may execute an intelligent app to process user input through the intelligent server 300 .

According to one embodiment, on the first screen 410, when the user terminal 201 recognizes a designated voice input (eg, wake up!) or receives an input through a hardware key (eg, a dedicated hardware key), the user terminal 201 hears a voice input. You can run intelligent apps to process your input. The user terminal 201 may execute an intelligent app, for example, in a state in which a schedule app is executed. According to an embodiment, the user terminal 201 may display an object (eg, icon) 411 corresponding to an intelligent app on the display 260 . According to an embodiment, the user terminal 201 may receive a voice input caused by a user's speech. For example, the user terminal 201 may receive a voice input saying "Tell me this week's schedule!". According to an embodiment, the user terminal 201 may display a user interface (UI) 213 (eg, an input window) of an intelligent app displaying text data of the received voice input on the display.

According to an embodiment, on the second screen 415, the user terminal 201 may display a result corresponding to the received voice input on the display. For example, the user terminal 201 may receive a plan corresponding to the received user input and display 'this week's schedule' on the display according to the plan.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^™ ) or on two user devices ( It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

Hereinafter, a configuration and operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 5 .

5 is a block diagram 500 of an electronic device according to an embodiment.

According to an embodiment, the electronic device 501 (eg, the electronic device 101 of FIG. 1 or the user terminal 201 of FIG. 2 ) displays a display 510 (eg, the display module 160 of FIG. 1 or Display 260 of FIG. 2), microphone 520 (eg, input module 150 of FIG. 1 or microphone 270 of FIG. 2), communication circuit 530 (eg, communication module 190 of FIG. 1) or communication interface 290 of FIG. 2 ), memory 540 (eg memory 130 of FIG. 1 or memory 230 of FIG. 2 ), and processor 550 (eg processor 120 of FIG. 1 ). Alternatively, it may include the processor 220 of FIG. 2 . According to one embodiment, display 510, microphone 520, communication circuitry 530, memory 540, and processor 550 may be operatively coupled to each other.

According to an embodiment, the display 510 may visually provide information to the outside of the electronic device 101 (eg, a user). For example, the display 510 may display a graphic user interface (GUI) of an application (or app) being executed.

According to an embodiment, the microphone 520 may receive external sound and convert it into an electrical signal. For example, the microphone 520 may receive user speech.

According to one embodiment, the communication circuit 530 is connected to an external device (eg, the server 108 of FIG. 1, the intelligent server 300 of FIG. 2, or the service server 400 of FIG. 2) to transmit and receive data. can be configured to

According to an embodiment, the memory 540 may store one or more instructions executed by at least one component (eg, the processor 550) of the electronic device 501. According to one embodiment, the memory 540 may store various data used when executing instructions. For example, the memory 540 may store an image of a screen output to the display 510 . For example, the memory 540 may store at least one of attribute information, location information, and size information of at least one element constituting a screen output on the display 510 . For example, the memory 540 may store text information extracted from an image. For example, the memory 540 may store voice control information generated based on attribute information, location information, and text information of at least one element constituting a screen output on the display 510 .

According to an embodiment, the processor 550 may output an execution screen of an application through the display 510 . According to an embodiment, the processor 550 may obtain a screen image output to the display 510 . According to an embodiment, the processor 550 may analyze the layout of the screen output to the display 510 . For example, the processor 550 may obtain at least one of attribute information, location information, and size information of at least one element constituting the screen.

According to an embodiment, the attribute information may include information about whether a user action (or interaction) is possible. For example, a property capable of a user action may include a selectable property (eg, clickable), a property capable of receiving text (eg, EditText), or a property receiving a password (eg, password). For example, the attribute information may include information on a selectable attribute, a text input attribute, or a password input attribute.

According to an embodiment, the processor 550 may identify an element capable of a user action based on attribute information of at least one element. According to an embodiment, the processor 550 may identify an element having a user actionable attribute among at least one element. For example, the processor 550 may identify an element that is selectable, can receive text, or receives a password.

According to an embodiment, the processor 550 may obtain location information and size information of an element capable of an identified user action. According to an embodiment, the processor 550 may obtain position information and size information of an element capable of a user action from position information and size information of at least one element constituting a screen output on the display 510 . For example, location information may be coordinate information. For example, the size information may be at least one of width information and length information.

According to an embodiment, the processor 550 may determine a region of an element capable of a user action on an image of a screen output to the display 510 based on position information and size information of elements capable of a user action. According to an embodiment, the processor 550 may extract text information of an element capable of a user action from the image of the determined area. For example, the processor 550 may obtain text information from an area of each element capable of a user action on a screen image by using an optical character recognition (OCR) function. According to an embodiment, the processor 550 obtains text information corresponding to an element capable of a user action, thereby generating a candidate list for spoken text that the user can use to control the electronic device 501 based on voice. can be obtained According to an embodiment, the electronic device 501 may reduce data throughput by obtaining text information only for elements capable of user actions. According to an embodiment, the processor 550 may store acquired attribute information, location information, and text information of elements capable of user actions by corresponding them to each other.

According to an embodiment, the processor 550 may perform a control operation corresponding to a user's speech received through the microphone 520 based on attribute information, location information, and text information of an element capable of a user action. According to an embodiment, the processor 550 may generate voice control information based on attribute information, location information, and text information of an element capable of a user action. According to an embodiment, the voice control information may be information for controlling a screen output to the electronic device 501 or the display 510 based on user speech. According to an embodiment, the voice control information may include a command for performing control corresponding to attribute information with respect to an element capable of a user action corresponding to location information. According to one embodiment, the command may be stored in association with text information.

For example, if the attribute information of the first element is a selectable attribute, the voice control information may output a command for selecting the first element corresponding to the location information of the first element and a screen according to the selection of the first element. It may contain commands that:

As another example, if the attribute information of the second element is an attribute capable of receiving text input, the voice control information is a command for positioning the cursor on the second element corresponding to the location information of the second element and the second element. It may include a command for outputting a screen on which a cursor is placed.

As another example, when the attribute information of the third element is an attribute for receiving a password, the voice control information is a command for positioning the cursor on the third element corresponding to the location information of the third element, and the cursor on the third element. It may include a command for outputting a screen on which is placed, and a command for outputting a screen displaying an input password so that it cannot be identified upon receiving a user input for inputting a password.

According to one embodiment, a command may be stored in association with text information of an element capable of a user action. According to an embodiment, the processor 550 may execute a command associated with the text information by identifying text information of an element capable of a user action. According to an embodiment, the processor 550 may identify text information of an element capable of a user action based on text data corresponding to a user's speech. According to an embodiment, the processor 550 may identify text information having the highest probability of matching text data corresponding to a user's utterance from among text information of an element capable of a user action. According to an embodiment, the processor 550 may perform a control operation corresponding to user speech by executing a command associated with text information having a highest probability of matching text data corresponding to user speech.

According to an embodiment, the processor 550 may convert user speech received through the microphone 520 into text data. According to an embodiment, the processor 550 may correct the converted text data to match the text on the screen output to the display 510 . According to an embodiment, the processor 550 may correct text data based on text information of an element capable of a user action. For example, the processor 550 may change text data to match text information most similar to text data among text information of an element capable of a user action. According to an embodiment, the processor 550 may use the corrected text data as text data corresponding to the above-described user utterance. According to an embodiment, the processor 550 may perform a control operation corresponding to a user's utterance by executing a command associated with text information having the highest probability of matching the corrected text data.

According to an embodiment, the processor 550 may perform a control operation based on attribute information and location information corresponding to text information having the highest probability of matching the corrected text data among text information of an element capable of a user action. there is. According to an embodiment, the processor 550 may perform control corresponding to attribute information on an element corresponding to location information. According to an embodiment, the processor 550 may change the screen output to the display 510 according to attribute information as the control operation is performed.

For example, the processor 550 determines the text information with the highest probability of matching the corrected text data, based on the fact that the attribute information corresponding to the text information with the highest probability of matching with the corrected text data is a selectable attribute. An element corresponding to the corresponding location information may be selected, and a screen according to the selection of the corresponding element may be output to the display 510 .

As another example, the processor 550 has the highest probability of matching the corrected text data based on the fact that the attribute information corresponding to the text information with the highest probability of matching the corrected text data is an attribute that can receive text. A cursor may be positioned at an element corresponding to location information corresponding to text information, and a screen where the cursor is positioned at the corresponding element may be output to the display 510 .

As another example, the processor 550 determines that the text information with the highest probability of matching the corrected text data is based on the fact that the attribute information corresponding to the text information with the highest probability of matching the corrected text data is an attribute for receiving a password. A cursor may be positioned on an element corresponding to location information corresponding to information, and a screen where the cursor is positioned on the corresponding element may be output to the display 510 . Upon receiving a user input for inputting a password, the processor 550 may output a screen on the display 510 in which the input password cannot be identified.

According to the above-described embodiment, the processor 550 may extract text information from an image of an area determined as an area of an element capable of a user action on an image of a screen output to the display 510 . According to an embodiment, the processor 550 may determine whether the extracted text information corresponds to a language set as a user language. According to an embodiment, the language set as the user language may be a language set as a default language by the electronic device 501 based on a user input. According to an embodiment, the processor 550 may translate the extracted text information into the user language upon determining that the extracted text information is not a language set as the user language. According to an embodiment, the processor 550 may perform a control operation corresponding to the user's speech further based on the translated text information. According to the above-described embodiment, the processor 550 may generate voice control information based on attribute information, location information, and text information of an element capable of a user action. According to an embodiment, text information of an element capable of a user action used by the processor 550 to generate voice control information may include translated text information. For example, the processor 550 performs a control operation based on attribute information and location information corresponding to text information having the highest probability of matching text data corresponding to a user's utterance among translated text information of an element capable of a user action. can be done

Hereinafter, software modules of an electronic device according to an exemplary embodiment will be described with reference to FIG. 6 .

6 is a diagram 600 illustrating software modules of an electronic device according to an exemplary embodiment. For example, the software module shown in FIG. 6 may be a program 640 stored in the memory 540 of FIG. 5 . For example, the program 640 (eg, the program 140 of FIG. 1 ) may be executed by the electronic device 501 of FIG. 5 or the processor 550 of the electronic device 501 .

According to one embodiment, the program 640 includes a layout analyzer 641, an optical character recognizer (OCR) 643, a voice control information generator 645, an automatic may include an automatic speech recognition module (ASR module) 647, a natural language understanding module (NLU module) 649, and a control action handler 651. .

According to an embodiment, the layout analyzer 641 may obtain an image of a screen output to a display (eg, the display 510 of FIG. 5 ) and analyze the layout of the screen. According to an embodiment, the layout analyzer 641 may obtain attribute information of at least one element constituting a screen. According to an embodiment, the attribute information may include information about whether a user action (or user interaction) is possible. For example, a property capable of a user action may include a selectable property (eg, clickable), a property capable of receiving text (eg, EditText), or a property receiving a password (eg, password). According to an embodiment, the layout analyzer 641 may identify an element capable of a user action based on attribute information of at least one element.

According to an embodiment, the layout analyzer 641 may obtain location information and size information of an element capable of an identified user action. According to an embodiment, the layout analyzer 641 may determine a region of an element capable of a user action on a screen image based on location information and size information of an element capable of a user action.

According to an embodiment, the optical character recognizer 643 may extract text information from an image of a region of an element capable of a user action determined by the layout analyzer 641 . According to an embodiment, the optical character recognizer 643 may obtain text information of an element included in a screen and capable of a user action.

According to an embodiment, the voice control information generator 645 uses the information acquired by the layout analyzer 641 and the optical character recognizer 643 to generate information necessary for controlling the electronic device based on the user's voice. can do. For example, information required to control an electronic device based on a user's voice may be referred to as voice control information. According to an embodiment, the voice control information may include attribute information, location information, and size information of an element included in a screen capable of a user action. According to an embodiment, the voice control information generator 645 may format the generated voice control information into a form that can be delivered to the automatic voice recognition module 647 or the natural language understanding module 649 and transmit it. According to an embodiment, the automatic voice recognition module 647 or the natural language understanding module 649 may selectively use the transmitted voice control information.

According to an embodiment, the automatic voice recognition module 647 and the natural language understanding module 649 may perform a voice recognition function of an electronic device. For example, the automatic speech recognition module 647 may correspond to the automatic speech recognition module 321 of FIG. 2 , and the natural language understanding module 649 may correspond to the natural language understanding module 323 of FIG. 2 . For example, the automatic speech recognition module 647 and the natural language understanding module 649 may be modules included in an electronic device as shown in FIG. It may be a module included in an external electronic device.

According to an embodiment, the automatic voice recognition module 647 may convert user speech received through a microphone (eg, the microphone 520 of FIG. 5 ) into text data. According to an embodiment, the automatic voice recognition module 647 may correct the converted text data to match the text on the screen based on the voice control information delivered from the voice control information generator 645. For example, the voice control information generator 645 The voice control information may include text information of an element capable of a user action included in a screen, for example, the automatic voice recognition module 647 converts text from a user's speech among text information of an element capable of a user action. The text data may be changed to match text information most similar to the data According to an embodiment, the automatic speech recognition module 647 may transmit corrected text data to the natural language understanding module 649.

According to an embodiment, the natural language understanding module 649 controls the control corresponding to the user's speech based on the voice control information transmitted from the voice control information generator 645 and the corrected text data transmitted from the automatic voice recognition module 647. information can be determined. For example, the voice control information transmitted from the voice control information generator 645 may include text information of an element included in a screen capable of a user action and one or more commands associated with the text information. According to an embodiment, the natural language understanding module 649 may determine, as control information corresponding to user utterance, a command associated with text information having the highest probability of matching corrected text data among text information of an element capable of a user action. . According to one embodiment, the natural language understanding module 649 may transmit the determined command to the control action executor 651 .

According to one embodiment, the control operation executor 651 may perform a control operation by executing a command transmitted from the natural language understanding module 649 . According to an embodiment, the command may be a command based on location information and attribute information of an element included in a screen capable of a user action. According to an embodiment, the command may be to perform control corresponding to attribute information on an element capable of a user action corresponding to location information. According to an embodiment, the command may include a command for changing a screen output on a display according to attribute information as a control operation is performed. For example, the control operation executor 651 may select an element corresponding to location information based on attribute information being a selectable attribute, and may output a screen according to the selection of the corresponding element to a display. As another example, the control operation executor 651 locates the cursor on an element corresponding to the location information based on the attribute information being an attribute capable of receiving text, and outputs a screen where the cursor is positioned on the corresponding element on a display. can As another example, the control operation executor 651 displays a screen on which the input password cannot be identified when a user input for entering a password is received based on the fact that the attribute information is an attribute for receiving a password ( 510) can be output.

Although not shown in FIG. 6 , the program 640 according to an embodiment may further include other software modules. According to one embodiment, the program 640 may further include a translation module (not shown). According to an embodiment, when the text information acquired by the optical character recognizer 643 is not a language set as the user language, the translation module (not shown) may translate the text information into the user language. According to an embodiment, the translation module (not shown) may transmit the translated text information to the voice control information generator 645, and the voice control information generator 645 adds the translated text information to generate voice control information. can do.

Hereinafter, a user voice-based control operation of an electronic device according to an embodiment will be described with reference to FIGS. 7 to 9 .

7 is a diagram 700 for explaining an operation of an electronic device according to an exemplary embodiment. Operations of the electronic device 701 described later may be performed by, for example, the electronic device 501 of FIG. 5 or the processor 550 of the electronic device 501 .

According to an embodiment, the electronic device 701 may output the first screen 710 to a display (eg, the display 510 of FIG. 5 ). According to an embodiment, the first screen 710 may include at least one element, and each element may include content such as an image or text. For example, the first screen 710 may be an execution screen of an over the top (OTT) service application, and at least one element included in the first screen 710 is for providing information on each media content. It may include at least one thumbnail or poster.

According to an embodiment, the electronic device 701 may analyze the layout of the first screen 710 . According to an embodiment, the electronic device 701 may obtain attribute information 750 of at least one element included in the first screen 710 through layout analysis of the first screen 710 . For example, at least one element included in the first screen 710 may include an element 711 having an attribute capable of a user action and an element 712 having no attribute capable of a user action. According to an embodiment, the electronic device 701 may identify an element 711 having an attribute capable of a user action based on attribute information 750 of at least one element. Attributes capable of user actions may include, for example, selectable attributes (eg, clickable), attributes capable of receiving text input (eg, EditText), or attributes receiving input of passwords (eg, password). For example, a thumbnail or a poster on the first screen 710 may be an element having a selectable attribute, and may be an element 711 having a user actionable attribute. A textbox representing a title may be an element 712 that is not an attribute capable of user action.

According to an embodiment, the electronic device 701 may obtain position information and size information of at least one element included in the first screen 710 through layout analysis of the first screen 710 . For example, the electronic device 701 may obtain location information and size information of an element 711 having a user actionable attribute based on the attribute information 750 of at least one element. According to an embodiment, the electronic device 701 may at least temporarily store information acquired through layout analysis of the first screen 710 in a memory (eg, the memory 540 of FIG. 5 ) for each element. For example, the electronic device 701 may store attribute information, location information, and size information of an element 711 having a user actionable attribute to correspond to each other.

8 is a diagram 800 for explaining an operation of an electronic device according to an exemplary embodiment. In the description of FIG. 8 , the description of FIG. 7 may be applied in the same or similar manner to components using the same reference numerals as those of FIG. 7 . Operations of the electronic device 701 described later may be performed by, for example, the electronic device 501 of FIG. 5 or the processor 550 of the electronic device 501 .

According to an embodiment, the electronic device 701 may determine a region of an element having a property capable of a user action on the first screen 710 based on the property information 750 of at least one element. According to an embodiment, the electronic device 701 may identify an element having a property capable of a user action based on the property information 750 of at least one element. For example, an attribute capable of a user action may include a selectable attribute (eg, clickable). For example, the electronic device 701 may identify a first element 811, a second element 812, a third element 813, and a fourth element 814, which are elements having selectable properties. . According to an embodiment, the electronic device 701 is based on position information and size information of each of the first element 811, the second element 812, the third element 813, and the fourth element 814. Areas of each of the first element 811 , the second element 812 , the third element 813 , and the fourth element 814 on the first screen 710 may be determined.

According to an embodiment, the electronic device 701 may extract text information 850 of an element capable of a user action from an image of each area of an element having a property capable of a user action on the first screen 710 . For example, the electronic device 701 may display an image of the region of the first element 811 , an image of the region of the second element 812 , an image of the region of the third element 813 , and an image of the region of the fourth element 814 . Text information of each element can be extracted from the image of the area of . For example, the electronic device 701 provides text information (eg, first text) of the first element 811 from an image corresponding to an area of the first element 811 as at least a part of the image of the first screen 710. ) can be extracted. For example, the electronic device 701 provides text information (eg, second text) of the second element 812 from an image corresponding to the area of the second element 812 as at least a part of the image of the first screen 710. ) can be extracted. For example, the electronic device 701 provides text information (eg, third text) of the third element 813 from an image corresponding to an area of the third element 813 as at least a part of the image of the first screen 710. ) can be extracted. For example, the electronic device 701 provides text information (eg, fourth text) of the fourth element 814 from an image corresponding to an area of the fourth element 814 as at least a part of the image of the first screen 710. ) can be extracted. According to an embodiment, the electronic device 701 includes elements capable of user actions (eg, the first element 811, the second element 812, the third element 813, and the fourth element 814). Text information 850 may be stored. For example, the electronic device 701 may store text information 850 of an element having a property capable of a user action so as to correspond to at least one of property information, location information, or size information for each element.

According to an embodiment, the electronic device 701 includes elements (eg, a first element 811 , a second element 812 , a third element 813 , and a fourth element 814 having attributes capable of user actions). Voice control information may be generated based on the attribute information, location information, and text information 850 of )). According to an embodiment, the voice control information may include a command for performing control corresponding to attribute information on an element having an attribute capable of a user action corresponding to location information. According to one embodiment, the command may be stored in association with text information 850 . For example, since the first element 811, the second element 812, the third element 813, and the fourth element 814 are elements having selectable attributes, the voice control information is location information of each element. It may include a command for selecting each element corresponding to and a command for outputting a screen according to each element being selected.

9 is a diagram 900 for explaining an operation of an electronic device according to an exemplary embodiment. In the description of FIG. 9 , the descriptions of FIGS. 7 and 8 may be applied identically or similarly to the elements in which the same reference numerals as those of FIGS. 7 and 8 are used. Operations of the electronic device 701 described later may be performed by, for example, the electronic device 501 of FIG. 5 or the processor 550 of the electronic device 501 .

According to an embodiment, the electronic device 701 may receive the user's speech 910 through a microphone (eg, the microphone 520 of FIG. 5 ). According to an embodiment, the electronic device 701 may convert the user's speech 910 into text data. According to an embodiment, the electronic device 701 may correct text data converted from user utterance 910 based on text information 850 of an element having a property capable of a user action. For example, the electronic device 701 may change the text data to match the most similar text information among the text information 850 having a user actionable attribute. According to an embodiment, the electronic device 701 may change the text data into text data displayed on the first screen 710 by correcting the text data converted from the user's speech 910 .

According to an embodiment, the electronic device 701 performs control based on voice control information including text information having the highest probability of matching the corrected text data among the text information 850 of an element having a property capable of a user action. action can be performed. According to one embodiment, the voice control information may include text information and a command associated with the text information. According to an embodiment, the command may be to perform control corresponding to attribute information with respect to an element having an attribute capable of a user action corresponding to location information. Location information and attribute information included in the command may correspond to text information associated with the command. According to an embodiment, the electronic device 701 may perform a control operation by executing a command associated with text information having the highest probability of matching the corrected text data among the text information 850 having user actionable attributes. there is. According to an embodiment, the electronic device 701 may output the second screen 920 and the third screen 930 to a display (eg, the display 510 of FIG. 5 ) as the control operation is performed.

For example, when text data corresponding to the user speech 910 (eg, text data converted from the user speech 910 or text data corrected from the converted text data) is 'obja', the electronic device 701 ) may perform a control operation based on location information and attribute information corresponding to text information (eg, fourth text) having the highest probability of matching 'obja' among the text information 850 . For example, location information corresponding to the fourth text may be a coordinate value of the fourth element 814, and attribute information corresponding to the fourth text may be a selectable attribute. For example, the electronic device 701 may perform control to select a position corresponding to the coordinate value of the fourth element 814 on the first screen 710 . Selecting a location corresponding to the coordinate value of the fourth element 814 by the electronic device 701 may be referred to as selecting the fourth element 814 . The electronic device 701 may output a second screen 920 for selecting the fourth element 814 . For example, the second screen 920 may be a screen in which a cursor 925 is displayed at a position corresponding to the coordinate value of the fourth element 814 and the cursor 925 is clicked. . For example, the second screen 920 may be a screen showing a process in which the electronic device 701 performs a control operation. The electronic device 701 may output the third screen 930 as the fourth element 814 is selected. For example, the third screen 930 may be a screen on which media content corresponding to the fourth element 814 is reproduced.

Referring to FIG. 9 , a case in which the electronic device 701 selects the fourth element 814 is exemplified, but is not limited thereto, and the electronic device 701 displays the first screen 710 according to the user's speech 910. ), another element (eg, the first element 811, the second element 812, or the third element 813) may be selected. Screens output when the electronic device 701 selects another element may be similar to the second screen 920 and the third screen 930 .

Hereinafter, a user voice-based control operation of an electronic device according to an exemplary embodiment will be described with reference to FIGS. 10 to 13 .

7 to 9 have been described above, a screen output as a control operation is performed on an element having a selectable attribute of an electronic device has been described, but referring to FIGS. 10 to 13 hereinafter, the electronic device may receive text input. A screen displayed as a control operation is performed on an element having an attribute having a password or an element having a password input attribute will be described.

10 is a diagram 1000 for describing an operation of an electronic device according to an exemplary embodiment. Operations of the electronic device 1001 described below may be performed by, for example, the electronic device 501 of FIG. 5 or the processor 550 of the electronic device 501 .

According to an embodiment, the electronic device 701 may output the first screen 1010 to a display (eg, the display 510 of FIG. 5 ). According to an embodiment, the first screen 1010 may include at least one element, and each element may include content such as an image or text. For example, the first screen 1010 may be a login screen of an application, and at least one element included in the first screen 1010 receives user information and based on the input user information It may include at least one textbox and at least one button for logging in.

According to an embodiment, the electronic device 1001 may analyze the layout of the first screen 1010 . According to an embodiment, the electronic device 1001 may obtain attribute information 1050 of at least one element included in the first screen 1010 through layout analysis of the first screen 1010 . According to an embodiment, the attribute information 1050 of at least one element may include an attribute capable of a user action. According to an embodiment, the electronic device 1001 may identify an element having a property capable of a user action based on the property information 1050 of at least one element. Attributes capable of user actions may include, for example, selectable attributes (eg, clickable), attributes capable of receiving text input (eg, EditText), or attributes receiving input of passwords (eg, password). For example, in the first screen 1010, the ID input window is an element having a property to receive text, the password input window is an element having a property to receive a password, and the login button has a property to select text. As an element with a user action, it can be a property that is possible.

According to an embodiment, the electronic device 1001 may obtain position information and size information of at least one element included in the first screen 1010 through layout analysis of the first screen 1010 . For example, the electronic device 1001 may obtain position information and size information of an element having an attribute capable of a user action based on the attribute information 1050 of at least one element. According to an embodiment, the electronic device 1001 may at least temporarily store information acquired through layout analysis of the first screen 1010 in a memory (eg, the memory 540 of FIG. 5 ) for each element. For example, the electronic device 1001 may store attribute information, location information, and size information of an element having a user actionable attribute to correspond to each other.

11 is a diagram 1100 for describing an operation of an electronic device according to an exemplary embodiment. In the description of FIG. 11 , the description of FIG. 10 may be applied in the same or similar manner to components using the same reference numerals as those of FIG. 10 . Operations of the electronic device 1001 described later may be performed by, for example, the electronic device 501 of FIG. 5 or the processor 550 of the electronic device 501 .

According to an embodiment, the electronic device 1001 may determine a region of an element having a property capable of a user action on the first screen 1010 based on the property information 1050 of at least one element. According to an embodiment, the electronic device 1001 may identify an element having a property capable of a user action based on the property information 1050 of at least one element. Attributes capable of user actions may include, for example, selectable attributes (eg, clickable), attributes capable of receiving text input (eg, EditText), or attributes receiving input of passwords (eg, password). For example, the electronic device 1001 includes a first element 1111 that is an element having a property to receive text, a second element 1112 that is an element that has a property to receive a password, and an element that has a selectable property. It is possible to identify the third element 1113 that is. According to an embodiment, the electronic device 1001 is displayed on the first screen 1010 based on position information and size information of each of the first element 1111, the second element 1112, and the third element 1113. Regions of each of the first element 1111 , the second element 1112 , and the third element 1113 may be determined.

According to an embodiment, the electronic device 1001 may extract text information 1150 of an element capable of a user action from an image of each region of an element having a property capable of a user action on the first screen 1010 . For example, the electronic device 1001 obtains text information of each element from an image of the region of the first element 1111, an image of the region of the second element 1112, and an image of the region of the third element 1113. can be extracted. For example, the electronic device 1001 provides text information (eg, first text) of the first element 1111 from an image corresponding to an area of the first element 1111 as at least a part of an image of the first screen 1010. ) can be extracted. For example, the electronic device 1001 provides text information (eg, second text) of the second element 1112 from an image corresponding to the region of the second element 1112 as at least a part of the image of the first screen 1010. ) can be extracted. For example, the electronic device 1001 provides text information (eg, third text) of the third element 1113 from an image corresponding to the area of the third element 1113 as at least a part of the image of the first screen 1010. ) can be extracted. According to an embodiment, the electronic device 1001 stores text information 1150 of elements capable of user actions (eg, the first element 1111, the second element 1112, and the third element 1113). can For example, the electronic device 1001 may store text information 1150 of an element having a property capable of a user action so as to correspond to at least one of property information, location information, or size information for each element.

According to an embodiment, the electronic device 1001 provides attribute information and locations of elements (eg, the first element 1111, the second element 1112, and the third element 1113) having attributes capable of user actions. Voice control information may be generated based on the information and the text information 1150 . According to an embodiment, the voice control information may include a command for performing control corresponding to attribute information on an element having an attribute capable of a user action corresponding to location information. According to one embodiment, the command may be stored in association with the text information 1150 . For example, since the first element 1111 is an element having a text input property, the voice control information is placed on the first element 1111 corresponding to the location information of the first element 1111 as a cursor. It may include a command for locating and a command for outputting a screen in which a cursor is positioned on the first element 1111. For example, since the second element 1112 is an element having a property of receiving a password, the voice control information places a cursor on the second element 1112 corresponding to the location information of the second element 1112. A command to output a screen with a cursor placed on the second element 1112, and a user input for entering a password are received to display the screen so that the input password cannot be identified. It may contain commands to do so. For example, since the third element 1113 is an element having a selectable attribute, the voice control information includes a command for selecting the third element 1113 corresponding to location information of the third element 1113 and the third element. It may include a command for outputting a screen according to the selection of 1113.

12 is a diagram 1200 for explaining an operation of an electronic device according to an exemplary embodiment. In the description of FIG. 12 , the descriptions of FIGS. 10 and 11 may be applied identically or similarly to components using the same reference numerals as those of FIGS. 10 and 11 . Operations of the electronic device 1001 described later may be performed by, for example, the electronic device 501 of FIG. 5 or the processor 550 of the electronic device 501 .

According to an embodiment, the electronic device 1001 may receive user speech 1210 (eg, user id) through a microphone (eg, the microphone 520 of FIG. 5 ). According to an embodiment, the electronic device 1001 may convert the user's utterance 1210 into text data. According to an embodiment, the electronic device 1001 may correct text data converted from user utterance 1210 based on text information 1150 of an element having a property capable of a user action. For example, the electronic device 1001 may change the text data to match the most similar text information among the text information 1150 having a user actionable attribute. According to an embodiment, the electronic device 1001 may change the text data into text data displayed on the first screen 1010 by correcting the text data converted from the user's speech 1210 .

According to an embodiment, the electronic device 1001 performs control based on voice control information including text information having the highest probability of matching the corrected text data among the text information 1150 of an element having a property capable of a user action. action can be performed. According to one embodiment, the voice control information may include text information and a command associated with the text information. According to an embodiment, the command may be to perform control corresponding to attribute information with respect to an element having an attribute capable of a user action corresponding to location information. Location information and attribute information included in the command may correspond to text information associated with the command. According to an embodiment, the electronic device 1001 may perform a control operation by executing a command associated with text information having the highest probability of matching the corrected text data among the text information 1150 having properties capable of user actions. there is. According to an embodiment, the electronic device 1001 may output the second screen 1220 and the third screen 1230 to a display (eg, the display 510 of FIG. 5 ) as the control operation is performed.

For example, when text data corresponding to the user speech 1210 (eg, text data converted from the user speech 1210 or text data corrected from the converted text data) is 'user id', the electronic device ( 1001 ) may perform a control operation based on location information and attribute information corresponding to text information (eg, first text) having the highest probability of matching 'user id' among the text information 1150 . For example, the location information corresponding to the first text may be a coordinate value of the first element 1111, and the attribute information corresponding to the first text may be a text input attribute. For example, the electronic device 1001 may perform control to position a cursor (eg, the text cursor 1235) at a position corresponding to the coordinate value of the first element 1111 on the first screen 1010. . Placing the cursor (eg, the text cursor 1235 ) at a location corresponding to the coordinate value of the first element 1111 by the electronic device 1001 may include selecting the first element 1111 . The electronic device 1001 may output a second screen 1220 for selecting the first element 1111 . For example, the second screen 1220 may be a screen in which the cursor 1225 is displayed at a position corresponding to the coordinate value of the first element 1111 and the cursor 1225 is clicked. For example, the second screen 1220 may be a screen showing a process in which the electronic device 1001 performs a control operation. The electronic device 1001 may output the third screen 1230 as the first element 1111 is selected. For example, the third screen 1230 may be a screen in which a text cursor 1235 is displayed on the first element 1111 . For example, the text cursor 1235 may be a cursor 1225 displayed on the second screen 1220 transformed into a vertical bar shape indicating that text can be input.

13 is a diagram 1300 for describing an operation of an electronic device according to an exemplary embodiment. In the description of FIG. 13 , the descriptions of FIGS. 10 and 11 may be applied identically or similarly to components using the same reference numerals as those of FIGS. 10 and 11 . Operations of the electronic device 1001 described later may be performed by, for example, the electronic device 501 of FIG. 5 or the processor 550 of the electronic device 501 .

The embodiment shown in FIG. 13 is different from the embodiment shown in FIG. 12 in that the user utterance is changed from “user id” to “password”. Hereinafter, differences from the embodiment shown in FIG. 12 will be mainly described.

According to an embodiment, the electronic device 1001 may receive user speech 1310 (eg, password) through a microphone (eg, the microphone 520 of FIG. 5 ). According to an embodiment, the electronic device 1001 may convert the user's utterance 1310 into text data. According to an embodiment, the electronic device 1001 may correct text data converted from user utterance 1310 based on text information 1150 of an element having a property capable of a user action.

According to an embodiment, the electronic device 1001 performs control based on voice control information including text information having the highest probability of matching the corrected text data among the text information 1150 of an element having a property capable of a user action. action can be performed. As for the voice control information, the above description with reference to FIG. 12 may be applied identically or similarly. According to an embodiment, the electronic device 1001 may perform a control operation by executing a command associated with text information having the highest probability of matching the corrected text data among the text information 1150 having properties capable of user actions. there is. According to an embodiment, the electronic device 1001 displays the second screen 1320, the third screen 1330, and the fourth screen 1340 as the control operation is performed (eg, the display 510 of FIG. 5 ). )) can be output.

For example, when text data corresponding to the user speech 1310 (eg, text data converted from the user speech 1310 or text data corrected from the converted text data) is 'password', the electronic device 1001 ) may perform a control operation based on location information and attribute information corresponding to text information (eg, second text) having the highest probability of matching 'password' among the text information 1150 . For example, the location information corresponding to the second text may be a coordinate value of the second element 1112, and the attribute information corresponding to the second text may be an attribute for receiving a password. For example, the electronic device 1001 may perform control to position a cursor (eg, the text cursor 1335) at a position corresponding to the coordinate value of the second element 1112 on the first screen 1010. . Placing the cursor (eg, the text cursor 1335 ) at a position corresponding to the coordinate value of the second element 1112 by the electronic device 1001 may include selecting the second element 1112 . The electronic device 1001 may output a second screen 1320 for selecting the second element 1112 . For example, the second screen 1320 may be a screen in which the cursor 1225 is displayed at a position corresponding to the coordinate value of the second element 1112 and the cursor 1225 is clicked. For example, the second screen 1220 may be a screen showing a process in which the electronic device 1001 performs a control operation. The electronic device 1001 may output the third screen 1330 as the second element 1112 is selected. For example, the third screen 1330 may be a screen on which a text cursor 1335 is displayed on the second element 1112 . For example, the text cursor 1335 may be a cursor 1325 displayed on the second screen 1320 transformed into a vertical bar shape indicating that text can be input. Upon receiving a user input for inputting a password, the electronic device 1001 may display a fourth screen 1340 in which the input password 1345 is not identifiable. For example, the electronic device 1001 may display the password 1345 in a shape designated so that it cannot be identified.

The control operation of the electronic device when the attribute information corresponding to the text information with the highest probability of matching the text data corresponding to the user's utterance is a selectable attribute is identical or similar to that described above with reference to FIGS. 7 to 9 . can be applied For example, the electronic device 1001 may perform control to select a position corresponding to the coordinate value of the third element 1113 on the first screen 710 . The electronic device 1001 may output a screen (not shown) for selecting the third element 1113 . For example, on a screen (not shown) for selecting the third element 1113, a cursor 1325 is displayed at a position corresponding to the coordinate value of the third element 1113, and the cursor 1325 is clicked. It may be a screen that The electronic device 1001 may output a screen (not shown) after logging in as the third element 1113 is selected. For example, a screen (not shown) after login may be a screen of a user account accessed based on an input user ID and password.

Hereinafter, an operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 14 .

14 is a flowchart 1400 illustrating an operation of an electronic device according to an exemplary embodiment. Operations of the electronic device described later may be performed by the electronic device 501 of FIG. 5 or the processor 550 of the electronic device 501 .

In operation 1401, the electronic device may obtain an image of the output screen. According to an embodiment, the electronic device may obtain a screen image output to a display (eg, the display 510 of FIG. 5 ). For example, the screen output on the display may be an execution screen of an application.

In operation 1403, the electronic device may obtain attribute information of at least one element constituting the screen. According to an embodiment, the electronic device may analyze the layout of the screen output on the display. For example, the electronic device may obtain attribute information of at least one element constituting the screen through analysis of the layout of the screen. According to an embodiment, the attribute information may include information about whether a user action (or interaction) is possible. For example, a property capable of a user action may include a selectable property (eg, clickable), a property capable of receiving text (eg, EditText), or a property receiving a password (eg, password). According to an embodiment, the attribute information may include information on a selectable attribute, a text input attribute, or a password input attribute.

In operation 1405, the electronic device may identify an element capable of a user action. According to an embodiment, the electronic device may identify an element capable of a user action based on attribute information of at least one element. According to an embodiment, the electronic device may identify an element having a user actionable attribute among at least one element. For example, the electronic device may identify an element that is selectable, capable of receiving text input, or inputting a password.

In operation 1407, the electronic device may obtain location information and size information of the identified element. According to an embodiment, the electronic device may obtain position information and size information of at least one element through layout analysis of a screen. According to an embodiment, the electronic device may obtain location information and size information of an element identified as an element capable of a user action from location information and size information of at least one element. For example, location information may be coordinate information. For example, the size information may be at least one of width information and length information.

In operation 1409, the electronic device may determine a region of the identified element on the image. According to an embodiment, the electronic device may determine a region of the identified element on the image based on location information and size information. According to an embodiment, the electronic device may determine a region of an element capable of a user action on an image of a screen output to a display based on location information and size information of an element capable of a user action.

In operation 1411, the electronic device may extract text information of the identified element from the image of the determined region. According to an embodiment, the electronic device may extract text information of an element capable of a user action from an image of a region of an element capable of a user action. According to an embodiment, the image of the region of the element capable of a user action is at least a part of the image of the screen output to the display, and may be an image corresponding to the region of the element capable of the user action determined on the screen. For example, the electronic device may obtain text information from an area of each element capable of a user action on a screen image by using an optical character recognition (OCR) function.

In operation 1413, the electronic device may perform a control operation corresponding to user utterance based on the attribute information, text information, and location information of the identified element. According to an embodiment, the electronic device may generate voice control information based on attribute information, location information, and text information of an element capable of a user action. According to an embodiment, the voice control information may be information for controlling a screen output to an electronic device or a display based on user speech. According to an embodiment, the voice control information may include a command for performing control corresponding to attribute information with respect to an element capable of a user action corresponding to location information. According to one embodiment, a command may be stored in association with text information of an element capable of a user action.

According to an embodiment, the electronic device may execute a command associated with the text information by identifying text information of an element capable of a user action. According to an embodiment, the electronic device may identify text information of an element capable of a user action based on text data corresponding to a user's speech. According to an embodiment, the text data corresponding to the user's speech is obtained by converting the user's speech received by the electronic device through a microphone (eg, the microphone 520 of FIG. 5) into text data, or converting the converted text data into text data. may have been corrected. According to an embodiment, the electronic device may correct text data to match text on a screen output on a display based on text information of an element capable of a user action. For example, the electronic device may change text data to match text information most similar to text data among text information of an element capable of a user action. According to an embodiment, the electronic device may use the corrected text data as text data corresponding to the above-described user utterance. According to an embodiment, the electronic device may identify text information having the highest probability of matching the corrected text data among text information of an element capable of a user action. According to an embodiment, the electronic device may perform a control operation corresponding to a user's speech by executing a command associated with text information having the highest probability of matching the corrected text data.

According to an embodiment, the electronic device may perform a control operation based on attribute information and location information corresponding to text information having the highest probability of matching the corrected text data among text information of an element capable of a user action. According to an embodiment, the electronic device may perform control corresponding to attribute information on an element corresponding to location information. According to an embodiment, the electronic device may change a screen output on a display according to property information as a control operation is performed.

For example, the electronic device determines the text information corresponding to the text information most likely to match the corrected text data based on the fact that the attribute information corresponding to the text information with the highest probability of matching the corrected text data is a selectable attribute. An element corresponding to the location information may be selected, and a screen according to the selection of the corresponding element may be output to the display.

As another example, the electronic device provides text information with the highest probability of matching the corrected text data based on the fact that the attribute information corresponding to the text information with the highest probability of matching the corrected text data is an attribute that can receive text. A cursor may be positioned on an element corresponding to location information corresponding to , and a screen where the cursor is positioned on the corresponding element may be output on a display.

As another example, the electronic device determines the text information with the highest probability of matching the corrected text data based on the fact that the attribute information corresponding to the text information with the highest probability of matching the corrected text data is an attribute that receives a password. A cursor may be positioned on an element corresponding to the corresponding location information, and a screen where the cursor is positioned on the corresponding element may be output on a display. Upon receiving a user input for inputting a password, the electronic device may output a screen displaying the input password so that it cannot be identified on the display.

Hereinafter, an operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 15 .

15 is a flowchart 1500 illustrating an operation of an electronic device according to an exemplary embodiment. Operations of the electronic device described later may be performed by the electronic device 501 of FIG. 5 or the processor 550 of the electronic device 501 .

In operation 1501, the electronic device may extract text information of an element capable of a user action. According to an embodiment, operation 1501 may correspond to operation 1411 of FIG. 14 . According to an embodiment, the electronic device may extract text information from an image of an area determined as an area of an element capable of a user action on an image of a screen output to a display (eg, the display 510 of FIG. 5 ).

In operation 1503, the electronic device may determine whether the extracted text information corresponds to a language set as a user language. According to an embodiment, the language set as the user language may be a language set as a default language by the electronic device based on a user input. According to an embodiment, the electronic device may perform operation 1509 when it is determined that the extracted text information is a language set as the user language, and operations 1505 and 1507 when it is determined that the extracted text information is a language set as the user language. , and operation 1509 can be performed.

In operation 1505, the electronic device may translate the extracted text information into a user language. According to an embodiment, the electronic device may translate the extracted text information into the user language upon determining that the extracted text information is not a language set as the user language.

In operation 1507, the electronic device may add translated text information. According to an embodiment, the electronic device may generate voice control information by adding translated text information. According to an embodiment, the same or similar content described above with reference to FIG. 14 may be applied to voice control information. According to an embodiment, the voice control information may include a command for performing control corresponding to attribute information with respect to an element capable of a user action corresponding to location information. According to an embodiment, the electronic device may associate and store a command with translated text information.

In operation 1509, the electronic device may perform a control operation corresponding to user utterance based on attribute information, text information, and location information of an element capable of a user action. According to one embodiment, operation 1509 may correspond to operation 1413 of FIG. 14 . According to an embodiment, when performing operation 1509 after performing operation 1507, the electronic device may perform a control operation corresponding to user utterance further based on translated text information of an element capable of a user action. According to an embodiment, the electronic device may match text corresponding to user speech among translated text information of elements capable of user actions (eg, text data converted from user speech or text data corrected from converted text). A control operation corresponding to a user's speech may be performed by executing a command associated with text information having the highest probability.

According to an embodiment disclosed in this document, an electronic device (eg, the electronic device 101 of FIG. 1 , the user terminal 201 of FIGS. 2 and 4 , the electronic device 501 of FIG. 5 , and the electronic device 501 of FIGS. 7 to 9 The electronic device 701 of or the electronic device 1001 of FIGS. 10 to 13) is a microphone (eg, the input module 150 of FIG. 1, the microphone 270 of FIG. 2, or the microphone 520 of FIG. 5) ), a display (eg, the display module 160 of FIG. 1, the display 260 of FIG. 2, or the display 510 of FIG. 5), a communication circuit (eg, the communication module 190, the communication interface of FIG. 2 ( 290), or the communication circuit 530 of FIG. 5), a memory (eg, the memory 130 of FIG. 1, the memory 230 of FIG. 2, or the memory 540 of FIG. 5), and the microphone, the display , the communication circuit, and a processor operatively coupled with the memory (eg, the processor 120 of FIG. 1, the processor 220 of FIG. 2, or the processor 550 of FIG. 5), wherein the When the memory is executed, the processor obtains an image of the screen output on the display, acquires attribute information of at least one element constituting the screen, and obtains attribute information based on the attribute information. to identify an element capable of a user action, obtain location information and size information of the identified element, determine a region of the identified element on the image based on the location information and size information, Extract text information of the identified element from the image of the determined region, and perform a control operation corresponding to a user's speech received through the microphone based on attribute information, text information, and location information of the identified element. It may store one or more instructions (instructions) to do.

According to an embodiment disclosed in this document, the attribute information may include information on a selectable attribute, a text input attribute, or a password input attribute.

According to an embodiment disclosed in this document, the instructions cause the processor to generate voice control information based on attribute information, location information, and text information of the identified element, and the voice control information comprises: and a command for performing control corresponding to the attribute information on an element corresponding to location information, and the command may be stored in association with the text information.

According to an embodiment disclosed in this document, the instructions include: the processor converts the user's utterance into text data, and corrects the text data to match text on the screen based on text information of the identified element. can make it

According to an embodiment disclosed in this document, the instructions may be performed by the processor based on attribute information and location information corresponding to text information having a highest probability of matching the corrected text data among the text information of the identified element. to perform a control operation, and the control operation may be to perform control corresponding to the attribute information on an element corresponding to the location information.

According to an embodiment disclosed in this document, the instructions may cause the processor to change the screen according to the attribute information as the control operation is performed.

According to an embodiment disclosed in this document, the instructions cause the processor to determine the location information based on attribute information corresponding to text information having the highest probability of matching the corrected text data, which is a selectable attribute. A screen can be displayed according to the selection of the corresponding element.

According to an embodiment disclosed in this document, the instructions are based on the attribute information corresponding to the text information having the highest probability of matching the corrected text data, the processor being an attribute capable of receiving text input, A screen in which a cursor is placed on an element corresponding to the location information may be output.

According to an embodiment disclosed in this document, the instructions cause the processor to determine the corrected text data based on the fact that attribute information corresponding to text information having the highest probability of matching the corrected text data is an attribute of receiving a password, A screen in which a cursor is placed on an element corresponding to information may be output, and upon receiving a user input for inputting a password, a screen displaying the input password unidentifiable may be displayed.

According to an embodiment disclosed in this document, the instructions are such that the processor determines whether the extracted text information is a language set as a user language, and determines that the extracted text information is not a language set as a user language. Accordingly, the extracted text information may be translated into the user's language, and a control operation corresponding to the user's utterance may be performed further based on the translated text information of the identified element.

According to an embodiment disclosed in this document, an electronic device (eg, the electronic device 101 of FIG. 1 , the user terminal 201 of FIGS. 2 and 4 , the electronic device 501 of FIG. 5 , and the electronic device 501 of FIGS. 7 to 9 The operating method of the electronic device 701 of or the electronic device 1001 of FIGS. 10 to 13 is a display (e.g., the display module 160 of FIG. 1, the display 260 of FIG. 2, or the display of FIG. 5) (510)) Acquire an image of a screen output, acquire attribute information of at least one element constituting the screen, and obtain an element capable of a user action based on the attribute information Identifying, obtaining location information and size information of the identified element, determining an area of the identified element on the image based on the location information and size information, and determining the identified element from the image of the determined area. Extract text information of the element, and based on the attribute information, text information, and location information of the identified element, a microphone (eg, the input module 150 in FIG. 1, the microphone 270 in FIG. 2, or the microphone 270 in FIG. 5) A control operation corresponding to the user's speech received through the microphone 520 may be performed.

According to an embodiment disclosed in this document, the attribute information may include information on a selectable attribute, a text input attribute, or a password input attribute.

According to an embodiment disclosed in this document, the method generates voice control information based on attribute information, location information, and text information of the identified element, and the voice control information corresponds to the location information. and a command for performing control corresponding to the attribute information on an element, and the command may be stored in association with the text information.

According to an embodiment disclosed in this document, the method may convert the user utterance into text data, and correct the text data to match text on the screen based on text information of the identified element.

According to an embodiment disclosed in this document, the method performs a control operation based on attribute information and location information corresponding to text information having the highest probability of matching the corrected text data among the text information of the identified element. and the control operation may be to perform control corresponding to the attribute information on an element corresponding to the location information.

According to an embodiment disclosed in this document, the method may change the screen according to the attribute information as the control operation is performed.

According to an embodiment disclosed in this document, the method selects an element corresponding to the location information based on attribute information corresponding to text information having the highest probability of matching the corrected text data, which is a selectable attribute. You can output the screen according to the selection.

According to an embodiment disclosed in this document, based on the fact that attribute information corresponding to text information with the highest probability of matching the corrected text data is an attribute capable of receiving text, an element corresponding to the location information The screen where the cursor is located can be output.

According to an embodiment disclosed in this document, the method, based on attribute information corresponding to text information having the highest probability of matching the corrected text data, is an attribute for receiving a password, the method corresponding to the location information. A screen on which a cursor is positioned on an element may be output, and a screen displaying the entered password so that it cannot be identified may be output as a user input for inputting a password is received.

According to an embodiment disclosed in this document, the method determines whether the extracted text information is a language set as the user language, and determines that the extracted text information is not a language set as the user language. Text information may be translated into the user's language, and a control operation corresponding to the user's utterance may be performed further based on the translated text information of the identified element.

Claims (20)

In electronic devices,
mike;
display;
communication circuit;
Memory; and
a processor operatively connected with the microphone, the display, the communication circuitry, and the memory;
The memory, when executed, the processor,
Obtaining an image of a screen output on the display;
Acquiring attribute information of at least one element constituting the screen;
Identifying an element capable of a user action based on the attribute information,
Acquiring location information and size information of the identified element;
determine a region of the identified element on the image based on the location information and the size information;
Extracting text information of the identified element from the image of the determined region;
An electronic device that stores one or more instructions for performing a control operation corresponding to a user's speech received through the microphone based on attribute information, text information, and location information of the identified element. The method of claim 1,
The attribute information includes information on a selectable attribute, a text input attribute, or a password input attribute. The method of claim 1,
The instructions, the processor,
To generate voice control information based on attribute information, location information, and text information of the identified element;
The voice control information includes a command for performing a control corresponding to the attribute information with respect to an element corresponding to the location information, and the command is stored in association with the text information. The method of claim 1,
The instructions, the processor,
converting the user utterance into text data;
The electronic device that corrects the text data to match the text on the screen based on the text information of the identified element. The method of claim 4,
The instructions, the processor,
Perform a control operation based on attribute information and location information corresponding to text information having the highest probability of matching the corrected text data among the text information of the identified element;
The control operation is to perform a control corresponding to the attribute information with respect to an element corresponding to the location information. The method of claim 5,
The instructions, the processor,
As the control operation is performed, the electronic device changes the screen according to the property information. The method of claim 6,
The instructions, the processor,
The electronic device outputs a screen according to selecting an element corresponding to the location information based on that attribute information corresponding to text information having the highest probability of matching the corrected text data is a selectable attribute. The method of claim 6,
The instructions, the processor,
Based on the fact that the attribute information corresponding to the text information with the highest probability of matching the corrected text data is an attribute that can receive text, a screen with a cursor placed on an element corresponding to the location information is output. An electronic device to do so. The method of claim 6,
The instructions, the processor,
Based on the fact that attribute information corresponding to text information with the highest probability of matching the corrected text data is an attribute for receiving a password, a screen in which a cursor is positioned at an element corresponding to the location information is output; An electronic device that, upon receiving a user input for entering a password, outputs a screen displaying the input password so that it cannot be identified. The method of claim 1,
The instructions, the processor,
determining whether the extracted text information is a language set as a user language;
Translating the extracted text information into the user language when it is determined that the extracted text information is not a language set as the user language;
and performing a control operation corresponding to the user's utterance further based on the translated text information of the identified element. In the operating method of the electronic device,
Acquire the image of the screen output on the display,
Acquiring attribute information of at least one element constituting the screen;
Identifying an element capable of a user action based on the attribute information,
Acquiring location information and size information of the identified element;
determine a region of the identified element on the image based on the location information and the size information;
Extracting text information of the identified element from the image of the determined region;
A method of performing a control operation corresponding to a user's speech received through a microphone based on attribute information, text information, and location information of the identified element. The method of claim 11,
The attribute information includes information on a selectable attribute, a text input attribute, or a password input attribute. The method of claim 11,
The method,
generating voice control information based on attribute information, location information, and text information of the identified element;
The voice control information includes a command for performing a control corresponding to the attribute information with respect to an element corresponding to the location information, and the command is stored in association with the text information. The method of claim 11,
The method,
converting the user utterance into text data;
The method of correcting the text data to match the text on the screen based on the text information of the identified element. The method of claim 14,
The method,
Performing a control operation based on attribute information and location information corresponding to text information having the highest probability of matching the corrected text data among the text information of the identified element;
The control operation is to perform a control corresponding to the attribute information with respect to an element corresponding to the location information. The method of claim 15
The method,
As the control operation is performed, the screen is changed according to the attribute information. The method of claim 16
The method,
The method of outputting a screen according to selecting an element corresponding to the location information based on that attribute information corresponding to text information having the highest probability of matching the corrected text data is a selectable attribute. The method of claim 16
Based on the fact that the attribute information corresponding to the text information with the highest probability of matching the corrected text data is an attribute that can receive text, a screen with a cursor placed on an element corresponding to the location information is output. How to. The method of claim 16
The method,
Based on the fact that attribute information corresponding to text information with the highest probability of matching the corrected text data is an attribute for receiving a password, a screen in which a cursor is positioned at an element corresponding to the location information is output; A method of outputting a screen displaying an input password not to be identified upon receiving a user input for entering a password. The method of claim 11,
The method,
determining whether the extracted text information is a language set as a user language;
Translating the extracted text information into the user language when it is determined that the extracted text information is not a language set as the user language;
and performing a control operation corresponding to the user utterance further based on the translated text information of the identified element.

Images