KR20180055638A - Electronic device and method for controlling electronic device using speech recognition - Google Patents

Abstract

According to various embodiments, an electronic device includes a microphone, a display, and a processor. The processor displays content including at least one object and at least one text corresponding to the at least one object obtained on the basis of resources constituting the content on the display, determines a first text corresponding to a speech received using the microphone in the at least one text, and executes a command corresponding to the received speech on a first object corresponding to the first text in the at least one object on the basis of at least one command registered to control the at least one object. Other embodiments are possible. When a user desires to control an object included in content on a screen using speech input, according to various embodiments, the electronic device and the method of controlling the electronic device using speech recognition allow the user to easily grasp an object that can be currently selected and executed and attributes of the object.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an electronic device and a control method of an electronic device using speech recognition,

Various embodiments relate to electronic devices and methods of controlling electronic devices using speech recognition.

Conventional electronic devices require additional input devices such as a mouse or keyboard, or a hardware device such as a touchable display for controlling the content displayed on the screen. For example, in the related art, a user can select a desired object from the content displayed on the screen of the electronic device using an input device such as a mouse to execute a specific function on the object, have.

In recent years, with the development of technology, electronic devices have been able to control content on the screen using speech recognition. Control technology using speech recognition has been developed so that an electronic device recognizes a vocabulary such as a promised word and executes the function of an application or can control by voice through separately created information.

In the case where there is no input device such as a mouse or a keyboard or an input for control of the electronic device through the conventional input method is difficult due to an environmental constraint such as a user moving on a distance or the like, Controlling the content can be difficult.

In addition, conventionally, additional information has to be written in order to control an electronic device through voice recognition.

In addition, the control of the electronic device through the conventional speech recognition technology can be performed only when the user displays the help text on the screen and the electronic device recognizes the user's voice to perform a specific function There was a limit.

According to various embodiments, it is possible to provide an electronic device and a control method of an electronic device using speech recognition so that control such as selecting and executing at least one object included in the displayed content using the user's voice can be provided . For example, when the distance between a user and an electronic device is too long, or an electronic device that does not support an input device such as a mouse or a keyboard, or a situation where the user is not free to use his / It is possible to make control such as selecting and executing an object using the voice of the user when the input for controlling the electronic device is difficult.

According to various embodiments, there is provided an electronic device for providing a function of controlling, by voice, at least one object of content, which is not content generated for control using speech recognition, without further modification, and an electronic device using speech recognition A control method can be provided.

According to various embodiments, there is provided an electronic device for providing a function of executing various attributes and controllable actions of the object by voice, rather than simply selecting and executing an object included in the content, A control method can be provided.

An electronic device according to various embodiments, comprising: a microphone; display; And a processor, wherein the processor displays on the display content including at least one object and at least one text corresponding to the at least one object obtained based on the resources that make up the content Determining a first text corresponding to a voice received using the microphone among the at least one text, and determining, based on the at least one command registered to control the at least one object, And executing a command corresponding to the received voice for a first object corresponding to the first text.

A method of controlling an electronic device using speech recognition in accordance with various embodiments, the method comprising: generating content including at least one object and at least one text corresponding to the at least one object obtained based on the resources constituting the content On the display; Determining a first text corresponding to a voice received using the microphone among the at least one text; And executing a command corresponding to the received voice for a first object corresponding to the first text of the at least one object based on at least one command registered to control the at least one object .

A machine-readable storage medium having recorded thereon a program for executing a method of controlling an electronic device using speech recognition in accordance with various embodiments, the method comprising the steps of: receiving content comprising at least one object; Displaying on the display at least one text corresponding to the at least one object obtained based on the resource; Determining a first text corresponding to a voice received using the microphone among the at least one text; And executing a command corresponding to the received voice for a first object corresponding to the first text of the at least one object based on at least one command registered to control the at least one object .

An electronic device according to various embodiments and a control method of an electronic device using voice recognition can perform an attribute of a specific object included in content displayed on an electronic device and various user control commands through voice input.

An electronic device according to various embodiments and a control method of an electronic device using speech recognition may be configured such that when a user wants to control an object included in content on the screen by voice input, Can be easily grasped.

1 is a configuration diagram of an electronic device according to various embodiments.
2 is a flow diagram of operations for creating and registering instructions for controlling an electronic device using speech recognition in accordance with various embodiments.
3 is a detailed operation flow diagram for generating an instruction for controlling an object to be displayed on a display of an electronic device according to various embodiments.
4 is a flowchart of operations for controlling objects included in content according to speech recognition of an electronic device (e.g., electronic device 101) according to various embodiments.
Figures 5 and 6 are illustrations of display screens of an electronic device according to various embodiments.
7A to 7C are exemplary diagrams of screens sequentially displaying the GUI and / or text of each of the objects according to the designated importance of the electronic device according to various embodiments.
8A to 8C are exemplary diagrams of screens sequentially displaying the GUI of each of the objects according to the designated importance of the electronic device according to various embodiments.
9 is an illustration of an operation for controlling objects included in content through speech recognition of an electronic device according to various embodiments.
10 is an illustration of an operation for controlling objects included in content through speech recognition of an electronic device according to various embodiments.
11 is an illustration of an operation for controlling objects contained in content via user input of an electronic device according to various embodiments.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. It is to be understood that the embodiments and terminologies used herein are not intended to limit the invention to the particular embodiments described, but to include various modifications, equivalents, and / or alternatives of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar components. The singular expressions may include plural expressions unless the context clearly dictates otherwise. In this document, the expressions "A or B" or "at least one of A and / or B" and the like may include all possible combinations of the items listed together. Expressions such as " first, "" second," " first, "or" second, " But is not limited to those components. When it is mentioned that some (e.g., first) component is "(functionally or communicatively) connected" or "connected" to another (second) component, May be connected directly to the component, or may be connected through another component (e.g., a third component).

In this document, the term " configured to (or configured) to "as used herein is intended to encompass all types of hardware, software, Quot ;, "made to do "," made to do ", or "designed to" In some situations, the expression "a device configured to" may mean that the device can "do " with other devices or components. For example, a processor configured (or configured) to perform the phrases "A, B, and C" may be implemented by executing one or more software programs stored in a memory device or a dedicated processor (e.g., an embedded processor) , And a general purpose processor (e.g., a CPU or an application processor) capable of performing the corresponding operations.

Electronic devices in accordance with various embodiments of the present document may be used in various applications such as, for example, smart phones, tablet PCs, mobile phones, videophones, electronic book readers, desktop PCs, laptop PCs, netbook computers, workstations, a portable multimedia player, an MP3 player, a medical device, a camera, or a wearable device. Wearable devices may be of the type of accessories (eg, watches, rings, bracelets, braces, necklaces, glasses, contact lenses or head-mounted-devices (HMD) (E.g., a skin-pads or tattoo), or a bio-implantable circuit. In some embodiments, the electronic device may be, for example, a television, a digital video disk (Such as Samsung HomeSync ^™ , Apple TV ^™ , or Google TV ^™ ), a refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, air purifier, set top box, home automation control panel, A game console (e.g., Xbox ^TM , PlayStation ^TM ), an electronic dictionary, an electronic key, a camcorder, or an electronic frame.

In an alternative embodiment, the electronic device may be any of a variety of medical devices (e.g., various portable medical measurement devices such as a blood glucose meter, a heart rate meter, a blood pressure meter, or a body temperature meter), magnetic resonance angiography (MRA) A navigation system, a global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), an automobile infotainment device, a marine electronic equipment (For example, marine navigation systems, gyro compasses, etc.), avionics, security devices, head units for vehicles, industrial or domestic robots, drones, ATMs at financial institutions, of at least one of the following types of devices: a light bulb, a fire detector, a fire alarm, a thermostat, a streetlight, a toaster, a fitness device, a hot water tank, a heater, a boiler, . According to some embodiments, the electronic device may be a piece of furniture, a building / structure or part of an automobile, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (e.g., Gas, or radio wave measuring instruments, etc.). In various embodiments, the electronic device is flexible or may be a combination of two or more of the various devices described above. The electronic device according to the embodiment of the present document is not limited to the above-described devices. In this document, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

1 is a configuration diagram of an electronic device 101 according to various embodiments.

The electronic device 101 may include a processor 120, a memory 130, an input / output interface 150, a display 160, and a communication interface 170. In some embodiments, the electronic device 101 may omit at least one of the components or additionally include other components.

Processor 120 may include one or more of a central processing unit, an application processor, or a communications processor (CP). The processor 120 may perform computations or data processing related to, for example, control and / or communication of at least one other component of the electronic device 101.

Memory 130 may include volatile and / or non-volatile memory. Memory 130 may store instructions or data related to at least one other component of electronic device 101, for example. According to one embodiment, the memory 130 may store software and / or programs. The program 140 may include, for example, a kernel, a middleware, an application programming interface (API), and / or an application program (or "application").

Output interface 150 may be configured to communicate commands or data entered from a user or other external device to another component (s) of the electronic device 101, or to another component (s) of the electronic device 101 ) To the user or other external device.

The input / output interface 150 may include a microphone 151, for example.

The display 160 may include a display such as, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectromechanical system (MEMS) display, or an electronic paper display . Display 160 may display various content (e.g., text, images, video, icons, and / or symbols, etc.) to a user, for example. Display 160 may include a touch screen and may receive a touch, gesture, proximity, or hovering input using, for example, an electronic pen or a portion of the user's body.

The communication interface 170 establishes communication between the electronic device 101 and an external device (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106) . For example, communication interface 170 may be connected to network 162 via wireless or wired communication to communicate with an external device (e.g., second external electronic device 104 or server 106).

The wireless communication may include, for example, LTE, LTE-A (LTE Advance), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) System for Mobile Communications), and the like. According to one embodiment, the wireless communication may be wireless communication, such as wireless fidelity (WiFi), Bluetooth, Bluetooth low power (BLE), Zigbee, NFC, Magnetic Secure Transmission, Frequency (RF), or body area network (BAN). According to one example, wireless communication may include GNSS. GNSS may be, for example, Global Positioning System (GPS), Global Navigation Satellite System (Glonass), Beidou Navigation Satellite System (Beidou) or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, "GPS" can be used interchangeably with "GNSS ". The wired communication may include, for example, at least one of a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard 232 (RS-232), a power line communication or a plain old telephone service have. Network 162 may include at least one of a telecommunications network, e.g., a computer network (e.g., LAN or WAN), the Internet, or a telephone network.

Each of the first and second external electronic devices 102, 104 may be the same or a different kind of device as the electronic device 101. According to various embodiments, all or a portion of the operations performed in the electronic device 101 may be performed in one or more other electronic devices (e.g., electronic devices 102, 104, or server 106). According to the present invention, when electronic device 101 is to perform a function or service automatically or on demand, electronic device 101 may perform at least some functions associated therewith instead of, or in addition to, (E.g., electronic device 102, 104, or server 106) may request the other device (e.g., electronic device 102, 104, or server 106) Perform additional functions, and forward the results to the electronic device 101. The electronic device 101 may process the received results as is or additionally to provide the requested functionality or services. For example, Cloud computing, distributed computing, or client-server computing technology may be utilized.

According to various embodiments, in an electronic device (e.g., electronic device 101), a microphone (e.g., microphone 151); Display (e.g., display 160); And a processor (e.g., processor 120), the processor comprising: at least one processor configured to process at least one object corresponding to the at least one object obtained based on the resource comprising the at least one object; Displaying the text on the display, determining a first one of the at least one texts corresponding to the received voice using the microphone, and based on at least one command registered to control the at least one object, And executing a command corresponding to the received voice for a first one of the at least one object corresponding to the first text.

According to various embodiments, the processor is configured to analyze the resource, to distinguish the at least one object within the content based on an analysis result of the resource, and to determine at least one text corresponding to the at least one object And registering the at least one command for controlling the at least one object based on an analysis result of the resource.

According to various embodiments, the processor determines at least one non-text object that is not text among the at least one object based on the resource, and determines the at least one text to the at least one non- And displaying on the display correspondingly.

According to various embodiments, the processor determines an object to which a graphical user interface is to be applied, based on an attribute of the at least one object obtained based on the resource, And displaying the determined object on the display in correspondence with a graphical user interface corresponding to the determined object attribute.

According to various embodiments, the processor is configured to determine, based on at least one of the size, position, type, and logging information of the at least one object, the size, color, and / or size of the at least one text corresponding to the at least one object. And at least one of a font and a font to display on the display.

According to various embodiments, the processor determines an order to display the at least one text corresponding to the at least one object on the display based on the determined importance, and sequentially On the display. ≪ RTI ID = 0.0 >

According to various embodiments, the processor may include determining the importance based on at least one of the size, location, type, and logging information of the at least one object.

According to various embodiments, the apparatus further includes a camera (not shown), wherein the processor determines an area of the user's gaze on the display based on the image acquired through the camera, And displaying on the display text corresponding to objects included in the determined area.

According to various embodiments, the processor is configured to determine, when displaying the at least one text on the display, whether there are any texts to be overlapped and displayed among the at least one text, And displaying only one of the texts to be displayed overlapping with each other based on the predetermined importance.

According to various embodiments, the at least one object may include at least one of text, images, moving images, icons, links, and input areas.

According to various embodiments, the content may comprise a web page.

Figure 2 is a flow diagram of operations for creating and registering an instruction to control an electronic device (e.g., electronic device 101) using speech recognition in accordance with various embodiments. 3 is a detailed operation flow diagram for generating an instruction to control an object to be displayed on a display of an electronic device (e.g., electronic device 101) according to various embodiments.

Referring to FIG. 2, an electronic device analyzes resources constituting content to determine at least one object included in the content and at least one operation to control the at least one object, and to voice the at least one object A command for controlling can be registered. The content may be a web page composed of the resource and may be displayed on the display of the electronic device or other external device connected to the electronic device. The resource may be a standardized document such as an HTML (Hyper Text Markup Language) document or an XML (Extensible Markup Language) document.

In operation 210, the electronic device may distinguish at least one object included in the content.

According to one embodiment, the electronic device may obtain an object list for the objects included in the content.

According to one embodiment, the electronic device can analyze the resources constituting the content and distinguish objects included in the content based on the analysis result of the resource. For example, the electronic device can distinguish objects included in the content according to the type of tag included in the resource. For example, the object may be a text, an image, a moving picture, an icon, a link, or an input area.

In operation 220, the electronic device may obtain operational information for controlling the at least one object.

According to one embodiment, the electronic device can obtain the operation information list for the operation information for controlling the object for each object.

According to one embodiment, the operation information may include a user operation (input) for the object and at least one control operation of the electronic device according to the user operation.

According to one embodiment, the electronic device may analyze the attributes on an object-by-object basis to obtain operation information for controlling each object.

For example, the electronic device can perform various operations such as user manipulation such as clicking, mouse-over, dragging, scrolling, and the like by voice output, text copy, And at least one control operation of an object of the electronic device according to the present invention.

For example, the electronic device can analyze the tags included in the resources constituting the content to obtain operation information that can be controlled on an object-by-object basis.

For example, when the attribute of the first object performs an operation of changing the color of the first object when the mouse cursor is moved, if the attribute of the first object is defined in the resource of the content including the first object, And the color changing operation of the first object according to the movement of the mouse cursor can be obtained as operation information for controlling the first object.

As another example, when the first object is a link (also referred to as a hyperlink), the attribute of the first object is moved to another part of the current screen through input such as click or touch, Lt; RTI ID = 0.0 > a < / RTI > The electronic device moves to another part of the current screen through input such as click or touch based on the attribute of the first object defined in the resource of the content, And can be obtained as operation information.

As another example, when the first object is text, the attributes of the first object may include audio output through application of TTS (text to speech) technology through touch, text selection through dragging, and / or search through drag and click Or may be defined in the resource of the content including the first object. The electronic device can perform a voice output by applying a text to speech (TTS) technique through touch, a text selection through dragging, and / or a search operation by drag and click based on the attribute of the first object defined in the resource of the content As operation information for controlling the first object.

According to another embodiment, the operation information may be previously specified for each object type.

For example, operation information may be specified in advance for each object type in the resource of the content. More specifically, the operation information may be defined for each type of the object in the tag included in the resource of the content.

As another example, the electronic device may have previously specified operation information for each object type.

In operation 230, the electronic device may generate and register an instruction based on the operation information of the at least one object and the at least one object.

According to one embodiment, the electronic device may generate and register at least one command (also referred to as a text command) based on the at least one object and the operation information of the at least one object.

According to one embodiment, the instruction may include a selection portion for specifying an object and a control portion for controlling the object. For example, the electronic device may generate or select a first instruction, which is a selection portion for specifying each object, by determining or selecting a corresponding text for each object. If the object is a text, the electronic device can select a word or a sentence included in the text as a first command, and if the object is non-text rather than text, the electronic device transmits the content of the content The first instruction can be determined based on the first instruction. Further, the electronic device can generate a second instruction, which is a control part corresponding to each operation information for each object.

The operation of generating the command will be described in detail with reference to FIG. Referring to FIG. 3, the command may be determined differently depending on whether the first object is text or not. For example, if the first object is text, the electronic device may determine all or some of the words of the text as the first instruction for selection of the first object. If the first object is a non-text such as an image or an icon, the electronic device may be provided with an automatic ID-type command such as a number or an alphabet as a first command for selecting the first object.

In operation 310, the electronic device may determine that the first object is text. For example, the electronic device may analyze the resource of the content including the first object to determine whether the first object is text. In 310 operation, if the electronic device determines that the first object is text, then 320 operation may be performed, otherwise 330 operation may be performed.

In operation 320, the electronic device may determine at least a portion (a word or a sentence) of text as a first object as a first instruction for selection of a first object.

According to one embodiment, the electronic device may store a condition for determining a first instruction and may use the stored condition to determine a first instruction.

For example, the electronic device can analyze a tag of a resource to identify a subject portion (a subject word or a subject sentence) of the text, and determine the subject portion as the first command.

As another example, the electronic device can determine the first sentence of the text by analyzing the tag of the resource, and determine the first sentence as the first instruction.

As another example, the electronic device may analyze resources to compare the text with relatively bold words or phrases, relatively large words or phrases, underlined words or phrases, and / or other words or phrases, Or a sentence or the like to determine an emphasized word or sentence, and the emphasized word or sentence can be determined as a first command.

As another example, the electronic device may determine all words and / or all sentences contained in the text as a first instruction.

In operation 330, the electronic device may assign text to a first object and determine the text as a first instruction for selection of the first object.

According to one embodiment, the text may be a number, an alphabet, or the like.

According to one embodiment, the text may be displayed as a tool tip when displayed on a screen.

According to one embodiment, the electronic device may analyze the attributes of the first object in the resource to render the text.

For example, in consideration of attributes such as the type of the first object, the position at which the first object is displayed, the position of the first object at the resource, and / or the size at the time when the first object is displayed, Can be given.

For example, if there is a non-text object in addition to the first object in the content including the first object, the text of the first object may be a text that is separated from the text of another non-text object have. That is, if there are a plurality of non-text objects in the content, the electronic device may assign different texts to each of the plurality of non-text objects based on the attribute. For example, if there are three non-text objects in the content, the electronic device will display the number 1 for the first non-text object to be displayed the largest and the number 1 for the second non- 3 for the remaining non-text objects, and the number 2 for the remaining non-text objects.

In operation 340, the electronic device may determine a second instruction to control the first object based on the first motion information obtained to control the first object.

According to an embodiment, the first operation information may include a first user operation and a first control operation according to the first user operation. For example, the electronic device can extract a first user operation in the first control operation and a first control operation in accordance with the first user operation. Further, the electronic device may generate a second instruction corresponding to the first user operation, and the instruction may correspond to the first control operation. For example, when the first user operation is a click, a mouse over, a drag, a scroll, a TTS, or a search, the second instruction may be click, mouse over, drag, scroll, TTS, or search.

According to one embodiment, the electronic device may generate a plurality of second instructions. For example, it is possible to generate a variety of possible second instructions that the user may utter for the first user operation so that the user may execute the first action even if only one of the second instructions is uttered have.

According to one embodiment, the second command may be predefined for each user operation included in the control operation, or the second command may be stored for each control operation based on stored user voice commands or voice command statistical information of a plurality of people, It may be one that the electronic device generates automatically.

In operation 350, the electronic device may generate instructions for controlling the first object based on the first instruction and the second instruction. For example, the electronic device may generate an instruction that includes a first instruction and a second instruction.

4 is a flowchart of operations for controlling objects included in content according to speech recognition of an electronic device (e.g., electronic device 101) according to various embodiments.

Referring to FIG. 4, when an electronic device displays content on a display, it can display text corresponding to a non-text object that is not a text object contained in the content. Further, the electronic device recognizes the voice of the user and can perform a control operation on the object.

According to one embodiment, when the user wants to click on a picture to which a command is assigned with the number "12 ", the user can utter the voice with" 12 clicks " The electronic device receiving the voice may determine whether there is a command corresponding to the uttered voice among the registered commands. For example, the electronic device may determine a command corresponding to the uttered voice, by comparing the uttered voice with registered commands. For example, the electronic device may recognize "12" as the first instruction of the selection portion for specifying the object in the instruction and "click " as the second instruction of the control portion of the recognized object 12. [ Accordingly, the electronic device can execute a control operation corresponding to the determined command, that is, an operation of clicking the object 12. [

In operation 410, the electronic device may display the content and at least one text corresponding to at least one object contained in the content.

According to one embodiment, the content may be a web page, and the at least one object may be an image, text, video, icon, and the like.

According to one embodiment, the electronic device receives content comprising at least one object, and the electronic device is operable to generate the commands described with reference to Figs. 2 and 3 above. In addition, the electronic device may display the received content on a display, and display the text assigned to each of the non-text objects included in the content to correspond to each of the non-text objects. The text assigned to each of the non-text objects is selected from among text included in the content to be displayed according to operation 330 of FIG. 3 during execution of an operation of generating a command, Lt; / RTI > The content and the text assigned to each of the non-text objects may be displayed on the display at the same time, or after the content is displayed on the display, Text may be displayed.

According to another embodiment, an electronic device receives content comprising at least one object, receives text assigned to each of the non-text objects contained in the content, and responds to each of the objects contained in the content Lt; RTI ID = 0.0 > 1 < / RTI > In addition, the electronic device may display the received content on a display, and display the text assigned to each of the non-text objects included in the content to correspond to each of the non-text objects. The content and the text assigned to each of the non-text objects may be displayed on the display at the same time, or after the content is displayed on the display, Text may be displayed.

In operation 420, the electronic device may receive a voice signal. For example, an electronic device can receive a user's voice using a microphone.

In operation 430, the electronic device can recognize the voice of the received voice signal. For example, the electronic device can recognize the voice of the received voice signal using voice recognition technology. The electronic device can convert the received voice signal into text and recognize the converted text.

In operation 440, the electronic device may select a registered command corresponding to the recognized voice.

For example, the electronic device may determine a command corresponding to the recognized voice, by comparing the recognized voice with registered commands. For example, the electronic device may determine an instruction comprising a first instruction of a selection portion for specifying an object in the content based on the recognized speech and a second instruction of a control portion for controlling the first object .

As another example, the electronic device may use natural language processing techniques to determine an instruction corresponding to a command intended by the recognized speech. For example, when the recognized voice is "touch the first object ", the electronic device can determine the intention of the recognized voice by applying a natural language processing technique, and the registered command" Quot; click "to determine the command of" first object click ".

In 450 operation, the electronic device may execute an operation corresponding to the instruction. For example, the electronic device may execute a second instruction contained in the instruction.

According to various embodiments, the electronic device further comprises a graphical user interface (GUI) in operation to display at least one text corresponding to at least one object contained in the content of the 410 operation and the content Can be displayed. For example, the electronic device can identify an attribute for each object included in the content based on the resource of the content, and can determine an object to which the GUI is applied among each object included in the content based on the identified attribute. As another example, the electronic device can determine an object to which the GUI is applied among each object included in the content based on a corresponding instruction for each object. Thereafter, the electronic device can display the attribute of the determined object or the GUI corresponding to the command on the determined object on the display. Thereby allowing the displayed GUI to act as a guide so that the user can speak clearly to control a particular object.

According to one embodiment, the GUI may be various icons, images, tooltips, and the like.

According to one embodiment, the attribute may be various, such as the type of the object and / or the operation information of the object. For example, when the operation information on the first object is text input other than the basic operation such as clicking, or the first button of the mouse and the second button of the second button, etc., The designated GUI can be displayed on the display in association with the first object.

5 is an exemplary view of a display screen of an electronic device (e.g., electronic device 101) according to various embodiments.

5, the electronic device displays tool tips (e.g., texts) assigned numbers 1 through 8 corresponding to the web page and image objects 501, 502, 503, 504, 505, 506, 507, (551, 552, 553, 554, 555, 556, 557, 558). In addition, the electronic device can display the green guide GUI 511 on the object 501 having the text input attribute. Also, the electronic device can display the guide GUIs 561, 562, 563, and 564 of yellow color on each of the objects 521, 522, 523, and 524 having the extended attribute through the mouseover.

According to the embodiment of FIG. 5 described above, the electronic device may display the corresponding text to overlap the image objects. In performing such an operation, if a text of the same size is displayed to be overlapped on all the objects, the user may have an unnatural feeling about the objects displayed on the electronic device. In addition, for example, on the layout of the screen of the electronic device, a relatively large image object may occupy an important part in the content. When the text is displayed reflecting such a characteristic, Command can be uttered.

Thus, according to various embodiments, the electronic device may be configured such that the size of the text to be displayed is differently applied, depending on the size of each image object included in the content displayed on the screen of the electronic device.

According to one embodiment, the electronic device may change and display the text size to correspond to the size of each of the image objects. The electronic device can identify a portion related to the size of each of the image objects in the resources constituting the content, and set the text size to correspond to the size of each of the objects. 6, a tool tip 653 of an image object 603 having the largest size among image objects 601, 602, 603, 604, 605, 606, 607, The tool tips 651, 652, 654, 655, 656, 657, and 658 of the first to sixth embodiments 601, 602, 604, 605, 606, 607, The electronic devices also include tool tips 651 and 652 of the image objects 601 and 602 having the smallest display size and tool tips 653 and 654 of the other objects 603, 604, 605, 606, 607, , 655, 656, 657, 658).

On the other hand, by applying the above-described embodiment of changing the text size of the image object to the GUI in the same manner, the GUI size can be changed and displayed so as to correspond to the size of each object to which the GUI is applied.

In general, if all the text corresponding to each of the GUI and image objects corresponding to the objects at one time is displayed together with the content such as a web page, the screen may appear dirty or the user may feel rejection. In order to solve this problem, according to various embodiments, the electronic device can classify objects according to the designated importance and determine the order of displaying the GUI and the text based thereon.

According to one embodiment, an electronic device classifies objects according to relative importance such as an input window, an object having a large size, a small object, an advertisement object advertising a specific product, and the like, / / You can determine the order layer that displays the text. For example, if there is a user input for a start trigger to perform a speech utterance, the GUI and / or text may be sequentially displayed according to a specified order layer.

According to another embodiment, the electronic device can determine the importance of the object based on the type of the object, the size of the object, and / or the position of the object, and the GUI corresponding to each object according to the importance of the object And / or the order in which the text is to be displayed. For example, the electronic device can determine that the larger the size of the object is, the higher the degree of importance, and display the GUI and / or text corresponding to each object in order of importance, i.e., in descending order of size, In the case of the same objects, GUI and / or text corresponding respectively to the corresponding order can be displayed. Alternatively, the electronic device can determine that the position of the object is more important in the center of the screen, and can display the GUI and / or text corresponding to each of the objects included in the designated area in order of importance, .

According to another embodiment, the electronic device can determine the importance of an object based on logging information (e.g., a selected number of times) of each of the objects included in the content, and responds to each of the objects according to the importance of the object And the order in which the GUI and / or text is to be displayed. The electronic device can use the number of times information of a specific object selected in the content is logged and used as metadata. For example, the electronic device can determine that the greater the number of selected objects is, the higher the degree of importance, and the GUI and / or text corresponding to each object can be displayed in order of importance, i.e., in a descending order of the number of times.

According to yet another embodiment, the electronic device can determine the importance of an object based on the user's gaze on the content, determine an order to display the GUI and / or text corresponding to each of the objects according to the importance of the object . For example, an electronic device can acquire an image of a user using a camera, and can determine at least a part of an area of a screen of an electronic device, The determined part of the area can be determined as a high importance area, and the other areas can be determined as a low importance area. For example, the electronic device may first display the GUI and / or text corresponding to each of the objects included in the determined high importance area, and then display the GUI corresponding to each of the objects included in the remaining area have.

7A to 7C are exemplary diagrams of screens sequentially displaying the GUI and / or text of each of the objects according to the designated importance of the electronic device according to various embodiments.

Referring to Figures 7A and 7C, the electronic device is configured to receive image objects 703, 704, 705, 706, 707, 708 for a link among the objects based on at least one of the above- 1 important object, designates an image object 701 for input included in the search word input window and an image object 702 for search execution as a second important object, and other menu objects 721, 722, 723 , 724) can be designated as the third important object.

When the electronic device executes the web page or executes the speech recognition function, the electronic device may first display a screen as shown in FIG. 7A based on the designated importance. Referring to FIG. 7A, the first tool tips 753, 754, 755, 756, 757, 758 (corresponding to each of the objects 703, 704, 705, 706, 707, 708 for the link designated as the first important object) ) Can be displayed.

The electronic device can display the screen as shown in FIG. 7B in accordance with the specified condition such as elapse of the designated time, user operation, and the like. For example, the electronic device may display an image object 701 for an input designated as a second important object, and a search execution for the input specified by the first tool tips 753, 754, 755, 756, 757, 752 corresponding to each of the image objects 702 for the input image 702 and a first GUI 711 corresponding to the image object 601 for input.

According to the above-specified conditions, the electronic device can display a screen as shown in FIG. 7C. For example, the electronic device may further include a third tool tip 753, 754, 755, 756, 757, 758, second tool tips 771, 752 and a first GUI 711, Each of the second GUIs 761, 762, 763, and 764 corresponding to each of the other menu objects 721, 722, 723, and 724 designated as an important object.

8A to 8C are exemplary diagrams of screens sequentially displaying the GUI of each of the objects according to the designated importance of the electronic device according to various embodiments.

8A to 8C are exemplary views of a GUI and / or text sequentially displayed on each of the objects according to the designated importance of the electronic device according to various embodiments.

Referring to FIGS. 8A and 8C, an electronic device may include image objects 803, 804, 805, 806, 807, and 808 for a link among objects based on at least one of the importance determination conditions described above. 1 as an important object and designates the image object 801 for the input included in the search word input window and the image object 802 for the search execution as the second important object and the other menu objects 821, 822, 823 , 824) can be designated as the third important object.

When the electronic device executes the web page or executes the voice recognition function, the electronic device may first display a screen as shown in FIG. 8A based on the designated importance level. 8A, first tool tips 853, 854, 855, 856, 857, 858 (corresponding to each of the objects 803, 804, 805, 806, 807, 808 for the link designated as the first important object) ) Can be displayed.

The electronic device can display the screen as shown in FIG. 8B in accordance with the specified condition such as elapse of the designated time, user operation, and the like. For example, after completing the display of the first tool tips 853, 854, 855, 856, 857, 858, the electronic device may display an image object 801 for the input designated as the second important object, Each of the second tool tips 851 and 852 corresponding to each of the image objects 802 and the first GUI 811 corresponding to the image object 601 for input can be displayed.

According to the above-specified conditions, the electronic device can display a screen as shown in FIG. 8C. For example, after terminating the display of the second tool tips 871, 852 and the first GUI 811, the electronic device may display the other menu objects 821, 822, 823, 824 designated as the third important object 862, 863, and 864 corresponding to the second GUIs 861, 862, 863, and 864, respectively.

According to various embodiments, the electronic device may display a GUI and / or text corresponding to at least one object located at a point of view of the user's view of the content. For example, an electronic device can acquire an image of a user using a camera, and can determine at least some areas of a screen of the electronic device that the user's line of sight is directed to in the image of the user obtained using the face recognition technology. The electronic device may display a GUI and / or text corresponding to each of the objects included in the determined area.

According to various embodiments, when displaying a GUI and / or text on each of the objects, when some GUI and / or text are expected to be overlapped, the electronic device may display the highest importance object of the GUI and / Only the corresponding GUI and / or text can be displayed on the screen. Alternatively, when displaying a GUI and / or text on each of the objects, it is expected that some GUI and / or text will overlap, but if the GUI and / or text obscured by the overlap is determined to be recognizable, The GUI and / or text may be displayed in a superimposed manner.

9 is an illustration of an operation for controlling objects included in content through speech recognition of an electronic device according to various embodiments.

Referring to Fig. 9, for example, when a user desires to display (extend) a submenu of "Departments ", the user can utter a voice command" Cursor on departments ". The electronic device receiving the voice command can determine the command corresponding to the uttered voice command by comparing the uttered voice command with the registered commands. For example, the electronic device recognizes "Departments" as a first instruction of a selection part for specifying an object in an instruction, and "Cursor on" Can be recognized. Accordingly, the electronic device can display the submenu of "Departments " by moving the control operation corresponding to the determined command, i.e., the cursor to" Departments ".

10 is an illustration of an operation for controlling objects included in content through speech recognition of an electronic device according to various embodiments.

Referring to FIG. 10, for example, when a user wants to execute a TTS function for a specific article in a displayed web page, the user can utter a voice command "Read Nobel Prize Winner". The electronic device receiving the voice command can determine the command corresponding to the uttered voice command by comparing the uttered voice command with the registered commands. For example, the electronic device recognizes the " Nobel Prize Winner " as the first instruction of the selection part for specifying an object in the command, and reads " Read " It can be recognized as a second instruction word. Thus, the electronic device can execute the TTS function for article articles containing the control action corresponding to the determined command, i.e., "Nobel Laureate Year ". Additionally, if the article does not display a full text, the electronic device may cause the article's full text to be displayed.

11 is an illustration of an operation for controlling objects contained in content via user input of an electronic device according to various embodiments.

11, a menu for displaying (expanding and displaying) the detailed menu such as "Department" includes a guide GUI 1101 indicating that the user is controllable through voice commands and / or a guide GUI (1102) can be displayed. Accordingly, the user can intuitively recognize the control method through the form of the guide GUIs 1101 and 1102. [ For example, if the user speaks a voice command "Cursor on departments" or moves the mouse cursor over the "Department" menu according to user input, the electronic device may display a submenu of "Departments".

According to various embodiments, there is provided a method of controlling an electronic device (e.g., electronic device 101) using speech recognition, the method comprising the steps of: receiving content including at least one object; Displaying at least one text corresponding to one object on the display; Determining a first text corresponding to a voice received using the microphone among the at least one text; And executing a command corresponding to the received voice for a first object corresponding to the first text of the at least one object based on at least one command registered to control the at least one object .

According to various embodiments, the operations of analyzing the resource, distinguishing the at least one object within the content based on the analysis result of the resource, and determining at least one text corresponding to the at least one object, And registering the at least one command for controlling the at least one object based on an analysis result of the resource.

According to various embodiments, the act of displaying on the display at least one text corresponding to the at least one object comprises the steps of: displaying at least one non- And displaying the at least one text on the display by mapping the at least one text to the at least one non-text object.

According to various embodiments, there is provided a method comprising: determining an object to which a graphical user interface is to be applied of the at least one object based on an attribute of the at least one object obtained based on the resource; And associating a graphical user interface corresponding to an attribute of the determined object with the graphic user interface to display on the display.

In accordance with various embodiments, the act of displaying on the display at least one text corresponding to the at least one object may include, based on at least one of the size, position, type, and logging information of the at least one object, Adjusting at least one of a size, a color, and a font of the at least one text corresponding to the at least one object to display on the display.

In accordance with various embodiments, the act of displaying on the display at least one text corresponding to the at least one object may include, based on at least one of the size, position, type, and logging information of the at least one object, Determining an order of importance, determining an order in which the at least one text corresponding to the at least one object is displayed on the display based on the determined importance, and determining the order in which the at least one And displaying the text on the display.

According to various embodiments, the method further comprises determining an area of the user's gaze on the display, based on the image obtained through the camera, wherein at least one text corresponding to the at least one object is displayed on the display May include displaying on the display text corresponding to an object included in the determined one of the at least one object.

According to various embodiments, the act of displaying on the display at least one text corresponding to the at least one object comprises: determining whether there are any of the at least one text to be displayed overlapping with each other, And determining that there are texts to be overlapped to be displayed, displaying only one of the texts to be overlapped and displayed based on the predetermined importance.

According to various embodiments, there is provided a machine-readable storage medium having recorded thereon a program for executing a method of controlling an electronic device (e.g., electronic device 101) using speech recognition, the method comprising: Displaying on the display at least one text corresponding to the at least one object obtained based on the content comprising the content and the resource comprising the content; Determining a first text corresponding to a voice received using the microphone among the at least one text; And executing a command corresponding to the received voice for a first object corresponding to the first text of the at least one object based on at least one command registered to control the at least one object .

As used herein, the term "module " includes units comprised of hardware, software, or firmware and may be used interchangeably with terms such as, for example, logic, logic blocks, components, or circuits. A "module" may be an integrally constructed component or a minimum unit or part thereof that performs one or more functions. "Module" may be implemented either mechanically or electronically, for example, by application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs) And may include programmable logic devices. At least some of the devices (e.g., modules or functions thereof) or methods (e.g., operations) according to various embodiments may be stored in a computer readable storage medium (e.g., memory 130) . ≪ / RTI > When the instruction is executed by a processor (e.g., processor 120), the processor may perform a function corresponding to the instruction. The computer-readable recording medium may be a hard disk, a floppy disk, a magnetic medium such as a magnetic tape, an optical recording medium such as a CD-ROM, a DVD, a magnetic-optical medium such as a floppy disk, The instructions may include code that is generated by the compiler or code that may be executed by the interpreter. Modules or program modules according to various embodiments may include at least one or more of the components described above Operations that are performed by modules, program modules, or other components, in accordance with various embodiments, may be performed in a sequential, parallel, iterative, or heuristic manner, or at least in part Some operations may be executed in a different order, omitted, or other operations may be added.

Claims (20)

In an electronic device,
MIC;
display; And
The processor comprising:
Displaying on the display at least one text corresponding to the at least one object obtained based on the resources constituting the content, the content including at least one object,
Determining a first text corresponding to a voice received using the microphone among the at least one text,
Executing instructions corresponding to the received voice for a first object corresponding to the first text of the at least one object based on at least one command registered to control the at least one object Electronic device. 2. The apparatus of claim 1,
Analyzes the resource,
Identify the at least one object within the content based on an analysis result of the resource and determine at least one text corresponding to the at least one object,
And registering the at least one instruction for controlling the at least one object based on an analysis result of the resource. 2. The apparatus of claim 1,
Determining at least one non-text object that is not text among the at least one object based on the resource,
And displaying the at least one text on the display by mapping the at least one text to the at least one non-text object. 2. The apparatus of claim 1,
Determining an object to which a graphical user interface is to be applied among the at least one object based on the attribute of the at least one object obtained based on the resource,
And displaying the determined object on the display in correspondence with a graphical user interface corresponding to the attribute of the determined object. 2. The apparatus of claim 1,
Adjusting at least one of the size, color, and font of the at least one text corresponding to the at least one object based on at least one of size, position, type, and logging information of the at least one object, Lt; RTI ID = 0.0 > 1, < / RTI > 2. The apparatus of claim 1,
Determine an order to display the at least one text corresponding to the at least one object on the display based on the determined importance,
And sequentially displaying the at least one text on the display in accordance with the determined order. 7. The apparatus of claim 6,
And determining the importance based on at least one of size, location, type, and logging information of the at least one object. The method according to claim 1,
Further comprising a camera,
The processor comprising:
Determining an area of the user's gaze on the display based on the image obtained through the camera,
And displaying on the display text corresponding to an object included in the determined one of the at least one object. 2. The apparatus of claim 1,
When displaying the at least one text on the display, determining whether there is any text to be displayed overlapping among the at least one text,
And displaying only one of the texts to be displayed overlapping with each other based on a predetermined importance, if it is determined that there is text to be displayed overlapping with each other. The method of claim 1, wherein the at least one object comprises:
A text, an image, a moving picture, an icon, a link, and an input area. The method of claim 1,
An electronic device comprising a web page. A method of controlling an electronic device using speech recognition,
Displaying on the display content comprising at least one object and at least one text corresponding to the at least one object obtained based on the resources comprising the content;
Determining a first text corresponding to a voice received using the microphone among the at least one text; And
Executing instructions corresponding to the received voice for a first object corresponding to the first text of the at least one object based on at least one command registered to control the at least one object A method for controlling an electronic device using speech recognition. 13. The method of claim 12,
Analyzing the resource,
Identifying the at least one object within the content based on an analysis result of the resource and determining at least one text corresponding to the at least one object, and
And registering the at least one command for controlling the at least one object based on an analysis result of the resource. 13. The method of claim 12, wherein displaying at least one text corresponding to the at least one object on the display comprises:
Determining at least one non-text object that is not text among the at least one object based on the resource; and
And displaying the at least one text on the display in correspondence with the at least one text to the at least one non-text object. 13. The method of claim 12,
Determining an object to which a graphical user interface is to be applied among the at least one object based on an attribute of the at least one object obtained based on the resource;
And displaying the determined object on the display in correspondence with a graphical user interface corresponding to a determined graphical user interface corresponding to the attribute of the determined object on the display, . 13. The method of claim 12, wherein displaying at least one text corresponding to the at least one object on the display comprises:
Adjusting at least one of the size, color, and font of the at least one text corresponding to the at least one object based on at least one of size, position, type, and logging information of the at least one object, The method comprising the steps of; 13. The method of claim 12, wherein displaying at least one text corresponding to the at least one object on the display comprises:
Determining the importance based on at least one of size, position, type, and logging information of the at least one object;
Determining an order to display the at least one text corresponding to the at least one object on the display based on the determined importance; and
And sequentially displaying the at least one text on the display in accordance with the determined order. 13. The method of claim 12,
Further comprising determining, based on the image obtained through the camera, an area facing the user's line of sight on the display,
Displaying at least one text corresponding to the at least one object on the display,
And displaying on the display text corresponding to an object included in the determined one of the at least one object. 13. The method of claim 12, wherein displaying at least one text corresponding to the at least one object on the display comprises:
Determining if there are any of the at least one text to be displayed overlapping with each other, and
And displaying only one of the texts to be overlapped and displayed based on the predetermined importance, when it is determined that there are texts to be displayed overlapping with each other. A machine-readable storage medium storing a program for executing a control method of an electronic device using speech recognition, the method comprising:
Displaying on the display content comprising at least one object and at least one text corresponding to the at least one object obtained based on the resources comprising the content;
Determining a first text corresponding to a voice received using the microphone among the at least one text; And
Executing instructions corresponding to the received voice for a first object corresponding to the first text of the at least one object based on at least one command registered to control the at least one object Lt; / RTI >

Images