KR20190024333A - Electric terminal and method for controlling the same - Google Patents

Abstract

Disclosed are an electronic device which can process information based on an artificial intelligence technology by receiving a voice of a user, and a control method thereof. According to the present invention, the electronic device comprises: a microphone formed to receive a voice; a wireless communication unit communicating with an external device for displaying content; a control unit recognizing a viewer of the content based on the voice received through the microphone and determining whether a viewing preference of the recognized viewer has been learned; and a learning data unit constituting information related to the viewing preference of the viewer based on a determination result of the control unit. In addition, the control unit outputs a signal related to a change of the content based on the constituted information, and the viewing preference of the viewer is continuously learned based on a feedback for the output signal.

Description

[0001] ELECTRIC TERMINAL AND METHOD FOR CONTROLLING THE SAME [0002]

The present invention relates to an electronic device capable of receiving speech of a user and processing based on artificial intelligence technology.

Recently, with the development of hardware, artificial intelligence technology which realizes human thinking process, that is, cognition, reasoning, learning, and the like, with computing technology has been dramatically developed.

Artificial intelligence technology is not only itself a research subject, it can be directly or indirectly combined with other fields of computer science to provide various functions. Especially, it is very active to introduce artificial intelligence elements in various fields of information technology and to utilize them in solving problems in the field.

Artificial intelligence is divided into strong artificial intelligence and weak artificial intelligence. Strong artificial intelligence is a technology that can make human-like thinking and decision-making, and it is a level of self-study and self-judgment. Weak artificial intelligence is a level of technology that provides optimal solutions by performing cognitive processes such as perception, reasoning, etc. through a computational model.

As a result of this development, artificial intelligence technology is applied to the most familiar electronic device, and attempts are being made to provide various functions.

Particularly, studies related to the connection of electronic devices with other devices to further expand the application of artificial intelligence technology are actively performed. By way of example, other devices may be connected based on the user's voice received at the electronic device, and input of control commands to the connected device may be performed via the electronic device.

However, up to now, the control of the connected device is performed only when there is an intentional input of the user, so that there is a limit in accessing the content or menu function desired by the user accurately and quickly among various contents or menu functions. In recent years, a smart TV providing a variety of contents has been provided. It is aimed at grasping and controlling a desired one without a user's operation, but the operation is still in a stage where operation is insufficient.

In order to solve the above-described problems, the present invention provides an electronic device capable of recognizing a viewer of a connected device based on a user's voice and performing processing based on artificial intelligence technology so that a viewer can access the desired content accurately and quickly, And a control method thereof.

Another object of the present invention is to provide an electronic device and a control method thereof, which can change the content or environment setting that is being watched according to a changed environment when a viewer's member changes while watching the content.

According to an aspect of the present invention, there is provided an electronic device including: a microphone configured to receive a voice; A wireless communication unit for communicating with an external device displaying content; A control unit for recognizing a viewer of the content based on the received voice and determining whether the viewing taste of the recognized viewer is learned; And a running data unit for configuring information related to a viewing preference of the viewer based on the determination result, wherein the controller outputs a signal related to the change of the content on the basis of the configured information, Is learned based on feedback on the output signal.

Further, in one embodiment, the control unit may transmit a signal for providing the recommended content corresponding to the learned viewing taste to the external device if the viewing taste of the recognized viewer has been previously learned, And outputs guidance information related to the content.

Further, in one embodiment, if the viewing preference of the recognized viewer has not been learned before, the control unit determines the recommended content based on the gender and the age range of the viewer who is grasped based on the recognized viewer's voice .

According to an embodiment of the present invention, the controller recognizes the viewer of the content based on the received voice and an image obtained through a camera provided in the external device.

Also, in one embodiment, the control unit recognizes a viewer of the content based on the received voice when a start word for connection with the external device is input to the microphone.

In addition, in one embodiment, when the change in the configuration of the viewer is detected on the basis of the received voice and image while the user is viewing the content, the control unit And a signal is generated.

Further, in one embodiment, when a plurality of viewers are recognized on the basis of the received voice and image, the control unit transmits a signal requesting to provide different recommended contents according to whether the viewer has learned a plurality of audiences To the external device.

Also, in one embodiment, if the learned viewing tastes of the plurality of viewers are different, the controller analyzes the correlation of the learned viewing tastes to determine whether there is a common viewing tastes, and based on the determination result And generates a signal for providing different kinds of recommended contents.

According to an embodiment of the present invention, the controller may provide a signal for recommending contents corresponding to a common viewing preference to the external device according to an analysis result of the degree of correlation, And learns taste.

According to an embodiment of the present invention, the controller may be configured to grasp the leader viewers of the plurality of viewers based on the received audio and video if the correlation view shows no common viewing preference, And transmits a signal for simultaneously recommending the first content corresponding to the taste and the second content corresponding to the viewing priority to the external device.

Also, in one embodiment, the recommended ratio of the first and second contents is adjusted according to the ratio of the leader viewers among the plurality of viewers.

Also, in one embodiment, when the plurality of viewers are switched to a single viewer based on the received voice and image, the control unit may set the viewing content to a preference corresponding to the viewing preference of the single viewer Value to the external device.

Also, in one embodiment, when it is detected that a plurality of viewers are converted into a single viewer based on the received voice and image, a control signal for connecting the registered device of the single viewer to the external device is output And outputting the generated connection information in response to the output control signal.

Also, in one embodiment, the control unit analyzes the information of the viewer based on the received voice, transmits a control signal for recommending a viewer account corresponding to the analysis result to the external device, And generates a signal related to the change of the content based on the content.

Also, in one embodiment, the control unit outputs guide information of a recommended viewer account, registers a viewer account based on feedback on the guide information, and based on the viewer's voice corresponding to the registered viewer account And learns watching tastes.

Also, in one embodiment, the control unit selects a specific area of the content based on the voice of the viewer received while watching the content, extracts the sound of the object included in the selected specific area, Up control signal to the external device.

In one embodiment, the control unit provides the external device with a control signal for zooming down the specific area when a predetermined voice signal is input while the sound of the object is extracted.

A method of controlling an electronic device according to an embodiment of the present invention includes the steps of: connecting to an external device that displays content; recognizing a viewer of the content based on a voice received by the microphone; Generating information related to the viewing preference of the viewer based on the determination result, outputting a signal related to the change of the content on the basis of the configured information, And receiving feedback on the received feedback to learn the viewer's taste of the viewer.

According to the electronic device and the control method thereof according to the embodiment of the present invention, it is possible to naturally grasp whether a viewer is a viewer of a connected device based on a user's voice, to recommend contents matching the taste of the viewer, It does. Accordingly, it is possible to solve the inconvenience and time consuming operation for inputting a desired viewing content or an appropriate viewing environment for the viewer. In addition, it is possible to change the content or environment setting value in accordance with the changed environment by recognizing the configuration change of the content viewer without user's manipulation. In particular, even when a plurality of viewers have different viewing preferences, .

1 is a conceptual diagram showing an electronic device related to the present invention communicating with an external device that outputs content.
FIG. 2A is a block diagram for explaining a detailed configuration of an electronic device related to the present invention, and FIG. 2B is an exemplary conceptual view of the electronic device of FIG. 2A.
2C is an exemplary conceptual diagram for explaining a process of recognizing a viewer of an external device in an electronic device related to the present invention.
Figure 3 is a representative flow diagram illustrating operations implemented in an electronic device associated with the present invention.
FIGS. 4A, 4B, and 4C are exemplary conceptual diagrams for explaining control of content output to a connected external device, based on the viewer's viewing taste of the recognized viewer, in an electronic device related to the present invention. FIG.
5 is a flowchart illustrating an operation related to recommending different contents based on a changed viewer when a change of a viewer is detected in an electronic device related to the present invention.
Figs. 6A, 6B, 6C, 6D, and 6E are exemplary conceptual diagrams showing how a recommended content is configured differently based on a viewer's change and a common viewing preference of a plurality of viewers in an electronic device related to the present invention .
7A and 7B are illustrative conceptual diagrams showing a point of time when a viewer is recognized and a screen configuration in an external device in the electronic device related to the present invention.
Figs. 8, 9A, 9B, and 9C are illustrative conceptual diagrams illustrating operations associated with control of an external device when a plurality of viewers are changed to a single viewer, in an electronic device related to the present invention.
10 and 11 are flow charts and illustrative conceptual diagrams illustrating an operation associated with recommending a viewer account based on a user's voice in an electronic device associated with the present invention.
12 and 13 are flowcharts and illustrative conceptual diagrams illustrating an operation related to perceiving objects of interest in the contents based on the viewer's voice and performing related functions in the electronic device related to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals are used to designate identical or similar elements, and redundant description thereof will be omitted. The suffix " module " and " part " for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

The electronic devices described herein may include a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a slate PC, A tablet PC, an ultrabook, a wearable device such as a smartwatch, a smart glass, and a head mounted display (HMD). have.

However, it will be readily apparent to those skilled in the art that configurations in accordance with the embodiments described herein may be applied to electronic devices such as digital TVs, desktop computers, digital signage, and the like, will be.

1 shows an electronic device associated with the present invention in communication with an external device for outputting content.

As shown in FIG. 1, the electronic device 100 according to the present invention may be implemented as a speaker. In addition, the electronic device 100 according to the present invention is capable of recognizing speech and performing a speech recognition function to control the operation of the electronic device 100, based on the recognized speech. Here, the voice recognition function is a function for controlling an electronic device through voice, and includes a natural language processing function for analyzing a natural language, a question and answer function for presenting an answer to a question, a language capable of processing a digital voice signal in a computer A voice processing function for converting the voice into a voice, and an automatic translation and translation function for converting voice in multiple languages.

In addition, the electronic device 100 according to the present invention can be wirelessly connected to an external device 200, for example, a smart TV, which outputs contents based on the recognized voice. After the electronic device 100 and the external device 200 are connected as described above, the output from the external device 200 can be controlled based on the voice recognized by the electronic device 100. [ Here, the control of the output includes, for example, changing, editing or adding the content output from the external device 200 or adjusting the setting environment of the content, for example, the volume, the aspect ratio, and the brightness can do.

Also, in the present invention, it is possible to control the external device 200 and another device 300, for example, a smart phone, a set-top box, etc., to be connected based on the voice recognized by the electronic device 100. In this case, the operation of another device 300 connected to the external device 200 can be controlled based on a voice command input to the electronic device 100 connected to the external device 200. At this time, the operation result of the other device 300 may be output through the electronic device 100. [

Hereinafter, FIG. 2A is a block diagram for explaining an electronic device related to the present invention, and FIG. 2B is a conceptual diagram showing an example of an electronic device related to the present invention, that is, an example of a speaker-type electronic device.

The electronic device 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, ), And the like. The components shown in FIG. 2A are not essential for implementing an electronic device, so that the electronic device described herein may have more or fewer components than the components listed above.

More specifically, the wireless communication unit 110 among the above-described components can communicate with the electronic device 100 and the wireless communication system, between the electronic device 100 and another electronic device 100, Lt; RTI ID = 0.0 > wireless < / RTI > In addition, the wireless communication unit 110 may include one or more modules that connect the electronic device 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short distance communication module 114, and a location information module 115 .

The input unit 120 includes a camera 121 or an image input unit for inputting a video signal, a microphone 122 for inputting an audio signal, an audio input unit, a user input unit 123 for receiving information from a user A touch key, a mechanical key, and the like). The voice data or image data collected by the input unit 120 may be analyzed and processed by a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of the information in the electronic device, the surrounding information surrounding the electronic device, and the user information. For example, the sensing unit 140 may include a proximity sensor 141, an illumination sensor 142, a touch sensor, an acceleration sensor, a magnetic sensor, A G-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor, a finger scan sensor, an ultrasonic sensor, A microphone 226, a battery gauge, an environmental sensor (for example, a barometer, a hygrometer, a thermometer, a radiation detection sensor, A thermal sensor, a gas sensor, etc.), a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). On the other hand, the electronic device disclosed in this specification can combine and use information sensed by at least two of the sensors.

The output unit 150 includes at least one of a display unit 151, an acoustic output unit 152, a haptic tip module 153, and a light output unit 154 to generate an output related to visual, auditory, can do. The display unit 151 may have a mutual layer structure with the touch sensor or may be integrally formed to realize a touch screen. Such a touch screen may function as a user input 123 that provides an input interface between the electronic device 100 and a user and may provide an output interface between the electronic device 100 and a user.

The interface unit 160 serves as a path to various kinds of external devices connected to the electronic device 100. The interface unit 160 is connected to a device having a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, And may include at least one of a port, an audio I / O port, a video I / O port, and an earphone port. In the electronic device 100, corresponding to the connection of the external device to the interface unit 160, it is possible to perform appropriate control with respect to the connected external device.

The memory 170 also stores data that supports various functions of the electronic device 100. The memory 170 may store a number of application programs (application programs or applications) running on the electronic device 100, data for operation of the electronic device 100, and instructions. At least some of these applications may be downloaded from an external server via wireless communication. At least some of these applications may also reside on the electronic device 100 since the time of shipment for the basic functions of the electronic device 100 (e.g., phone call incoming, outgoing, message receiving, origination functions). On the other hand, an application program may be stored in the memory 170, installed on the electronic device 100, and operated by the control unit 180 to perform the operation (or function) of the electronic device.

In addition to the operations associated with the application program, the control unit 180 typically controls the overall operation of the electronic device 100. The control unit 180 may process or process signals, data, information, and the like input or output through the above-mentioned components, or may drive an application program stored in the memory 170 to provide or process appropriate information or functions to the user.

In addition, the controller 180 may control at least some of the components illustrated in FIG. 1A in order to drive an application program stored in the memory 170. FIG. Further, the controller 180 may operate at least two of the components included in the electronic device 100 in combination with each other for driving the application program.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power to the components included in the electronic device 100. The power supply unit 190 includes a battery, which may be an internal battery or a replaceable battery.

At least some of the components may operate in cooperation with each other to implement an operation, control, or control method of an electronic device according to various embodiments described below. Also, the method of operation, control, or control of the electronic device may be implemented on an electronic device by driving at least one application program stored in the memory 170. [

Hereinafter, the components listed above will be described in more detail with reference to FIG. 1A, before explaining various embodiments implemented through the electronic device 100 as described above.

First, referring to the wireless communication unit 110, the broadcast receiving module 111 of the wireless communication unit 110 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more broadcast receiving modules may be provided to the mobile terminal 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

The mobile communication module 112 may be a mobile communication module or a mobile communication module such as a mobile communication module or a mobile communication module that uses technology standards or a communication method (e.g., Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution And an external terminal, or a server on a mobile communication network established according to a long term evolution (AR), a long term evolution (AR), or the like.

The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.

The wireless Internet module 113 is a module for wireless Internet access, and may be embedded in the electronic device 100 or externally. The wireless Internet module 113 is configured to transmit and receive a wireless signal in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, wireless LAN (WLAN), wireless fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, DLNA (Digital Living Network Alliance), WiBro Interoperability for Microwave Access, High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE) and Long Term Evolution-Advanced (LTE-A) 113 transmit and receive data according to at least one wireless Internet technology, including Internet technologies not listed above.

The wireless Internet module 113 for performing a wireless Internet connection through the mobile communication network can be used for wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE or LTE- May be understood as a kind of the mobile communication module 112.

The short-range communication module 114 is for short-range communication, and includes Bluetooth ™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB) (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology. The short range communication module 114 may be used to communicate between the electronic device 100 and the wireless communication system via the wireless area networks or between the electronic device 100 and another electronic device 100 or between the electronic device 100 ) And another electronic device 100 (or an external server) are located. The short-range wireless communication network may be a short-range wireless personal area network.

Herein, the other electronic device 100 is a wearable device (e.g., a smartwatch, a smart glass, etc.) capable of interchanging data with the electronic device 100 according to the present invention (smart glass), HMD (head mounted display)). The short range communication module 114 can detect (or recognize) a wearable device capable of communicating with the electronic device 100 around the electronic device 100. [ Further, the control unit 180 may control at least a part of the data processed in the electronic device 100 to be transmitted to the short distance communication module (or the short distance communication module), if the detected wearable device is an authorized device to communicate with the electronic device 100 according to the present invention 114 to the wearable device. Therefore, the user of the wearable device can use the data processed in the electronic device 100 through the wearable device. For example, according to this, the user can make a phone call through the wearable device when a telephone call is received in the electronic device 100, or when the message is received in the electronic device 100, It is possible to check the message.

The position information module 115 is a module for obtaining the position (or current position) of the electronic device, and representative examples thereof include a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, an electronic device can utilize a GPS module to obtain the position of an electronic device using a signal sent from a GPS satellite. As another example, an electronic device can utilize a Wi-Fi module to obtain the location of an electronic device based on information of a wireless access point (wireless AP) that transmits or receives a wireless signal with a Wi-Fi module. Optionally, the location information module 115 may perform any of the other modules of the wireless communication unit 110 to obtain data regarding the location of the electronic device, in addition or alternatively. The location information module 115 is a module used to obtain the location (or current location) of the electronic device, and is not limited to a module that directly calculates or obtains the location of the electronic device.

Next, the input unit 120 is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For inputting image information, Or a plurality of cameras 121 may be provided. The camera 121 processes image frames such as still images or moving images obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170. [ The plurality of cameras 121 provided in the electronic device 100 may be arranged to form a matrix structure and the electronic device 100 may be provided with various angles or foci through the camera 121 having the matrix structure A plurality of pieces of image information can be input. In addition, the plurality of cameras 121 may be arranged in a stereo structure to acquire a left image and a right image for realizing a stereoscopic image.

The microphone 122 processes the external acoustic signal into electrical voice data. The processed voice data can be utilized variously according to functions (or application programs being executed) being performed in the electronic device 100. Meanwhile, the microphone 122 may be implemented with various noise reduction algorithms for eliminating noise generated in receiving an external sound signal.

The user input unit 123 is for receiving information from a user and when the information is inputted through the user input unit 123, the control unit 180 can control the operation of the electronic device 100 to correspond to the input information . The user input unit 123 may include a mechanical input means (or a mechanical key such as a button located on the front, rear or side of the electronic device 100, a dome switch, a jog wheel, Jog switches, etc.) and touch-type input means. For example, the touch-type input means may comprise a virtual key, a soft key or a visual key displayed on the touch screen through software processing, The virtual key or the visual key can be displayed on the touch screen with various forms. For example, the virtual key or the visual key can be displayed on the touch screen, ), An icon, a video, or a combination thereof.

The running data portion 130 may be configured to receive, classify, store and output information to be used for data mining, data analysis, intelligent decision making, and machine learning algorithms and techniques. The running data unit 130 stores information received, detected, detected, generated, predefined information or other information output through the terminal, received by the terminal, received, detected, detected , And one or more memory units configured to store generated, predefined or output data.

The running data portion 130 may be integrated into an electronic device or may include a memory. In one embodiment, the running data portion 130 may be implemented via the memory 170. The running data portion 130 may be implemented in memory associated with the electronic device 100 (e.g., external memory coupled to the electronic device 100 (either wired or electrically) The running data portion 130 may be implemented in a memory that is stored in a cloud computing environment or in a memory that is communicatively coupled to the terminal 100 via a communication method such as a network. Lt; RTI ID = 0.0 > accessible by < / RTI >

The running data part 130 is used to identify, index, classify, manipulate, store, retrieve and output data for use in supervised or non-supervised learning, data mining, predictive analysis or other machine learning techniques, To one or more databases. The information stored in the running data section 130 may be used by the control section 180 using a different type of data analysis, machine learning algorithm and machine learning techniques, or by a plurality of controls included in the electronic device . Examples of such algorithms and techniques are K nearest neighbor systems, fuzzy logic (for example, possibility theory), neural networks, Boltzmann (Boltzmann such as vector quantization, pulsed neural nets, support vector machines, maximum margin classifiers, hill-climbing, inductive logic systems, But are not limited to, baysian networks, petri nets (e.g., finite state machines, mealy machines, moore finite state machines), classifiers trees (e.g., perceptron trees, support vector trees, markov trees, decision tree forests, random forests), horses All models and systems (pandemon ium models and systems, clustering, artificially intelligent planning, artificially intelligent forecasting, data fusion, sensor fusion, image fusion, Reinforcement learning, augmented reality, pattern recognition, and automated planning.

The control unit 180 may determine or predict at least one executable operation of the electronic device based on information determined or generated using data analysis, machine learning algorithms, and machine learning techniques. To this end, the control unit 180 can request, search, receive, or utilize the data of the running data unit 130. The controller 180 may perform various functions for implementing a knowledge-based system, an inference system, a knowledge acquisition system, and the like, and may include a system (e.g., a fuzzy logic system) for uncertain reasoning, an adaptation system, Neural networks, and the like.

The control unit 180 may be a sub-module that enables voice and natural language processing such as an I / O processing module, an environmental condition module, a voice-text (STT) processing module, a natural language processing module, a workflow processing module, Lt; / RTI > Each of the sub-modules may have access to one or more systems or data and models in the electronic device, or a subset or superset thereof. Here, the object to which each of the submodules has access rights may include a scheduling, a vocabulary index, a user data, a task flow model, a service model, and an automatic speech recognition (ASR) system. In another embodiment, the controller 180 or the electronic device may be implemented with the submodule, system, or data and model.

In some embodiments, based on the data in the running data portion 130, the controller 180 may be configured to detect and detect what the user requires based on contextual conditions or intentions of the user represented by user input or natural language input Lt; / RTI > In addition, the control unit 180 can actively derive or acquire the information necessary to completely determine what the user requests in accordance with the context condition or the user's intention. For example, the control unit 180 can detect and sense what the user requests by analyzing historical data including past input and output, pattern matching, unambiguous words, input intentions, and the like. In addition, the control unit 180 can determine a work flow for executing a function requested by a user according to a context condition or a user's intention. In addition, the control unit 180 can execute a workflow for satisfying a user's request based on a context condition or a user's intention.

In some embodiments, the controller 180 may be implemented as a microprocessor, such as a memory, a memristors, a transconductance amplifier, a pulsed neural circuit, an artificial intelligent nanotechnology system (e.g., an autonomous nano machine), or an artificial intelligent quantum mechanical system A neural network), and the like. In some embodiments, the controller 180 may include a pattern recognition system, such as a machine vision system, an acoustic recognition system, a handwriting recognition system, a data fusion system, a sensor fusion system, and a soft sensor. The machine vision system may include content based image retrieval, optical character recognition, augmented reality, egomotion, tracking and optical flow, and the like.

The control unit 180 can sense or receive information in the electronic device, surrounding information surrounding the electronic device, or other information through the sensing unit 140. [ In addition, the controller 180 can receive broadcast signals and / or broadcast-related information, wireless signals, wireless data, and the like through the wireless communication unit 110. Also, the control unit 180 may receive image information (or a corresponding signal), audio information (or a corresponding signal), data, or information input from a user, from an input unit.

The controller 180 collects information, processes or classifies the information (e.g., a knowledge graph, a command policy, a personalization database, a dialog engine, etc.), and transmits the processed or classified information to the memory 170 or the running data (130).

Based on the data analysis, the machine learning algorithm, and the machine learning technique, when the operation of the electronic device is determined, the controller 180 controls the components of the electronic device to perform the determined operation. Can be controlled. The control unit 180 can execute the determined operation by controlling the electronic device based on the control command.

In one embodiment, when a specific operation is performed, the controller 180 analyzes history information indicating performance of a specific operation through a data analysis, a machine learning algorithm, and a machine learning technique. Based on the analyzed information, An update to the learned information can be performed. Accordingly, the control unit 180 can improve the accuracy of the future performance of the data analysis based on the updated information, the machine learning algorithm, and the machine learning technology together with the running data unit 130.

Meanwhile, the sensing unit 140 senses at least one of information in the electronic device, surrounding environment information surrounding the electronic device, and user information, and generates a corresponding sensing signal. The control unit 180 may control the driving or operation of the electronic device 100 or perform data processing, function or operation related to the application program installed in the electronic device 100 based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 140 will be described in more detail.

First, the proximity sensor 141 refers to a sensor that detects the presence of an object approaching a predetermined detection surface, or the presence of an object in the vicinity of the detection surface, without mechanical contact by using electromagnetic force or infrared rays. The proximity sensor 141 may be disposed in the inner area of the electronic device or the proximity sensor 141 near the touch screen.

Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. In the case where the touch screen is electrostatic, the proximity sensor 141 can be configured to detect the proximity of the object with a change of the electric field along the proximity of the object having conductivity. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of explanation, the act of recognizing that the object is located on the touch screen in proximity with no object touching the touch screen is referred to as " proximity touch & The act of actually touching an object on the screen is called a " contact touch. &Quot; The position at which the object is closely touched on the touch screen means a position where the object corresponds to the touch screen vertically when the object is touched. The proximity sensor 141 can detect a proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, have. Meanwhile, the control unit 180 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 141 as described above, and further provides visual information corresponding to the processed data It can be output on the touch screen. Furthermore, the control unit 180 can control the electronic device 100 such that different operations or data (or information) are processed depending on whether the touch to the same point on the touch screen is a proximity touch or a touch contact .

The touch sensor uses a touch (or touch input) applied to the touch screen (or the display unit 151) by using at least one of various touch methods such as a resistance film type, a capacitive type, an infrared type, an ultrasonic type, Detection.

For example, the touch sensor may be configured to convert a change in a pressure applied to a specific portion of the touch screen or a capacitance generated in a specific portion to an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure at the time of touch, a capacitance at the time of touch, and the like where a touch object touching the touch screen is touched on the touch sensor. Here, the touch object may be a finger, a touch pen, a stylus pen, a pointer, or the like as an object to which a touch is applied to the touch sensor.

Thus, when there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and transmits the corresponding data to the controller 180. Thus, the control unit 180 can know which area of the display unit 151 is touched or the like. Here, the touch controller may be a separate component from the control unit 180, and may be the control unit 180 itself.

On the other hand, the control unit 180 may perform different controls or perform the same control according to the type of the touch object touching the touch screen (or a touch key provided on the touch screen). Whether to perform different controls or perform the same control depending on the type of the touch object can be determined according to the operating state of the current electronic device 100 or an application program being executed.

On the other hand, the touch sensors and the proximity sensors discussed above can be used independently or in combination to provide a short touch (touch), a long touch, a multi touch, a drag touch ), Flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, and the like. Touch can be sensed.

The ultrasonic sensor can recognize the position information of the object to be sensed by using ultrasonic waves. Meanwhile, the controller 180 can calculate the position of the wave generating source through the information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source can be calculated using the fact that the light is much faster than the ultrasonic wave, that is, the time when the light reaches the optical sensor is much faster than the time the ultrasonic wave reaches the ultrasonic sensor. More specifically, the position of the wave generating source can be calculated using the time difference with the time when the ultrasonic wave reaches the reference signal.

The camera 121 includes at least one of a camera sensor (for example, a CCD, a CMOS, etc.), a photo sensor (or an image sensor), and a laser sensor.

The camera 121 and the laser sensor may be combined with each other to sense a touch of the sensing object with respect to the three-dimensional stereoscopic image. The photosensor can be laminated to the display element, which is adapted to scan the movement of the object to be detected proximate to the touch screen. More specifically, the photosensor mounts photo diodes and TRs (Transistors) in a row / column and scans the contents loaded on the photosensor using an electrical signal that varies according to the amount of light applied to the photo diode. That is, the photo sensor performs coordinate calculation of the object to be sensed according to the amount of change of light, and position information of the object to be sensed can be obtained through the calculation.

The display unit 151 displays (outputs) information to be processed in the electronic device 100. For example, the display unit 151 may display execution screen information of an application program driven by the electronic device 100, or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information .

In addition, the display unit 151 may be configured as a stereoscopic display unit for displaying a stereoscopic image.

In the stereoscopic display unit, a three-dimensional display system such as a stereoscopic system (glasses system), an autostereoscopic system (no-glasses system), and a projection system (holographic system) can be applied.

The sound output unit 152 may output audio data received from the wireless communication unit 110 or stored in the memory 170 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output unit 152 also outputs sound signals related to functions (e.g., call signal reception tones, message reception tones, etc.) performed in the electronic device 100. [ The audio output unit 152 may include a receiver, a speaker, a buzzer, and the like.

The haptic module 153 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 153 may be vibration. The intensity and pattern of the vibration generated in the haptic module 153 can be controlled by the user's selection or the setting of the control unit. For example, the haptic module 153 may synthesize and output different vibrations or sequentially output the vibrations.

In addition to vibration, the haptic module 153 may be configured to perform various functions such as a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a touch on the skin surface, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.

The haptic module 153 can transmit the tactile effect through the direct contact, and the tactile effect can be felt by the user through the muscles of the finger or arm. At least two haptic modules 153 may be provided according to the configuration of the electronic device 100.

The light output unit 154 outputs a signal for notifying the occurrence of an event using the light of the light source of the electronic device 100. Examples of events that occur in the electronic device 100 may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, receiving information via an application, and the like.

The signal output by the light output unit 154 is implemented as the electronic device emits light of a single color or a plurality of colors to the front or rear surface. The signal output may be terminated by the electronic device sensing an event confirmation of the user.

The interface unit 160 serves as a path for communication with all the external devices connected to the electronic device 100. The interface unit 160 receives data from an external device or supplies power to each component in the electronic device 100 or allows data in the electronic device 100 to be transmitted to an external device. For example, a port for connecting a device equipped with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, an audio I / O port, a video I / O port, an earphone port, and the like may be included in the interface unit 160.

The identification module is a chip for storing various kinds of information for authenticating the usage right of the electronic device 100. The identification module includes a user identification module (UIM), a subscriber identity module (SIM) A universal subscriber identity module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the interface unit 160. [

The interface unit 160 may be a path through which power from the cradle is supplied to the electronic device 100 when the electronic device 100 is connected to an external cradle, And various command signals may be transmitted to the electronic device 100. [ Various command signals or power signals input from the cradle may be operated with a signal for recognizing that the electronic device 100 is correctly mounted on the cradle.

The memory 170 may store a program for the operation of the controller 180 and temporarily store input / output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 170 may store data related to vibration and sound of various patterns outputted when a touch is input on the touch screen.

The memory 170 may be a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), a multimedia card micro type ), Card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read memory, a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and / or an optical disk. The electronic device 100 may operate in association with a web storage that performs the storage function of the memory 170 on the Internet.

Meanwhile, as described above, the control unit 180 controls the operation related to the application program and the general operation of the electronic device 100. [ For example, when the state of the electronic device meets a set condition, the controller 180 can execute or release a lock state for restricting input of a user's control command to applications.

In addition, the control unit 180 performs control and processing related to voice communication, data communication, video call, or the like, or performs pattern recognition processing to recognize handwriting input or drawing input performed on the touch screen as characters and images, respectively . Further, the control unit 180 may control any one or a plurality of the above-described components in order to implement various embodiments described below on the electronic device 100 according to the present invention.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for operation of the respective components. The power supply unit 190 includes a battery, the battery may be an internal battery configured to be chargeable, and may be detachably coupled to the terminal body for charging or the like.

In addition, the power supply unit 190 may include a connection port, and the connection port may be configured as an example of an interface 160 through which an external charger for supplying power for charging the battery is electrically connected.

As another example, the power supply unit 190 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 190 may use at least one of an inductive coupling method based on a magnetic induction phenomenon from an external wireless power transmission apparatus and a magnetic resonance coupling method based on an electromagnetic resonance phenomenon Power can be delivered.

In the following, various embodiments may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

2B, an electronic device of a speaker type is shown as an example of the electronic device 100. FIG. As shown, the speaker-type electronic device 100 may include a microphone 122, a speaker 152, and a display unit 151. The frame 101 of the electronic device 100 may have a built-in lighting function so as to activate the write function in the frame in accordance with a predetermined input. In addition, a camera 121 is provided outside the electronic device 100 to acquire an image of the surroundings.

The electronic device 100 may operate in either a standby mode or a voice recognition mode. In the standby mode, before execution of the voice recognition function, it is detected whether voice related to execution of voice recognition function exists around the electronic device. To this end, the control unit 180 of the electronic device 100 may monitor (or monitor) whether sounds over a specific loudness are continuously detected through the microphone 122. Further, since it is not necessary to perform speech analysis in the standby mode, it can operate with only a low electric power, for example, a current of about 4.4 mA. This standby mode may also be referred to as a " listening phase ".

On the other hand, in the standby mode, when a sound of a specific volume or more is detected through the microphone 122, the standby mode of the electronic device 100 can be switched to the voice recognition mode. Alternatively, when a predetermined start word of a predetermined volume or more is detected through the microphone 122, the standby mode of the electronic device 100 may be switched to the voice recognition mode. Here, the initiator may include voice commands, such as " Hello ", " Wake up ", " Google ", and the like, for waking up the electronic device 100 in the standby mode.

The voice recognition mode refers to a state in which the control unit 180 of the electronic device 100 analyzes the voice input through the microphone 122. [ Since speech analysis is performed in the speech recognition mode, more current is consumed than in the standby state 210. That is, the electronic device according to the present invention may exist in the standby state 210 in which the voice analysis is not performed before the start word is received, thereby reducing the consumption current. On the other hand, the control unit 180 of the electronic device 100 may determine whether a start word that starts speech recognition in the speech recognition mode is received before the speech analysis. In other words, the control unit 180 can start speech analysis for speech uttered after the initiator.

In the speech recognition mode, the electronic device 100 can voice-analyze the voice input via the microphone 122 to control the operation of the electronic device 100. Such speech analysis may be performed using a separate third party application installed in the electronic device.

In addition, the controller 180 of the electronic device 100 may transmit to the predetermined artificial intelligence server or to the running data part 130 of the electronic device 100 in order to process the voice analysis of the voice information through the artificial intelligence algorithm The stored data can be used. The predetermined artificial intelligence server is a server that provides a function of learning a vast amount of information using an artificial intelligence algorithm and deriving optimal result information based on the learned information. Alternatively, the control unit 180 of the electronic device 100 may generate the result information in response to the input voice information, based on the data stored in the running data unit 130.

In addition, if the voice of the user or the appointed start word is not received in the voice recognition mode for a preset time, the voice recognition mode may be switched to the standby mode again.

Subsequently, the speaker 152 can output the response result of the inputted voice information. Such a response result may include not only the answer to the inputted voice information but also inquiry or guidance information for confirming ambiguous voice information or a search range. Also, the speaker 152 may output a preset alert sound when the above-described voice recognition mode is switched to the standby mode or when the standby mode is switched to the voice recognition mode.

The display unit 151 may output a graphic object or an image corresponding to a result of the response output through the speaker 152. Alternatively, when the speaker 152 is set to be silent according to the setting of the electronic device 100, the response result of the inputted voice information may be outputted only through the display unit 151. [ Meanwhile, if the display unit 151 is composed of the above-described touch screen, the user can perform the interaction with the electronic device 100 by controlling graphic objects or images based on various types of touch input.

The camera 121 may be removably mounted to one side of the electronic device 100. The image obtained through the camera 121 can be used to identify a user corresponding to the voice information input through the speaker 152. [ Alternatively, it may transmit the acquired image to another device according to a user request.

The following description is based on the assumption that the electronic device 100 is provided with the running data unit 130 itself and is configured to output the analysis result of the inputted voice information. However, the present invention is not limited to this, and may be implemented in a form in which the electronic device 100 communicates with the artificial intelligence server to receive a response corresponding to analysis of voice information, and the electronic device 100 operates based on the received response It is possible.

In the following drawings and descriptions, the electronic device 100 is implemented in the form of a speaker for outputting sound. However, the present invention is not limited to this and can be implemented by various types of electronic devices. To be clear to.

2C is an exemplary conceptual diagram illustrating a process of recognizing a viewer of an external device 200 that outputs a content through the microphone 121 of the electronic device 100 related to the present invention.

2C, it is assumed that User A is a viewer of broadcast content output from the external device 200, and User B is not a viewer. If a voice command that intentionally fires to control the operation of the voice or electronic device 100 or the external device 200 that naturally fires while User A watches the broadcast content is received through the microphone of the electronic device 100, And calculates the distance between the user A and the electronic device 100 that communicates with the external device 200. Based on the calculated distance, the user A determines whether the external device 200 is a viewer Can be determined.

Regarding the calculation of the distance based on the voice, it is possible to calculate the approximate separation distance by using the point where the treble is easily absorbed and the bass is not easily absorbed, so that the bass can be heard better at higher distances than at higher distances. In another example, the position of the user corresponding to the recognized voice may be recognized by recognizing the position of the remote control device (e.g., the remote controller) that controls the external device.

At this time, the distance from the electronic device 100 for determining that the user A is the viewer is calculated including the separation distance between the external device 200 and the electronic device 100. In the case of User B, the distance determined based on the amount of voice received in the electronic device 100 is outside the range determined by the viewer of the external device 200.

If the calculated separation distance is ambiguous, it is possible to confirm whether or not the viewer is a viewer based on the image obtained through the camera 221 provided in the external device 200. User A is determined to be a viewer because it is within the angle of view of the camera 221, and User B is located outside the angle of view of the camera 221, so that it can be determined that the user B is not a viewer. In one example, even if the user A is within the angle of view of the camera 221, the face direction, the posture, and the state (sleep state) of the user A are judged based on the obtained image to view the contents output from the external device 200 It is possible to specifically grasp whether or not the user is doing so.

On the other hand, when User B is not initially a viewer but when voice information is recognized at a position close to the external device 200 or the electronic device 100 or close to the electronic device 100, it is recognized as a viewer together with User A . In this case, User A and User B are recognized as a plurality of viewers, and content control related to them will be described in more detail below. Alternatively, the user A may move so as to deviate from the view angle range of the camera 221. In this case, User A may be excluded from the viewer.

Figure 3 is a representative flow diagram illustrating operations implemented in an electronic device associated with the present invention.

Referring to FIG. 3, an electronic device 100 according to the present invention is connected for communication with an external device 200 displaying contents (S10). For example, the external device 200 may be an analog TV, a digital TV, a 3D TV, a smart TV, an LED TV, an OLED TV, a plasma TV, a monitor, a smart phone, Do not. The external device 200 may be connected to a set-top box or an IP set-top box for providing multimedia communication services through a network.

Next, the electronic device 100 recognizes the viewer of the external device 200 based on the voice received through the microphone 122 (S20). At this time, the received voice includes ambient noise. The controller 180 of the electronic device 100 may filter the ambient noise from the voice received through the microphone 122 to extract a clear voice.

The received voice may be the voice of a plurality of users. In this case, the control unit 180 classifies the voice information of the plurality of users, analyzes the divided voice information to identify the location of the user, and based on the analysis result and the detected location of the user, It is possible to recognize whether the viewer is the viewer of the device 200 or which of the viewer is the viewer.

Also, the control unit 180 can determine whether the received voice is the voice of the registered viewer. To this end, the control unit 180 matches the voice corresponding to the viewer account stored in advance in the memory 160 with the voice of the currently received user, extracts information corresponding to the matching viewer account, . In other words, when it is determined whether the user corresponding to the received voice is to view the contents of the external device 200 ('first judgment'), it is determined whether the user who wants to view the contents has a registered viewer account 'Second judgment').

Next, the control unit 180 of the electronic device 100 determines whether the viewing preference of the viewer recognized based on the voice has been previously learned (S30). To this end, the memory 160 of the electronic device 100 may store voice information of the user that was previously received, and data related to the user's viewing preference matched to the stored voice information may be stored in the running data portion 130 . The data stored in the running data unit 130 can be continuously learned based on the voice of the same person received by the electronic device 100 or the external device 200. [

Here, the viewing preference of the viewer refers to a preference program of the viewer, a preference value (e.g., screen brightness, volume) of the viewer, a favorite genre, an average view duration, a channel conversion frequency, Whether or not the contents are related to the viewing of the contents, habits, characteristics, and the like.

When the viewer is recognized as a viewer close to the external device 200, the controller 180 determines whether the viewing preference of the viewer has been previously learned. Then, based on the determination result, information related to the viewing preference of the viewer is configured (S40).

If the viewing taste is the learned one, the viewing taste of the viewer is configured on the basis of the previous learning history. To this end, the learning data unit 130 of the electronic device 100 may continuously update and store the learning result of the viewing taste of the same viewer by using a learning algorithm based on machine learning. Here, such machine learning may be based on the voice received by the electronic device 100 while the same viewer watches the content through the external device 200. [ For example, the viewer's preference program may be generated as data related to viewing tastes by assigning scores by date and time. Such a viewing preference is formed separately for each user, and is used as data only when the user is recognized as a viewer.

On the other hand, if the viewer's taste is not previously learned (that is, the voice of the new user or the voice of the user whose account is not registered), the viewer's taste is estimated by analyzing the characteristics of the viewer based on the received voice . For example, the characteristics of the viewer's age, gender, emotional state and the like can be generated as an analysis result based on the received voice's strength, intensity, pitch (pitch), tone, kind of language, word level and the like. Accordingly, it is possible to estimate an average viewing preference according to the combination of the analyzed age group, sex, and emotion state. When it is difficult to grasp the characteristics of the viewer by only the received voice, information (e.g., image, location) collected through the external device 200 and the remote control device controlling the external device 200 can be further used.

On the other hand, if there are a plurality of viewers, the control unit 180 considers all the viewing preferences of a plurality of viewers. For example, when both User A and User B are recognized as viewers, it is determined whether the viewing tastes of User A and User B have been learned respectively, whether there is a viewing taste previously learned in the common viewing situation, It is possible to generate a common viewing preference which is estimated according to the relationship between the viewing preference and the viewing preference. For this purpose, data related to the viewing preference of the viewer stored in the running data section 130 can be searched, extracted, used, or combined.

The user can grasp the attitude or the state of the viewer through the camera 121 provided in the external device 200 if no audio is received for a predetermined time in the electronic device 100 even after the viewer is recognized or the viewer's viewing taste is grasped . For example, the viewer can determine whether the viewer is sleeping or concentrating on other tasks and exclude them from the viewer. Accordingly, when it is recognized that there is no viewer, the power consumption of the external device 200 may be reduced by at least adjusting the brightness of the screen or decreasing the volume.

Next, the control unit 180 of the electronic device 130 generates a signal related to the change of the content output to the external device 200 based on the information related to the viewing preference (S50). The information thus generated is reflected in the content transmitted to and outputted from the external device 200. [ Here, the signals related to the change of the content include not only control operations related to other contents such as recommendation of a preference program, reservation recording of a preference program, change of channel, setting of the content being watched, , And a control signal for performing a control operation such as an aspect ratio adjustment.

Then, the electronic device 100 learns the viewing preference of the viewer stored in the running data part 130 based on the feedback on the signal related to the change of the content. For example, if the signal related to the change of contents is provided on the menu screen for recommending the viewer's preference program, it is stored in the running data part 130 based on the viewer's response (e.g., rejection, selection, etc.) The viewing taste of the viewer can be learned. At this time, the viewer's response may be made through the voice (e. G., 'No', 'see-through', etc.) received at the electronic device 100.

As described above, according to the present invention, even if a user intentionally does not search for, select, or change contents to be viewed, it is possible to grasp whether the user is a viewer with only the voice uttered by the user and help the user to watch the contents according to the viewing preference corresponding thereto.

Figs. 4A, 4B and 4C show exemplary conceptual diagrams related to controlling content output to a connected external device, based on the viewer's viewing preference of the recognized viewer, in an electronic device associated with the present invention.

First, referring to FIG. 4A, when the user speaks a predetermined start word (e.g., "Turn on TV") 10, it is received by the microphone 122 of the electronic device 100, (100) and the smart TV (200).

The control unit 180 searches for a matching viewer account based on the received voice ("Turn on TV"), generates a control signal related to the recommended content to be viewed by the user, and transmits the generated control signal to the smart TV 200 .

The smart TV 200 receiving the control signal outputs information (" Hi, Jihoon ") corresponding to the viewer account and the recommended content corresponding to the viewing preference of the viewer to the screen 401. The recommended content is configured based on the learned viewing preference if the viewing preference of the viewer is previously learned. To this end, data stored in the running data part 130 of the electronic device 100 may be utilized as described above, and in another example, data stored in a server to which the artificial intelligence technology communicating with the electronic device 100 is applied may be used Or utilize the viewing preference of the viewer account stored in the smart TV 200 itself.

On the other hand, if the viewing preference of the viewer has not been learned before, the recommended content can be configured in order of popularity. In another example, if the viewing preference of the viewer has not been learned previously, the recommendation content can be determined based on at least one of the viewer's sex, age group, and emotion state, which is grasped based on the recognized viewer's voice. For example, if a person is identified as a man in his / her 30s based on voice, the recommended contents can be configured in the order of programs that are popular among men in their 30s. In addition, when it is determined that the voice of the viewer includes the emotional state such as stress or depression, the recommended content can include the comic genre.

When a viewer selects a specific broadcast program from the recommended content, such feedback is also transmitted to the electronic device 100 so that machine learning of the viewer's taste is naturally achieved. In addition, while the viewer watches the specific broadcast program 402, the environment settings matching the viewing preferences of the viewer, for example, the preferred screen brightness and volume size are automatically adjusted (410). The electronic device 100 may also allow the electronic device 100 to display a predetermined broadcast program 402 at a predetermined time interval (e.g., 5 minutes or settable) if the audio of the viewer is not received during the reference time, Or activates a camera provided in the smart TV 200 to monitor whether the viewer continues to watch or whether there is a change in the configuration of the viewer.

Further, in one example, the control unit 180 may output guidance information related to the recommended content while transmitting a signal for providing the recommended content corresponding to the learned viewing preference to the external device.

For example, referring to FIG. 4C, when it is determined that a favorite program corresponding to a viewing preference of a viewer starts soon while a viewer watches a specific program 403, Can be output at the bottom of the screen of FIG. At the same time, a guidance voice 20 for inducing a channel change for viewing the preference program through the speaker of the electronic device 100 may be output. Once the viewer performs a simple response ('huh!'), The channel to the preferred program is instantly switched. Even if the viewer wants to continue watching the program at present, it is sufficient to simply ignite the refusal doctor ('hate'). The results of the viewer 's responses are used to continuously learn watching tastes.

Hereinafter, referring to FIG. 5 and FIGS. 6A to 6E, a flowchart for explaining an operation related to recommending different contents based on a changed viewer when a change in the configuration of the viewer is detected in the electronic device 100. FIG. Here, the processing in the case where the number of viewers is changed due to a change in configuration of the viewer will be described in more detail.

First, referring to FIG. 5, it is assumed that, based on the voice received by the electronic device 100 (and / or the image obtained through the camera of the external device 200) even while viewing the content through the external device 200 The change in the configuration of the viewer can be recognized (S501).

FIG. 6A is an example of a case where a new viewer is introduced while viewing the content, and FIG. 6B is an example of a case where an existing viewer is leaving. For example, if User B is speaking 'what's', it can recognize that it is an inflow of a new viewer when it is close to the electronic device 100 or the external device 200. 6B, when a plurality of viewers are watching a content, if the user C speaks 'home', it recognizes the absence of the user D based on the image obtained through the camera 221 of the external device 200 Thus, it is possible to recognize the exiting of User D. In one example, if a voice associated with a change in the configuration of the viewer is detected, the camera 221 of the external device 200 may be activated to re-grasp the number of viewers.

The electronic device 100 determines the number of viewers (S502). If a plurality of viewers are identified, it is determined whether the viewer's taste is learned or not (S503). As a result of the determination in step S503, if there are a plurality of viewers whose viewing preferences are learned, the correlation between the plurality of viewing preferences is analyzed (S506).

The correlation analysis between the viewing preferences means the matching or similarity of the preference programs corresponding to a plurality of learned viewing preferences. In this regard, referring to FIG. 6C, if the learned viewing tastes of User A, User B, and User C are the viewing taste 1, the viewing taste 2, and the viewing taste 3, Lt; / RTI > In this correlation, when the data related to the viewing preference is a superordinate concept such as a 'preference genre', the date and time when the data is collected can be considered as an additional limiting condition. If the data related to the viewing taste is a sub-concept or a specific concept such as 'preference program', the additional limitation condition may be excluded or only the viewing frequency may be considered. In addition, although there are coincident favorite programs or genres, if the score is low in the individual preferences, the degree of correlation is considered to be very low and can be excluded from the following common viewing preferences.

As a result of the correlation analysis, it is determined whether there is a common viewing preference (S508), and if there is a common viewing preference, the corresponding content is recommended (S509). The control device 100 learns a common viewing preference based on the result of the response from the external device 200. [ The common viewing preference is managed separately from the individual viewing preference of the current viewer.

On the other hand, if there is no common viewing preference, it is determined whether the viewing preference is related to a plurality of viewers (S510). That is, it is determined whether or not the data is generated when there are a plurality of viewers.

If the data is not a common viewing preference but is generated when there are a plurality of viewers, it is determined whether the viewer type corresponding to the data is the leader (S511). Whether or not the viewer's type is a leader can be determined by comparing the similarity or the frequency of matching between the independent viewing preference of the viewer and the data generated when the viewer interacts with the viewer. For example, if the viewer views the program A at the same time zone alone while viewing the program A with a plurality of people, the viewer can be classified into a follow-up type rather than a reader. On the other hand, if the viewers' favorite programs do not differ greatly from when watching alone and when watching many people, they can be classified as a leader type. Alternatively, the viewer who grips the remote control device (not shown) for controlling the operation of the external device 200 may be classified into the reader type. In addition, there can be a plurality of viewers of this leader type.

If there is a plurality of viewers, if the viewer of the generated data is the leader type, the leader type content and the popular content are recommended together (S512). Specifically, the control unit 180 may transmit a signal to the external device 200 simultaneously recommending the first content corresponding to the viewing preference of the reader type viewer and the second content corresponding to the viewing priority order. The plurality of viewers can select the specific content desired from the recommendation list including the content of the leader type and the popular content.

In addition, in one example, the recommended ratio of the first and second contents may be adjusted differently according to the ratio of the leader type viewer among the plurality of viewers. 6D and 6E, if the viewer type ratio of the leader type to the total number of viewers is smaller than the majority, the recommendation ratio of popular contents ('Now Hot') in the content recommendation screen 610A as shown in FIG. 6D (612) is further increased. If the viewer type ratio of the leader type exceeds the total number of viewer types in accordance with the change of the configuration of the viewer, as shown in FIG. 6E, the content recommendation screen 610B displays content of 'You may like' The recommended ratio 613 is further increased. In this case, the recommendation rate of popular content ("Now Hot") is automatically reduced (614).

If the viewer of the data generated when there are a plurality of viewers is a viewer of the follow-type, the popular content expected to be enjoyed by a plurality of persons is determined as the recommended content (S507).

On the other hand, if it is determined in step S503 that the viewers whose viewing preferences are learned are singular, the content having a high popularity is recommended (S507). That is, if there are a large number of viewers, one of them does not recommend the preferred program according to the viewing preference. This takes into consideration that a plurality of viewers may have different viewing preferences.

On the other hand, if it is determined in step S502 that there is only one viewer, it is determined whether the viewing preference of the changed or remaining viewer has been learned (S504), and the content according to the viewing preference is provided to the external device 200 ). That is, at this time, the electronic device 100 can transmit the channel change signal to the external device 200, which switches to the channel corresponding to the preference program.

In addition, in one example, the controller 180 may estimate the relationship between a plurality of viewers and provide new recommendation contents different from existing viewing preferences. The relationship of the plurality of viewers can be analyzed by analyzing the image obtained through the camera 221 of the external device 200 at the time of recognizing the viewer. For example, if a plurality of viewers watch hands in a very friendly posture, for example, as a result of image analysis, a plurality of viewers can be estimated as 'lovers' or 'couples', and if there is no other viewer, A signal can be generated to provide the recommended contents of the genre. This can be performed irrespective of whether or not there is any difference between the previously learned personal viewing preferences or the existing learned common viewing preferences.

7A and 7B are illustrative conceptual diagrams showing a point of time when a viewer is recognized and a screen configuration in an external device in the electronic device related to the present invention. The control unit 180 of the electronic device 100 can recognize the viewer of the content based on the voice received through the microphone 122 and the image obtained through the camera provided in the external device 221. [ At this time, the camera 221 photographs the viewers adjacent to the external device 200, and can grasp the number of viewers.

The time point at which the viewer of the content is recognized in the electronic device 100 may be a time point when a start word (e.g., 'TV on') is input to the microphone 122 for connection to the external device 200. [ Alternatively, the external device 200 may be activated when the startup word is input.

As shown in FIG. 7A, when the number of the viewers grasped at the time of turning on the external device 200 is large, a specific broadcast program 701 determined according to the common viewing preference or popularity ranking of a plurality of viewers is displayed on the screen . In determining the specific broadcast program 701, the viewing preferences of each of a plurality of viewers are not reflected (except for coincidence).

On the other hand, as shown in FIG. 7B, if only one viewer is identified at the time when the sub-device 200 is turned on, the program 702 corresponding to the viewing preference of the viewer is displayed on the screen. At this time, the personal device or the peripheral device of the viewer can be automatically connected to the external device 200, and the information 710 indicating the connection state can be outputted through the screen of the external device 200. [

8, 9A, 9B, and 9C illustrate examples of operations related to control of an external device when the viewer of the content is changed from a plurality of viewers to a single viewer.

When it is detected that a plurality of viewers are converted into a single viewer in the electronic device 100, the control unit 180 transmits a signal for changing the content being viewed to a preference value corresponding to a viewing preference of a single viewer to the external device 200 ). For example, as shown in the left drawing of FIG. 8, the existing content 801 is changed to the brightness and volume-adjusted content 801 'optimized for the user A. At this time, information 811 informing the change of the environment setting value of the content is output in the screen 801 '.

As another example, when it is detected that a plurality of viewers are converted into a single viewer, the control unit 180 outputs a control signal for connecting a registered device (e.g., a smartphone) of a single viewer to the external device 200 And output the generated connection information in response to the output control signal. 8, the information 821 indicating that the smartphone of User A is connected to the smart TV 200 is output through the screen of the smart TV 200 or the speaker of the electronic device 100 .

9A shows that the screen of the smartphone 200A of the viewer connected to the external device 200 is outputted through the screen of the external device 200B. At this time, the screen of the smartphone 200A may be upscaled and provided to the external device 200 (902).

As another example, if it is detected that a plurality of viewers have been converted into a single viewer, the electronic device 100 receives the sound of the content 903 as shown in FIG. 9B, If the language used is different from the language used by the viewer, a signal for providing the caption 911 can be provided to the external device 200. At this time, the subtitles corresponding to the sound of the contents can be received and output in real time by communicating with the server to which the artificial intelligence technology is applied.

In another example, the subtitles may be output (912, 913) such that the dl is close to the objects of the content 904 being watched (e.g., in the form of speech bubbles) as shown in Figure 9c. For this, the position on the screen of the currently speaking object and the caption generation position can be matched and displayed in real time based on the source sound of the content 904.

10 and 11 illustrate examples of operations associated with recommending a viewer account based on the user's voice on which the electronic device 100 associated with the present invention is received.

10, the user's voice is received from the electronic device 100 connected to the external device 200 (S1010). Next, the electronic device 100 recognizes the viewer based on the received voice and analyzes the information of the viewer (S1020). For example, the age, gender, current mood, or emotional state of the viewer can be analyzed based on the characteristics of the received voice, for example, the tone, the quantity, the height, and the pitch of the voice. The viewer here assumes that there is no registered viewer account. If the age of the viewer is high, the optimum volume level can be set higher than the reference value. Further, if the age of the viewer who is grasped through voice is very low (for example, a child), the maximum viewing time can be automatically set so as not to watch for a long time.

11, the camera 221 of the external device 200 is activated to display the body information (e.g., the face image 1101) of the viewer, Gender, age, etc., from the subject). It is also possible to estimate the status on a member of the viewer based on another registered viewer account.

Then, the control unit 180 of the electronic device 100 recommends a viewer account corresponding to the viewer analysis result (S1030). The recommendation of the viewer account may be provided through the screen of the external device 200 or through the display unit 151 or the speaker 152 of the electronic device 100. [ For example, as shown in FIG. 11, when a result of analyzing information of a viewer is identified as 'Mom' among family members, a pop-up window 1102 asking whether to register it as a viewer account is provided on the screen of the external device 200 .

In one example, the control unit 180 may output information for inducing registration of a viewer account. Further, in one example, the viewer's taste of the viewer is learned based on the result of the response to the outputted information. To this end, the control unit 180 outputs guidance information of a recommended viewer account corresponding to the viewer analysis result. For example, as shown in FIG. 11, a viewer account can be registered based on the feedback on the guidance voice 1103 output through the speaker of the electronic device 100. On the other hand, a recommended viewer account may be modified or rejected based on the user's voice.

The registered viewer account is stored in the memory of electronic device 100 (and / or external device 200). Thereafter, when the user's voice corresponding to the registered viewer account is received, the viewer's taste is automatically learned.

Figs. 12 and 13 show an exemplary operation related to performing an associated function by grasping an object of interest in contents based on a voice of a viewer in an electronic device related to the present invention. Fig.

Referring to FIG. 12, the electronic device 100 may receive a voice of a viewer that is ignited during content viewing through the external device 200 connected to the electronic device 100 (S1210). At this time, the voice of the received viewer excludes voice related to controlling the operation of the external device 200. That is, here, it may be limited to audio information indicating natural feelings, feelings, feelings, and the like with respect to the portion that the viewer is interested in from the content currently being viewed.

Next, a signal for selecting a specific area of the content based on the received voice is generated (S1220). At this time, a window may be displayed in a specific area selected on the screen of the connected external device 200. For example, in FIG. 13, when the viewer watches the contents 1301 through the TV 200 and naturally utteres the appreciation of a specific person included in the contents (for example, 'John is cute'), The detected faces may be displayed as a window type 1311 in the content 1301. [ If a plurality of areas are selected based on the received voice, a window may be displayed for each area.

If a specific area is selected from the content being watched in this way, the sound of the object included in the selected specific area can be controlled to be extracted (S1230). Specifically, a sound matching with the object can be searched and extracted from the source sound of the content being watched. For example, in FIG. 13, a region in which a face image in a window matches 'John' is selected, and a voice uttered by 'John' can be extracted.

Next, the electronic device 100 may generate a signal for volume up the extracted sound and transmit the generated signal to the external device 200 (S1240). Accordingly, the extracted sound is output from the external device 200 in a volume-up state. Accordingly, as shown in FIG. 13, the voice of the extracted 'John' is volume-up provided (1312) and the other sound is automatically volume-down or filtered. At this time, in one example, as shown in FIG. 13, a zoomed-in screen 1302 can be provided centered on the selected region, that is, the face image of 'John'.

Further, if the above-described state continues for a predetermined time or longer, it is reflected in the viewing preference of the viewer. That is, the viewer's taste of the viewer is learned based on the selected object and the extracted sound (S1250).

In another example, the selection of a specific area in the content may be made through input to the external device 200 itself or a remote control device that controls the operation of the external device 200. [

As described above, according to the embodiments of the present invention, it is possible to naturally grasp whether a viewer is a viewer of a connected device based on a user's voice, to recommend content suitable for the taste of the viewer or to provide environment settings. Accordingly, it is possible to solve the inconvenience and time consuming operation for inputting a desired viewing content or an appropriate viewing environment for the viewer. In addition, it is possible to change the content or environment setting value in accordance with the changed environment by recognizing the configuration change of the content viewer without user's manipulation. In particular, even when a plurality of viewers have different viewing preferences, .

The present invention described above can be embodied as computer-readable codes on a medium on which a program is recorded. The computer readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the computer readable medium include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, , And may also be implemented in the form of a carrier wave (e.g., transmission over the Internet). Also, the computer may include a control unit 180 of the terminal. Accordingly, the above description should not be construed in a limiting sense in all respects and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

Claims (18)

A microphone configured to receive voice;
A wireless communication unit for communicating with an external device displaying content;
A control unit for recognizing a viewer of the content based on the received voice and determining whether the viewing taste of the recognized viewer is learned; And
And a running data section that configures information related to the viewing preference of the viewer based on the determination result,
Wherein the control unit outputs a signal related to the change of the content based on the configured information,
Wherein the viewing preference of the viewer is learned based on feedback on the output signal. The method according to claim 1,
Wherein,
And transmits a signal for providing a recommended content corresponding to the learned viewing preference to the external device if the viewing taste of the recognized viewer has been previously learned and outputs guidance information related to the recommended content Lt; / RTI > The method according to claim 1,
Wherein,
And determines the recommended content based on the gender and the age range of the viewer who is grasped based on the recognized voice of the viewer if the recognized viewer's taste has not been learned before. The method according to claim 1,
Wherein,
And recognizes the viewer of the content based on the received voice and an image obtained through a camera provided in the external device. The method according to claim 1,
Wherein,
And when a start word for connection with the external device is input to the microphone, the electronic device recognizes the viewer of the content based on the received voice. The method according to claim 1,
Wherein,
And generates a signal related to a change in the content being watched based on a viewing preference of the viewer when the configuration change of the viewer is detected based on the received voice and image while watching the content. The method according to claim 6,
Wherein,
And transmits to the external device a signal requesting to provide different recommended contents according to whether a plurality of viewers are recognized based on the received audio and video, Device. 8. The method of claim 7,
Wherein,
If the learned viewing tastes of the plurality of viewers are different, it is determined whether there is a common viewing taste by analyzing the correlation degree of the learned viewing tastes, and a signal for providing different kinds of recommended contents based on the determination result And generates an electric signal. 9. The method of claim 8,
Wherein,
Provides a signal for recommending contents corresponding to a common viewing preference to the external device according to a result of the analysis of the degree of correlation, and learns the common viewing preference based on a response result of the external device. 9. The method of claim 8,
Wherein,
And if it is determined that there is no common viewing preference, an audience viewer of the plurality of viewers is recognized based on the received voice and video,
And transmits a signal for simultaneously recommending the first content corresponding to the viewing preference of the leader viewer and the second content corresponding to the viewing priority to the external device. 11. The method of claim 10,
Wherein the recommended ratio of the first and second contents is adjusted according to the ratio of the leader viewers among the plurality of viewers. The method according to claim 6,
Wherein,
When a plurality of viewers are switched to a single viewer based on the received voice and image, a signal for changing the content being viewed to a preference value corresponding to the viewing preference of the single viewer is transmitted to the external device And the electronic device. The method according to claim 6,
When a plurality of viewers are switched to a single viewer on the basis of the received voice and image, outputs a control signal for connecting the registered device of the single viewer to the external device, And outputs the generated connection information. The method according to claim 1,
Wherein,
Analyzes a viewer's information based on the received voice, transmits a control signal for recommending a viewer account corresponding to the analysis result to the external device, and generates a signal related to the change of content based on the feedback on the control signal And the electronic device. 15. The method of claim 14,
Wherein,
And displays the guide information of the recommended viewer account, registers the viewer account based on the feedback on the guide information, and learns the viewing preference based on the voice of the viewer corresponding to the registered viewer account. . The method according to claim 1,
A control unit that selects a specific area of the content based on a voice of a viewer received while viewing the content, extracts sound of an object included in the selected specific area, and provides a control signal to the external device to increase the volume of the extracted sound . 17. The method of claim 16,
And provides a control signal for zooming down the specific area to the external device when a predetermined voice signal is inputted while the sound of the object is extracted. Connecting to an external device for displaying content;
Recognizing a viewer of the content based on a voice received by the microphone;
Determining whether a viewer's taste of the recognized viewer is learned;
Configuring information related to the viewing preference of the viewer based on the determination result;
Outputting a signal related to a change of the content based on the configured information; And
And receiving feedback on the output signal to learn the viewing preference of the viewer.

Images