KR102390685B1 - Electric terminal and method for controlling the same - Google Patents

Abstract

Disclosed are an electronic device capable of receiving a user's voice and processing information based on artificial intelligence technology, and a method for controlling the same. An electronic device according to the present invention includes a microphone configured to receive a voice, a wireless communication unit communicating with an external device for displaying content, and recognizing a viewer of content based on a voice received through the microphone, and recognizing the viewer's viewing It includes a control unit for determining whether the taste is learned, and a running data unit for configuring information related to the viewer's viewing preference based on the determination result of the control unit. In addition, the control unit outputs a signal related to content change based on the configured information, and the viewer's viewing preference is continuously learned based on the feedback on the output signal.

Description

ELECTRIC TERMINAL AND METHOD FOR CONTROLLING THE SAME

The present invention relates to an electronic device capable of receiving a user's voice and processing it based on artificial intelligence technology.

Recently, along with the development of hardware, artificial intelligence technology that implements human thinking processes, that is, cognition, reasoning, learning, etc. with computing technology, is developing rapidly.

Artificial intelligence technology can not only be a subject of research on its own, but can also be directly or indirectly combined with other fields of computer science to provide various functions. In particular, attempts are being made to introduce artificial intelligence elements in various fields of information technology and use them to solve problems in that field.

Artificial intelligence technology is divided into strong artificial intelligence and weak artificial intelligence. Strong AI is a technology that can make human-like thinking and decision-making, and it is a level of technology that learns by itself and makes decisions on its own. Weak AI is a level of technology that provides optimal solutions by performing cognitive processes such as perception and reasoning through computational models.

As part of the development of such techniques, attempts to provide various functions by applying artificial intelligence technology to electronic devices most familiar to users are increasing.

In particular, research related to further expanding the application of artificial intelligence technology by connecting an electronic device to other devices is being actively conducted. For example, another device may be connected based on a user's voice received by the electronic device, and a control command to the connected device may be input through the electronic device.

However, until now, control of the connected device is performed only when there is a user's intentional input, so there is a limit in accurately and quickly accessing the content or menu function that the user wants to view among various contents or menu functions. In addition, in recent years, smart TVs that provide various contents have been provided, and although the purpose is to grasp and control the desired items without user manipulation, the operation is still at an insufficient stage.

In order to solve the above-described problems, the present invention recognizes a viewer of a connected device based on the user's voice and performs processing based on artificial intelligence technology to enable accurate and quick access to the content desired by the viewer without user manipulation. An object of the present invention is to provide an apparatus and a method for controlling the same.

Another object of the present invention is to provide an electronic device capable of changing content being viewed or environment settings according to a changed environment when a member of a viewer changes while viewing content, and a method for controlling the same.

According to an aspect of the present invention, there is provided an electronic device comprising: a microphone configured to receive a voice; a wireless communication unit communicating with an external device that displays content; a controller for recognizing the viewer of the content based on the received voice and determining whether the recognized viewer's viewing preference has been learned; and a running data unit configuring information related to the viewer's viewing preference based on the determination result, wherein the control unit outputs a signal related to a change of the content based on the configured information, and the viewer's viewing preference It is characterized in that it is learned based on the feedback on the output signal.

Also, in an embodiment, if the recognized viewer's viewing preference has been previously learned, the controller transmits a signal for providing recommended content corresponding to the learned viewing preference to the external device, and the recommendation It is characterized in that the guide information related to the content is output.

In addition, in an embodiment, if the recognized viewer's viewing preference has not been previously learned, the controller determines the recommended content based on the viewer's gender and age group determined based on the recognized viewer's voice characterized in that

Also, in an embodiment, the controller is characterized in that the viewer of the content is recognized based on the received voice and an image obtained through a camera provided in the external device.

Further, in an embodiment, when a start-up word for connection with the external device is input to the microphone, the controller recognizes a viewer of the content based on the received voice.

In addition, in an embodiment, when a change in the configuration of the viewer is detected based on the received audio and image while viewing the content, the controller relates to a change in the content being viewed based on the changed viewing preference of the viewer. It is characterized in that it generates a signal.

In addition, in an embodiment, when a plurality of viewers are recognized based on the received voice and image, the controller sends a signal requesting to provide different recommended content according to whether there are a plurality of viewers whose viewing tastes have been learned. It is characterized in that it is transmitted to the external device.

Also, in an embodiment, if the learned viewing tastes of the plurality of viewers are different from each other, the controller determines whether there is a common viewing preference by analyzing the correlation between the learned viewing preferences, and based on the determination result It is characterized in that the signal for providing different types of recommended content is generated.

Also, in an embodiment, the control unit provides a signal for recommending content corresponding to a common viewing preference according to the analysis result of the correlation to the external device, and based on the response result of the external device, the common viewing It is characterized by learning tastes.

Also, in an embodiment, if there is no common viewing preference as a result of the analysis of the correlation, the controller identifies a leader viewer among the plurality of viewers based on the received voice and image, and views the leader viewer It is characterized in that a signal for simultaneously recommending the first content corresponding to the taste and the second content according to the viewing priority is transmitted to the external device.

Also, in an embodiment, the recommendation ratio of the first and second contents is characterized in that the ratio is adjusted according to the ratio of the reader viewer among the plurality of viewers.

Also, in an embodiment, when it is detected that a plurality of viewers have been converted to a single viewer based on the received audio and video, the controller sets an environment corresponding to the viewing preference of the single viewer for the content being viewed. It is characterized in that it transmits a signal that changes to a value to the external device.

Also, in an embodiment, when it is detected that a plurality of viewers have been converted to a single viewer based on the received audio and video, a control signal for connecting the registered device of the single viewer to the external device is output. and outputting connection information generated in response to the output control signal.

Also, in an embodiment, the control unit analyzes viewer information based on the received voice, transmits a control signal for recommending a viewer account corresponding to the analysis result to the external device, and provides feedback on the control signal It is characterized in that the signal related to the change of the content is generated based on the .

Also, in an embodiment, the controller outputs guide information of a recommended viewer account, registers a viewer account based on feedback on the guide information, and based on a viewer's voice corresponding to the registered viewer account It is characterized by learning viewing preferences.

Also, in an embodiment, the controller selects a specific area of the content based on a viewer's voice received while watching the content, extracts a sound of an object included in the selected specific area, and adjusts the volume of the extracted sound. It is characterized in that it provides a control signal to the external device to up.

Also, in an embodiment, when a preset voice signal is input while the sound of the object is extracted, the controller provides a control signal for zooming in the specific area to the external device.

A method of controlling an electronic device according to an embodiment of the present invention includes the steps of connecting to an external device that displays content, recognizing a viewer of the content based on a voice received by a microphone, and learning the recognized viewer's viewing preference Determining whether or not there is, based on the determination result, configuring information related to the viewer's viewing preference, outputting a signal related to a change of the content based on the configured information, and adding to the output signal It is characterized in that it comprises the step of learning the viewing preference of the viewer by receiving the feedback.

According to the electronic device and its control method according to the embodiment of the present invention, it is possible to naturally determine whether the user is a viewer of a connected device based on the user's voice, and recommends content suitable for the identified viewer's taste or provides environment settings by itself does it Accordingly, inconvenience and time consumption of manipulation and input for searching for content desired by a viewer or setting a suitable viewing environment are eliminated. In addition, it recognizes changes in the composition of content viewers without user manipulation and changes content or environment setting values according to the changed environment. to provide

1 is a conceptual diagram illustrating a state in which an electronic device related to the present invention communicates with an external device that outputs content.
FIG. 2A is a block diagram illustrating a detailed configuration of an electronic device related to the present invention, and FIG. 2B is an exemplary conceptual diagram of the electronic device of FIG. 2A.
2C is an exemplary conceptual diagram for explaining a process of recognizing a viewer of an external device in an electronic device related to the present invention.
3 is a representative flowchart for explaining an operation implemented in an electronic device related to the present invention.
4A, 4B, and 4C are exemplary conceptual diagrams for explaining that the electronic device related to the present invention controls content output to a connected external device based on a recognized viewing preference of a viewer.
5 is a flowchart illustrating an operation related to recommending different content based on a changed viewer when a change in a viewer is detected in the electronic device according to the present invention.
6A, 6B, 6C, 6D, and 6E are exemplary conceptual diagrams illustrating how recommended content is configured differently based on a change in a viewer and a common viewing preference of a plurality of viewers in an electronic device related to the present invention. .
7A and 7B are exemplary conceptual views illustrating a view point at which a viewer is recognized and a screen configuration in an external device in an electronic device related to the present invention.
8, 9A, 9B, and 9C are exemplary conceptual diagrams illustrating an operation related to control of an external device when a plurality of viewers is changed to a single viewer in an electronic device related to the present invention.
10 and 11 are flowcharts and exemplary conceptual diagrams for explaining an operation related to recommending a viewer account based on a user's voice in an electronic device related to the present invention.
12 and 13 are flowcharts and exemplary conceptual diagrams for explaining an operation related to performing a related function by recognizing an object of interest in content based on a viewer's voice in an electronic device related to the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number such as 1st, 2nd, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Electronic devices described in this specification include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, and slate PCs. , tablet PCs, ultrabooks, wearable devices, for example, watch-type terminals (smartwatch), glass-type terminals (smart glass), HMD (head mounted display), etc. may be included. there is.

However, those skilled in the art will readily appreciate that the configuration according to the embodiments described herein may also be applied to electronic devices such as digital TVs, desktop computers, digital signage, etc., except when applicable only to electronic devices. will be.

1 shows a state in which an electronic device related to the present invention is connected to and communicates with an external device that outputs content.

As shown in FIG. 1 , the electronic device 100 according to the present invention may be implemented in the form of a speaker. In addition, the electronic device 100 according to the present invention may recognize a voice and, based on the recognized voice, may execute a voice recognition function for controlling the operation of the electronic device 100 . Here, the voice recognition function is a function for controlling the electronic device through voice, and includes a natural language processing function for analyzing natural language, a question answering function for presenting an answer to a question, and a language capable of processing digital voice signals in a computer. It may include a voice processing function that converts voice into , an automatic interpretation and translation function that converts voice into multiple languages, and the like.

Also, the electronic device 100 according to the present invention may be wirelessly connected to an external device 200 that outputs content based on the recognized voice, for example, a smart TV. After the electronic device 100 and the external device 200 are connected as described above, the output from the external device 200 may be controlled based on the voice recognized by the electronic device 100 . Here, controlling the output includes, for example, changing, editing, or adding content output from the external device 200 or adjusting the setting environment of the content, for example, volume, aspect ratio, brightness, etc. can do.

Also, in the present invention, based on the voice recognized by the electronic device 100 , the external device 200 and another device 300 , for example, a smart phone, a set-top box, etc. may be controlled to be connected. In this case, the operation of another device 300 connected to the external device 200 may be controlled based on a voice command input to the electronic device 100 connected to the external device 200 . In this case, the operation result of the other device 300 may be output through the electronic device 100 .

Hereinafter, FIG. 2A is a block diagram for explaining an electronic device related to the present invention, and FIG. 2B is a conceptual diagram illustrating an example of an electronic device related to the present invention, that is, an example of an electronic device in the form of a speaker.

The electronic device 100 includes a wireless communication unit 110 , an input unit 120 , a sensing unit 140 , an output unit 150 , an interface unit 160 , a memory 170 , a control unit 180 , and a power supply unit 190 . ) and the like. The components shown in FIG. 2A are not essential for implementing the electronic device, and thus, the electronic device described in this specification may have more or fewer components than those listed above.

More specifically, among the components, the wireless communication unit 110 is between the electronic device 100 and the wireless communication system, between the electronic device 100 and another electronic device 100 , or between the electronic device 100 and an external server. It may include one or more modules that enable wireless communication between them. Also, the wireless communication unit 110 may include one or more modules for connecting the electronic device 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast reception module 111 , a mobile communication module 112 , a wireless Internet module 113 , a short-range communication module 114 , and a location information module 115 . .

The input unit 120 includes a camera 121 or an image input unit for inputting an image signal, a microphone 122 or an audio input unit for inputting an audio signal, and a user input unit 123 for receiving information from a user, for example, , a touch key, a push key, etc.). The voice data or image data collected by the input unit 120 may be analyzed and processed as a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of information in the electronic device, surrounding environment information surrounding the electronic device, and user information. For example, the sensing unit 140 may include a proximity sensor 141, an illumination sensor 142, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, and gravity. Sensor (G-sensor), gyroscope sensor, motion sensor, RGB sensor, infrared sensor (IR sensor: infrared sensor), fingerprint recognition sensor (finger scan sensor), ultrasonic sensor , optical sensors (eg, cameras (see 121)), microphones (see 122), battery gauges, environmental sensors (eg, barometers, hygrometers, thermometers, radiation detection sensors, It may include at least one of a thermal sensor, a gas sensor, etc.) and a chemical sensor (eg, an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the electronic device disclosed in the present specification may combine and utilize information sensed by at least two or more of these sensors.

The output unit 150 is for generating an output related to visual, auditory or tactile sense, and includes at least one of a display unit 151 , a sound output unit 152 , a haptip module 153 , and an optical output unit 154 . can do. The display unit 151 may implement a touch screen by forming a layer structure with the touch sensor or being integrally formed. Such a touch screen may function as the user input unit 123 providing an input interface between the electronic device 100 and the user, and may provide an output interface between the electronic device 100 and the user.

The interface unit 160 serves as a passage with various types of external devices connected to the electronic device 100 . The interface unit 160 connects a device equipped with a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and an identification module. It may include at least one of a port, an audio input/output (I/O) port, a video input/output (I/O) port, and an earphone port. In response to the connection of the external device to the interface unit 160 , the electronic device 100 may perform appropriate control related to the connected external device.

Also, the memory 170 stores data supporting various functions of the electronic device 100 . The memory 170 may store a plurality of application programs (or applications) driven in the electronic device 100 , data for operation of the electronic device 100 , and commands. At least some of these application programs may be downloaded from an external server through wireless communication. Also, at least some of these application programs may exist on the electronic device 100 from the time of shipment for basic functions (eg, incoming calls, outgoing functions, message reception, and outgoing functions) of the electronic device 100 . Meanwhile, the application program may be stored in the memory 170 , installed on the electronic device 100 , and driven to perform an operation (or function) of the electronic device by the controller 180 .

In addition to the operation related to the application program, the controller 180 generally controls the overall operation of the electronic device 100 . The controller 180 may provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the above-described components or by driving an application program stored in the memory 170 .

In addition, the controller 180 may control at least some of the components discussed with reference to FIG. 1A in order to drive an application program stored in the memory 170 . Furthermore, in order to drive the application program, the controller 180 may operate at least two or more of the components included in the electronic device 100 in combination with each other.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 to supply power to each component included in the electronic device 100 . The power supply unit 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the respective components may operate cooperatively to implement an operation, control, or control method of an electronic device according to various embodiments described below. Also, the operation, control, or control method of the electronic device may be implemented on the electronic device by driving at least one application program stored in the memory 170 .

Hereinafter, before looking at various embodiments implemented through the electronic device 100 as described above, the above-listed components will be described in more detail with reference to FIG. 1A .

First, referring to the wireless communication unit 110 , the broadcast reception module 111 of the wireless communication unit 110 receives a broadcast signal and/or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more of the broadcast reception modules may be provided to the mobile terminal 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

Mobile communication module 112, the technical standards or communication methods for mobile communication (eg, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (CDMA2000), EV -Enhanced Voice-Data Optimized or Enhanced Voice-Data Only (DO), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced, etc.) transmits and receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network constructed according to.

The wireless signal may include various types of data according to transmission/reception of a voice call signal, a video call signal, or a text/multimedia message.

The wireless Internet module 113 refers to a module for wireless Internet access, and may be built-in or external to the electronic device 100 . The wireless Internet module 113 is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

As wireless Internet technology, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), etc., and the wireless Internet module ( 113) transmits and receives data according to at least one wireless Internet technology within a range including Internet technologies not listed above.

From the point of view that wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A, etc. is made through a mobile communication network, the wireless Internet module 113 performs wireless Internet access through the mobile communication network. ) may be understood as a type of the mobile communication module 112 .

Short-range communication module 114 is for short-range communication, Bluetooth™, RFID (Radio Frequency Identification), infrared communication (Infrared Data Association; IrDA), UWB (Ultra Wideband), ZigBee, NFC At least one of (Near Field Communication), Wireless-Fidelity (Wi-Fi), Wi-Fi Direct, and Wireless Universal Serial Bus (USB) technologies may be used to support short-range communication. The short-range communication module 114, between the electronic device 100 and a wireless communication system, between the electronic device 100 and another electronic device 100, or the electronic device 100 through wireless area networks (Wireless Area Networks). ) and a network in which another electronic device 100 or an external server is located may support wireless communication. The local area network may be local area networks (Wireless Personal Area Networks).

Here, the other electronic device 100 is a wearable device capable of exchanging (or interworking with) data with the electronic device 100 according to the present invention, for example, a smart watch, smart glasses, and the like. (smart glass) or HMD (head mounted display)). The short-range communication module 114 may detect (or recognize) a wearable device capable of communicating with the electronic device 100 in the vicinity of the electronic device 100 . Furthermore, when the detected wearable device is a device authenticated to communicate with the electronic device 100 according to the present invention, the controller 180 transmits at least a portion of data processed by the electronic device 100 to the short-range communication module ( 114) to transmit to the wearable device. Accordingly, the user of the wearable device may use data processed by the electronic device 100 through the wearable device. For example, according to this, when a call is received by the electronic device 100, the user performs a phone call through the wearable device, or when a message is received by the electronic device 100, the user receives the received call through the wearable device It is possible to check the message.

The location information module 115 is a module for acquiring a location (or current location) of an electronic device, and a representative example thereof includes a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, if the electronic device utilizes the GPS module, it may acquire the location of the electronic device by using a signal transmitted from a GPS satellite. As another example, if the electronic device utilizes the Wi-Fi module, it may acquire the location of the electronic device based on information of the Wi-Fi module and a wireless access point (AP) that transmits or receives a wireless signal. If necessary, the location information module 115 may perform any function of the other modules of the wireless communication unit 110 to obtain data on the location of the electronic device as a substitute or additionally. The location information module 115 is a module used to obtain the location (or current location) of the electronic device, and is not limited to a module that directly calculates or obtains the location of the electronic device.

Next, the input unit 120 is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For input of image information, the electronic device 100 uses one Alternatively, a plurality of cameras 121 may be provided. The camera 121 processes an image frame such as a still image or a moving image obtained by an image sensor in a video call mode or a shooting mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170 . Meanwhile, the plurality of cameras 121 provided in the electronic device 100 may be arranged to form a matrix structure, and various angles or focal points may be applied to the electronic device 100 through the cameras 121 forming the matrix structure as described above. A plurality of image information with can be input. In addition, the plurality of cameras 121 may be arranged in a stereo structure to acquire a left image and a right image for realizing a stereoscopic image.

The microphone 122 processes an external sound signal as electrical voice data. The processed voice data may be utilized in various ways according to a function (or a running application program) being performed by the electronic device 100 . Meanwhile, various noise removal algorithms for removing noise generated in the process of receiving an external sound signal may be implemented in the microphone 122 .

The user input unit 123 is for receiving information from a user, and when information is input through the user input unit 123 , the controller 180 may control the operation of the electronic device 100 to correspond to the input information. . The user input unit 123 is a mechanical input means (or a mechanical key, for example, a button located on the front, rear or side of the electronic device 100, a dome switch, a jog wheel, jog switch, etc.) and a touch input means. As an example, the touch input means consists of a virtual key, a soft key, or a visual key displayed on the touch screen through software processing, or a part other than the touch screen. On the other hand, the virtual key or the visual key may be displayed on the touch screen while having various shapes, and for example, graphic, text ), an icon, a video, or a combination thereof.

The learning data unit 130 may be configured to receive, classify, store, and output information to be used for data mining, data analysis, intelligent decision-making, and machine learning algorithms and techniques. The running data unit 130 receives, detects, detects, generates, and stores predefined information or information output in other ways through the terminal, or receives, detects, detects by other configurations, devices and terminals , one or more memory units configured to store generated, predefined or output data.

The running data unit 130 may include an integrated or provided memory in the electronic device. In an embodiment, the running data unit 130 may be implemented through the memory 170 . However, the present invention is not limited thereto, and the running data unit 130 is implemented in a memory associated with the electronic device 100 (eg, an external memory connected to the electronic device 100 (eg, an external memory connected by wire (or electrically)), or electronic It may be implemented through a memory included in a server capable of communicating with the device 100. In another embodiment, the running data unit 130 is a memory maintained in a cloud computing environment, or a terminal through a communication method such as a network. may be implemented through another remote memory accessible by

The learning data portion 130 is generally configured to identify, index, classify, manipulate, store, retrieve, and output data for use in supervised or unsupervised learning, data mining, predictive analytics, or other machine learning techniques. is configured to be stored in one or more databases. The information stored in the learning data unit 130 may be used by the controller 180 or a plurality of controllers included in the electronic device using at least one of different types of data analysis, machine learning algorithm, and machine learning technology. . Examples of such algorithms and techniques include K-Nearest neighbor system, fuzzy logic (eg, possibility theory), neural networks, Boltzmann machines, vector quantization, pulsed neural nets, support vector machines, maximum margin classifiers, hill-climbing, inductive logic systems, Bayesian networks, petri nets (e.g., finite state machines, mealy machines, Moore finite state machines), classifiers trees) (eg perceptron trees, support vector trees, markov trees, decision tree forests, random forests), horses Pandemonium models and systems, clustering, artificially intelligent planning, artificially intelligent forecasting, data fusion, sensor fusion, image fusion (image fusion), reinforcement learning (reinforcement learning), augmented reality (augmented reality), pattern recognition (pattern recognition), automatic planning (automated planning), etc. there is.

The controller 180 may determine or predict at least one executable operation of the electronic device based on information determined or generated using data analysis, machine learning algorithms, and machine learning techniques. To this end, the controller 180 may request, search, receive, or utilize the data of the running data unit 130 . The controller 180 may perform various functions for implementing a knowledge-based system, an inference system, and a knowledge acquisition system, and may include a system for uncertain reasoning (eg, a fuzzy logic system), an adaptive system, a machine learning system, and an artificial intelligence system. It can perform various functions including neural networks and the like.

The controller 180 is a sub-module that enables voice and natural language processing, such as an I/O processing module, an environmental condition module, a speech-text (STT) processing module, a natural language processing module, a work flow processing module, and a service processing module. may include Each of the sub-modules may have access to one or more systems or data and models in the electronic device, or a subset or superset thereof. Here, the target to which each of the sub-modules has access right may include scheduling, vocabulary index, user data, task flow model, service model, and automatic speech recognition (ASR) system. In another embodiment, the controller 180 or the electronic device may be implemented with the sub-module, system, or data and model.

In some embodiments, based on the data in the running data unit 130, the control unit 180 based on the user's intention or context conditions expressed as user input or natural language input to detect and detect a user's request can be configured. In addition, the controller 180 may actively derive or acquire information necessary to completely determine what the user requests according to a context condition or a user's intention. For example, the controller 180 may detect and sense a user's request by analyzing past data including past inputs and outputs, pattern matching, unambiguous words, input intentions, and the like. Also, the controller 180 may determine a workflow for executing a function requested by the user according to a context condition or a user's intention. Also, the controller 180 may execute a work flow for satisfying a user's request based on a context condition or a user's intention.

In some embodiments, the controller 180 includes memistors, memristors, transconductance amplifiers, pulsed neural circuits, artificial intelligent nanotechnology systems (eg, autonomous nanomachines), or artificial intelligent quantum machine systems (eg, quantum Dedicated hardware elements for learning data processing including neural networks) may be implemented. In some embodiments, the controller 180 may include a pattern recognition system such as a machine vision system, an acoustic recognition system, a handwriting recognition system, a data fusion system, a sensor fusion system, and a soft sensor. Machine vision systems may include content-based image search, optical character recognition, augmented reality, egomotion, tracking and optical flow, and the like.

The controller 180 may sense or receive information in the electronic device, information on the surrounding environment surrounding the electronic device, or other information through the sensing unit 140 . Also, the controller 180 may receive a broadcast signal and/or broadcast-related information, a wireless signal, wireless data, and the like, through the wireless communication unit 110 . Also, the controller 180 may receive image information (or a corresponding signal), audio information (or a corresponding signal), data, or information input from a user from the input unit.

The control unit 180 collects information, processes or classifies it (eg, knowledge graph, command policy, personalization database, conversation engine, etc.), and stores the processed or classified information in the memory 170 or learning data unit It can be stored in (130).

Then, the running data unit 130 determines the operation of the electronic device based on data analysis, machine learning algorithm, and machine learning technology, and the controller 180 controls the components of the electronic device to execute the determined operation. can be controlled The controller 180 may execute the determined operation by controlling the electronic device based on the control command.

In an embodiment, when a specific operation is performed, the controller 180 analyzes history information indicating the performance of the specific operation through data analysis, machine learning algorithm, and machine learning technology, and based on this analysis information, It is possible to update the learned information. Accordingly, the control unit 180 can improve the accuracy of the future performance of the data analysis, machine learning algorithm, and machine learning technology based on the updated information together with the learning data unit 130 .

Meanwhile, the sensing unit 140 senses at least one of information in the electronic device, surrounding environment information surrounding the electronic device, and user information, and generates a sensing signal corresponding thereto. The controller 180 may control the driving or operation of the electronic device 100 or perform data processing, functions, or operations related to an application program installed in the electronic device 100 based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 140 will be described in more detail.

First, the proximity sensor 141 refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing in the vicinity without mechanical contact using the force of an electromagnetic field or infrared rays. The proximity sensor 141 may be disposed in an inner region of the electronic device covered by the touch screen as described above or near the touch screen.

Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, the proximity sensor 141 may be configured to detect the proximity of an object having conductivity by a change in an electric field according to the proximity of the object. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of description, the act of approaching an object on the touch screen without contacting it so that the object is recognized that it is located on the touch screen is called “proximity touch”, and the touch The act of actually touching an object on the screen is called "contact touch". The position where the object is touched in proximity on the touch screen means a position where the object is perpendicular to the touch screen when the object is touched in proximity. The proximity sensor 141 may detect a proximity touch and a proximity touch pattern (eg, proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement state, etc.) there is. Meanwhile, the controller 180 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 141 as described above, and further, provides visual information corresponding to the processed data. It can be printed on the touch screen. Furthermore, the controller 180 may control the electronic device 100 to process different operations or data (or information) according to whether a touch to the same point on the touch screen is a proximity touch or a contact touch. .

The touch sensor receives a touch (or touch input) applied to the touch screen (or the display unit 151) using at least one of various touch methods such as a resistive film method, a capacitive method, an infrared method, an ultrasonic method, and a magnetic field method. detect

As an example, the touch sensor may be configured to convert a change in pressure applied to a specific part of the touch screen or a change in capacitance occurring in a specific part of the touch screen into an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure at the time of touch, an electrostatic capacitance at the time of touch, etc. in which a touch object applying a touch on the touch screen is touched on the touch sensor. Here, the touch object is an object that applies a touch to the touch sensor, and may be, for example, a finger, a touch pen, a stylus pen, or a pointer.

As such, when there is a touch input to the touch sensor, a signal(s) corresponding thereto is sent to the touch controller. The touch controller processes the signal(s) and then transmits corresponding data to the controller 180 . Accordingly, the controller 180 can know which area of the display unit 151 has been touched, and the like. Here, the touch controller may be a component separate from the controller 180 , or may be the controller 180 itself.

Meanwhile, the controller 180 may perform different controls or may perform the same control according to the type of the touch object that touches the touch screen (or a touch key provided in addition to the touch screen). Whether to perform different control or the same control according to the type of the touch object may be determined according to the current operating state of the electronic device 100 or a running application program.

On the other hand, the above-described touch sensor and proximity sensor independently or in combination, a short (or tap) touch on the touch screen (short touch), long touch (long touch), multi touch (multi touch), drag touch (drag touch) ), flick touch, pinch-in touch, pinch-out touch, swype touch, hovering touch, etc. It can sense touch.

The ultrasonic sensor may recognize position information of a sensing target by using ultrasonic waves. Meanwhile, the controller 180 may calculate the position of the wave source through information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source may be calculated using the property that light is much faster than ultrasonic waves, that is, the time at which light arrives at the optical sensor is much faster than the time at which ultrasonic waves reach the ultrasonic sensor. More specifically, the position of the wave source may be calculated using a time difference from the time that the ultrasonic wave arrives using light as a reference signal.

On the other hand, the camera 121 as seen in the configuration of the input unit 120 includes at least one of a camera sensor (eg, CCD, CMOS, etc.), a photo sensor (or image sensor), and a laser sensor.

The camera 121 and the laser sensor may be combined with each other to detect a touch of a sensing target for a 3D stereoscopic image. The photo sensor may be stacked on the display device, and the photo sensor is configured to scan the motion of the sensing target close to the touch screen. More specifically, the photo sensor mounts photo diodes and transistors (TRs) in rows/columns and scans the contents placed on the photo sensors using electrical signals that change according to the amount of light applied to the photo diodes. That is, the photo sensor calculates the coordinates of the sensing target according to the amount of change in light, and through this, location information of the sensing target can be obtained.

The display unit 151 displays (outputs) information processed by the electronic device 100 . For example, the display unit 151 may display execution screen information of an application program driven in the electronic device 100 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information. .

Also, the display unit 151 may be configured as a stereoscopic display unit for displaying a stereoscopic image.

A three-dimensional display method such as a stereoscopic method (glasses method), an auto stereoscopic method (glassesless method), or a projection method (holographic method) may be applied to the stereoscopic display unit.

The sound output unit 152 may output audio data received from the wireless communication unit 110 or stored in the memory 170 in a call signal reception, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, and the like. The sound output unit 152 also outputs a sound signal related to a function (eg, a call signal reception sound, a message reception sound, etc.) performed by the electronic device 100 . The sound output unit 152 may include a receiver, a speaker, a buzzer, and the like.

The haptic module 153 generates various tactile effects that the user can feel. A representative example of the tactile effect generated by the haptic module 153 may be vibration. The intensity and pattern of vibration generated by the haptic module 153 may be controlled by a user's selection or setting of the controller. For example, the haptic module 153 may synthesize and output different vibrations or output them sequentially.

In addition to vibration, the haptic module 153 is configured to respond to stimuli such as a pin arrangement that moves vertically with respect to the contact skin surface, a jet force or suction force of air through a nozzle or an inlet, rubbing against the skin surface, contact of an electrode, electrostatic force, etc. Various tactile effects can be generated, such as the effect of heat absorption and the effect of reproducing a feeling of coolness and warmth using an element capable of absorbing heat or generating heat.

The haptic module 153 may not only deliver a tactile effect through direct contact, but may also be implemented so that the user can feel the tactile effect through a muscle sense such as a finger or arm. Two or more haptic modules 153 may be provided according to the configuration of the electronic device 100 .

The light output unit 154 outputs a signal for notifying the occurrence of an event by using the light of the light source of the electronic device 100 . Examples of the event generated in the electronic device 100 may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, information reception through an application, and the like.

The signal output from the light output unit 154 is realized when the electronic device emits light of a single color or a plurality of colors toward the front or rear. The signal output may be terminated when the electronic device detects the user's event confirmation.

The interface unit 160 serves as a passage with all external devices connected to the electronic device 100 . The interface unit 160 receives data from an external device, receives power and transmits it to each component inside the electronic device 100 , or allows data inside the electronic device 100 to be transmitted to an external device. For example, a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting a device equipped with an identification module (port), an audio I/O (Input/Output) port, a video I/O (Input/Output) port, an earphone port, etc. may be included in the interface unit 160 .

Meanwhile, the identification module is a chip storing various information for authenticating the usage right of the electronic device 100 , and includes a user identification module (UIM), a subscriber identity module (SIM), and general user authentication. It may include a universal subscriber identity module (USIM) and the like. A device equipped with an identification module (hereinafter, 'identification device') may be manufactured in the form of a smart card. Accordingly, the identification device may be connected to the terminal 100 through the interface unit 160 .

In addition, the interface unit 160 serves as a passage through which power from the cradle is supplied to the electronic device 100 when the electronic device 100 is connected to an external cradle, or is input from the cradle by a user. It may be a path through which various command signals are transmitted to the electronic device 100 . Various command signals or the power input from the cradle may be operated as signals for recognizing that the electronic device 100 is correctly mounted in the cradle.

The memory 170 may store a program for the operation of the controller 180 , and may temporarily store input/output data (eg, a phone book, a message, a still image, a moving picture, etc.). The memory 170 may store data related to vibrations and sounds of various patterns output when a touch input on the touch screen is input.

Memory 170 is a flash memory type (flash memory type), hard disk type (hard disk type), SSD type (Solid State Disk type), SDD type (Silicon Disk Drive type), multimedia card micro type (multimedia card micro type) ), card-type memory (such as SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read (EEPROM) -only memory), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The electronic device 100 may be operated in relation to a web storage that performs a storage function of the memory 170 on the Internet.

Meanwhile, as described above, the controller 180 controls an operation related to an application program and generally an overall operation of the electronic device 100 . For example, when the state of the electronic device satisfies a set condition, the controller 180 may execute or release a lock state that restricts input of a user's control command to applications.

In addition, the controller 180 performs control and processing related to voice calls, data communication, video calls, etc., or performs pattern recognition processing capable of recognizing handwriting input or drawing input performed on the touch screen as characters and images, respectively. can Furthermore, in order to implement various embodiments described below on the electronic device 100 according to the present invention, the controller 180 may control any one or a combination of the above-described components.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 to supply power required for operation of each component. The power supply unit 190 includes a battery, and the battery may be a built-in battery configured to be rechargeable, and may be detachably coupled to the terminal body for charging or the like.

In addition, the power supply unit 190 may include a connection port, and the connection port may be configured as an example of the interface 160 to which an external charger that supplies power for charging the battery is electrically connected.

As another example, the power supply unit 190 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 190 uses one or more of an inductive coupling method based on a magnetic induction phenomenon or a resonance coupling method based on an electromagnetic resonance phenomenon from an external wireless power transmitter. power can be transmitted.

Meanwhile, various embodiments below may be implemented in a computer-readable recording medium using, for example, software, hardware, or a combination thereof.

Continuing to refer to FIG. 2B , an electronic device in the form of a speaker is shown as an example of the electronic device 100 . As illustrated, the electronic device 100 in the form of a speaker may include a microphone 122 , a speaker 152 , and a display unit 151 . The frame 101 of the electronic device 100 has a built-in writing function, and may be implemented to activate the writing function in the frame according to a predetermined input. In addition, a camera 121 may be provided outside the electronic device 100 to acquire an image of the surroundings.

The electronic device 100 may operate in any one of a standby mode and a voice recognition mode. In the standby mode, it is detected whether a voice related to the execution of the voice recognition function exists in the vicinity of the electronic device before the execution of the voice recognition function. To this end, the controller 180 of the electronic device 100 may monitor (or monitor) whether a sound of a specific loudness or higher is continuously detected through the microphone 122 . In addition, since there is no need to perform voice analysis in the standby mode, it can operate with low power, for example, a current of about 4.4 mA. This standby mode may also be referred to as a “listening phase”.

Meanwhile, in the standby mode, when a sound greater than or equal to a specific volume is detected through the microphone 122 , the standby mode of the electronic device 100 may be switched to the voice recognition mode. Alternatively, the standby mode of the electronic device 100 may be switched to the voice recognition mode when a predetermined starting word greater than or equal to a specific volume is detected through the microphone 122 . Here, the startup word may include a predetermined voice command for waking up the electronic device 100 in the standby mode, for example, a voice command such as “Hello”, “Wake up”, “Google”, and the like.

The voice recognition mode refers to a state in which the controller 180 of the electronic device 100 analyzes a voice input through the microphone 122 . In the voice recognition mode, since voice analysis is performed, more current is consumed than in the standby state 210 . That is, before the start word is received, the electronic device according to the present invention exists in the standby state 210 in which voice analysis is not performed, thereby reducing current consumption. Meanwhile, before the voice analysis, the controller 180 of the electronic device 100 may determine whether or not a starting word for starting voice recognition is received in the voice recognition mode. In other words, the controller 180 may start the voice analysis of the voice uttered after the starting word.

In the voice recognition mode, the electronic device 100 may control the operation of the electronic device 100 by analyzing a voice input through the microphone 122 . Such voice analysis may be performed using a separate third-party application installed in the electronic device.

In addition, the controller 180 of the electronic device 100 transmits to a preset artificial intelligence server or to the running data unit 130 of the electronic device 100 in order to process voice analysis of voice information through an artificial intelligence algorithm. Stored data can be used. Here, the preset artificial intelligence server is a server that provides a function of learning a vast amount of information using an artificial intelligence algorithm and deriving optimal result information based on the learned information. Alternatively, the controller 180 of the electronic device 100 may generate result information in response to the input voice information based on data stored in the running data unit 130 .

In addition, if a user's voice or a predetermined startup word is not received for a preset time in the aforementioned voice recognition mode, the voice recognition mode may be switched back to the standby mode.

Subsequently, the speaker 152 may output a response result of the input voice information. The response result may include not only an answer to the input voice information, but also question or guidance information for confirming ambiguous voice information or a search range. In addition, the speaker 152 may output a preset notification sound when the aforementioned voice recognition mode is switched to the standby mode or from the standby mode to the voice recognition mode.

The display unit 151 may output a graphic object or an image corresponding to the response result output through the speaker 152 . Alternatively, when the speaker 152 is set to silent according to the setting of the electronic device 100 , a response result of the input voice information may be output only through the display unit 151 . On the other hand, when the display unit 151 is formed of the above-described touch screen, the user and the electronic device 100 may interact by controlling a graphic object or an image based on various types of touch inputs.

The camera 121 may be detachably mounted on one side of the electronic device 100 . The image acquired through the camera 121 may be used to identify a user corresponding to voice information input through the speaker 152 . Alternatively, the acquired image may be transmitted to another device according to a user's request.

Meanwhile, in the following description, it will be described based on the fact that the electronic device 100 has the running data unit 130 and is set to output the analysis result of the input voice information. However, the present invention is not limited thereto, and the electronic device 100 communicates with the artificial intelligence server to receive a response corresponding to the analysis of voice information, and the electronic device 100 operates based on the received response. may be

In addition, in the drawings and descriptions below, it has been described as an example that the electronic device 100 is implemented in the form of a speaker that outputs voice, but the present invention is not limited thereto, and those skilled in the art can realize that it can be implemented in various types of electronic devices. self-evident to

Hereinafter, FIG. 2C is an exemplary conceptual diagram for explaining a process of recognizing a viewer of the external device 200 that outputs content through the microphone 121 of the electronic device 100 related to the present invention.

First, in FIG. 2C , it is assumed that User A is a viewer of broadcast content output from the external device 200 and User B is not a viewer. When User A naturally utters a voice while watching broadcast content or a voice command intentionally utters to control the operation of the electronic device 100 or the external device 200 is received through the microphone of the electronic device 100, A separation distance between User A and the electronic device 100 communicating with the external device 200 is calculated by analyzing the received voice volume, etc., and based on the calculated separation distance, User A determines whether the external device 200 is a viewer can be judged

Regarding the separation distance calculation based on voice, it is possible to calculate the approximate separation distance by using the fact that the high tone is easily absorbed and the low tone is not easily absorbed, so that the low tone is more audible than the high tone even at a long distance. Also, in another example, a location of a user corresponding to a recognized voice may be recognized by recognizing a location of a remote control apparatus (eg, a remote controller) that controls an external device.

In this case, the separation distance from the electronic device 100 for determining that User A is a viewer is calculated including the separation distance between the external device 200 and the electronic device 100 . In the case of User B, the separation distance determined based on the volume of the voice received by the electronic device 100 is outside the range determined by the viewer of the external device 200 .

If the calculated separation distance is ambiguous, it may be confirmed whether the viewer is a viewer based on an image acquired through the camera 221 provided in the external device 200 . Since User A is within the range of view of the camera 221 , it is determined as a viewer, and User B is outside the range of view of the camera 221 , so it can be determined as not a viewer because the image is not captured. In one example, even if User A is within the angle of view of the camera 221 , the user A's face direction, posture, and state (sleep or not) are determined based on the acquired image and the content output from the external device 200 is viewed. You can be specific about what you are doing.

On the other hand, although User B was not initially a viewer, when voice information is recognized at a location close to the external device 200 or the electronic device 100 or close to the electronic device 100, it will be recognized as a viewer together with User A. can In this case, User A and User B are recognized as a plurality of viewers. Content control related thereto will be described in more detail below. Alternatively, User A may move out of the angle of view range of the camera 221 . In this case, User A may be excluded from the viewer.

3 is a representative flowchart for explaining an operation implemented in an electronic device related to the present invention.

Referring to FIG. 3 , a step in which the electronic device 100 according to the present invention is connected for communication with an external device 200 that displays content is started ( S10 ). Here, the external device 200 may be implemented as, for example, analog TV, digital TV, 3D-TV, smart TV, LED TV, OLED TV, plasma TV, monitor, smart phone, etc., but is not limited thereto. does not Also, the external device 200 includes a case in which a set-top box or an IP set-top box that provides a multimedia communication service is connected through a network.

Next, the electronic device 100 recognizes the viewer of the external device 200 based on the voice received through the microphone 122 ( S20 ). In this case, the received voice includes ambient noise, and the controller 180 of the electronic device 100 may extract a clear voice by filtering the ambient noise from the voice received through the microphone 122 .

The received voice may be voices of a plurality of users. In this case, the controller 180 classifies the voice information of a plurality of users, analyzes each of the divided voice information to determine the user's location, and based on the analysis result and the identified user's location, the plurality of users It is possible to recognize whether the device 200 is a viewer or which user of them is a viewer.

Also, the controller 180 may determine whether the received voice is the voice of a registered viewer. To this end, the controller 180 matches the voice corresponding to the viewer account previously stored in the memory 160 and the currently received voice of the user, and extracts information corresponding to the matched viewer account to specifically specify the characteristics of the viewer. can figure out In other words, when it is determined whether the user corresponding to the received voice wants to view the content of the external device 200 ('first determination'), determining whether the user who wants to view the content has a registered viewer account ( 'second judgment').

Next, the controller 180 of the electronic device 100 determines whether the viewer's viewing preference recognized based on the voice has been previously learned ( S30 ). To this end, the user's previously received voice information may be stored in the memory 160 of the electronic device 100 , and data related to the user's viewing preference matching the stored voice information is stored in the running data unit 130 . can be Data stored in the running data unit 130 may be continuously learned based on the voice of the same person received by the electronic device 100 or the external device 200 .

Here, the viewer's viewing preference refers to the viewer's preferred program, screen environment settings (eg, screen brightness, volume), preferred genre, average viewing duration, channel switching frequency, and whether the content is viewed in connection with other devices. It includes all of preference information, habits, characteristics, etc. related to content viewing, such as whether or not.

When a person close to the external device 200 is recognized as a viewer, the controller 180 determines whether the viewer's viewing preference has been previously learned. Then, based on the determination result, information related to the viewer's viewing preference is configured (S40).

If the viewing taste is a learned viewer, the viewer's viewing preference is configured based on the previous learning history. To this end, the learning data unit 130 of the electronic device 100 may continuously update and store the learning result of the viewing preference of the same viewer using a learning algorithm based on machine learning. Here, such machine learning may be performed based on a voice received by the electronic device 100 while the same viewer watches content through the external device 200 . For example, the viewer's preference program may be generated as data related to viewing preference by giving a score for each date and time. These viewing preferences are separately formed for each user, and are used as data only when the corresponding user is recognized as a viewer.

On the other hand, if the viewer's viewing preference has not been previously learned (that is, it is the voice of a new user or the voice of a user whose account is not registered), the viewer's preference is estimated by analyzing the characteristics of the viewer based on the received voice. . For example, characteristics such as age, gender, and emotional state of the viewer may be generated as an analysis result based on the received voice volume, strength, pitch (map), tone, type of language, word level, and the like. Accordingly, it is possible to estimate the average viewing taste according to the analyzed combination of age, gender, and emotional state. When it is difficult to identify the characteristics of the viewer only with the received voice, information (eg, image, location) collected through the external device 200 and a remote control device controlling the same may be further used.

On the other hand, if there are a plurality of viewers, the controller 180 considers all viewing tastes of the plurality of viewers. For example, if both User A and User B are recognized as viewers, whether the viewing preferences of User A and User B have been respectively learned, whether there are previously learned viewing preferences in a joint viewing situation, User A and User B A common viewing preference estimated according to the relationship between the two may be generated as an analysis result. To this end, it is possible to search for, extract, use, or combine data related to the viewer's viewing preference stored in the running data unit 130 .

If a voice is not received by the electronic device 100 for a predetermined time after the viewer is recognized or the viewer's viewing preference is recognized, the viewer's posture or state may be determined through the camera 121 provided in the external device 200 . . For example, you can determine whether a viewer is sleeping or concentrating on another task, so you can exclude them from your audience. Accordingly, if it is recognized that there is no viewer, power consumption of the external device 200 may be reduced by at least adjusting the screen brightness or decreasing the volume.

Next, the controller 180 of the electronic device 130 generates a signal related to a change of content output to the external device 200 based on the information related to the configured viewing preference (S50). The information generated in this way is transmitted to the external device 200 and reflected in the outputted content. Here, the signal related to content change refers to not only control operations related to other content such as recommendation of a preferred program, scheduled recording of preferred program, and channel change, but also environment setting of the content being viewed, for example, volume adjustment and screen brightness adjustment. , and includes all control signals for performing control operations such as aspect ratio adjustment.

Thereafter, the electronic device 100 learns the viewer's viewing preference stored in the running data unit 130 based on the feedback on the signal related to the change of the content. For example, if the signal related to the change of content is to provide a menu screen for recommending the viewer's preferred program, it is stored in the running data unit 130 based on the viewer's response (eg, rejection, selection, etc.) using the menu screen. A viewer's viewing preference may be learned. In this case, the viewer's response may be made through a voice (eg, 'No', 'I want to see this', etc.) received by the electronic device 100 .

As described above, in the present invention, even if the user does not intentionally search for, select, or change the content he/she wants to watch, the user can determine whether the viewer is a viewer only by the voice he utters, thereby helping the user to view the content according to his/her viewing preference.

4A, 4B, and 4C are conceptual diagrams illustrating examples of controlling content output to a connected external device based on a recognized viewing preference of a viewer in an electronic device related to the present invention.

First, referring to FIG. 4A , when a user utters a predetermined startup word (eg, “turn on TV”) 10 , the microphone 122 of the electronic device 100 receives it, and the electronic device according to the result of the voice recognition function (100) and the smart TV (200) are connected.

Then, the controller 180 searches for a matching viewer account based on the received voice (“turn on the TV”), generates a control signal related to the recommended content that the user wants to watch, and transmits the generated control signal to the smart TV 200 ) is sent to

Upon receiving the control signal, the smart TV 200 outputs information (“Hi, Jihoon”) corresponding to the viewer account and recommended content corresponding to the viewer's viewing preference on the screen 401 . If the viewer's viewing preference has been previously learned, the recommended content is configured based on the learned viewing preference. To this end, as described above, data stored in the running data unit 130 of the electronic device 100 may be utilized, and in another example, data stored in a server to which an artificial intelligence technology communicating with the electronic device 100 is applied is used. Alternatively, the viewing preference of the viewer account stored in the smart TV 200 itself may be utilized.

On the other hand, if the viewer's viewing preference has not been previously learned, the recommended content may be configured in order of popularity. In another example, if the viewer's viewing preference has not been previously learned, the recommended content may be determined based on at least one of the viewer's gender, age group, and emotional state, which is recognized based on the recognized viewer's voice. For example, if it is identified as a male in his 30s based on his voice, the recommended contents may be configured in the order of programs that are popular with males in his 30s. Also, when it is determined that emotional states such as stress and depression are included in the viewer's voice, the comic genre may be included in the recommended content.

When the viewer selects a specific broadcast program from the recommended content, this feedback is also transmitted to the electronic device 100 so that machine learning of the viewer's viewing preference is performed naturally. Also, while the viewer watches the specific broadcast program 402 , environment settings matching the viewer's viewing taste, for example, preferred screen brightness and volume, are automatically adjusted ( 410 ). In addition, if the user's voice is not received for a reference period of time while watching the specific broadcast program 402, the electronic device 100 performs a preset time interval (eg, 5 minutes or a settable) for the electronic device 100 . Alternatively, the camera provided in the smart TV 200 is activated to monitor whether the viewer continues to watch, whether there is a change in the viewer's configuration, and the like.

Also, in one example, the controller 180 may output guide information related to the recommended content while transmitting a signal for providing the recommended content corresponding to the learned viewing preference to the external device.

For example, referring to FIG. 4C , if it is determined that a preferred program corresponding to the viewer's viewing taste will start soon while the viewer watches a specific program 403 , a notification message notifying this is displayed on the smart TV 200 . may be output at the bottom of the screen. At the same time, a guide voice 20 for inducing a channel change for viewing a preferred program may be output through the speaker of the electronic device 100 . As long as the viewer performs a simple response ('Yeah!'), the channel switch to the preferred program is instantaneous. Even if the viewer wants to continue watching the current program, it is enough to simply say no ('no'). The viewer's response results are used to continuously learn viewing preferences.

Hereinafter, with reference to FIGS. 5 and 6A to 6E , when a change in the viewer's configuration is detected in the electronic device 100, it is a flowchart for explaining an operation related to recommending different content based on the changed viewer. Here, in particular, processing in the case where the number of viewers is changed due to a change in the configuration of the viewers will be described in more detail.

First, referring to FIG. 5 , based on a voice (and/or an image acquired through a camera of the external device 200 ) received by the electronic device 100 while watching content through the external device 200 . A change in the viewer's configuration may be recognized (S501).

6A is an example of a case in which a new viewer is introduced while watching content, and FIG. 6B is an example of a case in which an existing viewer leaves. For example, when User B comes close to the electronic device 100 or the external device 200 while uttering 'what do you see', it may be recognized as an influx of new viewers. In addition, as shown in FIG. 6B , when User C utters 'Come' while a plurality of viewers are watching content, the absence of User D is recognized based on the image acquired through the camera 221 of the external device 200 Thus, User D's exit can be recognized. In one example, when a voice related to a change in the viewer's configuration is detected, the camera 221 of the external device 200 may be activated to determine the number of viewers again.

The electronic device 100 determines the number of viewers (S502). If it is determined that there are a plurality of viewers, it is determined whether there are a plurality of viewers whose viewing tastes have been learned ( S503 ). As a result of the determination in step S503, if there are a plurality of viewers whose viewing tastes have been learned, a correlation between the plurality of viewing tastes is analyzed (S506).

The correlation analysis between the viewing tastes means the degree of matching or similarity of preference programs corresponding to a plurality of learned viewing tastes. In this regard, referring to FIG. 6C , when the learned viewing preferences of User A, User B, and User C are respectively viewing preference 1, viewing preference 2, and viewing preference 3, the number or ratio of mutually overlapping preferred programs or genres can be shown. In this correlation, when the data related to the viewing taste is a higher concept such as 'preferred genre', the date and time at which the data was collected may be considered as additional limiting conditions. If the data related to the viewing taste is a sub-concept or a specific concept such as 'preferred program', additional limiting conditions may be excluded or only the viewing frequency may be considered. In addition, if there is a matching preferred program or genre, but the score is low even in individual preferences, the correlation is considered to be very low and may be excluded from the following common viewing preferences.

As a result of the correlation analysis, it is determined whether there is a common viewing preference (S508), and if there is a common viewing preference, a corresponding content is recommended (S509). The control apparatus 100 learns a common viewing preference based on a response result from the external device 200 . Common viewing preferences are managed separately from individual viewing preferences of current viewers.

On the other hand, if there is no common viewing preference, it is determined whether the viewing preference is related to a plurality of viewers ( S510 ). That is, it is determined whether the data is generated when there are a plurality of viewers.

If the data is generated when there are a plurality of viewers but not a common viewing preference, it is determined whether the type of viewer corresponding to the data is a leader ( S511 ). Whether the viewer type is a leader may be determined by comparing the independent viewing preference of the corresponding viewer and the similarity or matching frequency of data generated when the viewer watches the video together with a large number of people. For example, if the viewer prefers program A when watching alone and watches program B of a different genre when watching with a large number of people in the same time zone, the viewer may be classified as a follower rather than a leader. On the other hand, if the viewer's preferred programs are not significantly different when viewed alone and when viewed by multiple people, they can be classified as a leader type. Alternatively, a viewer who grips a remote control device (not shown) that controls the operation of the external device 200 may be classified as a reader type. Also, there may be a plurality of viewers of this reader type.

When there are a plurality of viewers and the viewer of the generated data is a reader type, both reader-type content and popular content are recommended (S512). Specifically, the controller 180 may transmit to the external device 200 a signal for simultaneously recommending the first content corresponding to the viewing preference of the reader-type viewer and the second content according to the viewing priority. A plurality of viewers can select desired specific content from a recommendation list including reader-type content and popular content.

Also, in an example, the recommendation ratio of the first and second content may be differently adjusted according to the ratio of the reader type viewer among the plurality of viewers. Referring to FIGS. 6D and 6E , if the ratio of reader-type viewers to the total number of viewers is less than the majority, the recommendation ratio of popular content ('Now Hot') on the content recommendation screen 610A as shown in FIG. 6D . (612) is further increased. When the ratio of reader-type viewers exceeds the majority of the total number of viewers according to the change in the viewer's configuration, as shown in FIG. 6E , the content corresponding to the reader-type viewing preference in the content recommendation screen 610B ('You may like') The recommendation rate 613 is further increased. In this case, the recommendation rate of the popular content ('Now Hot') is automatically reduced (614).

If the viewer of the generated data is a follow-type viewer when there are a plurality of viewers, a popular content expected to be liked by several people is determined as the recommended content (S507).

On the other hand, if it is determined in step S503 that the number of viewers who have learned the viewing taste is singular, content with high popularity is recommended (S507). That is, if there are a plurality of viewers, a preferred program is not recommended according to the viewing preference of one of them. This takes into account that the viewing tastes of a plurality of viewers may be different from this.

On the other hand, if there is only one viewer in step S502, it is determined whether the viewing taste of the changed or remaining viewers has been learned (S504), and content according to the viewer's viewing preference is provided to the external device 200 (S505). ). That is, in this case, the electronic device 100 may transmit a channel change signal for switching to a channel corresponding to the preferred program to the external device 200 .

Also, in an example, the controller 180 may provide new recommended content different from the existing viewing taste by estimating the relationship between a plurality of viewers. The relationship between the plurality of viewers may be identified by analyzing an image obtained through the camera 221 of the external device 200 at the point in time of recognizing the viewer. For example, as a result of video analysis, if multiple viewers are watching in a very friendly posture, such as holding hands, it can be estimated that multiple viewers are 'lovers' or 'couples', and if there are no other viewers, A signal may be generated to provide recommended content of a genre. This may be performed regardless of whether there is a previously learned individual viewing preference that is different from or an existing learned common viewing preference.

7A and 7B are exemplary conceptual views illustrating a view point at which a viewer is recognized and a screen configuration in an external device in an electronic device related to the present invention. The controller 180 of the electronic device 100 may recognize a viewer of the content based on a voice received through the microphone 122 and an image obtained through a camera provided in the external device 221 . In this case, the camera 221 may capture the number of viewers by photographing viewers adjacent to the external device 200 .

As described above, the time point at which the electronic device 100 recognizes the viewer of the content may be the time point at which a startup word (eg, 'turn on TV') is input to the microphone 122 for connection with the external device 200 . Alternatively, it may be a time when the external device 200 is activated as the startup word is input.

As shown in FIG. 7A , if the number of viewers identified at the time the external device 200 is turned on is large, a specific broadcast program 701 determined according to a common viewing preference or popularity ranking of a plurality of viewers is output to the screen can do it In determining the specific broadcast program 701, individual viewing preferences of a plurality of viewers (except by chance) are not reflected.

On the other hand, as shown in FIG. 7B , if there is only one viewer identified when the auxiliary device 200 is turned on, a program 702 suitable for the viewer's viewing preference is output on the screen. In this case, the viewer's personal device or peripheral device may be automatically connected to the external device 200 , and information 710 indicating the connection state may be output through the screen of the external device 200 .

Next, FIGS. 8, 9A, 9B, and 9C show examples of operations related to control of an external device when a viewer of content is changed from a plurality of viewers to a single viewer.

When the electronic device 100 detects that a plurality of viewers have been converted to a single viewer, the controller 180 transmits a signal for changing the content being viewed to an environment setting value corresponding to the viewing preference of the single viewer to the external device 200 . ) can be sent to For example, as shown in the left diagram of FIG. 8 , the existing content 801 is changed to the content 801 ′ with brightness and volume adjusted optimized for User A. In this case, information 811 informing of a change in the environment setting value of the content is output on the screen 801 ′.

As another example, when it is detected that a plurality of viewers have been converted into a single viewer, the controller 180 outputs a control signal for connecting a registered device (eg, a smart phone) of a single viewer with the external device 200 . and output connection information generated in response to the output control signal. For example, as shown in the right diagram of FIG. 8 , information 821 indicating that User A's smart phone is connected to the smart TV 200 is output through the screen of the smart TV 200 or the speaker of the electronic device 100 . can be

9A shows that the screen of the viewer's smart phone 200A connected to the external device 200 is output through the screen of the external device 200B. At this time, the screen of the smart phone 200A may be up-scaled and provided to the external device 200 ( 902 ).

As another example, when it is detected that a plurality of viewers have been converted into a single viewer, as shown in FIG. 9B , the electronic device 100 receives the sound of the content 903 and analyzes the received sound as a result of language type After determining , if the type of language used is different from the language used by the viewer, a signal for providing the subtitle 911 may be provided to the external device 200 . In this case, the subtitle corresponding to the sound of the content may be received and output in real time by communicating with the server to which the artificial intelligence technology is applied.

In another example, as shown in FIG. 9C , the subtitle dl may be output in proximity to objects of the content 904 being viewed (eg, in the form of speech bubbles) 912 and 913 . To this end, based on the source sound of the content 904, the location of the currently uttering object on the screen and the location of the caption generation may be matched to display in real time.

10 and 11 show examples of operations related to recommending a viewer account based on a user's voice received by the electronic device 100 related to the present invention.

First, referring to FIG. 10 , the electronic apparatus 100 connected to the external device 200 receives a user's voice ( S1010 ). Next, the electronic device 100 recognizes the viewer based on the received voice and simultaneously analyzes the viewer's information ( S1020 ). For example, the age, gender, current mood or emotional state of the viewer may be analyzed based on the characteristics of the received voice, for example, the tone, volume, pitch, and pitch of the voice. Here, it is assumed that the viewer does not have a registered viewer account. If the age of the viewer identified through the voice is high, the optimal volume level may be set higher than the reference value. In addition, if the age of the viewer identified through the voice is very low (eg, a child), the maximum viewing time may be automatically set so as not to watch the video for a long time.

On the other hand, when the viewer's information identified based on the received voice is insufficient, as shown in FIG. 11 , the camera 221 of the external device 200 is activated to provide the viewer's body information (eg, face image 1101). gender, age, etc.) can be collected. It is also possible to estimate the status of the viewer on the basis of other registered viewer accounts.

Then, the controller 180 of the electronic device 100 recommends a viewer account corresponding to the viewer analysis result ( S1030 ). The recommendation of the viewer account may be provided through the screen of the external device 200 or may be provided through the display unit 151 or the speaker 152 of the electronic device 100 . For example, as shown in FIG. 11 , if it is identified as 'Mom' among family members as a result of analyzing the viewer's information, a pop-up window 1102 asking whether to register it as a viewer account will be provided through the screen of the external device 200. can

In an example, the controller 180 may output information inducing registration of a viewer account. In addition, in one example, the viewer's viewing preference is learned based on the result of the response to the output information. To this end, the controller 180 outputs guide information of the recommended viewer account corresponding to the viewer analysis result. For example, as shown in FIG. 11 , a viewer account may be registered based on the feedback on the guide voice 1103 output through the speaker of the electronic device 100 . Meanwhile, the recommended viewer account may be modified or registered based on the user's voice.

The registered viewer account is stored in the memory of the electronic device 100 (and/or the external device 200). Thereafter, when a user voice corresponding to the registered viewer account is received, the viewer's viewing preference is automatically learned.

12 and 13 show exemplary operations related to performing a related function by identifying an object of interest in content based on a viewer's voice in an electronic device related to the present invention.

Referring to FIG. 12 , the electronic apparatus 100 may receive a viewer's voice uttered while viewing content through the external device 200 connected to the electronic apparatus 100 ( S1210 ). In this case, the received voice of the viewer is excluded from the voice related to controlling the operation of the external device 200 . That is, here, it may be limited to voice information representing a natural emotion, appreciation, feeling, etc. of a part that the viewer is interested in viewing in the currently viewed content.

Next, a signal for selecting a specific area of content is generated based on the received voice (S1220). In this case, a window may be displayed in a specific area selected on the screen of the connected external device 200 . For example, in FIG. 13 , when a viewer naturally utters an appreciation for a specific person included in the content while watching the content 1301 through the TV 200 (eg, 'John is cute'), the face recognition function is Faces detected while performing may be displayed in the form of a window 1311 in the content 1301 . When a plurality of regions are selected based on the received voice, a window may be displayed for each region.

As such, when a specific region is selected from the content being viewed, it is possible to control to extract a sound of an object included in the selected specific region (S1230). Specifically, it is possible to search for and extract a sound matching the object from the source sound of the content being viewed. For example, in FIG. 13 , an area in which a face image in the window matches 'John' may be selected, and a voice uttered by 'John' may be extracted.

Next, the electronic apparatus 100 may generate a signal for increasing the volume of the extracted sound and transmit it to the external device 200 ( S1240 ). Accordingly, the volume of the extracted sound is increased in the external device 200 and output. Accordingly, as shown in FIG. 13 , the extracted voice of 'John' is provided with the volume up (1312), and other sounds are automatically reduced in volume or filtered. At this time, in one example, as shown in FIG. 13 , a zoomed-in screen 1302 may be provided based on the selected area, that is, the face image of 'John'.

In addition, if the above state continues for more than a predetermined time, this is reflected in the viewer's viewing preference. That is, the viewer's viewing preference is learned based on the selected object and the extracted sound ( S1250 ).

In another example, the selection of a specific region in the content may be made through the external device 200 itself or an input to a remote control device that controls the operation of the external device 200 .

As described above, according to embodiments of the present invention, it is possible to naturally determine whether a viewer is a connected device based on a user's voice, and recommends content suitable for the identified viewer's taste or provides environment settings by itself. Accordingly, inconvenience and time consumption of manipulation and input for searching for content desired by a viewer or setting a suitable viewing environment are eliminated. In addition, it recognizes changes in the composition of content viewers without user manipulation and changes content or environment setting values according to the changed environment. to provide

The present invention described above can be implemented as computer-readable codes on a medium in which a program is recorded. The computer-readable medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is also a carrier wave (eg, transmission over the Internet) that is implemented in the form of. In addition, the computer may include the control unit 180 of the terminal. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (18)

a microphone configured to receive voice;
a wireless communication unit communicating with an external device that displays content;
a controller for recognizing a viewer of the content based on the received voice and determining whether the recognized viewer's viewing preference has been learned; and
And a running data unit for configuring information related to the viewer's viewing preference based on the determined result,
The control unit is
When the learned viewing preferences of a plurality of viewers are different, it is determined whether there is a common viewing preference by analyzing the correlation between the learned viewing preferences, and a signal for providing different types of recommended content is generated based on the determination result. do,
A signal for recommending content corresponding to a common viewing preference is provided to the external device according to the analysis result of the correlation, and the common viewing preference is learned based on a response result of the external device;
If there is no common viewing preference as a result of the analysis of the correlation, a leader viewer is identified among the plurality of viewers based on the received audio and video;
and transmitting, to the external device, a signal for simultaneously recommending the first content corresponding to the viewing preference of the reader viewer and the second content according to the viewing priority. The method of claim 1,
The control unit is
output a signal related to a change of the content based on the configured information, and the viewer's viewing preference is learned based on feedback on the output signal,
When the recognized viewer's viewing preference has been previously learned, a signal for providing recommended content corresponding to the learned viewing preference is transmitted to the external device, and guide information related to the recommended content is output. electronic device. delete delete delete The method of claim 1,
The control unit is
When a change in the configuration of the viewer is detected based on the received voice and image while viewing the content, a signal related to the change of the content being viewed is generated based on the changed viewer's viewing preference,
When a plurality of viewers are recognized based on the received audio and video, a signal requesting to provide different recommended content is transmitted to the external device according to whether there are a plurality of viewers whose viewing tastes have been learned. Device. delete delete delete delete delete delete delete delete delete delete delete Connecting to an external device for displaying content;
recognizing a viewer of the content based on a voice received by a microphone;
determining whether the recognized viewer's viewing preference has been learned;
configuring information related to the viewer's viewing preference based on the determined result;
outputting a signal related to a change of the content based on the configured information; and
and receiving feedback on the output signal to learn the viewer's viewing preference,
The step of learning the viewing taste is,
When the learned viewing preferences of a plurality of viewers are different, it is determined whether there is a common viewing preference by analyzing the correlation between the learned viewing preferences, and a signal for providing different types of recommended content is generated based on the determination result. to do; and
providing a signal for recommending content corresponding to a common viewing preference according to the analysis result of the correlation to the external device, and learning the common viewing preference based on a response result of the external device;
If there is no common viewing preference as a result of the analysis of the correlation, a leader viewer is identified among the plurality of viewers based on the received audio and video;
and transmitting, to the external device, a signal for simultaneously recommending the first content corresponding to the viewing preference of the reader viewer and the second content according to the viewing priority.

Images