KR20160111777A - Set-top box, photographing apparatus for providing context-awareness services on the basis of multi-modal information to thereby learn and enhance user interface and user experience and method and computer-readable recording medium using the same - Google Patents

Abstract

According to one embodiment of the present invention, as to a set-top box which personally learns and improves a user interface and a user experience by performing context-awareness function on the basis of multi-modal information, once at least one sensing data are obtained through a communication unit which performs a communication with a broadcasting signal, a built-in sensor, and an external IoT terminal provided to a basic terminal by receiving and processing a broadcasting signal and the built-in sensor or the external IoT terminal, (i) specific context information data showing at least a partial states among figures, objects and environments to which the set-top box pertains are obtained as a result of analyzing at least one obtained sensing data, and (ii) with reference to the database for recording the information about an action item of the basic terminal, which is mapped per type of the context information data, or an action item of the external IoT terminal, provided is a set-top box comprising a processor that supports to perform an action item of the basic terminal or an action item of the external IoT terminal corresponding to the specific context information data. The set-top box according to one embodiment of the present invention provides a user experience that can interact with a user such as a pet by processing multi-modal information, recognizing a situation, and operating various devices in a house in a smart way.

Description

TECHNICAL FIELD [0001] The present invention relates to a set-top box, a photographing apparatus, a method using the same, and a computer readable recording medium for learning and improving a user interface and a user experience by performing a situation recognition function based on multi- USER INTERFACE AND USER EXPERIENCE AND METHOD AND COMPUTER-READABLE RECORDING MEDIUM USING THE SAME < RTI ID = 0.0 >

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a set-top box, a photographing apparatus, a method using the same and a computer using the same, which learn and improve a user interface and a user experience by performing a context-aware function based on multi- (A) acquiring at least one sensed data via an internal sensor or an external IoT terminal, (b) analyzing the obtained at least one sensed data, (C) acquiring specific situation information data indicating a state of at least a part of a person, an object and an environment in an indoor space to which the settop box belongs; (c) With reference to a database in which information on action items of the terminal is recorded, Response will be for the base terminal of an action item, or the external set-top box that can be supported so as to perform the action items in the IoT terminal or performing image pickup apparatus, and a method and computer-readable recording medium using the same.

With the development of IPTV technology, set-top box technology that provides various services such as providing content based on a network is being commercialized. For example, a user can receive contents such as video-on-demand (VOD) and network games through a set-top box, and can use services such as home shopping and Internet surfing.

However, the conventional set-top boxes have only performed a monotonous function of receiving the contents or services provided by the service provider and delivering the user's responses or requests to the service provider.

Recently, IoT (Internet of things) technology which can control objects from the outside by connecting with objects has been attracting attention. In connection with this, I have been interested in smart home technology that can monitor and control various devices in the house through communication network Interest is also rising.

Accordingly, the present inventors analyzed (multi-modal) sensing data obtained through various sensors to identify the context information in the indoor space in which the set-top box is placed, and learn the interface definition between the user and the device by themselves, We have come up with a new set-top box that provides a user experience that continuously improves with behavior that is consistent with contextual information.

SUMMARY OF THE INVENTION The present invention has been made to solve all the problems described above.

In addition, the present invention analyzes context data of an indoor space, identifies context information of a predetermined space, and various devices corresponding thereto learn an operation corresponding to the context information by themselves and continuously perform the improved function Another object of the present invention is to provide a set-top box, a photographing apparatus, a method using the same, and a computer-readable recording medium.

In addition, the present invention provides a user experience in which a set-top box or a photographing device learns and operates an action item corresponding to the situation information to define and continuously improve the interaction with the user, such as a pet, For other purposes.

The present invention also provides a technique of learning the remote control signal for controlling the basic terminal or the external IoT terminal and automatically controlling the operation of the basic terminal or the external IoT terminal according to a specific situation without the user's remote control operation in a specific situation For another purpose.

In order to accomplish the above object, a representative structure of the present invention is as follows.

According to an aspect of the present invention, there is provided a set-top box for learning and improving a user interface and a user experience by performing a context-aware function based on multi-modal information, (I) at least one sensing data is obtained through the built-in sensor or the external IoT terminal, (i) the acquired at least one As the result of analyzing the sensed data of the set top box, specific situation information data indicating at least a part of the person, object, and environment in the indoor space to which the set top box belongs; (ii) And the data item of the action item of the basic terminal or the action item of the external IoT terminal To the bus as a reference, the set-top box including a processor to perform a supporting or performing the base terminal of an action item or action item of the IoT external terminal corresponding to the particular status information data is provided.

According to another aspect of the present invention, there is provided a method using a set-top box for learning and improving a user interface and a user experience by performing a context-aware function based on multi-modal information, the method comprising the steps of: (a) The method comprising the steps of: (a) obtaining at least one sensed data through an IoT terminal; (b) analyzing at least one sensed data obtained from the settop box, And (c) recording information on an action item of the basic terminal or an action item of the external IoT terminal, the set-top box being mapped according to the type of the situation information data, Which is associated with the specific situation information data, And performing or performing an action item of the terminal or the action item of the external IoT terminal.

According to another aspect of the present invention, there is provided a photographing apparatus for learning and improving a user interface and a user experience by performing a context-aware function based on multi-modal information, the apparatus comprising: A communication unit for communicating with an external IoT terminal and at least one sensing data being obtained through the internal sensor or the external IoT terminal, the method comprising the steps of: (i) Acquiring specific situation information data indicating a state of at least a part of a person, an object, and an environment as a result of analyzing the sensing data, (ii) obtaining an action item of the basic terminal mapped to the type of the situation information data, With reference to a database in which information on action items of the specific situation information The image pickup apparatus includes a processor that supports performing the base terminal of an action item or action item of the IoT external terminal corresponding to the data or to perform is provided.

In addition to this, another method for implementing the present invention, a set top box, a photographing apparatus, and a computer readable recording medium for recording a computer program for executing the method are further provided.

The present invention has the following effects.

The present invention analyzes sensing data obtained in an indoor space, identifies context information on which a set-top box is placed, and allows a variety of devices corresponding thereto to learn an operation corresponding to context information by itself and perform a changed function. , A photographing apparatus, a method using the same, and a computer-readable recording medium.

In addition, the present invention has the effect of providing a user experience that continuously learns the interaction with the user, such as a pet, by learning the action item corresponding to the situation information by operating the set-top box or the photographing device.

The present invention also provides a technique of learning a remote control signal for controlling a basic terminal or an external IoT terminal and controlling the operation of a basic terminal or an external IoT terminal with a continuously improved function in accordance with a specific situation without a user's remote control operation in a specific situation .

1 is a conceptual diagram for schematically explaining an interaction between a set-top box and other devices according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a configuration and a control flow of a set-top box according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating an implementation technique of a set-top box according to an embodiment of the present invention.
4 is a view illustrating a concept of acquiring specific situation information data as a result of analyzing an infrared or near infrared ray image obtained by a set-top box according to an embodiment of the present invention or audio analyzed by a microphone.
FIG. 5 is a view illustrating a concept of acquiring specific situation information data by analyzing acquired infrared or near-infrared image or acquired voice information according to an embodiment of the present invention and recognizing characteristics of a person in the indoor space.
FIG. 6 is a view illustrating an embodiment for providing a service according to the characteristics of a person by analyzing the obtained infrared or near-infrared image or acquired voice information to identify characteristics of the person in the indoor space.
7 is a conceptual diagram illustrating an interaction between a set-top box, a mobile terminal, and an external IoT terminal according to another embodiment of the present invention.
8 is a view illustrating an operation of performing an action item displaying a content of a mobile terminal through a basic terminal according to another embodiment of the present invention.
9 is a view showing an embodiment in which an image obtained by a set-top box is provided to a mobile terminal and an action item according to a control signal received from the mobile terminal is performed.
FIG. 10 is a view showing an embodiment of performing an action item with reference to the energy use information of the IoT terminal.
11 is a conceptual diagram for schematically explaining an interaction between a photographing apparatus and other apparatuses according to another embodiment of the present invention.
12 is a block diagram showing a configuration and a control flow of a photographing apparatus according to another embodiment of the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

1 is a conceptual diagram for schematically explaining an interaction between a set-top box and other devices according to an embodiment of the present invention.

Referring to FIG. 1, a set-top box 100 according to an embodiment of the present invention can communicate with other devices via a wired or wireless network. First, the set-top box 100 can communicate with the base terminal 200. The basic terminal 200 may include a display device. In this case, the basic terminal 200 may be a UHD TV (Ultra High Definition Television). Specifically, when the basic terminal 200 includes a display device, the set top box 100 may receive a broadcast signal and provide a video signal to the basic terminal 200 as described later. However, the basic terminal 200 is not limited to the display device.

In addition, the set-top box 100 may communicate with at least one external IoT terminal 300. The external IoT terminal 300 may be any device capable of communicating with the set-top box 100. Here, the external IoT terminal 300 acquires sensing data for an indoor space while being located in a predetermined indoor space (i.e., an indoor space in which the set-top box 100 is present) and provides the sensing data to the set-top box 100, 100) or a device capable of receiving and responding to a control signal from a mobile terminal to be described later. For example, as shown in FIG. 1, in addition to the camera 300-1, the air conditioner 300-2, the speaker 300-3, the fire detector 300-4, and the microphone 300-5, A humidity sensor, an ambient light sensor, a refrigerator, a washing machine, and a gas range. The external IoT terminal 300 continuously provides sensing data to the set-top box 100 in response to a request from the set-top box 100.

The set-top box 100 can acquire specific situation information data as a result of receiving and analyzing the sensing data. Here, the specific situation information means specific context information of the indoor space in which the set-top box is placed. That is, the information indicates the state of at least a part of a person, an object, and an environment in an indoor space to which the set-top box belongs. For example, the specific situation information may include status information about whether a person in the living room to which the set-top box belongs, the viewing channel information, the number information of the viewer, the situation information of the person going out, , Weather information of a person, situation information of a person in a living room cleaning, situation information of a person listening to music, situation information of a person taking a sleep, situation information of a person cooking, situation information of a person exercising, A situation where a person is eating, a situation where a person takes a break, a situation where a person plays a game, a situation where a person studies, a situation where a person surfs the Internet, Top box 100, situation information in which a person is partying, situation information in which a person makes conversation, situation information in which a person approaches the set-top box 100, The situation where a person looks at a specific direction, the situation information when a person goes by or goes in front of the settop box 100, a household appliance such as a washing machine, an air conditioner, etc., Status information, status information of the door being opened or closed, status information of turning on or off the lamp, status information of the gas range being switched on or off, information of occurrence of a fire, and viewing date / time information. May mean a combination of some of them. However, as described later, the set-top box 100 may include at least one built-in sensor. In this case, the set-top box 100 may analyze the sensing data acquired through the built-in sensor to acquire specific situation information data.

The set top box 100 can acquire information about an action item of the basic terminal 200 or an action item of the external IoT terminal 300 that is mapped to specific situation information data by communicating with the database server 400 have. Alternatively, the set-top box 100 may store and refer to a database in its own storage. The database may include link node values of neural networks that are learned and updated and stored from the big data that constitute specific context information. That is, as described later, the set-top box 100 can learn specific situation information data and corresponding action items. As a result of learning, the database can be stored in the form of a neural network. Thus, the database may include a link node value for the mapping of action item to specific situation information data on the neural network.

The set-top box 100 may generate a control command for operating the basic terminal 200 or the external IoT terminal 300 according to the acquired information about the action item and transmit the control command to each terminal.

The set-top box 100 may also communicate with the mobile terminal 500. That is, the set-top box 100 may receive the control signal or the content from the mobile terminal 500 or transmit the sensing data obtained through the external IoT terminal 300 to the mobile terminal 500. In addition, the set-top box 100 may receive identification information such as an address from the mobile terminal 500 and use it to identify the user having the mobile terminal 500. [

Hereinafter, the configuration and operation of the set-top box 100 will be described in more detail.

FIG. 2 is a block diagram illustrating a configuration and a control flow of a set-top box according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating an implementation state of a set-top box according to an embodiment of the present invention.

2, a set-top box 100 according to an embodiment of the present invention includes a broadcast signal processing unit 110, a built-in sensor 120, a communication unit 130, and a processor 140.

The broadcast signal processing unit 110 receives the broadcast signal, processes the broadcast signal, and provides the processed broadcast signal to the basic terminal 200. The broadcast signal processing unit 110 may receive the broadcast signal through the tuner, but may receive the broadcast content from the web server via the Internet when the set-top box 100 provides the IPTV service. The broadcast signal processing unit 110 may appropriately process the received broadcast signal and provide it to the basic terminal 200 through a wired or wireless interface or a network.

The built-in sensor 120 is a sensor built in the set-top box. In one embodiment, the embedded sensor 120 may include an infrared camera or a near infrared camera as an embedded image sensor. Such an infrared camera or near-infrared camera has a merit for analyzing the indoor situation even when the room is dark. When a near-infrared camera is used as an internal image sensor, it can detect near infrared rays signals around 850 nm. Also, the built-in sensor 120 may be implemented by at least one of a CCD sensor (Charge Coupled Device Image Sensor) and a CMOS sensor (Complementary Metal Oxide Semiconductor Sensor).

Also, the built-in sensor 120 may include a microphone as an embedded audio sensor. The microphone can collect the user's voice, animal sounds, and ambient noise. Speech recognition techniques may be applied to the set-top box 100 as shown in FIG. 3 when the microphone collects the user's voice. In this case, the speech recognition module (not shown) or the processor 140, which will be described later, can recognize the collected speech. Alternatively, the collected voice data may be transmitted to an external server (not shown) to receive the voice recognition result.

The communication unit 130 is configured to perform communication with the external IoT terminal 300. In particular, the communication unit 130 can perform communication with the external IoT terminal 300 using a low-power communication technology. For example, as shown in FIG. 3, communication can be performed using a communication protocol of at least some of Z-Wave, Wi-Fi, Bluetooth, and Zigbee. However, it does not preclude the application of other communication technologies.

Also, the communication unit 130 may perform communication with the mobile terminal 500 owned by the user, if necessary. For example, the communication unit 130 may receive a control signal or content from the mobile terminal 500, and may transmit the sensing data obtained through the external IoT terminal 300 to the mobile terminal 500. Also, the communication unit 130 can receive identification information such as an address from the mobile terminal 500. [ Communication with the mobile terminal 500 may also be accomplished in a variety of ways including the communication protocols described above. For example, the communication unit 130 can communicate with the mobile terminal 500 using LTE technology.

The processor 140 may acquire specific situation information data as a result of analyzing the obtained at least one sensed data when at least one sensed data is acquired through the built-in sensor 120 or the external IoT terminal 300 have. At this time, the processor 140 may analyze the acquired at least one sensed data directly or may receive the analyzed result by another device. The at least one sensing data may be analyzed by a Big Data technique. Here, the specific situation information means specific context information in the indoor space in which the set-top box 100 is placed, as described above. That is, the information indicates the state of at least a part of a person, an object, and an environment in an indoor space to which the set-top box 100 belongs.

In addition, the processor 140 refers to a database in which action items of the basic terminal 200 mapped according to the type of the situation information data or action items of the external IoT terminal 300 are recorded, To perform or perform an action item of the basic terminal 200 or an action item of the external IoT terminal 300 corresponding to the information data. As described above, the database may exist separately in the external server and the set-top box 100 may store the database. However, in the case of the situation information data, which is information indicating the state of the person, it may be determined that there is a privacy problem, and the set-top box 100 may refer to its own database.

3, the processor 140 according to an exemplary embodiment of the present invention can operate an application written in Java using a virtual machine code supporting an interpreter and an AOT (Ahead on Time). In the case of Dalvik, a virtual machine similar to the conventional Java virtual machine (JVM), the JIT (Just in Time) support was able to improve the execution performance of the instruction repeatedly. However, It is difficult to maximize the performance of hardware. In addition, a separate memory required for compilation is required, and the program source code can be extracted from the bytecode in which the application is compiled. The processor 140 according to an embodiment of the present invention performs compilation using the AOT method, so that there is no need to perform compilation every time, and the above problems can be solved.

In addition, the processor 140 according to an embodiment of the present invention may be configured such that when an application is installed, such as an ART (Android RunTime), which is an acceleration technology of Java-Dalvik, adopted in the Android operating system, Download the application and load it. Therefore, execution performance is improved as compared with the conventional ART. At this time, the compiled file may be composed of translated code as a natural language used by the operating system. For example, the executable file downloaded by the processor 140 may be composed of C / C ++, which is a natural language, thereby eliminating the need to use the JNI, and thus the application execution speed can be further improved.

On the other hand, there may be a case where the information mapped to the obtained specific situation information data is not found in the database or it is determined that the acquired specific situation information data is not correctly recognized. Since this case can be regarded as a context in which the set top box 100 is experiencing for the first time, the processor 140 analyzes the obtained at least one sensed data to determine whether the set top box 100 is a person, New specific situation information data indicating the state of at least part of the object and the environment can be generated. In addition, the processor 140 may learn a new action item to be applied to the basic terminal 200 or a new action item to be applied to the external IoT terminal 300 in storing new specific situation information data, It can also be saved.

When the information mapped to the acquired specific situation information data is not stored in the database, it can be considered that the situation information and the action item have not yet been learned yet since the set-top box 100 has been installed in the room. For example, if the telephone sound is collected through the microphone while the user is watching the TV, the action item of the basic terminal 200, which is still mapped to specific situation information data, may not be stored in the database. In this case, when the user performs an operation of reducing the volume of the basic terminal 200, the processor 140 recognizes the pattern of the action item and stores the new specific situation information data in the database in correspondence with the new action item for reducing the volume New situation information and action items can be learned. In this way, the set-top box 100 learns new specific situation information data and corresponding new action items.

The built-in sensor 120 may include a sensor for sensing a remote control signal (not shown), or at least one of the external IoT terminals 300 may include a sensor for sensing a remote control signal (not shown). The sensor for detecting the remote control signal can detect the near infrared ray or infrared remote control signal of the indoor space. When at least one new action item information among the basic terminal 200 or the external IoT terminal 300 performed in response to the remote control signal detected through the remote control signal detecting sensor is obtained, And may generate the information data and map the generated new specific situation information data to the at least one new action item information among the basic terminal 200 or the external IoT terminal 300 and store the information in the database.

For example, if the room temperature is 28 degrees, assume that user A turned on the air conditioner through the remote control. If the specific situation information data corresponding to the information that the user A is present at the room temperature of 28 degrees does not exist in the database, the processor 140 acquires the remote control signal that activates the air conditioner through the sensor for detecting the remote control signal And can acquire, as new action item information, action information for operating the air conditioner corresponding to the obtained remote control signal. In this case, the processor 140 generates new specific situation information data indicating that the room temperature is 28 degrees and the user A exists, maps the generated new specific situation information data to the new action item information for operating the air conditioner, Can be stored. If the user A exists in a state where the room temperature is again 28 degrees, the controller can generate and transmit a control signal for operating the air conditioner by referring to the new action item information mapped to the new specific situation information data stored in the database.

Alternatively, in a state corresponding to the specific situation information data, at least one new action item information of the basic terminal 200 or the external IoT terminal 300, which is performed in response to the remote control signal detected through the remote control signal detecting sensor, The processor 140 may update the database by further mapping the specific situation information data to at least one of the new action item information of the primary terminal 200 or the external IoT terminal 300. [

Consider an example in which user B presses the air conditioner via the remote control when the room temperature is 28 degrees. Here, it is assumed that the specific situation information data corresponding to the information that the room temperature is 28 degrees and the user B exists is generated in advance, and the corresponding action item is set to turn on the fan. At this time, assuming that the room temperature is 28 degrees and the user B is present, the processor 140 obtains the remote control signal that activates the air conditioner through the sensor for detecting the remote control signal, The action information corresponding to the air conditioner can be obtained as the new action item information. In this case, the processor 140 updates the database by additionally mapping the specific situation information data to the new action item information for operating the air conditioner, since the specific situation information data indicating that the room temperature is 28 degrees and the user B exists already exists . Herein, the reason for referring to "additional" is that an action item "air conditioning on" can be further mapped in a situation in which an action item already mapped to the specific situation information data (ie, fan on) exists . On the other hand, if the room temperature is again 28 degrees after a few days, the processor 140 can generate and transmit a control signal for operating the air conditioner by referring to the new action item information mapped to the specific situation information data stored in the database. Of course, the fan can be operated at the same time.

The set-top box 100 according to an exemplary embodiment of the present invention may further include a remote controller signal transmitter (not shown) for generating and transmitting a remote controller signal. At this time, when the specific situation information data is obtained after the database is updated as described above, the processor 140 transmits at least one new action item of the basic terminal 200 or the external IoT terminal 300 through the remote control signal transmitting unit A remote control signal may be generated and transmitted or transmitted.

As described above, the set-top box 100 according to the present invention learns a remote control signal for controlling the basic terminal 200 or the external IoT terminal 300 and can automatically learn the remote control signal for controlling the basic terminal 200 or the external IoT terminal 300, So that the operation of the terminal 300 can be controlled.

Meanwhile, the processor 140 determines whether or not the action item of the basic terminal 200 corresponding to the acquired specific situation information data or the action item of the external IoT terminal 300 is performed, The action item of the external IoT terminal 300 or the action item of the external IoT terminal 300 may be learned so that the modified specific action information corresponds to the obtained specific situation information data and stored in the database. The action item of the basic terminal 200 corresponding to the acquired specific situation information data or the action item of the basic terminal 200 performed after the action item of the external IoT terminal 300 is performed a predetermined number of times or more, When the action item of the action information item 300 is modified, the mapping to the specific situation information data may be erroneous or the user may desire to change the action pattern. Therefore, in this case, the processor 140 may learn that the modified specific action information corresponds to the obtained specific situation information data and store the learned action information in the database.

For example, if a telephone sound is being collected through a microphone while the user is watching TV, the database may store an action item that reduces the volume, and accordingly the processor 140 may perform an action item that reduces the volume in this case can do. However, if the user manually turns off the TV three or more times after the action item that automatically reduces the volume has been automatically performed, the user would like to turn off the TV instead of reducing the volume in the same situation. In such a case, the processor 140 may learn the action item that turns off the TV, which is the modified action item, to correspond to the specific situation information data, and store the learned action item in the database.

As described above, the set-top box 100 according to the embodiment of the present invention can operate by learning an action item corresponding to the situation information, and thus can provide a user experience that can interact with a user like a pet.

FIG. 4 is a diagram illustrating a concept of obtaining specific situation information data as a result of analyzing infrared or near infrared rays images taken by a set-top box according to an embodiment of the present invention or analyzing audio obtained by a microphone.

First, the built-in sensor 120 may include an infrared camera or a near-infrared camera as an internal image sensor, or at least one of the external IoT terminals 300 may include an infrared camera or a near infrared camera as an external image sensor. The processor 140 may acquire, as big data, an image continuously captured at a predetermined time interval through the built-in image sensor or the external image sensor. 4, the processor 140 analyzes the at least one infrared or near infrared ray image of the indoor space obtained through the built-in image sensor or the external image sensor as the big data, Data can be acquired.

For example, the processor 140 may analyze at least one infrared or near-infrared image obtained through the built-in image sensor or the external image sensor as big data to obtain specific situation information data indicating that the user is under age have. In this case, for example, the processor 140 may support to perform or to perform an action item notifying the operation status of the basic terminal 200 or the external IoT terminal 300 to another external terminal (for example, a mobile terminal) . Or to perform or perform an action item that restricts the operation of the basic terminal 200 or the external IoT terminal 300 (e.g., restricts viewing of a specific channel of the TV).

In addition, when specific situation information data indicating that the identified person is a baby is obtained as a result of analyzing at least one infrared or near infrared ray image obtained through the built-in image sensor or the external image sensor as big data, For example, at least some of the action items that output a specific sound through an action item and a speaker that display a specific content are turned on (turned on) or set to a specific channel Or to perform.

4, the internal sensor 120 may further include a microphone as an internal audio sensor, or at least one external IoT terminal 300 may include a microphone as an external audio sensor. At this time, the processor 140 may acquire specific situation information data as a result of analyzing the voice information obtained through the built-in audio sensor or the external audio sensor as big data. For example, specific situation information data may be obtained in which the pet dog barks as a result of analyzing the infrared or near-infrared image and / or voice information obtained in FIG.

4, the processor 140 further analyzes the sensing data and other information obtained through the other external IoT terminal 300 or the built-in sensor 120 to obtain specific situation information data You can also get it. That is, the processor 140 receives temperature information obtained through a temperature sensor, which is at least one of the external IoT terminals 300, illuminance information obtained through an illuminance sensor, which is at least one of the external IoT terminals 300, 300, at least one of the external IoT terminal 300 and the external IoT terminal 300, the vibration information obtained through the vibration sensor, which is at least one of the external IoT terminal 300 and the external IoT terminal 300, At least one of fire information, access information obtained through an entrance sensor, at least one of the external IoT terminals 300, energy consumption information obtained from the external IoT terminal 300, TV viewing information, date / time information, To obtain the specific situation information data. As described above, the present invention can acquire specific situation information as a result of acquiring and analyzing the multimodal information.

For example, the processor 140 may analyze at least one infrared or near-infrared image obtained through the built-in image sensor or the external image sensor, access information and date / time information acquired through the access sensor as big data As a result, it is possible to indicate that the user is preparing to go to work or to obtain specific situation information data indicating that the user has left work. At this time, the processor 140 may include an action item for displaying a content by turning on or turning on a display device which is one of the basic terminals 120, an action item for outputting a specific sound through a speaker, May support or perform at least some of the action items that power on / off the lighting device, which is one of the IoT terminals 300. [

Similarly, when specific situation information data indicative of a situation in which a thief has been intruded is acquired, the processor 140 may support or perform at least some of the alert issue action item, the notification message transmission action item, and the crime prevention action item.

In addition, when specific situation information data indicative of a situation in which a fire has occurred is acquired, the processor 140 may assist in performing or performing at least some of the alert action action item, the notification message transmission action item, the fire report action item, and the fire suppression action item .

4, the processor 140 receives the specific situation information data as a result of recognizing only the sound obtained through the built-in microphone, which is one of the built-in sensors 120, or the external microphone, which is one of the external IoT terminals 300, It can also be obtained.

For example, the processor 140 acquires specific situation information data as a result of analyzing the sound acquired through the built-in microphone or the external microphone as the big data, and acquires the specific situation information data and transmits it to the basic terminal 200 or the external IoT terminal 300 ), A volume adjustment operation, a supplementary data provision operation, a PPR information provision operation, a video call operation, and a sound output operation of a speaker.

Of course, as another example, the processor 140 may acquire specific situation information data as a result of further analyzing the sensing data and other information obtained through the other external IoT terminal 300 or the built-in sensor 120 as big data You can do it.

FIG. 5 is a view illustrating a concept of obtaining specific situation information data more specifically by analyzing the obtained infrared or near-infrared image or voice information according to an embodiment of the present invention and recognizing characteristics of a person present in the indoor space .

According to an embodiment of the present invention, the built-in sensor 120 may include an infrared camera or a near-infrared camera as an internal image sensor or a microphone as an internal audio sensor, or at least one of the external IoT terminals 300 may be an infrared camera or a near- The camera may be included as an external image sensor or the microphone may be included as an external audio sensor. At this time, the processor 140 may analyze at least one infrared or near infrared ray image of the indoor space acquired through the built-in image sensor or the external image sensor as big data, as shown in FIG. 5, The voice information obtained through the external audio sensor is analyzed as big data, at least a part of the number of persons, gender, age and behavior existing in the indoor space can be grasped, and specific situation information data can be obtained through this.

FIG. 6 is a view illustrating an embodiment for providing a service according to the characteristics of a person by analyzing the obtained infrared or near-infrared image or acquired voice information to identify characteristics of the person in the indoor space.

Referring to FIG. 6, the processor 140 analyzes at least one infrared or near infrared ray image of the indoor space acquired through the built-in image sensor or the external image sensor as big data to find out that one user exists The sex is female and the age is 30. In addition, when the user takes an action to listen to music (for example, when a specific gesture is taken), it can detect an action item and output an action item for outputting a favorite music of a thirties woman through a speaker. When the power of the base terminal 200 is turned off, the set top box 100 can operate as described above. Of course, as described with reference to FIG. 5, such a service can be provided using an audio sensor, but a detailed description will be omitted.

Meanwhile, the user can be identified by analyzing at least one infrared or near infrared ray image obtained through the built-in image sensor or the external image sensor as big data, and such user identification information can be utilized as specific situation information data. However, since the set-top box 100 can acquire terminal identification information such as an address from the mobile terminal 500 held by the user, it is also possible to identify the user through the terminal identification information. In general, since many computing resources may be consumed in analyzing infrared or near-infrared images, the latter method has an advantage that users can be efficiently identified through terminal identification information.

In addition to receiving the terminal identification information from the mobile terminal 500 owned by the user, the communication unit 130 may perform an interaction with the mobile terminal 500 and the external IoT terminal 300 to provide various services. This embodiment will be described below.

7 is a conceptual diagram illustrating an interaction between a set-top box, a mobile terminal, and an external IoT terminal according to another embodiment of the present invention.

Referring to FIG. 7, when a control signal is input through the authenticated mobile terminal 500, the processor 140 recognizes the context information data corresponding to the control signal, Or to perform or execute an action item of the external IoT terminal 300 or an action item of the external IoT terminal 300. [ Accordingly, the user can easily operate the mobile terminal 500 and control the external IoT terminal 300 such as the lighting of the house, the air conditioner, and the washing machine.

The processor 140 may also send at least one sensed data obtained through the embedded sensor 120 or the external IoT terminal 300 to the authenticated mobile terminal 500 upon request from the authenticated mobile terminal 500. [ Lt; / RTI > For example, the processor 140 may transmit the temperature information of the house obtained through the temperature sensor, which is the external IoT terminal 300, to the authenticated mobile terminal 500, upon request from the authenticated mobile terminal 500 .

8 is a view illustrating an operation of performing an action item displaying a content of a mobile terminal through a basic terminal according to another embodiment of the present invention.

According to another embodiment of the present invention, the specific situation information data may be related to the mobile terminal 500 the user has (e.g., the situation information data corresponding to the pattern shaking the mobile terminal 500, ), The processor 140 displays the video of the content generated through the mobile terminal 500 through the basic terminal 200 or outputs the sound for the content To perform or perform an action item.

For example, referring to FIG. 8, the processor 140 may support performing or executing an action item for displaying an image photographed through the mobile terminal 500 through a display device, which is one of the basic terminals 200. [ Accordingly, the user can transmit the images photographed through the mobile terminal 500 to the remote family members so as to display the images on the TVs watched by the family members. In addition, for example, when an image photographed through the mobile terminal 500 is acquired in a situation where the user is leaving, the processor 140 displays the photographed image through the mobile terminal 500 through the basic terminal 200 To perform or perform an action item. However, when two or more people are in the indoor space, the image may not be displayed in consideration of the privacy problem.

9 is a view showing an embodiment in which an image obtained by a set-top box is provided to a mobile terminal and an action item according to a control signal received from the mobile terminal is performed.

Referring to FIG. 9, the processor 140 may transmit at least one image obtained through the built-in sensor 120 or the external IoT terminal 300 to the authenticated mobile terminal when a request is received from the authenticated mobile terminal. For example, the user can receive an image of a pet dog through the mobile terminal 500 in real time. At this time, the user can input a control signal through the mobile terminal 500. When the control signal is input through the mobile terminal 500, the processor 140 recognizes the context information data corresponding to the control signal and generates an action item of the basic terminal 200 corresponding to the recognized context information Or to perform or perform action items of the external IoT terminal 300. [ In the embodiment shown in FIG. 9, the processor 140 performs an action item for displaying an image of a master favorite dog through the basic terminal 200 according to a control signal. In addition, the processor 140 may perform a video call with a user of the mobile terminal 500 in accordance with a control signal.

FIG. 10 is a view showing an embodiment of performing an action item with reference to the energy use information of the IoT terminal.

Referring to FIG. 10, the communication unit 130 can acquire energy usage information of each of the external IoT terminals 300. The communication unit 130 can receive and manage energy usage information from each of the external IoT terminals 300 consuming electricity or gas at a predetermined time interval or after requesting it. Alternatively, the communication unit 130 may acquire energy usage information from the watt-hour meter or the gas meter, which is one of the external IoT terminals 300, at predetermined time intervals or upon request.

The processor 140 may support to perform or perform an action item of the basic terminal 200 or an action item of the external IoT terminal 300 with reference to the energy usage information of each of the obtained external IoT terminals 300 have. For example, if it is determined that the amount of electricity used in the household is greater than a predetermined value as a result of analyzing the amount of electricity obtained from the watt-hour meter, which is one of the external IoT terminals 300, as the big data, Or to perform or perform an action item that powers off the device. In particular, when the set-top box 100 learns a remote control signal for an electronic device in the home, the processor 140 determines whether the remote control signal transmitting unit of the set-top box 100 or the external IoT terminal 300 A remote control signal for turning off the power supply of the electronic device may be generated and transmitted.

If the energy usage information of at least some of the obtained external IoT terminals 300 corresponds to a predetermined condition, the processor 140 holds information on the use state of the external IoT terminal 300 by the user To the mobile terminal 500. For example, if it is determined that the air conditioner of the house is consuming excessive electricity, the processor 140 may provide this information to the user's mobile terminal 500 through the communication unit 130. The user can transmit a control command to the set top box 100 through the mobile terminal 500, such as turning off the air conditioner or raising the set temperature.

In addition, the processor 140 may further obtain EPG information about a broadcast signal provided to the basic terminal 200. In this case, the processor 140 may further refer to the obtained EPG information, To perform or to perform an action item of the IoT terminal 300. When the EPG information is further referred to, the action item of the basic terminal 200 or the external IoT terminal 300 that meets a specific situation can be performed more precisely. For example, in consideration of an accurate date of a broadcast signal provided to the basic terminal 200 (specific day information in the case of a drama broadcast on a specific day of the week), time information (accurate start time, end time and advertisement time of a specific broadcast) The operation of the base terminal 200 or the external IoT terminal 300 can be controlled.

The processor 140 further performs a function of controlling the flow of data between the broadcasting signal processing unit 110, the built-in sensor 120, and the communication unit 130. That is, the processor 140 controls the flow of data between each component of the set-top box 100 so that each component performs its own function. The processor 140 may be implemented by a hardware configuration such as a micro processing unit (MPU) or a central processing unit (CPU), a cache memory, and a data bus, and a software configuration of an operating system, .

Hereinafter, a photographing apparatus 100 'according to another embodiment of the present invention will be described.

FIG. 11 is a conceptual view for schematically explaining an interaction between a photographing apparatus and other apparatuses according to another embodiment of the present invention, and FIG. 12 is a view illustrating a configuration and a control flow of the photographing apparatus according to another embodiment of the present invention Block diagram.

For reference, the photographing apparatus 100 according to another embodiment of the present invention may refer to a closed circuit television (CCTV) or the like, but is not limited thereto.

Referring to FIG. 11, the photographing apparatus 100 'according to another embodiment of the present invention can communicate with other apparatuses via a wired or wireless network, like the set-top box 100 described above. The photographing apparatus 100 'may photograph a predetermined space and provide a video signal to a basic terminal (not shown). The base terminal can receive and display a video signal. Here, the predetermined space may be either an indoor space or an outdoor space. For example, the photographing apparatus 100 'may be installed inside the home or outside the home.

In addition, the photographing apparatus 100 'can communicate with at least one external IoT terminal 300 in the same manner as the set-top box 100 described above. The external IoT terminal 300 continuously provides the sensing data to the photographing apparatus 100 'at the request of the photographing apparatus 100'. The photographing apparatus 100 'can acquire specific situation information data as a result of receiving and analyzing the sensing data. In particular, the photographing apparatus 100 'can acquire specific situation information data as a result of analyzing the multimodal sensing data obtained from a plurality of external IoT terminals 300. [ The photographing apparatus 100 'may also communicate with the database server 400 to obtain information about an action item of the basic terminal or an action item of the external IoT terminal 300 that is mapped to specific situation information data have. Of course, the photographing apparatus 100 'may store and refer to a database in its own storage unit. The photographing apparatus 100 'may generate a control command for operating the basic terminal or the external IoT terminal 300 according to the obtained information about the action item and transmit the control command to each terminal.

In addition, the photographing apparatus 100 'may perform communication with the mobile terminal 500. That is, the photographing apparatus 100 'may receive the control signal or the contents from the mobile terminal 500 or transmit the sensing data obtained through the external IoT terminal 300 to the mobile terminal 500. In addition, the photographing apparatus 100 'may receive identification information such as an address from the mobile terminal 500 and use it to identify the user having the mobile terminal 500. [

Referring to FIG. 12, a photographing apparatus 100 'according to another embodiment of the present invention includes an image signal providing unit 150, a built-in sensor 120, a communication unit 130, and a processor 140.

The video signal providing unit 150 captures a predetermined space and provides a video signal to the base terminal. When the photographing apparatus 100 'is implemented by CCTV, the photographing apparatus 100' can provide a video signal obtained by photographing a predetermined space to a display apparatus in a control center or other building.

The processor 140 may determine that at least one sensing data is obtained through the built-in sensor 120 or the external IoT terminal 300 including at least the embedded image sensor and that (i) the result of analyzing the obtained at least one sensing data (Ii) obtaining an action item of the basic terminal mapped to the type of the context information data or an action item of the action item of the external IoT terminal 300 It is possible to support or execute an action item of the basic terminal corresponding to the specific situation information data or an action item of the external IoT terminal 300 with reference to the database recording the information.

The detailed operation of the processor 140, the built-in sensor 120, and the communication unit 130 is the same as that of the set-top box 100 described above,

As described above, the photographing apparatus 100 'according to another embodiment of the present invention analyzes the sensed data obtained in a predetermined space, identifies the context information on which the photographing apparatus 100' is placed, Can perform an operation corresponding to the context information.

In addition, the embodiments of the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specially designed and constructed for the present invention or may be those known and used by those skilled in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims.

Therefore, the spirit of the present invention should not be construed as being limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the following claims, I will say.

100: Set-top box
110: Broadcast signal processor
120: Built-in sensor
130:
140: Processor
200: basic terminal
300: External IoT terminal
400: database server
500: mobile terminal
100 ': photographing device
150: Video signal demultiplexer

Claims (48)

A set-top box for learning and improving a user interface and a user experience by performing a context-aware function based on multi-modal information,
A broadcast signal processor for receiving and processing a broadcast signal and providing the broadcast signal to the base terminal,
Built-in sensor,
A communication unit for performing communication with the external IoT terminal, and
(I) analyzing the acquired at least one sensed data to determine whether the at least one sensed data is present in the person, object, or environment in the interior space of the settop box Acquiring specific situation information data indicative of at least a part of the state, (ii) referring to a database which records information on action items of the basic terminal mapped to the types of the situation information data or action items of the external IoT terminal And performing or executing an action item of the primary terminal corresponding to the specific situation information data or an action item of the external IoT terminal. The method according to claim 1,
The processor comprising:
If the information mapped to the obtained specific situation information data is not in the database or it is determined that the mapping to the acquired specific situation information data is incorrect, Generates new specific situation information data indicating a state of at least a part of a person, an object and an environment in the indoor space to which it belongs, and corresponds to a new action item applied to the basic terminal or a new action item applied to the external IoT terminal And stores it in the database. 3. The method of claim 2,
Wherein the built-in sensor includes a sensor for sensing a remote control signal, or at least one of the external IoT terminals includes a sensor for sensing a remote control signal,
When at least one new action item information among the basic terminal or the external IoT terminal performed corresponding to the remote control signal detected through the remote control signal detecting sensor is obtained,
The processor comprising:
Generates the new specific situation information data and maps the generated new specific situation information data to at least one new action item information of the basic terminal or the external IoT terminal and stores the new specific situation information data in the database. The method according to claim 1,
The processor comprising:
The action item of the basic terminal corresponding to the acquired specific situation information data or the action item of the external terminal device corresponding to the acquired specific situation information data or after the action item of the external terminal device is performed a predetermined number of times or more, The modified action item is stored in correspondence with the obtained specific situation information data. The method according to claim 1,
Wherein the built-in sensor includes an infrared camera or a near-infrared camera as an internal image sensor, or at least one of the external IoT terminals includes an infrared camera or a near-infrared camera as an external image sensor,
The processor comprising:
And acquires the specific situation information data as a result of analyzing at least one infrared or near infrared ray image of the indoor space obtained through the built-in image sensor or the external image sensor. 6. The method of claim 5,
Wherein the internal sensor further comprises a microphone as an internal audio sensor, or at least one of the external IoT terminals includes a microphone as an external audio sensor,
The processor comprising:
Wherein the specific situation information data is acquired as a result of further analyzing the voice information obtained through the built-in audio sensor or the external audio sensor. 6. The method of claim 5,
The processor comprising:
Temperature information obtained through a temperature sensor, which is at least one of the external IoT terminals, illumination information obtained through an ambient light sensor, which is at least one of the external IoT terminals, vibration information obtained through a vibration sensor, The at least one of the position information obtained through the position sensor, at least one of the external IoT terminals, the energy consumption amount obtained from the external IoT terminal, the TV viewing information, the date / time information and the weather information, Lt; / RTI > The method according to claim 1,
Wherein the internal sensor includes an infrared camera or a near infrared camera as an internal image sensor or a microphone as an internal audio sensor, or at least one of the external IoT terminals includes an infrared camera or a near infrared camera as an external image sensor, Lt; / RTI >
The processor comprising:
Analyzing at least one infrared or near infrared ray image of the indoor space obtained through the built-in image sensor or the external image sensor, analyzing the voice information obtained through the built-in audio sensor or the external audio sensor, Determining at least a part of the number of existing persons, sex, age, and behavior, and acquiring the specific situation information data. The method according to claim 1,
Wherein,
Further performing communication with a mobile terminal that the user has,
The processor comprising:
When the control signal is inputted through the mobile terminal, the control unit recognizes the context information data corresponding to the control signal, performs an action item of the basic terminal corresponding to the recognized status information data or an action item of the external IoT terminal Wherein the set-top box comprises a set-top box. The method according to claim 1,
If the specific situation information data is about a mobile terminal that the user has,
The processor comprising:
And to perform or perform an action item for displaying an image of the content generated through the mobile terminal through the basic terminal or outputting a sound for the content. The method according to claim 1,
The processor comprising:
And transmits at least one sensed data obtained through the built-in sensor or the external IoT terminal to the authenticated mobile terminal when a request is received from the authenticated mobile terminal. The method according to claim 1,
Wherein,
Further obtaining energy usage information of each of the external IoT terminals,
The processor comprising:
And supports or executes an action item of the basic terminal or an action item of the external IoT terminal with reference to the energy usage information of each of the obtained external IoT terminals. 13. The method of claim 12,
If at least a part of the energy usage information of each of the acquired external IoT terminals corresponds to a predetermined condition,
The processor comprising:
And provides the information on the use state of the corresponding external IoT terminal to the mobile terminal that the user has. The method according to claim 1,
The processor comprising:
And acquires the specific situation information data as a result of recognizing the sound obtained through the built-in microphone, which is one of the built-in sensors, or the external microphone, which is one of the external IoT terminals. The method according to claim 1,
The specific situation information includes,
Top box, status information on whether a person in the living room to which the set-top box belongs is watching TV, viewing channel information, person number information, situation information on a person going out, person's room information, A situation in which a person performs cleaning, a situation in which a person listens to music, a situation in which a person takes sleep, information on a situation in which a person cooks, information on a situation in which a person performs exercise, , Situation information in which a person takes a break, information on a situation in which a person plays a game, information on a situation in which a person is studying, information on situations in which a person surfs the Internet, information on situations in which a person performs a TV shopping, Top box, situation information in which a person comes close to the set-top box, situation information in which a person goes away from the set-top box, Status information of a person looking at a specific direction, situation information of a person passing or going in front of the set-top box, situation information of a household appliance such as a washing machine and an air conditioner, The information on the closed state, the information on the state in which the lamp is turned on or off, the information on the state in which the gas range is turned on or off, the information on the occurrence of the fire, and the viewing date / time information, or a combination of at least some of them. box. The method according to claim 1,
Wherein the built-in sensor includes a sensor for sensing a remote control signal, or at least one of the external IoT terminals includes a sensor for sensing a remote control signal,
When at least one new action item information among the basic terminal or the external IoT terminal performed in response to the remote control signal detected through the remote control signal detecting sensor is acquired in a state corresponding to the specific situation information data,
The processor comprising:
And updates the database by further mapping the specific situation information data to at least one new action item information of the basic terminal or the external IoT terminal. 17. The method of claim 16,
Further comprising a remote control signal transmitting unit for generating and transmitting a remote control signal,
The processor comprising:
The remote control signal transmitting unit generates a remote control signal for performing at least one new action item out of the basic terminal or the external IoT terminal and transmits or transmits the remote control signal through the remote control signal transmitting unit Supporting a set-top box. The method according to claim 1,
The processor comprising:
And operates using a virtual machine code supporting an interpreter and an AOT (Ahead on Time). 19. The method of claim 18,
The processor comprising:
And downloading and executing the precompiled application from the server. The method according to claim 1,
The processor comprising:
Acquires EPG information on a broadcast signal provided to the basic terminal,
And supports or executes an action item of the basic terminal or an action item of the external IoT terminal by further referring to the obtained EPG information. A method using a set top box that learns and improves a user interface and a user experience by performing a context aware function based on multi-modal information,
(a) acquiring at least one sensing data via the built-in sensor or the external IoT terminal,
(b) obtaining, as a result of analyzing the obtained at least one sensed data, specific situation information data indicating a state of at least a part of a person, an object and an environment in an indoor space to which the set top box belongs , And
(c) the set-top box corresponds to the specific situation information data with reference to a database in which an action item of a basic terminal mapped according to the type of the situation information data or an action item of the external IoT terminal is recorded To perform or perform an action item of the basic terminal or an action item of the external IoT terminal. 22. The method of claim 21,
The step (c)
If it is determined that the information mapped to the specific situation information data is not in the database or that the mapping of the obtained specific situation information data is incorrect, the set top box analyzes the obtained at least one sensed data And generates new specific situation information data indicating a state of at least one of a person, an object and an environment in the indoor space to which the set-top box belongs, and generates new specific situation information data indicating a new action item applied to the primary terminal, And storing in the database corresponding to the action item. 23. The method of claim 22,
Wherein the built-in sensor includes a sensor for sensing a remote control signal, or at least one of the external IoT terminals includes a sensor for sensing a remote control signal,
The step (c)
When the new action item information of at least one of the primary terminal or the external IoT terminal performed in response to the remote control signal sensed by the remote control signal detecting sensor is acquired, generates the new specific situation information data, The specific situation information data is mapped to at least one new action item information of the basic terminal or the external IoT terminal and is stored in the database. 22. The method of claim 21,
(d) when the set-top box receives an action item of the basic terminal corresponding to the acquired specific situation information data or an action item of the basic terminal performed over a predetermined number of times after the action item of the external IoT terminal is performed, Further comprising the step of storing the modified action item in association with the obtained specific situation information data when the action item of the IoT terminal is modified. 22. The method of claim 21,
Wherein the built-in sensor includes an infrared camera or a near-infrared camera as an internal image sensor, or at least one of the external IoT terminals includes an infrared camera or a near-infrared camera as an external image sensor,
The step (b)
Wherein the set top box obtains the specific situation information data as a result of analyzing at least one infrared or near infrared ray image of the indoor space acquired through the built-in image sensor or the external image sensor. 26. The method of claim 25,
Wherein the internal sensor further comprises a microphone as an internal audio sensor, or at least one of the external IoT terminals includes a microphone as an external audio sensor,
The step (b)
Wherein the set top box obtains the specific situation information data as a result of further analyzing the voice information obtained through the built-in audio sensor or the external audio sensor. 26. The method of claim 25,
The step (b)
Wherein the set-top box comprises at least one of an external IoT terminal, temperature information obtained through a temperature sensor, at least one of external IoT terminals, illuminance information obtained through an illuminance sensor, which is at least one of the external IoT terminals, Referring to at least a part of the obtained vibration information, position information obtained through a position sensor, which is at least one of the external IoT terminals, energy consumption obtained from the external IoT terminal, TV viewing information, date / time information, And acquiring the situation information data. 22. The method of claim 21,
Wherein the internal sensor includes an infrared camera or a near infrared camera as an internal image sensor or a microphone as an internal audio sensor, or at least one of the external IoT terminals includes an infrared camera or a near infrared camera as an external image sensor, Lt; / RTI >
The step (b)
At least one infrared or near infrared ray image of the indoor space obtained through the built-in image sensor or the external image sensor is analyzed or the audio information obtained through the built-in audio sensor or the external audio sensor is analyzed, Wherein at least some of the number of persons present, gender, age, and behavior are identified and the specific situation information data is thereby obtained. 22. The method of claim 21,
(e) when the set-top box receives a control signal through the mobile terminal, recognizes the context information data corresponding to the control signal and transmits the action item of the basic terminal corresponding to the recognized context information or the external IoT Further comprising supporting the terminal to perform or perform an action item of the terminal. 22. The method of claim 21,
If the specific situation information data is about a mobile terminal that the user has,
The step (c)
Wherein the set-top box supports to perform or perform an action item for displaying an image of the content generated through the mobile terminal through the basic terminal or outputting a sound for the content. 22. The method of claim 21,
(f) transmitting the at least one sensed data acquired through the built-in sensor or the external IoT terminal to the authenticated mobile terminal when the set-top box is requested by the authenticated mobile terminal Lt; / RTI > 22. The method of claim 21,
The step (c)
When the energy use information of each of the external IoT terminals is further obtained, the processor refers to the energy use information of each of the obtained external IoT terminals and determines the action item of the basic terminal or the action item of the external IoT terminal To perform or to perform the method. 33. The method of claim 32,
The step (c)
If the energy usage information of at least some of the obtained external IoT terminals corresponds to a predetermined condition, the set top box provides information on the use state of the external IoT terminal to the mobile terminal that the user has ≪ / RTI > 22. The method of claim 21,
The step (b)
Wherein the set-top box acquires the specific situation information data as a result of recognizing the sound obtained through the built-in microphone, which is one of the built-in sensors, or the external microphone, which is one of the external IoT terminals. 22. The method of claim 21,
The specific situation information includes,
Top box, status information on whether a person in the living room to which the set-top box belongs is watching TV, viewing channel information, person number information, situation information on a person going out, person's room information, A situation in which a person performs cleaning, a situation in which a person listens to music, a situation in which a person takes sleep, information on a situation in which a person cooks, information on a situation in which a person performs exercise, , Situation information in which a person takes a break, information on a situation in which a person plays a game, information on a situation in which a person is studying, information on situations in which a person surfs the Internet, information on situations in which a person performs a TV shopping, Top box, situation information in which a person comes close to the set-top box, situation information in which a person goes away from the set-top box, Status information of a person looking at a specific direction, situation information of a person passing or going in front of the set-top box, situation information of a household appliance such as a washing machine and an air conditioner, A closing date and time information, a closed status information, a status information of turning on or off a light, a status information of a gas range being turned on or off, a fire occurrence status information and a viewing date / time information or a combination of at least some of them . 22. The method of claim 21,
Wherein the built-in sensor includes a sensor for sensing a remote control signal, or at least one of the external IoT terminals includes a sensor for sensing a remote control signal,
(g) when at least one new action item information among the basic terminal or the external IoT terminal performed corresponding to the remote control signal detected through the remote control signal detecting sensor is acquired in a state corresponding to the specific situation information data And further updating the database by further mapping the specific situation information data to at least one new action item information of the primary terminal or the external IoT terminal. 37. The method of claim 36,
(h) After the database is updated, when the specific situation information data is obtained, a remote control signal for executing at least one new action item of the basic terminal or the external IoT terminal is generated and transmitted through the remote control signal transmitting unit The method comprising the steps of: 22. The method of claim 21,
(c0) acquiring EPG information on a broadcast signal provided to the base station,
The step (c)
And performing or executing an action item of the basic terminal or an action item of the external IoT terminal with reference to the acquired EPG information. 39. A computer-readable recording medium having recorded thereon a computer program for executing the method according to any one of claims 21 to 38. 40. The method of claim 39,
The recording medium includes:
And a link node value of a neural network that is learned and updated and stored from the big data constituting the specific situation information. 1. A photographing apparatus for learning and improving a user interface and a user experience by performing a situation recognition function based on multi-modal information,
A video signal providing unit for providing a video signal of a predetermined space to a base terminal,
A built-in sensor that includes at least an embedded image sensor,
A communication unit for performing communication with the external IoT terminal, and
Wherein at least one sensing data is obtained through the built-in sensor or the external IoT terminal, the method comprising: (i) analyzing at least one sensed data to obtain specific situation information indicating at least a part of the person, (Ii) a database storing action item of the basic terminal mapped to the type of the situation information data or action item of the external IoT terminal, And a processor for supporting or performing the corresponding action items of the basic terminal or the action items of the external IoT terminal. 42. The method of claim 41,
The processor comprising:
When the information mapped to the obtained specific situation information data is not in the database or it is determined that the mapping to the obtained specific situation information data is incorrect, Generates new specific situation information data indicating at least a part of the environment and stores the new action item in the database in correspondence with a new action item to be applied to the primary terminal or a new action item to be applied to the external IoT terminal Shooting device. 42. The method of claim 41,
The processor comprising:
The action item of the basic terminal corresponding to the acquired specific situation information data or the action item of the external terminal device corresponding to the acquired specific situation information data or after the action item of the external terminal device is performed a predetermined number of times or more, The modified action item is stored in association with the acquired specific situation information data. 42. The method of claim 41,
Wherein the built-in sensor includes an infrared camera or a near-infrared camera as the built-in image sensor, or at least one of the external IoT terminals includes an infrared camera or a near infrared camera as an external image sensor,
The processor comprising:
And acquires the specific situation information data as a result of analyzing at least one infrared or near infrared ray image obtained through the built-in image sensor or the external image sensor. 45. The method of claim 44,
Wherein the internal sensor further comprises a microphone as an internal audio sensor, or at least one of the external IoT terminals includes a microphone as an external audio sensor,
The processor comprising:
And acquires the specific situation information data as a result of further analyzing the audio information acquired through the built-in audio sensor or the external audio sensor. 42. The method of claim 41,
Wherein the internal sensor includes an infrared camera or a near infrared camera as an internal image sensor or a microphone as an internal audio sensor, or at least one of the external IoT terminals includes an infrared camera or a near infrared camera as an external image sensor, Lt; / RTI >
The processor comprising:
Analyzing the at least one infrared or near infrared ray image obtained through the built-in image sensor or the external image sensor, analyzing the audio information obtained through the built-in audio sensor or the external audio sensor, Number, sex, age, and behavior of the user, and acquires the specific situation information data through the identification. 42. The method of claim 41,
Wherein,
Further performing communication with a mobile terminal that the user has,
The processor comprising:
When the control signal is inputted through the mobile terminal, the control unit recognizes the context information data corresponding to the control signal, performs an action item of the basic terminal corresponding to the recognized status information data or an action item of the external IoT terminal To be performed. 42. The method of claim 41,
The processor comprising:
And transmits at least one sensed data acquired through the built-in sensor or the external IoT terminal to the authenticated mobile terminal when a request is received from the authenticated mobile terminal.

Images