WO2016148398A1 - Boîtier décodeur et appareil de photographie pour exécuter une fonction sensible au contexte à base d'informations multi-modales pour apprentissage et amélioration automatiques d'interface utilisateur et d'expérience d'utilisateur - Google Patents

Boîtier décodeur et appareil de photographie pour exécuter une fonction sensible au contexte à base d'informations multi-modales pour apprentissage et amélioration automatiques d'interface utilisateur et d'expérience d'utilisateur Download PDF

Info

Publication number
WO2016148398A1
WO2016148398A1 PCT/KR2016/001332 KR2016001332W WO2016148398A1 WO 2016148398 A1 WO2016148398 A1 WO 2016148398A1 KR 2016001332 W KR2016001332 W KR 2016001332W WO 2016148398 A1 WO2016148398 A1 WO 2016148398A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
terminal
sensor
action item
external
Prior art date
Application number
PCT/KR2016/001332
Other languages
English (en)
Korean (ko)
Inventor
안태효
지대훈
이윤
Original Assignee
주식회사 스마트올웨이즈온
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 주식회사 스마트올웨이즈온 filed Critical 주식회사 스마트올웨이즈온
Publication of WO2016148398A1 publication Critical patent/WO2016148398A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/44—Receiver circuitry for the reception of television signals according to analogue transmission standards
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]

Definitions

  • the present invention provides a set-top box, a photographing apparatus, and a method and computer using the same, which perform a context-aware function based on MULTI-MODAL information to learn and improve a user interface and a user experience.
  • a readable recording medium more specifically, (a) obtaining at least one sensing data through an internal sensor or an external IoT terminal, and (b) analyzing the obtained at least one sensing data as a result of the analysis.
  • Acquire specific context information data representing a state of at least a part of a person, an object, and an environment in an indoor space to which the set-top box belongs, and (c) the action item or the external IoT of the basic terminal mapped by the type of context information data.
  • the specific situation information data is referred to with reference to a database that records information on action items of the terminal.
  • Response will be for the base terminal of an action item, or the external set-top box that can be supported so as to perform the action items in the IoT terminal or performing image pickup apparatus, and a method and computer-readable recording medium using the same.
  • set-top box technology that provides various services such as providing contents based on a network has been commercialized.
  • a user may be provided with content such as video on demand (VOD) or a network game through a set top box, and may use services such as home shopping and internet surfing.
  • VOD video on demand
  • a network game through a set top box, and may use services such as home shopping and internet surfing.
  • the set-top box has only performed a monotonous function of receiving content or service provided by a service provider and delivering a user's response or request to the service provider.
  • IoT Internet of Things
  • the present inventors analyze the sensing data (multi-modal) acquired through various sensors to identify the context information in the indoor space where the set-top box is located, and learn the interface definition between the user and the device by themselves.
  • a new concept set-top box has been developed that provides a user experience that continually improves with context-sensitive behavior.
  • the present invention aims to solve all the above-mentioned problems.
  • the present invention analyzes the sensing data of the indoor space to identify the context information (context information) of the predetermined space, and the corresponding various devices can learn the operation that fits the context information by themselves and continuously perform the improved function. It is another object of the present invention to provide a new concept set top box, a photographing apparatus, a method using the same, and a computer readable recording medium.
  • the present invention provides a user experience that the set-top box or the photographing device can learn and operate the action items corresponding to the situation information to define and continuously improve the interaction with the user like a pet. For other purposes.
  • the present invention is to provide a technology for automatically controlling the operation of the basic terminal or the external IoT terminal in accordance with a specific situation without learning the remote control of the user in a specific situation by learning the remote control signal for the control of the basic terminal or external IoT terminal. Another purpose.
  • a broadcast signal may be received and processed.
  • a broadcast signal processor provided to a terminal, a built-in sensor, a communication unit performing communication with an external IoT terminal, and the built-in sensor or the external IoT terminal, (i) the at least one acquired Acquiring specific context information data representing a state of at least a part of a person, an object, and an environment in the indoor space to which the set-top box belongs as a result of analyzing sensing data of (ii) the (m) mapped to each type of context information data.
  • a database that records information about an action item of a basic terminal or an action item of the external IoT terminal.
  • the set-top box including a processor to perform a supporting or performing the base terminal of an action item or action item of the IoT external terminal corresponding to the particular status information data is provided.
  • a method using a set-top box for learning and improving a user interface and a user experience by performing a context-aware function based on multi-modal information (a) the set-top box is built-in sensor or external Acquiring at least one sensing data through an IoT terminal; (b) the set-top box analyzes the acquired at least one sensing data, and among the people, objects, and environments in the indoor space to which the set-top box belongs.
  • a photographing apparatus for performing a situational awareness function based on multi-modal information to learn and improve a user interface and a user experience by providing a video signal photographing a predetermined space to a basic terminal.
  • a sensing data is obtained through an image signal providing unit, a built-in sensor, a communication unit performing communication with an external IoT terminal, and the built-in sensor or the external IoT terminal, (i) the obtained at least one
  • specific context information data representing at least some of the state of the person, the object, and the environment is obtained, and (ii) the action item of the basic terminal or the external IoT terminal mapped to each type of context information data.
  • the specific situation information with reference to a database that records information on an action item of
  • the image pickup apparatus includes a processor that supports performing the base terminal of an action item or action item of the IoT external terminal corresponding to the data or to perform is provided.
  • a computer readable recording medium for recording another method, a set top box, a photographing apparatus, and a computer program for executing the method for implementing the present invention.
  • the present invention analyzes the sensing data acquired in the indoor space to identify the context information on which the set-top box is placed, and sets various boxes corresponding to the set-top box to learn the behaviors corresponding to the context information and to perform the changed functions. It is effective to provide a photographing apparatus, a method using the same, and a computer-readable recording medium.
  • the present invention has the effect that the set-top box or the photographing device learns and operates the action item corresponding to the situation information, and learns the interaction with the user as a pet and provides a user experience that is continuously improved.
  • the present invention is a technology for controlling the operation of the base terminal or the external IoT terminal with a function that is continuously improved to a specific situation without learning the remote control of the user in a specific situation by learning the remote control signal for the control of the base terminal or external IoT terminal Has the effect of providing.
  • FIG. 1 is a conceptual diagram schematically illustrating an interaction between a set-top box and other devices according to an embodiment of the present invention.
  • FIG. 2 is a block diagram showing a configuration and control flow of a set-top box according to an embodiment of the present invention
  • Figure 3 is a view showing an implementation technology of a set-top box according to an embodiment of the present invention.
  • FIG. 4 is a diagram illustrating a concept of acquiring specific contextual information data as a result of analyzing an infrared or near infrared image obtained by a set-top box according to an embodiment of the present invention or analyzing audio acquired by a microphone.
  • FIG. 5 is a diagram illustrating a concept of acquiring specific context information data by analyzing characteristics of a person present in an indoor space by analyzing acquired infrared or near infrared images or acquired voice information according to an embodiment of the present invention.
  • FIG. 6 is a diagram illustrating an embodiment of providing a service suitable for a characteristic of a person by analyzing the acquired infrared or near infrared image or the acquired voice information to grasp the characteristic of the person present in the indoor space.
  • FIG. 7 is a conceptual diagram illustrating an interaction between a set top box, a mobile terminal and an external IoT terminal according to another embodiment of the present invention.
  • FIG. 8 is a diagram illustrating an operation of performing an action item displaying content of a mobile terminal through a base terminal according to another embodiment of the present invention.
  • FIG. 9 is a diagram illustrating an embodiment of providing an image acquired by a set top box to a mobile terminal and performing an action item according to a control signal received from the mobile terminal.
  • FIG. 10 is a diagram illustrating an embodiment of performing an action item with reference to energy use information of an IoT terminal.
  • FIG. 11 is a conceptual diagram schematically illustrating an interaction between a photographing apparatus and another apparatus according to another exemplary embodiment of the present invention.
  • FIG. 12 is a block diagram illustrating a configuration and a control flow of a photographing apparatus according to another embodiment of the present invention.
  • FIG. 1 is a conceptual diagram schematically illustrating an interaction between a set-top box and other devices according to an embodiment of the present invention.
  • the set top box 100 may communicate with other devices through a wired or wireless network.
  • the set top box 100 may communicate with the basic terminal 200.
  • the basic terminal 200 may include a display device.
  • the basic terminal 200 may be, for example, a UHD TV (Ultra High Definition Television).
  • the set top box 100 may receive a broadcast signal and provide an image signal to the basic terminal 200 as described below.
  • the basic terminal 200 is not limited to the display device.
  • the set top box 100 may communicate with at least one external IoT terminal 300.
  • the external IoT terminal 300 may be any device that can communicate with the set-top box 100.
  • the external IoT terminal 300 is located in a predetermined indoor space (that is, the indoor space in which the set-top box 100 exists) while acquiring sensing data for the indoor space and providing the sensing data to the set-top box 100 or set-top box ( 100) or a device capable of receiving and responding to a control signal from a mobile terminal to be described later.
  • the air conditioner 300-2, the speaker 300-3, the fire detector 300-4, the microphone 300-5 can be a variety of devices.
  • Humidity sensor, illuminance sensor, refrigerator, washing machine, gas range can be a variety of devices.
  • the external IoT terminal 300 provides sensing data to the set top box 100 continuously or at the request of the set top box 100.
  • the set top box 100 may obtain specific context information data as a result of receiving and analyzing the sensing data.
  • the specific context information refers to specific context information of the indoor space where the set top box is placed. That is, information representing the state of at least a part of a person, an object, and an environment in an indoor space to which the set-top box belongs.
  • certain contextual information includes contextual information about whether a person in the living room to which the set-top box belongs is watching TV, viewing channel information, number of viewers, situational information of a person going out, and information about a person's room.
  • Weather information of a person situation information of a person in the living room cleaning, situation information of a person listening to music, situation information of a person sleeping, situation information of a person cooking, situation information of a person exercising, Situation information when a person eats, situation information where a person takes a break, situation information where a person plays a game, situation information where a person studies, situation information where a person surfs the Internet, situation information where a person is doing TV home shopping,
  • Losing situation information situation information that a person is stationary, situation information that a person looks at a specific direction, situation information that a person passes or goes back and forth in front of the set-top box 100, a washing machine, and an air conditioner such as an air conditioner It may mean each of the situation information, the situation information that the door is opened or closed, the situation information that
  • the set top box 100 may communicate with the database server 400 to obtain information about an action item of the basic terminal 200 or an action item of the external IoT terminal 300 mapped to specific context information data. have.
  • the set top box 100 may store the database in its own storage and refer to it.
  • the database may include link node values of neural networks that are learned, updated, and stored from big data constituting specific context information. That is, as described below, the set-top box 100 may learn specific context information data and action items corresponding thereto, and as a learning result, the database may be stored in the form of a neural network.
  • the database may include link node values for mapping of specific contextual information data and action items on the neural network.
  • the set top box 100 may generate a control command for operating the basic terminal 200 or the external IoT terminal 300 according to the obtained information on the action item, and transmit the generated control command to each terminal.
  • the set top box 100 may communicate with the mobile terminal 500. That is, the set top box 100 may receive a control signal or content from the mobile terminal 500 or transmit sensing data acquired through the external IoT terminal 300 to the mobile terminal 500. In addition, the set-top box 100 may receive identification information such as an address from the mobile terminal 500 and may be used to identify a user having the mobile terminal 500.
  • FIG. 2 is a block diagram showing the configuration and control flow of a set-top box according to an embodiment of the present invention
  • Figure 3 is a view showing an implementation state of a set-top box according to an embodiment of the present invention.
  • the set top box 100 includes a broadcast signal processor 110, a built-in sensor 120, a communicator 130, and a processor 140.
  • the broadcast signal processor 110 is configured to receive and process a broadcast signal and provide the same to the basic terminal 200.
  • the broadcast signal processor 110 may receive a broadcast signal through a tuner, but when the set-top box 100 provides an IPTV service, the broadcast signal processor 110 may receive broadcast content from a web server through the Internet.
  • the broadcast signal processor 110 may properly process the received broadcast signal and provide it to the basic terminal 200 through a wired / wireless interface or a network.
  • the built-in sensor 120 is a sensor built in the set top box.
  • the built-in sensor 120 may include an infrared camera or a near infrared camera as a built-in image sensor.
  • the use of an infrared camera or near-infrared camera has an advantage in analyzing the indoor situation even in a dark room.
  • a near-infrared camera as a built-in image sensor, it can detect near-infrared signals around 850nm.
  • the embedded sensor 120 may be implemented as at least one of a CCD (Charge Coupled Device Image Sensor) or a CMOS Sensor (Complementary Metal Oxide Semiconductor Sensor).
  • the built-in sensor 120 may include a microphone as a built-in audio sensor.
  • the microphone can collect the user's voice, animal sounds and ambient noise.
  • speech recognition technology may be applied to the set top box 100 as shown in FIG. 3.
  • the voice recognition module (not shown) or the processor 140 to be described later may recognize the collected voice.
  • the collected voice data may be sent to an external server (not shown) to receive a voice recognition result.
  • the communication unit 130 is a component that communicates with the external IoT terminal 300.
  • the communication unit 130 may communicate with the external IoT terminal 300 using a low power communication technology.
  • communication may be performed using at least some communication protocols of Z-Wave, Wifi, Bluetooth, and Zigbee. However, this does not exclude the application of other communication technologies.
  • the communication unit 130 may communicate with the mobile terminal 500 held by the user as needed.
  • the communicator 130 may receive a control signal or content from the mobile terminal 500 and transmit the sensing data obtained through the external IoT terminal 300 to the mobile terminal 500.
  • the communication unit 130 may receive identification information such as an address from the mobile terminal 500. Communication with the mobile terminal 500 may also be accomplished in a variety of ways, including the communication protocol described above.
  • the communicator 130 may communicate with the mobile terminal 500 using LTE technology.
  • the processor 140 may acquire specific context information data as a result of analyzing the obtained at least one sensing data. have. In this case, the processor 140 may directly analyze the obtained at least one sensing data, or may receive a result analyzed by another device. The at least one sensing data may be analyzed by big data technology.
  • the specific context information refers to specific context information in the indoor space where the set top box 100 is located. That is, the information indicates the state of at least a part of a person, an object, and an environment in the indoor space to which the set-top box 100 belongs.
  • the processor 140 may refer to a database that records information on the action item of the base terminal 200 or the action item of the external IoT terminal 300, which is mapped for each type of context information data. It may support to perform or perform an action item of the base terminal 200 or an action item of the external IoT terminal 300 corresponding to the information data.
  • the database may exist separately in an external server, or the set top box 100 may store the database.
  • the set top box 100 may refer to its own database because it may be determined that there is a privacy problem.
  • the processor 140 may operate an application written in Java using an interpreter and virtual machine code that supports AOT (Ahead on Time). Even in the case of Dalvik, a virtual machine similar to the Java Virtual Machine (JVM), the performance of repeated instruction execution can be improved by supporting JIT (Just in Time), but this causes compilation every time an application is executed. It was difficult to get the most out of the hardware. In addition, a separate memory is required for compilation, and there is a problem in that security is vulnerable because program source code can be extracted from the compiled bytecode. Since the processor 140 according to an embodiment of the present invention compiles using the AOT method, there is no need to compile every time, thereby solving the above problems.
  • AOT Address on Time
  • the processor 140 may be precompiled by an application server without performing compilation when installing an application, such as ART (Android RunTime), which is an acceleration technology of Java-Dalvik adopted by the Android operating system. Download and load the application. Therefore, the execution performance is improved as compared with the conventional ART.
  • the compiled file may be composed of code translated into natural language used by the operating system.
  • the executable file downloaded by the processor 140 may be composed of natural language C / C ++, and thus, the application execution speed may be increased because the use of JNI is not required.
  • the set-top box 100 may be regarded as a context that is first experienced.
  • the processor 140 may include a person in an indoor space to which the set-top box 100 belongs, New specific contextual information data representing a state of at least some of the thing and the environment may be generated.
  • the processor 140 learns a new action item applied to the base terminal 200 or a new action item applied to the external IoT terminal 300 in storing the new specific context information data, and matches them to a database. You can also save.
  • the set top box 100 When the information mapped to the acquired specific context information data is not present in the database, it may be considered that the set top box 100 has not been learned about the context information and action items since the set-top box 100 has been installed indoors. For example, if a ring tone is collected through a microphone while a user is watching TV, the action item of the basic terminal 200 mapped to specific context information data may not be stored in the database yet. In this case, when the user performs an operation of reducing the volume of the basic terminal 200, the processor 140 identifies the pattern of the action item and stores new specific context information data in a database corresponding to the new action item of reducing the volume. By doing so, you can learn new situation information and action items. In this way, the set-top box 100 learns the new specific context information data and the new action item corresponding thereto.
  • the built-in sensor 120 may include a sensor for detecting a remote control signal (not shown) or at least one of the external IoT terminal 300 may include a sensor for detecting a remote control signal (not shown).
  • the sensor for detecting a remote control signal may detect a near infrared or infrared remote control signal in an indoor space.
  • the processor 140 acquires a remote control signal that operates the air conditioner through a sensor for detecting the remote control signal.
  • Action information for operating the air conditioner in response to the obtained remote control signal may be acquired as new action item information.
  • the processor 140 generates new specific contextual information data indicating that the room temperature is 28 degrees and user A exists, and maps the generated new specific contextual information data to new action item information for operating the air conditioner. Can be stored. Thereafter, if the user A is present while the room temperature becomes 28 degrees again, a control signal for operating the air conditioner may be generated and transmitted with reference to the new action item information mapped to the new specific context information data stored in the database.
  • the processor 140 may update the database by additionally mapping the specific context information data with at least one new action item information of the basic terminal 200 or the external IoT terminal 300.
  • the processor 140 may update the database by additionally mapping the specific situation information data with the new action item information for operating the air conditioner since the specific situation information data indicating that the room temperature is 28 degrees and the user B already exists. Can be.
  • the reason for referring to "additional” is that an action item "turning on the air conditioner” may be further mapped in a situation where an action item (that is, turning on the fan) that is already mapped to the specific situation information data exists.
  • the processor 140 may generate and transmit a control signal for operating the air conditioner with reference to the new action item information mapped to the specific context information data stored in the database.
  • the fan can also be operated at the same time.
  • the set-top box 100 may further include a remote control signal transmitter (not shown) for generating and transmitting a remote control signal.
  • a remote control signal transmitter for generating and transmitting a remote control signal.
  • the specific situation information data is obtained after the database is updated as described above, at least one new action item of the basic terminal 200 or the external IoT terminal 300 is obtained through the remote control signal transmitter. It may also support to generate and transmit or transmit a remote control signal for performing the operation.
  • the set-top box 100 learns a remote control signal for controlling the basic terminal 200 or the external IoT terminal 300 to automatically operate the basic terminal 200 or the external IoT without a user's remote control in a specific situation.
  • the operation of the terminal 300 can be controlled.
  • the processor 140 may perform the basic terminal 200 at least a predetermined number of times after the action item of the base terminal 200 or the action item of the external IoT terminal 300 corresponding to the acquired specific context information data is performed.
  • the modified action item may be learned to correspond to the obtained specific context information data and stored in a database.
  • the action item of the base terminal 200 or the action item of the external IoT terminal 300 corresponding to the acquired specific context information data has been performed, the action item of the base terminal 200 or the external IoT terminal has been performed a predetermined number of times or more.
  • the processor 140 may learn the corresponding action item to correspond to the obtained specific context information data and store the same in the database.
  • the database may store an action item for reducing the volume, and accordingly, the processor 140 performs an action item for decreasing the volume in such a case. can do.
  • the processor 140 may learn and store the action item of turning off the TV, which is the modified action item, to the specific context information data so as to correspond to the specific situation information data.
  • the set-top box 100 can provide a user experience that can interact with a user like a pet because the set-top box 100 can learn and operate action items corresponding to the situation information.
  • FIG. 4 is a diagram illustrating a concept of acquiring specific contextual information data as a result of analyzing an infrared or near infrared image photographed by a set-top box according to an embodiment of the present invention or analyzing audio acquired by a microphone.
  • the above-described built-in sensor 120 may include an infrared camera or a near infrared camera as a built-in image sensor, or at least one of the external IoT terminals 300 may include an infrared camera or a near infrared camera as an external image sensor.
  • the processor 140 may acquire, as big data, an image obtained by continuously photographing an indoor space at a predetermined time interval through the internal image sensor or the external image sensor.
  • the processor 140 analyzes at least one infrared or near-infrared image of the indoor space acquired through the built-in image sensor or the external image sensor as big data. Data can be obtained.
  • the processor 140 may obtain specific context information data indicating that the user is a minor result of analyzing at least one infrared or near infrared image acquired through the internal image sensor or the external image sensor as big data. have.
  • the processor 140 may support to perform or perform an action item that notifies another external terminal (eg, a mobile terminal) of an operation state of the base terminal 200 or the external IoT terminal 300.
  • another external terminal eg, a mobile terminal
  • the mobile terminal may support or perform an action item for restricting the operation of the basic terminal 200 or the external IoT terminal 300 (for example, limiting viewing of a specific channel of a TV).
  • the processor 140 may include: For example, at least some of an action item displaying specific content by turning on a power of a display device, which is one of the basic terminals 120, or setting a specific channel and outputting a specific sound through a speaker Or support to perform.
  • the built-in sensor 120 may further include a microphone as a built-in audio sensor or at least one of the external IoT terminal 300 may include a microphone as an external audio sensor.
  • the processor 140 may obtain specific context information data as a result of further analyzing the voice information acquired through the internal audio sensor or the external audio sensor as big data. For example, as a result of analyzing the infrared or near-infrared image and / or voice information obtained in FIG. 4, it is possible to obtain specific situation information data that the dog barks.
  • the processor 140 may further detect specific context information data as a result of further analyzing sensing data and other information obtained through another external IoT terminal 300 or the built-in sensor 120. You might get it. That is, the processor 140 may include temperature information obtained through a temperature sensor that is at least one of the external IoT terminals 300, illuminance information obtained through an illuminance sensor that is at least one of the external IoT terminals 300, and an external IoT terminal ( Vibration information obtained through at least one of the vibration sensor of 300, location information obtained through the position sensor of at least one of the external IoT terminal 300, through the fire detection sensor of at least one of the external IoT terminal 300.
  • Fire information at least one of access information acquired through an access sensor which is at least one of the external IoT terminals 300, energy consumption information obtained from the external IoT terminals 300, TV viewing information, date / time information, and weather information.
  • the specific contextual information data may be obtained by further referring to. As such, the present invention can obtain specific context information as a result of obtaining and analyzing multi-modal information.
  • the processor 140 may analyze, as big data, at least one infrared or near infrared image obtained through the internal image sensor or the external image sensor, and entrance information and date / time information obtained through the entrance sensor. As a result, specific situation information data indicating that the user is preparing to go to work or indicating that the user has gone home can be obtained.
  • the processor 140 may turn on the power of the display device, which is one of the basic terminals 120, or set the channel to a specific channel to display an action item, an action item to output a specific sound through a speaker, and an external device. It may support to perform or perform at least some of the action items for turning on / off the power of the lighting device, which is one of the IoT terminals 300.
  • the processor 140 may support to perform or perform at least some of an alert command action item, a notification message transmission action item, and a crime reporting action item.
  • the processor 140 may support to perform or perform at least some of an alert command action item, a notification message transmission action item, a fire report action item, and a fire suppression action item. Can be.
  • the processor 140 may recognize the specific context information data as a result of recognizing only a sound acquired through an internal microphone that is one of the internal sensors 120 or an external microphone that is one of the external IoT terminals 300. Can also be obtained.
  • the processor 140 acquires specific context information data as a result of analyzing the sound acquired through the internal microphone or the external microphone as big data and refers to the base terminal 200 or the external IoT terminal 300 by referring to the database. At least some of a channel change operation, a volume adjustment operation, an additional data provision operation, a PPR information provision operation, a video call operation, and a sound output operation of the speaker may be performed.
  • the processor 140 may acquire specific context information data as a result of further analyzing the sensing data and other information acquired through the other external IoT terminal 300 or the built-in sensor 120 as big data. You can do it.
  • FIG. 5 is a diagram illustrating a concept of obtaining specific context information data in detail by analyzing acquired infrared or near infrared image or voice information according to an embodiment of the present invention to identify characteristics of a person present in an indoor space. .
  • the built-in sensor 120 includes an infrared camera or a near infrared camera as a built-in image sensor, a microphone as a built-in audio sensor, or at least one of the external IoT terminal 300 is an infrared camera or near infrared
  • the camera may be included as an external image sensor or the microphone may be included as an external audio sensor.
  • the processor 140 may analyze at least one infrared or near-infrared image of the indoor space acquired through the built-in image sensor or the external image sensor as big data or as shown in FIG. 5. By analyzing the voice information acquired through the external audio sensor as big data, at least a part of the number, gender, age, and behavior of the persons present in the indoor space may be identified, and specific contextual information data may be obtained through this.
  • FIG. 6 is a diagram illustrating an embodiment of providing a service suitable for a characteristic of a person by analyzing the acquired infrared or near infrared image or the acquired voice information to grasp the characteristic of the person present in the indoor space.
  • the processor 140 analyzes at least one infrared or near-infrared image of the indoor space acquired through the internal image sensor or the external image sensor as big data to find that one user exists. You can see that the gender is female and age 30. In addition, when the user takes an action to listen to music (for example, when taking a specific gesture), the user may perform an action item that outputs the music of the girl in his 30s through the speaker. Even when the power of the basic terminal 200 is turned off, the set-top box 100 may operate as described above. Of course, as described in FIG. 5, such a service may be provided using an audio sensor, but a detailed description thereof will be omitted.
  • a user may be identified by analyzing at least one infrared or near infrared image acquired through the internal image sensor or the external image sensor as big data, and the user identification information may be used as specific context information data.
  • the set top box 100 may obtain terminal identification information such as an address from the mobile terminal 500 held by the user, the set top box 100 may identify the user through the terminal identification information.
  • the latter method since a large amount of computing resources may be consumed in analyzing an infrared or near infrared image, the latter method has an advantage of efficiently identifying a user through terminal identification information.
  • the communication unit 130 may perform interaction with the mobile terminal 500 and the external IoT terminal 300 to provide various services in addition to receiving terminal identification information from the mobile terminal 500 held by the user. This embodiment will be described below.
  • FIG. 7 is a conceptual diagram illustrating an interaction between a set top box, a mobile terminal and an external IoT terminal according to another embodiment of the present invention.
  • the processor 140 recognizes context information data corresponding to the control signal, and corresponds to a basic terminal corresponding to the recognized context information data. It may support to perform or perform the action item of 200 or the action item of the external IoT terminal 300. This allows the user to simply control the mobile terminal 500 to control the external IoT terminal 300, such as home lighting, air conditioning, washing machine.
  • the processor 140 receives the at least one sensing data obtained through the built-in sensor 120 or the external IoT terminal 300 and the authenticated mobile terminal 500. Can be sent to.
  • the processor 140 may transmit temperature information of a house acquired through a temperature sensor that is an external IoT terminal 300 to the authenticated mobile terminal 500. .
  • FIG. 8 is a diagram illustrating an operation of performing an action item displaying content of a mobile terminal through a base terminal according to another embodiment of the present invention.
  • the specific context information data relates to the mobile terminal 500 possessed by the user (for example, the context information data corresponding to the pattern of shaking the mobile terminal 500, the mobile terminal 500 ),
  • the processor 140 displays an image of the content generated through the mobile terminal 500 through the base terminal 200 or outputs a sound for the content. It may support to perform or perform an action item.
  • the processor 140 may support or perform an action item for displaying an image captured by the mobile terminal 500 through a display device, which is one of the basic terminals 200. Therefore, the user may transmit the image photographed through the mobile terminal 500 to the remotely located family to display on the TV watched by the family.
  • the processor 140 displays the image taken through the mobile terminal 500 through the basic terminal 200. It may support to perform or perform an action item. However, when two or more people are in an indoor space, the image may not be displayed in consideration of privacy issues.
  • FIG. 9 is a diagram illustrating an embodiment of providing an image acquired by a set top box to a mobile terminal and performing an action item according to a control signal received from the mobile terminal.
  • the processor 140 may transmit at least one image acquired through the built-in sensor 120 or the external IoT terminal 300 to the authenticated mobile terminal.
  • a user may receive an image of a dog taken in real time through the mobile terminal 500.
  • the user may input a control signal through the mobile terminal 500.
  • the processor 140 recognizes the context information data corresponding to the control signal and the action item of the basic terminal 200 corresponding to the recognized context information data. Or it may support to perform or perform the action item of the external IoT terminal 300.
  • the processor 140 performs an action item displaying an image of a pet favorite dog through the base terminal 200 according to a control signal.
  • the processor 140 may perform a video call between the user of the mobile terminal 500 and the pet dog according to the control signal.
  • FIG. 10 is a diagram illustrating an embodiment of performing an action item with reference to energy use information of an IoT terminal.
  • the communication unit 130 may obtain energy usage information of each of the external IoT terminals 300.
  • the communication unit 130 may receive and manage energy usage information from each of the external IoT terminals 300 consuming electricity or gas at a predetermined time interval or after making a request.
  • the communication unit 130 may obtain energy usage information at a predetermined time interval or on request from a power meter or a gas meter, which is one of the external IoT terminals 300.
  • the processor 140 may support or perform an action item of the basic terminal 200 or an action item of the external IoT terminal 300 with reference to the energy usage information of each of the obtained external IoT terminals 300. have. For example, as a result of analyzing, as big data, the electricity usage obtained from the electricity meter, which is one of the external IoT terminals 300, if the current electricity usage is determined to be greater than or equal to a predetermined value, the processor 140 may determine the specific electronics of the household. It may support or perform an action item that cuts off the power of the device.
  • the processor 140 is the remote control signal transmitting unit of the set-top box 100 or the external IoT terminal 300 is specified It may be controlled to generate and transmit a remote control signal for turning off the power of the electronic device.
  • the processor 140 holds information on the usage state of the corresponding external IoT terminal 300. It can be provided to the mobile terminal 500. For example, if it is determined that the air conditioner in the house consumes excessive electricity, the processor 140 may provide such information to the mobile terminal 500 of the user through the communication unit 130. The user may transmit a control command such as turning off the air conditioner or increasing a set temperature through the mobile terminal 500 to the set-top box 100.
  • the processor 140 may further obtain EPG information on the broadcast signal provided to the base terminal 200.
  • the processor 140 may further refer to the obtained EPG information to determine the action item or external appearance of the base terminal 200. It may support to perform or perform an action item of the IoT terminal 300. In this way, if the EPG information is further referred to, the action item of the base terminal 200 or the external IoT terminal 300 suitable for a specific situation may be more precisely performed.
  • the operation of the basic terminal 200 or the external IoT terminal 300 may be controlled.
  • the processor 140 further performs the function of controlling the flow of data between the above-described broadcast signal processor 110, built-in sensor 120, the communication unit 130. That is, the processor 140 controls the flow of data between each component of the set-top box 100, thereby controlling each component to perform its own function.
  • the processor 140 includes hardware configurations such as a micro processing unit (MPU) or a central processing unit (CPU), a cache memory, a data bus, and a software configuration of an operating system and an application performing a specific purpose. It may include.
  • FIG. 11 is a conceptual diagram schematically illustrating an interaction between a photographing apparatus and another apparatus according to another embodiment of the present invention
  • FIG. 12 illustrates a configuration and control flow of the photographing apparatus according to another embodiment of the present invention. It is a block diagram.
  • the photographing apparatus 100 may mean a CCTV (Closed Circuit Television) or the like, but is not limited thereto.
  • CCTV Consumer Circuit Television
  • the photographing apparatus 100 ′ may communicate with other devices through a wired or wireless network similarly to the set-top box 100 described above.
  • the photographing apparatus 100 ′ may photograph a predetermined space and provide an image signal to a basic terminal (not shown).
  • the basic terminal may receive and display an image signal.
  • the predetermined space may be either an indoor space or an outdoor space.
  • the photographing apparatus 100 ′ may be installed inside or outside the home.
  • the photographing apparatus 100 ′ may communicate with at least one external IoT terminal 300.
  • the external IoT terminal 300 continuously provides the sensing data to the photographing device 100 'or at the request of the photographing device 100'.
  • the photographing apparatus 100 ′ may obtain specific context information data as a result of receiving and analyzing the sensing data.
  • the photographing apparatus 100 ′ may obtain specific context information data as a result of analyzing the multi-modal sensing data acquired from the plurality of external IoT terminals 300.
  • the photographing apparatus 100 ′ may obtain information about an action item of the basic terminal or an action item of the external IoT terminal 300 by communicating with the database server 400 and mapped to specific context information data. have.
  • the photographing apparatus 100 ′ may store the database in its own storage and refer to it.
  • the photographing apparatus 100 ′ may generate a control command for operating the basic terminal or the external IoT terminal 300 according to the obtained information on the action item, and transmit the generated control command to each terminal.
  • the photographing apparatus 100 ′ may communicate with the mobile terminal 500. That is, the photographing apparatus 100 ′ may receive control signals or content from the mobile terminal 500 or transmit sensing data acquired through the external IoT terminal 300 to the mobile terminal 500. In addition, the photographing apparatus 100 ′ may receive identification information such as an address from the mobile terminal 500 and use the same to identify a user having the mobile terminal 500.
  • the photographing apparatus 100 ′ includes an image signal providing unit 150, an internal sensor 120, a communication unit 130, and a processor 140.
  • the video signal providing unit 150 is configured to photograph a predetermined space and provide a video signal to a basic terminal.
  • the photographing apparatus 100 ′ may provide an image signal obtained by photographing a predetermined space to a display device in a control center or other building.
  • the processor 140 may (i) analyze the obtained at least one sensing data.
  • Acquire specific context information data representing the state of at least some of the person, thing and environment, and (ii) for the action item of the base terminal or the action item of the external IoT terminal 300 mapped by the type of the context information data as With reference to the database recording the information, it is possible to support or perform the action item of the base terminal or the action item of the external IoT terminal 300 corresponding to the specific context information data.
  • the photographing apparatus 100 ′ analyzes sensing data acquired in a predetermined space to identify context information on which the photographing apparatus 100 ′ is placed, and various apparatuses corresponding thereto. May perform an operation corresponding to the context information.
  • the embodiments according to the present invention described above may be implemented in the form of program instructions that may be executed by various computer components, and may be recorded in a computer-readable recording medium.
  • the computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.
  • Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or may be known and available to those skilled in the computer software arts.
  • Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.
  • Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.
  • the hardware device may be configured to operate as one or more software modules to perform the process according to the invention, and vice versa.

Abstract

La présente invention concerne un boîtier décodeur qui exécute une fonction sensible au contexte en se basant sur des informations multimodales afin d'apprendre et de s'améliorer de manière autonome. Le boîtier décodeur comprend : une unité de traitement de signal de diffusion pour recevoir et fournir un signal de diffusion ; et un processeur pour (i) acquérir des données d'informations contextuelles spécifiques indiquant un état d'au moins une personne, un objet, et/ou un environnement dans un espace intérieur auquel le boîtier décodeur appartient, en tant que résultat de l'analyse d'au moins un élément parmi des données de détection acquises, et (ii) exécuter un élément d'action d'un terminal de base ou un élément d'action d'un terminal IoT externe, qui correspond aux données d'informations contextuelles spécifiques, en référence à une base de données qui enregistre des informations sur l'élément d'action du terminal de base ou sur l'élément d'action du terminal IoT externe, qui est mappé selon le type des données d'informations contextuelles.
PCT/KR2016/001332 2015-03-16 2016-02-05 Boîtier décodeur et appareil de photographie pour exécuter une fonction sensible au contexte à base d'informations multi-modales pour apprentissage et amélioration automatiques d'interface utilisateur et d'expérience d'utilisateur WO2016148398A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2015-0036253 2015-03-16
KR1020150036253A KR101671760B1 (ko) 2015-03-16 2015-03-16 멀티모달 정보를 기반으로 상황 인지 기능을 수행하여 사용자 인터페이스와 사용자 경험을 스스로 학습하고 개선하는 셋톱박스, 촬영 장치, 그리고 이를 이용한 방법 및 컴퓨터 판독 가능한 기록 매체

Publications (1)

Publication Number Publication Date
WO2016148398A1 true WO2016148398A1 (fr) 2016-09-22

Family

ID=56919123

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2016/001332 WO2016148398A1 (fr) 2015-03-16 2016-02-05 Boîtier décodeur et appareil de photographie pour exécuter une fonction sensible au contexte à base d'informations multi-modales pour apprentissage et amélioration automatiques d'interface utilisateur et d'expérience d'utilisateur

Country Status (2)

Country Link
KR (1) KR101671760B1 (fr)
WO (1) WO2016148398A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108574771A (zh) * 2017-03-10 2018-09-25 峰范（北京）科技有限公司 信息收集与处理系统及其语音播放装置、处理方法

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101961437B1 (ko) * 2017-07-14 2019-07-18 주식회사 아씨오 냉난방 제어 시스템, 방법 및 컴퓨터 판독 가능한 기록매체
KR102122095B1 (ko) * 2018-08-31 2020-06-11 주식회사 나눔기술 Ai컴패니언의 상황인지, 현황 및 표현 정보를 통신프로토콜을 이용하여 모니터링하는 인터페이스 시스템 및 그 방법
KR20200060197A (ko) 2018-11-21 2020-05-29 한국전자통신연구원 사용자 맞춤 증강현실 서비스 제공 방법 및 이를 위한 장치
EP4002858A4 (fr) * 2019-07-16 2023-03-29 Lg Electronics Inc. Dispositif d'affichage pour commander un ou plusieurs appareils électroménagers en fonction d'une situation de visualisation
KR102307093B1 (ko) * 2019-12-24 2021-09-30 주식회사 엘지유플러스 개인화된 iptv 서비스 제공 방법 및 장치
WO2024071625A1 (fr) * 2022-09-29 2024-04-04 삼성전자 주식회사 Dispositif d'affichage d'image pour afficher une image de réalité virtuelle, et procédé d'affichage associé

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20050046580A (ko) * 2003-11-14 2005-05-18 한국전자통신연구원 홈네트워크 환경에서의 멀티-모달 상황 인식어플리케이션을 위한 시스템 및 방법
JP2008206166A (ja) * 2008-03-10 2008-09-04 Funai Electric Co Ltd 放送信号受信システム
WO2010126321A2 (fr) * 2009-04-30 2010-11-04 삼성전자주식회사 Appareil et procédé pour inférence d'intention utilisateur au moyen d'informations multimodes
KR20130083764A (ko) * 2012-01-13 2013-07-23 주식회사 오비고 웹 애플리케이션에 포함되는 자바스크립트를 ａｏｔ 방식으로 컴파일하기 위한 방법, 단말 장치 및 컴퓨터 판독 가능한 기록 매체
KR20130108467A (ko) * 2011-01-19 2013-10-02 퀄컴 인코포레이티드 패턴 매칭을 통한 상황들의 학습

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20050046580A (ko) * 2003-11-14 2005-05-18 한국전자통신연구원 홈네트워크 환경에서의 멀티-모달 상황 인식어플리케이션을 위한 시스템 및 방법
JP2008206166A (ja) * 2008-03-10 2008-09-04 Funai Electric Co Ltd 放送信号受信システム
WO2010126321A2 (fr) * 2009-04-30 2010-11-04 삼성전자주식회사 Appareil et procédé pour inférence d'intention utilisateur au moyen d'informations multimodes
KR20130108467A (ko) * 2011-01-19 2013-10-02 퀄컴 인코포레이티드 패턴 매칭을 통한 상황들의 학습
KR20130083764A (ko) * 2012-01-13 2013-07-23 주식회사 오비고 웹 애플리케이션에 포함되는 자바스크립트를 ａｏｔ 방식으로 컴파일하기 위한 방법, 단말 장치 및 컴퓨터 판독 가능한 기록 매체

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108574771A (zh) * 2017-03-10 2018-09-25 峰范（北京）科技有限公司 信息收集与处理系统及其语音播放装置、处理方法

Also Published As

Publication number Publication date
KR101671760B1 (ko) 2016-11-02
KR20160111284A (ko) 2016-09-26

Similar Documents

Publication Publication Date Title
WO2016148398A1 (fr) Boîtier décodeur et appareil de photographie pour exécuter une fonction sensible au contexte à base d'informations multi-modales pour apprentissage et amélioration automatiques d'interface utilisateur et d'expérience d'utilisateur
US10499238B2 (en) Remote alarm hushing
US10869006B2 (en) Doorbell camera with battery at chime
US9294340B1 (en) Systems and methods for updating data across multiple network architectures
CN108831448A (zh) 语音控制智能设备的方法、装置及存储介质
WO2020173375A1 (fr) Procédé, dispositif et système de commande de liaison de plusieurs dispositifs intelligents
WO2017010794A1 (fr) Procédé de commande et appareil permettant de gérer un environnement de sommeil
WO2016186440A1 (fr) Boîtier décodeur utilisant des informations multimodales pour acquérir des informations utilisateur, serveur de gestion utilisant le boîtier décodeur, et support d'enregistrement lisible par ordinateur
EP2430767A2 (fr) Terminal mobile
US11743578B2 (en) Systems and methods of power-management on smart devices
CN112166350A (zh) 智能设备中的超声感测的系统和方法
CN112136102B (zh) 信息处理装置、信息处理方法以及信息处理系统
US20160148476A1 (en) Monitoring system
CN104410898A (zh) 智能家庭云盒系统及其控制方法
WO2016117737A1 (fr) Système de collecte et de traitement d'informations de dispositifs domestiques utilisant un internet des objets en nuage, et procédé de collecte et de traitement associé
US9705696B2 (en) Monitoring system
WO2019168377A1 (fr) Dispositif électronique et procédé de commande de dispositif électronique externe basé sur des informations de motif d'utilisation correspondant à un utilisateur
CN109196352B (zh) 用于通过使用声学信号来监视对象及其状态的系统和方法
WO2013180354A1 (fr) Procédé et dispositif domestique pour sortir une réponse à une entrée d'utilisateur
KR20160111777A (ko) 멀티모달 정보를 기반으로 상황 인지 기능을 수행하여 사용자 인터페이스와 사용자 경험을 스스로 학습하고 개선하는 셋톱박스, 촬영 장치, 그리고 이를 이용한 방법 및 컴퓨터 판독 가능한 기록 매체
WO2021256760A1 (fr) Dispositif électronique mobile et son procédé de commande
JP2020119425A (ja) 情報処理装置、情報処理システム、情報処理方法およびプログラム
WO2021125507A1 (fr) Dispositif électronique et son procédé de commande
WO2020111398A1 (fr) Appareil électronique et procédé de commande de celui-ci
KR20170126648A (ko) 월패드, 월패드 가상화 장치 및 이를 이용한 월패드 가상화 구현 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16765154

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 26.01.2018)

122 Ep: pct application non-entry in european phase

Ref document number: 16765154

Country of ref document: EP

Kind code of ref document: A1