KR101819499B1 - Multimedia device, multiple image sensors having different types and the method for controlling the same - Google Patents

Abstract

According to another aspect of the present invention, there is provided a method of controlling a multimedia device using a plurality of heterogeneous image sensors, the method comprising: controlling a first image sensor for extracting depth data and a second image sensor for extracting color data, Identifying at least one user located in the perimeter of the device; accessing a memory storing a per-user preferred channel; receiving, from the accessed memory, a preference channel for a preferred channel corresponding to the identified at least one user Controlling the switching to the corresponding preferred channel based on the extracted information, and monitoring a change of the user located in front of the multimedia device.

Description

TECHNICAL FIELD [0001] The present invention relates to a multimedia device, a plurality of heterogeneous image sensors, and a method of controlling the same. [0002] MULTIMEDIA DEVICE, MULTIPLE IMAGE SENSORS HAVING DIFFERENT TYPES AND THE METHOD FOR CONTROLLING THE SAME,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multimedia device technology, and more particularly, to a technology for controlling a multimedia device using a plurality of heterogeneous image sensors.

There have been many discussions about the technique of simply making a video call using a camera mounted on a PC or a notebook computer. In addition, recently, methods for meeting various functions and conditions desired by users due to rapid development of electric and electronic technology are being discussed. Furthermore, a camera and the like linked to a multimedia device have been proposed for research and application for utilizing various functions in addition to a conventional simple function of a video call. For example, various attempts have been made to utilize a camera coupled to a multimedia device to utilize a user's face recognition and recognition field, multimedia device control according to a user's gesture recognition, and the like. However, the control according to the recognition and recognition of the user using the existing camera has a problem that the level of recognition and recognition can not be achieved due to the environment in which the multimedia device is installed and the user's unexpected movements.

One embodiment of the present invention is to provide a solution for more accurately recognizing and recognizing users located in the vicinity of a multimedia device using a plurality of heterogeneous image sensors.

Another embodiment of the present invention provides a design method capable of simultaneously enhancing recognition and recognition distance and performance by improving problems caused in a recognition and recognition process provided by a camera interfaced with a multimedia device .

According to another embodiment of the present invention, the preferred channel service is provided in various forms according to whether the recognized user is singular or plural, and the processing speed is increased.

According to an embodiment of the present invention, there is provided a method of controlling a multimedia device using a plurality of heterogeneous image sensors, the method comprising: controlling a first image sensor for extracting depth data and a second image sensor for extracting color data, Identifying at least one user located in the perimeter of the device; accessing a memory storing a per-user preferred channel; receiving, from the accessed memory, a preference channel for a preferred channel corresponding to the identified at least one user Controlling the switching to the corresponding preferred channel based on the extracted information, and monitoring a change of a user located in the vicinity of the multimedia device.

Further, a multimedia device using a plurality of heterogeneous image sensors according to an embodiment of the present invention includes a broadcast network interface for receiving broadcast data, a first image sensor for extracting depth data, and a second image sensor for extracting color data. A face recognition module for identifying at least one user using information generated in accordance with the control of the image sensor, a memory for storing a preference channel for each user, and a memory for accessing the memory, A display unit for displaying a guide message for indicating the corresponding preference channel; and a display unit for displaying the corresponding preference channel when the signal for selecting the corresponding preference channel is applied, And a control unit for controlling the switching to the preferred channel do.

According to an embodiment of the present invention, a solution for more accurately identifying users located in the vicinity of a multimedia device using a plurality of heterogeneous image sensors is provided.

According to another embodiment of the present invention, there is provided a design method capable of enhancing recognition and recognition distance and performance at the same time by improving the problems occurring in the recognition and recognition process provided by the camera interlocked with the multimedia device .

According to another embodiment of the present invention, the preferred channel service is provided in various forms according to whether the recognized user is singular or plural, and the processing speed is improved.

The effects of the more specific invention will be described later in detail in the following table.

1 is a diagram schematically illustrating an example of an overall system including a multimedia device according to an embodiment of the present invention.
FIG. 2 is a diagram showing an example of the multimedia device shown in FIG. 1 in more detail.
FIG. 3 is a view showing a multimedia device using a plurality of heterogeneous image sensors according to an embodiment of the present invention and a photographing screen at the same time.
4 is a diagram illustrating a process of using detection data and recognition data in a plurality of heterogeneous image sensors and multimedia devices according to an embodiment of the present invention.
5 is a diagram for explaining a face vector stored in the database shown in FIG.
6 is a view for explaining an operation of a plurality of heterogeneous image sensors connected to a multimedia device into a hardware area and a software area according to an embodiment of the present invention.
7 is a view showing a plurality of heterogeneous image sensors and a multimedia device according to an embodiment of the present invention, respectively.
8 is a view showing a plurality of heterogeneous image sensors and a multimedia device according to another embodiment of the present invention, respectively.
FIG. 9 is a diagram illustrating a plurality of heterogeneous image sensors according to an exemplary embodiment of the present invention in more detail.
10 is a view showing an example of a first image sensor among a plurality of heterogeneous image sensors according to an embodiment of the present invention.
11 is a view showing another example of the first image sensor among the plurality of heterogeneous image sensors according to the embodiment of the present invention.
12 is a diagram for explaining a method of calculating a distance using the first image sensor shown in FIG.
13 is a view showing an image taken by a first image sensor among a plurality of heterogeneous image sensors according to an embodiment of the present invention.
14 is a view showing an image taken by a second image sensor among a plurality of heterogeneous image sensors according to an embodiment of the present invention.
15 is a block diagram of a multimedia device for providing a preferred channel for each user / family unit using a plurality of heterogeneous image sensors according to an embodiment of the present invention.
15 is a diagram illustrating a first example of a scene in which a multimedia device for providing a preferred channel per user / family unit by using a plurality of heterogeneous image sensors according to an exemplary embodiment of the present invention automatically selects a preferred channel to be.
Fig. 17 is a diagram showing an example of a database storing data obtained by Fig. 16. Fig.
18 is a view showing a second example of a scene in which a multimedia device for providing a preferred channel for each user / family unit by using a plurality of heterogeneous image sensors according to an embodiment of the present invention automatically selects a preferred channel to be.
FIG. 19 is a diagram showing an example of a database storing data obtained by FIG. 18. FIG.
20 is a diagram illustrating a third example of a scene in which a multimedia device for providing a preferred channel for each user / family unit by using a plurality of heterogeneous image sensors according to an embodiment of the present invention automatically selects a preferred channel to be.
FIG. 21 is a diagram showing an example of a database storing data acquired by FIG. 20. FIG.
22 to 24 illustrate a first example of a process of automatically switching a multimedia device for providing a preferred channel per user / family unit to a preferred channel using a plurality of heterogeneous image sensors according to an embodiment of the present invention FIG.
25 to 27 illustrate a second example of a process of automatically switching a multimedia device for providing a preferred channel for each user / family unit to a preferred channel using a plurality of heterogeneous image sensors according to an embodiment of the present invention FIG.
28 to 30 illustrate a third example of a process of automatically switching a multimedia device for providing a preferred channel for each user / family unit to a preferred channel using a plurality of heterogeneous image sensors according to an embodiment of the present invention FIG.
31 is a flowchart illustrating a method for providing a preference channel for each user / family unit using a plurality of heterogeneous image sensors according to an exemplary embodiment of the present invention.
Fig. 32 is a flowchart showing the step S2810 shown in Fig. 31 in more detail according to another embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings. Furthermore, the suffix "module" and " part "for components used in the following description are given merely for ease of description, and the" module " It can be designed in hardware or software.

Meanwhile, the multimedia device described herein corresponds to various types of devices that receive and process broadcast data, for example. Further, the multimedia device may correspond to Connected TV. In addition to the broadcast receiving function, the Connected TV is further provided with a wired / wireless communication device and the like, and is more convenient to use than a hand-held input device, a touch screen or a motion- Lt; / RTI > Also, it can be connected to the Internet and a computer by the support of a wired or wireless Internet function, and can perform functions such as e-mail, web browsing, banking or game. A standardized general-purpose OS may be used for these various functions.

Accordingly, the Connected TV can perform various user-friendly functions because various applications can be freely added or deleted on a general-purpose OS kernel, for example. More specifically, the Connected TV may be, for example, a web TV, an Internet TV, an HBBTV, a smart TV, a DTV, or the like, and may be applied to a smartphone according to circumstances.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG.

As used herein, terms used in the present invention are selected from general terms that are widely used in the present invention while taking into account the functions of the present invention, but these may vary depending on the intention or custom of a person skilled in the art or the emergence of new technologies. Also, in certain cases, there may be a term chosen arbitrarily by the applicant, in which case the meaning thereof will be described in the description of the corresponding invention. Therefore, it is intended that the terminology used herein should be interpreted based on the meaning of the term rather than on the name of the term, and on the entire contents of the specification.

1 is a diagram schematically illustrating an example of an overall broadcasting system including a multimedia device according to an embodiment of the present invention. The multimedia device of FIG. 1 may correspond to, for example, Connected TV, but the scope of the present invention is not limited to Connected TV, and the scope of rights of the present invention should be determined by the claims in principle.

1, an overall system including a multimedia device according to an embodiment of the present invention includes a content provider (CP) 10, a service provider (SP) 20, a network provider A network provider (NP) 30 and an HNED 40. The HNED 40 corresponds to, for example, a client 100 that is a multimedia device according to an embodiment of the present invention.

The content provider 10 produces and provides various contents. 1, a terrestrial broadcaster, a cable SO (System Operator), a MSO (Multiple System Operator), a satellite broadcaster, An Internet broadcaster, and the like may be exemplified. [0040] In addition, the content provider 10 may provide various applications in addition to the broadcast content.

The service provider 20 can provide the contents provided by the contents provider 10 in a service package. For example, the service provider 20 of FIG. 1 can package and provide the first terrestrial broadcast, second terrestrial broadcast, cable MSO, satellite broadcast, various Internet broadcasts, applications, etc. to the user.

The network provider 30 may provide a network for providing services to the client 100. The client 100 may be provided with a service by building a home network end user (HNED).

On the other hand, the client 100 can also provide the content through the network. In this case, conversely, conversely, conversely, the client 100 may be a content provider, and the content provider 10 may receive content from the client 100. [ In this case, there is an advantage that an interactive content service or a data service is possible.

FIG. 2 is a diagram showing an example of the multimedia device shown in FIG. 1 in more detail.

The multimedia device 200 includes a network interface 201, a TCP / IP manager 202, a service delivery manager 203, A demultiplexer 205, a PSIP and / or SI decoder 204, an audio decoder 206, a video decoder 207, a display unit A / and OSD Module 208, a Service Control Manager 209, a Service Discovery Manager 210, a Metadata Manager 212, an SI & Metadata DB 211, A manager 214, and a service manager 213. Further, a plurality of heterogeneous image sensors 260 are connected to the multimedia device 200, and are connected by, for example, a USB-type connection. 2, the plurality of heterogeneous image sensors 260 are designed as separate modules. However, the plurality of heterogeneous image sensors 260 may be designed to be housed in the multimedia device 200 It is possible.

The network interface unit 201 receives packets received from the network and transmits packets to the network. That is, the network interface unit 201 receives services, contents, and the like from the service provider through the network.

The TCP / IP manager 202 is involved in delivering packets from the multimedia device 200 to the multimedia device 200, that is, from the source to the destination. The service delivery manager 203 is responsible for controlling the received service data. For example, RTP / RTCP can be used to control real-time streaming data. When the real-time streaming data is transmitted using the RTP, the service delivery manager 203 parses the received data packet according to RTP and transmits it to the demultiplexer 205 or to the service manager 213 And stores it in the SI & Metadata DB 711. And feedbacks the network reception information to the server providing the service using the RTCP.

The demultiplexer 205 demultiplexes the received packets into audio, video, program specific information (PSI) data, and transmits them to the audio / video decoders 206 and 207 and the PSI & PSIP and / do.

The PSI & PSIP and / or SI decoder 204 receives and decodes a demultiplexed PSI section, a Program and Service Information Protocol (PSIP) section, or a SI (Service Information) section in the demultiplexer 205.

In addition, the PSI & PSIP and / or SI decoder 204 decodes the received sections to create a database of service information, and stores the database related to the service information in the SI & Metadata DB 211.

The audio / video decoder 206/207 decodes the video data and audio data received from the demultiplexer 205. [

The UI manager 214 provides a GUI (Graphic User Interface) for the user using an OSD (On Screen Display) or the like, and receives a key input from the user to perform a receiver operation according to the input. For example, upon receipt of a key input from the user regarding channel selection, the key input signal is transmitted to the service manager 213.

The service manager 213 controls the manager associated with the service, such as the service delivery manager 203, the service discovery manager 210, the service control manager 209, and the metadata manager 212.

The service manager 213 also creates a channel map and selects a channel using the channel map according to the key input received from the user interface manager 214. The service discovery manager 210 provides information necessary for selecting a service provider that provides the service. Upon receiving a signal regarding channel selection from the service manager 213, the service discovery manager 210 searches for the service using the information.

The service control manager 209 is responsible for selection and control of services. For example, when a user selects a service such as IGMP or RTSP when selecting a Live Broadcasting service such as an existing broadcasting method and selects a service such as VOD (Video On Demand), RTSP is used to select a service, control . The metadata manager 212 manages the metadata associated with the service and stores the metadata in the SI & Metadata DB 211.

The SI & Metadata DB 211 selects the service information decoded by the PSI & PSIP and / or SI Decoder 204, the metadata managed by the metadata manager 212, and the service provider provided by the service discovery manager 210 Store the necessary information. In addition, the SI & Metadata DB 211 may store setup data and the like for the system.

Meanwhile, the IG 250 is a gateway that collects functions necessary for accessing an IMS-based IPTV service.

The plurality of heterogeneous image sensors 260 shown in FIG. 2 are designed to capture a single image or a plurality of images for a person or object located in the vicinity of the multimedia device 200. More specifically, for example, the plurality of heterogeneous image sensors 260 are designed to operate on a single image or a plurality of images sequentially, periodically, at a selected time, or at a particular condition. A detailed description thereof will be described below.

FIG. 3 is a view showing a multimedia device using a plurality of heterogeneous image sensors according to an embodiment of the present invention and a photographing screen at the same time. Hereinafter, referring to FIG. 3, a multimedia device and a photographing screen using a plurality of heterogeneous image sensors according to an embodiment of the present invention will be described.

In general, the first image sensors associated with depth data processing are not suitable for long-distance facial recognition due to limited resolution (e.g., maximum VGA grade) and recognition distances (e.g., 3.5 m) . In addition, the second image sensors related to color data processing have a disadvantage that they are slow in recognition and are not robust to light conditions. Therefore, in order to compensate for the disadvantages of each image sensor, the multimedia device according to an embodiment of the present invention is designed to be interlocked with a hybrid image sensor module in which a first image sensor and a second image sensor are combined.

As the above-mentioned first image sensor, for example, an IR camera or a depth camera is used. More specifically, for example, TOF (Time Of Flight) method and structured light method are discussed with the IR camera or depth camera. In the TOF method, distance information is calculated using a time difference that is radiated by infrared rays. The structured light method radiates infrared rays in a specific pattern and calculates a distance by analyzing the deformed pattern. However, in the case of the first image sensor, there is an advantage in terms of depth data recognition and processing speed, and it is possible to easily sense objects and people even in a dark place. However, it has a disadvantage that the resolution is low at a long distance.

Furthermore, as the second image sensor described above, for example, a color camera or an RGB camera is used. More specifically, for example, the color camera or the RGB camera, the stereo camera method and the mono camera method are discussed. The stereo camera method detects and tracks a hand or a face on the basis of respective image parallax comparison information photographed through two cameras. The mono camera method detects and tracks a hand or a face based on shape and color information shot through a single camera. However, in the case of the second image sensor, the resolution is improved as compared with the first image sensor. However, the second image sensor is vulnerable to ambient light and is difficult to recognize in the dark. Particularly, there is a problem that accurate depth recognition is difficult.

In order to solve such conventional problems, as shown in FIG. 3, a multimedia device according to an embodiment of the present invention is designed to include both a first image sensor and a second image sensor. However, the image sensors may be designed to be embedded in the multimedia device or may be designed as a separate hardware module. First, as shown in the area (b) of FIG. 3, the first image sensor captures an image including users located in the vicinity of the multimedia device. The specific photographed images are sequentially shown in (1), (2), (3), and (4) in FIG.

On the other hand, when the photographing and data analysis of the first image sensor is completed, the second image sensor photographs an image of a specific user's face, as shown in the area (a) of FIG. The specific photographed images are sequentially shown in (5), (6), and (7) in FIG.

The first image sensor among the plurality of heterogeneous image sensors according to an embodiment of the present invention captures a first image located in the periphery of the multimedia device and extracts depth data from the captured first image. As shown in (1) of FIG. 3, it is possible to design the area of each object to be displayed in different shades according to the distance.

Further, the first image sensor can recognize and recognize at least one face of the user using the extracted depth data. That is, by using a pre-stored database or the like, Extracts the user's body information (e.g., a face, a hand, a foot, a joint, etc.) as shown in FIG. 3B, and further calculates position coordinates and distance information . More specifically, it is designed to calculate x, y, and z values, which are positional information of the face of the user, where x is the position of the face on the abscissa in the captured first image, The position of the face with respect to the vertical axis in the first image, and z denotes the distance between the face of the user and the first image sensor.

In addition, a second image sensor for extracting a color image among a plurality of heterogeneous image sensors according to an embodiment of the present invention photographs a second image of the recognized face of the user, Respectively.

On the other hand, when the second image sensor and the first image sensor shown in FIG. 3 are designed to be adjacent to each other, the error due to the physical position difference may be negligible. However, according to another embodiment of the present invention, the coordinate information or the distance information acquired by the first image sensor is corrected using the information on the physical position difference, It is designed to capture users by using information or distance information. In addition, if the first image sensor and the second image sensor are designed to be horizontal from the ground, information on the phisycal position difference may be set on the basis of a horizontal frame. The feature information is extracted from the photographed second image as shown in (3) of FIG. The feature information is, for example, data corresponding to a specific region (e.g., mouth, nose, eye, etc.) for identifying a plurality of users using the multimedia device. Furthermore, the second image sensor may zoom in on the area of the user's face based on coordinate values (x, y, z values) obtained through photographing of the first image sensor. This means a process of switching from (5) to (6) in FIG.

When the photographing and analysis of the first image sensor and the second image sensor are completed, the multimedia device according to an embodiment of the present invention accesses a memory storing data corresponding to the extracted feature information, And extracts information identifying a particular user stored in the memory.

If information identifying the specific user exists in the memory, the multimedia device provides a predetermined service for the specific user.

On the other hand, if the information identifying the particular user is not present in the memory, the multimedia device is designed to display a guide message for storing the recognized user in the memory.

As described above, according to the embodiment of the present invention, the first image sensor detects the user position information or the coordinate information of the face, and the second image sensor uses the data obtained from the first image sensor, .

Furthermore, according to another embodiment of the present invention, the second image sensor is designed to operate only in a specific condition, not unconditionally using the second image sensor. For example, if the distance information to the user obtained by the operation of the first image sensor is less than or equal to the first reference value, or if the recognition rate of the user's face acquired through the operation of the first image sensor is equal to or greater than the second reference value, Detects and recoginizes a user face located in the vicinity of the multimedia device with only the first image sensor. On the other hand, if the distance information to the user obtained through the operation of the first image sensor exceeds the first reference value, or if the recognition rate of the user's face acquired by the operation of the first image sensor is less than the second reference value, And is further designed to recognize the user's face using the second image sensor.

According to another embodiment of the present invention, in the process of recognizing the user's face, the second image sensor zooms in using the distance information acquired through the first image sensor, And is designed to capture only the face portion using the face coordinate information acquired through the sensor.

Therefore, when a plurality of different image sensors of different types are used, it is possible to realize a long-distance facial recognition, and the data processing speed is further improved than before.

4 is a diagram illustrating a process of using detection data and recognition data in a plurality of heterogeneous image sensors and multimedia devices according to an embodiment of the present invention.

Face detection and face recognition are different processes. The face detection includes a process of detecting a face region in one image. On the other hand, the face recognition is a process of recognizing whether a detected face corresponds to a specific user. Particularly, the process of executing the face detection process using the first image sensor and executing the face recognition process using the second image sensor according to an embodiment of the present invention will be described with reference to FIG.

4, a multimedia device according to an exemplary embodiment of the present invention includes a detection module 301, a recognition module 302, a database 303, a first image sensor 304, a second image sensor 305 And the like, and uses the detection data 306 and the recognition data 307 as needed. The detection data 306 may include, for example, information-based detection techniques, feature-based detection techniques, template matching techniques, may be generated based on appearance-based detection techniques. In addition, the recognition data 307 includes data such as eyes, nose, mouth, jaw, area, distance, shape, angle and the like for identifying a specific user.

Further, the detection module 301 determines the presence of the user's face by using the image data received from the first image sensor 304. In the process of estimating the area where the user's face is located, the above-described information-based detection techniques, feature-based detection techniques, matching techniques, and appearance-based detection techniques.

The recognition module 302 identifies whether the user is a specific user by using the image data received from the second image sensor 305. [ At this time, the received image data and the face vector information stored in the DB 303 are compared based on the recognition data 307 described above. This will be described in more detail with reference to FIG.

5 is a diagram for explaining a face vector stored in the database shown in FIG.

As shown in FIG. 5, face vectors for each user using a multimedia device according to an exemplary embodiment of the present invention are stored. The face vector is, for example, a data set for feature information appearing on faces of users, and is used for identifying each specific user.

6 is a view for explaining an operation of a plurality of heterogeneous image sensors connected to a multimedia device into a hardware area and a software area according to an embodiment of the present invention.

6, a configuration in which a multimedia device receives an image through a plurality of heterogeneous image sensors and performs an operation includes a hardware area 360 of the image sensor and a multimedia device for processing data received from the image sensor And a software area 350 will be described.

In FIG. 6, the hardware area 360 is shown as a separate module. However, the hardware area 360 may be integrated into a multimedia device that processes the software area 350.

First, the hardware area may include a data collection area 340 and a firmware area 330. [

The data collection area 340 receives the original data to be recognized by the multimedia device through the image sensor. The data collection area 340 includes an IR light projector, a depth image sensor, a color image sensor sensor, a microphone, and a camera chip.

In addition, the firmware area 330 is an area constituting a connection between the hardware area and the software area by being present in the hardware area and operating. In addition, it can be configured as a host application required by a specific application, and can perform downsampling, mirroring, and the like.

Accordingly, the data collection area 340 and the firmware area 330 operate in cooperation with each other, through which the hardware area 360 can be controlled. Also, the firmware area may be driven in the camera chip.

The software area 350 may also include an application programming interface (API) area 320 and a middleware area 310.

API region 320 may be implemented in the control of the multimedia device. Also, when the camera unit is configured as a separate external device from the multimedia device, the API region may be executed in a personal computer, a game console, a set-top box, or the like.

Also, the API area 320 may be a simple API that allows the multimedia device to drive the sensor in the hardware area.

The middleware area 310 may include a depth processing middleware as a recognition algorithm area. In addition, the middleware can provide an application with a clear user control API even when a user inputs a gesture through a hand or a gesture through a whole body area. In addition, the middleware region may include at least one of a middleware region, a middleware region, and a middleware region. The middleware region may include at least one of middleware, Algorithm. In addition, the algorithm can be operated using depth information, color information, infrared information, and voice information obtained in the hardware domain.

7 is a view showing a plurality of heterogeneous image sensors and a multimedia device according to an embodiment of the present invention, respectively. Hereinafter, a plurality of heterogeneous image sensors and multimedia devices according to an embodiment of the present invention will be described with reference to FIG. In FIG. 7, a plurality of heterogeneous image sensors and a multimedia device according to an exemplary embodiment of the present invention are shown separately. However, the multiple cameras may be embedded in the multimedia device.

7, the multimedia device 400 according to one embodiment of the present invention is designed as a module of a CPU (Central Processing Unit) 401 and a GPU (Graphic Processing Unit) 404, and the CPU 401 Includes an application 402 and a face recognition processing module 403. The plurality of heterogeneous image sensors 420 according to an exemplary embodiment of the present invention includes an application specific integrated circuit (ASIC) 421, an emitter 422, a first image sensor 423, 424). The multimedia device 400 and the plurality of heterogeneous image sensors 420 are connected to the wired or wireless interface 410 and may use a universal serial bus (USB) interface, for example. However, the modules of FIG. 7 are merely one embodiment, and the scope of rights of the present invention should be determined by the claims in principle.

The emitter 422 emits light to at least one or more users located around the multimedia device 400. Further, the first image sensor 423 may capture a first image using the divergent light, extract depth data from the captured first image, and use the extracted depth data to convert the at least one The user's face is detected. Also, the second image sensor 424 captures a second image of the detected face of the user, and extracts feature information from the captured second image.

Then, the extracted feature information is transmitted to the face recognition processing module 403 of the multimedia device through the interface 410. Although not shown in FIG. 7, the face recognition processing module 403 is designed to include, for example, a receiver, a memory, an extractor, a controller, and the like.

The receiving unit of the face recognition processing module 403 receives the feature information transmitted through the plurality of heterogeneous image sensors 420 and the interface 410. Further, the memory of the face recognition processing module 403 stores feature information and corresponding IDs for at least one user.

Accordingly, the extracting unit of the face recognition processing module 403 extracts an ID corresponding to the received feature information from the memory, and the control unit of the face recognition processing module 403 extracts predetermined functions corresponding to the ID It is designed to perform automatically.

On the other hand, when the face recognition processing module is designed to be performed by the CPU of the multimedia device as shown in FIG. 7, there is an advantageous effect in terms of expanding the features such as various facial recognition and various functions.

8 is a view showing a plurality of heterogeneous image sensors and a multimedia device according to another embodiment of the present invention, respectively. Hereinafter, a plurality of heterogeneous image sensors and multimedia devices according to another embodiment of the present invention will be described with reference to FIG. In FIG. 8, a plurality of heterogeneous image sensors and a multimedia device according to an exemplary embodiment of the present invention are shown separately. However, the multiple cameras may be embedded in the multimedia device.

8, the multimedia device 500 according to an exemplary embodiment of the present invention is designed as a module of a CPU (Central Processing Unit) 501 and a GPU (Graphic Processing Unit) 503, and the CPU 501 Includes an application 502. A plurality of heterogeneous image sensors 520 according to an embodiment of the present invention includes a face recognition processing module 521, an application specific integrated circuit (ASIC) 522, an emitter 523, a first image sensor 524 ), And a second image sensor 525 module. The multimedia device 500 and the plurality of heterogeneous image sensors 520 are connected to each other through a wired or wireless interface 510. For example, a universal serial bus (USB) interface may be used. However, the modules of FIG. 8 are merely one embodiment, and the scope of rights of the present invention should be determined by the claims in principle.

8 is different from FIG. 7 in that the face recognition processing module 521 is mounted on the plurality of heterogeneous image sensors 520, and the remaining description is omitted.

Meanwhile, when the face recognition processing module is designed to be performed by a plurality of heterogeneous image sensors 520 as shown in FIG. 8, it is possible to design more various types of cameras through independent platforms.

FIG. 9 is a diagram illustrating a plurality of heterogeneous image sensors according to an exemplary embodiment of the present invention in more detail. Hereinafter, referring to FIG. 9, a plurality of heterogeneous image sensors according to an embodiment of the present invention will be described in detail.

9, a plurality of heterogeneous image sensors according to an exemplary embodiment of the present invention includes a first image sensor group 610 and a second image sensor 620, a controller 630, a memory 640, An interface 650 and the like and is designed to receive audio data from the microphone 670 and the external audio source 660 under the control of the controller 630. [

The memory 640 may be, for example, a flash memory or the like, and the interface 650 may be designed with a USB interface, for example, and connected to an external multimedia device. Meanwhile, the first image sensor group 610 includes an emitter 680 and a first image sensor 690, and the emitter can be designed, for example, as an IR (Infra-Red) emitter Do.

Further, under the control of the controller 630, the light projector 682 of the emitter 680 projects the lens 681 to emit light to at least one or more users located in the vicinity of the multimedia device.

In addition, under the control of the controller 630, the first image sensor 690 captures a first image using light received through the lens 691, and extracts depth data from the captured first image And transmits it to the controller 630.

The controller 630 detects the face of the at least one user using the transmitted depth data, and then controls the second image sensor 620.

The second image sensor 620 captures a second image of the detached user's face, which is applied through the lens 621, under the control of the controller 630. Further, the second image sensor 620 transmits the feature information extracted from the photographed second image to the controller 620.

The controller 630 is designed to transmit the extracted feature information to the multimedia device using the interface 650. [ Accordingly, the multimedia device receiving the image data can quickly identify a specific user among the users stored in the DB.

10 is a view showing an example of a first image sensor among a plurality of heterogeneous image sensors according to an embodiment of the present invention. Hereinafter, with reference to FIG. 10, an example of a first image sensor among a plurality of heterogeneous image sensors according to an embodiment of the present invention will be described. The IR source 710 shown in Figure 10 may correspond to the emitter 680 of Figure 6 and the depth image processor 720 shown in Figure 10 may correspond to the first image sensor 690 of Figure 9 The description of the bar, Fig. 9 and Fig. 10 may be applied additionally. In addition, the camera shown in FIG. 10 can be designed by borrowing, for example, the structured light method described above.

As shown in FIG. 10, the IR source 710 is designed to project a coded pattern image successively to a target user 730. The depth image processor 720 estimates the position of the user using the information obtained by distorting the original pattern image by the target user 730.

11 is a view showing another example of the first image sensor among the plurality of heterogeneous image sensors according to the embodiment of the present invention. Hereinafter, another example of the first image sensor among the plurality of heterogeneous image sensors according to an embodiment of the present invention will be described with reference to FIG. The LED 810 shown in Fig. 11 may correspond to the emitter 680 of Fig. 9, and the depth image processor 820 shown in Fig. 11 may correspond to the first image sensor 690 of Fig. 9 , The description of Fig. 9 and Fig. 11 may be supplementarily applied. Further, the camera shown in Fig. 11 can be designed, for example, by using the TOF system described above.

As shown in FIG. 11, the light emitted by the LED 810 is transmitted to the target user 830. The reflected light from the target user 830 is then transmitted to the depth image processor 820. Unlike FIG. 10, the modules shown in FIG. 11 calculate the position of the target user 830 using information on a time difference. This will be described in more detail with reference to FIG.

12 is a diagram for explaining a method of calculating a distance using the first image sensor shown in FIG. Hereinafter, a method of calculating the distance using the first image sensor shown in FIG. 11 will be described with reference to FIG.

As shown in the left graph of FIG. 12, the arrival time t value can be obtained through the time difference between the emitted light and the reflected light.

12, the distance between the LED 810 and the target user 830 and the total distance from the target user 830 to the depth image processor 820 are determined by the following equation And the t value. Consequently, the distance to the LED 830 or the depth image processor 820 and the target user 830 is estimated as 1 / d.

13 is a view showing an image taken by a first image sensor among a plurality of heterogeneous image sensors according to an embodiment of the present invention. Hereinafter, a more specific image taken by the first image sensor among the plurality of heterogeneous image sensors according to an embodiment of the present invention will be described with reference to FIG.

As shown in FIG. 13, in the image photographed by the first image sensor, color information such as RGB is not displayed gorgeously. However, there is an advantage that the approximate position of an individual object can be quickly detected by expressing light and shade differentially according to the distance.

For example, as shown in FIG. 13, in the case of the farthest hall way, it is indicated by the darkest light and it is separated by about 10 m from the first image sensor. Further, in the case of a wall located at a medium distance level, it is indicated by the contrast of the middle tone, and it is confirmed that the distance is about 5 m from the first image sensor. In the case of a TV viewer located at a relatively short distance, it is displayed in the brightest tone and is separated from the first image sensor by about 4 m.

14 is a view showing an image taken by a second image sensor among a plurality of heterogeneous image sensors according to an embodiment of the present invention. Hereinafter, with reference to FIG. 14, a more specific image taken by the second image sensor among the plurality of heterogeneous image sensors according to an embodiment of the present invention will be described.

Unlike the first image sensor, since the second image sensor uses a clear RGB color, major components of the user's face can be easily identified. In particular, as illustrated in FIG. 14, the data for the eye periphery, the nose periphery, and the mouth periphery used for identifying a person are processed and designed to be used as feature information for face recognition. Of course, the eyes, nose and mouth are illustrated in FIG. 14, but other face components such as ear, forehead, hair color, wrinkle, skin color, face type, face size and the like may be used in some cases.

15 is a block diagram of a multimedia device for providing a preferred channel for each user / family unit using a plurality of heterogeneous image sensors according to an exemplary embodiment of the present invention. The multimedia device 1210 is coupled to a plurality of disparate image sensors 1240 via an interface 1250. The interface 1250 corresponds to, for example, data transmission / reception connection of wired / wireless, and more specifically, for example, a USB interface may be used. Further, the plurality of heterogeneous image sensors 1240 may be, for example, a depth camera and a color camera, and may be designed as a separate external module of the multimedia device 1210 as shown in FIG. 15, 1210 may be designed to be built-in.

The multimedia device 1210 includes a broadcast network interface 1211, a face recognition module 1212, a memory 1213, an extraction unit 1216, a control unit 1218, a display unit 1219, a speaker 1220, According to another embodiment of the present invention, the memory 1213 may further include a first generation unit 1214 and a second generation unit 1215. The extraction unit 1216 may include a determination unit 1217). However, the multimedia device shown in FIG. 15 corresponds to an embodiment for convenience of description, and the scope of rights of the present invention should be determined by the claims in principle.

The broadcast network interface 1211 receives broadcast data from the broadcast network 1230. Although not shown in FIG. 15, the video data of the received broadcast data is subjected to video processing, and then output through the display unit 1219. Further, among the received broadcast data, the audio data is subjected to audio processing, and then output through the speaker 1220.

The face recognition module 1212 identifies one or more users using information generated according to control of a first image sensor for extracting depth data and a second image sensor for extracting color data. 15, and the second image sensor corresponds to a depth camera among the plurality of heterogeneous image sensors 1240 shown in FIG. 15, and the second image sensor corresponds to the depth camera among the plurality of different image sensors 1240 shown in FIG. Or RGB camera.

The memory 1213 stores a favorite channel for each user. Based on the automatic face recognition result, Or a family-specific database.

The extraction unit 1216 accesses the memory 1213 and extracts information on a preferred channel corresponding to one or more users identified by the face recognition module 1212. [

The display unit 1219 is designed to display a guide message indicative of the corresponding preferred channel. Further, the controller 1218 controls the broadcast network interface 1211 to control switching to the corresponding preferred channel when a signal for selecting the corresponding preferred channel is applied.

Meanwhile, the face recognition module 1212 detects and recognizes one or more faces of the user from the first image captured through the first image sensor, and extracts depth data of the face of the user. Also, the face recognition module 1212 can capture the second image of the recognized face of the user using the depth data extracted by the second image sensor. In addition, the face recognition module 1212 extracts the feature information of the user face recognized from the second image photographed through the first image sensor, and accesses the memory storing the data corresponding to the extracted feature information Identify a specific user. Further, the depth data corresponds to, for example, information on the distance between the image sensor and the user or coordinate information on which the face of the user is located.

According to another embodiment of the present invention, the extracting unit 1216 is designed to further include a determining unit 1217 that determines whether the identified user is singular or plural. In addition, according to another embodiment of the present invention, the memory 1213 may store a first favorite list in which a single user stores a broadcast program of a channel viewed more than once on a specific day and at a specific time, And a second generating unit 1215 for generating a family favorite list storing a broadcast program of a channel that is simultaneously viewed by a plurality of users at least once on a specific day and at a specific time.

Accordingly, if the identified user is singular in accordance with the determination of the determination unit 1217, the extraction unit 1216 stores the individual favorite list generated by the first generation unit 1214 By accessing the memory 1213, information on the preferred channel corresponding to a single user is extracted.

If the number of the identified users is plural according to the control of the determination unit 1217, the extraction unit 1216 stores the family favorite list generated by the second generation unit 1215 The information on the preferred channel corresponding to the plurality of users is extracted.

In such a design, it is advantageous to separately manage the personalized / family preferred channel list and access it more quickly, and to improve the facial recognition accuracy and the facial recognition distance by using a plurality of heterogeneous image sensors There is also.

16 is a view illustrating a first example of a scene in which a multimedia device for providing a preferred channel per user / family unit by using a plurality of heterogeneous image sensors according to an embodiment of the present invention automatically selects a preferred channel to be. FIG. 17 is a diagram showing an example of a database storing data obtained by FIG. 16. FIG. As described above, the operation of the plurality of heterogeneous image sensors 1340 shown in FIG. 16 identifies the user A located in the vicinity of the multimedia device 1310. Further, the multimedia device 1310 and the plurality of heterogeneous image sensors 1340 according to the embodiment of the present invention may continuously monitor the user A to store the history of the user A viewing the specific channel, And creates a personal favorite list as shown in FIG.

The personal favorite list may be designed so that the user A shown in FIG. 17 stores a broadcast program of a channel viewed at least twice at a specific day and at a specific time. However, it is possible to store the details of the temporally watched contents as an error, so that an absolute amount of time (for example, watching for 30 minutes or more or 50 minutes or more) or a large amount of time (for example, 50% Or more than 75% of viewers).

18 is a view showing a second example of a scene in which a multimedia device for providing a preferred channel for each user / family unit by using a plurality of heterogeneous image sensors according to an embodiment of the present invention automatically selects a preferred channel to be. FIG. 19 is a diagram showing an example of a database storing data obtained by FIG. 18. FIG. As described above, the operation of the plurality of heterogeneous image sensors 1540 shown in Fig. 18 identifies the user B located in the vicinity of the multimedia device 1510. Fig. Further, the multimedia device 1510 and the plurality of heterogeneous image sensors 1540 according to the embodiment of the present invention continuously store the history of watching the user B by watching the user B, And creates a personal favorite list as shown in FIG. The description of the individual favorite list is similar to the description of FIGS. 16 and 17, and the same description will be omitted.

Fig. 19 is designed to additionally construct a database for user B in the database for user A stored in Fig. By designing in this way, it is possible to compare the same programs overlapping among the preferred channels for each user.

20 is a diagram illustrating a third example of a scene in which a multimedia device for providing a preferred channel for each user / family unit by using a plurality of heterogeneous image sensors according to an embodiment of the present invention automatically selects a preferred channel to be. FIG. 21 is a diagram showing an example of a database storing data obtained by FIG. 20. FIG. 16 to 19, the operation of the plurality of heterogeneous image sensors 1740 shown in Fig. 20 identifies a plurality of users A and B located in the vicinity of the multimedia device 1710. [ Further, the multimedia device 1710 and the plurality of heterogeneous image sensors 1740 according to the embodiment of the present invention continuously monitor the user A and the user B, which are the plurality of users, so that the plurality of users A and B Stores a history of watching a specific channel, and generates a family favorite list as shown in FIG.

The family favorite list may be designed to store a plurality of users A and B shown in FIG. 20 and a broadcast program of a channel viewed at least twice at a specific day and at a specific time. However, it is possible to store the details of the temporally watched contents as an error, so that an absolute amount of time (for example, watching for 30 minutes or more or 50 minutes or more) or a large amount of time (for example, 50% Or more than 75% of the time).

21 shows the family favorite list for the combination of the user A and the user B among the family members, the family favorite list for the combination of the users A, B and C among the family members, the family for the combination of the users B and D a favorite list, and a family favorite list for a combination of users A, B, C, and D that are all family members can be DB, respectively. Thus, in a combination of family members located around the multimedia device according to an exemplary embodiment of the present invention Accordingly, there is a unique effect that other preferred channel services can be performed.

22 to 24 illustrate a first example of a process of automatically switching a multimedia device for providing a preferred channel per user / family unit to a preferred channel using a plurality of heterogeneous image sensors according to an embodiment of the present invention FIG. However, as described in FIGS. 16 to 21, it is assumed that a database for a preferred channel is generated. Of course, according to another embodiment of the present invention, it is possible to design not to automatically generate the preference channel DB but to set the preference channel DB directly.

As shown in FIG. 22, when the operation of the plurality of heterogeneous image sensors 1940 identifies that the user A and the user B are located in the vicinity of the multimedia device 1910, the multimedia device 1910 displays a feedback screen Display. For example, as shown at 1950 in FIG. 22, a guide message indicating that the automatic preferred channel is being searched is displayed, or a guide message is displayed to guide that the user A and the user B are recognized.

When the 11th channel is detected as the favorite channel for the user A and the user B shown in FIG. 22, as shown in FIG. 23, the broadcast receiver 2010 transmits a guide message asking whether to switch to the favorite channel 11 2050 are output. Further, according to another embodiment of the present invention, the plurality of heterogeneous image sensors 2040 shown in FIG. 23 are continuously designed to monitor the change of users.

23, the multimedia device 2110 controls the broadcasting network interface and the like to switch to the preferred channel screen and displays a message 2150 for guiding the result to the preferred channel screen, as shown in FIG. . Further, according to another embodiment of the present invention, the plurality of heterogeneous image sensors 2140 shown in FIG. 24 are designed to continuously monitor the change of users. Of course, when there are a plurality of common favorite channels of user A and user B, a common favorite channel list in which a plurality of common favorite channels are displayed is output, and the screen is switched to a specific favorite channel screen selected by user A or user B.

Meanwhile, the common favorite channel for a plurality of users according to the embodiment of the present invention is classified and provided according to channels viewed together at the specific time zone described above, and is classified and provided according to the genre of a broadcast program viewed by a plurality of users together . That is, if the genre of the broadcast program frequently viewed by the users A and B is "Kids", the broadcast receiver may classify the broadcast programs related to the "Kids" genre and provide them as a list if both the user A and the user B are recognized .

25 to 27 illustrate a second example of a process of automatically switching a multimedia device for providing a preferred channel for each user / family unit to a preferred channel using a plurality of heterogeneous image sensors according to an embodiment of the present invention However, it is assumed that a database for a preferred channel is generated as described with reference to FIGS. 16 to 21. FIG. Of course, according to another embodiment of the present invention, it is possible to design not to automatically generate the preference channel DB but to set the preference channel DB directly.

As shown in FIG. 25, the operation of the plurality of heterogeneous image sensors 2240 detects whether the users located near the multimedia device 2210 are changed. That is, as shown in FIG. 25, when the user B of the user A and the user B who have watched the broadcast program together leave the predetermined range, there is a possibility that the favorite channel may be changed.

Here, the predetermined range may be defined as a range in which the detection of a user being viewed by the plurality of heterogeneous image sensors fails. The FOV (field-of-view) range of the plurality of heterogeneous image sensors .

On the other hand, the above-described angle (A) of the FOV (Field Of View) is defined, for example, as follows.

A = 2? = 2 tan -1 (K / 2f)

Here, f denotes the focal length, and K denotes the diagonal length of the image sensor. By using the above-described formula, it is possible to calculate an angle that varies according to the focal distance f and the diagonal length K of the sensor.

As shown in FIG. 26, when 9 is detected as the preferred channel for the user A left alone, the multimedia device 2310 outputs a guide message 2350 asking whether to switch to the preferred channel 9 . Further, according to another embodiment of the present invention, the plurality of heterogeneous image sensors 2340 shown in FIG. 26 are designed to continuously monitor whether the users are changing. That is, according to the exemplary embodiment of the present invention, when a specific user among a plurality of users viewing a broadcast program is not recognized as having departed from a preset predetermined area, the multimedia device may transmit a common preferred channel And can provide a common preferred channel list. On the other hand, in the multimedia device according to the embodiment of the present invention, when a new user who is not an existing user (s) watching a broadcast program is recognized within a preset predetermined area, To provide a common preferred channel list.

Meanwhile, provision of the common preferred channel for a plurality of users according to the embodiment of the present invention is not limited to a case where a plurality of users necessarily watch together. For example, even if the user A watches the C broadcast program on Mondays and Tuesdays, and the user B views the C broadcast program on Wednesdays and Thursdays and watches the same broadcast program frequently, the user B can be classified as the common favorite channel. That is, when users watching according to a specific broadcast program are recorded and viewed more than a predetermined number of times, the channel of the broadcast program may be classified as a common favorite channel.

27, the multimedia device 2410 controls the broadcasting network interface and the like to switch to the preferred channel screen and displays a message 2450 for guiding the result of the switching to the preferred channel screen. In the case of accepting the guide message 2350 shown in FIG. 26, . 28 to 30 illustrate a third example of a process of automatically switching a multimedia device for providing a preferred channel for each user / family unit to a preferred channel using a plurality of heterogeneous image sensors according to an embodiment of the present invention FIG. However, as described in FIGS. 16 to 21, it is assumed that a database for a preferred channel is generated. Of course, according to another embodiment of the present invention, it is possible to design not to automatically generate the preference channel DB but to set the preference channel DB directly.

As shown in FIG. 28, when the operation of the plurality of heterogeneous image sensors 2540 identifies that the user A, the user B, and the user C are located in the vicinity of the multimedia device 2510, the multimedia device 2510 Display the feedback screen. For example, as shown at 2550 in FIG. 28, a guide message indicating that the automatic preferred channel is being searched is displayed, or a guide message is displayed to guide that the user A, the user B, and the user C are recognized.

However, as a result of searching for the family favorite list or the favorite favorite list, it is possible that the user A, B, and C shown in FIG. In this case, as shown in FIG. 29, the multimedia device 2610 outputs a guide message 2650 that guides each user's preferred channel. Further, according to another embodiment of the present invention, the plurality of heterogeneous image sensors 2640 shown in FIG. 29 are designed to continuously monitor whether users are changed.

29, the multimedia device 2710 controls the broadcasting network interface and the like to switch to the preferred channel screen and guides the result to the preferred channel screen Message 2750 is output. Further, according to another embodiment of the present invention, the plurality of heterogeneous image sensors 2740 shown in FIG. 30 are designed to continuously monitor whether users are changed.

31 is a flowchart illustrating a method for providing a preference channel for each user / family unit using a plurality of heterogeneous image sensors according to an exemplary embodiment of the present invention. On the other hand, FIG. 31 to FIG. 33 can be interpreted by supplementing the description of the method invention and the description of the previous article invention.

According to another aspect of the present invention, there is provided a method of controlling a multimedia device using a plurality of heterogeneous image sensors, the method comprising: controlling a first image sensor for extracting depth data and a second image sensor for extracting color data, One or more users located in the periphery of the device are identified (S2810).

Further, the multimedia device accesses the memory storing the preferred channel for each user (S2820), and extracts information on the preferred channel corresponding to the identified one or more users from the accessed memory (S2830).

In addition, the multimedia device controls to switch to the corresponding preferred channel based on the extracted information (S2840), and monitors whether the users located in the vicinity of the multimedia device are changed (S2850).

According to another embodiment of the present invention, the step S2840 includes the steps of: displaying a guide message indicating the corresponding preferred channel; and when the signal for selecting the corresponding preferred channel is applied, And controlling the switching to the preferred channel to be performed.

According to another embodiment of the present invention, the step S2830 further includes a step of determining whether the number of the identified users is singular or plural. Further, the determining step may further include extracting information on a preferred channel corresponding to a single user by accessing a memory storing a favorite favorite list when the number of the identified users is a single number . The determining may further include extracting information on a preferred channel corresponding to a plurality of users by accessing a memory storing the family favorite list when the number of the identified users is plural .

In addition, the personal favorite list stores a broadcast program of a channel viewed by a single user at least twice on a specific day and a specific time, and the family favorite list is stored in a list of a plurality of users on a specific day and a specific time And the broadcast program of the channel that is simultaneously viewed at least two times is stored.

FIG. 32 is a flowchart illustrating in more detail step S2810 shown in FIG. 31, according to another embodiment of the present invention.

According to another embodiment of the present invention, the step S2810 shown in FIG. 31 is designed to include the steps S2811 through S2816 shown in FIG.

First, a first image located around the multimedia device is photographed using the first image sensor (S2811), and depth data related to the distance and position information of the user is extracted from the photographed first image (S2812 ).

If the user's face region is close-up using the extracted depth data, the multimedia device controls the second image sensor to capture a second image of the face of the user (S2813) And extracts feature information on the user's face (S2814).

The multimedia device accesses a memory storing data corresponding to the extracted feature information (S2815), and is designed to extract information for recognizing a specific user stored in the memory (S2816).

In this specification, both the invention and the invention of the method are explained, and the description of the two inventions can be supplementarily applied as necessary.

The method inventions according to the present invention can all be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

1210: Multimedia device
1230: Broadcast network
1240: a plurality of heterogeneous image sensors
1250: Interface

Claims (16)

A method of controlling a multimedia device using a plurality of heterogeneous image sensors,
identifying at least one user located in the periphery of the multimedia device under the control of a first image sensor for extracting depth data and a second image sensor for extracting color data;
Accessing a memory storing a preference channel for each user;
Extracting information on a preferred channel corresponding to the identified at least one user from the accessed memory;
Controlling to switch to the corresponding preferred channel based on the extracted information; And
Monitoring a change of a user located in front of the multimedia device,
Wherein the identifying comprises:
Capturing a first image located in the periphery of the multimedia device using the first image sensor;
Extracting depth data from the photographed first image;
Detecting at least one face of the user using the extracted depth data;
Capturing a second image of the detected face of the user using the second image sensor;
Extracting feature information from the photographed second image;
Accessing a memory storing data corresponding to the extracted feature information; And
Extracting information identifying a particular user stored in the memory
Further comprising a plurality of heterogeneous image sensors. delete The method according to claim 1,
Wherein the controlling comprises:
Displaying a guide message indicative of the corresponding preferred channel; And
When the signal for selecting the corresponding preferred channel is applied, controlling to switch to the corresponding preferred channel
Further comprising a plurality of heterogeneous image sensors. The method according to claim 1,
Wherein the extracting comprises:
Determining whether the identified number of users is singular or plural
Further comprising a plurality of heterogeneous image sensors. 5. The method of claim 4,
Wherein the determining step comprises:
A step of extracting information on a favorite channel corresponding to a single user by accessing a memory storing a favorite favorite list when the number of the identified users is a single number;
Further comprising a plurality of heterogeneous image sensors. 5. The method of claim 4,
Wherein the determining step comprises:
Extracting information on a preferred channel corresponding to a plurality of users by accessing a memory storing a family favorite list when the number of the identified users is a plurality
Further comprising a plurality of heterogeneous image sensors. 6. The method of claim 5,
The personal favorite list may include:
A method of controlling a multimedia device using a plurality of heterogeneous image sensors storing a broadcast program of a channel viewed by a single user at least twice at a specific day and at a specific time. The method according to claim 6,
The family favorite list includes:
A method for controlling a multimedia device using a plurality of heterogeneous image sensors storing a broadcast program of a channel viewed by a plurality of users at least two or more times at a specific day and at a specific time. In a multimedia device using a plurality of heterogeneous image sensors,
A broadcast network interface for receiving broadcast data;
a face recognition module for identifying at least one user using information generated according to a control of a first image sensor for extracting depth data and a second image sensor for extracting color data;
A memory storing user-preferred channels;
An extracting unit for accessing the memory and extracting information on a preferred channel corresponding to the identified at least one user;
A display unit for displaying a guide message indicative of the corresponding preferred channel; And
And a control unit for controlling switching to the corresponding preferred channel when a signal for selecting the corresponding preferred channel is applied,
Wherein the face recognition module comprises:
Capturing a first image located in the periphery of the multimedia device using the first image sensor;
Extracting depth data from the photographed first image,
Detecting at least one face of the user using the extracted depth data,
Capturing a second image of the detected face of the user using the second image sensor,
Extracting feature information from the photographed second image,
Accesses a memory storing data corresponding to the extracted feature information, and
And a plurality of disparate image sensors for extracting information identifying a specific user stored in the memory. 10. The method of claim 9,
Wherein the first image sensor corresponds to a depth camera and the second image sensor corresponds to an RGB camera
A multimedia device using a plurality of heterogeneous image sensors. 10. The method of claim 9,
Wherein the first image sensor and the second image sensor comprise:
Embedded in the multimedia device or designed as a separate external module
A multimedia device using a plurality of heterogeneous image sensors. delete 10. The method of claim 9,
The extracting unit extracts,
A determination unit for determining whether the number of the identified users is singular or plural,
Further comprising a plurality of disparate image sensors. 14. The method of claim 13,
The memory comprising:
A first generating unit for generating a favorite favorite list storing a broadcasting program of a channel viewed by a single user at least twice at a specific day and at a specific time; And
A second generating unit for generating a family favorite list in which a plurality of users store broadcast programs of a channel viewed at least two times at a specific day and at a specific time,
Further comprising a plurality of disparate image sensors. 15. The method of claim 14,
According to the judgment of the judging unit, when the number of the identified users is a single number,
The extracting unit extracts,
The memory storing the individual favorite list generated by the first generating unit is accessed to extract information on the favorite channel corresponding to a single user
A multimedia device using a plurality of heterogeneous image sensors. 15. The method of claim 14,
When the number of the identified users is plural according to the control of the determination unit,
The extracting unit extracts,
And accesses the memory storing the family favorite list generated by the second generation unit, thereby extracting information on a favorite channel corresponding to a plurality of users
A multimedia device using a plurality of heterogeneous image sensors.

Images