KR101758271B1 - Method for recognizing user gesture in multimedia device and multimedia device thereof - Google Patents

Abstract

The present invention relates to a method of recognizing a user gesture of a multimedia device capable of smoothly recognizing a user gesture by outputting a predetermined message to a user when the user recognizes the user gesture in the multimedia device, and a multimedia device therefor, The image sensing unit recognizes the user gesture in the image input through the image sensing unit and if the user gesture is not recognized in the image input through the image sensing unit, When the user gesture is not recognized in the multimedia device, the user can easily recognize the cause of the user gesture not being recognized and solve the user gesture A user gesture recognition method of a multimedia device, and a multimedia device therefor.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a method and apparatus for recognizing a user gesture of a multimedia device,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a multimedia device and an operation method thereof, and more particularly, to an operation method and a multimedia device therefor that can improve the usability of a user.

More particularly, the present invention relates to a method of operating a multimedia device and a multimedia device therefor by outputting a predetermined message to a user in the case of recognizing a user gesture in a multimedia device.

The multimedia device is, for example, a device having a function of receiving and processing a broadcast image that a user can view. The multimedia device displays, for example, a broadcast selected by a user among broadcast signals transmitted from the broadcast station on the display. Currently, broadcasting is changing from analog broadcasting to digital broadcasting around the world.

Digital broadcasting refers to broadcasting in which digital video and audio signals are transmitted. Compared to analog broadcasting, digital broadcasting is strong against external noise and has a small data loss, is advantageous for error correction, has a high resolution, and provides a clear screen. Also, unlike analog broadcasting, digital broadcasting is capable of bidirectional service.

Also, in order to use the digital broadcasting, the multimedia device of today has higher performance and versatility than the existing multimedia device, and various services available in the multimedia device such as Internet service, VOD (Video On Demand), and electronic album service are diversified .

However, as the various functions and services are available in the multimedia device, the conventional multimedia device has inconvenience that the user can use the desired functions through complicated operations. In order to improve the user interface, a method of using a gesture of a user as a means of inputting a command has been tried. However, according to the related art, a problem that the recognition rate of the gesture is affected according to the surrounding environment, There is a problem that it is difficult to recognize the gesture of a specific user.

Accordingly, there is a need to develop a gesture recognition method of a multimedia device for accurately recognizing a user's gesture in a multimedia device and a multimedia device therefor by solving the problems of the related art.

It is an object of the present invention to provide a multimedia device capable of acquiring information on a surrounding environment of a multimedia device and displaying a message or a menu according to the surrounding environment to accurately recognize a user gesture, And a multimedia device according to the gesture recognition method.

It is another object of the present invention to provide a gesture recognition method of a multimedia device and a multimedia device therefor that can reduce a malfunction of a multimedia device by displaying various user interface screens.

According to another aspect of the present invention, there is provided a method of recognizing a user gesture of a multimedia device, the method comprising: receiving a user image by operating an image sensing unit; Recognizing a user gesture in an image input through the image sensing unit; Displaying an image and an error message input through the image sensing unit when the user gesture is not recognized; And returning to receiving the user image.

According to another aspect of the present invention, there is provided a method of recognizing a user gesture of a multimedia device, the method comprising: receiving a user image by operating an image sensing unit; Recognizing environmental information of a photographing environment of the image sensing unit in an image input through the image sensing unit; Comparing the recognized environment information with an optimal environment for user recognition; And outputting a guide message for adjusting the photographing environment or adjusting a display screen if the comparison result does not match the optimal environment.

According to another aspect of the present invention, there is provided a multimedia device including: an image sensing unit receiving a user image; A recognition unit for recognizing a user gesture in an image input through the image sensing unit; A message generating unit for generating an error message when the recognition unit does not recognize the user gesture; And a controller for displaying the image input through the image sensing unit and the error message generated by the message generator, and controlling the image sensing unit to receive the user image again.

According to another aspect of the present invention, there is provided a multimedia device including: an image sensing unit receiving a user image; A recognition unit for recognizing environmental information of a shooting environment of the multimedia device from the image input through the image sensing unit; A storage unit for storing information on an optimal environment for the user recognition; And a control unit for displaying a guide message for adjusting the photographing environment or controlling a display screen when the environment information recognized by the recognizing unit does not match information about the stored optimal environment.

According to the embodiment of the present invention, when the user gesture is not recognized in the multimedia device, the multimedia device displays a message regarding the cause of the user gesture not being recognized, so that the user can easily input the user gesture.

According to another embodiment of the present invention, even when a plurality of users simultaneously use one multimedia device, only the gesture of a specific user can be recognized, so that a malfunction in the user gesture recognition process can be reduced.

Meanwhile, according to another embodiment of the present invention, various user interfaces can be provided in the multimedia device, and the multimedia device can be operated conveniently.

1 is a diagram schematically illustrating an example of an overall system including a multimedia device according to an embodiment of the present invention.
FIG. 2 is a diagram showing an example of the multimedia device shown in FIG. 1 in more detail.
FIG. 3 is a view showing a multimedia device using a plurality of heterogeneous image sensors according to an embodiment of the present invention and a photographing screen at the same time.
4 is a diagram illustrating a process of using detection data and recognition data in a plurality of heterogeneous image sensors and multimedia devices according to an embodiment of the present invention.
5 is a diagram for explaining a face vector stored in the database shown in FIG.
6 is a view for explaining an operation of a plurality of heterogeneous image sensors connected to a multimedia device into a hardware area and a software area according to an embodiment of the present invention.
7 is a view showing a plurality of heterogeneous image sensors and a multimedia device according to an embodiment of the present invention, respectively.
8 is a view showing a plurality of heterogeneous image sensors and a multimedia device according to another embodiment of the present invention, respectively.
FIG. 9 is a diagram illustrating a plurality of heterogeneous image sensors according to an exemplary embodiment of the present invention in more detail.
10 is a view showing an example of a first image sensor among a plurality of heterogeneous image sensors according to an embodiment of the present invention.
11 is a view showing another example of the first image sensor among the plurality of heterogeneous image sensors according to the embodiment of the present invention.
12 is a diagram for explaining a method of calculating a distance using the first image sensor shown in FIG.
FIG. 13 is a view showing an example of the multimedia device shown in FIGS. 1 and 2 in more detail.
14 illustrates a user and multimedia device in accordance with an embodiment of the present invention.
15 is a detailed block diagram of an image sensing unit of a multimedia device according to an embodiment of the present invention.
16 is a view showing an image of a depth image sensor that recognizes coordinates of each part of a user's body according to an embodiment of the present invention.
17 is a flowchart illustrating a user gesture recognition process of a multimedia device according to an embodiment of the present invention.
FIG. 18 is a flowchart illustrating a process of displaying the recognition image of FIG. 8 in detail.
FIG. 19 is a flowchart illustrating a user gesture recognition process for searching for a location change of a user of a multimedia device according to an exemplary embodiment of the present invention. Referring to FIG.
20 is a flowchart illustrating a user gesture recognition process for searching an imaging environment of a multimedia device according to an embodiment of the present invention.
FIG. 21 is a flowchart illustrating a user gesture recognition process when a plurality of users of a multimedia device according to another embodiment of the present invention is present.
22 is a view illustrating a display screen including a user gesture recognition error message according to an embodiment of the present invention.
23 is a view showing a display screen including an error message including a cropping image according to an embodiment of the present invention.
FIG. 24 is a view showing a display screen including an error message for user location change according to an embodiment of the present invention.
25 is a view illustrating a display screen including a user location guide message according to an embodiment of the present invention.
FIG. 26 is a view showing a display screen including a photographing environment adjustment message according to an embodiment of the present invention.
27 is a view illustrating a display screen including a photographing environment guide message according to an embodiment of the present invention.
28 is a view illustrating a display screen including a user selection menu according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings. Furthermore, the suffix "module" and " part "for components used in the following description are given merely for ease of description, and the" module " It can be designed in hardware or software.

Meanwhile, the multimedia device described herein corresponds to various types of devices that receive and process broadcast data, for example. Further, the multimedia device may correspond to Connected TV. In addition to the broadcast receiving function, the Connected TV is further provided with a wired / wireless communication device and the like, and is more convenient to use than a hand-held input device, a touch screen or a motion- Lt; / RTI > Also, it can be connected to the Internet and a computer by the support of a wired or wireless Internet function, and can perform functions such as e-mail, web browsing, banking or game. A standardized general-purpose OS may be used for these various functions.

Accordingly, the Connected TV can perform various user-friendly functions because various applications can be freely added or deleted on a general-purpose OS kernel, for example. More specifically, the Connected TV may be, for example, a Web TV, an Internet TV, a Hybrid Broad-Band TV (HBBTV), a Smart TV, a DTV, and the like.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG.

As used herein, terms used in the present invention are selected from general terms that are widely used in the present invention while taking into account the functions of the present invention, but these may vary depending on the intention or custom of a person skilled in the art or the emergence of new technologies. Also, in certain cases, there may be a term chosen arbitrarily by the applicant, in which case the meaning thereof will be described in the description of the corresponding invention. Therefore, it is intended that the terminology used herein should be interpreted based on the meaning of the term rather than on the name of the term, and on the entire contents of the specification.

1 is a diagram schematically illustrating an example of an overall broadcasting system including a multimedia device according to an embodiment of the present invention. The multimedia device of FIG. 1 may correspond to, for example, Connected TV, but the scope of the present invention is not limited to Connected TV, and the scope of rights of the present invention should be determined by the claims in principle.

1, an overall system including a multimedia device according to an embodiment of the present invention includes a content provider (CP) 10, a service provider (SP) 20, a network provider A network provider (NP) 30 and an HNED 40. The HNED 40 corresponds to, for example, a client 100 that is a multimedia device according to an embodiment of the present invention.

The content provider 10 produces and provides various contents. 1, a terrestrial broadcaster, a cable SO (System Operator), a MSO (Multiple System Operator), a satellite broadcaster, An Internet broadcaster, and the like may be exemplified. [0040] In addition, the content provider 10 may provide various applications in addition to the broadcast content.

The service provider 20 can provide the contents provided by the contents provider 10 in a service package. For example, the service provider 20 of FIG. 1 can package and provide the first terrestrial broadcast, second terrestrial broadcast, cable MSO, satellite broadcast, various Internet broadcasts, applications, etc. to the user.

The network provider 30 may provide a network for providing services to the client 100. The client 100 may be provided with a service by building a home network end user (HNED).

On the other hand, the client 100 can also provide the content through the network. In this case, conversely, conversely, conversely, the client 100 may be a content provider, and the content provider 10 may receive content from the client 100. [ In this case, there is an advantage that an interactive content service or a data service is possible.

FIG. 2 is a diagram showing an example of the multimedia device shown in FIG. 1 in more detail.

The multimedia device 200 includes a network interface 201, a TCP / IP manager 202, a service delivery manager 203, A demultiplexer 205, a PSIP and / or SI decoder 204, an audio decoder 206, a video decoder 207, a display unit A / and OSD Module 208, a Service Control Manager 209, a Service Discovery Manager 210, a Metadata Manager 212, an SI & Metadata DB 211, A manager 214, and a service manager 213. Further, a plurality of heterogeneous image sensors 260 are connected to the multimedia device 200, and are connected by, for example, a USB-type connection. 2, the plurality of heterogeneous image sensors 260 are designed as separate modules. However, the plurality of heterogeneous image sensors 260 may be designed to be housed in the multimedia device 200 It is possible.

The network interface unit 201 receives packets received from the network and transmits packets to the network. That is, the network interface unit 201 receives services, contents, and the like from the service provider through the network.

The TCP / IP manager 202 is involved in delivering packets from the multimedia device 200 to the multimedia device 200, that is, from the source to the destination. The service delivery manager 203 is responsible for controlling the received service data. For example, RTP / RTCP can be used to control real-time streaming data. When the real-time streaming data is transmitted using the RTP, the service delivery manager 203 parses the received data packet according to RTP and transmits it to the demultiplexer 205 or to the service manager 213 And stores it in the SI & Metadata DB 711. And feedbacks the network reception information to the server providing the service using the RTCP.

The demultiplexer 205 demultiplexes the received packets into audio, video, program specific information (PSI) data, and transmits them to the audio / video decoders 206 and 207 and the PSI & PSIP and / do.

The PSI & PSIP and / or SI decoder 204 receives and decodes a demultiplexed PSI section, a Program and Service Information Protocol (PSIP) section, or a SI (Service Information) section in the demultiplexer 205.

In addition, the PSI & PSIP and / or SI decoder 204 decodes the received sections to create a database of service information, and stores the database related to the service information in the SI & Metadata DB 211.

The audio / video decoder 206/207 decodes the video data and audio data received from the demultiplexer 205. [

The UI manager 214 provides a GUI (Graphic User Interface) for the user using an OSD (On Screen Display) or the like, and receives a key input from the user to perform a receiver operation according to the input. For example, upon receipt of a key input from the user regarding channel selection, the key input signal is transmitted to the service manager 213.

The service manager 213 controls the manager associated with the service, such as the service delivery manager 203, the service discovery manager 210, the service control manager 209, and the metadata manager 212.

The service manager 213 also creates a channel map and selects a channel using the channel map according to the key input received from the user interface manager 214. The service discovery manager 210 provides information necessary for selecting a service provider that provides the service. Upon receiving a signal regarding channel selection from the service manager 213, the service discovery manager 210 searches for the service using the information.

The service control manager 209 is responsible for selection and control of services. For example, when a user selects a service such as IGMP or RTSP when selecting a Live Broadcasting service such as an existing broadcasting method and selects a service such as VOD (Video On Demand), RTSP is used to select a service, control . The metadata manager 212 manages the metadata associated with the service and stores the metadata in the SI & Metadata DB 211.

The SI & Metadata DB 211 selects the service information decoded by the PSI & PSIP and / or SI Decoder 204, the metadata managed by the metadata manager 212, and the service provider provided by the service discovery manager 210 Store the necessary information. In addition, the SI & Metadata DB 211 may store setup data and the like for the system.

Meanwhile, the IG 250 is a gateway that collects functions necessary for accessing an IMS-based IPTV service.

The plurality of heterogeneous image sensors 260 shown in FIG. 2 are designed to capture a single image or a plurality of images for a person or object located in the vicinity of the multimedia device 200. More specifically, for example, the plurality of heterogeneous image sensors 260 are designed to operate on a single image or a plurality of images sequentially, periodically, at a selected time, or at a particular condition. A detailed description thereof will be described below.

FIG. 3 is a view showing a multimedia device using a plurality of heterogeneous image sensors according to an embodiment of the present invention and a photographing screen at the same time. Hereinafter, referring to FIG. 3, a multimedia device and a photographing screen using a plurality of heterogeneous image sensors according to an embodiment of the present invention will be described.

In general, the first image sensors associated with depth data processing are not suitable for long-distance facial recognition due to limited resolution (e.g., maximum VGA grade) and recognition distances (e.g., 3.5 m) . In addition, the second image sensors related to color data processing have a disadvantage that they are slow in recognition and are not robust to light conditions. Therefore, in order to compensate for the disadvantages of each image sensor, the multimedia device according to an embodiment of the present invention is designed to be interlocked with a hybrid image sensor module in which a first image sensor and a second image sensor are combined.

As the above-described first image sensor, for example, an IR image sensor or a depth image sensor is used. More specifically, for example, a TOF (Time Of Flight) method and a structured light method are discussed as the IR image sensor or the depth image sensor. In the TOF method, distance information is calculated using a time difference that is radiated by infrared rays. The structured light method radiates infrared rays in a specific pattern and calculates a distance by analyzing the deformed pattern. However, in the case of the first image sensor, there is an advantage in terms of depth data recognition and processing speed, and it is possible to easily sense objects and people even in a dark place. However, it has a disadvantage that the resolution is low at a long distance.

Furthermore, as the above-described second image sensor, for example, a color image sensor or an RGB image sensor is used. More specifically, for example, as the color image sensor or RGB image sensor, a stereo image sensor method and a mono image sensor method are discussed. The stereo image sensor method detects and tracks a hand or a face based on each image parallax comparison information photographed through the two image sensors. The mono image sensor method detects and tracks a hand or a face based on shape and color information shot through one image sensor. However, in the case of the second image sensor, the resolution is improved as compared with the first image sensor. However, the second image sensor is vulnerable to ambient light and is difficult to recognize in the dark. Particularly, there is a problem that accurate depth recognition is difficult.

In order to solve such conventional problems, as shown in FIG. 3, a multimedia device according to an embodiment of the present invention is designed to include both a first image sensor and a second image sensor. However, the image sensors may be designed to be embedded in the multimedia device or may be designed as a separate hardware module. First, as shown in the area (b) of FIG. 3, the first image sensor captures an image including users located in the vicinity of the multimedia device. The specific photographed images are sequentially shown in (1), (2), (3), and (4) in FIG.

On the other hand, when the photographing and data analysis of the first image sensor is completed, the second image sensor photographs an image of a specific user's face, as shown in the area (a) of FIG. The specific photographed images are sequentially shown in (5), (6), and (7) in FIG.

The first image sensor among the plurality of heterogeneous image sensors according to an embodiment of the present invention captures a first image located in the periphery of the multimedia device and extracts depth data from the captured first image. As shown in (1) of FIG. 3, it is possible to design the area of each object to be displayed in different shades according to the distance.

Further, the first image sensor can recognize and recognize at least one face of the user using the extracted depth data. That is, by using a pre-stored database or the like, Extracts the user's body information (e.g., a face, a hand, a foot, a joint, etc.) as shown in FIG. 3B, and further extracts coordinate information and distance information . More specifically, it is designed to calculate x, y, and z values, which are distance information of the face of the user, where x is the position of the face on the abscissa in the captured first image, The position of the face with respect to the vertical axis in the first image, and z denotes the distance between the face of the user and the first image sensor.

In addition, a second image sensor for extracting a color image among a plurality of heterogeneous image sensors according to an embodiment of the present invention photographs a second image of the recognized face of the user, Respectively.

On the other hand, when the second image sensor and the first image sensor shown in FIG. 3 are designed to be adjacent to each other, the error due to the physical position difference may be negligible. However, according to another embodiment of the present invention, the coordinate information or the distance information acquired by the first image sensor is corrected using the information on the physical position difference, It is designed to capture users by using information or distance information. In addition, if the first image sensor and the second image sensor are designed to be horizontal from the ground, information on the phisycal position difference may be set on the basis of a horizontal frame. The feature information is extracted from the photographed second image as shown in (3) of FIG. The feature information is, for example, data corresponding to a specific region (e.g., mouth, nose, eye, etc.) for identifying a plurality of users using the multimedia device. Furthermore, the second image sensor may zoom in on the area of the user's face based on coordinate values (x, y, z values) obtained through photographing of the first image sensor. This means a process of switching from (5) to (6) in FIG.

When the photographing and analysis of the first image sensor and the second image sensor are completed, the multimedia device according to an embodiment of the present invention accesses a memory storing data corresponding to the extracted feature information, And extracts information identifying a particular user stored in the memory.

If information identifying the specific user exists in the memory, the multimedia device provides a predetermined service for the specific user.

On the other hand, if the information identifying the particular user is not present in the memory, the multimedia device is designed to display a guide message for storing the recognized user in the memory.

As described above, according to the embodiment of the present invention, the first image sensor detects the user distance information or the coordinate information of the face, and the second image sensor uses the data obtained from the first image sensor, .

Furthermore, according to another embodiment of the present invention, the second image sensor is designed to operate only in a specific condition, not unconditionally using the second image sensor. For example, if the distance information to the user obtained by the operation of the first image sensor is less than or equal to the first reference value, or if the recognition rate of the user's face acquired through the operation of the first image sensor is equal to or greater than the second reference value, Detects and recoginizes a user face located in the vicinity of the multimedia device with only the first image sensor. On the other hand, if the distance information to the user obtained through the operation of the first image sensor exceeds the first reference value, or if the recognition rate of the user's face acquired by the operation of the first image sensor is less than the second reference value, And is further designed to recognize the user's face using the second image sensor.

According to another embodiment of the present invention, in the process of recognizing the user's face, the second image sensor zooms in using the distance information acquired through the first image sensor, And is designed to capture only the face portion using the face coordinate information acquired through the sensor.

Therefore, when a plurality of different image sensors of different types are used, it is possible to realize a long-distance facial recognition, and the data processing speed is further improved than before.

4 is a diagram illustrating a process of using detection data and recognition data in a plurality of heterogeneous image sensors and multimedia devices according to an embodiment of the present invention.

Face detection and face recognition are different processes. The face detection includes a process of detecting a face region in one image. On the other hand, the face recognition is a process of recognizing whether a detected face corresponds to a specific user. Particularly, the process of executing the face detection process using the first image sensor and executing the face recognition process using the second image sensor according to an embodiment of the present invention will be described with reference to FIG.

4, a multimedia device according to an exemplary embodiment of the present invention includes a detection module 301, a recognition module 302, a database 303, a first image sensor 304, a second image sensor 305 And the like, and uses the detection data 306 and the recognition data 307 as needed. The detection data 306 may include, for example, information-based detection techniques, feature-based detection techniques, template matching techniques, may be generated based on appearance-based detection techniques. In addition, the recognition data 307 includes data such as eyes, nose, mouth, jaw, area, distance, shape, angle and the like for identifying a specific user.

Further, the detection module 301 determines the presence of the user's face by using the image data received from the first image sensor 304. In the process of estimating the area where the user's face is located, the above-described information-based detection techniques, feature-based detection techniques, matching techniques, and appearance-based detection techniques.

The recognition module 302 identifies whether the user is a specific user by using the image data received from the second image sensor 305. [ At this time, the received image data and the face vector information stored in the DB 303 are compared based on the recognition data 307 described above. This will be described in more detail with reference to FIG.

5 is a diagram for explaining a face vector stored in the database shown in FIG.

As shown in FIG. 5, face vectors for each user using a multimedia device according to an exemplary embodiment of the present invention are stored. The face vector is, for example, a data set for feature information appearing on faces of users, and is used for identifying each specific user.

6 is a view for explaining an operation of a plurality of heterogeneous image sensors connected to a multimedia device into a hardware area and a software area according to an embodiment of the present invention.

6, a configuration in which a multimedia device receives an image through a plurality of heterogeneous image sensors and performs an operation includes a hardware area 360 of the image sensor and a multimedia device for processing data received from the image sensor And a software area 350 will be described.

In FIG. 6, the hardware area 360 is shown as a separate module. However, the hardware area 360 may be integrated into a multimedia device that processes the software area 350.

First, the hardware area may include a data collection area 340 and a firmware area 330. [

The data collection area 340 receives the original data to be recognized by the multimedia device through the image sensor. The data collection area 340 includes an IR light projector, a depth image sensor, a color image sensor sensor, a microphone, and an image sensor chip.

In addition, the firmware area 330 is an area constituting a connection between the hardware area and the software area by being present in the hardware area and operating. In addition, it can be configured as a host application required by a specific application, and can perform downsampling, mirroring, and the like.

Accordingly, the data collection area 340 and the firmware area 330 operate in cooperation with each other, through which the hardware area 360 can be controlled. Also, the firmware area may be driven in the image sensor chip.

The software area 350 may also include an application programming interface (API) area 320 and a middleware area 310.

API region 320 may be implemented in the control of the multimedia device. Also, when the image sensing unit is configured as a separate external device from the multimedia device, the API region may be executed in a personal computer, a game console, a set-top box, or the like.

Also, the API area 320 may be a simple API that allows the multimedia device to drive the sensor in the hardware area.

The middleware area 310 may include a depth processing middleware as a recognition algorithm area. In addition, the middleware can provide an application with a clear user control API even when a user inputs a gesture through a hand or a gesture through a whole body area. In addition, the middleware region may include at least one of a middleware region, a middleware region, and a middleware region. The middleware region may include at least one of middleware, Algorithm. In addition, the algorithm can be operated using depth information, color information, infrared information, and voice information obtained in the hardware domain.

7 is a view showing a plurality of heterogeneous image sensors and a multimedia device according to an embodiment of the present invention, respectively. Hereinafter, a plurality of heterogeneous image sensors and multimedia devices according to an embodiment of the present invention will be described with reference to FIG. In FIG. 7, a plurality of heterogeneous image sensors and a multimedia device according to an exemplary embodiment of the present invention are shown separately, but the multiple image sensor may be embedded in the multimedia device.

7, the multimedia device 400 according to one embodiment of the present invention is designed as a module of a CPU (Central Processing Unit) 401 and a GPU (Graphic Processing Unit) 404, and the CPU 401 Includes an application 402 and a face recognition processing module 403. The plurality of heterogeneous image sensors 420 according to an exemplary embodiment of the present invention includes an application specific integrated circuit (ASIC) 421, an emitter 422, a first image sensor 423, 424). The multimedia device 400 and the plurality of heterogeneous image sensors 420 are connected to the wired or wireless interface 410 and may use a universal serial bus (USB) interface, for example. However, the modules of FIG. 7 are merely one embodiment, and the scope of rights of the present invention should be determined by the claims in principle.

The emitter 422 emits light to at least one or more users located around the multimedia device 400. Further, the first image sensor 423 may capture a first image using the divergent light, extract depth data from the captured first image, and use the extracted depth data to convert the at least one The user's face is detected. Also, the second image sensor 424 captures a second image of the detected face of the user, and extracts feature information from the captured second image.

Then, the extracted feature information is transmitted to the face recognition processing module 403 of the multimedia device through the interface 410. Although not shown in FIG. 7, the face recognition processing module 403 is designed to include, for example, a receiver, a memory, an extractor, a controller, and the like.

The receiving unit of the face recognition processing module 403 receives the feature information transmitted through the plurality of heterogeneous image sensors 420 and the interface 410. Further, the memory of the face recognition processing module 403 stores feature information and corresponding IDs for at least one user.

Accordingly, the extracting unit of the face recognition processing module 403 extracts an ID corresponding to the received feature information from the memory, and the control unit of the face recognition processing module 403 extracts predetermined functions corresponding to the ID It is designed to perform automatically.

On the other hand, when the face recognition processing module is designed to be executed by the CPU of the multimedia device as shown in FIG. 7, there is an advantageous effect in terms of expanding the features such as various face recognition and function addition by lowering the image sensor design cost.

8 is a view showing a plurality of heterogeneous image sensors and a multimedia device according to another embodiment of the present invention, respectively. Hereinafter, a plurality of heterogeneous image sensors and multimedia devices according to another embodiment of the present invention will be described with reference to FIG. However, although a plurality of heterogeneous image sensors and a multimedia device according to an embodiment of the present invention are shown separately in FIG. 8, the multiple image sensor may be embedded in the multimedia device.

8, the multimedia device 500 according to an exemplary embodiment of the present invention is designed as a module of a CPU (Central Processing Unit) 501 and a GPU (Graphic Processing Unit) 503, and the CPU 501 Includes an application 502. A plurality of heterogeneous image sensors 520 according to an embodiment of the present invention includes a face recognition processing module 521, an application specific integrated circuit (ASIC) 522, an emitter 523, a first image sensor 524 ), And a second image sensor 525 module. The multimedia device 500 and the plurality of heterogeneous image sensors 520 are connected to each other through a wired or wireless interface 510. For example, a universal serial bus (USB) interface may be used. However, the modules of FIG. 8 are merely one embodiment, and the scope of rights of the present invention should be determined by the claims in principle.

8 is different from FIG. 7 in that the face recognition processing module 521 is mounted on the plurality of heterogeneous image sensors 520, and the remaining description is omitted.

Meanwhile, when the face recognition processing module is designed to be performed by a plurality of heterogeneous image sensors 520 as shown in FIG. 8, it is possible to design more various types of image sensors through independent platforms.

FIG. 9 is a diagram illustrating a plurality of heterogeneous image sensors according to an exemplary embodiment of the present invention in more detail. Hereinafter, referring to FIG. 9, a plurality of heterogeneous image sensors according to an embodiment of the present invention will be described in detail.

9, a plurality of heterogeneous image sensors according to an exemplary embodiment of the present invention includes a first image sensor group 610 and a second image sensor 620, a controller 630, a memory 640, An interface 650 and the like and is designed to receive audio data from the microphone 670 and the external audio source 660 under the control of the controller 630. [

According to an embodiment, the first image sensor may be a depth image sensor.

The depth image sensor is an image sensor in which a pixel value recognized in an image photographed through the depth image sensor is a distance from the depth image sensor.

The first image sensor group 610 may include an emitter 680 and a first image sensor 690 and the emitter may be designed as an Infra-Red (IR) emitter, for example Do.

 In order to acquire an image through the first image sensor group 610, an infrared ray is emitted from an emitter 680, and from the phase difference between the emitted infrared ray and infrared ray reflected from the object, A time of flight (TOF) method of acquiring distance information between image sensors and an infrared ray pattern (many infrared points) are emitted from an emitter 680, There is a structured light method of acquiring distance information between a subject and a depth image sensor based on a pattern in which the patterns are distorted by the image sensor 690.

That is, the multimedia device can grasp the distance information of the subject through the depth image sensor. Particularly, when the subject is a person, skeleton information and coordinate information of each part of the body can be obtained, and information on the concrete operation of the body can be obtained by searching for each partial movement of the body.

Further, under the control of the controller 630, the light projector 682 of the emitter 680 projects the lens 681 to emit light to at least one or more users located in the vicinity of the multimedia device.

In addition, under the control of the controller 630, the first image sensor 690 captures a first image using light received through the lens 691, and extracts depth data from the captured first image And transmits it to the controller 630.

According to an embodiment, the second image sensor 620 may be an RGB image sensor. The RGB image sensor is an image sensor that acquires color information as a pixel value.

The second image sensor 620 may include three image sensors (CMOS) that obtain information about each color R (Red), G (Green), and B (Blue).

Also, the second image sensor 620 can obtain a relatively high resolution image as compared with the depth image sensor.

The second image sensor 620 captures a second image of the subject, which is applied through the lens 621, under the control of the controller 630. Further, the second image sensor 620 may transmit the feature information extracted from the second image, to the controller 620. [0052] FIG.

In addition, when the first image sensor group 610 and the second image sensor 620 acquire the distance information of the user, the first image sensor group 610 and the second image sensor 620 continuously acquire the distance information of the user, And can recognize the operation of the user through the tracked information.

The controller 630 controls the operation of each module. That is, the controller 630 controls the first image sensor group 610 and the second image sensor 620 to photograph the subject when the photographing start signal using the image sensing unit is received, and analyzes the photographed image And may control the first image sensor group 610 and the second image sensor 620 by loading the setting information in the memory 640. [

The controller 630 is also designed to transmit the extracted feature information to the multimedia device using the interface 650. [ Accordingly, the multimedia device receiving the feature information can acquire the feature information according to the photographed image.

The memory 640 may store set values of the first image sensor group 610 and the second image sensor 620. That is, when a signal for capturing a subject is inputted using the image sensing unit from the user, the image sensing unit analyzes the image input from the image sensing unit through the controller 630, The image sensor setting values may be loaded according to the result to set the shooting environments of the first image sensor group 610 and the second image sensor 620. [

The memory 640 may be, for example, a flash memory or the like, and the interface 650 may be designed with a USB interface, for example, and connected to an external multimedia device.

Through the above-described configuration, the user can input a predetermined image and voice to the multimedia device, and can control the multimedia device through the input image or voice.

10 is a view showing an example of a first image sensor among a plurality of heterogeneous image sensors according to an embodiment of the present invention. Hereinafter, with reference to FIG. 10, an example of a first image sensor among a plurality of heterogeneous image sensors according to an embodiment of the present invention will be described. The IR source 710 shown in FIG. 10 may correspond to the emitter 680 of FIG. 9 and the depth image processor 720 shown in FIG. 10 may correspond to the first image sensor 690 of FIG. 9 The description of the bar, Fig. 9 and Fig. 10 may be applied additionally. In addition, the image sensor shown in FIG. 10 can be designed, for example, by borrowing the structured light method described above.

As shown in FIG. 10, the IR source 710 is designed to project a coded pattern image successively to a target user 730. The depth image processor 720 estimates the position of the user using the information obtained by distorting the original pattern image by the target user 730.

11 is a view showing another example of the first image sensor among the plurality of heterogeneous image sensors according to the embodiment of the present invention. Hereinafter, another example of the first image sensor among the plurality of heterogeneous image sensors according to an embodiment of the present invention will be described with reference to FIG. The LED 810 shown in Fig. 11 may correspond to the emitter 680 of Fig. 9, and the depth image processor 820 shown in Fig. 11 may correspond to the first image sensor 690 of Fig. 9 , The description of Fig. 9 and Fig. 11 may be supplementarily applied. Further, the image sensor shown in Fig. 11 can be designed by borrowing, for example, the TOF method described above.

As shown in FIG. 11, the light emitted by the LED 810 is transmitted to the target user 830. The reflected light from the target user 830 is then transmitted to the depth image processor 820. Unlike FIG. 10, the modules shown in FIG. 11 calculate the position of the target user 830 using information on a time difference. This will be described in more detail with reference to FIG.

12 is a diagram for explaining a method of calculating a distance using the first image sensor shown in FIG. Hereinafter, a method of calculating the distance using the first image sensor shown in FIG. 11 will be described with reference to FIG.

As shown in the left graph of FIG. 12, the arrival time t value can be obtained through the time difference between the emitted light and the reflected light.

12, the distance between the LED 810 and the target user 830 and the total distance from the target user 830 to the depth image processor 820 are determined by the following equation And the t value. Consequently, the distance to the LED 830 or the depth image processor 820 and the target user 830 is estimated as 1 / d.

FIG. 13 is a view showing an example of the multimedia device shown in FIGS. 1 and 2 in more detail.

The multimedia device 100 shown in FIG. 13 may be connected to a broadcast network and an Internet network. The multimedia device 100 may include, for example, a connected TV, an intelligent TV, a hybrid broad-band TV (HBBTV), a set-top box, a DVD player, a Blu-ray player,

Referring to FIG. 13, a multimedia device 100 according to an embodiment of the present invention includes a broadcast receiving unit 105, an external device interface unit 135, a storage unit 140, a user input interface unit 150, A display unit 180, an audio output unit 185, a power supply unit 190, and an image sensing unit 190. The broadcast receiving unit 105 may include a tuner 110, a demodulator 120, and a network interface unit 130. Of course, the tuner 110 and the demodulator 120 may be provided, but the network interface 130 may not be included. Alternatively, the tuner 110 and the demodulator 120 may be included, It is also possible to design the demodulation unit 120 not to include it.

The tuner 110 selects an RF broadcast signal corresponding to a channel selected by the user or all previously stored channels among RF (Radio Frequency) broadcast signals received through the antenna. Also, the selected RF broadcast signal is converted into an intermediate frequency signal, a baseband image, or a voice signal.

The tuner 110 can receive an RF broadcast signal of a single carrier according to an Advanced Television System Committee (ATSC) scheme or an RF broadcast signal of a plurality of carriers according to a DVB (Digital Video Broadcasting) scheme.

 The demodulation unit 120 may perform demodulation and channel decoding, and then output a stream signal TS. At this time, the stream signal may be a signal in which a video signal, a voice signal, or a data signal is multiplexed. For example, the stream signal may be an MPEG-2 TS (Transport Stream) multiplexed with an MPEG-2 standard video signal, a Dolby AC-3 standard audio signal, or the like.

The stream signal output from the demodulation unit 120 may be input to the controller 170. The control unit 170 performs demultiplexing, video / audio signal processing, and the like, and then outputs the video to the display unit 180 and audio to the audio output unit 185.

The external device interface unit 135 can connect the multimedia device 100 with an external device.

The external device interface unit 135 can be connected to an external device such as a DVD (Digital Versatile Disk), a Blu-ray, a game device, an image sensor, a camcorder, a computer (notebook) The external device interface unit 135 transmits external video, audio, or data signals to the controller 170 of the multimedia device 100 through a connected external device. Also, the control unit 170 can output the processed video, audio, or data signal to the connected external device. To this end, the external device interface unit 135 may include an A / V input / output unit (not shown) or a wireless communication unit (not shown).

The A / V input / output unit includes a USB terminal, a CVBS (Composite Video Banking Sync) terminal, and a component (not shown) so that the video and audio signals of an external device can be input to the multimedia device 100. [

(Analog) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (High Definition Multimedia Interface) terminal, an RGB terminal, a D-SUB terminal, and the like.

The wireless communication unit can perform short-range wireless communication with other electronic devices. The image display device 100 may be a digital camera such as Bluetooth, Radio Frequency Identification (RFID), infrared data association (IrDA), Ultra Wideband (UWB), ZigBee, Digital Living Network Alliance ) According to the communication standard.

Also, the external device interface unit 135 may be connected to the various set-top boxes via at least one of the various terminals described above to perform input / output operations with the set-top box.

The network interface unit 130 provides an interface for connecting the multimedia device 100 to a wired / wireless network including the Internet network. The network interface unit 130 may include an Ethernet terminal or the like for connection to a wired network and may be a WLAN (Wireless LAN) (Wi- Fi wireless broadband (Wibro), World Interoperability for Microwave Access (Wimax), and HSDPA (High Speed Downlink Packet Access) communication standards.

The network interface unit 130 can transmit or receive data with other users or other electronic devices via a network connected thereto or another network linked to the connected network.

The storage unit 140 may store a program for each signal processing and control in the control unit 170 or may store the processed video, audio, or data signals.

The storage unit 140 may also function to temporarily store video, audio, or data signals input from the external device interface unit 135 or the network interface unit 130. In addition, the storage unit 140 may store information on a predetermined broadcast channel through the channel memory function.

In addition, the storage unit 140 may store control information and a command database for user gesture recognition of the multimedia device.

The control information is information necessary for combining a plurality of pieces of information obtained from the image when a predetermined image is input through the image sensing unit 190 of the multimedia device.

That is, the control information includes a plurality of image sensors capable of acquiring different information from the image sensing unit 190, and when combining information obtained from the respective image sensors, According to the distance, information on the weight of the information obtained through each image sensor may be included in the gesture recognition object (hand, foot, head) by which the user inputs the gesture.

In addition, the instruction database stores mapping data for a specific instruction corresponding to a user gesture.

That is, the command database stores feature information of each user gesture, and stores a list of executable operations in association with each feature information in the multimedia device. Accordingly, when the user gesture image is inputted through the image sensing unit 190 and the feature information is extracted from the input image, the multimedia device stores the specific information corresponding to the feature information in the database stored in the storage unit 140 The command can be retrieved and executed.

In addition, the storage unit 140 may store a database 141 including data necessary for recognizing users and user gestures in the multimedia device.

That is, data on the user recognition information and data on the user gesture recognition information can be stored in the database 141. When predetermined information is acquired from the image input through the image sensing unit 190, It is possible to search the database 141 based on the user and specify an instruction corresponding to the user gesture.

In addition, shape information and color information of each object can be stored through the database 141. [ That is, for example, the database 141 is searched based on the size information and the type information extracted from the image input through the image sensing unit 190, and an object or a user included in the input image is recognized . That is, the database 141 may store size, shape, vector, and color information for each user's face.

In addition, the database 141 may store the feature information of the expected recognition obstacle when the user gesture recognition is not completed in the multimedia device. Wherein the expected recognition obstacle is located between the user and the image sensing unit of the multimedia device so as to block the user gesture recognition of the multimedia device by blocking the portion of the user inputting the user gesture from the image sensing unit of the multimedia device .

That is, objects such as a bowl, a trash can, a cup, and a water bottle may be included in the anticipated recognition obstacle as an object whose size is within a certain range and whose position can be easily changed.

Accordingly, the multimedia device stores feature information on the expected recognition obstacles in the database 141, and when the subject is recognized according to the analysis of the image input from the multimedia device, It is possible to compare the feature information of the expected recognition obstacles of the database and search whether there is an object to be matched.

In addition, the database 141 may store environment information on the optimum environment in user gesture recognition of the multimedia device. The optimal environment may be different for each content or service available in the multimedia device. For example, the distance between the user of the optimal environment and the multimedia device for inputting the user gesture for adjusting the channel of the terrestrial broadcast in the multimedia device may be determined by the user of the optimal environment for controlling the game character of the game contents available in the multimedia device May be closer than the distance between the multimedia devices.

The storage unit 140 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD Memory, etc.), RAM, ROM (EEPROM, etc.), and the like.

FIG. 13 shows an embodiment in which the storage unit 140 is provided separately from the control unit 170, but the scope of the present invention is not limited thereto. The storage unit 140 may be included in the controller 170.

The user interface unit 150 transmits a signal input by the user to the control unit 170 or a signal from the control unit 170 to the user.

For example, the user interface unit 150 may be configured to turn on / off the power, select the channel, and set the screen from the remote control device 200 according to various communication methods such as a radio frequency (RF) communication method and an infrared Or the like, or to transmit a control signal from the control unit 170 to the remote control device 200. [0064]

For example, the user interface unit 150 may transmit a control signal input from a sensing unit (not shown) that senses a user's gesture to the control unit 170, (Not shown). Here, the sensing unit (not shown) may include a touch sensor, an audio sensor, a position sensor, an operation sensor, and the like.

The control unit 170 demultiplexes the input stream or processes the demultiplexed signals through the tuner 110 or the demodulation unit 120 or the external device interface unit 135 to generate a signal for video or audio output Can be generated and output.

The video signal processed by the controller 170 may be input to the display unit 180 and displayed as an image corresponding to the video signal. The video signal processed by the controller 170 may be input to the external output device through the external device interface 135. [

The audio signal processed by the control unit 170 may be output to the audio output unit 185 through audio. The voice signal processed by the control unit 170 may be input to the external output device through the external device interface unit 135.

The display unit 180 converts an image signal, a data signal, an OSD signal processed by the control unit 170 or a video signal and a data signal received from the external device interface unit 135 into R, G, and B signals, respectively Thereby generating a driving signal.

Meanwhile, a sensing unit (not shown) having at least one of a touch sensor, a voice sensor, a position sensor, and an operation sensor may be further included in the image display device 100 to detect a user's gesture. The signal sensed by the sensing unit (not shown) may be transmitted to the controller 170 through the user interface unit 150.

The image sensing unit 190 may further include an image sensing unit 190 for sensing a user. The image information photographed by the image sensing unit 190 may be input to the controller 170.

The image sensing unit 190 may include a plurality of cameras capable of acquiring different information, which will be described in detail with reference to FIG.

In addition, the image sensing unit 190 may track the position of the user by continuously acquiring distance information of the recognized user when the user is recognized. That is, when the user moves, information on the movement path, the movement direction, and the movement degree can be obtained.

The control unit 170 may sense the gesture of the user by combining the images captured from the image sensing unit 190 or the sensed signals from the sensing unit (not shown).

Accordingly, the control unit 170 includes a recognition unit 171 for recognizing user and user gestures in the image input through the image sensing unit, and a message generating unit 172 for generating a message related to the user and the user gesture .

The recognition unit 171 may store data on user recognition information and user gesture recognition information in a database and if predetermined information is acquired from the image input through the image sensing unit 190, The user and the user gesture can be recognized.

In addition, the recognition unit 171 can recognize environment information on the shooting environment of the multimedia device in the image input through the image sensing unit of the multimedia device. That is, the multimedia device analyzes the inputted image and obtains information about the distance between the user and the multimedia device, the user location, the location of the light source, the brightness and the intensity in the photographing environment.

 The message generating unit 172 may generate a related message in the user and user gesture recognition process of the multimedia device. The message may include an error message that is displayed when the multimedia device does not recognize the user and the user gesture.

If the user and the user gesture are not recognized, the message generating unit 172 recognizes an object included in the input image through the image sensing unit 190, An expected error of the recognition obstacle may be searched to generate an error message including information on the searched recognition obstacle.

Size information, vector information, and color information of each object included in the image input through the image sensing unit 190 may be extracted and used for the object recognition and the recognition obstacle search.

That is, based on the extracted information, the error message may be generated by searching the database 141 for storing information on an object and information on a predicted recognition obstacle stored in the storage unit 140.

In addition, if the user and the user gesture are not recognized, the message generating unit 172 may generate an error message including tracking information of the user position of the image sensing unit 190. [

That is, according to the tracking information, when the user is out of the recognizable area of the image sensing unit 190, information including the current user position, a guide message for moving the user to the recognizable area, Can be generated. That is, the guide message may include information on directions, distances, and the like to which the user should move.

 The message generation unit 172 may generate the optimal distance information for recognizing the user gesture of the multimedia device or the user of the multimedia device stored in the storage unit 140 and the user distance information obtained through the image sensing unit 190 And generate an error message including the guide message according to the comparison result. That is, the guide message may include information on directions, distances, and the like to which the user should move.

The message generating unit 172 generates environment information on the shooting environment of the multimedia device recognized by the recognizing unit 171 and environment information on the optimum environment for user gesture recognition stored in the storage unit 140 And may include a guide message for adjusting the shooting environment of the multimedia device when the environment information does not match. That is, the guide message may include a message requesting the user to brighten the illumination around the multimedia device.

The control unit 170 may control the display unit 180 to display the image input through the image sensing unit 190 and the error message generated by the message generating unit 172, After the image and the error message are displayed, it is possible to control the image sensing unit 190 to capture an image for user gesture recognition.

The control unit 170 crops an image of a portion of the image input through the image sensing unit 190 through which the recognized obstacle is located through the message generating unit 172, And display the pinged image and the error message together.

In addition, the control unit 170 searches the face of the user in the image input through the image sensing unit 190, and when the plurality of user faces are searched, the image of the portion including each recognized user face is cropped Cropping the face image, displaying the cropped facial image through the display unit 180, receiving one of the facial image selection signals, and recognizing the user's gesture corresponding to the selected facial image can do.

The control unit 170 compares the environmental information of the shooting environment of the multimedia device recognized by the recognition unit 171 with the environment information of the optimum environment for user gesture recognition stored in the storage unit 140, If the environment information does not match, it is possible to control an operation necessary for adjusting the shooting environment of the multimedia device. That is, when the shooting environment of the multimedia device is dark as a result of the comparison of the environment information, a bright screen is displayed on the display unit of the multimedia device, and the brightness of the area photographed by the multimedia device can be brightened.

The remote control device 200 transmits the user input to the user interface unit 150. [ To this end, the remote control apparatus 200 can use Bluetooth, RF (radio frequency) communication, infrared (IR) communication, UWB (Ultra Wideband), ZigBee, or the like.

Also, the remote control device 200 can receive the video, audio or data signal output from the user interface unit 150 and display it on the remote control device 200 or output sound or vibration.

14 illustrates a user and multimedia device in accordance with an embodiment of the present invention.

According to the embodiment, the multimedia device 1900 can acquire the user image through the image sensing units 1901 and 1902 of the multimedia device to recognize the user 1903.

In addition, the image sensing unit may include two image sensor modules 1901 and 1902 that acquire different information from each other to accurately recognize a user. That is, according to an embodiment, the image sensing unit may include a depth image sensor, and an RGB image sensor (RGB cam). This is described in detail in FIG.

As shown in the figure, the image sensing units 1901 and 1902 of the multimedia device 1900 are positioned at the lower end of the multimedia device and are easy to detect the user's body center, , It is possible to recognize smooth user and user gesture.

Through the above-described configuration, the multimedia device acquires information on the user and the user location, and can provide a viewing environment suitable for the user based on the obtained information.

15 is a diagram illustrating an image of a depth image sensor that recognizes coordinates of each part of a user's body according to an embodiment of the present invention.

Referring to FIG. 15, the multimedia device can acquire coordinate information for each part of the user's body through the image photographed through the depth image sensor.

That is, when the user 2401 is photographed through the depth image sensor, the depth image sensor can obtain distance information of each part of the user's body as image information.

For example, coordinate information for only the right elbow 2402 of the user's body can be obtained. In addition, various numerical information such as a head size, a shoulder width, and an arm length of the user can be obtained through the obtained distance information of each part of the user's body, thereby enabling the user to be recognized.

When the face of the user is to be recognized, the coordinates of the head part of the obtained distance information is transmitted to the RGB image sensor of the image sensing part, so that the RGB image sensor detects the head part of the user according to the coordinates of the head part To acquire a zooming image for the object.

In addition, when a user moves a body and performs a specific action, the movement of the distance information of each part of the body can be sensed to recognize the action.

16 is a flowchart illustrating a user gesture recognition process of a multimedia device according to an embodiment of the present invention.

According to the embodiment, the multimedia device receives the user gesture image through the image sensing unit of the multimedia device (S2001).

When the selection signal of the menu item or the hot-key input signal for starting the user gesture recognition process is received from the user, the multimedia device starts shooting the image for receiving the user gesture through the image sensing unit.

In addition, the image sensing unit may include a plurality of image sensors for acquiring different information from each other for accurate user gesture recognition. That is, for example, it is possible to acquire distance information and distance information for a user through the depth image sensor, including a depth image sensor and an RGB image sensor, and acquire color information for a user through the RGB image sensor can do.

That is, the user may input a predetermined user gesture for controlling the multimedia device when the multimedia device starts capturing an image.

Next, the multimedia device recognizes the user gesture in the image input through the image sensing unit of the multimedia device (S2002).

When a user gesture image is input through the image sensing unit of the multimedia device, the multimedia device extracts the feature information of the user gesture included in the image and searches for a specific command corresponding to the extracted feature information. The specific command includes an instruction to execute a certain function in the multimedia device, or to receive any service or content.

If the specific command is found, the multimedia device executes the searched specific command (S2004). Through this, the user can input a user gesture, thereby allowing the multimedia device to perform a desired operation.

On the other hand, if the feature information of the user gesture is not extracted in the recognition step (S2002), or if there is no specific instruction corresponding to the extracted feature information, the image input through the image sensing unit of the multimedia device and the error Message is displayed (S2003).

That is, in the image inputted through the image sensing unit of the multimedia device, when the other obstacle is located and the user does not input the image of the part where the user's gesture is input, or the image sensing environment is dark, The user can not execute a command corresponding to the user gesture inputted by the user and can output an error message indicating that the user can not recognize the user gesture.

In addition, an image input through the image sensing unit of the multimedia device can be output together with the error message. This will be described in detail below with reference to FIG.

In addition, an image input through the image sensing unit of the multimedia device can crop and display a part of the image input through the image sensing unit. This will be described in detail below with reference to FIG. 17 and FIG.

In this case, when the user gesture recognition is not proceeded in inputting the user gesture to the multimedia device, the user can check the image inputted through the image sensing unit of the multimedia device together with the error message, It is possible to easily carry out the operation necessary for the operation.

FIG. 17 is a flowchart illustrating a process of displaying the recognition image of FIG. 15 in detail.

According to the embodiment, if the user gesture is not recognized in the recognition step S2002 of FIG. 15, the multimedia device analyzes the image input through the image sensing unit of the multimedia device to recognize the user gesture (S2101 ).

The multimedia device may search for information about each object included in the image for recognizing the user gesture. To this end, the multimedia device extracts size information, shape information, and color information of each object from the image to recognize which object or user is included in the image, and based on the extracted information, Can search for an object or a user, respectively.

That is, for example, when the subject whose size and shape matches the face of the person is searched according to the size information and the type information extracted from the image, the subject can be recognized as a person. In addition, for this purpose, the multimedia device can store shape information and color information of each object through a database.

Next, when it is determined that each object included in the image is an object or a user, the multimedia apparatus searches for a recognized recognition obstacle based on the recognized object (S2102).

The multimedia device can store the feature information of the expected recognition obstacle in the case where the user gesture recognition is not completed in the multimedia device through the database.

Wherein the expected recognition obstacle is located between the user and the image sensing unit of the multimedia device so as to block the user gesture recognition of the multimedia device by blocking the portion of the user inputting the user gesture from the image sensing unit of the multimedia device .

That is, objects such as a bowl, a trash can, a cup, and a water bottle may be included in the anticipated recognition obstacle as an object whose size is within a certain range and whose position can be easily changed.

Accordingly, the multimedia device may store the feature information on the expected recognition obstacles in the database, and if the subject is recognized according to the analysis of the image input from the multimedia device, The feature information of the recognition obstacles is compared with each other to search for a matching object.

That is, for example, when an image input through the image sensing unit of the multimedia device is analyzed, if a user, a table, a frame, and a bowl for inputting a user gesture are recognized as a subject included in the image, When a bowl of the recognized objects exists between the user and the image sensing unit of the multimedia device, the vessel can be searched as a predicted recognition obstacle.

Next, when the expected recognition obstacle is found, the multimedia device crops the image of the part where the expected recognition obstacle exists and displays it together with the error message (S2103).

That is, when a predicted recognition obstacle is searched in an image input through the image sensing unit of the multimedia device, the multimedia device searches a part of the image where the expected recognition obstacle is located in the image, And temporarily stores the image in the memory.

In addition, an error message including information on a predicted recognition obstacle included in the cropping image may be generated based on the temporarily stored cropping image and displayed together with the cropping image.

The display screen related to this will be described in detail below with reference to FIG.

When the steps are completed, the multimedia device returns to the step of receiving the user gesture image through the image sensing unit of the multimedia device, and performs the operation for recognizing the user gesture.

Through this, the user can check the information and the image of the expected recognition obstacle together with the error message, so that the user can more easily perform the operation for smooth user gesture input.

FIG. 18 is a flowchart illustrating a user gesture recognition process for searching for a user's location movement in a multimedia device according to an exemplary embodiment of the present invention.

According to the embodiment, the multimedia device receives the user gesture image through the image sensing unit of the multimedia device (S2201).

The step of receiving the user gesture image (S2201) is the same as the step of receiving the user gesture image of FIG. 8 (S2001), and a detailed description thereof will be omitted.

Next, the multimedia device tracks the position of the user inputting the user gesture (S2202).

That is, if the distance information of the user is recognized in the image input step S2201, the multimedia device continuously tracks the distance to the user through the image sensing unit of the multimedia device, thereby tracking the position of the user.

For example, it is possible to track the user position by continuously photographing an image including the user's distance information through a depth camera of the image sensing unit, and can measure the distance information, And stores the information about it.

Next, the user gesture is recognized through the input image through the image sensing unit of the multimedia device (S2203).

The step S2203 of recognizing the user gesture and the step S2206 of executing a corresponding command when the user gesture is recognized are the same as the steps 2002 and S2004 of FIG.

On the other hand, if the user gesture is not recognized in step S2203, it is determined whether the tracked user position is located in a recognizable area through the image sensing unit of the multimedia device (S2204).

The multimedia device has a limited area that can be photographed through the image sensing unit of the multimedia device. That is, since only an image within a photographing angle range of the image sensor included in the image sensing unit can be photographed, when the user moves in a range other than the photographing angle, the user can not photograph the image.

Therefore, the multimedia apparatus searches based on the tracked user position whether the point where the current user is located is within the shooting angle range of the image sensing unit of the multimedia device. In addition, when the point where the current user is located is outside the photographing angle range, it is possible to specifically search whether the point where the current user is located deviates from the photographing angle range in which direction.

Next, the multimedia device displays an error message including information on the searched user location (S2205). In addition, an image input through the image sensing unit of the multimedia device can be displayed together with the error message.

That is, the error message may include merely an error message indicating that the user position is out of the recognizable area of the image sensing unit of the multimedia device, and the user position may include an image capturing angle of the image sensing unit of the multimedia device And may include an error message that includes information as to which direction from the range, and to what extent, is out of range. This will be described in detail below with reference to FIGS. 23 and 24. FIG.

Accordingly, the multimedia device can display the error message as described above. The user can easily recognize that his / her location is out of the area where the user gesture can be recognized through the image sensing unit of the multimedia device, Thereby performing an operation for inputting a user gesture.

19 is a flowchart illustrating a user gesture recognition process for searching an imaging environment of a multimedia device according to an embodiment of the present invention.

According to the embodiment, the multimedia device receives the user gesture image through the image sensing unit of the multimedia device (S2301).

The step of receiving the user gesture image (S2301) is the same as the step of receiving the user gesture image of FIG. 8 (S2001), and a detailed description thereof will be omitted.

Next, the environment information of the shooting environment of the multimedia device is retrieved from the image input through the image sensing unit of the multimedia device (S2302).

The environment information includes all the information about the photographing environment using the image sensing unit of the multimedia device. The environment information includes a distance between the image sensing unit of the multimedia device and the user, a user location, a location, type, And the like.

Accordingly, the multimedia device can analyze the input image through the image sensing unit of the multimedia device to acquire the environment information.

Next, based on the obtained environment information, it is determined whether the shooting environment is an environment suitable for image recognition including a user gesture (S2303).

The multimedia device may store environment information on the optimum environment in the user gesture recognition in the multimedia device through the database. In addition, the optimum environment may be different for each content or service available in the multimedia device. For example, the distance between the user of the optimal environment and the multimedia device for inputting the user gesture for adjusting the channel of the terrestrial broadcast in the multimedia device may be determined by the user of the optimal environment for controlling the game character of the game contents available in the multimedia device May be closer than the distance between the multimedia devices.

Accordingly, the multimedia device compares the environment information obtained in the environment information searching step (S2302) with the environment information on the optimal environment, and determines that the shooting environment of the current multimedia device is the optimal environment for recognizing the user gesture in the multimedia device It is possible to judge whether or not they match.

In addition, if the environment information of the shooting environment of the multimedia device coincides with the environment information of the optimal environment, the multimedia device continues the operation of recognizing the user gesture in the image input through the image sensing unit of the multimedia device S2305).

The step S2305 of continuing the operation for recognizing the user gesture is the same as the operation after the step S2002 of recognizing the user gesture in Fig.

Accordingly, the multimedia device can extract the feature information of the user gesture from the image input through the image sensing unit of the multimedia device, search the specific instruction corresponding to the extracted feature information, and execute the searched specific instruction.

On the other hand, if it is determined in step S2303 that the environment of the multimedia device is not suitable for recognizing the user gesture, the multimedia device may output a notification message in step S2304.

The notification message includes information on the shooting environment of the multimedia device, information on the optimal environment in the user gesture recognition of the multimedia device, information on the comparison result between the shooting environment of the multimedia device and the optimal environment, and the like . The notification message will be described later in detail with reference to FIG. 24 to FIG.

In addition, according to an embodiment, the multimedia device can display an image input through the image sensing unit of the multimedia device together with the notification message.

In addition, according to the embodiment, the multimedia device may display the color of one director on the display screen of the multimedia device to correct the shooting environment of the multimedia device. That is, for example, when the brightness of the photographing environment is low as a result of comparing the photographing environment and the environment information of the optimal environment, the multimedia device displays a white screen on the display unit of the multimedia device, The environment can be corrected.

Next, the multimedia device can take an image including the user gesture again through the image sensing unit of the multimedia device.

Accordingly, the user can recognize the information about whether the shooting environment of the current multimedia device is suitable in advance before recognizing the user gesture in the multimedia device, so that the user can perform an operation for correct user gesture recognition.

20 is a flowchart illustrating a user gesture recognition process when a plurality of users of a multimedia device according to another embodiment of the present invention are present.

According to the embodiment, a plurality of users can simultaneously receive contents or services using the multimedia device. However, when a predetermined command is input to the multimedia device using the user gesture, a user having control over the multimedia device can be specified for smooth command input.

Accordingly, the image is received through the image sensing unit of the multimedia device (S2401), and the face of the user is recognized in the input image (S2402).

The step of receiving the image (S2401) is the same as the step of receiving the user gesture image of FIG. 16 (S2001), and a detailed description thereof will be omitted.

In addition, the multimedia device extracts the feature information of the objects included in the input image, compares the extracted feature information with the feature information of the face of the person, and displays the object whose feature information matches the face of the user . The feature information may include information on the size and shape of the object.

Accordingly, the multimedia apparatus extracts an image of a portion where the user's face is located, based on the feature information, from the image input through the image sensing unit of the multimedia device, and recognizes the user corresponding to the extracted facial image .

That is, in the extracted facial image, distance information and vector information for each part of the face are acquired, and a database storing facial information of each user is retrieved based on the obtained information, Can be recognized.

Next, it is determined whether the recognized user face is plural (S2403).

That is, since a plurality of users can simultaneously use contents or services through one multimedia device, a plurality of users may exist in an image input through the image sensing unit of the multimedia device.

Accordingly, if a plurality of users exist as a result of the determination, the user list including the facial images of the recognized users is displayed (S2404).

The multimedia device may display a user selection menu including the user list to receive a selection signal of a user having control of inputting a user gesture among a plurality of users using a predetermined content or service through the multimedia device have. This will be described in detail below with reference to FIG.

Next, the multimedia device receives one or more user selection signals from the users included in the user list (S2405).

The user can select a user having the control right to input the user gesture through a local key input of a predetermined remote controller or a multimedia device.

In addition, the multimedia apparatus identifies a user having the control right among the plurality of users when the selection signal is received, and recognizes only the user gesture inputted from the selected user as a control signal of the multimedia apparatus.

On the other hand, when only one face of the user is recognized in the determination step S2403, it is not necessary to specify a user having the control right, and the user gesture of the recognized one user is recognized as a control signal of the multimedia device .

That is, even when a plurality of users simultaneously use one multimedia device through the steps shown in FIG. 20, the multimedia device can specify a user who inputs the user gesture, and can recognize recognition rate and reliability in user gesture recognition There is an effect that can be improved.

21 is a view showing a display screen 2501 including a user gesture recognition error message according to an embodiment of the present invention.

According to an embodiment, when a selection signal of a menu item or a hot-key input signal for starting a user gesture recognition process is received from a user, the multimedia device captures an image for receiving a user gesture through the image sensing unit And recognizes the user's gesture. If the user gesture can not be recognized in the input image, the user can display the error message 2502.

The error message 2502 may include an image 2603 captured through the image sensing unit of the multimedia device.

That is, the user can check the image 2503 captured through the image sensing unit of the multimedia device together with the error message through the display screen as shown in FIG. 21, It is possible to accurately recognize the cause of occurrence of the error message.

The user removes a cause of the multimedia device not recognizing the user gesture based on the error message 2502 and inputs a selection signal of the confirmation menu item 2504 so that the multimedia device displays the user gesture It can be re-recognized.

22 is a view showing a display screen 2601 including an error message including a cropping image according to an embodiment of the present invention.

According to the embodiment, when the user gesture is not recognized in the image input through the image sensing unit of the multimedia device, the multimedia device may crop the portion of the image captured by the image sensing unit of the multimedia device, and an error message 2602 for user gesture recognition including an image 2603 cropped.

That is, as shown in FIG. 17, when the user gesture is not recognized in the image input through the image sensing unit of the multimedia device, the multimedia device displays the image gesture in the image input through the image sensing unit of the multimedia device It is possible to search for information about a subject, recognize each of the subjects, and identify an expected recognition obstacle among the recognized subjects. In addition, it is possible to crop only the image of the portion where the expected recognition obstacle is located in the image input through the image sensing unit of the multimedia device, and to display the cropping image in the error message.

That is, since the multimedia device can provide the display screen as shown in FIG. 22, the user can view the clipping image 2603 for the expected recognition obstacle among the images captured through the image sensing unit of the multimedia device through the error message 2602 ) Can be checked together, so that the cause of the occurrence of the error message can be recognized more accurately.

The user removes the cause of the multimedia device not recognizing the user gesture based on the error message 2602 and inputs a selection signal of the confirmation menu item 2604 so that the multimedia device displays the user gesture It can be re-recognized.

23 is a view showing a display screen 2701 including an error message for user location change according to an embodiment of the present invention.

According to an embodiment, the multimedia device may track the user location through the image sensing unit of the multimedia device and may display an alert message 2702 if the user location is outside the area recognizable by the image sensing unit have.

Through the warning message 2702, the user can recognize that his or her own position is out of the recognition area of the image sensing unit, and perform an operation to move back to the recognition area of the image sensing unit.

Also, when a selection signal of the confirmation menu item 2703 included in the warning message 2702 is received from the user, the image for receiving the user gesture again is shot through the image sensing unit, and the selection of the cancel menu item 2704 When a signal is received, the operation for the user gesture input can be canceled.

Accordingly, when the user moves out of the recognizable area of the image sensing unit of the multimedia device during input of the user gesture, the user can recognize that the position is out of order and return to the position where the user gesture can be input again.

24 is a view illustrating a display screen 2701 including a user location guide message according to an embodiment of the present invention.

According to an embodiment, a multimedia device may acquire information on a user's position of a user inputting a user gesture through an image sensing unit of the multimedia device, and the obtained user distance information may be optimized for recognizing the user gesture If it does not match the user location, a guide message 2702 for adjusting the user location to the optimal location may be displayed.

The guide message may include information that can recognize whether the user is out of the optimal user location.

In addition, the user can recognize the movement path and the movement distance for moving the current user position to the optimum position through the guide message 2802. [

Also, if the confirmation menu item 2803 is selected from the user, the multimedia device may start the resampling through the image sensing unit of the multimedia device, and continue the operation for recognizing the user gesture.

Through this, the user can recognize whether the current user's position is appropriate, and move to a position suitable for inputting the user's gesture to perform a smooth user's gesture input.

25 is a view showing a display screen 2901 including a photographing environment adjustment message according to an embodiment of the present invention.

According to an embodiment of the present invention, a multimedia device searches environment information of a shooting environment of a multimedia device in an image input through an image sensing unit of a multimedia device, compares the searched environment information with optimal environment information for user gesture recognition The photographing environment can be adjusted, and the confirmation menu 2902 can be included in adjusting the photographing environment.

In addition, the environment information may include information on the brightness of the photographed image.

That is, for example, when the multimedia device searches the environment information of the shooting environment of the input image through the image sensing unit of the multimedia device, if the brightness of the image is dark, the multimedia device turns on the illumination unit of the multimedia device , The brightness of the shooting environment can be improved by displaying a display screen including bright colors on the display unit of the multimedia device.

However, it is possible to receive a user's confirmation command through the confirmation menu 2902 before performing the operation of adjusting the photographing environment. Accordingly, when the confirmation menu item 2903 of the confirmation menu 2902 is selected from the user, the multimedia device performs an operation for adjusting the shooting environment. On the other hand, when the cancel menu item 2904 of the confirmation menu 2902 is selected from the user, an operation for recognizing the user gesture in the current state is performed without performing an operation for adjusting the shooting environment.

Accordingly, the user can recognize, through the confirmation menu 2902, how the image input to the multimedia device has a problem, and to inform the multimedia device of what action the multimedia device will perform in order to overcome the difference . ≪ / RTI >

26 is a view showing a display screen 3001 including a photographing environment guide message according to an embodiment of the present invention.

According to an embodiment of the present invention, a multimedia device searches environment information of a shooting environment of a multimedia device in an image input through an image sensing unit of a multimedia device, compares the searched environment information with optimal environment information for user gesture recognition , The user can display a guide message 3002 for the shooting environment

In addition, the environment information may include information on the brightness of the photographed image.

That is, for example, when the multimedia device searches the environment information of the shooting environment of the input image through the image sensing unit of the multimedia device, the multimedia device displays a guide for guiding the brightness adjustment of the shooting environment, A message 3002 can be displayed. In addition, the user can recognize the darkness of the brightness of the photographing environment through the guide message 3002 and improve the problem of the photographing environment through a predetermined operation such as turning on another illumination.

In addition, the user can select the confirmation menu item 3003 to allow the multimedia device to perform the reshooting in the improved shooting environment.

That is, through the guide message 3002, the user can be aware of a problem with the current shooting environment of the multimedia device, thereby obtaining a smooth user gesture image.

FIG. 27 is a view showing a display screen 3101 including a user selection menu according to an embodiment of the present invention.

According to an embodiment of the present invention, a multimedia device includes a user selection menu for receiving a user from an image input through an image sensing unit of the multimedia device, and receiving a user selection for inputting a user gesture when the user is a plurality of recognized users 3102 can be displayed.

That is, since a plurality of users can use predetermined contents or services through the multimedia device at the same time, a user gesture for controlling the multimedia device, having control of the multimedia device among the plurality of users, A user selection menu 3102 for specifying a user to be displayed on a display screen.

The user selection menu 3102 may include user information 3103 and 3104 for each user recognized by the multimedia device.

27, the user information is obtained by searching the face of each user from the image input through the image sensing unit of the multimedia device, cropping the image of the part where the searched face part is located, 3103 < / RTI >

In addition to the image information including the face of each user, a user name or a user ID is searched based on the information about the face of each user, and information including the searched name or ID Can be displayed.

When the user selection menu 3102 is displayed, the user can move the pointer 3105 to select any one of the user information 3103 and 3104 for each user, When the selection signal is received, only the user gesture input by the user corresponding to the selected user information can be recognized as a control signal of the multimedia device.

In addition, the user can input a selection signal of the cancel menu item 3106 of the user selection menu 3102 to the multimedia device to remove the user selection menu 3102 from the display screen.

That is, if the control information is not included among the users displayed in the user information 3103 and 3104, or if the user intends to terminate the step of inputting the user gesture to the multimedia device, 3105 may be selected to return to the state before the user selection menu 3102 is displayed.

Accordingly, even when a plurality of users simultaneously use the multimedia device, the user can input a user gesture through the user selection menu 3102 to select a specific user to control the multimedia device, thereby enabling smooth user gesture input.

The multimedia device and its operation method according to the present invention are not limited to the configuration and method of the embodiments described above but the embodiments may be modified so that all or part of each embodiment Or may be selectively combined.

Meanwhile, the method of operating the multimedia device of the present invention can be implemented as a code that can be read by a processor on a recording medium readable by a processor included in the multimedia device. The processor-readable recording medium includes all kinds of recording apparatuses in which data that can be read by the processor is stored. Examples of the recording medium that can be read by the processor include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and may also be implemented in the form of a carrier wave such as transmission over the Internet . In addition, the processor-readable recording medium may be distributed over network-connected computer systems so that code readable by the processor in a distributed fashion can be stored and executed.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the present invention.

100: Multimedia device
105: Broadcast receiver
110: tuner
120: demodulator
130: Network interface unit
135: External device interface unit
140:
150: user input interface unit
170:
180:
185: Audio output section
190: Power supply
200: remote control device

Claims (14)

delete delete delete delete delete delete delete 1. A multimedia device capable of user gesture recognition,
An image sensing unit for receiving a user image;
A recognition unit for recognizing a user gesture in an image input through the image sensing unit;
A message generating unit for generating an error message when the recognition unit does not recognize the user gesture; And
And a control unit for displaying an image input through the image sensing unit and an error message generated by the message generating unit and controlling the image sensing unit to receive a user image again,
The image sensing unit includes:
Tracking the position of the user,
Wherein the message generator comprises:
And generates an error message including information on a current user position when the position of the tracked user is out of the recognizable area of the image sensing unit. 9. The method of claim 8,
The multimedia device includes:
In the case where the user does not recognize the gesture, a storage unit
Further comprising:
Wherein the message generator comprises:
Recognizing an object included in the input image through the image sensing unit, searching an expected obstacle among the recognized objects through the storage unit, generating an error message for the detected obstacle,
Wherein,
Cropping an image of a portion where the searched recognition obstacle is located in the image input through the image sensing unit, and controlling to display the cropped image and the error message
A multimedia device capable of user gesture recognition. delete 9. The method of claim 8,
The multimedia device includes:
A storage unit for storing optimal distance information of a user for user gesture recognition based on contents in use in the multimedia apparatus,
Further comprising:
The image sensing unit includes:
Acquiring distance information of the user,
Wherein the message generator comprises:
Compares the optimum distance information stored in the storage unit with the distance information of the user, and generates an error message including a guide message according to the comparison result
A multimedia device capable of user gesture recognition. 9. The method of claim 8,
Wherein,
A face of the user is searched for in the image input through the image sensing unit, and when the searched face is a plurality of images, the image of the portion including the recognized face in the input image is cropped, And displays the pinged facial image, receives one of the facial image selection signals, and controls the user to recognize the user's gesture corresponding to the selected facial image
A multimedia device capable of user gesture recognition. delete delete

Images