KR20150054656A - Digital device and method for processing three dimensional image thereof - Google Patents

Abstract

The present invention relates to a digital device which can obtain a color image and a depth image, and a method for processing three-dimensional image thereof, which comprises the steps of: converting resolution of an optic acquisition part from first resolution into second resolution which is lower than the first resolution; detecting a visible ray and an infrared ray from a certain object; extracting color image information from a visible ray detected in a first detection part of the optic acquisition part during a first time; extracting depth image information from the infrared ray detected in a second detection part of the optic acquisition part during a second time; determining whether the extraction of color image information and depth image information of the object is completely finished; and displaying a three dimensional image of the object based on the extracted color image information and depth image information, if the extraction of color image information and depth image information is completely finished.

Description

TECHNICAL FIELD [0001] The present invention relates to a digital device and a three-

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a digital device for processing a three-dimensional image, and more particularly, to a digital device and a three-dimensional image processing method thereof capable of acquiring a color image and a depth image together .

In general, a digital device for processing three-dimensional images may include a color camera for acquiring color images and a depth camera for acquiring depth images.

Here, the color camera is a camera employing a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and detects a visible light reflected from a subject receiving sunlight or flash light, Images can be acquired.

And, a depth camera can acquire a ray of light reflected by a laser light or an infrared ray to a subject, and obtain a depth image of the object.

In this case, the depth may mean the distance from the camera to the subject.

Then, a three-dimensional image of the subject can be acquired using the obtained two-dimensional color image and depth image.

However, a digital device for processing such a three-dimensional image requires a separate color camera for acquiring a color image and a depth camera for acquiring a depth image, so that the digital device is complicated in configuration, takes a long time for image processing, The overall cost may increase.

Therefore, there is a demand for a digital device for processing a three-dimensional image capable of acquiring a color image and a depth image together in the future.

An embodiment of the present invention is to provide a digital device having a simple configuration and a three-dimensional image processing method using the light receiving unit capable of acquiring both a color image and a depth image.

According to another aspect of the present invention, there is provided a method of converting a resolution of a light receiving unit into a first resolution and a second resolution lower than a first resolution, contrast of a three-dimensional image and a three-dimensional image processing method thereof.

A three-dimensional image processing method of a digital device according to an embodiment of the present invention includes the steps of converting a resolution of a light receiving unit to a second resolution lower than a first resolution at a first resolution; A step of extracting color image information from the visible light sensed by the first sensing unit of the light receiving unit during a first time and a step of extracting color image information from the infrared light sensed by the second sensing unit of the light receiving unit during the second time, Determining whether extraction of the color image information and the depth image information with respect to the subject is completed; and if the extraction of the color image information and the depth image information with respect to the subject is completed, And embodying a three-dimensional image of the object based on the depth image information.

A digital device according to an embodiment of the present invention includes a receiving unit for receiving a two-dimensional or three-dimensional image mode request signal, a first sensing unit for sensing, from a predetermined subject, visible light corresponding to a visible region of the optical spectrum, And a second sensing unit that senses infrared light corresponding to an infrared region of the optical spectrum; and a second sensing unit that extracts color image information from the first sensing unit during a first time, Dimensional image of the object based on the extracted color image information and the depth image information, and a controller for receiving the three-dimensional image mode request signal, If the resolution of the light receiving unit is the first resolution, the resolution of the light receiving unit is changed from the first resolution to the second resolution Converted to, and may comprise a control unit for controlling the light receiving portion, a video processor, and the 3D image implementing parts.

According to an embodiment of the present invention, a color image and a depth image can be simultaneously processed using a light receiving unit including a first sensing unit for sensing visible light and a second sensing unit for sensing infrared light, , The three-dimensional image processing time and the overall cost can be reduced.

According to an embodiment of the present invention, the resolution of the light receiving unit is converted from the first resolution to the second resolution lower than the first resolution, and the infrared light between the previous frame end time of the color image information and the next frame start time The sensitivity and contrast of the depth image can be improved, and the depth sense of the three-dimensional image can be improved.

1 is a block diagram illustrating a digital device according to an embodiment of the present invention.
2 is a block diagram illustrating a digital device according to another embodiment of the present invention.
3 is a block diagram showing a three-dimensional image processing apparatus of a digital device according to the present invention
Fig. 4 is a view showing a bayer pattern according to the light receiving portion of Fig. 3
5 is a view showing a unit pixel of a light receiving unit according to the first embodiment
6 is a view showing a unit pixel of a light receiving unit according to a second embodiment
7 is a cross-sectional view showing a unit pixel of the light-
8 is a cross-sectional view of a fourth embodiment
9 is a cross-sectional view showing a unit pixel of the light-
10 is a cross-sectional view of a sixth embodiment
Fig. 11 is a block diagram showing the image processing unit of Fig. 3
12 is a block diagram showing the first image processing unit of FIG.
13 is a block diagram showing the second image processing unit of FIG.
FIG. 14 is a block diagram showing another embodiment of the second image processing unit of FIG.
15 is a block diagram showing the first depth image information extracting unit of FIG.
16 and 17 are flowcharts showing a method of processing a three-dimensional image of a digital device according to the present invention
18 to 23 are schematic views for explaining a three-dimensional image processing process of the digital device according to the present invention
24 is a view showing ON / OFF of the light emitting portion according to time;
25 is a view showing the image information processing of the image processing unit according to time;
FIG. 26 is a detailed flowchart showing a method of extracting the color image information of FIG.
FIG. 27 is a detailed flowchart showing a method of extracting the depth image information of FIG.
28 is a view showing a bayer pattern in which the first and second depth image information are not combined
29 is a view showing the distribution of infrared light sensed by the light receiving unit
30 is a view showing a Bayer pattern in which the first and second depth image information are synthesized
FIG. 31 is a detailed flowchart showing a method of extracting the second depth image information of FIG.
32 is a graph showing the light sensitivity characteristics of the first and second sensing units
33 and 34 are diagrams comparing contrasts of depth images according to whether or not the first depth image information and the second depth image information are combined

Hereinafter, various embodiments (s) of a digital device and a three-dimensional image processing method thereof according to the present invention will be described in detail with reference to the drawings.

The suffix "module "," part ", and the like for components used in the present specification are given only for ease of specification, and both may be used as needed. Also, even when described in ordinal numbers such as " 1st ", "2nd ", and the like, it is not limited to such terms or ordinal numbers.

In addition, although the terms used in the present specification have been selected from the general terms that are widely used in the present invention in consideration of the functions according to the technical idea of the present invention, they are not limited to the intentions or customs of the artisan skilled in the art, It can be different. However, in certain cases, some terms are arbitrarily selected by the applicant, which will be described in the related description section. Accordingly, it should be understood that the term is to be interpreted based not only on its name but on its practical meaning as well as on the contents described throughout this specification.

It is to be noted that the contents of the present specification and / or drawings are not intended to limit the scope of the present invention.

Hereinafter, the digital device described in this specification refers to a device that transmits, receives, processes, and outputs data, content, service, application, And includes all devices that perform at least one or more. The digital device can be paired or connected (hereinafter, referred to as 'pairing') with another digital device, an external server, or the like through a wire / wireless network, Can be transmitted / received. At this time, if necessary, the data may be appropriately converted before the transmission / reception. The digital device may be a standing device such as a network TV, a Hybrid Broadcast Broadband TV (HBBTV), a Smart TV, an IPTV (Internet Protocol TV), a PC (Personal Computer) And a mobile device or handheld device such as a PDA (Personal Digital Assistant), a smart phone, a tablet PC, a notebook, and the like. In order to facilitate understanding of the present invention and to facilitate the description of the present invention, FIG. 1, which will be described later, illustrates a digital TV, and FIG. 2 illustrates and describes a mobile device as an embodiment of a digital device. In addition, the digital device described in this specification may be a configuration having only a panel, a configuration such as a set-top box (STB), a device, a system, etc. and a set configuration .

Meanwhile, the wired / wireless network described in this specification refers to a communication network that supports various communication standards or protocols for pairing and / or data transmission / reception between digital devices or a digital device and an external server. Such a wired / wireless network includes all of the communication networks to be supported by the standard now or in the future, and is capable of supporting one or more communication protocols therefor. Such a wired / wireless network includes, for example, a USB (Universal Serial Bus), a Composite Video Banking Sync (CVBS), a Component, an S-Video (Analog), a DVI (Digital Visual Interface) A communication standard or protocol for a wired connection such as an RGB or a D-SUB, a Bluetooth standard, a radio frequency identification (RFID), an infrared data association (IrDA), an ultra wideband (UWB) (ZigBee), DLNA (Digital Living Network Alliance), WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) A long term evolution (LTE-Advanced), and Wi-Fi direct, and a communication standard or protocol for the network.

In addition, when the term is simply referred to as a digital device in this specification, the meaning may mean a fixed device or a mobile device depending on the context, and may be used to mean both, unless specifically stated otherwise.

On the other hand, the digital device is an intelligent device that supports, for example, a broadcast receiving function, a computer function or a support, at least one external input, and the like. The digital device includes e-mail, web browsing, Banking, game, application, and so on. In addition, the digital device may include an interface for supporting at least one input or control means (hereinafter, 'input means') such as a handwriting input device, a touch-screen, .

In addition, the digital device can use a standardized general-purpose OS (Operating System), but in particular, the digital device described in this specification uses the Web OS as an embodiment. Therefore, a digital device can handle adding, deleting, amending, and updating various services or applications on a general-purpose OS kernel or a Linux kernel. And through which a more user-friendly environment can be constructed and provided.

Meanwhile, the above-described digital device can receive and process an external input. The external input is connected to an external input device, that is, the digital device, through the wired / wireless network, Input means or digital device. For example, the external input may be a game device such as a high-definition multimedia interface (HDMI), a playstation or an X-Box, a smart phone, a tablet PC, a pocket photo devices such as digital cameras, printing devices, smart TVs, Blu-ray device devices and the like.

In addition, the server described in the present specification refers to a digital device or system that supplies data to or receives data from the above-described digital device, that is, a client, and is also referred to as a processor. The server may include a portal server for providing a web page, a web content or a web service, an advertising server for providing advertising data, A content server providing content, an SNS server providing a social network service (SNS), a service server provided by a manufacturer, a video on demand (VoD) service or a streaming service And a service server that provides a Multichannel Video Programming Distributor (MVPD) and a pay service, for example.

In addition, in the following description for convenience of explanation, only the application is described in the context of the present invention, and the meaning may include not only the application but also the service based on the context and the like.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a digital device according to an exemplary embodiment of the present invention.

The digital device 200 includes a network interface 201, a TCP / IP manager 202, a service delivery manager 203, an SI decoder 204, A demultiplexer (demux or demultiplexer) 205, an audio decoder 206, a video decoder 207, a display A / V and OSD module 208, a service management manager 209, a service discovery manager 210, an SI & metadata DB 211, a metadata manager 212, a service manager 213, A UI manager 214, and the like.

The network interface unit 201 receives the IP packet (s) (IP packet (s)) or the IP datagram (s) ) Is transmitted / received. For example, the network interface unit 201 can receive services, applications, content, and the like from the service provider 20 of FIG. 1 through a network.

The TCP / IP manager 202 determines whether the IP packets received by the digital device 200 and the IP packets transmitted by the digital device 200 are packet delivery (i.e., packet delivery) between a source and a destination packet delivery). The service discovery manager 210, the service control manager 209, the meta data manager 212, the service discovery manager 210, the service control manager 209, and the metadata manager 212. The TCP / IP manager 202 classifies the received packet (s) ) Or the like.

The service delivery manager 203 is responsible for controlling the received service data. For example, the service delivery manager 203 may use RTP / RTCP when controlling real-time streaming data. When the real-time streaming data is transmitted using the RTP, the service delivery manager 203 parses the received data packet according to the RTP and transmits the packet to the demultiplexing unit 205 or the control of the service manager 213 In the SI & meta data database 211. [ Then, the service delivery manager 203 feedbacks the network reception information to the server providing the service using RTCP.

The demultiplexer 205 demultiplexes the received packets into audio, video, SI (System Information) data, and transmits them to the audio / video decoder 206/207 and the SI decoder 204, respectively.

The SI decoder 204 decodes the demultiplexed SI data, that is, Program Specific Information (PSI), Program and System Information Protocol (PSIP), Digital Video Broadcasting Service Information (DVB-SI), Digital Television Terrestrial Multimedia Broadcasting / Coding Mobile Multimedia Broadcasting). Also, the SI decoder 204 may store the decoded service information in the SI & meta data database 211. The stored service information can be read out and used by the corresponding configuration, for example, by a user's request.

The audio / video decoder 206/207 decodes each demultiplexed audio data and video data. The decoded audio data and video data are provided to the user through the display unit 208. [

The application manager may include, for example, the UI manager 214 and the service manager 213 and may perform the functions of the controller of the digital device 200. [ In other words, the application manager can manage the overall state of the digital device 200, provide a user interface (UI), and manage other managers.

The UI manager 214 provides a GUI (Graphic User Interface) / UI for a user using an OSD (On Screen Display) or the like, and receives a key input from a user to perform a device operation according to the input. For example, the UI manager 214 receives the key input regarding the channel selection from the user, and transmits the key input signal to the service manager 213.

The service manager 213 controls the manager associated with the service such as the service delivery manager 203, the service discovery manager 210, the service control manager 209, and the metadata manager 212.

In addition, the service manager 213 generates a channel map and controls the selection of a channel using the generated channel map according to a key input received from the UI manager 214. [ The service manager 213 receives the service information from the SI decoder 204 and sets an audio / video PID (Packet Identifier) of the selected channel in the demultiplexer 205. The PID thus set can be used in the demultiplexing process described above. Accordingly, the demultiplexer 205 filters (PID or section) audio data, video data, and SI data using the PID.

The service discovery manager 210 provides information necessary for selecting a service provider providing the service. Upon receiving a signal regarding channel selection from the service manager 213, the service discovery manager 210 searches for the service using the information.

The service control manager 209 is responsible for selection and control of services. For example, the service control manager 209 uses IGMP or RTSP when a user selects a live broadcasting service such as an existing broadcasting system, and uses RTSP when selecting a service such as VOD Service selection and control. The RTSP protocol may provide a trick mode for real-time streaming. In addition, the service control manager 209 can initialize and manage a session through the IMS gateway 250 using an IP Multimedia Subsystem (IMS) and a Session Initiation Protocol (SIP). The protocols are one embodiment, and other protocols may be used, depending on the implementation.

The metadata manager 212 manages the metadata associated with the service and stores the metadata in the SI & metadata database 211.

The SI & meta data database 211 stores the service information decoded by the SI decoder 204, the meta data managed by the meta data manager 212, and the information necessary for selecting a service provider provided by the service discovery manager 210 do. In addition, the SI & meta data database 211 may store set-up data for the system and the like.

The SI & meta data database 211 may be implemented using a non-volatile RAM (NVRAM) or a flash memory.

Meanwhile, the IMS gateway 250 is a gateway that collects functions necessary for accessing the IMS-based IPTV service.

2 is a block diagram illustrating a digital device according to another embodiment of the present invention.

If FIG. 1 described above illustrates a fixed device as one embodiment of a digital device, FIG. 2 illustrates a mobile device as another embodiment of a digital device.

2, the mobile device 300 includes a wireless communication unit 310, an A / V input unit 320, a user input unit 330, a sensing unit 340, an output unit 350, A memory 360, an interface unit 370, a control unit 380, a power supply unit 390, and the like.

Hereinafter, each component will be described in detail.

The wireless communication unit 310 may include one or more modules that enable wireless communication between the mobile device 300 and the wireless communication system or between the mobile device and the network in which the mobile device is located. For example, the wireless communication unit 310 may include a broadcast receiving module 311, a mobile communication module 312, a wireless Internet module 313, a short range communication module 314, and a location information module 315 .

The broadcast receiving module 311 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. Here, the broadcast channel may include a satellite channel and a terrestrial channel. The broadcast management server may refer to a server for generating and transmitting broadcast signals and / or broadcast related information, or a server for receiving broadcast signals and / or broadcast related information generated by the broadcast management server and transmitting the generated broadcast signals and / or broadcast related information. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, a data broadcast signal, and a broadcast signal in which a data broadcast signal is combined with a TV broadcast signal or a radio broadcast signal.

The broadcast-related information may mean information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast-related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 312.

The broadcast-related information may exist in various forms, for example, in the form of an EPG (Electronic Program Guide) or an ESG (Electronic Service Guide).

The broadcast receiving module 311 may be, for example, an ATSC, a Digital Video Broadcasting-Terrestrial (DVB-T), a Satellite (DVB-S), a Media Forward Link Only And Integrated Services Digital Broadcast-Terrestrial (DRS). Of course, the broadcast receiving module 311 may be adapted to not only the above-described digital broadcasting system but also other broadcasting systems.

The broadcast signal and / or broadcast related information received through the broadcast receiving module 311 may be stored in the memory 360.

The mobile communication module 312 transmits and receives radio signals to at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signal may include various types of data depending on a voice signal, a video call signal, or a text / multimedia message transmission / reception.

The wireless Internet module 313 may be embedded or external to the mobile device 300, including a module for wireless Internet access. WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) and the like can be used as wireless Internet technologies.

The short-range communication module 314 is a module for short-range communication. Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IRDA), Ultra Wideband (UWB), ZigBee, RS-232 and RS-485 are used as short range communication technology. .

The position information module 315 is a module for acquiring position information of the mobile device 300, and may be a GPS (Global Position System) module.

The A / V input unit 320 is for inputting audio and / or video signals. The A / V input unit 320 may include a camera 321, a microphone 322, and the like. The camera 321 processes an image frame such as a still image or moving image obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame can be displayed on the display section 351. [

The image frame processed by the camera 321 may be stored in the memory 360 or transmitted to the outside via the wireless communication unit 310. [ At least two cameras 321 may be provided depending on the use environment.

The microphone 322 receives an external sound signal by a microphone in a communication mode, a recording mode, a voice recognition mode, or the like, and processes it as electrical voice data. The processed voice data can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 312 in the case of the communication mode, and output. The microphone 322 may be implemented with various noise reduction algorithms for eliminating noise generated in receiving an external sound signal.

The user input unit 330 generates input data for a user to control the operation of the terminal. The user input unit 330 may include a key pad, a dome switch, a touch pad (static pressure / static electricity), a jog wheel, a jog switch, and the like.

The sensing unit 340 senses the current state of the mobile device 300 such as the open / closed state of the mobile device 300, the position of the mobile device 300, the user's contact, the orientation of the mobile device, And generates a sensing signal for controlling the operation of the mobile device 300. For example, when the mobile device 300 is moved or tilted, it may sense the position, slope, etc. of the mobile device. It is also possible to sense whether power is supplied to the power supply unit 390, whether the interface unit 370 is connected to an external device, and the like. Meanwhile, the sensing unit 240 may include a proximity sensor 341 including NFC (Near Field Communication).

The output unit 350 may include a display unit 351, an acoustic output module 352, an alarm unit 353, and a haptic module 354 for generating output related to visual, auditory, have.

The display unit 351 displays (outputs) information processed by the mobile device 300. [ For example, if the mobile device is in call mode, it displays a UI or GUI associated with the call. When the mobile device 300 is in the video communication mode or the photographing mode, the photographed and / or received video or UI and GUI are displayed.

The display unit 351 may be a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED) flexible display, and a three-dimensional display.

Some of these displays may be transparent or light transmissive so that they can be seen through. This can be referred to as a transparent display, and a typical example of the transparent display is TOLED (Transparent OLED) and the like. The rear structure of the display portion 351 may also be of a light transmission type. With this structure, the user can see an object located behind the terminal body through the area occupied by the display unit 351 of the terminal body.

There may be two or more display units 351 depending on the implementation of the mobile device 300. [ For example, in the mobile device 300, the plurality of display portions may be spaced apart or integrally arranged on one surface, and may be disposed on different surfaces, respectively.

(Hereinafter, referred to as a 'touch screen') in which a display unit 351 and a sensor (hereinafter, referred to as 'touch sensor') for sensing a touch operation form a mutual layer structure It can also be used as a device. The touch sensor may have the form of, for example, a touch film, a touch sheet, a touch pad, or the like.

The touch sensor may be configured to convert a change in a pressure applied to a specific portion of the display portion 351 or a capacitance generated in a specific portion of the display portion 351 into an electrical input signal. The touch sensor can be configured to detect not only the position and area to be touched but also the pressure at the time of touch.

If there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and transmits corresponding data to the controller 380. Thus, the control unit 380 can know which area of the display unit 351 is touched or the like.

A proximity sensor 341 may be disposed in the interior area of the mobile device or in proximity to the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or a nearby object without mechanical contact using the force of an electromagnetic field or infrared rays. The proximity sensor has a longer life span than the contact sensor and its utilization is also high.

Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. And to detect the proximity of the pointer by the change of the electric field along the proximity of the pointer when the touch screen is electrostatic. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, the act of recognizing that the pointer is positioned on the touch screen while the pointer is not in contact with the touch screen is referred to as "proximity touch & The act of actually touching the pointer on the screen is called "contact touch. &Quot; The position where the pointer is proximately touched on the touch screen means a position where the pointer is vertically corresponding to the touch screen when the pointer is touched.

The proximity sensor detects a proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, a proximity touch movement state, and the like). Information corresponding to the detected proximity touch operation and the proximity touch pattern may be output on the touch screen.

The sound output module 352 can output audio data received from the wireless communication unit 310 or stored in the memory 360 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output module 352 also outputs sound signals associated with functions performed on the mobile device 300 (e.g., call signal receive tones, message receive tones, etc.). The sound output module 352 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 353 outputs a signal for notifying the occurrence of an event of the mobile device 300. [ Examples of events that occur in the mobile device include receiving a call signal, receiving a message, inputting a key signal, and touch input. The alarm unit 353 may output a signal for informing occurrence of an event in a form other than the video signal or the audio signal, for example, vibration. The video signal or the audio signal may be output through the display unit 351 or the audio output module 352 so that they may be classified as a part of the alarm unit 353.

The haptic module 354 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 354 is vibration. The intensity and pattern of the vibration generated by the haptic module 354 are controllable. For example, different vibrations may be synthesized and output or sequentially output. In addition to the vibration, the haptic module 354 may be arranged in a variety of ways, such as a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a spit on the skin surface, contact with an electrode, Various effects such as an effect of heat generation and an effect of reproducing a cool / warm feeling using a heat absorbing or heatable element can be generated. The haptic module 354 can be implemented not only to transmit the tactile effect through direct contact but also to allow the user to feel the tactile effect through the muscular sensation of a finger or an arm. More than one haptic module 354 may be provided according to the configuration of the mobile device 300.

The memory 360 may store a program for the operation of the control unit 380 and temporarily store input / output data (e.g., phone book, message, still image, moving picture, etc.). The memory 360 may store data on vibration and sound of various patterns outputted when a touch is input on the touch screen.

The memory 360 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory) (Random Access Memory), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM) A magnetic disk, and / or an optical disk. The mobile device 300 may operate in association with a web storage that performs storage functions of the memory 360 on the Internet.

The interface unit 370 serves as a pathway to all the external devices connected to the mobile device 300. The interface unit 370 receives data from an external device or receives power from the external device and transfers the data to each component in the mobile device 300 or transmits data in the mobile device 300 to an external device. For example, it may be provided with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port connecting a device with an identification module, an audio I / O port, A video I / O port, an earphone port, and the like may be included in the interface unit 370.

The identification module is a chip for storing various information for authenticating the usage right of the mobile device 300 and includes a user identification module (UIM), a subscriber identity module (SIM), a general user authentication module A Universal Subscriber Identity Module (USIM), and the like. A device having an identification module (hereinafter referred to as 'identification device') can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 200 through the port.

The interface unit 370 may be a communication path through which the power from the cradle is supplied to the mobile device 300 when the mobile device 300 is connected to an external cradle, A command signal may be the path through which it is communicated to the mobile device. The various command signals or the power supply input from the cradle may be operated with a signal for recognizing that the mobile device is correctly mounted on the cradle.

The control unit 380 typically controls the overall operation of the mobile device 300. The control unit 380 performs related control and processing, for example, for voice call, data communication, video call, and the like. The control unit 380 may include a multimedia module 381 for multimedia playback. The multimedia module 381 may be implemented in the control unit 380 or separately from the control unit 380. The control unit 380 can perform pattern recognition processing for recognizing handwriting input or drawing input performed on the touch-screen as characters and images, respectively.

The power supply unit 390 receives external power and internal power under the control of the controller 380 and supplies power necessary for operation of the respective components.

The various embodiments described herein may be implemented in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

According to a hardware implementation, the embodiments described herein may be implemented as application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays May be implemented using at least one of a processor, a controller, micro-controllers, microprocessors, and an electrical unit for performing other functions. In some cases, the implementation described herein Examples may be implemented by the control unit 380 itself.

According to a software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. Software code may be implemented in a software application written in a suitable programming language. Here, the software code is stored in the memory 360 and can be executed by the control unit 380. [

FIG. 3 is a block diagram showing a three-dimensional image processing apparatus of a digital device according to the present invention.

3, the 3D image processing apparatus of the digital device includes a light receiving unit 1100, an image processing unit 1200, a three-dimensional image implementing unit 1300, a control unit 1400, a light emitting unit 1500, (1600).

In addition, it may further include a timer 1700. [

Here, the light-receiving unit 1100 can receive visible light corresponding to the visible region of the light spectrum and infrared light corresponding to the infrared region of the light spectrum from the predetermined object 1800. [

In some cases, the light receiving unit 1100 may include a first sensing unit 1110 and a second sensing unit 1120.

At this time, the first sensing unit 1110 of the light receiving unit 1100 can sense visible light corresponding to the visible region of the optical spectrum, for example, can sense light having a wavelength range of about 350-700 nm.

The second sensing unit 1120 of the light receiving unit 1100 can sense infrared light corresponding to the infrared region of the optical spectrum. For example, the infrared light can sense light having a wavelength range of about 700 nm or more .

Further, the light receiving unit 1100 can convert into a high-resolution mode and a low-resolution mode according to a control signal of the control unit 1400. [

In one example, the low resolution mode may be a resolution of about 1/4 or less of the high resolution mode.

Here, when the three-dimensional image is implemented, the light receiving unit 1100 can convert from the high resolution mode to the low resolution mode according to the control signal of the controller 1400. [

In addition, when the two-dimensional image is implemented, the light receiving unit 1100 can convert from the low resolution mode to the high resolution mode according to the control signal of the control unit 1400. [

In some cases, when the two-dimensional image is implemented, the light receiving unit 1100 may maintain the low resolution mode without converting from the low resolution mode to the high resolution mode according to the control signal of the controller 1400. [

Also, even when the light receiving unit 1100 is converted to the low resolution mode, the image processing unit 1200 can extract the depth image information or the color image information at the same time as the high resolution mode.

For example, when the light receiving unit 1100 of the low resolution mode operates at 60 frames, the image processing unit 1200 may have an interval of about 1/120 seconds between frames.

As described above, according to the present invention, by arranging the light receiving portion 1100 capable of simultaneously receiving the visible light corresponding to the visible region of the light spectrum and the infrared light corresponding to the infrared region of the light spectrum from the predetermined object 1800, The color image information can be obtained from the visible light and the depth image information can be obtained from the infrared light.

Therefore, since the acquired color image information and depth image information can be simultaneously processed, the present invention is simple in configuration, and the three-dimensional image processing time and the overall cost can be reduced.

The image processing unit 1200 extracts the color image information from the first sensing unit 1110 during the first time and outputs the depth image from the first sensing unit 1110 and the second sensing unit 1120 during the second time, Information can be extracted.

As described above, the image processing unit 1200 of the present invention can acquire the color image information and the depth image information at different times without acquiring the color image information and the depth image information at the same time.

That is, the image processing unit 1200 of the present invention may have a different time from acquiring the color image information and acquiring the depth image information.

The reason for this is that a depth image processing method capable of improving the sensitivity and contrast of depth images must be performed independently.

For example, the image processing unit 1200 performs a binning process on the acquired color image information.

Here, the binning process refers to combining pixels of a plurality of sensors of the light receiving unit 1100 to obtain a new pixel. Since the binning process combines two or more pixels, the sensitivity and SNR ( Signal to noise ratio) is improved, so that a clear color image can be obtained even in dark lighting.

In addition, the image processing unit 1200 performs a sub-sampling process for the IR pixel with respect to the acquired depth image information.

Here, the sub-sampling process is a process of collecting only the depth information of the IR pixels in the IR frame. Since only the depth information of the infrared pixels is collected and output, the sensitivity and contrast of the depth image can be improved.

If the image processing unit 1200 separately acquires the color image information and the depth image information, the image processing unit 1200 may receive the depth image information from the second sensing unit 1120 of the light receiving unit 1100 Since the depth image information can be extracted not only by the infrared light received but also by the infrared light received from the first sensing unit 1110 of the light receiving unit 1100, the sensitivity and contrast for the depth image are improved, The depth feeling of the image can be improved.

Accordingly, the image processing unit 1200 acquires the color image information acquisition operation and the depth image information acquisition operation for each unit pixel of the light receiving unit 1100 including the first sensing unit 1110 and the second sensing unit 1120 Can be performed separately.

Next, the three-dimensional image implementing unit 1300 can implement a three-dimensional image of the subject 1800 based on the color image information and the depth image information extracted from the image processing unit 1200.

The control unit 1400 may control the light receiving unit 1100, the image processing unit 1200, and the three-dimensional image implementing unit 1300.

Here, when receiving the 3D image mode request signal, the controller 1400 checks the resolution mode of the light receiving unit 1100, and if the resolution of the light receiving unit 1100 is the first resolution, the controller 1400 determines the resolution of the light receiving unit 1100 To a second resolution lower than the first resolution at the first resolution.

The reason for changing the resolution of the light receiving unit 1100 is to reduce crosstalk.

Here, crosstalk refers to interference between an RGB frame for acquiring color image information and an IR frame for acquiring depth image information.

That is, IR illumination is required for the IR frame, and IR illumination is not required for the RGB frame. The crosstalk is a phenomenon in which the IR illumination of the IR frame affects the RGB frame, and noise or distortion occurs in the RGB image Phenomenon.

Therefore, in order not to affect the crosstalk influence on the RGB frames located before and after the IR frame, the IR illumination must be turned on instantaneously.

However, if the on-time of the IR illumination is very short, the IR illumination can not be recognized in the IR frame, and therefore the sensitivity of the depth image is inevitably lowered.

Therefore, in the present invention, in order to eliminate such a crosstalk phenomenon, in the three-dimensional image mode, the resolution of the light receiving unit 1100 is changed from the first resolution to the second resolution lower than the first resolution.

For example, if the resolution of the light receiving unit 1100 is reduced to about 1/4 of the resolution at the highest resolution, the time interval between the RGB frame and the IR frame is increased.

Therefore, the ON time of the IR illumination is increased by an increased time interval, and the IR illumination can be sufficiently recognized in the IR frame, and the sensitivity of the depth image can be improved.

In addition, the control unit 1400 can control the light emitting unit 1500 and the timer 1700.

For example, the light emitting unit 1500 may be driven according to a control signal of the control unit 1400 as an apparatus for generating infrared light.

That is, the light emitting unit 1500 can emit infrared light for a second time period without emitting infrared light for the first time period, in accordance with the control signal of the control unit 1400. [

Here, the driving time of the light emitting unit 1500 may be between the end time of the previous frame of the color image information and the start time of the next frame.

The timer 1700 controls the first time for turning off the driving of the light emitting unit 1500 and the driving of the light emitting unit 1500 on the basis of the control signal of the controller 1400 ) Can be measured.

Therefore, the control unit 1400 can control the driving of the light emitting unit 1500 according to the time measured by the timer 1700. [

The storage unit 1600 can store the extracted color image information and depth image information by the image processing unit 1200. [

Here, since the image processing unit 1200 acquires the color image information and acquires the depth image information from the storage unit 1600, the storage unit 1600 stores the extracted image information for a predetermined time And can perform a buffer function.

As described above, since the color image and the depth image can be processed at the same time by using the light receiving unit including the first sensing unit for sensing visible light and the second sensing unit for sensing infrared light, Dimensional image processing time and the overall cost can be reduced.

The present invention also provides a method for converting the resolution of a light receiving unit to a second resolution lower than the first resolution at a first resolution to increase the exposure time of infrared light between a previous frame end time of color image information and a next frame start time The sensitivity and contrast for the depth image are improved, and the depth feeling of the three-dimensional image can be improved.

The depth image information can be extracted not only by the infrared light received from the second sensing unit of the light receiving unit but also by the infrared light received by the first sensing unit of the light receiving unit, The sensitivity and contrast for the depth image are improved, and the depth feeling of the three-dimensional image can be improved.

FIG. 4 is a view showing a bayer pattern according to the light receiving unit of FIG. 3. FIG. 4A is a view showing a pixel arrangement of the light receiving unit, and FIG. 4B is a view showing unit pixels of the light receiving unit.

As shown in FIG. 4, the light receiving unit 1100 may include pixels capable of sensing visible light in the red, green, and blue wavelength ranges, and pixels capable of sensing infrared light.

For example, the light receiving unit 1100 includes a plurality of unit pixels including a first sensing unit 1110 for sensing visible light of red, green, and blue wavelengths, and a second sensing unit 1120 for sensing infrared light Lt; / RTI >

Here, the first sensing unit 1110 may include a first pixel that senses light of a red wavelength band, a second pixel that senses light of a green wavelength band, and a third pixel that senses light of a blue wavelength band.

The first sensing unit 1110 may include a first pixel that senses light in a yellow color wavelength band, a second pixel that senses light in a cyan color wavelength band, a second pixel that senses light in a cyan color wavelength band, And a third pixel that senses light.

As another example, the first sensing unit 1110 may include a first pixel that senses light in a red wavelength band, a second pixel that senses light in a green wavelength band, a third pixel that senses light in a blue wavelength band, And a fourth pixel for sensing light having a wavelength band of any one of colors, yellow, cyan, and magenta.

5A is a view showing a unit pixel of the light receiving unit, FIG. 5A shows an arrangement structure in which the pixels of the first sensing unit surround the second sensing unit, FIG. 5B is a view And the sub-pixels are arranged side-by-side.

5, the light receiving unit includes unit pixels including a first sensing unit 1110 that senses visible light of red, green, and blue wavelengths, and a second sensing unit 1120 that senses infrared light. can do.

Here, the first sensing unit 1110 includes a first pixel 1112 that senses light of a red wavelength band, a second pixel 1114 that senses light of a green wavelength band, a third pixel 1114 that senses light of a blue wavelength band 1116).

At this time, the areas of the first, second, and third pixels 1112, 1114, and 1116 may be equal to each other.

The area of the first, second, and third pixels 1112, 1114, and 1116 may be the same as the area of the second sensing unit 1120 that senses infrared light.

6 is a second embodiment showing a unit pixel of a light receiving unit.

6, the light receiving unit includes unit pixels including a first sensing unit 1110 for sensing visible light of red, green, and blue wavelengths, and a second sensing unit 1120 for sensing infrared light. The first sensing unit 1110 includes a first pixel 1112 that senses light of a red wavelength band, a second pixel 1114 that senses light of a green wavelength band, a third pixel 1114 that senses light of a blue wavelength band, Pixel < / RTI >

Here, the areas of the first, second, and third pixels 1112, 1114, and 1116 may be equal to each other, but the area of any one of the first, second, and third pixels 1112, 1114, The area may be larger than the area of the second sensing unit 1120 that senses infrared light.

As described above, according to the present invention, even if the area of the second sensing unit 1120, which affects the sensitivity and contrast of the depth image, is relatively small compared to the first sensing unit 1110, The sensitivity and the contrast for the depth image and the color image can be simultaneously improved.

A detailed description thereof will be given later.

7A and 7B show an arrangement structure in which the pixels of the first sensing unit are surrounded by the second sensing unit, and FIG. 7B is a view showing an arrangement structure of the first sensing unit on one side of the second sensing unit. And the sub-pixels are arranged side-by-side.

7, the light-receiving unit includes a first sensing unit 1110 that senses visible light in yellow, cyan, and magenta wavelength ranges, and a second sensing unit 1120 that senses infrared light. . ≪ / RTI >

The first sensing unit 1110 includes a fourth pixel 1117 that senses light of a yellow color wavelength band, a fifth pixel 1118 that senses light of a cyan color wavelength band, a magenta And a sixth pixel 1119 for sensing light of a color wavelength band.

At this time, the areas of the fourth, fifth, and sixth pixels 1117, 1118, and 1119 may be equal to each other.

The area of the fourth, fifth, and sixth pixels 1117, 1118, and 1119 may be the same as the area of the second sensing unit 1120 that senses infrared light.

8 is a fourth embodiment showing a unit pixel of the light receiving unit.

8, the light receiving unit includes a first sensing unit 1110 that senses visible light of yellow, cyan, and magenta wavelength ranges, and a second sensing unit 1120 that senses infrared light. The first sensing unit 1110 includes a fourth pixel 1117 for sensing light of a yellow color wavelength band, a fifth pixel 1118 for sensing light of a cyan color wavelength band, And a sixth pixel 1119 for sensing light of a magenta color wavelength band.

Here, the areas of the fourth, fifth, and sixth pixels 1117, 1118, and 1119 may be the same as each other. However, among the fourth, fifth, and sixth pixels 1117, 1118, and 1119, The area may be larger than the area of the second sensing unit 1120 that senses infrared light.

As described above, even if the area of the second sensing unit 1120, which affects the sensitivity and contrast of the depth image, is relatively smaller than that of the first sensing unit 1110, using the depth image processing method of the present invention , The sensitivity and the contrast for the depth image and the color image can be simultaneously improved.

9A is a view showing a unit pixel of a light receiving unit, FIG. 9A shows an arrangement structure in which pixels of the first sensing unit are surrounded by the second sensing unit, FIG. 9B is a view And the sub-pixels are arranged side-by-side.

9, the light-receiving unit includes a first sensing unit 1110 that senses visible light in the red, green, and blue wavelength ranges and senses visible light in any one of the white, yellow, cyan, and magenta wavelength ranges And a second sensing unit 1120 for sensing infrared light.

Here, the first sensing unit 1110 includes a first pixel 1112 that senses light of a red wavelength band, a second pixel 1114 that senses light of a green wavelength band, a third pixel 1114 that senses light of a blue wavelength band And a seventh pixel 1111 for sensing light having a wavelength band of any one of white, magenta, yellow, cyan, and magenta colors.

At this time, the areas of the first, second, third, and seventh pixels 1112, 1114, 1116, and 1111 may be equal to each other.

The area of the first, second, third, and seventh pixels 1112, 1114, 1116, and 1111 may be the same as the area of the second sensing unit 1120 that senses infrared light.

10 is a sixth embodiment showing a unit pixel of a light receiving unit.

10, the light receiving unit includes a first sensing unit 1110 that senses visible light in the red, green, and blue wavelength ranges and senses visible light in any one of the white, yellow, cyan, and magenta wavelength ranges And a second sensing unit 1120 for sensing infrared light. The first sensing unit 1110 includes a first pixel 1112 for sensing light of a red wavelength band, A second pixel 1114 for sensing light of a wavelength band, a third pixel 1116 for sensing light of a blue wavelength band, a white color, a yellow color, a cyan color, a magenta color And a seventh pixel 1111 for sensing light having a wavelength band of any one of the colors.

The areas of the first, second, third and seventh pixels 1112, 1114, 1116 and 1111 may be equal to each other, but the first, second, third and seventh pixels 1112, 1114 and 1116 , And 1111 may be larger than the area of the second sensing unit 1120 that senses infrared light.

As described above, even if the area of the second sensing unit 1120, which affects the sensitivity and contrast of the depth image, is relatively smaller than that of the first sensing unit 1110, using the depth image processing method of the present invention , The sensitivity and the contrast for the depth image and the color image can be simultaneously improved.

11 is a block diagram showing the image processing unit of FIG.

As shown in FIG. 11, the image processing unit 2000 may include a first image processing unit 2100 and a second image processing unit 2300.

Here, the first image processor 2100 extracts the color image information from the visible light sensed by the first sensing unit of the light receiving unit during the first time, and stores the extracted color image information in the storage unit 1600 of FIG. 3 .

The second image processor 2300 extracts the second depth image information from the infrared light sensed by the second sensing unit of the light receiving unit during the second time, (1600).

In some cases, the second image processor 2300 may extract the first depth image information from the infrared light sensed by the first sensing unit of the light receiving unit during the second time period, and may detect the noise of the first depth image information And the first depth image information from which the noise has been removed can be stored in the storage unit 1600 of FIG.

The second image processor 2300 extracts the second depth image information from the infrared light sensed by the second sensing unit of the light receiving unit during the second time, (1600).

Next, the second image processor 2300 can synthesize the first depth image information and the second depth image information from which noise has been removed.

12 is a block diagram showing the first image processing unit of FIG.

12, the first image processing unit 2100 may include a first detecting unit 2120, a first converting unit 2140, and a color image information extracting unit 2160.

Here, the first detection unit 2120 can detect the light amount of the visible light sensed by the first sensing unit of the light receiving unit for the first time.

For example, the first detection unit 2120 detects the amount of red light from the first pixel of the first sensing unit in the unit pixel of the light receiving unit, detects the amount of green light from the second pixel of the first sensing unit, The light amount of the blue light can be detected from the third pixel of the sensing part and the amount of infrared light can be detected from the second sensing part.

Depending on the case, the first detection unit 2120 may detect the amounts of white light, yellow light, cyan light, and magenta light in addition to red light, green light, and blue light, depending on the type of the light receiving unit.

Next, the first conversion unit 2140 can convert the light amount of the detected visible light into an electrical signal.

Next, the color image information extracting unit 2160 can extract the color image information from the visible light converted into the electrical signal.

Here, the color image information extracting unit 2160 performs a binning process on the acquired color image information.

Herein, the binning process refers to combining pixels of a plurality of sensors of the light receiving unit 1100 to obtain a new pixel. Since the binning process combines two or more pixels, the sensitivity and SNR ( Signal to noise ratio) is improved, so that a clear color image can be obtained even in dark lighting.

Accordingly, the first image processor 2100 can acquire the color image information from the visible light sensed by the first sensing unit of the light receiving unit for the first time.

13 is a block diagram showing a second image processing unit of FIG.

13, the second image processing unit 2300 may include a second detecting unit 2310, a second converting unit 2330, and a depth image information extracting unit 2340.

Here, the second detection unit 2310 can detect the amount of infrared light monitored by the second sensing unit of the light receiving unit during the second time.

For example, the second detection unit 2310 can detect the amount of infrared light from the second sensing unit in the unit pixel of the light receiving unit.

Then, the second conversion section 2330 can convert the light amount of the detected infrared light into an electrical signal.

Next, the depth image information extracting unit 2340 can extract the depth image information from the infrared light sensed by the second sensing unit of the light receiving unit.

Here, the depth image information extracting unit 2340 performs a sub-sampling process on the acquired IR image information.

At this time, the sub-sampling process is a process of collecting only the depth information of the IR pixels in the IR frame. Since only the depth information of the infrared pixels is collected and output, the sensitivity and contrast of the depth image can be improved.

Accordingly, the second image processor 2300 can acquire the depth image information from the infrared light sensed by the second sensing unit of the light receiving unit for the second time.

FIG. 14 is a block diagram showing another embodiment of the second image processing unit of FIG. 11. FIG.

14, the second image processing unit 2300 includes a second detecting unit 2310, a second converting unit 2330, first and second depth image information extracting units 2350 and 2370, 2390 < / RTI >

Here, the second detection unit 2310 can detect the amount of infrared light monitored from the first sensing unit and the second sensing unit of the light receiving unit during the second time.

For example, the second detection unit 2310 detects the amount of red light and infrared light from the first pixel of the first sensing unit in the unit pixel of the light receiving unit, and outputs the light amount of the green light and infrared light from the second pixel of the first sensing unit The light amount of the blue light and the infrared light can be detected from the third pixel of the first sensing unit and the amount of infrared light can be detected from the second sensing unit.

Then, the second conversion section 2330 can convert the light amount of the detected infrared light into an electrical signal.

Next, the first depth image information extracting unit 2350 extracts the first depth image information from the infrared light sensed by the first sensing unit of the light receiving unit, removes the noise of the extracted first depth image information, The removed first depth image information may be stored in the storage unit 600 of FIG.

The second depth image information extracting unit 2370 extracts the second depth image information from the infrared light sensed by the second sensing unit of the light receiving unit and outputs the extracted second depth image information to the storage unit 1600 ). ≪ / RTI >

The combining unit 2390 combines the first depth image information extracted from the first depth image information extracting unit 2350 and the second depth image information extracted from the second depth image information extracting unit 2370, The final depth image information can be generated.

Thus, the second image processing section 2300 can extract the depth image information by the following equation (1).

[Equation 1]

if (? (R ₂ - R ₁ )> R _th ), IR ₂ = IR ₂ +? (R ₂ - R ₁ )

_{if (β (G 2 - G} 1)> G th), IR 2 = IR 2 + β (G 2 - G 1)

(B ₂ - B ₁ )> B _th ), IR ₂ = IR ₂ + γ (B ₂ - B ₁ )

Here, IR ₂ is the second depth image information,

R ₁ , G ₁ , and B ₁ represent a first noise value for the infrared light of the extracted first sensing unit,

R ₂ , G ₂ , and B ₂ represent a second noise value for the infrared light of the extracted first sensing unit during the second time,

alpha, beta, and gamma are weights considering the sensitivity characteristics of the first sensing unit, and

R _{th, th} G, B _th is a physical noise threshold, the first sensing unit.

In addition, α (R ₂ - R ₁ ) is the first depth image information extracted from the first pixel of the first sensing unit that senses the red light and the infrared light in the unit pixel of the light receiving unit, ₂ - G _1), in the unit pixel of the light receiving portion, is extracted from the second pixels, a first sensing unit for sensing the green light and infrared light, a first depth image information with noise _{removed, γ (B 2 - B 1} ) Is the first depth image information extracted from the third pixel of the first sensing unit that senses blue light and infrared light in the unit pixel of the light receiving unit and is removed from the noise.

15 is a block diagram showing the first depth image information extracting unit of FIG.

As shown in FIG. 15, the first depth image information extracting unit 2370 may include a noise removing unit 2410, a comparing unit 2430, and a final depth value determining unit 2450.

Here, the noise removing unit 2410 can remove the noise of the infrared light sensed by the first sensing unit of the light receiving unit, and calculate the depth value from which the noise is removed.

For example, the noise removing unit 2410 may detect the first noise value for the infrared light detected from the first sensing unit and the second noise value for the infrared light sensed from the first sensing unit during the second time, 2 Noise value can be extracted.

Here, the first noise value may be visible light other than the infrared light sensed from the first sensing unit during the first time period

The second noise value may be visible light other than the infrared light sensed by the first sensing unit during the second time.

Next, the noise removing unit 2410 may calculate a difference value obtained by subtracting the first noise value from the extracted second noise value, and multiply the calculated difference value by a weight to calculate a noise-removed depth value.

Here, the difference value between the second noise value and the first noise value may be a depth value for the pure external light from which the noise is removed.

The weighting value may be a value considering the sensitivity characteristic for each pixel of the first sensing unit.

Then, the comparing unit 2430 can compare whether the depth value of which the noise is removed is larger than the physical noise value of the first sensing unit.

Next, if the result of the comparison is that the depth value from which the noise is removed is greater than the physical noise value of the first sensing unit, the final depth value determiner 2450 determines that the noise- Can be determined by the depth value.

Here, if the depth value with noise removed is larger than the physical noise value of the first sensing unit, it may have an effect on improving the sensitivity and contrast for the depth image. However, if the depth value from which the noise is removed is less than the physical noise of the first sensing unit If the value is less than or equal to the value, it improves the sensitivity and contrast for the depth image, which is not significant and can be ignored.

The 3D image processing method of the digital device according to the present invention will now be described.

16 and 17 are flowcharts illustrating a method of processing a three-dimensional image of a digital device according to the present invention, which will be described with reference to a three-dimensional image processing apparatus of FIG.

16 and 17, when receiving a video mode request signal from the outside, the control unit 1400 checks whether the video mode request signal is a 3D video mode request signal (S11). (S13 )

If the video mode request signal is a three-dimensional video mode request signal, the controller 1400 determines whether the resolution mode of the light receiver 1100 is the first resolution (S15).

If the resolution of the light receiving unit 1100 is the first resolution, the control unit 1400 can convert the resolution of the light receiving unit 1100 to a second resolution lower than the first resolution at the first resolution. (S17)

If it is determined in step S19 that the resolution mode of the light receiving unit 1100 is the second resolution (S21), the control unit 1400 determines whether the resolution mode of the light receiving unit 1100 is the second resolution.

If the resolution of the light receiving unit 1100 is the second resolution, the control unit 1400 converts the resolution of the light receiving unit 1100 to a first resolution higher than the second resolution at the second resolution, ) Two-dimensional image can be implemented. (S25)

In this manner, the control unit 1400 can convert the resolution of the light receiving unit 1100 from the first resolution to the second resolution lower than the first resolution. (S100)

The reason for changing the resolution of the light receiving unit 1100 is to reduce crosstalk.

Here, crosstalk refers to interference between an RGB frame for acquiring color image information and an IR frame for acquiring depth image information.

That is, IR illumination is required for the IR frame, and IR illumination is not required for the RGB frame. The crosstalk is a phenomenon in which the IR illumination of the IR frame affects the RGB frame, and noise or distortion occurs in the RGB image Phenomenon.

Therefore, in order not to affect the crosstalk influence on the RGB frames located before and after the IR frame, the IR illumination must be turned on instantaneously.

However, if the on-time of the IR illumination is very short, the IR illumination can not be recognized in the IR frame, and therefore the sensitivity of the depth image is inevitably lowered.

Therefore, in the present invention, in order to eliminate such a crosstalk phenomenon, in the three-dimensional image mode, the resolution of the light receiving unit 1100 is changed from the first resolution to the second resolution lower than the first resolution.

For example, if the resolution of the light receiving unit 1100 is reduced to about 1/4 of the resolution at the highest resolution, the time interval between the RGB frame and the IR frame is increased.

Therefore, the ON time of the IR illumination is increased by an increased time interval, and the IR illumination can be sufficiently recognized in the IR frame, and the sensitivity of the depth image can be improved.

Next, referring to the time measurement of the timer 1700, the control unit 1400 can turn off the driving of the light emitting unit 1500 so as not to emit the infrared light to the subject 1800 during the first time (S110)

Then, the light receiving unit 1100 can receive visible light from the object 1800. (S120)

The control unit 1400 controls the image processing unit 1200 to extract the color image information from the visible light sensed by the first sensing unit 1110 of the light receiving unit 1100 for the first time. )

Next, the control unit 1400 can refer to the time measurement of the timer 1700 and turn on the driving of the light emitting unit 1500 to emit the infrared light to the subject 1800 during the second time (S140)

Here, the driving time of the light emitting unit 1500 may be between the end time of the previous frame of the color image information and the start time of the next frame.

Then, the light receiving unit 1100 can receive infrared light from the object 1800. (S150)

The control unit 1400 may control the image processing unit 1200 to extract the depth image information from the infrared light sensed by the second sensing unit 1120 of the light receiving unit 1100 for a second time. S160)

In some cases, the light receiving unit 1100 can receive both visible light and infrared light from the object 1800. [

The control unit 1400 controls the image processing unit 1200 so that the depth image from the infrared light sensed by the first sensing unit 1110 and the second sensing unit 1120 of the light receiving unit 1100 during the second time, You can also extract all of the information.

Next, the control unit 1400 can determine whether the extraction of the color image information and the depth image information with respect to the subject 1800 is complete (S170)

If the control unit 1400 determines that the extraction of the color image information and the depth image information for the object 1800 is completed, the control unit 1400 controls the three-dimensional image implementation unit 1300 to extract the extracted color image information and the depth image Dimensional image of the subject 1800 on the basis of the information (S180)

However, if the extraction of the color image information and the depth image information for the subject 1800 is not completed, steps S110 to S160 may be repeated.

In some cases, steps S140 - S160 may precede steps S110 - S130.

That is, according to the present invention, driving of the light emitting unit 500 is turned off during the first time period after the driving of the light emitting unit 500 is turned on for the second time, And extract the color image information.

18 to 23 are schematic views for explaining a three-dimensional image processing process of the digital device according to the present invention.

As shown in FIG. 18, the 3D image processing apparatus of the digital device can extract the color image information during the first time and extract the depth image information during the second time in the three-dimensional image mode.

That is, the present invention utilizes the Frame by Frame technology that receives RGB frames and IR frames alternately.

As described above, the present invention can acquire the color image information and the depth image information at different times without acquiring the color image information and the depth image information at the same time.

19 and 20, the present invention can convert the resolution of the light receiving unit from the high resolution mode to the low resolution mode when a three-dimensional image is implemented.

Thus, the reason for changing the resolution of the light receiving unit is to reduce crosstalk.

Here, crosstalk refers to interference between an RGB frame for acquiring color image information and an IR frame for acquiring depth image information.

That is, IR illumination is required for the IR frame, and IR illumination is not required for the RGB frame. The crosstalk is a phenomenon in which the IR illumination of the IR frame affects the RGB frame, and noise or distortion occurs in the RGB image Phenomenon.

Therefore, in order not to affect the crosstalk influence on the RGB frames located before and after the IR frame, the IR illumination must be turned on instantaneously.

However, as shown in Fig. 19, if the on-time of the IR illumination is very short, the IR illumination can not be recognized in the IR frame, and therefore the sensitivity of the depth image is inevitably lowered.

Therefore, in the present invention, in order to eliminate such a crosstalk phenomenon, in the three-dimensional image mode, the resolution of the light receiving unit is changed from the first resolution to the second resolution lower than the first resolution.

For example, if the resolution of the light receiving unit is reduced to about 1/4 of the resolution at the highest resolution, the time interval between the RGB frame and the IR frame is increased.

Therefore, as shown in Fig. 20, the on-time of the IR illumination is increased by an increased time interval, and in the IR frame, the IR illumination can be sufficiently recognized and the sensitivity of the depth image can be improved.

On the other hand, the present invention can acquire the color image information and the depth image information at different times without acquiring the color image information and the depth image information at the same time.

That is, the time for acquiring the color image information and the time for acquiring the depth image information may be different from each other in the present invention.

The reason for this is that a depth image processing method capable of improving the sensitivity and contrast of depth images must be performed independently.

For example, as shown in FIG. 21, the present invention performs a binning process on acquired color image information.

Herein, the binning process refers to combining pixels of a plurality of sensors of a light receiving unit to obtain a new pixel. Since the binning process combines two or more pixels, the sensitivity and SNR (Signal to Noise Ratio) is improved, so that a clear color image can be obtained even in dark lighting.

In addition, as shown in FIG. 22, the present invention performs a sub-sampling process for an IR pixel with respect to the acquired depth image information.

Here, the sub-sampling process is a process of collecting only the depth information of the IR pixels in the IR frame. Since only the depth information of the infrared pixels is collected and output, the sensitivity and contrast of the depth image can be improved.

As an example, FIG. 23 is a diagram showing the IR illumination time when the RGB-IR sensor, which is a light receiving unit, is a low resolution mode which is 1/4 of high resolution and operates at about 60 fps.

Here, the RGB frame operates at about 30 fps, and the IR frame can operate at about 30 fps.

The RGB-IR sensor, which is a light receiving unit, operates in a high-resolution mode and implements a three-dimensional image when implementing a two-dimensional image as shown in FIG. 23, Separate frames into frames and IR frames.

In the RGB frame, a binning process is performed to extract color image information, and in an IR frame, a sub-sampling process can be performed to extract depth image information.

Here, when the resolution of the light receiving unit is lowered at about the resolution of about 1/4 at the highest resolution, the time interval between the RGB frame and the IR frame is increased. Therefore, the ON time of the IR illumination is increased by the increased time interval, The IR illumination can be sufficiently recognized and the sensitivity of the depth image can be improved.

That is, the driving time of the IR light emitting portion may be between the end time of the previous RGB frame and the start time of the next RGB frame.

24, the light emitting unit 1500 of the present invention is controlled by the control signal of the control unit 1400 to turn on / off the light emitting unit during the first time, as shown in FIG. 24, A step of emitting no external light, and a step of emitting infrared light during a second time period can be alternately repeated.

25, the image processing unit 1200 of the present invention, by the control signal of the control unit 1400, displays the image information processed by the image processing unit during the first time, Extracting the color image information, and extracting the depth image information during the second time period, may be alternately repeated.

FIG. 26 is a detailed flowchart illustrating a method of extracting the color image information of FIG. 16, which will be described with reference to the first image processing unit of FIG.

12 and 26, the first detection unit 2120 of the first image processing unit 2100 detects the amount of visible light detected from the first sensing unit 1100 of FIG. 3 during a first time period (S132)

In step S134, the color image information extracting unit 2160 extracts the color image information from the visible light in accordance with the control signal from the control unit. The color image information extracting unit 2160 extracts the color image information from the visible light (S135)

Here, the first image processing unit 2100 may detect infrared light from the first and second sensing units 1100 and 1200 in FIG. 3, but the amount of infrared light incident from the subject is small and can be ignored.

That is, although the infrared light may act as noise of the color image information, since the infrared light is not emitted from the light emitting portion, the amount of the infrared light to be sensed is very small.

Therefore, the first image processing unit 2100 does not have a large problem even if the additional noise removal operation is not performed because the noise on the extracted color image information is very small.

FIG. 27 is a detailed flowchart showing a method of extracting the depth image information of FIG. 16, which will be described with reference to the second image processing unit of FIG.

14 and 27, the second detection unit 2310 of the second image processing unit 2300 detects the first detection unit 1110 and the second detection unit 1120 of FIG. 1 during the first time, It is possible to detect the amount of infrared light detected from the light source (S162)

Then, the second conversion section 2330 converts the detected light amount of the visible light into an electrical signal (S164)

Then, the first depth image information extracting unit 2350 extracts the first depth image information from the infrared light sensed by the first sensing unit, and removes the noise of the extracted first depth image information (S166 )

Also, the second depth image information extracting unit 2370 can extract the second depth image information from the infrared light sensed by the second sensing unit (S166)

Next, the composing unit 2390 can synthesize the first depth image information from which the noise has been removed and the second depth image information (S168)

As described above, the present invention extracts first depth image information from which noises have been removed through the first sensing unit 1110 of FIG. 3, and synthesizes the extracted first depth image information into second depth image information , It is possible to improve the sensitivity and contrast of the depth image information with respect to the subject.

If a three-dimensional image of a subject is implemented using only the obtained second depth image information through the second sensing unit 1120 of FIG. 3 without the first depth image information being synthesized, the depth image information about the subject is degraded .

FIG. 28 is a view showing a bayer pattern in which first and second depth image information are not combined, FIG. 29 is a view showing a distribution of infrared light sensed by a light receiving unit, FIG. FIG. 5 is a view showing a bayer pattern in which depth image information is synthesized. FIG.

28, the light receiving unit 1100 may include a plurality of unit pixels including a first sensing unit that senses visible light of red, green, and blue wavelengths, and a second sensing unit that senses infrared light have.

Here, the first image processing unit of the present invention extracts the color image information from the first sensing unit, and the second image processing unit of the present invention does not extract the first depth image information from the first sensing unit, In the case of extracting only the second depth image information, as shown in FIG. 26, the color image information includes first color image information (R1 - R4) extracted from the first pixel of the first sensing unit that senses light in the red wavelength band, (G1 - G4) extracted from the second pixel of the first sensing unit for sensing the light of the first sensing unit (B1) extracted from the third pixel of the first sensing unit that senses the light of the blue wavelength band, - B4), and the depth image information may include only the depth image information IR1 - IR4 extracted from the second sensing unit for sensing infrared light.

Therefore, when the 3D image of the subject is implemented, the depth information and the contrast of the depth image are reduced only by the depth image information IR1-IR4 extracted only from the second sensing unit, .

29, the light receiving unit 1100 may detect the infrared light 1130 reflected from the subject, and the infrared light 1130 may include all or a part of the infrared light 1130, Only a part can be entered.

Therefore, since the light amount of the entire infrared light that can be detected is small, the light-receiving unit 1100 may be degraded in sensitivity and contrast with respect to the depth image extracted from the infrared light.

Therefore, the present invention extracts the first depth image information from which noises have been removed through the first sensing unit 1110 of FIG. 3, synthesizes the extracted first depth image information with the second depth image information, The sensitivity and the contrast of the depth image with respect to the subject can be improved.

30, the light receiving unit 1100 may include a first pixel 1112 of the first sensing unit 1110 for sensing the light of the red wavelength band by the first sensing unit 1110, A second pixel 1114 of the first sensing unit 1110 that senses the light of the blue wavelength band, a third pixel 1116 of the first sensing unit 1110 that senses the light of the blue wavelength band, The first depth information extracting unit 1120 extracts the noise-removed first depth image information through the first sensing unit 1110 and outputs the extracted first depth image information to the second depth image information It is possible to improve the sensitivity and contrast of the depth image with respect to the object.

FIG. 31 is a detailed flowchart showing a method of extracting the second depth image information of FIG. 16, which will be described with reference to the first depth image extracting unit of FIG.

As shown in FIGS. 14 and 31, the noise eliminator 2410 is configured to detect, during a first time period, a first noise value for the infrared light sensed from the first sensing unit and a second noise value for the second time from the first sensing unit The second noise value for the detected infrared light can be extracted.

Here, the first noise value may be visible light other than the infrared light sensed from the first sensing unit during the first time period

The second noise value may be visible light other than the infrared light sensed by the first sensing unit during the second time.

Next, the noise removing unit 2410 may calculate a difference value obtained by subtracting the first noise value from the extracted second noise value, and multiply the calculated difference value by a weight to calculate a depth value from which noise has been removed. (S1682 )

Here, the difference value between the second noise value and the first noise value may be a depth value for the pure external light from which the noise is removed.

The weighting value may be a value considering the sensitivity characteristic for each pixel of the first sensing unit.

32 is a graph showing light sensitivity characteristics of the first and second sensing units.

As shown in FIG. 32, it can be seen that the first and second sensing units exhibit different light sensitivity characteristics depending on the wavelength band of light.

For example, as shown in FIG. 28, the sensitivity of each pixel of the first sensing unit and the sensitivity of the second sensing unit may differ by about 3.5 times.

That is, in the first pixel of the first sensing unit that senses the light of the red wavelength band, the second pixel of the first sensing unit senses the light of the green wavelength band, and the third pixel of the first sensing unit senses the light of the blue wavelength band, Green light, and blue light, the sensitivity of the second sensing unit for sensing infrared light is about 3.5 times greater than that of the first sensing unit.

Therefore, when calculating the depth value from which the noise is removed, the difference between the second noise value and the first noise value is multiplied by a weight to remove the sensitivity difference.

Next, the comparison unit 2430 can compare whether the depth value of the noise is greater than the physical noise value of the first sensing unit (S1683)

Next, if the result of the comparison is that the depth value from which the noise is removed is greater than the physical noise value of the first sensing unit, the final depth value determiner 2450 determines that the noise- And can be determined as the depth value (S1684)

Here, if the depth value with noise removed is larger than the physical noise value of the first sensing unit, it may have an effect on improving the sensitivity and contrast for the depth image. However, if the depth value from which the noise is removed is less than the physical noise of the first sensing unit If the value is less than or equal to the value, it improves the sensitivity and contrast for the depth image, which is not significant and can be ignored.

33 and 34 are diagrams comparing contrasts of the depth images according to whether the first depth image information and the second depth image information are combined. FIG. 33 is a view for comparing the first depth image information and the second depth image information FIG. 34 shows a case where the first depth image information and the second depth image information are combined.

As shown in FIG. 33, when a three-dimensional image of a subject is implemented using only the acquired second depth image information through the second sensing unit 1120 of FIG. 3 without combining the first depth image information, The depth image information may be degraded.

However, as shown in FIG. 34, the noise extracted first depth image information is extracted through the first sensing unit, the second depth image information is extracted through the second sensing unit, and then the first depth image information It can be seen that the sensitivity and contrast of the depth image with respect to the subject are greatly improved when synthesized into the second depth image information.

Accordingly, the present invention can simultaneously process a color image and a depth image using a light receiving unit including a first sensing unit for sensing visible light and a second sensing unit for sensing infrared light, The image processing time and the overall cost can be reduced.

The present invention also provides a method for converting the resolution of a light receiving unit to a second resolution lower than the first resolution at a first resolution to increase the exposure time of infrared light between a previous frame end time of color image information and a next frame start time The sensitivity and contrast for the depth image are improved, and the depth feeling of the three-dimensional image can be improved.

The depth image information can be extracted not only by the infrared light received from the second sensing unit of the light receiving unit but also by the infrared light received by the first sensing unit of the light receiving unit, The sensitivity and contrast for the depth image are improved, and the depth feeling of the three-dimensional image can be improved.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the scope of the invention as defined by the appended claims. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention.

1100: light receiving unit 1110: first sensing unit
1120: second sensing unit 1200: image processing unit
1300: 3D image implementation unit 1400:
1500: light emitting portion 1600:
1700: Timer 1800: Subject

Claims (20)

A three-dimensional image processing method of a digital device including a light receiving unit including a first sensing unit for sensing visible light and a second sensing unit for sensing infrared light, and a light emitting unit for emitting the infrared light,
Converting the resolution of the light receiving unit from a first resolution to a second resolution lower than the first resolution;
Sensing the visible light and the infrared light from a predetermined subject;
Extracting color image information from the visible light sensed by the first sensing unit of the light receiving unit during a first time;
Extracting depth image information from the infrared light detected by the second sensing unit of the light receiving unit during a second time;
Determining whether extraction of color image information and depth image information for the subject is completed; And,
Dimensional image of the subject on the basis of the extracted color image information and depth image information if extraction of the color image information and depth image information for the subject is completed, Dimensional image processing method. The method of claim 1, wherein the step of converting the resolution of the light receiving unit to a second resolution lower than the first resolution at the first resolution comprises:
Receiving a video mode request signal;
Confirming a resolution mode of the light receiving unit if the video mode request signal is a three-dimensional video mode request signal; And,
And converting the resolution of the light receiving unit to a second resolution lower than the first resolution in the first resolution if the resolution of the light receiving unit is the first resolution, Dimensional image processing method. 3. The method of claim 2,
Confirming a resolution mode of the light receiving unit if the video mode request signal is a two-dimensional video mode request signal; And,
And converting the resolution of the light receiving unit to a first resolution higher than the second resolution at the second resolution if the resolution of the light receiving unit is a second resolution Dimensional image processing method. The method of claim 1, further comprising, prior to sensing the visible light and the infrared light,
Wherein the control unit turns off the driving of the light emitting unit to emit the infrared light during the first time or turns on the driving of the light emitting unit during the second time to emit the infrared light to the subject, The method further comprising the step of: 5. The method of claim 4, wherein the driving time of the light emitting unit is between an end time of a previous frame of the color image information and a start time of a next frame. 5. The digital device according to claim 4, wherein the step of not emitting infrared light during the first time period and the step of emitting the infrared light during the second time period are alternately repeated. Image processing method. The method of claim 1, wherein the step of extracting the color image information during the first time period and the step of extracting the depth image information during the second time period are alternately repeated. Dimensional image processing method. 2. The method of claim 1, wherein during the first time,
Detecting an amount of visible light detected from the first sensing unit during the first time;
Converting the light amount of the detected visible light into an electrical signal; And,
And extracting the color image information from the visible light. The method of claim 1, wherein, during the second time period,
Detecting the amount of infrared light detected from the second sensing unit during the second time;
Converting the amount of the detected infrared light into an electrical signal;
And extracting depth image information from the infrared light. A receiving unit for receiving a two-dimensional or three-dimensional video mode request signal;
A light receiving unit including a first sensing unit for sensing visible light corresponding to a visible region of the light spectrum from a predetermined subject and a second sensing unit sensing infrared light corresponding to the infrared region of the light spectrum;
An image processing unit for extracting color image information from the first sensing unit during a first time and extracting depth image information from the second sensing unit during a second time;
A three-dimensional image implementing unit for implementing a three-dimensional image of the subject based on the extracted color image information and depth image information; And,
And when the resolution of the light receiving unit is the first resolution, the resolution of the light receiving unit is changed from the first resolution to the first resolution And a control unit for controlling the light receiving unit, the image processing unit, and the three-dimensional image implementing unit. The apparatus of claim 10, wherein the first sensing unit comprises:
A first pixel for sensing red light, a second pixel for sensing green light, and a third pixel for sensing blue light. The apparatus of claim 10, wherein the first sensing unit comprises:
A fourth pixel for sensing yellow color light, a fifth pixel for sensing cyan color light, and a sixth pixel for sensing magenta color light. The apparatus of claim 10, wherein the first sensing unit comprises:
A first pixel for sensing red light, a second pixel for sensing green light, a third pixel for sensing blue light, a white color, a yellow color, a cyan color, and a magenta color And a seventh pixel for sensing color light. 11. The method of claim 10,
Further comprising a light emitting unit for emitting the infrared light for the second time period without emitting the infrared light for the first time period in accordance with the control signal of the control unit. 15. The digital device of claim 14, wherein the driving time of the light emitting unit is between an end time of a previous frame of the color image information and a start time of a next frame. 15. The method of claim 14,
And a timer for measuring a first time for turning off the driving of the light emitting unit and a second time for turning on the driving of the light emitting unit according to a control signal of the control unit . 11. The method of claim 10,
And a storage unit for storing the color image information and the depth image information extracted from the image processing unit. The image processing apparatus according to claim 10,
A first image processing unit for extracting color image information from the visible light sensed by the first sensing unit during the first time,
And a second image processor for extracting the second depth image information from the infrared light sensed by the second sensing unit during the second time. The apparatus of claim 18, wherein the first image processor comprises:
A first detection unit for detecting the amount of visible light detected from the first sensing unit during the first time;
A first conversion unit for converting the light amount of the detected visible light into an electrical signal; And,
And a color image information extracting unit for extracting color image information from the visible light. The apparatus of claim 18, wherein the second image processor comprises:
A second detection unit for detecting the amount of infrared light detected from the second sensing unit during the first time;
A second conversion unit for converting the light amount of the detected infrared light into an electrical signal; And,
And a depth image information extracting unit for extracting the depth image information from the infrared light.

Images