KR100446528B1 - Apparatus and method for processing stereoscopic image in portable phone - Google Patents

Abstract

According to an embodiment of the present invention, a transmitting apparatus of a portable terminal includes image data of one of the two left and right image data for the same subject photographed by two image input devices and difference information between the two image data. The data is coded by a predetermined compression scheme and transmitted as packet data, and according to the present invention, the receiving apparatus of the mobile terminal receives the packet data, decodes the one video data, and then uses another video data based on the difference information. It is configured to decode and output.

Description

3D image processing apparatus and method in a mobile terminal {APPARATUS AND METHOD FOR PROCESSING STEREOSCOPIC IMAGE IN PORTABLE PHONE}

The present invention relates to an apparatus and method for stereoscopic image processing, and more particularly, to an apparatus and method for processing stereoscopic images in a mobile terminal.

In general, Code Division Multiple Access (hereinafter referred to as CDMA) mobile communication system has been developed from the conventional mobile communication standard with a focus on voice, and has been developed to enable high-speed data transmission as well as voice. In the next generation mobile communication standard, services such as voice, moving picture, and internet search are possible. With the development of such a mobile communication system, a portable terminal capable of transmitting and receiving image information such as an International Mobile Telecommunication in the year 2000 (IMT2000) terminal has recently emerged.

1 is a block diagram of a portable terminal capable of image processing according to the prior art.

Referring to FIG. 1, the central processing unit 100 controls the overall operation of the communication terminal and performs various processing on the baseband signal. The RF unit 109 processes the radio frequency (RF) from the base station 111 received through the antenna as a baseband signal, and processes the baseband signal from the central processing unit 100 into a radio signal Send via The image input device 103 converts an optical image signal of a subject input through a lens into an electrical signal through a solid-state image sensor, and digitally processes the electrical signal and outputs the electrical signal to the central processing unit 100. The display unit 104 displays the operation state of the communication terminal on the liquid crystal display (LCD) under the control of the controller 100. In addition, the display unit 104 displays an image captured by the image input apparatus 103 and an image received. The liquid crystal display window is a flat image output device. On the other hand, the speaker 105 and the microphone 107 are devices for inputting and outputting voice signals.

The portable terminal having the above-described configuration codes and packetizes the image photographed by the image input apparatus 103 by the central processing unit 100, and transmits the image to the other side via the RF unit 109. The packet having the image data received through the RF unit 109 is restored to the image signal by the central processing unit 100 and displayed on the display unit 104 which is a flat image output device.

That is, the portable terminal according to the prior art is intended to transmit a planar image and cannot process a stereoscopic image. On the other hand, a lot of researches on the stereoscopic image processing apparatus has been made and the stereoscopic image TV and the camera is commercialized. The biggest problem of the commercialized stereoscopic image technology is the large number of calculations and transmission amounts and the display method of the stereoscopic image. That is, since it is a stereoscopic image, the image information is more than that of a flat image, and in order to view a stereoscopic image, an observation point (eye position) is fixed within a predetermined range in front of a display device (e.g., a TV screen) or glasses for viewing stereoscopic images Auxiliary devices are needed.

However, the mobile terminal has a merit that the position of the eyeball is fixed to the front of the screen because only one observer sees the display (screen), unlike a TV which requires a plurality of viewers to view one screen as a personal device. . Therefore, if the portable terminal is applied to the stereoscopic image processing method, it can be said to be easy to implement.

Accordingly, an object of the present invention is to provide a transmission and reception apparatus and method for processing a stereoscopic image in a mobile terminal.

Another object of the present invention is to provide an apparatus and method for transmitting a plurality of images photographed through at least two image input apparatuses in a portable terminal.

It is still another object of the present invention to provide an apparatus and method for displaying a plurality of images photographed through at least two image input apparatuses on a planar image output apparatus in a portable terminal.

The present invention for achieving the above object, in the transmission device of the mobile terminal for transmitting the packet data to the wireless network, two image inputs spaced at a predetermined interval, and outputs two image data left and right by photographing the same subject And a controller for coding and packetizing data including one image data of the two left and right image data and difference information between the two image data in a predetermined compression method, and the packetization from the controller. And a radio frequency (RF) unit for converting the transmitted image data into a radio signal and transmitting the same.

According to an aspect of the present invention, there is provided a receiving apparatus of a mobile terminal for receiving and displaying two image data, left and right, for a same subject photographed by two image input apparatuses as a packet, wherein the packet is stored in the left and right sides. One image data among two image data and difference information between the two image data, and receiving the packet to decode the one image data, and then decoding another image data based on the difference information. And a display for storing the decoded two image data, and alternately selecting the two stored image data at a predetermined time period, reproducing the image data as a video signal, and displaying the image signal on the display window.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram of a cellular phone capable of image processing according to the prior art.

2 is a block diagram of a portable terminal capable of stereoscopic image processing according to an exemplary embodiment of the present invention.

3 is a diagram showing an external configuration of a mobile terminal according to the present invention;

4 is a view for explaining the principle of recognition of a three-dimensional image according to the present invention.

5 is a view for explaining the principle of recognition of a three-dimensional image according to the present invention.

6 is a diagram illustrating a procedure for packetizing and transmitting a plurality of image data input from two cameras according to an exemplary embodiment of the present invention.

7 is a diagram illustrating a procedure for receiving and displaying image data photographed by two cameras according to an exemplary embodiment of the present invention.

8 is a view showing an example of a block configuration for compressing a stereoscopic image according to an embodiment of the present invention.

9 is a view showing a relationship between left and right image data and disparity according to the present invention;

10 is a view showing another example of a block configuration for compressing a stereoscopic image according to an embodiment of the present invention.

FIG. 11 is a diagram for describing a method of receiving and restoring data compressed by the compression method of FIG. 10. FIG.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the present invention, if it is determined that the detailed description of the related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

Hereinafter, a method of displaying a plurality of images captured by at least two image input devices (eg, cameras) on a display unit. That is, a method of displaying a plurality of images through a polarization filter or alternately displaying the plurality of images to allow a user to recognize a three-dimensional image, and to compress and transmit a plurality of images effectively. Will be explained.

2 is a block diagram of a portable terminal capable of stereoscopic image processing according to an exemplary embodiment of the present invention. The portable terminal illustrated in FIG. 2 is characterized by having two left and right image input devices 203-1 and 203-2.

Referring to FIG. 2, the central processing unit 200 controls the overall operation of the communication terminal and performs various processing on the baseband signal. According to the present invention, the central processing unit 200 codes (or compresses) and packetizes a plurality of image data received from at least two image input devices 203-1 and 203-2 and outputs them to the RF unit 209. The packet including the plurality of image data received from the RF unit 209 is depacketized and decoded to be restored to the original image data and output to the display unit 204. A method of coding the plurality of image data will be described later in detail. The RF unit 209 processes a radio frequency (RF) from a base station 211 received through an antenna as a baseband signal, and processes the baseband signal from the central processing unit 200 as a radio signal. Transmit via antenna

The first and second image input devices 203-1 and 203-2 convert the optical image signal of the subject input through the lens into an electrical signal through a solid-state image sensor, and digitally process the electrical signal. The image data is output to the central processing unit 200. For example, the first and second image input devices 203-1 and 203-2 may be a camera that is a general charge coupled device (CCD) input device. The display unit 204 is an operating state of the communication terminal under the control of the control unit 200, and the display unit 204 displays the images captured by the first and second image input devices 203-1 and 203-2 and the received images. It is displayed on a liquid crystal display (LCD). According to the present invention, the display unit 204 displays a plurality of images photographed at the same time point provided by the central processing unit 200 at different refraction angles using polarization filters or alternately displays the plurality of images. On the other hand, the speaker 205 and the microphone 207 are devices for inputting and outputting voice signals.

That is, the signal reconstructed by the central processing unit 200 is the left and right two image signals, and the user should be able to view the two images independently through the left and right eyes, respectively. A method of displaying two left and right images on one screen includes a method of separating the left and right images at different refractive angles using a polarization filter so that the left and right eyes are displayed separately, and using glasses having a screen blocking function. When the display device equipped with the polarization filter is used, the stereoscopic image may be viewed when the position of the eyeball is fixed. In the present invention, the method in the case of using the eyeglasses as an auxiliary device is described, and the method of using the polarizing filter can be explained by the same principle.

3 shows an external configuration of a portable terminal according to the present invention.

As shown, the two cameras corresponding to the image input apparatuses 203-1 and 203-2 may be connected to the side of the terminal as A-L and A-R, or may be connected to an external device such as A. FIG. In order to view stereoscopic images, glasses 204 connected to a mobile phone, such as D, or a polarizing filter is installed on the surface of the screen B so that a user can recognize images in three dimensions without assistive devices such as D. do.

4 and 5 are diagrams for explaining the principle of three-dimensional recognition of an image according to the present invention.

As shown in FIG. 4, when two cameras spaced apart by a predetermined distance photograph the front side, an object (subject) is disparity on the screen in proportion to the distance between the camera and the object. As far as is displayed. When the user views each of these images independently through the left and right eyes, the user recognizes the images by three-dimensional fusion. That is, as shown in FIG. 5, when the user independently recognizes two images displayed by being separated by the disparity on the screen, each of them is recognized as an image that protrudes or enters from the screen according to the difference. do.

On the other hand, since the data input from the two cameras is a large amount of data, it must be compressed for transmission. FIG. 6 illustrates a procedure for packetizing and transmitting a plurality of image data input from two cameras according to an exemplary embodiment of the present invention. The procedure of FIG. 6 may be shared by the image input device 203 and the central processing unit 200 in the configuration of FIG. 2.

Referring to FIG. 6, in operation 601, two image input apparatuses 203-1 and 203-2, left and right, photograph a subject to generate left and right image signals, and in step 603, digital image is converted by digitally converting the left and right image signals. The data is generated and provided to the central processing unit 200. In operation 605, the central processing unit 200 stores left and right image data from the two image input apparatuses 203-1 and 203-2 in different image memories. In operation 607, the CPU 200 encodes the left and right image data. Here, the encoding will be described in detail later. Thereafter, the central processing unit 200 makes the encoded two image data into one packet data for transmission in step 609, and transmits the packet data through the RF unit 209 in step 611.

7 illustrates a procedure for receiving and displaying image data photographed by two cameras according to an exemplary embodiment of the present invention.

Referring to FIG. 7, first, in step 701, packet data including image data captured by two cameras is received. In operation 703, the CPU 200 separates left and right image data from the received packet data. Here, the separated left and right image data are encoded data. Therefore, the central processing unit 200 decodes the separated left and right image data in step 705 and outputs the restored left and right image data to the display unit 204. Then, the display unit 204 stores the left and right image data in different image memories. In operation 708, the display unit 204 alternately selects the left and right image data in frame units, and in step 709, converts the selected image data into analog video and reproduces the image signal. In operation 711, the controller 200 displays the reproduced video signal on a liquid crystal display (LCD). In this case, since the left and right image signals are alternately displayed on the screen, when the user wears glasses that are alternately blocked left and right and views the screen, the displayed image may be three-dimensionally recognized. As another example, a polarization filter may be used more conveniently. In this method, the polarization angles of the left and right images are adjusted and displayed. When viewed from an appropriate distance from the screen, the left and right images are respectively illuminated according to the left and right eyes. That is, it is a method suitable for a portable terminal in which the position of the eyeball is fixed.

The video data captured by the two cameras is very large and needs to be compressed. In addition, it should be possible to display the image data photographed from the two cameras as a planar image for the receiver that does not have a stereoscopic image display unit 204 according to the present invention. To this end, the present invention proposes a stereoscopic image compression method as follows.

In general, since there is a large correlation between two images, a process of reducing redundancy is needed. On the other hand, a method of compressing by removing redundant information in one image is already developed and used. For example, a JPEG still image compression method is mentioned. The present invention uses a method of compressing one image data (for example, a left image) using a conventional image compression method, and compressing another image data (for example, a right image) using the left image. In such a method, a terminal having a stereoscopic image display unit decodes and displays both images, and in the case of a general terminal, only a left image using a conventional image compression method can be decoded and displayed. .

8 shows an example of a block configuration for compressing a stereoscopic image according to an embodiment of the present invention.

Referring to FIG. 8, the encoder 801 encodes and outputs left image data input in units of frames. For example, the encoder 801 may use a JPEG encoder. Here, the frame unit is a unit of image data corresponding to one screen, and the frame data is composed of blocks of a predetermined size (Block_BR_n) as shown in FIG. 9. The predetermined size block generally has a size of 8 pixels x 8 pixels.

The disparity / motion estimation (803) performs a function of finding a similar block from the left image (A_L) data and the right image data input in units of frames. That is, the difference estimator 803 searches the x-axis to find a similar block between the left image A_L and the right image A_R, and subtracts the disparity information corresponding to the distance between the two similar blocks. 805 and the difference coding unit 809. Here, the difference estimator 803 is similar to the motion estimation used in MPEG and the like, and unlike MPEG, which finds similar parts from two images having a time difference, finds similar parts from left and right images. Thus, in MPEG, the search area is on a two-dimensional plane, but in the present invention, it is on a horizontal line. That is, the difference between the left and right images is displayed on the horizontal line. Therefore, the motion vector has only a horizontal axis (x axis), and the amount of computation is small because the search area is reduced by that amount. In detail, as shown in FIG. 9, the difference estimator 803 finds the most similar block (Block_BL_x) in the left image with respect to an arbitrary block (Block_BR_x) of the right image, and compares the distance between the two similar blocks ( disparity).

Referring back to FIG. 8, the subtraction unit 805 inputs data corresponding to the left image A_L and the difference information from the difference estimating unit 803, and the left image A_L and the left image A_L. The residual image is constructed and output using the difference between the block data of the right image A_R. Here, the data corresponding to Block BR_x of the right image in the residual image is composed of (Block BL_x)-(Block_BR_x). That is, the residual image may have a large compression effect when coding the residual image because the information of the image is mostly close to zero.

The encoder 807 encodes and outputs data of the residual image from the subtractor 805. Here, the encoder 807 may use a JPEG encoder. The difference coding unit 809 codes and outputs difference information provided from the difference estimating unit 803. The packet generator 811 may include first compressed data d from the encoder 801, second compressed data e from the encoder 807, and third compressed data from the difference coding unit 809. Combine to generate packet data. The data transmission unit 813 converts the packet data from the packet generation unit 811 according to the transmission standard and transmits the converted data. This compression method reduces the loss of information and can transmit it, but has a large amount of data. Therefore, the compression method that can further reduce the amount of data is as follows.

10 shows another example of a block configuration for compressing a stereoscopic image according to an embodiment of the present invention. 10 illustrates a method of transmitting only compressed left image A_L and disparity information. In this case, since only the difference information is transmitted, the data amount is small, but the receiver needs to restore the right image with the difference information. This method is particularly effective when modifying the block size, which is the basic unit for calculating the disparity information. That is, it is effective to use (8 * 4 pixel) or (4 * 2 pixel) rather than the general size (8 * 8 pixels).

Referring to FIG. 10, the encoder 1001 encodes and outputs left image data input in units of frames. For example, the encoder 1001 may use a JPEG encoder. The difference estimator 1003 searches the x-axis to find a similar block between the left image and the right image A_R, and outputs the distance between the two similar blocks as disparity information. The difference coding unit 1005 codes and outputs the difference information from the difference estimating unit 1003. Here, the difference coding unit 1005 codes the difference information using a general 'Run length' compression scheme or an 'entropy' coding scheme, and the packet generator 1007 encodes the left image from the encoder 1001. Packet data is generated by combining the first compressed data compressed with data and the second compressed data compressed with the difference information from the difference coding unit 1005. The data transmitter 1009 generates the packet data. The packet data from 1007 is converted and transmitted according to a predetermined transmission standard.

Then, as shown in FIG. 11, the receiver restores the right image R from the reconstructed left image L. FIG. That is, the right image R1 is restored by shifting the corresponding left (deparity) from the decoded left image L1, the right image R2 is restored from the left image L2, and the right image R3 is subsequently restored from the left image L3. Goes. In this process, an overlapping part occurs like R12, and an area without data like R23 occurs. At this time, the portion corresponding to R12 is replaced by averaging the overlapping data of the left image L1 and L2 as shown (R12 = (lp1 + lp21) / 2), and the portion corresponding to R23 is the last pixel of L2 ( lp22) and the first pixel (lp3) of L3 are reproduced by linear interpolation.

As described above, the present invention compresses and transmits left and right image data from two image input units by two methods. One is a method of coding after combining the left image (A_L), residual image (Residual Image) and the disparity information as shown in FIG. 8, respectively, and the other is left image (A_L) as shown in FIG. And the code for combining and disparity information, respectively, are transmitted. It is effective to use the above two methods in parallel according to channel capacity or user's choice. In other words, if there are many users between the base station and the portable terminal and there is little space in the communication channel, the latter method with a small amount of transmission is used. Otherwise, the former method is used. In addition, regardless of the channel capacity, one of the two methods may be used according to the user's setting.

Meanwhile, in the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the scope of the following claims, but also by those equivalent to the scope of the claims.

As described above, the present invention can transmit or receive a stereoscopic image through a simple algorithm in the mobile terminal. That is, it is possible to send and receive stereoscopic images by attaching a camera, and there is an advantage of supporting various types of stereoscopic image data services such as product advertisements, movies, and animations.

Claims (8)

A transmitter of a portable terminal for transmitting packet data to a wireless network, Two image input devices spaced at predetermined intervals and outputting two left and right image data by photographing the same subject; A controller for coding and packetizing one image data of the two left and right image data and data including difference information between the two image data in a predetermined compression method; And a radio frequency (RF) unit for converting the packetized video data from the controller into a radio signal and transmitting the radio signal. delete A reception device of a portable terminal for receiving and displaying two left and right image data of a same subject photographed by two image input devices as a packet, The packet includes one image data of the two left and right image data and difference information between the two image data. A controller which receives the packet and decodes the one image data and then decodes and outputs another image data based on the difference information; And a display unit which stores the decoded two image data and alternately selects the stored two image data at predetermined time periods, reproduces them as image signals, and displays them on a display window. A reception device of a portable terminal for receiving and displaying two left and right image data of a same subject photographed by two image input devices as a packet, The packet includes one image data of the two left and right image data and difference information between the two image data. A controller which receives the packet and decodes the one image data and then decodes and outputs another image data based on the difference information; And a display unit configured to display the decoded two image data on a display window at different refractive angles using a polarization filter. In the transmission method of a mobile terminal for transmitting packet data to a wireless network, Two image input devices spaced at predetermined intervals, photographing the same subject to generate two image data at right and left; Coding and packetizing data including one image data among the two left and right image data and difference information between the two image data by a predetermined compression method; And converting the packetized video data into a wireless signal and transmitting the converted video data. delete In the receiving method of a mobile terminal for receiving and displaying two left and right image data of the same subject photographed by two image input devices in a packet, Receiving data including one image data of the two left and right image data and difference information between the two image data; Receiving the packet, decoding the one image data, and decoding and outputting another image data based on the difference information; Storing the decoded two image data; And alternately selecting the two stored image data at a predetermined time period, reproducing the image data as a video signal, and displaying the same on a display window. In the receiving method of a mobile terminal for receiving and displaying two left and right image data of the same subject photographed by two image input devices in a packet, Receiving data including one image data of the two left and right image data and difference information between the two image data; Receiving the packet, decoding the one image data, and decoding and outputting another image data based on the difference information; And displaying the decoded two pieces of image data on the display window using different polarization filters at different refractive angles.