KR100297831B1 - An apparatus for converting two dimensional image to stereoscopic image - Google Patents

Abstract

PURPOSE: A device for converting three-dimensional picture of two-dimensional continuous picture is provided to generate three-dimensional picture using motion of continuous two-dimensional pictures. CONSTITUTION: A device for converting three-dimensional picture of two-dimensional continuous picture includes a motion estimating part(200). The motion estimating part has a motion estimate controlling part(210) for receiving start signal in a frame unit and controlling a motion estimating process, a memory access address generating part(220) for receiving start address of a block to be processed and generating address of a memory to read out picture data of the block in order, a register(230a) for storing picture data read by the memory according to the memory access address generating part, subtracters(240a,240b) for obtaining difference between the read picture data, accumulative calculators(250a,250b) for obtaining a maximum value of the subtracter output according to signal of the memory access address generating part and accumulating the maximum values, and comparators/storages(260a,260b) for comparing output of the accumulative calculator with the previous output and storing the compared value.

Description

An apparatus for converting two dimensional image to stereoscopic image}

The present invention relates to a three-dimensional image conversion apparatus of a two-dimensional continuous image using the movement of the continuous two-dimensional image, that is, the conversion of the motion parallax to the horizontal parallax.

1 is a flowchart illustrating a conventional 3D image generating method using a correction time difference. This flowchart is a method of generating a conventional three-dimensional image, using a modified time difference (MTD) developed by Sanyo Electric Co., Ltd., Japan. This method is briefly described as follows.

First, a moving region is extracted from a continuous two-dimensional image (first step).

Next, the movement speed and direction of the extracted area are extracted (second step). Based on the extracted motion speed and direction of the current image, a delay direction and a delay time are determined from a previous image frame (step 3).

Next, according to the determined delay direction and the delay time, it is determined whether the delayed image and the current image are to be displayed in which eyes of the left and right eyes of the person, respectively (step 4).

A process of determining a left / right image in the method of converting a 2D continuous image into a 3D image in this order is conceptually illustrated in FIG. 2. The conceptual diagram of FIG. 2 conceptually illustrates a process of determining a left / right image when a camera for capturing a continuous two-dimensional image is fixed and an object (■) in the image is moving. The method of converting a 2D continuous image into a 3D image is an image conversion method using a corrected time difference. As shown in FIG. 2, when an object (■) moves to the right, the original image is displayed to the right eye. This method provides binocular disparity to a person by showing a time-delayed image in the left eye. On the other hand, when the object is moving to the left, the original image is shown to the left eye and the time-delayed image is displayed to the right eye, thereby providing a binocular disparity opposite to the above. The three-dimensional image generation method using the correction time difference has the following problems.

First, since one of the previous frames is selected as the delayed image based only on the motion information of the original image, another depth of each region of the original image is ignored. By ignoring the different depths possessed by the regions as described above, there is no depth when viewing stereoscopic images. For example, a three-dimensional effect may be sensed for a moving object, but a three-dimensional feeling may not be felt for an object having a small movement such as a background of a screen. Second, when the motion of the object between the original image and the delayed image has a vertical component as shown in FIG. 2, if the 3D image is generated by the method using the correction time difference, the human visual structure that obtains a sense of depth in the horizontal parallax Due to the mismatch of eye fatigue occurs.

Third, when the amount of motion of the object in the original image is very large, even if any frame is selected as a delayed image from a plurality of previous frames, the stereoscopic image generated by the original image and the delayed image at this time has a large parallax and thus human parallax recognition There is a problem outside of the limits.

The present invention was devised to improve the above problems, and after measuring the motion of a two-dimensional image, by converting the horizontal parallax according to the characteristics of the motion to generate a new image, it has a sense of depth for each area of the image, the object in the original image It provides a three-dimensional image conversion device of a two-dimensional continuous image that has a natural three-dimensional feeling even if the motion of the vertical component, and generates a three-dimensional image that can be recognized by human parallax even if the motion of the object in the original image is very large Its purpose is to.

1 is a flowchart illustrating a conventional 3D image generating method using a correction time difference;

FIG. 2 is a conceptual diagram illustrating a process of determining a left / right image in the method illustrated in FIG. 1;

3 is a schematic overall block diagram of a three-dimensional image conversion apparatus of a two-dimensional continuous image according to the present invention;

4 is a detailed block diagram of a motion estimation unit shown in FIG. 3;

FIG. 5 is a detailed block diagram of a memory access address generation unit shown in FIG. 4;

6 is a detailed block diagram of the horizontal parallax generating unit shown in FIG. 3.

<Description of the symbols for the main parts of the drawings>

10.ADC 20.First Frame Memory

21. Second Frame Memory 22. Third Frame Memory

100. Input unit 200. Motion estimation unit

210. Motion estimation controller 220. Memory access address generator

221. Counter to increase address 222. Horizontal block start address

223. Starting block of vertical block 230a. register

230b. Register 240a. Subtractor

240b. Subtractor 250a. Absolute value and accumulator

250b. Absolute value and accumulator 260a. Comparator

260b. Comparator 270a. Horizontal movement vector

270b. Vertical motion vector 300. Horizontal parallax generator

400. Composite image generator 401. User control terminal

500.Output part 510.DAC

520a. Left video output device 520b. Right video output device

In order to achieve the above object, the memory for dividing the digital video signal converted from the analog video signal into a brightness signal and a color signal and storing the original image frame into blocks of a predetermined size, and in the memory corresponding to each block. In the three-dimensional image conversion apparatus for a two-dimensional continuous image according to the present invention having a motion estimation unit for comparing the stored image data and the image data of each block of the previous frame to measure a motion vector, the motion estimation unit starts in units of frames A motion estimation adjusting means for receiving a signal and adjusting a motion estimation process, and receiving a start address of a block to be processed from the motion estimation adjusting means, so as to sequentially read image data in the block from the memory. Generate memory access address A subtractor for obtaining a difference between the image means, the register storing the image data read from the memory according to the signal of the memory access address generation means, the subtractor for obtaining the difference between the read image data, and the signal of the memory access address generation means. An absolute value calculator / accumulator for obtaining an absolute value of the subtractor output, accumulating the absolute value, and a comparator / store for receiving and receiving a signal from the motion estimation adjusting means and comparing and storing the output of the accumulator with a previous output. desirable.

Here, the motion estimation adjusting means determines the position of the next block to be compared according to the comparison result of the comparator, and the motion estimation block in the current frame to estimate the motion to the memory access address generating means according to the positioning result. When the comparison of the image data between the motion estimation block and the previous frame blocks is finished, the position difference between the motion estimation block and the block of the previous frame having the smallest difference is determined according to the comparison result of the comparator. It is preferable to output the data, and to read and process the image data of the compared blocks of the previous image frame located in the horizontal direction of the block in the frame to estimate the motion and to be located in the vertical direction of the block in the frame to estimate the motion Compared of previous video frame In order to independently perform the operation of reading and processing the image data of the locks, the memory access address generating means may be provided with one or two, and the reading speed is doubled, and the register, subtractor, absolute value calculator / accumulator, Preferably, two comparators / stores are provided or one is provided, and the speed is doubled. The memory access address generating means includes a lookup table and an adder, starting with a block start signal of the motion estimation adjusting means. Preferably, the address of the lookup table is increased while the counter is increased, and the address of the memory is sequentially output while adding the output value of the lookup table and the block start address provided from the motion estimation adjusting means.

The horizontal parallax generating unit, which is further included in the 3D image converting apparatus of the 2D continuous images, and obtains horizontal parallaxes from the motion vectors of the respective blocks measured according to the motion characteristics of the original image, moves in the vertical direction with the horizontal motion vectors. The distance of the motion vector is obtained by using a lookup table including a calculation result of a vector that may occur in the process of calculating the distance of the vector, and the result is subtracted from the maximum horizontal parallax. It is preferable to re-adjust according to the output value and use it as horizontal parallax.

Hereinafter, an apparatus for converting a 3D image of a 2D continuous image according to the present invention will be described in detail with reference to the accompanying drawings.

The three-dimensional image conversion apparatus of the two-dimensional continuous image according to the present invention measures the movement of the two-dimensional image, and then converts to horizontal parallax according to the movement characteristics to generate a new image, thereby having a sense of depth for each area of the image, within the original image Even if the motion of the object has a vertical component, it has a natural three-dimensional feeling and generates a three-dimensional image that can be recognized as human parallax even if the motion of the object in the original image is very large, that is, converting the motion parallax into horizontal parallax An apparatus for performing a video signal processing to convert a two-dimensional continuous image to a three-dimensional image in real time. Specifically, by receiving a composite video signal such as TV, VCR, etc., the video signals corresponding to the left and right eyes are generated in real time and the left and right eyes of the observer are separately displayed and shown in two dimensions. It is a device that allows you to watch video information in three dimensions.

As described above, in order to process a video signal for converting a continuous two-dimensional image into a three-dimensional image, a three-dimensional image conversion apparatus for a two-dimensional continuous image according to the present invention, as shown in FIG. An input unit 100 that converts a digital image signal into a brightness signal and a color signal and stores the same in a memory, and a motion adder that measures a motion vector using a previous image frame for each block for each block unit of a predetermined size. A horizontal parallax generating unit 300 which obtains horizontal parallaxes from the motion vectors of each block according to the motion characteristics of the current image, and synthesizes corresponding to the current image by moving each block in the horizontal direction according to each horizontal parallax. An image generating unit 400 for generating an image and an output unit 500 for displaying a three-dimensional image consisting of the current image and the composite image. The user generation terminal 401 is added to the image generator 400. The three-dimensional image conversion apparatus of the two-dimensional continuous image having such a configuration does not use two cameras, that is, a stereoscopic camera, and a two-dimensional composite video signal input through an image reproducing apparatus such as a TV, a VCR, a CD, an HDTV, or a DVD By using it as it is to be able to watch a three-dimensional image, in brief, there is a feature that generates a three-dimensional image using a two-dimensional continuous image taken by a monocular video camera. To this end, in particular, a real-time image signal processing unit called a block motion estimator 200 and a horizontal parallax generator 300 is provided.

The three-dimensional image conversion apparatus of the two-dimensional continuous image configured as described above generates a horizontal parallax therefrom after measuring a motion vector between two consecutive images in the two-dimensional continuous image photographed with a monocular video camera, and then generates a horizontal parallax from the original image. By generating a new image moved in the horizontal direction, a three-dimensional image consisting of the original image and the new image is generated. To this end, as shown in FIG. 3, the input unit 100 receives a continuous two-dimensional image captured by a monocular video camera, that is, an analog form of two-dimensional image signal, using an analog / digital converter (ADC) 10. Convert to a digital video signal. The converted digital image signal is divided into a brightness signal (or a luminance signal) and a color signal and stored in the memories 20, 21, and 22 for each frame. The memory should be able to store video signals of three frames. That is, a memory for storing a video signal that is currently being input and a memory for storing a video signal of two previous frames necessary for estimating a motion should be provided. On the other hand, the three memories are provided to store the image signal F (k) being input, to store the image input F (k-1) before the image F (k-2). Storage is sequentially performed. The process of assigning the role to each memory or toggling the role for each time is implemented by converting the role of each memory using a vertical synchronization signal of the input image in the central controller. The input image stored in the memory is divided into luminance and color signals. In the motion estimation operation, only the luminance signal is used.

As illustrated in FIG. 4, the motion estimation unit 200 divides and processes an image signal of one frame into block units having a predetermined size. Here, the predetermined size is the number of pixels in the block, and the number of vertical and horizontal pixels is represented by m × n. It is preferable that m and n are 4 or 8. As such, the reason for blocking the image is to estimate the motion by distinguishing the main object having motion in the image from the background with little motion. The motion estimation measures the magnitude and direction of the motion, that is, the motion vector by comparing the blocks of the input image F (k-1) with several blocks corresponding to the previous image F (k-2). . For any block B of image F (k-1), find the image block most similar to block B in the previous image F (k-2) and display the difference in its position as a vector, that is, a horizontal component and a vertical component. One is a motion vector, and to find the similarity between blocks, one must sum the difference with the corresponding pixel value for every pixel value in the block. In this process, when the size of the block is m = n = 8, 8 * 8 = 64 pairs of pixel values should be compared to compare a pair of blocks. Since the operation must be performed on all blocks of the previous image near the position of the current block, a large amount of computation is required. The motion estimation unit 200 requires a large amount of computation in the 3D image conversion apparatus of the 2D continuous image of the present invention to make real-time processing difficult. In view of this point, the present invention includes a motion estimator for real-time motion estimation as shown in FIG. 4 to reduce the amount of computation by calculating the horizontal and vertical directions of the motion separately to enable parallel processing. Of course, if the horizontal and vertical directions of motion are calculated separately and processed in parallel, it is difficult to estimate the exact motion.However, in the case of a stereoscopic image, the motion is less than the concept of finding a block of the previous image having the smallest difference from the original block. Since there is a greater purpose in estimating, we can get acceptable results. The motion estimation unit 200 as shown in FIG. 4 operates as follows.

When a signal indicating the start of one frame is transmitted to the motion estimation controller 210, the motion estimation controller 210 starts sequential image signal processing in units of blocks. The motion estimation adjusting unit 210 first sends the start address of the block to be processed and the start command of the block to the memory access address generating unit 220. The memory access address generation unit 220 generates address numbers in order of blocks so that the data of the corresponding block can be read from the two frame memories F (k-2) 20 and F (k-1) 21. Do what you do. In the image data of each frame memory, data in the horizontal direction is input from the top to the bottom of the screen in order. Therefore, when one block data is to be read in order, the address to read the data should start from the start address of the block and approach the block size horizontally, and then jump by the number of pixels in the horizontal direction. As an example, FIG. 5 is a block diagram of a memory access address generation unit constituting circuit when the size of a block is 8 * 8 and the size of an image is 720 * 480. In order to process the actual image data, a very fast processing speed is required. In particular, the motion estimation unit 200 has the largest amount of computation, and the access address generator 220 requires the fastest processing speed. Therefore, the access address generation unit 220 has to increase the processing speed with a simple structure, but the present invention uses a look up table (LUT). When the start address of the block is given from the motion estimation controller 210 together with the block start signal, the memory access address generator 220 sequentially increases the value of the address increasing counter 221 so that the LUT values are sequentially output. do. The value of the output LUT is added to the start address of the block, and the result is sequentially output as the memory access address. Therefore, the value determined according to the block size and the horizontal size of the image should be recorded in the LUT to sequentially output the block values. As an example, it can be seen that the value increased by 720, which is the horizontal size of the image, is recorded in the LUT shown in FIG. In addition, the memory access address generation unit 220 may repeatedly perform the horizontal and vertical directions once to generate the memory access address, and may simultaneously process the horizontal and vertical motion vectors. Therefore, in the LUT of FIG. 5, the block start address includes two horizontal and vertical blocks 222 and 223. The memory access address generator 220 adjusts the horizontal block start address and the vertical block start address to be sequentially selected. do. The image data read from the memory by the above process is stored in the registers 230a and 230b by the data read signal given from the memory address generator 220 and then subtracted from the memory address generator 220 by the subtractors 240a and 240b. ) And the difference between the images is accumulated by the subtraction and accumulation signals given to the absolute value / accumulators 250a and 250b, respectively. When the above process is performed for one block, the difference between the front and rear image blocks is output as a result, and the memory address generator 220 outputs the block completion signal to the motion estimation controller 210. When the block completion signal is given to the motion estimation adjusting unit 210, the motion estimation adjusting unit 210 stores the value when the difference value of the calculated block is smaller than that of the previous block, and the first frame memory F at that time. (k-2) The position difference value between the block in the second frame memory F (k-1) 21 and the block in the second frame memory F (k-1) 21 is calculated for the corresponding block of the second frame memory F (k-1) 21. Store as a horizontal or vertical motion vector. At the same time, the motion estimation adjusting unit 210 transmits the start address and the block start signal of the block in the F (k-2) 20 to be compared next to the memory access address generator 220 with reference to the comparison result. When the above process is completed for one block of the image data of the second frame memory F (k-1) 21, the state of the associated registers 230a, 230b, 240a, 240b, 250a, 250b is initialized and the second frame memory F (k-1) The processing for the next block in 21 is started. After the processing of all blocks of the image data of the second frame memory F (k-1) 21 is completed, the motion estimation adjusting unit 210 sends a frame completion signal to the main adjusting unit (not shown) and waits for the next frame start signal.

The motion vector value obtained above should be converted into a horizontal parallax value for image synthesis. The horizontal parallax value is the sum of two horizontal and vertical vectors as shown in the following equation.

px = root {Vx ^ 2 + Vy ^ 2}

Here, Vx and Vy are motion vectors in the horizontal direction and the vertical direction, respectively.

In this process, a device for calculating square and square root values is required, but in the present invention, a table for all possible horizontal and vertical values is created and the computing device is referred to to replace the computing device. This simplifies the implementation of a complex device. The table is not large because the values of the possible horizontal and vertical vectors are predefined within a few. As an example, 23 types of vectors may be determined, respectively, horizontally and vertically. The calculated horizontal parallax value is subtracted from the already defined maximum possible horizontal parallax value. The subtracted result is readjusted and output according to the user's adjustment value, and the result is used by the synthesized image generator 400 to synthesize the image of the corresponding block.

The composite image generator 400 generates a composite image corresponding to the current image by moving each block in the horizontal direction according to each horizontal parallax.

The output unit 500 includes a digital / analog converter 510 for converting digital image signals into an analog image signal and left and right image output devices 520a and 520b, respectively. The composite image signals are converted into analog image signals, and the converted analog image signals are displayed through output terminals of the left and right image output devices 520a and 520b. At this time, the original image is displayed in the left eye of the person based on the positive parallax, the composite image is displayed in the right eye of the person, and based on the positive parallax, the horizontal parallax calculator 310 as shown in FIG. ), A subtractor 320 and an adder / subtracter 330, to synthesize the image using the result of subtracting the horizontal parallax value calculated above from the maximum horizontal parallax as possible reduces eye fatigue and looks more natural. .

As described above, the three-dimensional image conversion apparatus of the two-dimensional continuous image according to the present invention has a sense of depth for each region of the image, the camera movement and the vertical movement and rapid movement of the object in the image that cannot be solved by conventional techniques Even if this occurs, there is an advantage of generating a three-dimensional image that has a natural three-dimensional feeling and recognizable by human parallax. In particular, the apparatus for implementing a 3D image conversion real-time 2D continuous image according to the present invention generates a 3D image in real time using a 2D image captured by a monocular video camera without using two cameras, that is, a stereoscopic camera. It enables to watch stereoscopic images through image reproducing devices such as TV, VCR, DVD, HDTV, and CD, and also applies to medical field using endoscope and ultrasound to read stereoscopic images to improve diagnostic efficiency. In addition to these applications, there are advantages that can be easily applied in many areas, including the entertainment area for making animations in three dimensions. In addition, in the present invention, by configuring a device capable of real-time processing with a simple structure using the vertical horizontal separation of the movement and the look up table (LUT), there is an effect that the manufacturing cost is significantly reduced.

Claims (5)

The memory for dividing the digital video signal converted from the analog video signal into a brightness signal and a color signal and storing the original image frame into blocks of a predetermined size, and the image data stored in the memory corresponding to each block and the previous frame Motion estimation unit for measuring motion vectors by comparing the image data for each block In the three-dimensional image conversion apparatus of a two-dimensional continuous image having a motion estimation unit, Motion estimation adjusting means for adjusting a motion estimation process by receiving a start signal in units of frames; Memory access address generation means for receiving a start address of a block to be processed from the motion estimation adjusting means and generating an address of the memory so as to sequentially read image data in the block from the memory; A register for storing image data read from the memory in response to a signal of the memory access address generation means; A subtractor for obtaining a difference between the read image data; An absolute value calculator / accumulator for obtaining an absolute value of the subtractor output according to a signal of the memory access address generation means and accumulating the absolute value; And a comparator / storer for receiving the signal of the motion estimation adjusting means and comparing and storing the output of the accumulator with a previous output. According to claim 1, The motion estimation adjusting means determines the position of the next block to be compared according to the comparison result of the comparator, and starts the motion estimation block in the current frame to estimate the motion to the memory access address generating means according to the positioning result. When a comparison of the image data between the motion estimation block and the previous frame blocks is finished, the address difference is determined and output according to the comparison result of the comparator and determining the position difference between the motion estimation block having the smallest difference and the block of the previous frame. 3D image conversion apparatus for a two-dimensional continuous image, characterized in that. According to claim 1, The operation of reading and processing the image data of the compared blocks of the previous image frame located in the horizontal direction of the block in the frame to estimate motion and the comparison of the previous image frames located in the vertical direction of the block in the frame to estimate motion In order to independently perform an operation of reading and processing image data of blocks, two or more memory access address generating means may be provided, and the reading speed is doubled, the register, subtractor, absolute value calculator / accumulator, A three-dimensional image conversion apparatus for a two-dimensional continuous image, characterized in that each provided with two comparison / storage or one, but the speed is doubled. The method according to any one of claims 1 to 3, The memory access address generating means includes a lookup table and an adder, increases an address of the lookup table while increasing a counter starting with a block start signal of the motion estimation adjusting means, and outputs the motion and the output value of the lookup table. And a block start address provided from an estimation adjusting means, and sequentially outputs the address of the memory. The apparatus of claim 4, wherein the horizontal parallax generating unit further comprises a horizontal parallax generating unit configured to obtain horizontal parallaxes from the motion vectors of the respective blocks measured according to the motion characteristics of the original image. The distance of the motion vector is obtained by using a lookup table including a calculation result of a vector generated in the process of calculating the distance between the horizontal motion vector and the vertical motion vector, and the resulting horizontal parallax. Subtracting from the three-dimensional image conversion apparatus for a two-dimensional continuous image, characterized in that to use the horizontal parallax by re-adjusting it according to the output value of the terminal adjusted by the user.