AU2004200237A1

AU2004200237A1 - Image processing apparatus with frame-rate conversion and method thereof

Info

Publication number: AU2004200237A1
Application number: AU2004200237A
Authority: AU
Inventors: Jeong-Woo Kang; Jong-Sul Min; Young-Wook Sohn
Original assignee: Samsung Electronics Co Ltd
Current assignee: Samsung Electronics Co Ltd
Priority date: 2003-02-03
Filing date: 2004-01-22
Publication date: 2004-08-19
Anticipated expiration: 2024-01-22
Also published as: CN1520180A; KR20040070487A; KR100467625B1; AU2004200237B2; CN100594723C

Description

L_

AUSTRALIA

Patents Act COMPLETE SPECIFICATION

(ORIGINAL)

Class Int. Class Application Number: Lodged: Complete Specification Lodged: Accepted: Published: Priority Related Art: Name of Applicant: Samsung Electronics Co., Ltd Actual Inventor(s): Jeong-woo Kang, Jong-sul Min, Young-wook Sohn Address for Service and Correspondence: PHILLIPS ORMONDE FITZPATRICK Patent and Trade Mark Attorneys 367 Collins Street Melbourne 3000 AUSTRALIA Invention Title: IMAGE PROCESSING APPARATUS WITH FRAME-RATE CONVERSION AND METHOD

THEREOF

Our Ref: 711891 POF Code: 460249/462172 The following statement is a full description of this invention, including the best method of performing it known to applicant(s): -1- Soeq IMAGE PROCESSING APPARATUS WITH FRAME-RATE CONVERSION AND METHOD THEREOF This application claims the priority of Korean Patent Application No.

2003-6537, filed on February 3, 2003, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a frame-rate converter using a technique of motion estimation, and more particularly, to an image processing apparatus with the frame-rate conversion and method thereof to provide appropriate filtering and compensating for'a motion vector.

2. Description of the Related Art Typically, an image processing method in a high picture quality display, such as a digital television or a plasma display panel (PDP), uses a progressive scan method. In the progressive scan method, a whole image of a video frame is rendered at a timein the unit of a frame, just as a film is projected on a screen.

FIG. 1 is a block diagram of a part of an image processing apparatus in a conventional digital display device. The portion includes a de-interlacing unit 100, an 1-frame delaying unit 110, a first pixel block 120, a second pixel block 130, a sum-of-absolute difference map (SAD map) 140, a motion vector extraction unit 150, a frame-rate conversion (FRC) unit 160, and an interlacing unit 170.

The de-interlacing unit 100 converts input images from fields into frames.

The 1 -frame delaying unit 110 delays the frames to enable a previous frame (P frame),to be compared with a current frame (C frame).

The first pixel block 120 extracts a C x C pixel block from the C frame, where C 2".

The second pixel block 130 extracts a Cx C pixel block from the P frame.

The SAD map 140 compares the pixel blocks of the C frame and the P frame with each other to obtain difference between the pixel blocks and stores the result.

The motion vector extraction unit 150 estimates a motion vector of the current pixel block using the value in the SAD map 140.

The FRC 160 performs a frame-rate conversion for the extracted motion vector.

The interlacing unit 170 re-converts the frame-rate converted image back into a field.

In the image processing apparatus of FIG. 1, in order to perform the frame-rate conversion accurately, C frame and P frame are at the same resolution.

However, an input interlaced image comprising a current and a previous field contains a pixel difference between the two fields, the pixel difference existing in the vertical direction. The pixel difference between the two fields may lead to problems in comparing the two fields and extracting a vector motion. Therefore, the de-interlacing unit 100 performs a vertical image interpolation and protects a pixel difference between two images. The de-interlaced image is stored in a memory (not shown), and a predetermined block extracted from the current de-interlaced image (C frame) is compared with a block in the same position of the previous de-interlaced image (P frame) which has been delayed by the 1-frame delaying unit 110. The blocks from the previous frame and the current frame are compared by the SAD map 140 in order to estimate where and by how much the previous frame has moved, thereby the motion vector extraction unit 150 can determine a motion vector of the image. The predetermined block is a C x C square block. The C x C square block is used for motion vector estimation because the number of pixels in the vertical direction is doubled through de-interlacing. When using an interlace display.format, there are twice as many pixels in the horizontal direction as pixels in the vertical direction and a Cx C/2 block used for the motion vector estimation. The estimated motion vector is input to the FRC unit which makes a new frame and outputs it to the interlacing unit 170 which adapts the frame to an output display format in which the number of vertical pixels of the new frame is diminished in half.

In the image processing apparatus described above, since an input image is de-interlaced at the early stage, a higher memory capacity is required and the amount of operation is increased in proportion to the memory capacity. Also when the image is interlaced at the final stage, additional memory is further required. Due to the need for the large memory capacity and the amount of operation, the hardware for the image processing is not only enlarged but also leads to heating problems and errors in signal processing. In addition, an image which has undergone de-interlacing and interlacing is not the same as the original image and the quality of the image deteriorates.

The discussion of the background to the invention herein is included to explain the context of the invention. This is not to be taken as an admission that any of the material referred to was published, known or part of the common general knowledge in Australia as at the priority date of any of the claims.

Doanrt2 SUMMARY OF THE INVENTION The present invention provides an image processing apparatus with frame-rate conversion and method thereof, in which a stage of de-interlacing of an input image is eliminated to improve an efficiency of operation of image processing.

According to an aspect of the present invention, there is provided an apparatus which processes images with frame-rate conversion (FRC), the apparatus i ncl udi ng,a block extraction unit which extracts a predetermined block from an input field; a motion vector extraction unit which determines a motion vector of the predetermined block by comparing blocks of two sequentially input fields with each other; a motion filter which determines whether there has been any motion by using the motion vectors of pixels in a predetermined range of the block; a motion compensation unit which passes values of the motion vectors intact if it is determined that there has been a motion in the predetermined range of the block, and changes the values of the motion vectors to 0, otherwise; and a frame-rate conversion unit which performs a frame-rate conversion on the motion vectors received from the motion compensation unit.

It is preferable that the block extracted by the block extraction unit has dimensions Cx C/2, where C=2" It is preferable that the block extraction unit further comprises, a delay unit which delays the input interlaced image by one field; a first block extraction unit which extracts a block from the delayed input interlaced image; and a second block extraction unit which extracts a block from the current input interlaced image.

It is preferable that the apparatus further comprises a sum-of-absolute-difference (SAD) map which compares pixels in the first block extraction unit and in the second block extraction unit with each other to obtain differences therebetween.

It is preferable that the motion vector extraction unit estimates motion vectors of the input fields from the SAD map.

It is preferable that the motion filter sets a predetermined mask block, calculates absolute values of motion vectors output from the motion filter, and determines whether the number of the motion vectors which have an absolute value which is equal to or less than a predetermined number of pixels is equal to or more than a predetermined number.

It is preferable that the motion compensation unit converts the absolute values of the motion vectors in the mask block to 0 if the number of the motion vectors which have an absolute value which is equal to or less than a predetermined number of pixels is equal to or more than the predetermined number.

It is preferable that the motion compensation unit passes the motion vectors in the mask block intact if the number of the motion vectors which have an absolute value which is equal to or less than a predetermined number of pixels is less than a predetermined number.

It is preferable that the predetermined number of pixels is 1, and the predetermined number is the number of pixels covered by the mask block.

According to another aspect of the invention, there is provided a method of image processing with frame-rate conversion, the method including, extracting a predetermined block from an input interlaced image; estimating motion vectors of the pixels in the predetermined block by comparing blocks of two sequentially input interlaced images; determining whether there has been a motion by considering values of the motion vectors of pixels in a predetermined range of the block; passing the values of the motion vectors of pixels in the predetermined range of the block intact if it is determined that there has been any motion, and forcing the values of the motion vectors of pixels in the predetermined range of the block to 0 if it is determined that there has been no motion; and performing a frame-rate conversion (FRC) on the resulting motion vectors.

It is preferable that the predetermined block is extracted to have dimensions CxC/2, where C=2" It is preferable that the step of extracting the predetermined block comprises, delaying the input interlaced image by a field and extracting a first block from the delayed input interlaced image; and extracting a second block from the current input interlaced image.

It is preferable that the method further comprises comparing corresponding pixels in the first block and the second block to obtain differences therebetween.

it is preferable that the method further comprises estimating motion vectors of the input interlaced images from the differences and storing the results.

It is preferable that the step of determining if there has been a motion comprises, setting a predetermined mask block; calculating absolute values of the motion vectors; and determining whether the number of motion vectors of pixels covered by the predetermined mask block which have an absolute value which is equal to or less than a predetermined number of pixels is equal to or more than a predetermined number.

It is preferable that the step of determining if there has been a motion comprises forcing the absolute values Of the motion vectors covered with the mask block to 0 if the number of the motion vectors which have an absolute value which is equal to or less than a predetermined number of pixels is equal to or more than a predetermined number.

It is preferable that the step of determining if there has been a motion comprises passing the motion vectors in the mask block intact if the number of the motion vectors which have an absolute value which is equal to or less than a predetermined number of pixels is less than a predetermined number.

It is preferable that the predetermined mask block has dimensions of 3x3 pixels, and the predetermined number of pixels is 9.

According to yet another aspect of the invention, there is provided an apparatus which processes images with frame-rate conversion (FRC), the apparatus including a motion filter which determines whether a motion vector estimated from an input interlaced image is caused by a line deviation between odd and even fields; and a motion compensation unit which passes the motion vector intact if it has not been caused by the line deviation, and changes the motion vector to 0, otherwise.

According to still another aspect of the invention, there is provided a method of image processing with frame-rate conversion (FRC), the method including, determining whether a motion vector estimated from an input interlaced image is caused by a line deviation between odd and even fields; and compensating for the motion vector by passing the motion vector intact if it has not been caused by the line deviation, and changing the motion vector into a null vector, otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other aspects and advantages of the present invention will become more apparent by describing in detail preferred embodiments thereof with reference to the attached drawings in which: FIG. 1 is a block diagram of a conventional image processing apparatus; FIG. 2 is a block diagram of image processing apparatus according to a preferred embodiment of the present invention; FIG. 3 is a flowchart of a motion filtering method according to a preferred embodiment of the present invention; FIG. 4 is an example of an original input image; FIG. 5 shows a partial result of image processing for the original input image of FIG. 4, the image processing having de-interlacing and interlacing stages; FIG. 6 shows a partial result of image processing for the original input image of FIG. 4, the image processing not having a de-interlacing stage; and FIG. 7 shows a partial result of image processing for the original input image of FIG. 4 according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION FIG. 2 is a block diagram of an image processing apparatus according to a first embodiment of the present invention. The apparatus includes an 1-field delay unit 200, a first image block extraction unit 210, a second image block extraction unit 220, a sum-of-absolute difference map (SAD map) 230, a motion vector extraction unit 240, a storage unit 250, a motion filter 260, a motion vector compensation unit 270 and a frame-rate conversion (FRC) unit 280.

The 1-field delay unit 200 delays an image input in the unit of a field to allow a comparison of a current input image with a previous input image.

The first image block extraction unit 210 extracts a block from the current input image. The block has a size of CxC/2 pixels, wherein and- n As described above, a field which has been de-interlaced has half the number of pixels in the vertical direction as the frame which results from de-interlacing.

The second image block extraction unit 220 extracts a block from the previous input image. The shape and position of the block is identical to the block extracted from the current input image.

The SAD map 230 compares the blocks from the first image block extraction unit 210 and the second image block extraction unit 220 to obtain pixel differences, between the blocks, and stores the pixel difference values.

The motion vector extraction unit 240 determines a motion vector of each of the pixels of the the block of the current input image using values of the SAD map 230.

The storage unit 250 stores the motion vector value extracted by the motion vector extraction unit 240.

The motion filter 260 picks out incorrectly determined motion vector values in the storage unit 250 based on the absolute value of the motion vector. In other words, the motion filter 260 checks if a motion vector estimated from an input image, which is an interlaced image, is based on a pixel difference of line deviation between an odd field and an even field, and if so, determines the motion vector to be an incorrect motion vector.

The motion vector compensation unit 270 changes the values of the incorrcet motion vectors received from the motion filter 260 to and leaves the correct motion vectors intact.

The FRC unit 280 performs frame-rate conversion on the motion vector received from the motion vector compensation unit 270.

An operation of the apparatus of FIG. 2 will now be described.

The 1-field delay unit 200 delays an input (interlaced) image by a field so that, rectangular pixel blocks in the form of CxC/2, where, for example C= 32, extracted from the previous input image delayed by a field can be compared with the blocks from the current input image.

The SAD map 230 compares the block extracted from the previous input image delayed by a field with the corresponding pixel of the block extracted from the current input image, and obtains a pixel difference between them.

Using the pixel difference, the motion-vector extraction unit 240 determines a motion vector MV(vx, vy) for each of the pixels in an input image and stores the motion vectors in the storage unit 250.

The motion filter 260 corrects a misinterpretation of the motion vector received from the storage unit 250 by taking the distance between consecutively input images which are even and odd field images (the input order can be reversed) caused by "motion" of the input images.

The motion filter 260 sets a mask (block) of the form of Ax B, where, for example, A B 3. The motion filter 260 then calculates the absolute values of the motion vectors of pixels covered by the mask, and determines whether the number for which the absolute values of the motion vectors are equal to or less than a predetermined pixel difference is equal to or more than a predetermined number. The predetermined pixel difference may be 1, that is, a distance between even and odd field images, and in this case, the predetermined number is preferably the same as the number of pixels covered by the mask block since A B 3, the predetermined number is 9).

When the number of the absolute values of the motion vectors of pixels covered by the mask block which are equal to or less than is equal to 9, the motion vector compensation unit 270 converts the absolute values of the motion vectors to a null vector, and interprets that there was no "motion" of the corresponding block. If the number of absolute values of the motion vectors of pixels covered by the mask block which are equal to or less than is less than 9, the motion vector compensation unit 270 determines that there was a "motion" for the corresponding block and outputs the motion vectors intact.

The dimension of the mask block can be arbitrarily defined by a user.

The absolute value of the motion vector is obtained because only information about the extent of movement of the block is required, regardless of the direction of the movement. It is determined if the absolute value of a coordinate of the motion vector is less than a value of a predetermined threshold coordinate (thl, th2), and if so, a count value is increased. It is preferable that the predetermined threshold coordinate (thl, th2) is The comparison of the absolute value of the motion vector of CxC/2 block with the predetermined threshold coordinate is performed over each of the pixels in the mask block.

The motion filtering ensures that a still image is not given a false motion. An input (still) interlaced image has vy=1 as its vertical coordinate value even when there is no motion, because the even and the odd field images have a 1 pixel discrepancy between them. A field to be interpolated between the even and odd field images selects a value of or for its vy, so that a straight line deviated by 1 pixel can be observed in the output still image.

The motion vector compensation unit 270 makes vy consistently for a still image to prevent a deviation of the image when the image is output.

FIG. 3 is a flowchart of the motion filtering method according to a preferred embodiment of the present invention.

It is assumed that an estimated motion vector of pixels in a predetermined block of CxC/2 from two field images input sequentially is represented by a coordinate of MV(vx,vy).

Motion vectors are stored in a storage unit and are subjected to motion filtering one by one.

To begin the motion filtering, a mask block AxB is set to check if the estimated and stored motion vector is a real motion vector in step 300.

An absolute value of the motion vector MV(vx 1 ,vy 1 of a pixel positioned at j) in the mask block AxB is calculated in step 310. It can be expressed as in equation 1.

x i abs(vx 1 (1) yj =abs(vyj) The calculation of the absolute value of the motion vector decreases the probability of determining a motionless image to be an image in motion by only obtaining the extent of movement of the image regardless of the direction of the image.

It is determined whether the absolute value of the coordinate (vxi,vyj) exceeds a threshold (absolute) value of a predetermined coordinate (thl, th2) in step 320. Each coordinate value of (thi, h2) may be a number greater than one (the number of a single pixel), for example, 2).

If the absolute value of the motion vector coordinate is less than the threshold value (thl, th2), a count value is increased by 1 in step 330.

It is checked if the steps 320 arid 330 are performed over all the pixels covered by the mask block in step 340. If the mask block is a 3x3 block, the steps 320 and 330 should be performed over 9 pixels.

In step 350, it is determined if the count value reaches a threshold value th3, the count value being increase by 1 whenever the absolute value of the motion vector is less than the threshold value of (thl, th2). That is, to check if absolute 25. .values of the motion vectors of all the pixels in the mask block are less than or equal to 1. In step 350, if the count value has not exceeded the threshold value th3, it is determined that the absolute values of the motion vectors of some of the pixels in the mask block are greater than 2, in which case, it is determined that there has been a motion in the mask block. Thus, the motion vectors MV(vx,vy) in the predetermined block CxC/2 are passed through the motion filter with their original values in step 360.

In step 350, if the count value is equal to or greater than the threshold value th3, it is determined that the absolute values of all or most of pixels in the mask block are equal to or less than in which case, it is determined that there has been no motion in the mask block. Thus, in step 370, values of the motion vectors for the pixels covered by the mask block are forced to be 0 to prevent the deviation of a straight line in an image output, due to the value of the motion vector which appears to have even though there was no motion.

So far, a method of motion filtering has been described. After the process of motion filtering, the final motion vectors filtered in steps 360 and 370 are FRC converted and output in step 380.

The present invention as described above leads to the following effects in terms of hardware implementation: 1. Memory consumption and corresponding amounts of operation can be cut in half, because the block used for determining the motion vector is cut in half Cx C CxC/2 2. Image processing becomes simpler, which leads to a reduction in signal processing and a speed increase, because the de-interlacing and the interlacing processes are omitted.

Despite the effects in hardware implementation, the de-interlacing cannot be omitted in the conventional image processing apparatus as an error in finding the correct position of an image due to a discrepancy in the positions of pixels in odd and even field images may occur. However in the preferred embodiment of the present invention, a motion filter algorithm is provided to solve this problem. The motion filter algorithm forces the motion vector values of the motion vectors which seem to have a motion when there is actually no motion to to protect the deviation of a straight line in an image output.

FIG. 4 is an example of an original input image.

FIG. 5 shows a partial result of image processing for the original input image of FIG. 4, the image processing having de-interlacing and interlacing stages.

FIG. 6 shows a. partial result of image processing for the original input image of FIG. 4, the image processing, not having a de-interlacing stage. Omitting the de-interlacing causes an error in processing motion vector. In FIG. 6, deviations of straight lines can be observed in a marked circle.

FIG. 7 shows partial result of image processing for the original input image of FIG. 4 according to the present invention, in the present invention, de-interlacing is omitted, and instead, a motion filter is provided to compensate for errors in the motion vectors. In FIG. 7, line deviations are not observed.

According to the present invention, using a motion filter can reduce the hardware required for image processing with frame-rate conversion to lower costs and increase operational efficiency.

While this invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. An apparatus which processes images with frame-rate conversion (FRC), the apparatus including: a block extraction unit which extracts a predetermined block from an input field; a motion vector extraction unit which determines a motion vector of the predetermined block by comparing blocks of two sequentially input fields with each other; a motion filter which determines whether there has been any motion by using the motion vectors of pixels in a predetermined range of the block; a motion compensation unit which passes values of the motion vectors intact if it is determined that there has been a motion in-the predetermined range of the block, and changes the values of the motion vectors to 0, otherwise; and a frame-rate conversion unit which performs a frame-rate conversion on the motion vectors received from the motion compensation unit.

2. The apparatus of claim 1, wherein the block extracted by the block extraction unit has dimensions CxC/2, where C=2" (n

3. The apparatus of claim 1 or 2, wherein the block extraction unit further comprises: a delay unit which delays the input interlaced image by one field; a first block extraction unit which extracts a block from the delayed input interlaced image; and a second block extraction unit which extracts a block from the current input interlaced image.

4. The apparatus of claim 3, further comprising a sum-of-absolute-difference (SAD) map which compares pixels in the first block extraction unit and in the second block extraction unit with each other to obtain differences therebetween. The apparatus of claim 4, wherein the motion vector extraction unit estimates motion vectors of the input fields from the SAD map.

6. The apparatus of any one of the preceding dclaims, wheein the moton fter sets a predetermined mask block, calculates absolute values of motion vectors output from the motion filter, and determines whether the number of the motion vectors which have an absolute value which is equal to or less than a predetermined number of pixels is equal to or more than a predetermined number.

7. The apparatus of claim 6, wherein the motion compensation unit converts the absolute values of the motion vectors in the mask block to 0 if the number of the motion vectors which have an absolute value which is equal to or less than a predetermined number of pixels is equal to or more than the predetermined number.

8. The apparatus of claim 6, wherein the motion compensation unit passes the motion vectors in the mask block intact if the number of the motion vectors which have an absolute value which is equal to or less than a predetermined number of pixeis is less than a predetermined number.

9. The apparatus of claim 8, wherein the predetermined number of pixels is 1, and the predetermined number is the number of pixels covered by the mask block. A method of image processing with frame-rate conversion, the method includng: extracting a predetermined block from an input interlaced image; estimating motion vectors of the pixels in the predetermined block by comparing blocks of two sequentially input interlaced images; determining whether there has been a motion by considering values of the motion vectors of pixels in a predetermined range of the block; passing the values of the motion vectors of pixels in the predetermined range of the block intact if it is determined that there has been any motion, and forcing the values of the motion vectors of pixels in the predetermined range of the block to 0 if it is determined that there has been no motion; and performing a frame-rate conversion (FRC) on the resulting motion vectors.

11. The. method of claim 10, wherein the predetermined block is extracted to have dimensions CxC/2,where C=2" (n 12 The method ofdaim 10or11,werei the stepof acting the predetermined block comprising: delaying the input interlaced imnage by a field and extracting a first block from the delayed input interlaced image; and extracting a second block from the current input interlaced image.

13. The method of claim 12, further comprising comparing corresponding pixels in the first block and the second block to obtain differences therebetween.

14. The method of claim 13, further comprising estimating motion vectors of the input interlaced images from the differences and storing the results. The metad d arny one ofdaims 10 to 14, wher the step of detei~ if theae has been a motion comprises: setting a predetermined mask block; calculating absolute values of the motion vectors; and determining whether the number of motion vectors of pixels covered by the predetermined mask block which have an absolute value which is equal to or less than a predetermined number of pixels is equal to or more than a predetermined number.

16. The method of claim 15, wherein determining if there has been a motion comprises forcing the absolute. values of the motion vectors covered with the mask block to 0 if the number of the motion vectors which have an absolute value which is equal to or less than a predetermined number of pixels is equal to or more than a predetermined number.

17. The method of claim 15, wherein the determining if there has been a motion comprises passing the motion. vectors in the mask block intact if the number of the motion vectors which have an absolute value which is equal to or less than a predetermined number of pixels is less than a predetermined number.

18. The method of claim 16 or claim 17, wherein the predetermined numberof pixels is 1, and the predetermined number is the number of pixels covered by the mask block.

19. The method of claim 18, wherein the predetermined mask block has dimensions of 3x3 pixels, and the predetermined number of pixels is 9. An apparatus which processes images with frame-rate conversion (FRC), the apparatus including: a motion filter which determines whether a motion vector estimated from an input interlaced image is caused by a line deviation between odd and even fields; and a motion compensation unit which passes the motion vector intact if it has not been caused by the line deviation, and changes the motion vector to 0, otherwise.

21. A method of image processing with frame-rate conversion (FRC), the method i ncludin hg: determining whether a motion vector estimated from an input interlaced image is caused by a line deviation between odd and even fields; and compensating for the motion vector by passing the motion vector intact if it has not been caused by the line deviation, and changing the motion vector into a null Vector, otherwise.

22. An apparatus which processes images with frame-rate conversion substantially as herein described with reference to Figures 2 to 7 of the accompanying drawings.

23. A method of image processing with frame-rate conversion substantially as herein described with reference to Figures 2 to 7 of the accompanying drawings. DATED: 16 January, 2004 PHILLIPS ORMONDE FITZPATRICK Attorneys for: SAMSUNG ELECTRONICS CO. LTD.