WO2010005270A2 - Apparatus and method for optical flow estimation with subpixel resolution - Google Patents

Apparatus and method for optical flow estimation with subpixel resolution Download PDF

Info

Publication number
WO2010005270A2
WO2010005270A2 PCT/KR2009/003807 KR2009003807W WO2010005270A2 WO 2010005270 A2 WO2010005270 A2 WO 2010005270A2 KR 2009003807 W KR2009003807 W KR 2009003807W WO 2010005270 A2 WO2010005270 A2 WO 2010005270A2
Authority
WO
WIPO (PCT)
Prior art keywords
resolution
pixel
node
subpixel
optical flow
Prior art date
Application number
PCT/KR2009/003807
Other languages
French (fr)
Other versions
WO2010005270A3 (en
Inventor
Hong Jeong
Intae Na
Original Assignee
Postech Academy-Industry Foundation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Postech Academy-Industry Foundation filed Critical Postech Academy-Industry Foundation
Publication of WO2010005270A2 publication Critical patent/WO2010005270A2/en
Publication of WO2010005270A3 publication Critical patent/WO2010005270A3/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/269—Analysis of motion using gradient-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators

Definitions

  • the present invention relates to optical flow estimation; and, more particularly, to an apparatus and method for optical flow estimation that estimates optical flows with subpixel resolution by using optical flow vectors with pixel resolution.
  • Optical flow estimation is a computer vision technology widely used for, e.g., motion estimation and image compression.
  • optical flow estimation two time-successive images are compared to estimate displacement vectors of image pixels.
  • Optical flow estimation includes a procedure to find corresponding points between two images, which is an ill-posed problem having high computational complexity and causes high hardware complexity and long computation time.
  • Such high computational complexity makes it difficult to apply optical flow estimation to real-time applications, e.g., video monitoring systems and vehicle vision systems.
  • brightness constancy assumption means that corresponding points in a reference image and a comparison image have an identical brightness.
  • changes in pixel brightness between successive images are caused only by relative motion of a camera or an object.
  • the brightness constancy assumption contributes to reduction of complexity in optical flow estimation.
  • the local schemes find corresponding points by using information on local regions neighboring an image pixel.
  • Examples of the local schemes based on pixel masking are SAD (Sum of Absolute Difference) and NCC (Normalized Cross Correlation).
  • the local schemes may use a block matching in which images are subdivided into small blocks to find correspondence between blocks.
  • the global schemes estimate globally optimized results by using information on entire image pixels.
  • graph-cut or belief-propagation algorithms based on MRF (Markov random field) models have attracted much attention as a new technique for optical flow estimation or stereo vision in computer vision applications, because of their low error rates and robustness against data discontinuity or occlusion.
  • the brightness constancy assumption can be violated by changes in pose and shape of an object due to its motion, illumination differences due to shadows and occlusion, which results in a large amount of errors.
  • the local optical flow estimation schemes may suffer from an aperture problem and produce a large amount of errors due to ambiguous or periodic image patterns.
  • the global optical flow estimation schemes handle information on entire image pixels. Therefore, in the global optical flow estimation schemes, a small increase in the amount of input image data causes an extremely large increase in computation time, thereby constraining the number of estimatable discrete states and producing estimation results with a large amount of errors.
  • the present invention provides an apparatus and method for optical flow estimation that measures optical flows at a subpixel level to reduce quantization errors due to constraint in the number displacement states, while reducing computational load.
  • an optical flow estimation apparatus with subpixel resolution including:
  • a data cost calculation unit for computing a pixel-resolution data cost and a subpixel-resolution data cost for each node with respect to a first input image and a second input image, the first input image and the second input image being successive in time;
  • a message calculation unit for iteratively computing a pixel-resolution message and a subpixel-resolution message between the node and each neighboring node of the node based on the pixel-resolution data cost and the subpixel-resolution data cost, respectively, and propagating the pixel-resolution message and the subpixel-resolution message to the neighboring node;
  • a result output unit for estimating a pixel-resolution optical flow vector of each node based on the pixel-resolution data cost of the node and the pixel-resolution messages received from neighboring nodes of the node
  • the data cost calculation unit and the message calculation unit respectively compute the subpixel-resolution data cost and the subpixel-resolution message of each node by using the pixel-resolution optical flow vector of the node;
  • the result output unit estimates a subpixel-resolution optical flow vector of each node based on the subpixel-resolution data cost of the node and the subpixel-resolution messages received from the neighboring nodes of the node.
  • the data cost calculation unit includes a pixel data cost calculator for computing the pixel-resolution data cost of each node based on pixel brightness of the node; and a subpixel data cost calculator for computing subpixel-resolution data cost of each node based on the pixel-resolution optical flow vector of the node.
  • the pixel-resolution data cost and the subpixel-resolution data cost of each node are absolute difference between pixel brightness of a pixel in the first input image and that of a pixel in the second input image, the pixels in the first input image and in the second input image corresponding to the node.
  • the subpixel data cost calculator includes an oversampler for obtaining an over-sampled pixel brightness of the node by over-sampling pixel brightness at vicinity of a pixel in the second input image, the pixel having a position corresponding to a sum of a pixel-resolution optical flow vector of the node on a Markov random field and pixel coordinates of the node in the first input image; and an absolute difference calculator for computing absolute difference between the over-sampled pixel brightness and pixel brightness of the node in the first input image.
  • the pixel data cost calculator computes the pixel-resolution data cost by obtaining absolute difference between pixel brightness of a pixel in the first input image and that of a pixel in the second input image, the pixels in the first input image and in the second input image corresponding to the node.
  • the message calculation unit includes a pixel message calculator for computing the pixel-resolution message between the node and the neighboring node on a pixel-resolution Markov random field; and a subpixel message calculator for computing the subpixel-resolution message between the node and the neighboring node on a subpixel-resolution Markov random field.
  • the subpixel message calculator includes an adder for adding together the subpixel-resolution data cost of the node and the subpixel-resolution messages received at a previous iteration from the neighboring nodes other than the neighboring node to which the subpixel-resolution message is to be sent at a current iteration; a dynamic state allocator for allocating to addition result of the adder a message value in which positional relationship between the estimated pixel-resolution optical flow vector of the node and that of the neighboring node is reflected; and a message calculator for obtaining the subpixel-resolution message by adding allocation result of the dynamic state allocator and a smoothness cost function value between the node and the neighboring node.
  • the dynamic state allocator calculates difference between the pixel-resolution optical flow vector of the node and that of the neighboring node and dynamically allocates the message value according to a magnitude of the difference.
  • the result output unit includes a pixel-resolution output unit for estimating the pixel-resolution optical flow vector of the node by using the pixel-resolution data cost and the pixel-resolution messages; a subpixel-resolution output unit for estimating subpixel-resolution optical flow vector of the node by using the subpixel-resolution data cost and the subpixel-resolution messages; and a final result output unit for outputting a final optical flow vector of the node by using the pixel-resolution optical flow vector and the subpixel-resolution optical flow vector.
  • an optical flow estimation method with subpixel resolution including:
  • said estimating the subpixel-resolution optical flow vector includes computing a subpixel-resolution data cost of a pixel corresponding to each node; computing a subpixel-resolution message between the node and neighboring nodes of the node by using the subpixel-resolution data cost; and estimating the subpixel-resolution optical flow vector of the node by using the subpixel-resolution data cost and the subpixel-resolution messages.
  • said computing the subpixel-resolution data cost includes over-sampling pixel brightness at vicinity of a pixel in the second input image via interpolation, the pixel in the second input image having a position corresponding to a sum of the pixel-resolution optical flow vector of the node and pixel coordinates of the node in the first input image; and computing the subpixel-resolution data cost by using the over-sampled pixel brightness and pixel brightness of the pixel in the first input image.
  • the subpixel-resolution data cost is absolute difference between the over-sampled pixel brightness and the pixel brightness of the pixel in the first input image.
  • said computing each subpixel-resolution message includes adding together the subpixel-resolution data cost of the node and the subpixel-resolution messages received at a previous iteration from the neighboring nodes other than a neighboring node to which the subpixel-resolution message is to be sent at a current iteration; allocating to the addition result a message value in which positional relationship between the estimated pixel-resolution optical flow vector of the node and that of the neighboring node is reflected; and obtaining the subpixel-resolution message by adding the allocation result and a smoothness cost function value between the node and the neighboring node, the smoothness cost function value representing state continuity between the node and the neighboring node.
  • subpixel-resolution information e.g., data costs and messages
  • subpixel-resolution displacement vectors are estimated by using the data costs and messages.
  • optical flow estimation is performed based on information on entire image pixels, thereby reducing errors due to discontinuities caused by occlusion and object boundaries. Further, since the optical flow estimation uses parallel operations, e.g., message passing, computation time can be shortened while error rates are minimized.
  • Fig. 1 illustrates a block diagram of an optical flow estimation apparatus in accordance with an embodiment of the present invention
  • Fig. 2 illustrates an exemplary Markov random field
  • Fig. 3 illustrates a block diagram of the subpixel data cost calculator of Fig. 1;
  • Fig. 4 illustrates a block diagram of the subpixel message calculator of Fig. 1;
  • Figs. 5 to 7 respectively illustrate an exemplary view of dynamic state allocation
  • Figs. 9 to 10 respectively illustrate an explanatory view of subpixel-resolution displacement state estimation procedure
  • Fig. 11 illustrates a flowchart of an optical flow estimation method using the apparatus of Fig. 1.
  • Fig. 1 illustrates a block diagram of an optical flow estimation apparatus in accordance with an embodiment of the present invention.
  • the optical flow estimation apparatus includes a data cost calculation unit 100, a message calculation unit 110 and a result output unit 120.
  • the data cost calculation unit 100 computes a pixel-resolution data cost and a subpixel-resolution data cost for each node in two successive input images and , and provides thus computed data costs to the message calculation unit 110.
  • the data cost calculation unit 100 includes a pixel data cost calculator 102 for computing the pixel-resolution data cost based on a pixel value of each node; a subpixel data cost calculator 104 for computing the subpixel-resolution data cost based on a pixel-resolution optical flow vector; and a data cost memory 106 for storing therein outputs of the pixel data cost calculator 102 and the subpixel data cost calculator 104.
  • the data costs mean absolute difference between pixel brightness of a pixel in an original image and that of a pixel, corresponding to the pixel in the original image, in a comparison image.
  • the original image and the comparison image are two successive input images, and hereinafter, the pixel in the original image and the pixel in the comparison image will be referred to as "original pixel”and “comparison pixel”, respectively.
  • the pixel data cost calculator 102 receives two successive input images and to compute a pixel-resolution data cost of a node , having a coordinates vector at time frame and a pixel-resolution optical flow vector (displacement vector) at time frame , on a Markov random field of Fig. 2, as in Equation 1.
  • the pixel data cost calculator 102 calculates the pixel-resolution data cost by computing absolute difference between pixel brightness of pixels in two successive input images and .
  • Fig. 3 illustrates a block diagram of the subpixel data cost calculator 104 of Fig. 1.
  • the subpixel data cost calculator 104 includes an oversampler 300 and an absolute difference calculator 302.
  • the oversampler 300 obtains over-sampled pixel brightness by over-sampling pixel brightness at vicinity of the comparison pixel having a position corresponding to the sum of coordinates of the original pixel and a pixel-resolution optical flow vector output from the result output unit 120.
  • the absolute difference calculator 302 computes absolute difference between the over-sampled pixel brightness obtained by the oversampler 300 and the pixel brightness of the original image. That is, the subpixel data cost calculator 104 utilizes information on pixels which do not exist in the original image.
  • Equation 2 pixel brightness at vicinity of the comparison pixel having a position corresponding to the sum of the coordinates vector and the pixel-resolution optical flow vector estimated via pixel-resolution belief propagation is over-sampled via interpolation.
  • the belief propagation is to find optimal solution on an MRF via iterations and the over-sampling level depends on subpixel resolution.
  • the subpixel-resolution data cost in case of the displacement vector is calculated as Equation 2:
  • the oversampler 300 of the subpixel data cost calculator 104 receives as input the comparison image and the displacement vector , i.e., the optical flow vector of the node , and outputs the pixel brightness in the comparison image.
  • the absolute difference calculator 302 computes an absolute difference between the over-sampled pixel brightness obtained by the oversampler 300 and the pixel brightness of the original image to thereby obtain the subpixel-resolution data cost.
  • the data costs computed by the pixel data cost calculator 102 and subpixel data cost calculator 104 are stored in the data cost memory 106.
  • the message calculation unit 110 computes messages for each node on the MRF and provides the computed messages to neighboring nodes. Each message is stochastic information including displacement state of each node and information on neighboring nodes.
  • the message calculation unit 110 includes a pixel message calculator 112 for computing pixel-resolution messages between nodes on a pixel-resolution MRF and a subpixel message calculator 114 for computing subpixel-resolution messages between nodes on a subpixel-resolution MRF.
  • the pixel message calculator 112 of the message calculation unit 110 computes a pixel-resolution message delivered from a node having a state to its neighboring node having a state based on a smoothness cost function and the pixel-resolution data cost.
  • the pixel-resolution message at iteration is computed as Equations 3 and 4:
  • Equation 3 denotes neighbors of the node excluding the node , and denotes a normalization value.
  • the smoothness cost function is given by the square of state difference when nodes have discrete states, and and denote a weighting factor and an upper bound, respectively. The upper bound prevents a message value from becoming too large even when the difference between states is large.
  • Fig. 4 illustrates a block diagram of the subpixel message calculator 114 of Fig. 1.
  • the subpixel message calculator 114 includes an adder 400 for adding the subpixel-resolution data cost of each node with a given state and the subpixel-resolution messages at the previous iteration together; a dynamic state allocator 402 for allocating to the addition result of the adder 400 a message value in which positional relation between estimated pixel-resolution optical flow vector of a node and that of its neighboring node is reflected; and a message calculator 404 for obtaining each subpixel-resolution message of each node by adding a smoothness cost function value to the allocation result of the dynamic state allocator 402.
  • the subpixel message calculator 114 of the message calculating unit 110 is driven, after completion of pixel-resolution state estimation, to obtain the subpixel-resolution message by using the estimated pixel-resolution states and then performs subpixel-resolution state estimation by using the subpixel-resolution message.
  • the subpixel resolution state estimation includes following three steps.
  • a first step subpixel-resolution data cost and messages at a previous iteration of a node having a state are summed together as Equation 5:
  • Equation 6 the sum of the subpixel-resolution data cost and messages computed in the first step is adjusted by using the estimated pixel-resolution state, as in Equation 6:
  • Equation 6 difference between the discrete states and is reflected in a subpixel-resolution message.
  • Figs. 5 to 7 respectively illustrate an exemplary view of the second step, which will be hereinafter referred to as "dynamic state allocation”.
  • a final subpixel-resolution message is computed by using the value calculated in the second step and a smoothness cost function, as in Equations 7 and 8.
  • the result output unit 120 estimates pixel-resolution optical flow state and subpixel-resolution optical flow state based on the messages and data costs output from the message calculation unit 110, and outputs thus estimated optical flow states.
  • the result output unit 120 includes a pixel-resolution output unit 122 for estimating pixel-resolution optical flow vectors by using data costs and messages of nodes on a pixel-resolution MRF; a subpixel-resolution output unit 124 for estimating subpixel-resolution optical flow vectors by using data costs and messages of nodes on a subpixel-resolution MRF; and a final result output unit 126 for outputting final optical flow vectors of nodes by using the results estimated by the pixel-resolution output unit 122 and the subpixel-resolution output unit 124.
  • the final state of the node can be estimated through collecting the data cost of the node and messages from neighboring nodes, as in Equation 9.
  • the final states with subpixel resolution are estimated by using a two-stage iteration algorithm. That is, like existing belief propagation, pixel-resolution displacement states are estimated for two time-successive images as shown in Figs. 8 and 9. Then, as shown in Fig. 10, subpixel-resolution state information is obtained by using the estimated pixel-resolution state information, and finally subpixel-resolution displacement states are estimated by using the pixel-resolution state information and the subpixel-resolution state information.
  • Fig. 11 illustrates a flowchart of an optical flow estimation method using the apparatus of Fig. 1.
  • the pixel data cost calculator 102 of the data cost calculation unit 100 computes a pixel-resolution data cost of a node having a displacement state on an MRF (step S702).
  • the data cost can be computed as Equation 1.
  • the pixel message calculator 112 of the message calculation unit 110 computes pixel-resolution messages by using the pixel-resolution data cost calculated in the step S702 (step S704).
  • the pixel-resolution message to be sent from the node having the state to a neighboring node having a state is computed by using Equation 3 on the basis of the pixel-resolution data cost and a smoothness cost function given by Equation 4.
  • step S706 It is determined whether message computation for all nodes on the MRF is completed. If it is determined in the step S706 that the message computation is completed, the pixel-resolution output unit 122 estimates a pixel-resolution optical flow state of each node, i.e., a pixel-resolution optical flow vector (step S708).
  • the subpixel data cost calculator 104 of the data cost calculation unit 100 performs over-sampling pixel brighness at vicinity of a comparison pixel in the image via interpolation, the composition pixel having a position corresponding to the sum of the pixel-resolution optical flow vector estimated in the step S708 and coordinates of an original pixel in the image , and computes a subpixel-resolution data cost with a displacement state with respect to the over-sampled pixel brightness in the image and that of the original pixel in the image by using Equation 2 (step S710).
  • the adder 400 of the subpixel message calculator 114 adds the computed subpixel-resolution data cost and messages at the previous iteration together (step S712).
  • the dynamic state allocator 402 allocates to the addition result in the step S712 a message value in which positional relationship between the estimated pixel-resolution optical flow vector of the node and that of its neighboring node is reflected (step S714).
  • the message calculator 404 obtains the final subpixel-resolution message by adding the allocation result in the step S714 and a smoothness cost function value between nodes together (step S716).
  • the subpixel-resolution output unit 124 estimates a subpixel-resolution optical flow state of each node based on subpixel-resolution data cost of each node and subpixel-resolution messages of its neighboring nodes (step S720).

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

An optical flow estimation apparatus with subpixel resolution includes a data cost calculation unit for computing a pixel-resolution data cost and a subpixel-resolution data cost for each node with respect to a first and a second input image successive in time; a message calculation unit for iteratively computing a pixel-resolution message and a subpixel-resolution message between the node and each neighboring node based on the pixel-resolution data cost and the subpixel-resolution data cost, respectively; and a result output unit for estimating a pixel-resolution optical flow vector of each node based on the pixel-resolution data cost of the node and the pixel-resolution messages received from neighboring nodes of the node. The result output unit estimates a subpixel-resolution optical flow vector of each node based on the subpixel-resolution data cost and the subpixel-resolution messages which are computed by using the pixel-resolution optical flow vector of the node.

Description

APPARATUS AND METHOD FOR OPTICAL FLOW ESTIMATION WITH SUBPIXEL RESOLUTION
The present invention relates to optical flow estimation; and, more particularly, to an apparatus and method for optical flow estimation that estimates optical flows with subpixel resolution by using optical flow vectors with pixel resolution.
Optical flow estimation is a computer vision technology widely used for, e.g., motion estimation and image compression. In optical flow estimation, two time-successive images are compared to estimate displacement vectors of image pixels.
Optical flow estimation includes a procedure to find corresponding points between two images, which is an ill-posed problem having high computational complexity and causes high hardware complexity and long computation time. Such high computational complexity makes it difficult to apply optical flow estimation to real-time applications, e.g., video monitoring systems and vehicle vision systems.
In optical flow estimation, it is generally assumed that brightness of corresponding points between two successive images is constant (hereinafter, referred to as "brightness constancy assumption". That is, the brightness constancy assumption means that corresponding points in a reference image and a comparison image have an identical brightness. Under the brightness constancy assumption, changes in pixel brightness between successive images are caused only by relative motion of a camera or an object. The brightness constancy assumption contributes to reduction of complexity in optical flow estimation.
Existing approaches to corresponding point identification in optical flow estimation can be classified into local and global schemes.
The local schemes find corresponding points by using information on local regions neighboring an image pixel. Examples of the local schemes based on pixel masking are SAD (Sum of Absolute Difference) and NCC (Normalized Cross Correlation). Also, the local schemes may use a block matching in which images are subdivided into small blocks to find correspondence between blocks.
The global schemes estimate globally optimized results by using information on entire image pixels. In particular, graph-cut or belief-propagation algorithms based on MRF (Markov random field) models have attracted much attention as a new technique for optical flow estimation or stereo vision in computer vision applications, because of their low error rates and robustness against data discontinuity or occlusion.
In conventional optical flow estimation schemes, the brightness constancy assumption can be violated by changes in pose and shape of an object due to its motion, illumination differences due to shadows and occlusion, which results in a large amount of errors.
The local optical flow estimation schemes may suffer from an aperture problem and produce a large amount of errors due to ambiguous or periodic image patterns.
The global optical flow estimation schemes handle information on entire image pixels. Therefore, in the global optical flow estimation schemes, a small increase in the amount of input image data causes an extremely large increase in computation time, thereby constraining the number of estimatable discrete states and producing estimation results with a large amount of errors.
In view of the above, the present invention provides an apparatus and method for optical flow estimation that measures optical flows at a subpixel level to reduce quantization errors due to constraint in the number displacement states, while reducing computational load.
In accordance with an aspect of the present invention, there is provided an optical flow estimation apparatus with subpixel resolution, including:
a data cost calculation unit for computing a pixel-resolution data cost and a subpixel-resolution data cost for each node with respect to a first input image and a second input image, the first input image and the second input image being successive in time;
a message calculation unit for iteratively computing a pixel-resolution message and a subpixel-resolution message between the node and each neighboring node of the node based on the pixel-resolution data cost and the subpixel-resolution data cost, respectively, and propagating the pixel-resolution message and the subpixel-resolution message to the neighboring node; and
a result output unit for estimating a pixel-resolution optical flow vector of each node based on the pixel-resolution data cost of the node and the pixel-resolution messages received from neighboring nodes of the node,
wherein the data cost calculation unit and the message calculation unit respectively compute the subpixel-resolution data cost and the subpixel-resolution message of each node by using the pixel-resolution optical flow vector of the node; and
wherein the result output unit estimates a subpixel-resolution optical flow vector of each node based on the subpixel-resolution data cost of the node and the subpixel-resolution messages received from the neighboring nodes of the node.
Preferably, the data cost calculation unit includes a pixel data cost calculator for computing the pixel-resolution data cost of each node based on pixel brightness of the node; and a subpixel data cost calculator for computing subpixel-resolution data cost of each node based on the pixel-resolution optical flow vector of the node.
Preferably, the pixel-resolution data cost and the subpixel-resolution data cost of each node are absolute difference between pixel brightness of a pixel in the first input image and that of a pixel in the second input image, the pixels in the first input image and in the second input image corresponding to the node.
Preferably, the subpixel data cost calculator includes an oversampler for obtaining an over-sampled pixel brightness of the node by over-sampling pixel brightness at vicinity of a pixel in the second input image, the pixel having a position corresponding to a sum of a pixel-resolution optical flow vector of the node on a Markov random field and pixel coordinates of the node in the first input image; and an absolute difference calculator for computing absolute difference between the over-sampled pixel brightness and pixel brightness of the node in the first input image.
Preferably, the pixel data cost calculator computes the pixel-resolution data cost by obtaining absolute difference between pixel brightness of a pixel in the first input image and that of a pixel in the second input image, the pixels in the first input image and in the second input image corresponding to the node.
Preferably, the message calculation unit includes a pixel message calculator for computing the pixel-resolution message between the node and the neighboring node on a pixel-resolution Markov random field; and a subpixel message calculator for computing the subpixel-resolution message between the node and the neighboring node on a subpixel-resolution Markov random field.
Preferably, the subpixel message calculator includes an adder for adding together the subpixel-resolution data cost of the node and the subpixel-resolution messages received at a previous iteration from the neighboring nodes other than the neighboring node to which the subpixel-resolution message is to be sent at a current iteration; a dynamic state allocator for allocating to addition result of the adder a message value in which positional relationship between the estimated pixel-resolution optical flow vector of the node and that of the neighboring node is reflected; and a message calculator for obtaining the subpixel-resolution message by adding allocation result of the dynamic state allocator and a smoothness cost function value between the node and the neighboring node.
Preferably, the dynamic state allocator calculates difference between the pixel-resolution optical flow vector of the node and that of the neighboring node and dynamically allocates the message value according to a magnitude of the difference.
Preferably, the result output unit includes a pixel-resolution output unit for estimating the pixel-resolution optical flow vector of the node by using the pixel-resolution data cost and the pixel-resolution messages; a subpixel-resolution output unit for estimating subpixel-resolution optical flow vector of the node by using the subpixel-resolution data cost and the subpixel-resolution messages; and a final result output unit for outputting a final optical flow vector of the node by using the pixel-resolution optical flow vector and the subpixel-resolution optical flow vector.
In accordance with another aspect of the present invention, there is provided an optical flow estimation method with subpixel resolution, including:
receiving a first input image and a second input image, the first input image and the second input image being successive in time;
estimating a pixel-resolution optical flow vector of each node with respect to the first input image via iterative belief propagation on a pixel-resolution Markov random field;
estimating a subpixel-resolution optical flow vector of the node on a subpixel-resolution Markov random field by using the pixel-resolution optical flow vector; and
computing a final optical flow vector of the node having subpixel resolution by using the pixel-resolution optical flow vector and the subpixel-resolution optical flow vector.
Preferably, said estimating the subpixel-resolution optical flow vector includes computing a subpixel-resolution data cost of a pixel corresponding to each node; computing a subpixel-resolution message between the node and neighboring nodes of the node by using the subpixel-resolution data cost; and estimating the subpixel-resolution optical flow vector of the node by using the subpixel-resolution data cost and the subpixel-resolution messages.
Preferably, said computing the subpixel-resolution data cost includes over-sampling pixel brightness at vicinity of a pixel in the second input image via interpolation, the pixel in the second input image having a position corresponding to a sum of the pixel-resolution optical flow vector of the node and pixel coordinates of the node in the first input image; and computing the subpixel-resolution data cost by using the over-sampled pixel brightness and pixel brightness of the pixel in the first input image.
Preferably, the subpixel-resolution data cost is absolute difference between the over-sampled pixel brightness and the pixel brightness of the pixel in the first input image.
Preferably, said computing each subpixel-resolution message includes adding together the subpixel-resolution data cost of the node and the subpixel-resolution messages received at a previous iteration from the neighboring nodes other than a neighboring node to which the subpixel-resolution message is to be sent at a current iteration; allocating to the addition result a message value in which positional relationship between the estimated pixel-resolution optical flow vector of the node and that of the neighboring node is reflected; and obtaining the subpixel-resolution message by adding the allocation result and a smoothness cost function value between the node and the neighboring node, the smoothness cost function value representing state continuity between the node and the neighboring node.
According to the present invention, subpixel-resolution information, e.g., data costs and messages, is obtained by using estimated pixel-resolution displacement vectors, and subpixel-resolution displacement vectors are estimated by using the data costs and messages. Thus, the amount of computation can be effectively reduced.
Further, the optical flow estimation is performed based on information on entire image pixels, thereby reducing errors due to discontinuities caused by occlusion and object boundaries. Further, since the optical flow estimation uses parallel operations, e.g., message passing, computation time can be shortened while error rates are minimized.
The above features of the present invention will become apparent from the following description of embodiments, given in conjunction with the accompanying drawings, in which:
Fig. 1 illustrates a block diagram of an optical flow estimation apparatus in accordance with an embodiment of the present invention;
Fig. 2 illustrates an exemplary Markov random field;
Fig. 3 illustrates a block diagram of the subpixel data cost calculator of Fig. 1;
Fig. 4 illustrates a block diagram of the subpixel message calculator of Fig. 1;
Figs. 5 to 7 respectively illustrate an exemplary view of dynamic state allocation;
Figs. 9 to 10 respectively illustrate an explanatory view of subpixel-resolution displacement state estimation procedure; and
Fig. 11 illustrates a flowchart of an optical flow estimation method using the apparatus of Fig. 1.
Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which form a part hereof.
Fig. 1 illustrates a block diagram of an optical flow estimation apparatus in accordance with an embodiment of the present invention. The optical flow estimation apparatus includes a data cost calculation unit 100, a message calculation unit 110 and a result output unit 120.
[Corrected under Rule 26 07.10.2009]
The data cost calculation unit 100 computes a pixel-resolution data cost and a subpixel-resolution data cost for each node in two successive input images
Figure PCTKR2009003807-appb-I000001
and
Figure WO-DOC-FIGURE-2
, and provides thus computed data costs to the message calculation unit 110. The data cost calculation unit 100 includes a pixel data cost calculator 102 for computing the pixel-resolution data cost based on a pixel value of each node; a subpixel data cost calculator 104 for computing the subpixel-resolution data cost based on a pixel-resolution optical flow vector; and a data cost memory 106 for storing therein outputs of the pixel data cost calculator 102 and the subpixel data cost calculator 104. Here, the data costs mean absolute difference between pixel brightness of a pixel in an original image and that of a pixel, corresponding to the pixel in the original image, in a comparison image. The original image and the comparison image are two successive input images, and hereinafter, the pixel in the original image and the pixel in the comparison image will be referred to as "original pixel"and "comparison pixel", respectively.
[Corrected under Rule 26 07.10.2009]
The pixel data cost calculator 102 receives two successive input images
Figure WO-DOC-FIGURE-3
and
Figure WO-DOC-FIGURE-4
to compute a pixel-resolution data cost
Figure WO-DOC-FIGURE-5
of a node
Figure WO-DOC-FIGURE-6
, having a coordinates vector
Figure WO-DOC-FIGURE-7
at time frame
Figure WO-DOC-FIGURE-8
and a pixel-resolution optical flow vector (displacement vector)
Figure WO-DOC-FIGURE-9
at time frame
Figure WO-DOC-FIGURE-10
, on a Markov random field of Fig. 2, as in Equation 1.
[Corrected under Rule 26 07.10.2009]
MathFigure 1
Figure WO-DOC-MATHS-1
[Corrected under Rule 26 07.10.2009]
As shown in Equation 1, the pixel data cost calculator 102 calculates the pixel-resolution data cost
Figure WO-DOC-FIGURE-11
by computing absolute difference between pixel brightness of pixels in two successive input images
Figure WO-DOC-FIGURE-12
and
Figure WO-DOC-FIGURE-13
.
Fig. 3 illustrates a block diagram of the subpixel data cost calculator 104 of Fig. 1.
[Corrected under Rule 26 07.10.2009]
As shown in Fig. 3, the subpixel data cost calculator 104 includes an oversampler 300 and an absolute difference calculator 302. The oversampler 300 obtains over-sampled pixel brightness by over-sampling pixel brightness at vicinity of the comparison pixel having a position corresponding to the sum of coordinates of the original pixel and a pixel-resolution optical flow vector output from the result output unit 120. The absolute difference calculator 302 computes absolute difference between the over-sampled pixel brightness obtained by the oversampler 300 and the pixel brightness of the original image. That is, the subpixel data cost calculator 104 utilizes information on pixels which do not exist in the original image. In order to calculate the subpixel-resolution data cost of the node
Figure WO-DOC-FIGURE-14
, pixel brightness at vicinity of the comparison pixel having a position corresponding to the sum of the coordinates vector
Figure WO-DOC-FIGURE-15
and the pixel-resolution optical flow vector
Figure WO-DOC-FIGURE-16
estimated via pixel-resolution belief propagation is over-sampled via interpolation. Here, the belief propagation is to find optimal solution on an MRF via iterations and the over-sampling level depends on subpixel resolution. The subpixel-resolution data cost
Figure WO-DOC-FIGURE-17
in case of the displacement vector
Figure WO-DOC-FIGURE-18
is calculated as Equation 2:
[Corrected under Rule 26 07.10.2009]
MathFigure 2
Figure WO-DOC-MATHS-2
[Corrected under Rule 26 07.10.2009]
wherein
Figure WO-DOC-FIGURE-19
denotes over-sampled pixel brightness in the comparison image and
Figure WO-DOC-FIGURE-20
denotes pixel brightness of an original pixel. Further,
Figure WO-DOC-FIGURE-21
and
Figure WO-DOC-FIGURE-22
denote a weighting factor and an upper bound, respectively.
[Corrected under Rule 26 07.10.2009]
Referring to Equation 2, the oversampler 300 of the subpixel data cost calculator 104 receives as input the comparison image
Figure WO-DOC-FIGURE-23
and the displacement vector
Figure WO-DOC-FIGURE-24
, i.e., the optical flow vector of the node
Figure WO-DOC-FIGURE-25
, and outputs the pixel brightness
Figure WO-DOC-FIGURE-26
in the comparison image.
The absolute difference calculator 302 computes an absolute difference between the over-sampled pixel brightness obtained by the oversampler 300 and the pixel brightness of the original image to thereby obtain the subpixel-resolution data cost.
Referring back to Fig. 1, the data costs computed by the pixel data cost calculator 102 and subpixel data cost calculator 104 are stored in the data cost memory 106.
After completing the above-described data cost computation, the message calculation unit 110 computes messages for each node on the MRF and provides the computed messages to neighboring nodes. Each message is stochastic information including displacement state of each node and information on neighboring nodes. The message calculation unit 110 includes a pixel message calculator 112 for computing pixel-resolution messages between nodes on a pixel-resolution MRF and a subpixel message calculator 114 for computing subpixel-resolution messages between nodes on a subpixel-resolution MRF.
[Corrected under Rule 26 07.10.2009]
The pixel message calculator 112 of the message calculation unit 110 computes a pixel-resolution message delivered from a node
Figure WO-DOC-FIGURE-27
having a state
Figure WO-DOC-FIGURE-28
to its neighboring node
Figure WO-DOC-FIGURE-29
having a state
Figure WO-DOC-FIGURE-30
based on a smoothness cost function and the pixel-resolution data cost. In the pixel-resolution belief propagation, the pixel-resolution message
Figure WO-DOC-FIGURE-31
at
Figure WO-DOC-FIGURE-32
iteration is computed as Equations 3 and 4:
[Corrected under Rule 26 07.10.2009]
MathFigure 3
Figure WO-DOC-MATHS-3
[Corrected under Rule 26 07.10.2009]
MathFigure 4
Figure WO-DOC-MATHS-4
[Corrected under Rule 26 07.10.2009]
wherein
Figure WO-DOC-FIGURE-33
is a smoothness cost function in the pixel-resolution belief propagation. In Equation 3,
Figure WO-DOC-FIGURE-34
denotes neighbors of the node
Figure WO-DOC-FIGURE-35
excluding the node
Figure WO-DOC-FIGURE-36
, and
Figure WO-DOC-FIGURE-37
denotes a normalization value. In Equation 4, the smoothness cost function
Figure WO-DOC-FIGURE-38
is given by the square of state difference when nodes have discrete states, and
Figure WO-DOC-FIGURE-39
and
Figure WO-DOC-FIGURE-40
denote a weighting factor and an upper bound, respectively. The upper bound
Figure WO-DOC-FIGURE-41
prevents a message value from becoming too large even when the difference between states is large.
Fig. 4 illustrates a block diagram of the subpixel message calculator 114 of Fig. 1.
As shown in Fig. 4, the subpixel message calculator 114 includes an adder 400 for adding the subpixel-resolution data cost of each node with a given state and the subpixel-resolution messages at the previous iteration together; a dynamic state allocator 402 for allocating to the addition result of the adder 400 a message value in which positional relation between estimated pixel-resolution optical flow vector of a node and that of its neighboring node is reflected; and a message calculator 404 for obtaining each subpixel-resolution message of each node by adding a smoothness cost function value to the allocation result of the dynamic state allocator 402.
The subpixel message calculator 114 of the message calculating unit 110 is driven, after completion of pixel-resolution state estimation, to obtain the subpixel-resolution message by using the estimated pixel-resolution states and then performs subpixel-resolution state estimation by using the subpixel-resolution message.
[Corrected under Rule 26 07.10.2009]
The subpixel resolution state estimation includes following three steps. In a first step, subpixel-resolution data cost and messages at a previous iteration of a node having a state
Figure WO-DOC-FIGURE-42
are summed together as Equation 5:
[Corrected under Rule 26 07.10.2009]
MathFigure 5
Figure WO-DOC-MATHS-5
[Corrected under Rule 26 07.10.2009]
wherein
Figure WO-DOC-FIGURE-43
denotes a subpixel-resolution message sent to the node
Figure WO-DOC-FIGURE-44
from a neighboring node
Figure WO-DOC-FIGURE-45
at the previous iteration
Figure WO-DOC-FIGURE-46
, and
Figure WO-DOC-FIGURE-47
denotes a normalization value to prevent data overflow due to iterative operations.
In a second step, the sum of the subpixel-resolution data cost and messages computed in the first step is adjusted by using the estimated pixel-resolution state, as in Equation 6:
[Corrected under Rule 26 07.10.2009]
MathFigure 6
Figure WO-DOC-MATHS-6
[Corrected under Rule 26 07.10.2009]
wherein
Figure WO-DOC-FIGURE-48
and
Figure WO-DOC-FIGURE-49
denote discrete states of nodes
Figure WO-DOC-FIGURE-50
and
Figure WO-DOC-FIGURE-51
, respectively. In Equation 6, difference between the discrete states
Figure WO-DOC-FIGURE-52
and
Figure WO-DOC-FIGURE-53
is reflected in a subpixel-resolution message.
Figs. 5 to 7 respectively illustrate an exemplary view of the second step, which will be hereinafter referred to as "dynamic state allocation".
In a third step, a final subpixel-resolution message is computed by using the value calculated in the second step and a smoothness cost function, as in Equations 7 and 8.
[Corrected under Rule 26 07.10.2009]
MathFigure 7
Figure WO-DOC-MATHS-7
[Corrected under Rule 26 07.10.2009]
MathFigure 8
Figure WO-DOC-MATHS-8
After completion of iterative message transmission and update, the result output unit 120 estimates pixel-resolution optical flow state and subpixel-resolution optical flow state based on the messages and data costs output from the message calculation unit 110, and outputs thus estimated optical flow states. The result output unit 120 includes a pixel-resolution output unit 122 for estimating pixel-resolution optical flow vectors by using data costs and messages of nodes on a pixel-resolution MRF; a subpixel-resolution output unit 124 for estimating subpixel-resolution optical flow vectors by using data costs and messages of nodes on a subpixel-resolution MRF; and a final result output unit 126 for outputting final optical flow vectors of nodes by using the results estimated by the pixel-resolution output unit 122 and the subpixel-resolution output unit 124.
[Corrected under Rule 26 07.10.2009]
In other words, the final state
Figure WO-DOC-FIGURE-54
of the node
Figure WO-DOC-FIGURE-55
can be estimated through collecting the data cost of the node
Figure WO-DOC-FIGURE-56
and messages from neighboring nodes, as in Equation 9.
[Corrected under Rule 26 07.10.2009]
MathFigure 9
Figure WO-DOC-MATHS-9
As described above, in the present invention, the final states with subpixel resolution are estimated by using a two-stage iteration algorithm. That is, like existing belief propagation, pixel-resolution displacement states are estimated for two time-successive images as shown in Figs. 8 and 9. Then, as shown in Fig. 10, subpixel-resolution state information is obtained by using the estimated pixel-resolution state information, and finally subpixel-resolution displacement states are estimated by using the pixel-resolution state information and the subpixel-resolution state information.
Below, an optical flow estimation method using the above-described optical flow estimation apparatus will be described with reference to Fig. 11.
Fig. 11 illustrates a flowchart of an optical flow estimation method using the apparatus of Fig. 1.
[Corrected under Rule 26 07.10.2009]
First, two successive images
Figure WO-DOC-FIGURE-57
and
Figure WO-DOC-FIGURE-58
are received (step S700). The pixel data cost calculator 102 of the data cost calculation unit 100 computes a pixel-resolution data cost
Figure WO-DOC-FIGURE-59
of a node
Figure WO-DOC-FIGURE-60
having a displacement state
Figure WO-DOC-FIGURE-61
on an MRF (step S702). The data cost
Figure WO-DOC-FIGURE-62
can be computed as Equation 1.
[Corrected under Rule 26 07.10.2009]
The pixel message calculator 112 of the message calculation unit 110 computes pixel-resolution messages by using the pixel-resolution data cost
Figure WO-DOC-FIGURE-63
calculated in the step S702 (step S704). To be specific, the pixel-resolution message to be sent from the node
Figure WO-DOC-FIGURE-64
having the state
Figure WO-DOC-FIGURE-65
to a neighboring node
Figure WO-DOC-FIGURE-66
having a state
Figure WO-DOC-FIGURE-67
is computed by using Equation 3 on the basis of the pixel-resolution data cost
Figure WO-DOC-FIGURE-68
and a smoothness cost function given by Equation 4.
It is determined whether message computation for all nodes on the MRF is completed (step S706). If it is determined in the step S706 that the message computation is completed, the pixel-resolution output unit 122 estimates a pixel-resolution optical flow state of each node, i.e., a pixel-resolution optical flow vector (step S708).
[Corrected under Rule 26 07.10.2009]
When the pixel-resolution optical flow state estimation is completed, the subpixel data cost calculator 104 of the data cost calculation unit 100 performs over-sampling pixel brighness at vicinity of a comparison pixel in the image
Figure WO-DOC-FIGURE-69
via interpolation, the composition pixel having a position corresponding to the sum of the pixel-resolution optical flow vector estimated in the step S708 and coordinates of an original pixel in the image
Figure WO-DOC-FIGURE-70
, and computes a subpixel-resolution data cost with a displacement state
Figure WO-DOC-FIGURE-71
with respect to the over-sampled pixel brightness in the image
Figure WO-DOC-FIGURE-72
and that of the original pixel in the image
Figure WO-DOC-FIGURE-73
by using Equation 2 (step S710).
After computation of the subpixel-resolution data cost, the adder 400 of the subpixel message calculator 114 adds the computed subpixel-resolution data cost and messages at the previous iteration together (step S712).
The dynamic state allocator 402 allocates to the addition result in the step S712 a message value in which positional relationship between the estimated pixel-resolution optical flow vector of the node and that of its neighboring node is reflected (step S714).
The message calculator 404 obtains the final subpixel-resolution message by adding the allocation result in the step S714 and a smoothness cost function value between nodes together (step S716).
If it is determined that final subpixel-resolution messages for all nodes are computed (step S718), the subpixel-resolution output unit 124 estimates a subpixel-resolution optical flow state of each node based on subpixel-resolution data cost of each node and subpixel-resolution messages of its neighboring nodes (step S720).
While the invention has been shown and described with respect to the embodiments, it will be understood by those skilled in the art that various changes and modification may be made without departing from the scope of the invention as defined in the following claims.

Claims (14)

  1. An optical flow estimation apparatus with subpixel resolution, comprising:
    a data cost calculation unit for computing a pixel-resolution data cost and a subpixel-resolution data cost for each node with respect to a first input image and a second input image, the first input image and the second input image being successive in time;
    a message calculation unit for iteratively computing a pixel-resolution message and a subpixel-resolution message between the node and each neighboring node of the node based on the pixel-resolution data cost and the subpixel-resolution data cost, respectively, and propagating the pixel-resolution message and the subpixel-resolution message to the neighboring node; and
    a result output unit for estimating a pixel-resolution optical flow vector of each node based on the pixel-resolution data cost of the node and the pixel-resolution messages received from neighboring nodes of the node,
    wherein the data cost calculation unit and the message calculation unit respectively compute the subpixel-resolution data cost and the subpixel-resolution message of each node by using the pixel-resolution optical flow vector of the node; and
    wherein the result output unit estimates a subpixel-resolution optical flow vector of each node based on the subpixel-resolution data cost of the node and the subpixel-resolution messages received from the neighboring nodes of the node.
  2. The optical flow estimation apparatus of claim 1, wherein the data cost calculation unit includes:
    a pixel data cost calculator for computing the pixel-resolution data cost of each node based on pixel brightness of the node; and
    a subpixel data cost calculator for computing subpixel-resolution data cost of each node based on the pixel-resolution optical flow vector of the node.
  3. The optical flow estimation apparatus of claim 1, wherein the pixel-resolution data cost and the subpixel-resolution data cost of each node are absolute difference between pixel brightness of a pixel in the first input image and that of a pixel in the second input image, the pixels in the first input image and in the second input image corresponding to the node.
  4. The optical flow estimation apparatus of claim 2, wherein the subpixel data cost calculator includes:
    an oversampler for obtaining an over-sampled pixel brightness of the node by over-sampling pixel brightness at vicinity of a pixel in the second input image, the pixel having a position corresponding to a sum of a pixel-resolution optical flow vector of the node on a Markov random field and pixel coordinates of the node in the first input image; and
    an absolute difference calculator for computing absolute difference between the over-sampled pixel brightness and pixel brightness of the node in the first input image.
  5. The optical flow estimation apparatus of claim 2, wherein the pixel data cost calculator computes the pixel-resolution data cost by obtaining absolute difference between pixel brightness of a pixel in the first input image and that of a pixel in the second input image, the pixels in the first input image and in the second input image corresponding to the node.
  6. The optical flow estimation apparatus of claim 1, wherein the message calculation unit includes:
    a pixel message calculator for computing the pixel-resolution message between the node and the neighboring node on a pixel-resolution Markov random field; and
    a subpixel message calculator for computing the subpixel-resolution message between the node and the neighboring node on a subpixel-resolution Markov random field.
  7. The optical flow estimation apparatus of claim 6, wherein the subpixel message calculator includes:
    an adder for adding together the subpixel-resolution data cost of the node and the subpixel-resolution messages received at a previous iteration from the neighboring nodes other than the neighboring node to which the subpixel-resolution message is to be sent at a current iteration;
    a dynamic state allocator for allocating to addition result of the adder a message value in which positional relationship between the estimated pixel-resolution optical flow vector of the node and that of the neighboring node is reflected; and
    a message calculator for obtaining the subpixel-resolution message by adding allocation result of the dynamic state allocator and a smoothness cost function value between the node and the neighboring node.
  8. The optical flow estimation apparatus of claim 7, wherein the dynamic state allocator calculates difference between the pixel-resolution optical flow vector of the node and that of the neighboring node and dynamically allocates the message value according to a magnitude of the difference.
  9. The optical flow estimation apparatus of claim 1, wherein the result output unit includes:
    a pixel-resolution output unit for estimating the pixel-resolution optical flow vector of the node by using the pixel-resolution data cost and the pixel-resolution messages;
    a subpixel-resolution output unit for estimating subpixel-resolution optical flow vector of the node by using the subpixel-resolution data cost and the subpixel-resolution messages; and
    a final result output unit for outputting a final optical flow vector of the node by using the pixel-resolution optical flow vector and the subpixel-resolution optical flow vector.
  10. An optical flow estimation method with subpixel resolution, comprising:
    receiving a first input image and a second input image, the first input image and the second input image being successive in time;
    estimating a pixel-resolution optical flow vector of each node with respect to the first input image via iterative belief propagation on a pixel-resolution Markov random field;
    estimating a subpixel-resolution optical flow vector of the node on a subpixel-resolution Markov random field by using the pixel-resolution optical flow vector; and
    computing a final optical flow vector of the node having subpixel resolution by using the pixel-resolution optical flow vector and the subpixel-resolution optical flow vector.
  11. The optical flow estimation method of claim 10, wherein said estimating the subpixel-resolution optical flow vector includes:
    computing a subpixel-resolution data cost of a pixel corresponding to each node;
    computing a subpixel-resolution message between the node and neighboring nodes of the node by using the subpixel-resolution data cost; and
    estimating the subpixel-resolution optical flow vector of the node by using the subpixel-resolution data cost and the subpixel-resolution messages.
  12. The optical flow estimation method of claim 11, wherein said computing the subpixel-resolution data cost includes:
    over-sampling pixel brightness at vicinity of a pixel in the second input image via interpolation, the pixel in the second input image having a position corresponding to a sum of the pixel-resolution optical flow vector of the node and pixel coordinates of the node in the first input image; and
    computing the subpixel-resolution data cost by using the over-sampled pixel brightness and pixel brightness of the pixel in the first input image.
  13. The optical flow estimation method of claim 12, wherein the subpixel-resolution data cost is absolute difference between the over-sampled pixel brightness and the pixel brightness of the pixel in the first input image.
  14. The optical flow estimation method of claim 11, wherein said computing each subpixel-resolution message includes:
    adding together the subpixel-resolution data cost of the node and the subpixel-resolution messages received at a previous iteration from the neighboring nodes other than a neighboring node to which the subpixel-resolution message is to be sent at a current iteration;
    allocating to the addition result a message value in which positional relationship between the estimated pixel-resolution optical flow vector of the node and that of the neighboring node is reflected; and
    obtaining the subpixel-resolution message by adding the allocation result and a smoothness cost function value between the node and the neighboring node, the smoothness cost function value representing state continuity between the node and the neighboring node.
PCT/KR2009/003807 2008-07-11 2009-07-10 Apparatus and method for optical flow estimation with subpixel resolution WO2010005270A2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020080067677A KR100955044B1 (en) 2008-07-11 2008-07-11 Apparatus and method for estimating a subpixel resolution
KR10-2008-0067677 2008-07-11

Publications (2)

Publication Number Publication Date
WO2010005270A2 true WO2010005270A2 (en) 2010-01-14
WO2010005270A3 WO2010005270A3 (en) 2012-10-04

Family

ID=41507602

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2009/003807 WO2010005270A2 (en) 2008-07-11 2009-07-10 Apparatus and method for optical flow estimation with subpixel resolution

Country Status (2)

Country Link
KR (1) KR100955044B1 (en)
WO (1) WO2010005270A2 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102265315B1 (en) * 2015-04-17 2021-06-15 삼성전자주식회사 Method and apparatus for estimating image optical flow

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6507661B1 (en) * 1999-04-20 2003-01-14 Nec Research Institute, Inc. Method for estimating optical flow
WO2003036334A2 (en) * 2001-10-25 2003-05-01 Koninklijke Philips Electronics N.V. Method and apparatus for motion estimation
US6628715B1 (en) * 1999-01-15 2003-09-30 Digital Video Express, L.P. Method and apparatus for estimating optical flow
WO2007083204A2 (en) * 2006-01-20 2007-07-26 Nokia Corporation Method for optical flow field estimation using adaptive filtering

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080018788A1 (en) * 2006-07-20 2008-01-24 Samsung Electronics Co., Ltd. Methods and systems of deinterlacing using super resolution technology

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6628715B1 (en) * 1999-01-15 2003-09-30 Digital Video Express, L.P. Method and apparatus for estimating optical flow
US6507661B1 (en) * 1999-04-20 2003-01-14 Nec Research Institute, Inc. Method for estimating optical flow
WO2003036334A2 (en) * 2001-10-25 2003-05-01 Koninklijke Philips Electronics N.V. Method and apparatus for motion estimation
WO2007083204A2 (en) * 2006-01-20 2007-07-26 Nokia Corporation Method for optical flow field estimation using adaptive filtering

Also Published As

Publication number Publication date
KR20100007170A (en) 2010-01-22
KR100955044B1 (en) 2010-04-28
WO2010005270A3 (en) 2012-10-04

Similar Documents

Publication Publication Date Title
WO2018225967A1 (en) Device having cameras with different focal lengths and method of implementing cameras
WO2013125768A1 (en) Apparatus and method for automatically detecting object and depth information of image photographed by image pickup device having multiple color filter aperture
EP3639243A1 (en) Camera pose determination and tracking
JP6622503B2 (en) Camera model parameter estimation apparatus and program thereof
WO2016053067A1 (en) 3-dimensional model generation using edges
WO2017160026A2 (en) Location estimation method and apparatus using access point in wireless communication system
EP2321793A2 (en) Apparatus and method for frame interpolation based on accurate motion estimation
WO2015163542A1 (en) Method for effectively estmating three-dimensional location by using trilateration in wireless network, and recording medium in which program is recorded for carrying out same
WO2013168998A1 (en) Apparatus and method for processing 3d information
WO2014185710A1 (en) Method for correcting 3d image in tiled display and apparatus therefor
WO2020230977A1 (en) Metacognition-based high-speed environmental search method and device
WO2017099510A1 (en) Method for segmenting static scene on basis of image statistical information and method therefor
WO2012115435A2 (en) Method and apparatus for encoding and decoding multi view video
WO2015160052A1 (en) Method for correcting image from wide-angle lens and device therefor
WO2022037389A1 (en) Reference plane-based high-precision method and system for estimating multi-degree-of-freedom attitude of object
WO2010005270A2 (en) Apparatus and method for optical flow estimation with subpixel resolution
WO2014010820A1 (en) Method and apparatus for estimating image motion using disparity information of a multi-view image
WO2024056030A1 (en) Image depth estimation method and apparatus, electronic device and storage medium
WO2020130211A1 (en) Joint model registration device and method
WO2009096721A2 (en) Method and apparatus for encoding and decoding video signal using motion compensation based on affine transformation
WO2022098050A1 (en) A method and an electronic device for video processing
WO2022260392A1 (en) Method and system for generating image processing artificial neural network model operating in terminal
RU2407058C2 (en) Calculating transformation parametres for image processing
Rajagopalan et al. Motion optimization of ordered blocks for overlapped block motion compensation
CN202931463U (en) Characteristic block based video image stabilization device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 09794676

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 09794676

Country of ref document: EP

Kind code of ref document: A2