KR101841122B1 - Block-based optical flow estimation of motion pictures - Google Patents

Abstract

According to another aspect of the present invention, there is provided an optical flow estimation method comprising: estimating optical flows of reference blocks divided according to a size of a storage area in a reference image; And estimating an optical flow of the reference image based on the optical flows of the estimated reference blocks.

Description

[0001] Block-Based Optical Flow Estimation [

This specification relates to optical flow estimation of moving images. More particularly, this specification relates to estimating block-based optical flow of moving images.

Optical flow refers to a velocity field that indicates how the displacement of two points, selected from one image and another from successive images, has changed. Such an optical flow can express a change in position of an object or the like included in successive images constituting a moving picture. The optical flow can be used for compression / decompression of moving pictures, for example, on the basis of a characteristic representing a positional change of an object or the like.

Methods for estimating optical flows from successive images are presented. In particular, Thomas Brox et al. Proposed a total variation optical flow estimation algorithm: see Brooks et al., 'High Precision Optical Flow Estimation Based on Warping Theory' (T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, "High Accuracy Optical Flow Estimation Based on Warping," ECCV, European Conference on Computer Vision, 2004).

It takes much calculation time to apply the estimation method based on the optimization technique to the entire image to be estimated by the image processing apparatus that estimates the optical flow. In addition, hardware implementations that estimate optical flows of large images (e.g., HD-level images) using such an overall variation-based algorithm may impose constraints on the size of the cache and channel bandwidth .

This specification is intended to provide an estimation method that can reduce the size constraints of the cache and bandwidth that the image processing apparatuses can employ to estimate the optical flow.

In one embodiment, a method is provided for an image processing apparatus having a storage area to estimate an optical flow. The method includes estimating an optical flow of reference blocks divided according to a size of the storage area in a reference image; And estimating an optical flow of the reference image based on the optical flows of the estimated reference blocks.

The one or more embodiments may include one or more of the following features.

The step of estimating the optical flow of the reference blocks may include estimating each of the reference blocks based on a target block in the target image corresponding to each of the reference blocks. In addition, the step of estimating the optical flow of the reference blocks may be to estimate the optical flow according to the specific order of the reference blocks. In addition, the reference block may be a size that can be stored in the storage area and is separated from the reference image. The optical flows of the reference blocks may be stored in the storage area in the reference block, and may be estimated based on the stored image information and the image information in the target block. The optical flow of each of the reference blocks may be estimated using a solution of a linear system formed based on the image information in the reference block and the image information in the target block.

The optical flow of each of the reference blocks may be one in which the image information in an area including the reference block is sampled in the reference image and is stored in the storage area and estimated based on the stored image information and the image information in the target block .

The reference block may be one in which the other reference block and the image are partially overlapped.

On the other hand, an image processing apparatus for estimating an optical flow of an image in another embodiment is disclosed. The image processing apparatus includes a storage unit for storing image information of reference blocks in a reference image; And storing the image information of the reference blocks separated from the reference image in a size storable in the storage unit in the storage unit and estimating an optical flow of the reference blocks based on the stored image information and the image information in the target block, And a controller for estimating an optical flow of the reference image based on the estimated optical flows of the reference blocks.

The above or other embodiments may include one or more of the following features.

The control unit may sample the image information in the region including the reference block in the reference image and store the sampled image information in the storage unit. In addition, the reference block may be divided so that a part of the image overlaps with the other reference block.

According to embodiments of the block-based optical flow estimation method disclosed in this specification, the estimation operation can be performed by hardware implemented to include a storage space such as a cache having a fixed size by performing block-based estimation operation.

Also, the image processing apparatus according to the embodiment disclosed herein can estimate the optical flow using the hardware having the predetermined size of the cache, and reduce the size of the channel bandwidth for the estimation.

In addition, the image processing apparatus according to the embodiments disclosed herein can increase the inter-block diffusion of the optical flow estimated by applying the shape of the reference block or the estimation method differently according to the size of the image.

Also, the image processing apparatus according to the embodiments disclosed herein can reduce the error in the block boundary by ensuring the flatness of the optical flow while performing the estimation calculation of the block-based optical flow.

1 illustrates a method for constructing a block used for block-based optical flow estimation disclosed herein.
FIG. 2 illustrates a process of estimating an optical flow according to a total variation method.
3 is a method of constructing a reference block applicable to the optical flow estimation disclosed herein.
Fig. 4 shows a calculation method of optical flow estimation according to a block Gauss-Cydel method.
Figure 5 shows a calculation method for estimating an optical flow according to an extended block range method.
Figure 6 illustrates a method for estimating the optical flow of a reference block in accordance with an extended block range method.
FIG. 7 shows an operation method of optical flow estimation according to the divided block range method.
8 shows a method for estimating the optical flow of a reference block according to the divided block range method.

The technique disclosed in this specification is applied to a block-based optical flow estimation method for a moving picture and an apparatus using the same. However, the technology disclosed in this specification is not limited thereto, but may be applied to all optical flow estimation methods and apparatuses, image processing methods and apparatuses to which the technical idea of the above-described technology can be applied, and image handling apparatuses and methods using optical flow estimation .

It is noted that the technical terms used herein are used only to describe specific embodiments and are not intended to limit the scope of the technology disclosed herein. Also, the technical terms used herein should be interpreted as being generally understood by those skilled in the art to which the presently disclosed subject matter belongs, unless the context clearly dictates otherwise in this specification, Should not be construed in a broader sense, or interpreted in an oversimplified sense. In addition, when a technical term used in this specification is an erroneous technical term that does not accurately express the concept of the technology disclosed in this specification, it should be understood that technical terms which can be understood by a person skilled in the art are replaced. Also, the general terms used in the present specification should be interpreted in accordance with the predefined or prior context, and should not be construed as being excessively reduced in meaning.

Also, the singular forms "as used herein include plural referents unless the context clearly dictates otherwise. In this specification, the terms "comprising ", or" comprising "and the like should not be construed as necessarily including the various elements or steps described in the specification, Or may be further comprised of additional components or steps.

Further, the suffix "module" and "part" for components used in the present specification are given or mixed in consideration of ease of specification, and do not have their own meaning or role.

Furthermore, terms including ordinals such as first, second, etc. used in this specification can be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals denote like or similar elements, and redundant description thereof will be omitted.

Further, in the description of the technology disclosed in this specification, a detailed description of related arts will be omitted if it is determined that the gist of the technology disclosed in this specification may be obscured. It is to be noted that the attached drawings are only for the purpose of easily understanding the concept of the technology disclosed in the present specification, and should not be construed as limiting the spirit of the technology by the attached drawings.

Optical Fluctuation Estimation of Total Variation

Optical Flow refers to a velocity field that indicates how the displacement of two pixels, selected from one image and successive other images, has changed. Such an optical flow can express a change in position of an object or the like contained in successive images constituting a moving picture.

An image processing apparatus for estimating an optical flow performs a process of estimating an optical flow from one image and another image successive thereto. Hereinafter, among the images used by the image processing apparatus to estimate the optical flow, an image serving as a current reference is referred to as a reference image, and an optical flow of the reference image is estimated as an image continuous with the reference image The target image is referred to as a target image.

Although the present specification describes an optical flow estimation method as an example of an optical flow estimation algorithm of the entire variation method disclosed in 'Brooks et al.', The optical flow estimation method according to the present invention is not limited to using only the above algorithm.

The method for estimating the optical flow of the entire variation method between the reference image and the target image is performed based on the following several assumptions.

First, the gray value of the moved pixel is kept constant according to the movement direction (first assumption, Gray Value Constancy Assumption). The first hypothesis corresponds to the definition of the optical flow for pixels between consecutive images.

Secondly, the gradient of the brightness value of the pixel moved according to the movement direction is kept constant (second assumption, gradient constraint assumpption). The second hypothesis is to be able to determine the optical flow of pixels by compensating for the first hypothesis in the case that the brightness for the pixels changes due to the influence of illumination or the like. The inclination of the brightness value of the pixel shifted according to the movement direction according to the second assumption does not vary.

Third, the optical flow vector field should be smooth (third assumption, Smoothness Assumption). The third hypothesis relates to how the interrelationship of the gradient of the optical flow of a pixel and its surrounding pixels is consistent in the spatial axis direction. An optimized optical flow has discontinuous values near the object boundary in the image. Accordingly, the boundary of the optical flow vector field is formed by using the energy of the total variation type having the edge-preserving characteristic.

는 수학식 1과 같이 표현될 수 있다.The energy values derived from these three assumptions are as follows. First, the data energy derived from the first and second hypotheses,

Can be expressed as Equation (1).

는 시공간 축 상에서의 좌표,

는 벡터의 형태로 표시된 옵티컬 플로, I는 밝기 값,

는 전체 도메인(domain)을 나타내고

,

이다. 다음으로, 상기 제 3 가정으로부터 유도되는 평탄성 에너지(Smoothness Energy) 는 수학식 2와 같이 표현될 수 있다.here,

Coordinates on a space-time axis,

Is an optical flow in the form of a vector, I is a brightness value,

Represents the entire domain

,

to be. Next, the smoothness energy derived from the third assumption can be expressed by Equation (2).

는 다음의 수학식 3과 같이 표현될 수 있다.This total energy, which is the sum of data energy and planar energy

Can be expressed by the following equation (3).

는 보정 계수

를 이용하여 상기 데이터 에너지

및 상기 평탄성 에너지

를 종합하여 표현한 것이다. Here, the total energy

Correction coefficient

Lt; RTI ID = 0.0 >

And the flat energy

.

를 최소화하는 u, v는 전술된 수학식 1 내지 수학식 3에서의 u, v에 대한 Euler-Lagrange 방정식을 만족시킨다. 따라서, 상기 영상 처리 장치는 상기 수학식 1 내지 수학식 3에서 u, v에 대한 Euler-Lagrange 방정식을 만족시키는 해를 구함으로써 총 에너지를 최소화하는 u, v의 값을 추정할 수 있다. A method for estimating an optical flow in which the image processing apparatus minimizes total energy is as follows. According to variational calculus, total energy

U and v satisfy the Euler-Lagrange equations for u and v in the above-described equations (1) to (3). Therefore, the image processing apparatus can estimate the value of u, v that minimizes the total energy by obtaining a solution satisfying the Euler-Lagrange equation for u, v in Equations (1) to (3).

On the other hand, the problem of minimizing the total energy is transformed into a problem of solving the partial differential equation. At this time, since the modified partial differential equation is a nonlinear equation, the image processing apparatus linearizes the equations through Taylor expansion and fixed point iteration. The image processing apparatus can use an iteration-based linear solver such as SOR or Gauss-Seidel to solve the linearized equation. The linearized equation can be expressed by the following equations (4) and (5).

,

,

,

,

,

,

,

이며, k와 l은 각각 outer 와 inner fixed point iteration의 index이다.here,

,

,

,

,

,

,

,

, Where k and l are the indices of outer and inner fixed point iteration, respectively.

When the image processing apparatus is implemented to include hardware for estimating the optical flow, there are implementation limitations. That is, the image processing apparatus must solve equations (4) and (5) to estimate the optical flow for each pixel in the reference image. For example, if the reference image is a 1920x1080 image of HD resolution, the image processing apparatus performs an operation using a matrix of about 4M x 4M to estimate the optical flow of the reference image. At this time, in order to solve the above equation, the hardware implemented in the image processing apparatus needs to include a cache for storing values for processing a 4M x 4M matrix. However, it is not realistic to implement such an arrangement that a cache of such a size is included in an integrated circuit of the image processing apparatus in order to perform an operation of estimating an optical flow by the entire variation method for the entire reference image. In addition, even if the integrated circuit of the image processing apparatus is implemented to estimate the optical flow through a partial operation, the implementation cost is increased due to a large channel bandwidth.

Block-based optical flow estimation

A block-based optical estimation method disclosed herein is disclosed. The block-based optical estimation may divide the reference image into one or more reference blocks and include the reference blocks in the divided reference blocks, as opposed to estimating the optical flow with a total variation method for the pixels included in the entire reference image. Optical flow is estimated by the total variation method on a block-by-block basis as in the estimation method for the entire image with respect to the pixels that are estimated by the block-by-block basis, and optical flows of the entire reference image are estimated by using the optical flows estimated for each block.

To this end, a method is disclosed herein in which a reference image is divided into one or more blocks to estimate an optical flow. Since the optical flow estimation method disclosed in this specification is divided and estimated for each block without performing an operation for estimating the entire image at a time, the integrated circuit implemented with the estimation method can be implemented to include a relatively small size cache have.

In addition, the hardware in which the block-based optical flow estimation disclosed herein is implemented may include a cache of a predetermined size used for the operation for the block-based estimation.

Hereinafter, as methods applicable to the block-based optical flow estimation disclosed herein, there are a method of constructing a reference block to be an estimation object, a method of constructing a target block used for estimation, a method of securing inter-block flatness, A method of increasing liver diffusion and the like are disclosed.

Configuration of blocks used for optical flow estimation

Hereinafter, a reference image and a target image block applicable to block-based optical flow estimation according to the embodiment disclosed herein will be described with reference to FIG. 1 illustrates a method for constructing a block used for block-based optical flow estimation disclosed herein.

The image processing apparatus for performing the block-based optical flow estimation divides the reference image 10 to be estimated into a plurality of blocks having a specific size (hereinafter referred to as reference blocks). The size of the reference block may be determined in consideration of the size of the cache of the hardware in which the estimation method is implemented.

In addition, the image processing apparatus divides the target image 20, which is continuous with the reference image 10, into a plurality of blocks (hereinafter, referred to as target blocks) so as to correspond to the reference blocks. The size of the target block may be different from the size of the reference block. The size of the target block may be determined to be larger than the size of the reference block in consideration of the motion of the pixels so that there are corresponding pixels for all the pixels existing in the divided reference blocks.

Referring to FIG. 1 (a), for example, the reference image 10 is divided into eight reference blocks 11 to 18, and the target image 20 is divided into a plurality of reference blocks And blocks 21 through 28. [ Referring to FIG. 1B, for example, one of the target blocks 23 has a search range 23 '' as well as a position corresponding to one of the reference blocks 13 Is included in the region 23 '. The target block divided in the area 23 'including the search range can be determined according to various methods.

Block-based optical flow estimation method

Hereinafter, a process of using the total variation method for block-based optical flow estimation will be described with reference to FIG. FIG. 2 illustrates a process of estimating an optical flow according to a total variation method. As described above, the process of estimating the optical flow of each of the one or more reference blocks included in the reference image is similar to the process of estimating the optical flow with the entire variation method for the entire reference image. The block-based optical flow estimation includes a process of estimating the entire optical flow using the optical flows estimated for the respective reference blocks.

Referring to FIG. 2, the image processor performs a warping operation based on the reference image, the gradient of the reference image, the target image, the gradient of the target image, and the optical flow calculated in the previous step. In the notation shown in FIG. 2, g _{x, 1} represents the derivative of the reference block in the x direction, g _{x, 2} represents the derivative of the target block in the x direction.

 The image processor constructs a linearized equation that can solve the problem of minimizing the total energy, as described above, based on the result of the warping operation and the gradient of the optical flow. The image processor then solves the equation using an iterative based linear equation solution and estimates the optical flow based on the solution.

Meanwhile, an image processing apparatus according to the embodiments disclosed herein can estimate optical flows in a block-based manner on a reference image at each pyramidal level. That is, the image processing apparatus can perform block-based optical flow estimation at each pyramid level by applying the optical flow estimation of the entire variation method described with reference to FIG. 2 to each pyramid level of the reference image.

In this case, since the hardware including the cache of a predetermined size used for the block-based optical flow estimation is implemented in the image processing apparatus as described above, the image processing apparatus performs all the pyramidal level ) To perform only operations on a linear system of one size.

Ensuring block-to-block smoothness

Hereinafter, a method for ensuring block-to-block flatness applicable to the block-based optical flow estimation disclosed herein will be described. 3 is a method of constructing a reference block applicable to the optical flow estimation disclosed herein. Referring to Fig. 3, a method of constructing a reference block for ensuring the flatness of the optical flow is disclosed.

If the reference blocks used for estimating the optical flow based on the block are simply distinguished from the reference image, the flatness of the optical flow at the boundary of the reference blocks is not guaranteed. Therefore, in the block-based optical flow estimation method disclosed in the present specification, the reference block is configured such that the image processing apparatus further includes one or more pixels than the block to which the optical flow is to be estimated.

Referring to FIG. 3, when the reference image 10 is divided into blocks, if the block 13 'to be estimated for the optical flow has a size of (mxm), it is referred to for actual optical flow estimation The reference block 13 has a size of (m + 2 x m + 2). The image processing apparatus determines the reference block 13 to be larger than the block 13 'to be subjected to the optical flow estimation, The intensity and the inclination of the optical flow can be accurately calculated. Also, the image processing apparatus can apply inter-block flatness by referring to the optical flow outside the block (13) '.

Increase diffusion between blocks

Hereinafter, a method for enhancing inter-block diffusion applicable to the block-based optical flow estimation disclosed herein will be described.

Estimation of block-based optical flows requires a similar amount of computation with less spread between blocks compared with the full image-based estimation method, unless a method of increasing inter-block spreading is used. Accordingly, the block-based optical flow estimation disclosed in the present specification can be configured to include a process for increasing inter-block spreading considering characteristics of a reference image.

In some embodiments, the image processing apparatus for estimating a block-based optical flow may apply inter-block spreading by applying a modified fixed point iteration.

The modified fixed-point iterative method can be used, for example, by using a block Gauss-Seidel (BGS) method in which a Gauss-Cilel method is modified from a repetition-based linear equation solution used to solve a linear equation Lt; / RTI > The block Gauss-cidel method may also be referred to as an alternate block coverage (ABC) method.

Fig. 4 shows a calculation method of optical flow estimation according to a block Gauss-Cydel method. Referring to FIG. 4, the Gauss-cadel method is a method in which all reference blocks are processed by performing a total of k outer fixed point iterations for one reference block as shown in FIG. 4 (a) , The block Gaussian-cryptanalysis method performs only one iteration for each reference block for all reference blocks as shown in FIG. 4 (b) Repeated times. The use of the block Gauss cipher method increases diffusion between blocks while keeping the total amount of computation for optical flow estimation similar.

The order of the reference blocks to which the iterative operation is applied in the block Gaussian cipher method may be sequential or non-sequential.

Further, in some embodiments, the image processing apparatus for estimating a block-based optical flow can increase inter-block spreading by applying an extended reference block. The method of using the extended reference block may be referred to as an Extended Block Coverage (EBC) method.

In the extended block range method, the size of the reference block for estimating the optical flow is smaller than the size of the range to be calculated. That is, a range to be an operation target of the extended block range method includes a reference block.

FIG. 5 shows an operation method for optical flow estimation according to an extended block range method. Referring to FIG. 5, the image processing apparatus performs an estimation operation using an extended block 17, for example, to estimate an optical flow of a reference block 17 'in FIG. 5 (a) . In this case, a block-based estimation process is performed using the target block 27 shown in FIG. 5 (b). The method of estimating the optical flow according to the extended block range method using the reference blocks thus determined is as shown in FIG. 6, which is the same as the operation of the total variation method as described with reference to FIG. 2 Is omitted. The image processing apparatus can estimate the optical flow of the whole image in which the inter-block spread is increased by performing the extended block range method sequentially or non-sequentially on the reference blocks separated from the reference image 10.

In the extended block range method in which the object of calculation is widened in this manner, the diffusion strength between the reference blocks becomes strong as the diffusion intensity inside the block to be computed, but the computation object becomes wider, have. Therefore, the image processing apparatus keeps the matrix size of the linear system the same by sampling the gradient of the image and the optical flow with respect to the widened calculation object. A sampling rate used for the extended block range method is determined such that an image corresponding to the size of the extended block range is less than or equal to a block range used for the optical flow estimation through the sampling.

Further, in some embodiments, the image processing apparatus for estimating a block-based optical flow can increase inter-block spreading by performing an operation so that blocks overlap. The method of using such overlapped blocks may be referred to as the Separated Block Coverage (SBC) method.

FIG. 7 shows an operation method of optical flow estimation according to the divided block range method. 7, the image processing apparatus performs an operation for estimating a block-based optical flow for the reference block 12 of FIG. 7A, for example, And then performs an estimation operation on the next reference block 13 that partially overlaps the reference block 13. In this case, for each block-based optical flow estimation operation, block-by-block optical flow estimation is performed using the target blocks 22 and 23 corresponding to the reference blocks shown in FIG. 7B. Each of the target blocks may be determined to have a different range from the blocks 22 'and 23' corresponding to the reference block in which the operation is performed. The image processing apparatus estimates an optical flow sequentially or non-sequentially with respect to the reference blocks separated from the reference image 10. In this case, a part of the reference blocks are overlapped with each other, Optical flow increases the inter-block spread.

When the method using the overlapping block is used, the image processing apparatus can keep the matrix size of the linear system the same by sampling the gradient of the image and the optical flow with respect to the wider calculation object. In this case, the ratio used for the sampling is determined in consideration of the size of the block to be computed and the size of the block used for the optical flow estimation through the sampling.

In addition, a method of estimating each optical flow for the determined reference blocks according to the method of using the overlapping block is as shown in FIG. 8, And is omitted.

Estimation method based on pyramid level

A method of determining an optical flow estimation method according to a pyramid level is disclosed in an image processing apparatus according to embodiments disclosed herein.

The image processing apparatus according to embodiments disclosed herein may use an image pyramid of the reference image to estimate an optical flow of the reference image on a block basis.

The image processing using the image pyramid is for dividing and processing the image to be processed according to the resolution. The image processing method using the image pyramid is used to reduce image processing time. For example, when the amount of data of a reference image to be subjected to optical flow estimation is large, the reference image is used as a raw image, the size of the image is reduced while maintaining its characteristics, and an optical flow estimation operation is performed on the reduced image By doing so, the time required for processing large amounts of data can be saved.

The image pyramid refers to a set of images having different sizes according to each pyramidal level. L0, which represents the lowest level, such as the pyramid shape, represents the largest image size, and the image size decreases as the level increases.

In the case where the hardware implementing the above-described block-based optical flow estimation method has a cache for processing a reference block of a specific size, the image processing apparatus configured to include the hardware generates the pyramid level Desc / Clms Page number 5 > determining the optical flow estimation method in the first embodiment.

If the resolution of the reference image to be estimated for the block-based optical flow is, for example, 1920x1080 corresponding to Full HD, the image pyramid of the reference image has L0 of 1920x1080 according to each pyramid level, L1 is 960x540 , L2 is 480x270, L3 is 240x135, L4 is 120x68, L5 is 60x34, L6 is 30x17, and L7 is a set of images having a resolution of 15x9.

For example, if the hardware has a cache for processing a 16x16 reference block, the reference image of L7 can be processed using the hardware, but the reference image of L7 can be used as the cache size of the hardware And the size of the reference block is smaller than the size of the reference block that can be processed. Therefore, the feature of the block-based estimation method is not disclosed.

Also, the image processing apparatus may apply the block Gauss-cadel method for images corresponding to L5 to L6 among the image pyramids. The image processing apparatus including hardware having a cache capable of processing a 16x16 size can process an optical flow by dividing a reference image corresponding to L5 to L6 into several reference blocks.

In addition, the image processing apparatus may apply the extended block range method or the overlapped block range method to an image corresponding to L3 or less among the image pyramids. The image processing apparatus can perform block-based optical flow estimation and increase inter-block spreading by performing sampling in consideration of the cache size in a process of estimating an optical flow for an image corresponding to L3 or less.

However, the above-mentioned image processing apparatus determines the block-based optical estimation method considering the size of the image according to the pyramid level of the reference image, but it is not limited thereto.

A method for performing block-based optical flow estimation in accordance with the embodiments disclosed herein may be implemented in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof have. The image processing apparatus may be configured to include a control unit for performing block-based optical flow estimation and a cache for storing reference block information for solving the linear system in the estimation process.

According to a hardware implementation, the controller for performing the block-based optical flow estimating methods described above may be implemented by application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs) , FPGAs (field programmable gate arrays, processors, controllers, micro-controllers, microprocessors, and other units for performing other functions) . The cache may be implemented in hardware of the type included in the controller, but may be implemented in a memory separate from the controller.

According to a software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. Software code can be implemented in a software application written in a suitable programming language. The software code may be stored in a memory of the image processing apparatus and executed by the control unit.

The scope of the present invention is not limited to the embodiments disclosed in this specification, and the present invention can be modified, changed, or improved in various forms within the scope of the present invention and the claims.

Claims (11)

A method of estimating an optical flow of an image processing apparatus having a storage area,
Estimating an optical flow of reference blocks divided according to the size of the storage area in the reference image; And
Estimating an optical flow of the reference image based on optical flow of each of the estimated reference blocks,
Wherein estimating the optical flow of the reference blocks comprises:
Determining extension blocks including each reference block and having a larger extent than each reference block;
Further comprising the steps of: performing, for each of the extension blocks, sampling based on the image gradient and the pre-calculated optical flow, and estimating an optical flow of the sampled images to estimate an optical flow of the reference blocks,
The sampling rate of the sampling may be,
Wherein the size of the sampled images is determined to be less than or equal to a block range used for optical flow estimation of the reference blocks. 2. The method of claim 1, wherein estimating the optical flow of the reference blocks comprises:
Wherein each of the reference blocks corresponds to each of the reference blocks and is estimated based on a target block in the target image to be estimated for the optical flow. 3. The method of claim 2, wherein estimating the optical flow of the reference blocks comprises:
And estimating an optical flow according to a specific order of each of the reference blocks. delete delete delete delete delete 1. An image processing apparatus for estimating an optical flow of an image,
A storage unit for storing image information of reference blocks in a reference image; And
Storing the image information of the reference blocks separated from the reference image in a size that can be stored in the storage unit in the storage unit and estimating an optical flow of the reference blocks based on the stored image information and the image information in the target block And a controller for estimating an optical flow of the reference image based on the estimated optical flows of the reference blocks,
Wherein,
Determining extension blocks each containing a reference block and having a greater range than each reference block, performing sampling based on the image gradient and the pre-calculated optical flow for each of the determined extension blocks, Estimating an optical flow of the reference blocks by estimating an optical flow,
The sampling rate of the sampling may be,
Wherein the size of the sampled images is determined to be less than or equal to a block range used for optical flow estimation of the reference blocks. 10. The apparatus of claim 9, wherein the control unit
Wherein the image processing unit samples image information in an area including the reference block in the reference image and stores the sampled image information in the storage unit. 10. The method of claim 9,
Wherein the reference block is divided so that a part of the image overlaps with the other reference block.

Images