JP4409526B2 - Optical flow processor - Google Patents

Description

  The present invention relates to an optical flow process for obtaining a motion vector for each pixel in a moving image recognition process.

  Image recognition processing technology that can be expected to be applied to applications with high social contributions, such as vehicle safety systems, intelligent robot systems, intelligent surveillance systems, etc. is a very important research subject. Therefore, real-time image recognition processing with high accuracy and high resolution is required. Optical flow processing for obtaining a high-definition motion vector for each pixel is an elemental technology that can be widely applied to various moving image recognition processing fields.

The optical flow is a motion vector that represents how much each pixel in the image has moved. By using the optical flow, detailed recognition is possible, and a high-definition moving image recognition system can be realized. However, the optical flow derivation process that calculates the motion vector of each pixel requires real-time processing for high-resolution images because it requires several tens of GOPS to calculate the optical flow in real time and for all pixels. The system has not been put into practical use. In software processing, flow derivation processing for only feature points and high-speed flow derivation processing at the expense of accuracy are currently performed. In addition, real-time processing hardware is limited to low resolution or low accuracy using FPGA, ASIC, or the like (see Non-Patent Documents 1 and 2).
MV Correia and AC Campilho, ICPR, vol. 4, pp. 247-250, 2002 J. Diaz, E. Ros, S. Mota, F. Pelay and EM Ortigosa, Early Cognitive Vision Workshop, Talk 21, 2004 BKP Horn and BG Schunck, "Determining optical Flow", AI vol.17, pp.185-204, 1981

  An object of the present invention is to provide an optical flow processor that processes a high calculation load in real time.

（ここに、ｕ，ｖは、ｘ方向とｙ方向のオプティカルフローであり、ｎ＋１は繰返し回数であり、ave_uとave_vはｘ方向とｙ方向の平均オプティカルフローであり、E_x、E_y、E_tはｘ、ｙ、ｔ方向での輝度勾配であり、αは重み係数である）で表される演算を反復して画素ごとに動きベクトルを求める。このオプティカルフロープロセッサは、演算に必要なデータを入力し、解像度の異なる複数階層レベルでの階層画像作成、輝度勾配算出、オプティカルフロー導出と内挿、補間画像作成のための反復演算を行う共通演算器を備え、この共通演算器は、入力データについて加算を行う加算器と、入力データおよび／または加算器からのデータを演算する第１，第２，第３および第４の処理演算器と、第１から第４の処理演算器の演算結果を加算する累算器からなる。第１から第４の処理演算器の各々は、入力データを平均する平均化ブロックと、入力データの積和演算をする第１積和ブロックと、入力データの積和演算をする第２積和ブロックと、第１積和ブロックおよび／または第２積和ブロックからの入力データの除算、加算および減算を行う除算・加減算ブロックと、除算・加減算ブロックからの入力データおよび内部メモリからの入力データに対して積和演算をする第３積和ブロックと、第３積和ブロックからの入力データおよび内部メモリからの入力データに対して積和演算をする第４積和ブロックとからなる。このオプティカルフロープロセッサは、逐次的に実行される演算の種類に応じて、入力データを変更し、平均化ブロックと、第１積和ブロックと、第２積和ブロックと、除算・加減算ブロックと、第３積和ブロックと、第４積和ブロックを選択的に用い、データパスを変更して、複数階層レベルで階層画像を作成して、輝度勾配を算出して輝度勾配メモリに記憶し、最上位階層レベルについてオプティカルフローを導出してオプティカルフローメモリに記憶し、より解像度の大きい階層化画像について、上位階層レベルのオプティカルフローから下位階層レベルのオプティカルフローに変換して前記オプティカルフローメモリに記憶する内挿処理と、得られたオプティカルフローを用いて動き補償をする補間画像作成とを順次実行して、最終的なオプティカルフローを出力する。 The optical flow processor according to the present invention uses the following equation for an input moving image:

(Where u and v are optical flows in the x and y directions, n + 1 is the number of repetitions, ave_u and ave_v are average optical flows in the x and y directions, and E_x, E_y, and E_t are x , Y, and t, and α is a weighting coefficient) to obtain a motion vector for each pixel. This optical flow processor is a common operation that inputs the data necessary for the operation and performs iterative operations for creating hierarchical images, luminance gradient calculations, optical flow derivation and interpolation, and interpolated images at multiple levels with different resolutions. The common computing unit includes: an adder that performs addition on input data; first, second, third, and fourth processing computing units that operate on input data and / or data from the adder; It consists of an accumulator that adds the operation results of the first to fourth processing operation units. Each of the first to fourth processing arithmetic units includes an averaging block that averages the input data, a first product-sum block that performs a product-sum operation on the input data, and a second product-sum that performs a product-sum operation on the input data. A block, a division / addition / subtraction block that performs division, addition, and subtraction of input data from the first product-sum block and / or second product-sum block, input data from the division / addition / subtraction block, and input data from the internal memory A third product-sum block that performs a product-sum operation on the input and a fourth product-sum block that performs a product-sum operation on the input data from the third product-sum block and the input data from the internal memory. The optical flow processor changes the input data according to the type of operation executed sequentially , an averaging block, a first product-sum block, a second product-sum block, a division / addition / subtraction block, The third product-sum block and the fourth product-sum block are selectively used, the data path is changed , hierarchical images are created at a plurality of hierarchical levels, the luminance gradient is calculated and stored in the luminance gradient memory, and the The optical flow is derived for the upper hierarchical level and stored in the optical flow memory, and the hierarchical image having a higher resolution is converted from the optical flow of the upper hierarchical level to the optical flow of the lower hierarchical level and stored in the optical flow memory. and interpolation processing, and sequentially executes a creation interpolated image motion compensation using the obtained optical flow, final To output an optical flow.

  The optical flow processor preferably further includes an input buffer for receiving data from the external and output buffers, an internal memory for storing at least a part of the operation result, and an output for storing and outputting the operation result of the common arithmetic unit. The common arithmetic unit inputs data necessary for the operation from the input buffer and the internal memory. The internal memory includes, for example, a luminance gradient memory that stores a luminance gradient and an optical flow memory that stores an optical flow.

In the optical flow processor, preferably, each of the first, second, third, and fourth processing arithmetic units is configured to calculate luminance gradients E_x, E_y, E_t for each of the four parallel pixels when the optical flow is derived. The optical flows befor_u and befor_v of the previous frame are input, the averaging block calculates average optical flows ave_u and ave_v, the first product-sum block calculates out_bel = E_x ² + E_y ² + α ² , The second product-sum block calculates out_ber = E_x * ave_u + E_y * ave_v + E_t, the division / addition / subtraction block calculates div_add = out_ber / out_bel, and the third product-sum block calculates u = ave_u-E_x * div_add And v = ave_u-E_y * div_add, and the fourth product-sum block calculates tmp = (befor_u-u) ² + (befor_v-v) ² .

  In the optical flow processor, it is preferable that the first, second, third, and fourth processing calculators calculate luminance values E_mx, LPF filter coefficients lpf_x of the previous frame, the current frame, and the next frame when calculating the luminance gradient. Enter the diff filter coefficient diff_x (where x = l, m, n). Here, in the first processing arithmetic unit, the first product-sum block calculates out_bel = E_ml * lpf0 + E_mm * lpf1 + E_mn * lpf0, and the second product-sum block outputs out_ber = E_ml * diff0 + E_mm * diff1 + E_mn * diff0 is calculated, and the third product-sum block calculates Tmp_lpf = (E_l + E_n) * lpf0 + E_m * lpf1 in the y direction. In the second processing arithmetic unit, the first product-sum block calculates out_bel = E_ml * lpf0 + E_mm * lpf1 + E_mn * lpf0, and the second product-sum block outputs out_ber = E_ml * diff0 + E_mm * diff1 + E_mn * diff0 is calculated, the division / addition / subtraction block outputs out_bel and out_ber, and the third product-sum block inputs out_bel and out_ber from the first, second, and third processing arithmetic units. Tmp_lpf = (E_l + E_n) * lpf0 + E_m * lpf1 is calculated in the x direction, and the fourth product-sum block calculates diff_0 = diff0 * (E_i-E_k) and diff_1 = diff0 * (E_j-E_g) Calculate. In the third processing arithmetic unit, the first product-sum block calculates out_bel = E_ml * lpf0 + E_mm * lpf1 + E_mn * lpf0, and the second product-sum block outputs out_ber = E_ml * diff0 + E_mm * diff1 + E_mn * diff0 is calculated, the division / addition / subtraction block outputs out_bel and out_ber, and the third product-sum block inputs out_bel and out_ber from the first, second, and third processing arithmetic units. Tmp_lpf = (E_l + E_n) * lpf0 + E_m * lpf1 is calculated in the x direction, and the fourth product-sum block inputs the LPF coefficient and tmp = (E_i + E_k) * lpf0 + (E_j + E_g) * Lpf1 is calculated. In the fourth processing arithmetic unit, the third product-sum block calculates Tmp_lpf = (E_l + E_n) * lpf0 + E_m * lpf1 in the y direction.

  In the optical flow processor, for example, a plurality of the common arithmetic units are arranged in parallel. Then, wiring is performed so that the input data is shared between the two adjacent processing operation units between the adjacent common operation units, and the other common operation unit is connected to the common operation unit at one end of the plurality of common operation units. The previous optical flow data is transferred from the arithmetic unit.

  High-speed and high-precision calculations are realized in a small-scale circuit by optimizing the optical flow calculation and sharing the arithmetic unit in the sequential calculation including luminance gradient calculation and optical flow derivation.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.
FIG. 1 is a graph showing the performance of an optical flow processor in terms of the number of processed pixels per unit time and average angle error (MAE). MAE corresponds to the reciprocal of accuracy. The conventional optical flow processors described in Non-Patent Documents 1 and 2 have the performances indicated as Diaz and Miguel, respectively. The present invention realizes a real-time optical flow processor with higher accuracy and faster processing. In the figure, the number of processed pixels 3M and 9M [pels / sec] per unit time corresponds to a 30 fr / s CIF image and a 30 fr / s VGA image, respectively. The optical flow processor of the present invention can realize the high-precision and high-speed processing shown in the figure by using an architecture described later, and can process a high calculation load in real time. This VLSI architecture is scalable with accuracy and resolution. By arranging four common arithmetic units using this scalability, real-time processing of 30 fr / s VGA images is possible. Further, the average angle error (MAE) of the conventional processor is 10 degrees or more, while the processor of the present invention can achieve less than MAE 10 degrees.

（ここに、ｕ，ｖは、ｘ方向とｙ方向のオプティカルフローであり、ｎ＋１は繰返し回数であり、 ave_uとave_vはｘ方向とｙ方向の平均オプティカルフローであり、E_x、E_y、E_tはｘ、ｙ、ｔ方向での輝度勾配であり、αは重み係数である。） Various algorithms for calculating an optical flow have been proposed so far. The optical flow is a motion vector that represents how much each pixel in the image has moved. Here, a highly accurate Horn & Schunck algorithm (Non-Patent Document 3) is used. This algorithm is a gradient method based on a global assumption that adjacent optical flows change smoothly, and an optical flow is obtained by repeatedly solving the following update equation.

(Where u and v are the optical flows in the x and y directions, n + 1 is the number of repetitions, ave_u and ave_v are the average optical flows in the x and y directions, and E_x, E_y, and E_t are x ), Y, and t directions, and α is a weighting factor.)

  For VLSI implementation, the HOE (Hierarchical Optical-Flow Estimation) algorithm using the highly accurate Horn & Schunck algorithm is adopted. The feature of this algorithm is that it employs a multidimensional gradient filter for calculating the luminance gradient and performs a hierarchization process. The Horn & Schunck algorithm requires 7 pixels before and after LPF processing. Here, a multi-dimensional gradient filter using 3 frames can be used to detect a highly accurate flow without increasing the frame delay. In addition, it is possible to cope with detection of a large movement by the hierarchization processing. Hierarchization processing is an effective technique for derivation of a large flow with high accuracy, and is a technique for deriving a flow in order from an image with a smaller resolution by creating a hierarchical image by a Gaussian filter and 2: 1 subsampling. . Here, a three-level hierarchy process is used. In the hierarchization processing, the number of itr necessary for the update formula to converge is reduced in most images, so that the amount of calculation can be reduced.

FIG. 2 shows a flowchart of the HOE algorithm. In the flow of the optical flow process, first, a hierarchical image is created by filtering and sub-sampling, and stored in the hierarchical data buffer (S1). Next, image data (original image or hierarchized image) is input to calculate a luminance gradient, and stored in the luminance gradient memory (S2). Here, the brightness is calculated using the following equation using a multidimensional gradient filter:
E0 * lpf0 + E1 * lpf1 + E2 * lpf0
Is required. Here, E0, E1, and E2 are luminance values, and lpf0 and lpf1 are LPF filter coefficients used for the multidimensional gradient filter. The luminance gradient in the x, y, and z directions is derived from a multidimensional gradient equation with the luminance values of three frames as inputs. Next, an approximate optical flow ν2 is derived from the top three frames (previous frame, current frame and next frame) (S3), and optical flow is interpolated using the derivation of the optical flow, and the optical flow at the lower hierarchy level is calculated. Conversion to ν1 (S5). This interpolation process is a bilinear interpolation process with uniform weights. Here, an optical flow of four surrounding pixels is input, and an average of the four pixels is calculated and output. Next, frame interpolation, that is, motion compensation is performed from the previous frame and the next frame using the obtained optical flow ν1 (S6). The interpolated image is obtained by the following equation.
Emage [x + y * Xsize] = (1.0-s) * (1.0-t) * SrcImage [X + Y * Xsize]
+ (1.0-s) * t * SrcImage [X + (Y + inc_y) * Xsize]
+ s * (1.0-t) * SrcImage [X + inc_X + Y * Xsize]
+ s * t * SrcImage [X + inc_x + (Y + inc_y) * Xsize]
Here, s and t are the fractional parts of the optical flow, SrcImage is the luminance value of the next frame or the previous frame, and inc is the integer part of the optical flow. The optical flow and the frame luminance value are input, and the interpolated image luminance value of the next frame or the previous frame is calculated and output. Next, the process proceeds to a further lower level process (S7), and the above process is repeated to derive an optical flow. When the optical flow in the lowest hierarchy is obtained, the final optical flow is obtained by adding the obtained results (S8). Then, the process returns to the first step S1 to process the next frame.

In order to realize processing with high accuracy and high processing speed in hardware implementation, parameters of the HOE algorithm were optimized by simulation. For the number of iterations itr that greatly affects the amount of computation, the update formula computation is repeated until the average update amount of the entire image flow for each computation is, for example, 10 ^-4 or less, but in order to facilitate hardware control Aborts the repeated processing at a fixed number of times itr. The optimum number of iterations itr is 150. The optimum weight parameter α was α = 10.

  In addition, the HOE algorithm performs a large number of calculations that require decimal point precision, and therefore the hardware cost becomes very high. Therefore, by reducing the operation word length to the maximum bit length that does not impair the flow detection accuracy, low-load processing is realized. Here, various reduction patterns were defined, and bit length optimization simulation was performed. The reduction pattern adopted here uses 16-bit and 24-bit fixed-point numbers, and the bit allocation in the integer and fractional parts is adaptive, with the same average angle error (MAE) and itr as in the floating-point case. The number of times can be maintained. In this way, the number of ITRs and operation word length of the HOE algorithm were optimized for VLSI.

  In the HOE algorithm shown in FIG. 2, five processes of hierarchical image creation, luminance gradient creation, optical flow derivation, interpolation image creation, and bilinear interpolation are sequentially performed in real time (see FIG. 3). If the processing amount of these five processes is uniform, the operation frequency can be lowered and the power consumption can be reduced by pipeline parallel processing, but the optical flow deriving unit including the repetitive processing occupies most of the entire calculation. An efficient pipeline cannot be built. Processes such as optical flow derivation, luminance gradient calculation, and hierarchical image creation basically perform product-sum operations. Therefore, in the hardware implementation of the HOE algorithm, a common arithmetic unit CE that performs all operations has been developed to reduce the circuit scale. Here, a plurality of blocks for performing a product-sum operation are provided, and the arithmetic unit is shared by changing the data path for each process. This common arithmetic unit CE uses a dedicated 4-way SIMD data path circuit capable of simultaneous processing of four pixels in order to realize real-time arithmetic processing.

  FIG. 4 shows the overall architecture of an optical flow processor using a common arithmetic unit CE. The optical flow processor is connected to an external memory (SDRAM) and a CPU. The external memory includes an original image data memory, a final optical flow data memory, and a hierarchical data memory. Data necessary for each process is transferred from an external memory to a constant storage buffer and a luminance value storage buffer, which are input buffers, by a control signal from the CPU. The common arithmetic unit performs arithmetic processing using the data of any input buffer or the data of the internal memory input via the selector. Then, the obtained processing result is transferred to an output buffer or an internal memory (optical flow storage memory or luminance gradient storage memory). The common arithmetic unit CE includes a sequence controller (not shown). Note that only the brightness gradient data and the optical flow data are stored in the built-in memory due to a trade-off between the memory bus bandwidth and the built-in memory capacity. Note that whether or not to use the internal memory and what kind of data to store when using the internal memory may be appropriately selected.

  FIG. 5 shows an overall architecture including internal blocks of the common arithmetic unit CE. The common arithmetic unit CE includes a plurality of arithmetic blocks (processing arithmetic unit PE) for performing product-sum processing, a plurality of addition blocks (ADD) for performing addition processing of input data to the arithmetic block, and a plurality of arithmetic blocks (PE). It consists of an accumulation block (ACC) that performs the result addition process. Here, in order to perform parallel processing at the time of deriving the optical flow with the largest calculation amount, four processing arithmetic units PE are arranged in parallel, and one pixel is processed by one processing arithmetic unit PE. To do. In addition, the processing for each pixel can be realized with a throughput of 1 even during other processing.

  FIG. 6 is an internal block diagram of the processing arithmetic unit PE in the common arithmetic unit CE. Almost all processes such as optical flow derivation, luminance gradient calculation, and hierarchical image creation basically perform product-sum processing. Therefore, a plurality of blocks for performing product-sum operations are provided, and the arithmetic units are shared by changing the data path for each process. In the common arithmetic unit CE, data necessary for each calculation is transferred from the external memory to the buffer by the control signal from the CPU, and the data path can be changed by the control signal of the CPU. As a result, all the processing necessary for deriving the optical flow is performed by one common arithmetic unit CE.

  The processing arithmetic unit PE has six arithmetic elements, that is, an averaging block (AVE) for averaging, addition and shift processing, a product-sum block (BEL and BER) as first and second product-sum filters, A division / addition / subtraction block (Div / ADD) that performs division and addition / subtraction, a Calc / LPF block that is a third product-sum filter, and a difference block (Diff) that is a fourth product-sum filter. For example, in optical flow derivation, external data is input to the first product-sum filter (BEL block) on the one hand, and on the other hand, the second product-sum filter (BER block) via the averaging block (AVE). Is input. The division / addition / subtraction block (Div / ADD) performs division by inputting the processing results of both the product-sum filters BEL and BER. The result is output directly, or after being processed by the Calc / LPF block which is the third product-sum filter, the difference from the input data is taken by the difference block Diff which is the fourth product-sum filter, The result is output.

FIG. 7 shows a data path when the optical flow is derived. At the time of optical flow derivation, computation is performed using computation blocks in all processing computing units PE. Here, four-pixel parallel processing is performed using four processing computing units PE. First, optical flow data is transferred from the optical flow storage memory to the averaging block AVE, and local averages ave_u and ave_v are obtained. Next, in the BEL and BER blocks, the denominator Ex ² + Ey ² + α ² of the optical flow update formula and the numerator ave_u * Ex + ave_v * Ey + Et are calculated, and the division / addition / subtraction block (Div / ADD) divides these Find div = numerator / denominator. Next, the Calc / LPF block calculates u _{n + 1} = ave_u−Ex * div, v _{n + 1} = ave_v−Ey * div, and updates the optical flow to u _{n + 1} and v _{n + 1} . . The updated optical flow is written into an optical flow storage memory that is an internal memory. In the difference block (Diff), the optical flow update amount (ave_u−u _{n + 1} ) ² + (ave_v−v _{n + 1} ) ² is calculated. The accumulation block (ACC) adds the optical flow update amount of 4 pixels. This value is compared with a threshold value. Table 1 below shows the processing performed by each block of the processing arithmetic unit PE when the optical flow is derived.

  FIG. 8 shows a data path when calculating the luminance gradient. Here, calculation blocks necessary for calculating the luminance gradient are selected, and calculation processing is performed using them. As shown in the figure, the four processing calculators PE0, PE1, PE2, and PE3 perform different processes. The processing calculators PE0, PE1, and PE2 calculate luminance gradients in the x, y, and t directions, respectively. In the processing arithmetic unit PE0, the Diff block does not operate. Further, only the Calc_LPF block operates in the processing arithmetic unit PE3. First, the luminance values of the previous frame, the current frame, and the next frame are transferred from the external memory to the input buffer. In the BEL and BER blocks, LPF filtering (out_bel = E_ml * lpf0 + E_mm * lpf1 + E_mn * lpf0) and Diff filtering (out_ber = E_ml * diff0 + E_mm * diff1 + E_mn * diff0) are performed in the t direction. Next, LPF filtering (Tmp_lpf = (E_l + E_n) * (lpf0 + E_m * lpf1)) is performed in the x direction or the y direction in the Calc_LPF block. Here, E_mx represents a luminance value, lpfx represents an LPF filter coefficient used for the multidimensional gradient filter, diffx represents a diff filter coefficient used for the multidimensional gradient filter, diff_0 and diff_1 are luminance gradients in the x and y directions Represents. Finally, the Diff block performs Diff filtering in the x and y directions and LPF filtering in the y direction. Thus, the luminance gradients in the x, y, and t directions of one pixel are obtained using the four processing arithmetic units PE. In this way, the x-direction luminance gradient is obtained by processing the LPF filter in the t direction → the LPF filter in the y direction → the Diff filter in the x direction in the processing calculator PE0. In PE1, LPF filter in the t direction → LPF filter in the x direction → Diff filter processing in the y direction. Further, the t-direction luminance gradient is obtained by processing the Diff filter in the t direction, the LPF filter in the x direction, and the LPF filter in the y direction in the processing calculator PE2.

Table 2 below shows the processing of each block in the processing computing unit PE when calculating the luminance gradient. Here, 0, 1, 2, and 3 after the block name represent blocks in the processing arithmetic units PE0, PE1, PE2, and PE3.

  FIG. 9 shows a data path when creating a hierarchical image. The hierarchical image can be obtained by multiplying the luminance value data of 25 pixels by three processing arithmetic units and creating a one-pixel hierarchical image. First, the luminance value data is transferred from the external memory to the input buffer. Then, an addition value img_x of the luminance value of the original image is obtained by the addition blocks ADD1 and ADD2, and Gaussian for one row is obtained by each of the five product-sum blocks BER and BEL in the 0th, 1st and 2nd processing arithmetic units. Filter processing multiplication out_bel = img_a1 * A + img_a2 * B + img_a3 * C, out_bel = img_a1 * C + img_a2 * D + img_a3 * F, ..., out_bel = img_e1 * B + img_e2 * D + img_e3 * F And so on. Here, A, B, C, D, E, and F are coefficients of the Gaussian filter. Further, the multiplication result is added by 5 lines in the division / addition / subtraction block Div_Add and the accumulation block ACC, and the luminance value img (= img_a1 * A + img_a2 * B + img_a3 * C + img_b1 * of the upper layer image) B + img_b2 * C + img_b3 * D + img_c1 * C + img_c2 * D + img_c3 * E + img_d1 * B + img_d2 * D + img_d3 * E + img_e1 * B + img_e2 * D + img_e3 * F).

  FIG. 10 shows a data path at the time of bilinear interpolation. The optical flow data is transferred to the averaging block AVE, and addition and shift processing are performed. Then, the processing result is transferred again to the optical flow storage memory.

  FIG. 11 shows a data path at the time of vector addition. First, the optical flow data interpolated in the upper hierarchy from the external memory is transferred to the input buffer. Then, the optical flow data in the internal memory and the optical flow data in the input buffer are added by the averaging block AVE. The processing result is stored in the optical flow memory.

  FIG. 12 shows a data path when creating an interpolation image. At the time of creating an interpolation image, the luminance value data of the image is stored in the luminance gradient memory. First, the optical flow integer part data is transferred from the optical flow storage memory to the luminance gradient memory, and the luminance value data of the original image is transferred to the Calc_LPF block and the Diff block. At the same time, the fractional data of the optical flow is transferred to the sum-of-products block BER and multiplied, the coefficient st is determined, and finally the four coefficients st, s (1-t), t in the division / addition / subtraction block Div_Add (1-t), (1-s) (1-t) are determined. In the Calc_LPF block and Diff block, the coefficient obtained by the division / addition / subtraction block Div_Add is multiplied by the luminance value transferred from the luminance gradient memory, and Tmplpf = E_c * t (1-s) + E_d * (1-s) (1 -t) and tmp = E_a * st + E_b * s (1-t) are obtained, and finally, accumulation processing of both is performed in the accumulation block ACC and transferred to the output buffer.

  13, FIG. 15, FIG. 15, FIG. 16, FIG. 17, FIG. 18, FIG. 19, FIG. 20, and FIG. 21 respectively show the AVE operation block, BER operation block, BEL operation block, and Calc_LPF operation in the common operation unit CE. FIG. 4 shows arithmetic circuits of a block, a Div_add arithmetic block, a Diff arithmetic block, an ADD1 arithmetic block, an ADD2 arithmetic block, and an ACC arithmetic block. Tables 3 to 11 show signals handled by these calculation blocks.

Table 3 shows signals handled in the AVE operation block.

Table 4 shows signals handled by the BER calculation block.

Table 5 shows signals handled by the BEL operation block.

Table 6 shows signals handled in the Calc_LPF calculation block.

Table 7 shows signals handled by the Div_add operation block.

Table 8 shows signals handled by the diff operation block.

Table 9 shows signals handled in the ADD1 operation block.

Table 10 shows signals handled in the ADD2 operation block.

Table 11 shows signals handled in the ACC calculation block.

  FIG. 22 and FIG. 23 respectively show a data path at the time of optical flow derivation and a data path at the time of luminance gradient calculation of a BER calculation block that is one of the internal elements of the processing calculator PE. The arrow indicates the direction of the data path in the multiplexer. Although not shown in the figure, other input elements and data paths are similarly switched according to the processing contents. In this way, by changing the data path by the multiplexer according to the control signal of the CPU, the common arithmetic unit can cope with any operation by selectively using the internal elements.

  The optical flow processor described above enables efficient real-time processing using dedicated data bus circuits (parallelization and pipeline processing). As shown in FIG. 1, CIF30fps with 150 itr times per layer. Can be processed with high accuracy (average MAE of all test images = 5.2 degrees). That is, a high-precision optical flow with MAE (index of accuracy) <10 or less can be extracted in real time for a sequence of CIF 30 or higher.

  The optical flow processor described above has realized a scalable architecture corresponding to high precision and high resolution. Here, in an application that requires high accuracy and high resolution, high accuracy and high resolution can be supported by simply arranging a plurality of the common arithmetic units described above. Here, the connection is changed so that the common arithmetic units CE arranged in parallel share data. For example, wiring is performed so that input data is shared between two adjacent processing operation units PE between adjacent common operation units, and the common operation unit at one end of the plurality of common operation units Transfer the previous optical flow data from the common arithmetic unit. Thereby, it can cope with many applications. Since it is precision scalable and resolution scalable, it can be applied to many moving image recognition processing systems as motion feature amount extraction means.

  For example, when two optical flow processors are arranged in parallel, the number of repetitions can be doubled in the optical flow deriving unit, and high-precision correspondence is possible. FIG. 24 shows a pixel of a frame processed by each processor when two optical flow processors are arranged. In this case, 8-pixel parallel processing is performed, and the number of repetitions is doubled to obtain a highly accurate optical flow. FIG. 25 shows an architecture when two optical flow processors are arranged. In order to obtain an optical flow of 4 pixels with one common arithmetic unit CE, a total of 6 pixels, that is, four pixels and one pixel on the left and right sides thereof are required. Therefore, adjacent common arithmetic units, that is, the first common arithmetic unit CE1. And the 0th common arithmetic unit CE0 need to share data. Data wiring can be changed for data sharing. Specifically, each column that handles adjacent pixels between the two common arithmetic units CEO and CE1 is transferred to each other, and at the same time, the first common arithmetic unit CE1 to the zeroth common for the next column processing. It is necessary to transfer the optical flow data for two columns to the internal memory for the computing unit CEO.

  Also, a larger pixel size can be accommodated by arranging optical flow processors in parallel. FIG. 26 shows a processing frame pixel image when the NTSC size is subjected to four parallel processing. By dividing the frame to be processed in the horizontal direction, the size of the optical flow memory and the luminance gradient memory of the common arithmetic unit CE does not change even if the pixel size in the vertical direction changes. For this reason, it is possible to cope with a larger image size by simply connecting an optical flow processor. The common arithmetic units are connected in the same way as in FIG. 24, and the four are arranged, and the optical flow data in the rightmost two columns corresponding to the third common arithmetic unit CE3 are transferred to the memory storing the previous flow data of the zeroth common arithmetic unit CEO. Connect to do. For example, when four common arithmetic units CE are arranged, it is possible to process an image of the VGA 30 with 150 itr times per layer.

A graph representing the performance of an optical flow processor in terms of the number of processing pixels per unit time and the mean angle error (MAE) HOE algorithm flow chart Diagram showing five processes in the HOE algorithm Diagram showing the overall architecture of the optical flow processor Diagram showing the overall architecture including the internal blocks of the common arithmetic unit Internal block diagram of the processing arithmetic unit in the common arithmetic unit Diagram showing data path when optical flow is derived Diagram showing the data path when calculating the brightness gradient Diagram showing the data path when creating a hierarchical image Diagram showing data path during bilinear interpolation Diagram showing data path when adding vectors Diagram showing the data path when creating an interpolated image Calculation circuit diagram of AVE calculation block Calculation circuit diagram of BER calculation block Calculation circuit diagram of BEL calculation block Calculation circuit diagram of Calc_LPF calculation block Calculation circuit diagram of Div_add calculation block Arithmetic circuit diagram of Diff operation block Calculation circuit diagram of ADD1 calculation block Calculation circuit diagram of ADD2 calculation block Calculation circuit diagram of ACC calculation block Diagram showing data path when BER calculation block optical flow is derived The figure which shows the data path at the time of luminance gradient calculation of BER calculation block Diagram of frame pixels processed by each common arithmetic unit Diagram for explaining scalable architecture Image of processing frame pixel image when NTSC size is processed in 4 parallel

Explanation of symbols

  PE processing arithmetic unit, ADD1, ADD2 addition block, ACC accumulation block, AVE averaging block, BEL first product-sum block, BER second product-sum block, Div / ADD division / addition / subtraction block, Calc / LPF third product-sum Block, Diff Fourth product-sum block.

Claims (6)

For the input video, the following formula

(Where u and v are optical flows in the x and y directions, n + 1 is the number of repetitions, ave_u and ave_v are average optical flows in the x and y directions, and E_x, E_y, and E_t are x An optical flow processor that obtains a motion vector for each pixel by repeating an operation expressed by a luminance gradient in the, y, and t directions, where α is a weighting coefficient.
It is equipped with a common arithmetic unit that inputs data necessary for computation, performs hierarchical image creation at multiple hierarchical levels with different resolutions, luminance gradient calculation, optical flow derivation and interpolation, and iterative computation for interpolation image creation ,
The common arithmetic unit includes an adder that performs addition on input data, first, second, third, and fourth processing arithmetic units that calculate input data and / or data from the adder, and first to first It consists of an accumulator that adds the operation results of 4 processing operation units,
Each of the first to fourth processing arithmetic units is:
An averaging block that averages the input data;
A first product-sum block that performs a product-sum operation on input data;
A second product-sum block that performs a product-sum operation on input data;
A division / addition / subtraction block for performing division, addition and subtraction of input data from the first product-sum block and / or the second product-sum block;
A third product-sum block that performs a product-sum operation on the input data from the division / addition / subtraction block and the input data from the internal memory;
A fourth product-sum block that performs a product-sum operation on the input data from the third product-sum block and the input data from the internal memory,
The input data is changed according to the type of operation executed sequentially , the averaging block, the first product-sum block, the second product-sum block, the division / addition / subtraction block, and the third product-sum block, The fourth product-sum block is selectively used, the data path is changed , hierarchical images are created at a plurality of hierarchical levels, the luminance gradient is calculated and stored in the luminance gradient memory, and the optical flow is performed for the highest hierarchical level. An interpolation process for deriving and storing in an optical flow memory, converting a higher resolution hierarchical image from an upper level optical flow to a lower level optical flow, and storing the same in the optical flow memory; The interpolated image creation that performs motion compensation using the optical flow is sequentially executed , and the final optical flow is output .
Optical flow processor. Furthermore, an input buffer for receiving data from the external and output buffers, an internal memory for storing at least a part of the operation result, and an output buffer for storing and outputting the operation result of the common arithmetic unit,
2. The optical flow processor according to claim 1, wherein the common arithmetic unit inputs data necessary for an operation from an input buffer and an internal memory. The internal memory, the luminance gradient memory, characterized in that it comprises an optical flow memory for storing the optical flow, the optical flow processor of claim 2 for storing the luminance gradient. 2. The optical flow processor according to claim 1, wherein each of the first, second, third, and fourth processing arithmetic units is configured to obtain a luminance gradient E_x, E_y for each of the four parallel pixels when the optical flow is derived. , E_t and the optical flow befor_u, befor_v of the previous frame,
The averaging block calculates average optical flows ave_u and ave_v,
The first sum-of-products block calculates out_bel = E_x ² + E_y ² + α ² ,
The second product-sum block calculates out_ber = E_x * ave_u + E_y * ave_v + E_t,
The division / addition / subtraction block calculates div_add = out_ber / out_bel,
The third product-sum block calculates u = ave_u-E_x * div_add and v = ave_u-E_y * div_add,
The fourth product-sum block calculates tmp = (befor_u-u) ² + (befor_v-v) ² . 2. The optical flow processor according to claim 1, wherein the first, second, third, and fourth processing computing units are configured to calculate luminance values E_mx, LPF filters of a previous frame, a current frame, and a next frame when calculating a luminance gradient. Enter the coefficient lpf_x and the diff filter coefficient diff_x (where x = l, m, n)
In the first processing arithmetic unit,
The first product-sum block calculates out_bel = E_ml * lpf0 + E_mm * lpf1 + E_mn * lpf0,
The second product-sum block calculates out_ber = E_ml * diff0 + E_mm * diff1 + E_mn * diff0,
The third sum-of-products block calculates Tmp_lpf = (E_l + E_n) * lpf0 + E_m * lpf1 in the y direction,
In the second processing arithmetic unit,
The first product-sum block calculates out_bel = E_ml * lpf0 + E_mm * lpf1 + E_mn * lpf0,
The second product-sum block calculates out_ber = E_ml * diff0 + E_mm * diff1 + E_mn * diff0,
The division / addition / subtraction block outputs out_bel and out_ber,
The third product-sum block inputs out_bel and out_ber from the first, second, and third processing calculators, calculates Tmp_lpf = (E_l + E_n) * lpf0 + E_m * lpf1 in the x direction,
The fourth product-sum block calculates diff_0 = diff0 * (E_i-E_k) and diff_1 = diff0 * (E_j-E_g),
In the third processing arithmetic unit,
The first product-sum block calculates out_bel = E_ml * lpf0 + E_mm * lpf1 + E_mn * lpf0,
The second product-sum block calculates out_ber = E_ml * diff0 + E_mm * diff1 + E_mn * diff0,
The division / addition / subtraction block outputs out_bel and out_ber,
The third product-sum block inputs out_bel and out_ber from the first, second, and third processing calculators, calculates Tmp_lpf = (E_l + E_n) * lpf0 + E_m * lpf1 in the x direction,
The fourth product-sum block inputs LPF coefficients and calculates tmp = (E_i + E_k) * lpf0 + (E_j + E_g) * Lpf1
In the fourth processing arithmetic unit,
The third product-sum block calculates Tmp_lpf = (E_l + E_n) * lpf0 + E_m * lpf1 in the y direction. An optical flow processor according to any one of claims 1 to 5,
A plurality of the common arithmetic units are arranged in parallel,
Wiring to share input data in two adjacent processing arithmetic units between adjacent common arithmetic units;
An optical flow processor, wherein the previous optical flow data is transferred from another common arithmetic unit to a common arithmetic unit at one end of the plurality of common arithmetic units.

Images