WO2014175480A1

WO2014175480A1 - Hardware apparatus and method for generating integral image

Info

Publication number: WO2014175480A1
Application number: PCT/KR2013/003520
Authority: WO
Inventors: 최병호; 김제우; 이상설; 황영배; 장성준; 김정호
Original assignee: 전자부품연구원
Priority date: 2013-04-24
Filing date: 2013-04-24
Publication date: 2014-10-30

Abstract

Disclosed are a hardware apparatus and a method for generating an integral image. Herein, a hardware apparatus for calculating the features and a descriptor of an image comprises: an image fading unit for fading the entirety of a received black and white image and reducing the maximum number of bits to a previously defined number of bits; an integral image generation unit for sequentially calculating the pixel values of the black and white image outputted by the image fading unit, and buffering, by an offset, the accumulated pixel values up to the previous line, and then storing, as a differential value, the pixel values accumulated starting from the first pixel of the next line; and a first integral value acquisition unit for calculating the pixel values of a first integral image needed to extract the features by means of the offset and the differential value.

Description

How to Create Hardware Devices and Integral Images

TECHNICAL FIELD The present invention relates to a hardware device and a method for generating an integrated image, and more particularly, to the field of object, scene recognition and hardware system-on-chip technology.

SURF (Speeded Up Robust Feature) algorithm is one of representative algorithms for extracting feature points of objects.

The SURF algorithm regenerates the integrated image internally based on the input black and white image to perform object and scene recognition. However, as the input resolution increases, the maximum cumulative value also increases. As a result, the number of bits required to express this increases, which increases the total memory usage.

For 640x480 resolution, the required maximum number of bits in the integrated image is 27 bits. Storing 307200 (640 x 480) values of 27 bits is practically impossible for hardware and system-on-chip implementations. In order to implement the algorithm, an integrated image memory usage reduction method is urgently needed.

Accordingly, it is an object of the present invention to provide a hardware device and an integrated image generating method for reducing the size of an integrated image memory.

According to an aspect of the present invention, a hardware device is a hardware device for calculating a feature point and a descriptor of an image, and includes: an image fading unit for fading the entire input black and white image to reduce the maximum number of bits to a predefined bit, the image fading unit An integrated image generator which sequentially calculates pixel values of the black and white image output from the buffer and buffers the pixel values accumulated up to the previous line as an offset, and stores the pixel values accumulated from the first pixel of the next line as a difference value, and the offset And a first integrated value obtaining unit configured to calculate pixel values of a first integrated image required to extract the feature points using the difference values.

The hardware device,

An offset buffer in which an offset, which is an accumulated pixel value of a last row or a last column of a previous line, of the black and white image output from the image fading unit is stored, and the last row of the next line calculated from the first pixel of the next line of the previous line or A first integrated image memory in which a difference value, which is a accumulated pixel value of a last column, is stored;

The first integrated value obtaining unit,

Access to the offset buffer to obtain an offset of a previous line corresponding to the first integrated image; access to the first integrated image memory to obtain a difference value of a next line of the previous line; The pixel values of the first integrated image may be calculated by adding the difference values.

The offset buffer,

An offset, which is an integral value of accumulated pixels of the last row or the last column in the group, may be indexed and stored for each group in a group unit including one or more lines.

The first integrated value acquisition unit is a plurality of scale units,

The plurality of first integration value acquirers may share the offset buffer and the first integrated image memory.

The hardware device,

A second integrated image memory which is calculated from the first pixel of the next line of the previous line and stores the difference value, which is the accumulated pixel value of the last row or last column of the next line, and a second necessary to access the offset buffer to generate the descriptor. Obtain an offset corresponding to a second integrated image, access the second integrated image memory to obtain a difference value of a line following the offset, and add the obtained offset and the obtained difference value to sum the pixel value of the second integrated image; Can be calculated.

According to another aspect of the present invention, an integrated image generating method is a method of generating an integrated image for calculating a feature point and a descriptor of an input image by a hardware device, and the maximum number of bits for the entire input black and white image is defined as a predetermined bit. Performing fading fading; calculating an offset which is a cumulative pixel value of a last row or a last column of a previous line of the faded monochrome image; the next line calculated and accumulated from the first pixel of a next line of the previous line Calculating a difference value that is a pixel value of a last row or a last column of, and calculating a pixel value of the integrated image by using the offset and the difference value.

The step of calculating the difference value,

And calculating and storing a first difference value of the first integrated image required to extract the feature point, and calculating and storing a second difference value of the second integrated image required to extract the descriptor.

The calculating step,

Calculating the pixel value of the first integrated image by summing the offset and the first difference value, and calculating the pixel value of the second integrated image by summing the offset and the second difference value. can do.

According to an embodiment of the present invention, the number of bits of the input image is reduced through a fading technique. In addition, the pixel integrated value of the input image is stored as an offset by storing the accumulated value in the last column of one line separately, and when the integrated image of the new line is generated, the offset is accumulated again from the first pixel. By forming the difference value and generating the integrated image by summing the offset and the difference value, it is not necessary to store all of the integrated values of all the pixels, thereby reducing the size of the integrated image memory.

1 is a block diagram of a SURF hardware device according to an embodiment of the present invention.

2 is a configuration diagram of an offset buffer according to an embodiment of the present invention.

3 is a block diagram of an integrated image memory according to an exemplary embodiment of the present invention.

4 is a diagram illustrating an implementation of a scale processor and a feature point extractor according to an exemplary embodiment of the present invention.

5 is a flowchart illustrating a method of generating and obtaining an integrated image according to an exemplary embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, except to exclude other components unless specifically stated otherwise.

In addition, the terms "… unit", "... module" described in the specification means a unit for processing at least one function or operation, which may be implemented in hardware or software or a combination of hardware and software.

Hereinafter, a hardware device and an integrated image generating method according to an exemplary embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a speeded up robust feature (SURF) hardware device according to an embodiment of the present invention, Figure 2 is a block diagram of an offset buffer according to an embodiment of the present invention, Figure 3 is an embodiment of the present invention 4 is a block diagram of an integrated image memory according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating an implementation of a scale processor and a feature point extractor according to an exemplary embodiment of the present invention.

First, referring to FIG. 1, the SURF hardware device 100 generates a feature point and a descriptor. At this time, image pyramids are generated to represent a scale space for extracting feature points, and feature points are extracted from the generated image pyramid. At this time, the integral image used is the core of the SURF algorithm.

The SURF hardware apparatus 100 includes an image storage unit 101, an image fading unit 103, an integrated image generation unit 105, an offset buffer 107, a first integrated image memory 109, and a second integrated image memory ( 111, the scale processing unit 113, the first integrated value obtaining unit 115, the Hessian calculating unit 117, the feature point extracting unit 119, the feature point storing unit 121, and the second integrated value obtaining unit 123. , Rotation calculation unit 125, descriptor calculation unit 127, and descriptor storage unit 129.

At this time, the left and right sides are generated as two large blocks with the feature point storage unit 121 as a boundary, and the left side is divided into the feature point extraction unit and the right side the descriptor generation unit.

The image storage unit 101 stores the input black and white image.

The image fading unit 103 fades the entire black-and-white image stored in the image storage unit 101 to reduce the maximum number of bits to a predetermined bit.

In the case of black and white video, the analog input is quantized from 0 to 255 to use digital values. Here, the fading technique is to reduce the number of 256 quantization levels. When the number of quantization levels is reduced, the contrast or color of the image may change, but the image itself does not change significantly. Since object recognition is not based on the contrast or the color of the image, the recognition rate has little effect on the number of quantization levels.

When the number of quantization levels is reduced from 256 to 192, the representation bits are reduced from 8 bits to 6 bits, and the use of integrated image memory is reduced by about 8% in 640 × 480 resolution images.

The integrated image generator 105 sequentially calculates and accumulates pixel values of the pixels constituting the faded monochrome image received from the image fading unit 103 in the line direction.

The line may include a plurality of pixels arranged in units of rows of an image or a plurality of pixels arranged in units of columns of an image.

The integrated image generation unit 105 sequentially calculates and accumulates pixel values in group units including one or more lines. At this time, the accumulated pixel value of the last row or the last column of the group is calculated as an offset of the group and stored in the offset buffer 107.

In addition, the integrated image generator 105 calculates a differential value based on the offset and stores the differential value in the first integrated image memory 109 and the second integrated image memory 121, respectively.

In this case, the integrated image generating unit 105 newly calculates and accumulates the difference value starting from the pixel value of the first row or the first column of the next line of the previous line regardless of the accumulated pixel value of the previous line.

The offset buffer 107 indexes and stores the offset calculated by the integrated image generator 105 for each group. In addition, since the offset buffer 107 is implemented as a register, a collision does not occur even when the feature extraction unit and the descriptor generation unit simultaneously access.

In this case, the offset buffer 107 may be implemented as shown in FIG. 3.

Referring to FIG. 3A, the integrated image generation unit 105 may include an offset P1 buffered in group units including N lines (Row or Column # 0 to Row or Column #N). As shown in (b), they are stored in the offset buffer 107 for each group.

That is, the first group RG # 0 includes N lines (Row or Column # 0 to Row or Column #N), and the offset of the first group RG # 0 is the Nth line (Row or Column #). The accumulated pixel value of the last row or last column of N) is stored. Similarly, the offset of the second group RG # 1 stores the accumulated pixel value of the last row or the last column of the 2N th line (Row or Column # 2N).

In this case, in the case of a 640 × 480, 8-bit black and white image, when the offset is calculated in a 2-line group unit, the representation bit of the pixel value of the integrated image is reduced from the conventional 27 bits to 19 bits. This can reduce the usage of integrated image memory by about 30%.

Referring again to FIG. 1, the first integrated image memory 109 stores the difference value of the integrated image required for feature point extraction.

The second integrated image memory 111 stores the difference value of the integrated image required for the descriptor extraction.

The reason why the integrated image memory is divided into the first integrated image memory 109 and the second integrated image memory 111 and configured independently is the first integrated value obtaining unit 115 and the second integrated value obtaining unit 123. This is because can operate simultaneously without sequentially operating. If integrated image memory is implemented as one, an integrated image memory access conflict may occur during simultaneous operation.

In addition, since the first integral value obtaining unit 115 and the second integral value obtaining unit 123 are interfaced with a FIFO, the first integral value obtaining unit 115 and the second integral value obtaining unit 123 mutually interact with each other. Can work independently

The first integrated image memory 109 is a dedicated memory for feature point extraction. The plurality of scale processing units 113 are memories shared with each other.

The first integrated image memory 109 filters the pixel values of the integrated image in a specific rectangular area within the faded input black and white image, but the size of the specific rectangular area is not large. In this case, the specific rectangular area may be a rectangular box having a size of 51 × 51 or more.

The second integrated image memory 111 is a dedicated memory for descriptor extraction. The second integrated image memory 111 stores all the pixel values of the integrated image of the faded input black and white image frame. At this time, the integrated image of a considerably larger area is stored around the feature point.

Here, the first integrated image memory 109 and the second integrated image memory 111 may be implemented as shown in FIG. 4.

Referring to FIG. 4, in the first integrated image memory 109 and the second integrated image memory 111, differential values are indexed and stored for each entire line (Row or Column # 0 to Row or Column #N) of the image. That is, the difference value is a cumulative pixel value of the last row or the last column that is sequentially calculated from the pixel values of the first row or the first column (0,0) of the line corresponding to each line.

Again, referring to FIG. 1, the scale processor 113 performs parallel operations on a plurality of scales at the time of feature point extraction. In this case, the scale is a filtering operation for a specific rectangular region of the first integrated image memory 109 described above, and may include, for example, six scales. Therefore, the scale processor 113 may be implemented in plural as the number of scales. It is implemented as a pipeline and can be composed of six or more parallel operations.

At this time, each scale processing unit 113 filters by varying the size of a specific rectangle. For example, each scale processor 113 may be implemented with filter sizes of 9 × 9, 15 × 15, 21 × 21, 27 × 27, 39 × 39, and 51 × 51.

Each of the scale processor 113 includes a first integral value acquirer 115 and a Hessian calculator 117.

Here, the first integrated value obtaining unit 115 adds the offset obtained by approaching the offset buffer 107 and the difference value obtained by accessing the first integrated image memory 109, and then, in the Hessian calculation unit 117. Calculate the pixel value of the required integrated image.

That is, the first integrated value acquirer 115 obtains, from the offset buffer 107, an index indexed to a corresponding group among a plurality of lines constituting the integrated image required by the Hessian calculator 117. Then, the difference value of the next line of the group is obtained from the first integrated image memory 109. The offset and difference values thus obtained are added to calculate the integrated image pixel value and output to the Hessian calculator 117.

Referring to FIGS. 3 and 4 again, the first integrated value obtaining unit 115 may offset the first group RG # 0 and the next line Row or Column N + 1 of the first group RG # 0. The integrated image pixel value may be calculated by calculating the difference value of.

The Hessian calculator 117 calculates a Hessian determinant by performing a box filter operation using the integrated image pixel value received from the first integral value obtainer 115.

The feature point extractor 119 extracts the feature points by generating an image pyramid using the Hessian determinant calculated by the Hessian calculator 117.

In this case, the feature point extractor 119 may be formed in plural in correspondence with the number of the scale processor 113, and may be implemented as shown in FIG. 5.

Referring to FIG. 5, if the scale processing unit 113 has a total of six, the feature point extracting unit 119 is implemented as a total of four. That is, the indices of the six scale processing units 113 are S0, S1, S2, S3, S4, and S5, respectively, and the indices of the four feature point extracting units 119 are F0, F1, F2, and F3, respectively.

Here, if the scale processing unit 113 is six, it is obvious that the first integral value obtaining unit 115 and the Hessian calculating unit 117 are also implemented in six.

At this time, F0 is connected to S0, S1, S2, F1 is connected to S0, S1, S3, F2 is connected to S1, S3, S4, and F3 is connected to S2, S4, S5. As such, one feature point extractor 119 extracts the final feature point based on the feature points output by the three scale processing units 113.

The feature point storage unit 121 stores the feature points extracted by the feature point extractor 119 and is implemented as a FIFO (First In First Out).

The second integral value obtaining unit 123 obtains, from the offset buffer 107, an offset indexed to a corresponding group among a plurality of lines constituting the integrated image required for the descriptor generation. The difference value of the next line of the group is obtained from the second integrated image memory 111. The offset and difference values thus obtained are added to calculate the integrated image pixel values and output to the rotation calculator 125 and the descriptor calculator 127, respectively.

The rotation calculation unit 125 calculates a rotation value based on the integrated image pixel value obtained by the second integration value acquisition unit 123. Based on the coordinates and scales of the feature points extracted by the feature point storage unit 121, the main direction of the feature points is calculated based on the integral value of the specific area. This shows how much of the current feature point is rotated, which is used as a criterion for determining the area of the integral value used when calculating the descriptor.

The descriptor calculating unit 127 calculates the descriptor based on the integrated image pixel value obtained by the second integral value obtaining unit 123. The descriptor is assigned to the feature point stored in the feature point storage unit 121. Descriptor uses integral value of specific area according to scale around feature point, and after Harr-wavelet filtering, it is calculated and expressed in total of 4, dx, dy, | dx |, | dy |. Also, the area of integral value used to calculate the descriptor is adjusted by the rotation (angle) previously calculated.

The descriptor storage unit 129 stores descriptors calculated by the descriptor calculator 127.

5 is a flowchart illustrating a method of generating and obtaining an integrated image according to an exemplary embodiment of the present invention. That is, a flowchart illustrating an integrated image generation and acquisition operation of the SURF hardware device 100 described above.

Referring to FIG. 5, the image fading unit 103 reduces the maximum number of bits to a predetermined bit by applying a fading technique to the entire black-and-white image received (S101).

The integrated image generator 105 sequentially calculates pixel values in a group unit including one or more lines of a faded black and white image, and offsets the accumulated pixel values of the last row or the last column of the group. The calculation is performed (S103) and stored in the offset buffer 107 (S105).

In addition, the integrated image generation unit 105 calculates (S107) the accumulated pixel value of the last row or the last column of the line for each line as a differential value of the corresponding line, and thereby the first integrated image memory 109 and the second. Each is stored in the integrated image memory 121 (S109).

Next, the first integral value obtaining unit 115 obtains, from the offset buffer 107, an offset indexed to a corresponding group among a plurality of lines constituting an integrated image required by the Hessian calculating unit 117 (S111). ). The difference value of the next line of the group is obtained from the first integrated image memory 109 (S113).

Next, the first integrated value obtainer 115 calculates an integrated image pixel value by adding the offset acquired in step S111 and the difference value obtained in step S113 (S115) and outputs the integrated image pixel value to the Hessian calculator 117.

In addition, the second integrated value obtaining unit 123 obtains, from the offset buffer 107, an offset indexed to a corresponding group among a plurality of lines constituting the integrated image required for the descriptor generation (S117). The difference value of the next line of the group is obtained from the second integrated image memory 111 (S119).

Next, the second integral value obtaining unit 123 calculates the integrated image pixel value by adding the offset obtained in step S119 and the difference value obtained in step S121 (S121), and then rotates the calculating unit 125 and the descriptor calculating unit ( 127) respectively.

The embodiments of the present invention described above are not only implemented through the apparatus and the method, but may be implemented through a program for realizing a function corresponding to the configuration of the embodiments of the present invention or a recording medium on which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims

A hardware device for calculating feature points and descriptors of an image,

An image fading unit for fading the entire input black and white image to reduce the maximum number of bits to a predetermined bit;

An integrated image generator which sequentially calculates pixel values of the black and white image output from the image fading unit, buffers pixel values accumulated up to the previous line as offsets, and stores the pixel values accumulated from the first pixel of the next line as differential values. , And

A first integrated value obtaining unit configured to calculate a pixel value of a first integrated image required to extract the feature point using the offset and the difference value

Hardware device comprising a.
The method of claim 1,

An offset buffer in which an offset which is an accumulated pixel value of a last row or a last column of a previous line of a monochrome image output from the image fading unit is stored; and

A first integrated image memory which is calculated from the first pixel of the next line of the previous line and stores a difference value which is a accumulated pixel value of the last row or last column of the next line,

The first integrated value obtaining unit,

Access to the offset buffer to obtain an offset of a previous line corresponding to the first integrated image; access to the first integrated image memory to obtain a difference value of a next line of the previous line; And summing the difference values to calculate pixel values of the first integrated image.
The method of claim 2,

The offset buffer,

And an offset, which is an integral value of accumulated pixels of the last row or the last column in the group, is indexed and stored for each group in a group unit including one or more lines.
The method of claim 2,

The first integrated value acquisition unit is a plurality of scale units,

And a plurality of the first integral value obtaining units share the offset buffer and the first integrated image memory.
The method of claim 3,

A second integrated image memory which stores a difference value which is calculated from the first pixel of the next line of the previous line and is a accumulated pixel value of the last row or last column of the next line;

Access to the offset buffer to obtain an offset corresponding to the second integrated image required to generate the descriptor, access to the second integrated image memory to obtain a difference value of the line following the offset, and obtain the obtained offset and And summing the difference values to calculate pixel values of the second integrated image.
A method for generating an integrated image for calculating a feature point and a descriptor of an input image by a hardware device,

Performing fading to reduce the maximum number of bits to a predetermined bit on the entire black-and-white image received;

Calculating an offset which is a cumulative pixel value of a last row or a last column of a previous line of the faded monochrome image;

Calculating a difference value that is a pixel value of the last row or the last column of the next line, which is calculated and accumulated from the first pixel of the next line of the previous line, and

Calculating a pixel value of the integrated image using the offset and the difference value

Integrated image generation method comprising a.
The method of claim 6,

The step of calculating the difference value,

Calculating and storing a first difference value of a first integral image required to extract the feature point, and

Calculating and storing a second difference value of a second integrated image required to extract the descriptor

Integrated image generation method comprising a.
The method of claim 7, wherein

The calculating step,

Calculating the pixel value of the first integrated image by summing the offset and the first difference value, and

Calculating a pixel value of the second integrated image by summing the offset and the second difference value;

Integrated image generation method comprising a.