WO2015083857A1

WO2015083857A1 - Surf hardware apparatus, and method for managing integral image

Info

Publication number: WO2015083857A1
Application number: PCT/KR2013/011270
Authority: WO
Inventors: 최병호; 장성준; 이상설
Original assignee: 전자부품연구원
Priority date: 2013-12-05
Filing date: 2013-12-06
Publication date: 2015-06-11
Also published as: KR101531038B1; KR20150065532A

Abstract

Disclosed are a speeded-up robust feature (SURF) hardware apparatus and a method for managing integral images. The SURF hardware apparatus is an apparatus for calculating features and descriptors of an image and comprises: an integral image generation unit for generating an integral image on the basis of an input black and white image; a multi-integral image memory for storing the pixel values of the integral image, the memory comprising a plurality of sub-integral images; and a memory management unit for managing the clock and power source of the plurality of sub-integral image memories sequentially on the basis of results of real-time monitoring of the pixel values of the integral image.

Description

SVR hardware and integrated image memory management methods

The present invention relates to a speeded up robust feature (SURF) hardware device and an integrated image memory management method.

The SURF (Speeded Up Robust Feature) algorithm is one of representative algorithms for extracting feature points of objects. The SURF algorithm performs object and scene recognition by regenerating an integrated image based on the input black and white image.

Due to the nature of the SURF algorithm, the integrated image memory is frequently accessed because it calculates the Hessian by reading the integral image value pixel by pixel. However, when the integrated image is stored in the external memory, performance deterioration is severe in terms of speed, so most implementations configure it as dual port-based memory internally.

However, a large amount of memory is required for the integrated image storage, which causes a serious power consumption problem.

An object of the present invention is to provide a Speed Up Robust Feature (SURF) hardware device and an integrated image memory management method operating at low power by minimizing integrated image memory power consumption.

According to one aspect of the invention, the SURF hardware device is a Speed Up Robust Feature (SURF) hardware device for calculating the feature points and descriptors of the image, the integrated image generation unit for generating an integrated image based on the input black and white image, the integral Multi-integrated image memory including a plurality of sub-integrated images, and the clock and power of the plurality of sub-integrated image memories are sequentially managed based on the result of real-time monitoring of the pixel values of the integrated image. And a memory management unit.

The plurality of sub-integrated image memories may be in a bit group unit.

The memory manager,

When the clock is applied to the first sub-integration image memory and the power is turned on, the pixel value of the integrated image is stored, and the pixel value of the integrated image output from the integrated image generator is real-time monitored to satisfy a predetermined threshold value. The power may be turned on by applying a clock to a second sub-integration image memory of the first sub-integration image memory.

According to another aspect of the present invention, an integrated image memory management method is a method of managing an integrated image memory by a SURF (Speeded Up Robust Feature) hardware device that calculates a feature point and a descriptor of an image. Generating and sequentially managing clocks and powers of the plurality of sub-integrated image memories based on a result of real-time monitoring of pixel values of the integrated image.

The managing step,

Applying a clock to a first sub-integrated image memory and turning on power, storing pixel values of the integrated image in the first sub-integrated image memory, and real-time monitoring of pixel values of the integrated image to define a predetermined threshold Determining whether a value is satisfied, and applying a clock to a second sub-integration image memory of the first sub-integration image memory to turn on the power when the predetermined threshold is satisfied.

The plurality of sub-integrated image memories may include a multi-integrated image memory in units of bit groups.

According to an embodiment of the present invention, the power consumption of the integrated image memory is minimized to operate the SURF hardware device at low power.

1 is an exemplary diagram of an integral value for an arbitrary image applied to an embodiment of the present invention.

2 illustrates a typical single integrated image memory structure.

3 illustrates a bit group multi-integrated image memory structure according to an embodiment of the present invention.

4 is a configuration diagram of a speeded up robust feature (SURF) hardware device according to an embodiment of the present invention.

5 is a flowchart illustrating an integrated image memory management method according to an exemplary embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise.

Hereinafter, a SURF hardware apparatus and an integrated image memory management method according to an exemplary embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an integrated value of an arbitrary image applied to an embodiment of the present invention, FIG. 2 illustrates a general single integrated image memory structure, and FIG. 3 illustrates a bit group multi-integrated image memory structure according to an embodiment of the present invention. 4 is a configuration diagram of a speeded up robust feature (SURF) hardware device according to an embodiment of the present invention.

Referring to FIG. 1, an image received by a SURF hardware device is an 8-bit gray-scale image.

The value of one pixel is represented by 8 bits, that is, it is between 0 and 255. At this time, the maximum value of one pixel is 255 when white, and the minimum value is 0 when black. In other words, the more colors close to black are in the frame, the smaller the total sum (integral value) becomes.

The integral value of the input image of FIG. 1A is '0xd1b2b0', and the integral value of the input image of FIG. 1B is '0x6529ab'. At this time, it can be seen that the image of (B) has more colors closer to black than the image of (A). Accordingly, it can be seen that the integral value of FIG. 1B having more black pixels is smaller than that of FIG. 1A.

The hardware design always considers the worst case, i.e. for a 640x480 image, when the value of every pixel is 255, the final pixel value of the integrated image is 78,336,000, which is allocated 27 bits. However, in the general situation (picture), most of them need less than 27 bits.

Therefore, unlike the conventional single integrated image memory structure as shown in FIG. 2, an embodiment of the present invention manages the bit group multi-integrated image memory structure as shown in FIG. 3.

Referring to FIG. 4, an image signal processor (ISP) 1 and a SURF feature extraction IP 3 are connected to a central processing processor (CPU) 7 and a high speed interface via a bus 5. Interfaces 9, Low Speed Interfaces 11 and External Memory 13.

The central processing processor (CPU) 33 drives an operating system (OS) and firmware (Firmware) for operating the internal register setting and SURF in the SURF hardware device.

High Speed Interfaces 35 are controller modules for providing high speed external interfaces such as USB.

Low Speed Interfaces 37 are controller modules for providing low speed external interfaces such as Universal Asynchronous Receiver / Transmitter (UART) and Serial Peripheral Interface (SPI).

The external memory 13 includes an external memory PHY 15 in which a black and white image is stored and an external memory controller 17 that controls the operation of the external memory pie.

The image signal processing unit (ISP) 1 generates an RGB (Red-Green-Blue) / Y (luminance) U (difference between the luminance signal and the blue component) V (difference between the luminance signal and the red component) image.

The image signal processor (ISP) 1 includes a gray-scale image converter (Gray-Scale Image Converter) 19 and a write direct memory access (W-DMA) module 1 (21).

Here, the black and white image converter 19 converts the original image into a black and white image. The converted black and white image is stored in the external memory 13 through the W-DMA module 1 (21).

The integrated image memory 23 is a dedicated memory for extracting feature points, and stores the integrated image generated by the integrated image generator 27. The integrated image memory 23 is a memory shared by a plurality of Hessian calculators 31.

The memory manager 25 monitors the pixel integrated value in units of frames in real time and manages the clock and power of the integrated image memory 23 based on the real time monitoring.

In this case, the integrated image memory 23 is implemented as a bit group multi-integrated image memory structure as shown in FIG. 3. That is, the memory manager 25 manages a single integrated image memory as shown in FIG. 2 in a structure in which three integrated image memories are layered in bit groups (8-bit units) as shown in FIG. 3. The memory manager 25 initially applies a clock only to the first integrated image memory 23-1 and turns on the power to store the integrated value. When the pixel integrated value meets a predetermined threshold value by real-time monitoring of the pixel integrated value in a frame unit, a clock is applied to the next integrated image memory 23-3 and the power is turned on. Subsequently, the pixel integrated value is monitored in real time on a frame basis, and when the pixel integrated value satisfies a predetermined threshold value, the clock is applied to the next integrated image memory 23-5 and the power is turned on.

In this way, the memory manager 25 monitors the integral value generated by the integrated image generator 23 in real time, and based on the threshold value, each sub-integral image memory (Sub-Integral Image MEM0, Sub-Integral Image MEM1,...) Manages the clock and power of Sub-Integral Image MEMn sequentially (Integral Image Monitor & Integral Image Memory Clock / Power Manager). At this time, the turn-on of the clock and the power supply and the settling time after the change are set to a margin.

The integrated image generator 27 reads a black and white image from the external memory 13 using a read direct memory access (R-DMA) module 29 to generate an integrated image.

In this case, the integrated image generator 27 regenerates and uses an integrated image obtained by integrating a monochrome image, which is an input image, for a quick box filtering operation. In the case of 640 x 480 size and 8-bit black and white input image, the maximum value of one pixel can be 255, and when all pixels are 255, the maximum value of the integrated image is 78,336,000, which requires a maximum of 27 bits.

The plurality of Hessian calculation units 31 patch the integrated image pixel values from the integrated image memory 23 through the data fatching unit 33 according to the box filter pattern to perform a box filter operation. Perform. The Hessian values calculated by the plurality of Hessian calculation units 31 are stored in each Hessian memory 35.

The feature point extractor 37 extracts the feature points based on the result of the box filtering operation stored in each of the Hessian memories 35. At this time, the pixel having the largest Hessian value is extracted as the feature point, compared to the result of the box filtering operation having another size.

The main controller 39 controls the added block, that is, the integrated image memory 23 and the memory manager 25, in addition to the function of controlling the entire block.

The integrated image memory 23, the memory manager 25, the integrated image generator 27, the R-DMA module 29, the Hessian calculator 31, the data patching unit 33, the Hessian memory ( 35), the SURF feature extraction IP including the feature extractor 37, the main controller 39, and the W-DMA module 2 41 may be implemented as a system-on-chip.

Based on the above description, the integrated image memory management method is as follows.

Referring to FIG. 5, the memory manager 25 monitors the pixel integrated value generated by the integrated image generator 23 in real time (S101).

As a result of monitoring in step S101, the memory manager 25 determines whether the pixel integrated value satisfies a predefined threshold (S103). Here, the threshold is determined by how many sub-integrated image memories the integral image memory 23 is divided into.

According to an embodiment, the integrated image memory having a data width of 27 bits may be divided into two groups, a 7-bit memory (upper bit) and a 20-bit memory (lower bit). At this time, up to a value of 2 ²⁰ , that is, 0xFFFFF, may be stored in the ^20- bit memory for the lower bit. If the integral is greater than that, you are turning on 7-bit memory for the higher bits. In addition, the threshold may include a turn on margin. In other words, the time to stabilize by turning on a clock or power can be turned on earlier by margining at a threshold.

On the other hand, if it is not satisfied in step S103, step S101 is restarted.

On the other hand, if satisfied, the power is turned on by applying the clock of the next sub-integrated image memory (S105).

Although the preferred embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims

A speeded up robust feature (SURF) hardware device that computes feature points and descriptors in an image.

An integrated image generation unit generating an integrated image based on the input black and white image,

A multi-integrated image memory for storing pixel values of the integrated image and including a plurality of sub-integrated images, and

A memory manager configured to sequentially manage clocks and powers of the plurality of sub-integrated image memories based on a result of real-time monitoring of pixel values of the integrated image

SURF hardware device comprising a.
The method of claim 1,

The plurality of sub-integrated image memories

SURF hardware device in bit family.
The method of claim 2,

The memory manager,

When the clock is applied to the first sub-integration image memory and the power is turned on, the pixel value of the integrated image is stored, and the pixel value of the integrated image output from the integrated image generator is real-time monitored to satisfy a predetermined threshold value. And turning on a power by applying a clock to a second sub-integration image memory of the first sub-integration image memory.
A method of managing an integrated image memory by a speeded up robust feature (SURF) hardware device that computes feature points and descriptors of an image,

Generating an integrated image based on the input black and white image, and

Sequentially managing clocks and power supplies of a plurality of sub-integrated image memories based on a result of real-time monitoring of pixel values of the integrated image;

Integrated image memory management method comprising a.
The method of claim 4, wherein

The managing step,

Applying a clock to the first sub-integration image memory and turning on power;

Storing the pixel value of the integrated image in the first sub-integrated image memory, real-time monitoring of the pixel value of the integrated image to determine whether a predetermined threshold value is satisfied, and

Turning on the power by applying a clock to a second sub-integration image memory of the first sub-integration image memory when the predefined threshold value is satisfied;

Integrated image memory management method comprising a.
The method of claim 5,

And the plurality of sub-integrated image memories includes a multi-integrated image memory in units of bit groups.