KR101864000B1 - Multi-purpose image processing core - Google Patents

Abstract

Object detection, recognition, and tracking algorithms are used in many applications of vision. The output of these algorithms is essential for context awareness and decision making. The accuracy and processing latency of these algorithms are important parameters for the success of the system. This invention enables a neural network based on techniques and meets latency constraints.

Description

Multi-Purpose Image Processing Core {MULTI-PURPOSE IMAGE PROCESSING CORE}

The present invention relates to an image processing method of an FPGA for analyzing video frames embedded in a platform and operating in real time, using a neural network-based technique.

The neural network of vision is used more often because of its performance in complex tasks such as large-scale classification [ref 1] or multimodal [ref 2]. This success can be attributed to learning unspecified functions from unclassified data [ref 3], ref [4], layer processing via deep structures [ref 5] - ref 7 and recursive processing [ref 3] And the development of a broad range of statistical dependencies using [8]. The neural network approach is an orthonormous approach to the kernel method: the input is projected into the nonlinear high-dimensional space of the hidden unit, and the linear hyperplane can partition the data [ref 9]. This nonlinear projection is a powerful representation of visual data and is available for multiple other tasks such as classification, projection, tracking, clustering, point of interest detection, and the like. Thus, the image or video block is "analyzed" by the neural network via multiple layer processing, hidden layer activity representing visual input can be multiplexed into many different tasks as needed, ].

Demand for real-time embedded visual processing is growing, and demand for intelligent robot platforms such as UAV (Unmanned Aerial Vehicles) is also increasing. Such a system is expected to operate and navigate in an automated manner, which requires successful implementation of image and video knowledge functions. Scene recognition, detection of a specific object in an image, classification of a moving object, and object tracking are some of the essential visual functions required in an automated robot system. The weight and energy specifications of these systems are limited both by the visual processing function, the complexity and the number of reductions in operating capacity. At least a subset of these functions can mitigate the constraints of a typical time processing core.

In the present invention, sparse and complete image representations are formed in the hidden layers of the neural network and provide useful power in many ways [ref 4]. In particular, it can be embedded in a UAV platform for monitoring and recognition purposes.

[1] A. Krizhevsky, I. Sutskever, and G. Hinton, "Imagenet classification with deep convolutional neural networks," in Advances in Neural Information Processing Systems 25, 2012, pp. 1106-1114.

[2] N. Srivastava and R. Salakhutdinov, "Multimodal learning with deep boltzmann machines," in Advances in Neural Information Processing Systems 25, 2012, pp. 2231-2239.

[3] G. E. Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets," Neural computation, vol. 18, no. 7, pp. 1527-1554, 2006.

[Reference 4] B. A. Olshausen et al., "Emergence of simple-cell receptive field properties by learning a sparse code for natural images," Nature, vol. 6583, pp. 607-609, 1996.

[5] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324,1998.

[Reference 6] M. Ranzato, F. J. Huang, Y.-L. Boureau, and Y. Lecun, "Unsupervised learning of invariant feature hierarchies with application to object recognition," in Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on. IEEE, 2007, pp. 1-8.

[7] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layerwise training of deep networks," Advances in neural information processing systems, vol. 19, p. 153, 2007.

[8] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," Cognitive modeling, vol. 1, p. 213, 2002.12

[9] A. Coates, A. Y. Ng, and H. Lee, "An analysis of single-layer networks in unsupervised feature learning," in International Conference on Artificial Intelligence and Statistics, 2011, pp. 215-223.

[10] T. S. Lee, D. Mumford, R. Romero, and V. A. Lamme, "The role of the primary visual cortex in higher level vision," Vision research, vol. 38, no. 15, pp. 2429-2454, 1998.

[Reference 11] Y.-L. Boureau, F. Bach, Y. LeCun, and J. Ponce, "Learning Mid-level Features for Recognition," IEEE Vision and Pattern Recognition (2010). IEEE, 2010, pp. 2559-2566.

The goal of the present invention is to provide an FPGA implementation of a neural network that relies on an image processing core.

According to one aspect, the inventive multi-purpose image processing core 101 comprises at least one image analyzer 102 block; And at least one memory interface 103 block.

According to one embodiment, the image analyzer 102 block includes at least one feature extraction block 104; At least one feature summing (105) block; And at least one classification 106 block.

According to one embodiment, the image analyzer 102 block comprises a video frame 107; Feature dictionary 108; A classification matrix 110; Feature calculation request 109; And a scarcity multiplier 114.

According to one embodiment, the image analyzer 102 block includes a feature vector 111; And a classification label 112.

According to one embodiment, the feature extraction 104 block includes a patch 310 acquisition process; P vector (302) building process; P vector average value (303) calculation process; A process of building a binary PB vector 304; A bit flipping distance vector (DV) 305 calculation process of the PB together with a dictionary (D); An average value (306) calculation process of DV; A standard deviation value (307) calculation process of DV; Activation threshold (AT) 308 computation process of DV; And a pixel feature vector (PFV) 309 of DV.

According to one embodiment, the patch acquisition process includes writing each new incoming video line to a bottom line FIFO (401); Reading the previous video line from the bottom line FIFO (401) and writing to the upper line FIFO (401) if the next video line is coming; Continuing the first two steps until all line FIFOs 401 are populated with the required lines to build 204; If all lines are available, reading K times from the line FIFO 401 and acquiring (240) the patch; Obtaining the next pixel patch using the reading of K + 1 from the line FIFO (401); Continuing the read operation until all patches have been acquired through the video line; And moving the video line to an upper line FIFO (401) to produce a downward movement of the patch (204).

According to one embodiment, the inventive versatile image processing core 101 uses a binary PB (603) vector instead of the scalar P (501) vector for the distance calculation 305 between the feature dictionary D 701.

According to one embodiment, the inventive versatile image processing core 101 uses the binary feature dictionary D 701 instead of the scalar feature dictionary for the distance calculation 305.

According to one embodiment, the inventive multipurpose image processing core 101 has the possibility to load another feature dictionary (D) 701 during the operation according to the scenario.

According to one embodiment, the bit flipping distance calculation 305 process includes comparing the PB 603 vector to all columns 702 of feature dictionary D 701 by using the xor (801) operation; computing (802) said number of entries equal to "1" after xor operation 801; And constructing the distance vector DV (804) (803).

According to one embodiment, the inventive versatile image processing core 101 maintains the number of entries equal to "1" instead of the result of all xor operations 801,

According to one embodiment, the pixel feature vector (PFV) computation process 309 includes comparing the entry of DV 805 with AT 901; Assigning "0" if the entry of DV 805 is larger than AT 901; Assigning "1" if the entry of DV 805 is smaller than AT 901; And constructing a pixel feature vector (PFV) 309 (904).

According to one embodiment, the pixel feature vector (PFV) 904 maintains the result of the comparison between AT and DV entries as a binary value.

According to one embodiment, the feature summing (105) block comprises at least one integrated vector calculator (1001) for calculating the unified vector IV (1201); At least one address calculator (1002) for calculating the internal RAM (1004) according to the feature calculation request (109); At least one feature calculation request FIFO (1003) for storing the feature calculation request (109); And at least one sub-block of internal RAM 1004 that stores the PFV 309.

According to one embodiment, the feature summing (105) block includes four identical sub-regions; The boundary coordinate 1101 of the feature calculation request 109 is divided into the quadrants 1103, 1104, 1105 and 1106 as pixel values and the other coordinates 1102 are calculated.

According to one embodiment, the feature summing 105 block reads the PFV 309 from the external memory 113 and writes the internal RAM 1004 to make the calculation faster.

According to one embodiment, the sub-block of the unified vector calculator 1001 includes reading the PFV 309 from the internal RAM 1004; And computing the quadrant integration vector QIV 1103-1 by adding all of the entries of the previous PFV 904 both in the horizontal and vertical dimensions.

According to one embodiment, the inventive multipurpose image processing core 101 obtains all values of the first PVF 309 as "0" to omit the subtraction operation between the quadrant integration vector and the first PVF.

According to one embodiment, the classification block 106 comprises at least one C matrix row arbiter 1303 for controlling the multiplication of the C matrix row and the FV 111; And at least one multiplication & add (1304) operator to implement the matrix vector multiplication operation.

A multipurpose image processing (IP) core for meeting the objectives of the present invention is described in the accompanying drawings.
Figure 1 shows the IP core of an FPGA with external elements.
Figure 2 shows a video and patch structure.
Figure 3 shows the flow of a function extractor.
4 is a structure of a patch acquisition process.
5 shows the structure of a P vector.
Figure 6 shows the structure of a binary PB vector.
7 is a dictionary D.
Fig. 8 shows the structure of the distance vector DV.
9 is a calculation of the pixel function vector PFV.
10 is a structure of a functional summing unit.
11 shows the structure of a quadrant.
12 is a calculation of the function vector FV.
13 is a calculation of the hierarchical label CL.

In a preferred embodiment of the present invention, the inventive multi-purpose image processing core 101 is implemented in the FPGA 100. [ The core comprises two main sub-blocks; An image analyzer 102 and a memory interface 103.

The memory interface 103 performs data transfer between the image analyzer 102 and the memory interface 103. The image analyzer 102 includes three sub-blocks; Feature extraction (104) block; Feature summing (105) blocks; And classification (106) blocks. The image analyzer 102 block includes five types of inputs from outside the FPGA 100; A video frame 107; Feature dictionary 108; A classification matrix 110; Feature calculation request 109; And a scarcity multiplier 114. [ The video frame 107 includes two parameters; Resolution and frame rate. The resolution is M (row) 201 N (column) 202 and the frame rate is the number of frames 203 captured in one second. Other inputs; The feature dictionary 108, hierarchical matrix 110, feature computation request 109 and scarcity multiplier 114 are described in the following sections.

The feature extraction 104 block begins the patch acquisition process. This process captures the selected relative coordinates of the patch 204 from the video frame 107. To capture the associated coordinates, the incoming video line (row 201 of video frame 107) is written to line FIFO 401. [ Patch 204 The patch acquisition 301 process, in accordance with dimension (K), uses K line FIFOs 401. Each incoming video line is first written to the bottom of the line FIFO 401 and if the next video line comes in, the previous line is read from the bottom of the line FIFO 401 and written to the upper line FIFO 401. This step lasts until all the line FIFOs 401 are filled with the required lines to build the patches 204. [ If all lines are available, the next line comes in, and the pixel value is read from the line FIFO 401. After K read operations, the patch prepares for further operation. A read of K + 1 from the line FIFO 401 confer the next pixel patch. This step continues until all the patches 204 are captured through the line. During a patch read from line FIFO 401, the new line continues to move to upper line FIFO 401. This movement creates a downward movement of the patch 204 through the video frame 107.

P vector 501 is constructed 302 by utilizing the value of the patch 204 pixel to be captured. In practice, this build process is a simple registration assignment. There is a KxK registration from L1P1 402 to LKPK 403, and all registrations retain the associated pixel values. The bit size of the registration is determined by the maximum possible pixel value.

All pixel values of the patch 204 must be added and divided by the total number of pixels to calculate the average value of the P-vector (P 占 602). The additional process may be implemented by an adder, and the number of inputs of the adder may vary depending on the FPGA capability. The number of inputs of the adder can affect the pipeline clock latency and the number of adders used. After all the pixel values are calculated, the whole is divided by K * K.

After computing P? 602, each entry of P vector 501 is compared to P? 602 and binarized 304 to build PB vector 603. The P? The binarization step is essential to implement this image processing algorithm in current FPGAs. At a value smaller than P? (602), "0" is assigned. At the same or larger value, "1" is assigned. After all the values are compared 601 to the average value, a binary P (501) vector PB 603 is obtained. PB 603 is T (604) with a "1" bit vector if T (604) equals K * K.

All binary vectors PB 603 constructed from all the patches 204 of the image are converted into functional vectors using a pre-computed dictionary with Z (703) numbers of visual words. Dictionary D 701 is T (604) with a Z (703) bit matrix. The entity of D 701 may be a binary value; Quot; 1 "or" 0 ". The column of the D (701) matrix (DC1 - DCZ 702) is registered in the internal register of the FPGA 100. The dictionary is loaded into the FPGA by means of a communication interface such as PCI, VME, or the like. An entry in a dictionary can be updated at any time after an entry is stored in an internal register.

The bit flipping (or hamming) distance calculation 305 calculates the similarity between all the columns (DC1 - DCZ 702) of the two vectors: PB 603 and D 701. 0 "if the entry of the PB 603 and the DCX 702 are the same, and is otherwise assigned" 1 ". This action is implemented by xor (801). The total number of "1's" after xor (801) operation is a measure of the difference between two binary vectors. DV 804 contains the hamming distance of a single PB 603 vector in all visualization words (column 702) of the dictionary. The entry 805 of DV 804 maintains a number of "1" s, and thus the entry is an integer value and can be represented by a lower bit when compared to PB 603 or DCX 702. DV 804 is H (806) with a Z (703) bit vector. H 806 is the minimum number of bits that can define a scalar value T (604).

The average value of DV 804 (DVμ) is calculated similar to Pμ (602). To compute the standard deviation of DV 804 (DV?) 307, DV? Is subtracted from each entry of DV 805. The squares of these subtractions are calculated and all the squares are summed. Next, the entire value is divided by Z (703). Finally, the square root is calculated and DV? Is obtained. The activation threshold value AT (901) is calculated (307) by the equation (1). This threshold is used to construct a scarcity representation via nullifying a distance value that is greater than a particular value.

(1)

(One)

Each entry of the DV 804 to build the pixel feature vector 309 is compared 902 with the AT 901. If the entry 805 is larger than the AT 901, assigns "0" to the associated entry of the PFV 904. The result is "1" by the Z pixel feature vector PFV (904).

As a result, in each pixel of the video frame 107, a bit vector (pixel feature vector PFV 904) by Z 703 is obtained. The PFV 904 is sent to the memory interface 103 and recorded in the external memory 113.

The feature calculation request 109 is recorded in the feature calculation request FIFO 1003, and the request is recorded as pixel coordinates. The CPU transmits the coordinates of two boundary pixels (top left and bottom right, black point 1101), and the FPGA calculates the remaining (white point 1102) coordinates of the bottom area. The main idea is that the region has four identical sub-regions; Is divided into quadrants 1103, 1104, 1105, and 1106, and collects the pixel feature vector (PFVs 904) and links internal feature vectors to obtain the feature vector (FV 111).

In accordance with the pixel coordinates, the internal RAM 1004 address is computed by the address calculator 1002 block. The block knows the contents of the RAM, and the line coordinates are stored. To speed up the calculation, the PFV 904 value is read from the external memory 113 and written into the internal RAM. The RAM can store R x N (202) x Z (703) bits of data. R is the maximum number of lines that can be processed simultaneously.

The internal vector calculator 100 reads the required PFVs 904 from the internal RAM 1004 and allows the internal vector 1201 to be computed. The unified vector IV (1201) entry is the sum of all entries of the previous PFVs (904) on both the horizontal and vertical dimensions. For example, IV11 (1201-11) is the same as PFV11 (904-11), IV12 (1201-12) is the same as the sum of IV11 (1201-11) and PFV12 (904-12) 21) is equal to the sum of PFV11 (904-11) and PFV21 (904-21). The final result is the quadrant integrator vector IV22 (1201-22). The recruitment feature operation requires a difference between IV22 (1201-22) and IV11 (1201-11). Therefore, obtaining PFV11 (904-11) as "0" is the same. All of the integrated vectors IV22 (1201-22) are the difference and are the same as QIV (1103-1).

Because there are four quadrants (Q1 1103, Q2 1104, Q3 1105 and Q4 1106), the results of all quadrants 1103-1, 1104-1, 1105-1 and 1106-1, And the final feature vector FV 1202 is obtained. FV 1202 is a G x S bit vector. S is the minimum number of bits that can store a "1" in the quadrant. G is the same as 4 * Z (703). The vector is stored in the FPGA's internal RAM. This feature vector FV represents an image patch defined by boundary coordinates and can be used for classification and clustering purposes and is executed in the FPGA or CPU via memory transfer.

After the image area is terminated, that is, when pooling with the required coordinates in the area is terminated, the internal RAM 1004 is updated with a new line and a new pooling calculation is started. This process is controlled by the integrated vector calculator 1001 with the help of the address calculator 1002. [

The classifier block 106 generates a vector similar to the hierarchical label using a linear classification method. This performs a matrix vector multiplication of the hierarchical matrix C 1301 using the FV 111. [ Hierarchical matrix C 1301 is loaded into the FPGA as feature dictionary D 701. The row arbiter 1303 controls the C (1301) matrix row management for the FV multiplication 111. The C (1301) matrix is a J (1302) x G x S bit matrix. The result is a hierarchical label CL (112) vector. The entry of the CL 112 is a multiplication of the C (1301) row and the FV (111). CL 112 is sent to the CPU for further processing, classification, detection, and the like.

Claims (19)

In an inventive multi-purpose image processing core,
At least one image analyzer block; And
At least one memory interface block
/ RTI >
The image analyzer block comprising:
At least one feature extraction block;
At least one feature summing block; And
At least one classification block
/ RTI >
Wherein the feature extraction block comprises:
Patch acquisition process;
A P vector construction process for assigning each pixel value of the patch to each vector;
An average value calculation process of the P vector;
A binary PB vector construction process for binarizing the P vector based on the average value of the P vector;
A bit flipping distance vector (DV) calculation process of a PB using a dictionary (D);
The average value calculation process of DV;
The standard deviation value calculation process of DV;
An activation threshold (AT) calculation process of DV; And
The pixel feature vector (PFV) calculation process of DV
Gt; image processing core. ≪ / RTI >
delete The method according to claim 1,
The image analyzer block
Video frames;
Features Dictionary;
Classification matrix;
Feature calculation request; And
Scarcity multiplier
Gt; A multi-purpose image processing < / RTI >
The method according to claim 1,
The image analyzer block
Feature vector; And
Sort labels
Gt; A multi-purpose image processing < / RTI >
delete The method according to claim 1,
The patch acquisition process
Writing each new incoming video line to a bottom line FIFO;
Reading the previous video line from the bottom line FIFO and writing to the upper line FIFO when the next video line comes;
Continuing with writing to the bottom line FIFO until all line FIFOs are filled with lines required to build a patch and writing to the upper line FIFO;
If all the lines are available, reading from the line FIFO K times and obtaining the patch;
Obtaining a next pixel patch using a read of K + 1 from the line FIFO;
Continuing the read operation until all patches have been acquired through the video line; And
Moving the video line to an upper line FIFO to produce a downward movement of the patch
Gt; image processing core. ≪ / RTI >
The method according to claim 1,
Characterized by using a binary PB vector instead of a P vector for the distance vector (DV) computation.
The method according to claim 1,
Characterized by using a binary feature dictionary D instead of a scalar feature dictionary for the distance vector (DV) calculation.
The method according to claim 1,
Characterized by having the possibility to load another feature dictionary during operation according to a scenario. ≪ RTI ID = 0.0 > [0002] < / RTI >
The method according to claim 1,
The distance vector (DV) calculation process
comparing the PB vector with all the columns of the feature dictionary D by using the xor operation;
calculating a number of entries equal to "1" after the xor operation; And
Constructing the distance vector (DV)
Gt; image processing core. ≪ / RTI >
11. The method of claim 10,
Wherein said distance vector (DV) is characterized by maintaining the number of entries equal to "1" instead of the result of all xor operations.
The method according to claim 1,
The pixel feature vector (PFV) calculation process
Comparing an entry of DV with an AT;
Assigning "0" if the DV entry is larger than the AT;
Assigning a "1" if the DV entry is smaller than the AT; And
Constructing a pixel feature vector (PFV)
Gt; image processing core. ≪ / RTI >
13. The method of claim 12,
Wherein the pixel feature vector (PFV) is characterized by maintaining the result of the comparing step between AT and DV entries as a binary value.
The method according to claim 1,
The feature summing block
At least one integral vector calculator for calculating an integral vector IV;
At least one address calculator for calculating an internal RAM address in accordance with a feature calculation request;
At least one feature calculation request FIFO storing a feature calculation request; And
At least one internal RAM < RTI ID = 0.0 > (PFV) < / RTI &
Gt; image processing core. ≪ / RTI >
15. The method of claim 14,
Wherein the feature summing block is characterized by receiving boundary coordinates of a feature calculation request as pixel values to calculate the remaining coordinates to divide the region into quadrants, which are four identical sub-regions.
15. The method of claim 14,
Characterized in that the feature summing block is characterized by reading the pixel feature vector (PFV) from an external memory and writing an internal RAM to make the calculation faster.
15. The method of claim 14,
The sub-block of the integral vector calculator
Reading the pixel feature vector (PFV) from the internal RAM; And
Calculating the quadrant integral vector QIV by adding all the entries of the previous PFV both in the horizontal and vertical dimensions
Further comprising: an image processing unit for processing the image data;
18. The method of claim 17,
Characterized by making all values of the first pixel feature vector "0" to omit a subtraction operation between a quadrant integration vector and a first pixel feature vector.
The method according to claim 1,
The classification block
At least one C matrix row arbiter for controlling multiplication of a C matrix row and an FV; And
At least one multiplication and addition operator for implementing a matrix vector multiplication operation
Gt; image processing core. ≪ / RTI >

Images