CN110210346A - A kind of optimization method that video template matching is handled in real time - Google Patents

Abstract

The invention discloses an optimization method for real-time processing of video template matching. The method includes the following steps: reading and storing pixel value data of a template image; constructing a sliding window with the same size as the template image T, and using the sliding window to adjust the background image Carry out slide sampling, each slide corresponds to get a sub-image, read the pixel value data of the sub-image, and store and update the pixel value data of the sub-image in real time; use the minimum error method to obtain the sub-image corresponding to each slide The similarity between the pixel value data of the template image and the pixel value data of the template image, and then the matching position of the template image is obtained. The present invention adopts template matching based on similarity and a global search strategy of traversing background images with sliding windows, and on the basis of ensuring the effect, uses a field programmable gate array to accelerate calculation, and finally realizes real-time video processing.

Description

An Optimal Method for Real-time Processing of Video Template Matching

technical field

The invention belongs to the technical field of image processing, and further relates to an optimization method for real-time processing of video template matching, which can be used to match template images in background images and output matching positions in real time.

Background technique

Template matching is a technology to find a given target template image in an image, which is widely used in the fields of image processing and signal processing, such as image retrieval, image registration, image recognition and so on. However, with the rapid development of science and technology, the information processing capacity of image template matching processing is increasing, and people's demand for real-time image processing is also increasing. In the process of using general-purpose processors to realize real-time image template matching, the image processing speed has become the bottleneck of this technology, and it is difficult to achieve real-time processing for high-definition images. Some excellent matching algorithms can be executed correctly on desktops or workstations, but after they are applied to embedded systems, the algorithms must be simplified due to limitations such as main frequency and storage resources, and the simplified algorithms often perform poorly , and some cannot even be executed. So with the rapid development of hardware, it has become a trend to use Field Programmable Gate Array (FPGA) to solve real-time image processing problems.

Existing implementations of template matching are not ideal in terms of real-time processing, and the main factors affecting the speed are the operations of search calculation and matching. In some schemes, a large amount of image data is stored in order to traverse and search images, and its frequent reading reduces the speed; in other schemes, the design of the calculation part is unreasonable, which limits the clock frequency of the system and affects the overall processing speed. In addition, a large amount of storage and computing will also occupy a large amount of hardware resources, increasing the cost of implementation.

Contents of the invention

In order to solve the above problems, the purpose of the present invention is to propose an optimization method for real-time processing of video template matching. The present invention comprehensively considers the characteristics of the matching effect and field programmable gate array that the calculation speed is fast but not good at complex logic. The present invention adopts The template matching based on similarity and the global search strategy of traversing the background image through the sliding window, on the basis of ensuring the effect, use the field programmable gate array to accelerate the calculation, and finally realize the real-time processing of the video.

In order to achieve the above object, the present invention adopts the following technical solutions.

An optimization method for real-time processing of video template matching, comprising the following steps:

Step 1. Set the size of the template image T as m×n, and the size of the background image G as M×N; read and store the pixel value data ref[m×n] of the template image.

Step 2, construct a sliding window A with the same size as the template image T, that is, the size of the sliding window A is m×n; the sliding window A is used to perform sliding sampling on the background image G, and each sliding corresponds to a pair of m×n , read the pixel value data img _p [m×n] of the sub-image, and store and update the pixel value data img _p [m×n] of the sub-image in real time.

Among them, img _p [m×n] represents the sub-image pixel value data obtained by the p-th slide.

Step 3: Use the minimum error method to obtain the similarity between the pixel value data img _p [m×n] of the sub-image corresponding to each slide and the pixel value data ref[m×n] of the template image, and then obtain the template image’s Match the position to complete the matching of the template image and the background image.

Compared with prior art, the beneficial effect of the present invention is:

The invention is based on the processing of the template image and the background image by the programmable logic array, and improves the speed of the whole processing by designing a reasonable data storage and update method in the realization process, and saves the resources of the field programmable gate array. The basic idea is: perform a global search on the background image through the sliding window, and then use the similarity calculation to match the template image; by using multiple memories to store and update the data of the sub-image obtained by the sliding window, the data storage is accelerated. With the update speed, resources are saved, and the matching position of the final output template image is achieved.

Description of drawings

The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

FIG. 1 is a schematic flowchart of an optimization method for template matching real-time processing provided by the present invention.

FIG. 2 is a schematic diagram of a sliding process of a sliding window.

Detailed ways

The embodiments and effects of the present invention will be further described in detail below in conjunction with the accompanying drawings.

Step 1, set the size of the template image T as m×n, and the size of the background image G as M×N; read and store the pixel value data ref[m×n] of the template image;

The pixel value data of the m×n template image is read from the external memory. Since the template image is fixed and relatively small for a matching process, the pixel value data of the template image is stored in In the register, the pixel value data of the template image is recorded as ref[m×n].

Step 2, construct a sliding window A with the same size as the template image T, that is, the size of the sliding window A is m×n; the sliding window A is used to perform sliding sampling on the background image G, and each sliding corresponds to a pair of m×n sub-image, read the pixel value data img _p [m×n] of the sub-image, and store and update the pixel value data img _p [m×n] of the sub-image in real time;

The sliding sampling is as follows: the sliding window A starts from the upper left corner of the background image G, slides from left to right and from top to bottom, and slides once every clock cycle, and slides one pixel each time.

The pixel value data img _p [m×n] of the read sub-image is specifically:

When the sliding window A slides to the leftmost of each line of the background image, that is, when the coordinates of the upper left corner of the sliding window A in the background image G are (0, y), after m clock cycles, the background image is read The leftmost m×n pixel value data;

When the sliding window A is not located at the leftmost of each row of the background image, the read data consists of (m-1) columns of pixel value data at the previous sliding position, plus the last column of pixel value data at the current sliding position, that is Each slide corresponds to an n-row m-column pixel value data;

The pixel value data img _p [m × n] of the sub-image is stored in real time, and its specific steps are:

For the pixel value data obtained by sliding each time, the first (n-1) rows are correspondingly stored in (n-1) memories, and each row uses one memory; then each clock cycle from (n-1) memories Read the (n-1) pixel value data of the previous sliding position as the first (n-1) data of the last column of the current sliding position; read a data of the current sliding position from the external memory as the current The nth data of the last column of the sliding position, that is, the last column of data of the current sliding position.

The pixel value data img _p [m × n] of the sub-image is updated in real time, and its specific steps are:

Data update in memory: first, when the sliding window slides one pixel to the right, the data at address 0 in the memory is updated, and the data at address 0 in the second row is written to address 0 in the first row to complete the The update of the first row of data, and so on, write the data at the address 0 of the (n-1) row to the address 0 of the (n-2) row, and complete the update of the (n-2) row data ;Secondly, the data at the nth row address read from the external memory is written to the (n-1) row address at 0, and the update of the (n-1) row data is completed; finally, when the sliding window Slide one pixel to the right again to update the data at the address 1 in the memory, write the data at the address 1 in the second row to the address 1 in the first row, and complete the update of the data in the first row, and so on , write the data at the (n-1) row whose data address is 1 to the (n-2) row whose address is 1, and complete the update of the (n-2) row data; secondly, read from the external memory The data at the address 1 of the nth row is written to the address 1 of the (n-1)th row, and the update of the (n-1)th row data is completed. By analogy, the update of data in the memory is completed each time the sliding window slides.

Store the pixel value data of m×n background images in the current window A in the register each time; when the sliding window slides, update the data in the register correspondingly.

In this way, when the sliding window A moves down, there is already updated data in the memory, and you only need to repeat the previous operation until the sliding window A slides to the lower right corner of the background image, that is, the bottom right corner of the sliding window A. The coordinates of the lower right corner are consistent with the coordinates of the lower right corner of the entire image, that is, the sliding matching process is completed.

Step 3: Use the minimum error method to obtain the similarity between the pixel value data img _p [m×n] of the sub-image corresponding to each slide and the pixel value data ref[m×n] of the template image, and then obtain the template image’s Match the position to complete the matching of the template image and the background image.

The main idea of similarity-based template matching is to calculate the similarity between the template image and the area data covered by the sliding window in the image to be matched according to different similarity functions. Comprehensively considering the template matching effect and calculation complexity, the present invention uses the minimum error method to calculate the similarity.

The main idea of the minimum error method is to represent their correlation according to the difference between the template image and the area image covered by the sliding window. After the sliding window traverses all the pixels in the background image, the area with the smallest difference is the position of the template image in the background image. Common minimum error functions are variance, mean square error, mean absolute error, and absolute error. The calculation of the minimum error method mainly involves addition and subtraction, and rarely uses multiplication and division. Therefore, the minimum error method has the characteristics of simple implementation and small amount of calculation.

Each time the sliding window is slid, the similarity between the sub-image and the template image at the current sliding window position is obtained, and the similarity corresponding to the current sliding window position is compared with the similarity corresponding to the previous sliding window position. The position with higher similarity is is the position of the currently matched template image; when the sliding window traverses the background image, the position of the sliding window with the highest similarity to the template image is obtained, which is the matching position of the template image;

Among them, the similarity is represented by the sum of absolute errors, and the higher the similarity, the smaller the sum of absolute errors.

The minimum error function adopted in the present invention is the sum of absolute errors (SAD).

The formula for calculating the absolute error sum function is: G(i,j) is the pixel value of the background image in the sliding window, T(i,j) is the pixel value of the template image, |·| is the absolute value symbol, (i,j) represents the pixel coordinates in the image , D is the absolute error sum of the background image and the template image.

That is to subtract the pixel values at the corresponding positions of the template image and the background image, and then take the absolute value of the difference, and finally, add the m×n absolute values to obtain the absolute error sum of the two images, the smaller the absolute error sum , indicating that the greater the similarity, the position with the smallest absolute error sum is the matching template image position.

The present invention adopts DSP48E1 to calculate the absolute error sum of background image and template image, DSP48E1 is a kind of field programmable gate array hardware resource, it is a digital signal processing unit, can realize operations such as multiplication, addition, multiplication and addition, uses this resource to carry out calculation Additional resources of the FPGA can be saved. There are more addition and subtraction operations when calculating the similarity, and DSP48E1 can be used to save resources.

Using DSP48E1 to calculate the absolute error sum of the background image and the template image, the specific calculation process is:

First, calculate the corresponding difference: Subtract the corresponding pixel values of the template image and the sub-image of the background image; where, each pixel value is 8-bit data, and DSP48E1 can perform addition and subtraction of 48-bit data, four pixels Values are combined and subtracted, that is, the difference between four pixel values is calculated each time; then add 0 before the subtrahend for borrowing, and add 0 before the subtrahend, so that each data becomes 9 digits, and the combination is 36 number of digits, and then call DSP48E1 primitives to perform subtraction, and after the calculation is completed, divide the corresponding digits of the difference and restore them to four pixel values; traverse the background image to obtain the correspondence between the sub-images of each background image and the template image difference;

Secondly, the absolute value is taken for each corresponding difference value: the highest bit of the corresponding difference value is a sign bit, and the lower 8 bits are data; whether the highest bit of the corresponding difference value is borrowed is judged, if the highest bit is 0 , then borrow, indicating that the corresponding difference is a negative number, then invert the lower 8 bits into a positive number to obtain the absolute value of the corresponding difference; otherwise, the lower 8 bits of the corresponding difference are the corresponding the absolute value of the difference;

Finally, the absolute values of all corresponding differences are accumulated and summed to obtain the absolute error sum of the background image and the template image;

Specifically: first sum the absolute values of all corresponding differences in pairs to obtain the first-level summation result, and then sum the first-level summation results in pairs to obtain the second-level The summation result, and so on, is the sum of the absolute values of all corresponding differences, which is the absolute error sum of the background image and the template image. The above-mentioned accumulation and summation of all absolute values can also be performed by using DSP48E1.

The above matching process can also be expressed as: each slide corresponds to the current SAD value, compare the current SAD value with the SAD value with high similarity obtained in the previous slide, and use the smaller SAD value as the current SAD value with high similarity. SAD value, when the sliding ends, the smallest SAD value is the SAD value with the highest similarity, and the position corresponding to this window is the matching position, that is, the matching between the template image and the background image is completed.

The above steps are performed in a pipelined manner, and a SAD value can be obtained in each clock cycle. After the background image is input, the SAD values of all positions have been traversed, and the comparison of all SAD values is completed, and the matching position is output. At the same time, the final clock frequency can reach 100MHz, and the matching position of each frame of image can be quickly obtained, that is, the image can be processed in real time.

All or part of the steps of the present invention can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. The medium includes: various mediums capable of storing program codes such as ROM, RAM, magnetic disk or optical disk.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (8)

1. an optimization method for video template matching real-time processing, is characterized in that, comprises the following steps: Step 1, set the size of the template image T as m×n, and the size of the background image G as M×N; read and store the pixel value data ref[m×n] of the template image; Read the pixel value data of the template image T from the external memory, and store it in the register; Step 2, construct a sliding window A with the same size as the template image T, that is, the size of the sliding window A is m×n; the sliding window A is used to perform sliding sampling on the background image G, and each sliding corresponds to a pair of m×n sub-image, read the pixel value data img _p [m×n] of the sub-image, and store and update the pixel value data img _p [m×n] of the sub-image in real time; Among them, img _p [m×n] represents the sub-image pixel value data obtained by the p-th slide; Step 3: Use the minimum error method to obtain the similarity between the pixel value data img _p [m×n] of the sub-image corresponding to each slide and the pixel value data ref[m×n] of the template image, and then obtain the template image’s Match the position to complete the matching of the template image and the background image. 2. the optimization method of a kind of video template matching real-time processing according to claim 1, is characterized in that, described sliding sampling is: sliding window A starts from the upper left corner of background image G, from left to right, from top to bottom Slide down and down in turn, slide once per clock cycle, and slide one pixel at a time. 3. the optimization method of a kind of video template matching real-time processing according to claim 1, is characterized in that, the pixel value data img _p [m × n] of described reading sub-image is specifically: When the sliding window A slides to the leftmost of each line of the background image, that is, when the coordinates of the upper left corner of the position of the sliding window A in the background image G are (0, y), after m clock cycles, the background image is read The leftmost m×n pixel value data; When the sliding window A is not located at the leftmost of each row of the background image, the read data consists of (m-1) columns of pixel value data at the previous sliding position, plus the last column of pixel value data at the current sliding position, that is Each slide corresponds to an n-row m-column pixel value data. 4. the optimization method of a kind of video template matching real-time processing according to claim 3, is characterized in that, described pixel value data img _p [m × n] of sub-image is stored in real time, and its specific steps are: For the pixel value data obtained by sliding each time, the first (n-1) rows are correspondingly stored in (n-1) memories, and each row uses one memory; then each clock cycle from (n-1) memories Read the (n-1) pixel value data of the previous sliding position as the first (n-1) data of the last column of the current sliding position; read a data of the current sliding position from the external memory as the current The nth data of the last column of the sliding position, that is, the last column of data of the current sliding position. 5. the optimization method of a kind of video template matching real-time processing according to claim 4, is characterized in that, described pixel value data img _p [m × n] of sub-image is updated in real time, and its concrete steps are: Data update in memory: first, when the sliding window slides one pixel to the right, the data at address ₀ in the memory is updated, and the data at address 0 in the second row is written to address 0 in the first row to complete the The update of the first row of data, and so on, write the data at the address 0 of the (n-1) row to the address 0 of the (n-2) row, and complete the update of the (n-2) row data ;Secondly, the data at the address of the nth row read from the external memory is 0 and written to the address of the (n-1) row at 0, and the update of the data in the (n-1) row is completed; finally, when the sliding window Slide one pixel to the right again to update the data at the address 1 in the memory, write the data at the address 1 in the second row to the address 1 in the first row, and complete the update of the data in the first row, and so on , write the data at the address 1 of the (n-1) row to the address 1 of the (n-2) row, and complete the update of the (n-2) row data; secondly, read from the external memory Write the data at the address of the nth row of 1 to the address of the (n-1)th row at 1, and complete the update of the (n-1)th row of data; and so on, complete the data in the memory of each sliding window sliding renew; Store the pixel value data of m×n background images in the current window A in the register each time; when the sliding window slides, update the data in the register correspondingly. 6. the optimization method of a kind of video template matching real-time processing according to claim 1, is characterized in that, the minimum error function that described minimum error method adopts is absolute error sum, and its computing formula is: Among them, G(i, j) is the pixel value of the background image in the sliding window, T(i, j) is the pixel value of the template image, |·| is the absolute value symbol, (i, j) represents the Pixel coordinates, D is the sum of absolute errors between the background image and the template image. 7. the optimization method of a kind of video template matching real-time processing according to claim 1, is characterized in that, described respectively obtains the pixel value data img _p [m × n] of the sub-image corresponding to each slide and the template image The similarity of the pixel value data ref[m×n], and then the matching position of the template image is obtained. The specific steps are: Each time the sliding window is slid, the similarity between the sub-image and the template image at the current sliding window position is obtained, and the similarity corresponding to the current sliding window position is compared with the similarity corresponding to the previous sliding window position. The position with higher similarity is is the position of the currently matched template image; when the sliding window traverses the background image, the position of the sliding window with the highest similarity to the template image is obtained, which is the matching position of the template image; Among them, the similarity is represented by the sum of absolute errors, and the higher the similarity, the smaller the sum of absolute errors. 8. the optimization method of a kind of video template matching real-time processing according to claim 1, is characterized in that, adopts DSP48E1 to calculate the absolute error of background image and template image and, its concrete calculation process is: First, calculate the corresponding difference: Subtract the corresponding pixel values of the template image and the sub-image of the background image; where, each pixel value is 8-bit data, and DSP48E1 can perform addition and subtraction of 48-bit data, four pixels Values are combined and subtracted, that is, the difference between four pixel values is calculated each time; then add 0 before the subtrahend for borrowing, and add 0 before the subtrahend, so that each data becomes 9 digits, and the combination is 36 number of digits, and then call DSP48E1 primitives to perform subtraction, and after the calculation is completed, divide the corresponding digits of the difference and restore them to four pixel values; traverse the background image to obtain the correspondence between the sub-images of each background image and the template image difference; Secondly, the absolute value is taken for each corresponding difference value: the highest bit of the corresponding difference value is a sign bit, and the lower 8 bits are data; whether the highest bit of the corresponding difference value is borrowed is judged, if the highest bit is 0 , then borrow, indicating that the corresponding difference is a negative number, then invert the lower 8 bits into a positive number to obtain the absolute value of the corresponding difference; otherwise, the lower 8 bits of the corresponding difference are the corresponding the absolute value of the difference; Finally, the absolute values of all corresponding differences are accumulated and summed to obtain the absolute error sum of the background image and the template image; Specifically: first sum the absolute values of all corresponding differences in pairs to obtain the first-level summation result, and then sum the first-level summation results in pairs to obtain the second-level The summation result, and so on, is the sum of the absolute values of all corresponding differences, which is the absolute error sum of the background image and the template image.