JPH0969159A - Template matching device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a template matching device eliminating the necessity of increase of operation frequency even when a retrieving area is extended and operating by a low power consumption by forming the device so that the supply of picture element data in a retrieving area from a picture storage means to an arithmetic means can be executed on the same chip. SOLUTION: The template matching device is constituted of one chip mounting a picture memory 108, an operation cell array 102, a result storing part 105, and an I/O control part 106 on its surface. The picture memory 108 consists of plural line memories 101 and stores the picture element data of one picture in each line. The I/O control part 106 controls the I/O of data from/to an external bus 107 and the cell array 102 consists of plural operation cells 111 to 114. Since picture element data equivalent to one screen are stored in the memory 108 and picture data in a retrieving area are transferred in the same chip, the necessity of transfer of retrieving data through the external bus 107 can be eliminated, and even when the number of picture elements in the retrieving area is increased, the increase of operation frequency is unnecessary, reducing the power consumption.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a template matching device used in a tracking system or the like in digital image processing, and in particular, processing can be performed in a short time.
In addition, the present invention relates to a template matching device that consumes less power.

[0002]

2. Description of the Related Art In digital image processing such as graphic recognition, pixel data of a predetermined template (template) and pixel data of a search area (hereinafter referred to as search data)
A template matching method is often used, in which the similarity is determined based on some criterion and the region closest to the template is detected from the search region. Also in moving images, this template matching method is often used to detect the moving distance or moving position of a moving object on the screen.

The template matching method will be described below by taking as an example the case of detecting a portion that is substantially the same as the template (for example, the image at the present time) from the search region (for example, the image that is previous in time). Note that the template is an image region 20 of n rows × n rows as shown in FIG.
1. The search area is (n + k) rows × as shown in FIG.
It is assumed that the image area 203 has (n + k) rows. In addition, the pixel 2 at the i-th row and the j-th column in the template image 201
02 is m _i , _j , and the pixel 204 in the g-th row and the h-th column in the search area
Be S _g , _h .

In the search area 203 of FIG. 3B, (k
+1) ^two n rows × n rows of regions (hereinafter, although referred to as candidate blocks) exist, those that the pixel 204 of the g th row and column h of the candidate block and the pixel in the upper left corner of CB _{Expressed as gh} .

In the template matching method, each candidate block 205 is

[0006]

[Equation 1]

The absolute value error sum D _g , _h defined by the above is evaluated, and the smallest candidate block of D _g , _h is selected as the one closest to the template 201.

The calculation of the equation (1) requires n ² absolute value error calculations and additions to obtain D _g , _h for one candidate block. Therefore, in the template matching process, the calculation amount of order O (n ² k ² ) is required in order to obtain D _g , _h for (k + 1) ² candidate blocks. As described above, since the template matching processing requires a large amount of calculation, a template matching device that performs parallel processing is usually used for real-time processing of moving images.

FIG. 11 is a block diagram showing the structure of a conventional template matching device (a block diagram when the number of pixels of the template is four). In this template matching device, the image data in the image memory 702 is transferred to the external bus 704 based on the operation of the input control unit 701.
Then, the data is supplied to the calculation cells 711, 712, 721, 722 and the register 731, and the calculation is performed. And
The result is sent to the result storage unit 704, and the result is output based on the operation of the output control unit 705.

FIG. 12 is a diagram for explaining an example of a template matching method using the template matching device 700 of FIG. In FIG. 12, image data I ₁ and I ₂ are image data, and are sent from the outside through the external bus 704 in the order of I ₁ and I ₂ . Here, a part of the image data I ₂ is used as a template, and the image data I ₁
A case will be described where template matching is performed using a part of the above as a search area.

First, the image memory 751 stores the image I ₁ sent through the external bus (step 1).

Next, the image memory 752 stores the image I ₂ sent through the external bus (step 2).

Then, the template data is preloaded from the image memory 752 to the template matching device 700 through the external bus 704 (step 3).

Data in the search area for the preloaded template is transferred from the image memory 751 to the external bus 704.
Through the template matching device 700. The template matching device 700 calculates the portion of the search area that most resembles the template (step 4).

According to the above steps 1 to 4, 1
Template matching processing is performed for one template. When the template matching is performed on all the image data I _2, it is necessary to repeat the above steps 3 to 4.

Next, the template matching device 700
The internal processing of will be specifically described. Here, FIG.
The input image (current image) 401 shown in FIG.
It is assumed that template matching is performed with the reference image (image temporally previous) 501 shown in FIG.
1 is divided into blocks of 2 pixels × 2 pixels, and each block is used as a template for processing. At this time,
Since 20 blocks are obtained for the input image 401, template matching is executed 20 times. In addition, in order to simplify the description, FIG.
The processing of the screen 401 having horizontal 10 pixels × vertical 8 pixels as shown in (a) will be described. Generally, the screen used has a larger number of pixels, for example, one screen in the NTSC standard, which is one standard of television. Is horizontal 720 pixels x vertical 4
It has 80 pixels.

First, the process when a certain block of interest 402 (see FIG. 4A) in the input image 401 is a template will be described. Here, the search area is the reference image 5
In FIG. 5, a region 502 in which one pixel is added to the block in the same portion as the template 402 in the vertical and horizontal directions (see FIG.
(See (a)). Also, template 4
02 and each pixel of the search area 502 are respectively shown in FIG.
(B), as shown in FIG. 5 (b), m _{11 to} m ₁₂ , s _{11 to} s
₄₄ .

FIG. 6 shows nine ((2
+1) is a diagram showing a candidate block CB ₁₁ to CB ₃₃ for ^two). In the template matching method, the nine blocks CB _{11 to} CB ₃₃ are compared with the template 402,
A candidate block having the smallest absolute value error sum defined by Expression (1) is obtained. 13 to 18 are diagrams showing the arithmetic processing steps, and in these drawings, the arithmetic cells 711 and 71 are shown.
2, 721, 722, register 731, result storage unit 70
The states of No. 4 at the same time are shown side by side.

First, at time t = 1 (see FIG. 13), the pixel data (hereinafter referred to as template data) m ₁₁ of the template 402 is input to the operation cell 722 in the template matching device according to the instruction of the input control unit 701. To be done. Then, at time t = 2, 3, 4, m ₂₁ , m ₁₂ , and m ₂₂ are sequentially input into the template matching device. At this time, the previously input template data is sequentially transferred to the adjacent operation cells 712, 721, 711, and t = 4.
In, all template data of the template 402 is stored in the operation cells 711, 721, 712 and 722, one by one.

After time t = 5 in FIG. 14, FIG.
The pixel data s _{11 to} s _{44 of} the search area 502 shown in (4) are input to the operation cell 722 via the input control unit 701. At this time, each time new data is input to the operation cell 722, the search data is transferred from each operation cell to the adjacent operation cell, but in the case of transfer of the search data, the operation cell 71
After 2 is temporarily held in the register 731 (time t = 7
(Reference) to the arithmetic cell 721 (see time t = 8). This is data buffering for adjusting the combination of the template and the search area, and the required number of registers is determined by the vertical pixel difference between the template and the search area. Until time t = 8 in FIG. 14, only the search data is loaded, and no substantial calculation is performed.

Next, the calculation is started from time t = 9 shown in FIG. It should be noted that, in FIGS. 15 to 18, the operation cells that are active are shown in ^a practice frame, and d _{ab in} the figure is | s _b −.
The calculation result of m _a | is shown.

The calculation cell 721 at time t = 9 stores the template data m ₂₁ and the search data s ₂₁ in the register inside the cell, and the absolute value error d ²¹ ₂₁ = | s ₂₁ -m
₂₁ | is calculated. Similarly, the calculation cell 711 calculates the absolute value error d ¹¹ ₁₁ = | s ₁₁ −m ₁₁ |.

At time t = 10, the operation cells 721, 71
The calculation result of 1 is transferred to the calculation cells 722 and 712, respectively. Then, the operation cell 722 receives the absolute value error d obtained from m ₂₂ and s ₂₂ stored in its own register.
²² ₂₂ = | s ₂₂ −m ₂₂ |, and then the operation cell 72
It calculates a sum d ^{₂₁ 21} + d ^{₂₂ 22} a d ²¹ ₂₁ transferred from 1. Similarly, the calculation cell 712 has a sum of absolute value errors d ¹² ₁₂
= | Computes the sum d ^{₁₁ 11} + d ^{₁₂ 12} a d ¹¹ ₁₁ transferred from the operation cell 711 on which the operation was performed in | s ₁₂ -m _12.

At time t = 11, the arithmetic cell 7
The calculation results of 11, 712 are transferred to the result storage unit 704. Then, they are added and the absolute value error sum D ₁ , ₁
Is obtained. This absolute value error sum D ₁ , ₁ is the candidate block CB ₁₁ shown in FIG. 6 and the template 4 shown in FIG.
It is the sum of absolute value error with 02.

Operation cells 711, 721, 712, 72
2. In the result storage unit 704, the processing similar to the above is performed after the time t = 11, and the absolute value error sum for each candidate block is obtained as shown in FIGS. Then, at t = 21, all the absolute value error sums are obtained. By comparing these absolute value error sums, the candidate block closest to the template 402 can be known.

At t = 22 to 25 shown in FIG. 18, the operation for discharging the search data in the operation cells 711, 721, 712, 722 is performed, and the operation is not executed.

The template matching with the template 402 is performed by the above-described processing of FIGS. Therefore, for the entire input image 401 of FIG. 4A, the processing of FIGS.
It is necessary to do it 0 times.

[0028]

In the template matching device 700 shown in FIG. 11, the arithmetic cells 711 to 711 are used.
Since 22 executes operations in parallel at the same time, many operations can be executed at the same time, which is suitable for template matching with a large amount of operations.

However, in the above template matching apparatus, since the search data is sequentially supplied to the arithmetic cells through the external bus, there is a problem that a large amount of information passes through the external bus. For example, the number of pixels in one screen is p pixels in the horizontal direction, q pixels in the vertical direction, the size of the template is n pixels × n pixels, and the size of the search area is (n + k) pixels ×
Assuming that (n + k) pixels, in template matching for the entire screen, (n + k) ² × {(p / n) ×
(Q / n)} = (n + k) ² pq / n ² data needs to be transferred through the external bus. Generally, since the processing time per screen is constant, the larger the number of pixels in the screen and the number of pixels in the search area, the higher the operating frequency needs to be in order to perform real-time processing. Will increase.

The present invention has been made to solve the above problems, and an object of the present invention is to provide a template matching device that operates with low power consumption without the need to raise the operating frequency even if the search area is widened. And

[0031]

A template matching device according to a first aspect of the present invention comprises an image storage means for storing all pixel data of a reference image and an arithmetic unit having a number of pixels equal to or more than the number of pixels in a search area. Each pixel data of the search area supplied from the means is held by a different arithmetic unit, and the pixel data of the template is sequentially introduced to perform arithmetic operation for template matching processing. In other words, the pixel data of the search area can be supplied from the image storage means to the calculation means in the same chip.

In such a configuration, all the pixel data of the reference image is stored in the template matching device, so the only data that needs to be transferred through the external bus is the template pixel data. Then, since the pixel data in the search area is transferred in parallel from the image memory to the arithmetic unit using a plurality of transfer paths in the same chip, it is not necessary to increase the operating frequency even if the search area is widened, and the power consumption is reduced. Can be reduced.

A template matching apparatus according to a second aspect is the template matching apparatus according to the first aspect, wherein the image storage means comprises a plurality of row storage means for storing all pixel data of the reference image row by row. is there.

In this configuration, the row storage means stores the pixel data of the reference image row by row, and the pixel data can be read in parallel from the row storage means. In the data transfer in the chip, it is possible to omit the transfer of the pixel data which overlap in different search areas. Therefore, the calculation can be performed in a shorter time.

The template matching device according to a third aspect is the template matching device according to the second aspect, in which the image storage means and the arithmetic means are formed separately on a plurality of chips, and each chip is formed. At least a plurality of row storage means and a number of arithmetic units equal to or larger than the number of pixels in the search area are formed.

In the above configuration, even if the template matching device is divided into a plurality of chips, the data transfer from the image storage means to the calculation means can be performed within the same chip. Therefore, even when the number of pixels of the reference image is large and the arithmetic unit and the image storage unit cannot be integrated in one chip, it is possible to operate with low power consumption.

[0037]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the template matching apparatus of the present invention will be described below.

FIG. 1 is a block diagram showing an example of the configuration of the template matching apparatus of the present invention (a block diagram showing an example of the configuration when the number of pixels in the search area is 16 or less). This template matching device is composed of one chip, and includes an image memory (image storage means in the claims) 108, an operation cell array (operation means in the claims) 102, a result storage section 105, and an input / output control section 106. It consists of The image memory 108 is a plurality of row memories (row storage means in claims) 101.
And stores the pixel data of one screen in row units. The input / output control unit 106 controls input / output of data with the external bus 107. Arithmetic cell array 1
Reference numeral 02 denotes an arithmetic cell (an arithmetic unit in claims) 11
It consists of 1-144.

FIG. 2 is a diagram for explaining an example of a template matching method using the template matching device having the above structure. In FIG. 2, I ₁ and I ₂ are image data, which are sent from the outside through the external bus 107 in the order of I ₁ and I ₂ . Hereinafter, a part of the image data I ₂ is used as a template, and a part of the image data I ₁ is used as a search area.
A case where template matching is performed using 0b will be described.

First, the data of the image data I ₁ sent through the external bus 107 is stored in the image memory 108a via the input / output control unit 106a (step 1).

Next, the image data I ₂ is stored in the image memory 108b via the input / output control unit 106b (step 2).

Template (part of image data I ₂ )
Search data (a part of the image data I ₁ ) corresponding to
Based on the instruction from the input / output control unit 106a, the image memory 108a of the template matching apparatus 100a is preloaded into the arithmetic cell array 102a (step 3).

The input / output control unit 10 receives the template data.
6a and 106b, the template matching device 100b is connected to the external bus 1 from the image memory 108b.
It is supplied to the template matching apparatus 100a via 07, and the part of the search area most similar to the template is calculated in the operation cell array 102a of the template matching apparatus 100a (step 4).

According to the above steps 1 to 4, 1
Template matching processing is performed for one template. When performing template matching on the entire screen of the image data I ₂ , the above step 3 is performed.
~ Step 4 may be repeated. At this time, in the present embodiment, the image memory 108 is composed of a plurality of row memories 101, and data is stored in row units. Therefore, in the search data preload in step 3, the preload of the portion overlapping with other search data is omitted. It becomes possible to do. Therefore, the calculation can be performed in a shorter time.

In this embodiment, the amount of information (number of pixels) that needs to pass through the external bus 107 for calculation is
Since the number of pixels needs to be transferred from the template matching device 100b to the template matching device 100b, one screen has p pixels in the horizontal direction and q pixels in the vertical direction.
If the template is n pixels × n pixels, (n × n) ×
{(P / n) × (q / n)} = pq pixels, which does not depend on the number of pixels in the search area ((n + k) × (n + k) pixels) as in the conventional case (conventional: pq + (n + k)) ² pq / n ²
(Pixel), it is possible to reduce the amount of information that needs to pass through the external bus 107. Therefore, power consumption can be reduced.

Next, the internal processing of this template matching device will be specifically described. Here, for simplification, a case where the same processing as the processing described in the conventional example is performed will be described. That is, a case will be described where template matching is performed between the 10 pixel × 8 pixel input image shown in FIG. 4 and the 10 pixel × 8 pixel reference image shown in FIG. Here, it is assumed that the template has 2 pixels × 2 pixels and the search area has 4 pixels × 4 pixels.

First, the template is set to 402 in FIG.
Then, a case will be described in which the template matching is performed as the search region as a region 502 (see FIG. 5A) in which one pixel is added to the block in the same portion as the template 402 in the reference image 501 in the vertical and horizontal directions. In addition, each pixel of the template 402 is set to m as shown in FIG.
_{11 to} m ₂₂ and each pixel of the search area 502 is set to s ₁₁ to s ₄₄ as shown in FIG. 5B.

FIG. 6 is a diagram showing nine ((2 + 1) ² ) candidate blocks CB _{11 to} CB ₃₃ in the search area 502. In the template matching method, this 9
The blocks 402 CB _{11 to} CB ₃₃ are compared with the template 402 to obtain a candidate block having the smallest absolute value error sum defined by the equation (1).

7 to 9 are diagrams for explaining the process of the template matching device of FIG. The operation of the template matching device will be described below with reference to these figures. Note that in FIGS. 8 9, d ^a _b template data m _a and the absolute value error between the search data s _b | s _b -m _a | indicates a, D _{g, h} are shown in Formula (1) And the absolute value error sum (absolute value error sum with the candidate block having the upper left corner at the g-th row and the h-th column).

At time t = 1 to 4, arithmetic cells 111 to 1
The search data is supplied to 44. First, time t =
1, the search data s ₁₁ ,
Operation cells 11 in rows corresponding to s ₂₁ , s ₃₁ , and s ₄₁
4,124,134,144. Then, at time t = 2, the arithmetic cells 114, 124, 1
The data inside 34 and 144 are the calculation cells 113 and
123, 133, 143, and the operation cells 114,
Search data s12, s22, s32, and s42 are input from the image memory 108 to 124, 134, and 144, respectively. Similarly, at times t = 3 and 4, the data transfer to the adjacent operation cells and the operation cells 114, 124 and 13 are performed.
New data is input to 4,144. Then, at time t = 4, all the search data are set in one operation cell.

Then, at time t = 5 shown in FIG.
The pixel data m ₁₁ of the template 402 is supplied to the calculation cells 111 to 114 and 131 to 134, and the calculation cell 12
1 to 124 and 141 to 144 have a template 402.
Pixel data m ₂₁ of Then, the arithmetic cell 11
The absolute value error is calculated in the operation cells other than 4,124,134,144. For example, in the arithmetic cell 111,
Using the template data m ₁₁ and the search data s ₁₁ , d ¹¹
₁₁ = | s ₁₁ −m ₁₁ | is calculated. 8 and 9, active arithmetic cells (arithmetic cells in which arithmetic operations are performed) are surrounded by solid lines.

At time t = 6, the arithmetic cells 111 to 114
And 131 to 134 are supplied with the template data m ₁₂ , and the arithmetic cells 121 to 124 and 141 to 144 are supplied with the template data m ₂₂ . Then, the absolute value error is calculated in the operation cells other than the operation cells 111, 121, 131, 141, and the sum of the calculation result and the operation result at t = 5 transferred from the operation cell on the left of each operation cell. Is calculated.

Further, at time t = 7, the calculation result at t = 6 is transferred to the calculation cell on the right side. At this time, the calculation result is transferred from the calculation cells 114, 124, 134, 144 at the right end to the result storage unit 105.
Then, in the result storage unit 105, the absolute value error sums D ₁ , ₃ and D ₃ , _{3 of} the candidate blocks CB ₁₃ and CB ₃₃ shown in FIG. 6 and the template 402 shown in FIG. 4 are obtained.

Thereafter, by repeating the same procedure,
For all candidate blocks shown in FIG. 6, the absolute value error sum with the template 402 shown in FIG. 4 is the result storage unit 1.
Obtained at 05.

The same processing is basically performed in the active operation cells from time t = 5 to t = 12. For example, at time t = 5, since there is no operation data transferred from the adjacent cell, there is a function necessary for executing the operation processing at time t = 6 without performing significant addition processing. Thus, the process at time t = 5 can be executed.

As described above, according to the present invention, since the image memory 108 and the operation cell array 102 are included in one chip, the search data 502 is supplied to the operation cells 111 to 144 without passing through the external bus 107. It Therefore, it is possible to perform the calculation with low power consumption. However, if the number of pixels of the reference screen increases, it becomes difficult to form the image memory 108 and the operation cell array 102 in one chip.

In the above case, the template matching device may be constructed as shown in FIG. That is, the image memory is divided into a plurality of row memories and formed on different chips (formed on three chips T _{1 to} T _{3 in} FIG. 10). Then, arithmetic cells having the number of pixels in the search region or more are formed on each chip, and the search data is transferred from the image memory to the arithmetic cells in the same chip. With this configuration, it is not necessary to transfer the search data 502 between chips, because only the partial sum for obtaining the absolute value error sum needs to be transferred between chips. Further, even when the search range is increased, the amount of information that needs to be transferred via the external bus does not change, so there is no need to increase the operating frequency when performing real-time processing, and power consumption is reduced. be able to. In FIG. 10, the input / output control unit 106
The result storage unit 105 is omitted.

[0058]

According to the template matching apparatus of the first aspect, the pixel data for one screen is stored in the image memory, and the pixel data in the search area is transferred within the same chip. Therefore, the search is performed via the external bus. No need to transfer data. Therefore, it is not necessary to increase the operating frequency even if the number of pixels in the search area is large, so that power consumption can be reduced.

Further, in the template device according to the second aspect, since the image memory is the row memory, preloading from the image memory to the operation cell array can be efficiently performed, and the processing speed can be increased. .

Further, according to the template matching apparatus of the third aspect, even if the template matching apparatus is divided into a plurality of chips, the data transfer from the image memory to the operation cell is performed in the same chip. can do. Therefore, even when the number of pixels of the reference image is large and the image memory and the operation cell array cannot be configured in one chip, it is possible to operate with low power consumption.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration example of a template matching device of the present invention.

FIG. 2 is a diagram illustrating an example of a template matching method.

FIG. 3 is a diagram illustrating a template and search data.

FIG. 4 is a diagram showing an example of an input image.

FIG. 5 is a diagram showing an example of a reference image.

FIG. 6 is a diagram showing candidate blocks.

FIG. 7 is a diagram illustrating a processing process (t = 1 to 4) of the template matching device in FIG.

FIG. 8 is a diagram illustrating a processing process (t = 5 to 9) of the template matching device in FIG.

FIG. 9 is a diagram illustrating a processing process (t = 10 to 12) of the template matching device of FIG.

FIG. 10 is a block diagram showing a configuration example of a template matching device formed on a plurality of chips.

FIG. 11 is a block diagram showing a configuration of a conventional template matching device.

FIG. 12 is a diagram illustrating an example of a conventional template matching method.

FIG. 13 is a diagram illustrating a processing process (t = 1 to 4) of a conventional template matching device.

FIG. 14 is a diagram illustrating a processing process (t = 5 to 8) of a conventional template matching device.

FIG. 15 is a diagram illustrating a processing process (t = 9 to 13) of a conventional template matching device.

FIG. 16 is a diagram illustrating a processing process (t = 14 to 17) of a conventional template matching device.

FIG. 17 is a diagram illustrating a processing process (t = 18 to 21) of the conventional template matching device.

FIG. 18 is a diagram illustrating a processing process (t = 22 to 25) of the conventional template matching device.

[Explanation of symbols]

101 row memory 102 operation cell array 105 result storage unit 106 input / output control unit 108 image memory 111 to 144 operation cell 402 template 502 search data T _{1 to} T ₃ chips m _{11 to} m ₂₂ template data s _{11 to} s ₄₄ search data

Claims (3)

[Claims] 1. A predetermined calculation is performed on pixel data of a search area and pixel data of a template in a reference image,
In a template matching device for extracting a region closest to the template from the search region, an image storage unit for storing all pixel data of the reference image, and an arithmetic unit having a number of pixels equal to or more than the number of pixels in the search region, While holding each pixel data of the search area supplied from the image storage means in each of the different arithmetic units, the pixel data of the template is sequentially introduced, the arithmetic unit for performing the predetermined calculation, A template matching device, characterized in that it is formed so that the pixel data of the search area can be supplied from the image storage means to the calculation means in the same chip. 2. The template matching device according to claim 1, wherein the image storage means comprises a plurality of row storage means for storing all pixel data of the reference image in units of rows of the reference image. Template matching device. 3. The template matching apparatus according to claim 2, wherein the image storage means and the calculation means are formed separately on a plurality of chips, and each chip has at least a plurality of the row storage means. A template matching device, characterized in that the arithmetic units are formed in a number equal to or larger than the number of pixels in the search area.