JPS62137668A - Picture signal processor using convolution - Google Patents

Abstract

PURPOSE:To find convolution in each picture element data at a relative high speed with a few multipliers by performing the multiplication of a corresponding picture element data shifting it in order against a load coefficient of one line, and estimating a result. CONSTITUTION:One line share out of the load coefficients of N-number of lines and N-number of rows in a coefficient memory 41 is set at N-number of registers 421, 422,..., and also, the picture element data of a frame memory 2 corresponding to a set load coefficient is inputted in order to the first shift register 3, and input controls for all of the lines in the coefficient memory 41 are performed. And the multiplication between the value of the first register 3 and the values of the N-number of registers are found at N-number of multipliers 51, 52,..., and outputs are added at an adder 71. The convolution can be obtained by repeating an operation that the output of the adder 71 is inputted to the second shift register 73, and the output of the shift register 73 and the output of the adder 71 are added with an adder register 72, and returned again to the shift register 73.

Description

DETAILED DESCRIPTION OF THE INVENTION [Field of Industrial Application] The present invention relates to an image signal processing device using convolution. The device according to the invention is used, for example, in connection with visual sensors for identifying objects in industrial robots.

[Prior art and problems to be solved by the invention]

An image output captured by an imaging device (image sensor) such as a CCD camera is converted into digital information as an image signal. In order to sharpen this image, it is necessary to extract the line 2 contour, and convolution is performed on each pixel data for one frame that is the target of image signal processing.

Conventionally, when performing a convolution operation, a plurality of multipliers are used to perform the operation and the integration or addition is performed, which increases the storage capacity of the memory, increases the capacity and size of the image processing device, and inevitably increases the price.

The prior art will be explained using FIGS. 3 and 4.

The first three rows by three columns of convolutions are shown at the top of FIG. Processing target pixel data F (i, j) and weight coefficient W
When (i, j) are arranged as shown in the figure, the convolution calculation result G of the central pixel data F (2, 2)
(2,2) is expressed by the following formula.

For convenience of explanation, we use 匈 (1, 1) X F (1, 1) + W (2, 1) X
F (2, 1) ten (3, 1) x F (3, 1) is calculated in the first column, W (1, 2) XF (1, 2) + K (2
,2) x F(2,2)+K(3,2)xF(3,2
) is the second column operation, W(1,3) xF(1,3
) +W(2,3) xF(2,3) +W(3,3)
XF(3,3) is called the third column operation.

Conventionally, the calculation using the above convolution is shown in Figure 3.
The system is executed by the device shown in FIG. That is, FIG. 3 is used for general calculations, and uses one multiplier and one adder. F(i, j) is input to the first input of the multiplier, W(i, j) is input to the second input in sequence, and F (II
J) xW(i, j) is calculated, and in the case of a 3-row, 3-column convolution, a total of 9 results are integrated using an adder. As a result of integration, if the feature extraction is to be grasped at the central part F (2, 2) of the element arranged in 3 rows and 3 columns, the convolution result G (2, 2) can be easily obtained.

The device shown in Fig. 3 has a small number of components, which can be considered an advantage, but it takes a long time to obtain the result of the composite operation for one pixel data, for example, 256 x 256 pixels. This method has the problem that it is not suitable for calculating convolution operations for a large number of pixels, and that the storage capacity of the memory becomes enormous.

In the example of FIG. 4, a total of nine multipliers Mll to M19 and one adder All are provided, F(i, j) is input to the first input of each multiplier, and F(i, j) is input to the second input of each multiplier. Different W(i
, j) are input, parallel processing operations are performed, and the results are added together using an adder.

According to the apparatus shown in FIG. 4, the convolution operation of one pixel data can be obtained at approximately 1/9 the speed, but this multiplier is large and expensive, and such a multiplier is The apparatus shown in FIG. 4, which uses as many as nine of these, generally has problems in that it is expensive and large in size.

An object of the present invention is to obtain convolution of each of a plurality of pixel data to be processed stored in a frame memory at a relatively high speed using a small number of multipliers, and to further reduce the types of weighting coefficients required for convolution. Based on this idea, it is an object of the present invention to provide an image signal processing device in which the storage capacity of a multiplier RAM is reduced.

[Means and effects for solving the problem] The present invention includes a frame memory that digitally converts and stores an image signal of an image sensor, and a coefficient memory that stores N rows and N columns of weight coefficients in advance. and an image signal processing device that performs convolution of each of a plurality of pixel data to be processed stored in the frame memory using weighting coefficients of the coefficient memory, a first shift register having N stages; N registers in which N weight coefficients are set; N RAM multipliers for multiplying the output of the register corresponding to the output of each stage of the first shift register;
an adder that adds the outputs of the N multipliers; a second shift register having a number of stages equal to the number of pixel data to be processed in the row direction of the frame memory; an output of the adder and an output of the second shift register; an addition register that adds and stores the addition result and adds the addition result to the second shift register;
A load coefficient for one row is set from the coefficient memory to the N registers, and pixel data of the frame memory corresponding to the load coefficient is transferred to the first shift register for all rows of the load coefficient. By the time the last row of weighting coefficients is input, each row of pixel data to be processed is convolved using the addition register output data, and equal weighting coefficients are An image signal processing device is provided that is capable of performing matrix convolution so as not to have duplicate multiplication results.

In the device according to the invention, for example, a 3 row by 3 column convolution is performed as follows. That is,
The first shift register 3 has three stages of registers 31, 32 .
33, and three registers 421 to 423 are used. A total of 25 files are stored in frame memory 2 in a predetermined order.
6×256 pixel data F(x,y) are arranged. The pixel data from the 2nd row to the 254th row is targeted for processing, and a predetermined weighting coefficient W(i, j) is set.

Initially, W (1, 1) is written in the three registers 421 to 423.
.

W(2,1) and W(3,1) are set, F(0,0) is placed in the second stage 32 of the first shift register 3, and F(1,0) is placed in the first stage 33. is set. As a result, the output of the adder 71 of cylinder 1 becomes the first operation result of F (0, 1), and the first row of processing addition register (72) is configured not to add the output of the second shift register 73. Therefore, the first operation result of F(0,1) is input to the second shift register 73.

Next, F is placed in the third stage (33) of the first shift register 3.
(2,0) is input, the contents of the first row are shifted into the second row, and the contents of the second row are shifted into the third row. As a result, the output of the adder 72 becomes the first operation result of F(1,1), which is input to the second shift register 73.

Thereafter, F(3,0) to F(255,0) and finally, for example, O are sequentially input to the first stage of the first shift register 3, so that F(3,0) to F(255,0) is input to the second shift register 73. (
0,1) to F (255,1) are set.

Next, the addition operation of the addition register 72 is started, and W(1, 2) is written to the three registers 421 to 423.

Set W(2,2) and W(3,2), and set F(0,1) to the second stage of the first shift register 3 and F(0,1) to the first stage.
(1, 1) respectively, the output of the first adder 71 becomes the second operation result of F(0, 1), and the addition register 72 outputs F(0, 1), which was set in the second shift register 73. , 1), and this added value is returned to the second shift register (73) again. When such operations are performed on all the pixel data in the first row, the contents of the second shift register 73 become F(0,
1) to F (255, 1) is the sum of the first calculation result and the second calculation result.

Next, write W(1,3) to the three registers 421 to 423,
If you set W(2,3) and W(3,3) and set F(1,2) in the first stage and F(0,2) in the second stage of the first shift register 3, , the first adder (71
) becomes the third operation result of F(0,1), and the first . It is added to the sum of the second calculation results, and the convolution output G (
0, 1) are input to the control circuit.

Similarly, F(2
, 2) to F (255, 2) are shifted in, F(0, 1) to F (255,
1) convolution output G(0,1)~G(2
55,1) is obtained.

Through the above process, the convolution of each of the pixel data in the first row is completed, and operation calculations are performed on the pixel data in the second and subsequent rows through the same process.

〔Example〕

FIG. 1 is a block diagram of the main parts of the present invention. Figure 2 shows an example of an image signal processing device adopted as a real sensor of an industrial robot.
An A/D converter 12 converts the image signal sent from the frame memory 1 into digital information, and the output of the converter is stored in the frame memory 2. In practice, the frame memory 2 is often composed of multiple frame memories, and if necessary, it can be configured with RAM, serial access memory (S
AM).

The plurality of pixel data to be processed stored in the frame memory 2 are digitized, lines and contours of the image to be detected are extracted by convolution, and the image is sharpened by signal processing. It will be done.

In FIG. 1, each convolution of a plurality of pieces of pixel data to be processed stored in the frame memory 2 is performed using N rows and N columns of weight coefficients stored in advance in the coefficient memory 41.

The first shift register 3 composed of registers 31, 32, 33, . . . has N stages (N=3 in the figure). The coefficient memory 41 also includes N registers 421 and 42 in which N weight coefficients are set.
2.423. . . via multipliers 51, 52, 53, .
...is connected to... These multipliers 51.52゜53
, . . . are each stage 31°32 of the first shift register 3.
, 33. Registers 421 and 42 corresponding to the output of ...
The outputs of 2°423, . . . are multiplied. Also, N multipliers 51.52.53. ... are added by an adder 71. The second shift register 73 has a number of stages equal to the number of pixel data to be processed in the row direction of the frame memory 2, and the output of the second shift register 73 and the output of the adder 71 are added, and Second result
There is an addition register 72 which is added to the shift register 73 of . The output of the second shift register 73 passes through a gate 74 before being applied to the adder 71, and the passage of the signal is controlled by applying an inhibit signal.

N registers 421.422.423. . . . to set the load coefficient for one row of the coefficient memory 41, and
Frame memo U2 corresponding to this set load coefficient
pixel data are sequentially input to the first shift register 3 to perform input control for all rows of the coefficient memory 41, and while this control is being performed for the if& row of the coefficient memory 41, pixel data from the addition register 72 are inputted sequentially to the first shift register 3. The output data is output as a matrix convolution calculation result for each row of pixel data to be processed.

The result of multiplying the multiplier of the combination part by the variable is stored in the RAM, and the result is obtained by looking up the result in a table using the variable, and the multiplication operation is performed. Therefore, when convolution is performed once or continuously, the storage capacity of the multiplier RAM can naturally be reduced by reducing the number of multipliers required for the calculation.

Many of the weighting coefficient matrices for line/contour extraction used in -m-type image processing have a symmetrical arrangement. -to -columnproce
ss) has been adopted. As a result,
O', 45°, 90°, 135°...275°,
Vector convolution can be performed on each angle, such as 315°.

In the device shown in Figure 1, when vector convolution is performed using a column-to-column method and then matrix convolution is performed in a time-division manner, the symmetrical shape of the matrix prevents the multiplication results of equal weight coefficients from being duplicated. It turns out. Thereby, RAM with multiplier function
storage capacity can be saved.

In connection with the apparatus of FIG. 1, an example of a load matrix used for line/contour extraction will be described below.

In the matrix array, there are many coefficients in a symmetric form, and in the case of a symmetric form, the coefficients are considered to be unchanged.

Also, if the coefficients are the same, there is no need to load the previous one.
It is possible to use. When loading the multiplier, it is added to the multiplier output from the interface via the buffer and sent to the adder, and the buffer input is applied to each input simultaneously.

An example of a load matrix for line/contour extraction is shown below.

Gradient example: Gij matrix 3X3fx matrix 3X3fy matrix 4X4fx Laplacian example: Lij matrix 3x3 matrix 4x4 Note that if you need to further improve the accuracy after vector convolution , matrix conpolusion is performed, and the calculation can be carried out by not having duplicate multiplication results with equal multipliers due to symmetry.

Figure 5 shows frame memory as serial access memory (
Another embodiment configured with a dynamic RAM (d-RAM) with SAM will be shown. In Fig. 5, there are two accesses between the vector convolution processor and the frame memory: sequential access in the 0° force direction via serial access memory (SAM) and RAM access via d-RAM in the 45°, 90°, and 135° directions. It is equipped with an image data bus that is connected by the O.

It is characterized by particularly high-speed directional convolution.

Conventionally, the frame memory has been configured with a static RAM, and vector convolution in all directions can be performed at high speed, but there is a problem in that it is very expensive.

In FIG. 5, the frame memory usually has a capacity for four to five screens, so its storage capacity is large, and an inexpensive d-RAM is used to reduce the cost of the device. Especially d-R with serial access memory (SAM)
If the frame memory is sequentially accessed row by row from the serial access memory using AM, it becomes possible to operate at a speed comparable to that of high-speed S-RAM.

Therefore, if a vector convolution processor that operates at the same speed as access in the 0° direction that can be accessed sequentially is installed, the processing time will be O.

Direction vector convolution can be performed quickly. Matrix convolution is performed by repeatedly performing 0° direction vector convolution and adding up the sums. The above operation is
This is an important function for convolution processors.

〔Effect of the invention〕

According to the present invention, convolution of each of a plurality of pieces of pixel data to be processed stored in a frame memory is obtained at a relatively high speed using a small number of multipliers, and the number of types of weighting coefficients required for convolution is reduced. , it is possible to realize an image signal processing device in which the storage capacity of the multiplier RAM is reduced.

[Brief explanation of drawings]

FIG. 1 is a diagram showing a schematic configuration of an image signal processing device as an embodiment of the present invention, FIG. 2 is a diagram showing a configuration of a coefficient matrix for NXN convolution used in the device shown in FIG. 1, and FIG. FIG. 4 is a diagram explaining the functions of a conventional convolution device, and FIG. 5 shows another example of a frame memory used in the vector convolution processor used in the device of FIG. 1. 2... Frame memory, 3... First shift register, 31.32.33... Register, 11... Image sensor, 12... A/D converter, 41... Coefficient memory , 42... Mapping memory, 421.422, 423... Register, 51.52.
53... Multiplier, 61... Interface, 71... Adder,

Claims (1)

[Claims] 1. It has a frame memory that digitally converts and stores the image signal of the image sensor, and a coefficient memory that stores N rows and N columns of weighting coefficients in advance, and each of the plurality of processing target pixel data stored in the frame memory. An image signal processing device that performs convolution using weighting coefficients of the coefficient memory, comprising: a first shift register having N stages; N registers in which N weighting coefficients are set; a multiplier using N RAMs that multiplies the output of each stage of the first shift register by the output of the corresponding register; an adder that adds the outputs of the N multipliers; a processing target in the row direction of the frame memory a second shift register having a number of stages equal to the number of pixel data; an addition register that adds and stores the output of the adder and the output of the second shift register, and adds the addition result to the second shift register; , a load coefficient for one row is set from the coefficient memory to the N registers, and pixel data of the frame memory corresponding to the load coefficient is shifted to the first shift for all rows of the load coefficient. Each row of pixel data to be processed is sequentially input to the register, and until the last row of the weighting coefficients is inputted, each row of pixel data to be processed is convolved using the addition register output data, and the weighting factor selection signal is used to convolve each row of pixel data to be processed. 1. An image signal processing device using convolution, characterized in that the coefficients are repeatedly used, and the coefficients are further associated with a RAM-based multiplier by a mapping memory, thereby enabling sequential specification in matrix convolution.