GB2220289A - Picture processing apparatus - Google Patents

Abstract

A stream of pixels 5 produced by column by column sampling of a rectangular pixel image array, is fed to a corresponding rectangular array of columnal shift registers, the array of shift registers being sub-dividable into sub-arrays of sub-registers, the first columnal shift sub-register of each sub-array being fed in parallel with the stream of pixels and the end (base) of each columnal sub-register being connected to the input (top) of the next columnal sub-register, the stages in each sub-array forming the first processing stage of a multi-stage pyramidal processing operator 1, chosen, in combination with the shape of the sub-array, so that the output of the operator performs an adaptive desired operation on the image, possibly programmed to suit the content of the picture. <IMAGE>

Description

Description PICTURE PROCESSING APPARATUS This invention relates to picture processing apparatus for detecting specified features in a picture which may have been derived by scanning a natural scene containing small man-made objects to be detected. More particularly it relates to picture processing apparatus for applying a picture processing operator to a picture comprising rows and columns of picture elements, each picture element having a numerically defined intensity, said operator comprising an array of rows and columns of inputs adapted to register with corresponding sub-sets of adjacent picture elements of the picture in turn, the apparatus comprising first means for sampling the picture elements of the picture in turn from top to bottom, column by column, to obtain the numerical intensities, second means for feeding the intensities in turn into the top of the first column of a series of columns of shift register stages corresponding one each to the picture columns, the bottom of each shift register column being connected to feed outgoing intensities into the top of the next column of the series, third means for stepping the intensities through the columns of the series at the rate the picture elements are sampled, the operator input points being connected to a rectangular sub-set of adjacent stages of the shift register columns corresponding to the operator array, and the operator comprising a pyramid of layers of data processsing nodes, a node in a layer being connected to receive the data output by a cluster of adjacent nodes in the next lower layer, to perform a processing operation upon the data, and to output a result to the next higher layer of the pyramid, the top-most layer of the pyramid comprising a single node outputting the result of the operator.

Such picture processing apparatus is described in an article entitled "Pyramid Algorithms and Machines" by Charles R. Dyer in a textbook entitled "Multicomputers and Image Processing Algorithms and Programs" Edited by Preston and Uhr, Academic Press 1982.

Such pyramidal operators are pre-wired assemblies with a fixed set of processing operations. It is an object of the invention to provide a picture processing apparatus, which can be adapted "on-the-fly" to the content of pictures, particularly from natural scenes, as it changes. It is desirable to be able to change the number of picture elements to which picture processing is applied and to change the logical operations applied to the chosen picture elements.

Accordingly, the invention provides picture processing apparatus as set out in the opening paragraph characterised in that the columns of shift register stages comprise switching means to separate the shift register stages connected to the pyramid base into a plurality of sub-arrays of stages formed of columns of stages, successive columns of each sub-array being connected bottom to top, shift register stages being added to each column of each sub-array so that the total number of stages in each column equals the number of picture elements in a picture column, and the input of each sub-array being connected to receive the same sequence of sampled picture elements. Thus the base of the pyramidal picture processor can be broken up into a variable number of sub-arrays of pixel storage latches or stages to each of which the same set of pixels are applied.Thus several operators can be applied simultaneously to a given set of pixels.

In many picture processing operators the values of the incoming pixels need to be given a weight in dependence on their position in the operator array or sub-arrays. Accordingly the invention may be further characterised in that the nodes in the bottom layer of the pyramid each include a multiplying device comprising means for multiplying a picture element value in a shift register stage by a programmable coefficient and means for outputting the result to the next layer up the pyramid.

To further extend the scope of programming of the operators the invention may be yet further characterised in that the data processing nodes in layers above the bottom layer of the pyramid comprise means for switching the nodes to any one of a predetermined set of processing functions.

In this way the invention not only allows the size and/or number of simultaneously applied operators to be varied at will but the nature of the picture processing operations applied can also be varied at will.

The predetermined set of functions may include means for selecting either the absolute values or the magnitudes of the values input to a node for use in that node. Having selected the form of input value to be used in this way, the set of functions may include means for adding the data values output by the cluster of adjacent nodes in the next lower layer of the pyramid.

Additionally the set of functions may include means for sorting the maximum or the minimum data value output by the cluster of adjacent nodes in the next lower layer of the pyramid. Thus various combinations of processing functions can be carried out at a node in the layer.

An embodiment of the invention will now be described, by way of example, with reference to the accompanying drawings in which Figure 1 shows a known pyramidal systolic picture processing apparatus, Figures 2 to 6 inclusive shows five possible connection arrangements at the base of an 8 x 16 pyramidal picture processor in accordance with the invention, Figure 7 shows the switching arrangement for a quarter of the 8 x 16 pyramid base, Figure 8 shows the whole switching arrangement for the 8 x 16 pyramid base, Figure 9 shows a pixel value multiplying device for the bottom layer of the pyramid, Figure 10 shows the detailed arrangement of the connections between pyramid layers, and Figure 11 shows a programmable data processing node used in layers of the pyramid other than the bottom layer.

Referring to Figure 1 there is shown schematically, by way of setting the invention in its background, a pyramidal systolic array 1 connected to process the data of the picture elements of a picture scanned in columns. In this schematic example a picture, not shown, is scanned, by means not shown, in columns of sixteen picture elements per column from top to bottom. In practice, a larger number of picture elements per column is used, typically a hundred or more. The intensity of each picture element (pixel) is digitised, by means not shown, and applied in turn to terminal 2 connected to the first stage 3 of a 16 stage shift register 4. The final stage 5 of register 4 is connected to the first stage 6 of a second 16 stage shift register 7 and 16 stage shift registers 8 and 9 are similarly connected in series.Shift pulses are applied, by means not shown, to all stages of all registers in synchronism with the picture sampling rate.

The pixel values passing down a column of a shift register are in digital form. The binary digits of each pixel value may be passed serially from one shift register stage to the next in a column along one conductor from a set of binary latches, one latch for each bit, in one stage to a corresponding set of binary latches in the next stage. Alternatively, the digital pixel values may be passed in parallel from the binary latches of one stage to those of the next stage along a corresponding number of parallel conductors.

Thus after several columns of the picture have been sampled, laterally adjacent stages of the register contain pixel values of laterally adjacent picture elements in the picture scanned. As scanning continues the effect is to "roll" the digitised picture through the shift registers, advancing it one pixel to the right after each scanned column.

The pyramidal array 1 has a number of data processing layers.

The first layer 10 has a rectangular 4 x 4 array of 16 inputs 11 connected one each to the top four stages of the four columns.

Thus at each shift register pulse the digitised pixels move one pixel relative to the 16 pyramid connections. The successive layers 12,13,14,15 and 16 of the pyramid have successively decreasing numbers of data processing nodes. Each node in a layer will, in general, have connections to a selection of adjacent nodes in the next lower layer and possibly also to another selection of adjacent nodes in the same layer and an output connection to the next higher layer. Output 17 of the last layer presents, at each shift pulse, a single numerical value which represents the result of applying the whole operator pyramid to a 4 x 4 array of adjacent picture elements.

The data processing operation of the pyramid is referred to as being systolic since successive samples of data pass rythmically through a chain of processing nodes, all processed samples being passed on simultaneously from one processing node to the next.

Thus there are two alternating phases in the systolic operation. In the first phase, the processed data in each node is passed on to the next in the chain. In the second phase, processing operations are carried out at each node on the newly received data. Thus there is a pump phase alternating with a stationary phase hence the term systolic.

The nature of the processing operations carried on at each node are chosen to select some specific feature or set of features in the scanned picture. It is a disadvantage of the arrangement described above that the number of pixels applied to the bottom of the pyramid is fixed and also that the operation carried at each node is a single fixed calculation. For greater generality in the assessment of images it would be desirable to remove these limitations. For some features a large array of pixels is needed, for others a small array. Also, a range of data processing operations should be available at each node to cope with different features.

By way of example, the invention will be illustrated by reference to a pyramid having a rectangular base of 8 x 16 connections. As will be seen the number of connections in the base layer can be varied widely, this 8 x 16 example being chosen to illustrate the wide range of switchable functions that can be obtained even with such a relatively small number of base layer connections.

Figures 2 to 6 inclusive show five different possible connection arrangements of the shift register stages X at the bottom of the pyramid. X is chosen as the symbol for each shift register stage since it not only comprises the latch which stores a pixel value between shift pulses but also has a multiplying device for taking the numerical value in the latch, multiplying it by an externally programmable factor, and presenting this multiplied value to the base of the pyramid. The pixel value in the latch passes on unchanged to the next shift register stage or latch in the shift register column at the next shift pulse.

In Figure 2, the area covered by the pyramid is 8 x 16, the largest possible, which allows fairly extensive matched filters to be applied to the picture S as will be explained later. The triangles at the base of each column are shift registers having a number of stages equal to the number of pixels in a picture column minus eight. Hence the total number of shift register stages equals the number of pixels in a picture column as required in the Figure 1 scheme.

In Figure 3, the shift register stages are divided into two 8 x 8 sub-arrays, each being fed simultaneously with the picture S. The triangle delays have the same value as in Figure 2. With this arrangement two quite different operators, or filters, could be applied to the picture, the final value from the top of the pyramid being a programmable logical combination of their results.

In Figure 4, the stages are divided into four 8 x 4 sub-arrays, each fed simultaneously with the picture S, the triangle delays being unchanged from Figures 2 and 3. Four different, though smaller, operators could now be applied simultaneously to the picture. In Figure 5, the stages are divided into four 4 x 8 sub-arrays, the triangle delays now being four less than the number of pixels in a picture column. The sub-arrays are the same size as those of Figure 4 but are rotated by 90', a desirable feature when looking first for vertical and then for horizontal picture features.

Finally, in Figure 6, the stages are divided into eight 4 x 4 sub-array, each fed simultaneously with the picture S. Certain kinds of picture feature detector require an operator to be presented to the eight pixels surrounding any given pixel in the eight possible orientations. With the invention this can all be done simultaneously.

Figure 7 shows the switching arrangement for the 8 x 4 first quarter of the 8 x 16 array of shift register stages which provides the various sub-arrays of Figures 2 to 6 inclusive. Also shown on the far right of this figure is the first column of the next quarter of the 8 x 16 array with its switching arrangements. In this figure the switches are shown as mechanical switches shown ganged together where required by a chain dotted line. In practice solid state switches would be used due, in part, to the need for speeds much higher than could be attained by mechanical switches.

Also, in this figure it is assumed that the picture is scanned in columns of n pixels, the triangle delays for each column arrangement being shown in number of pixels, n, n-4 and n-8.

The twelve switches 20 to 31 inclusive are ganged together for control by input Al to be switched to either of two states. In the state shown the output from the bottom of the first upper half-column 32 of four stages is routed to the top of the next upper half-column 33 via a delay of n-4 and switch 20. The state of switch 28 applies S to the top of the first lower half-column 34 of four stages and thence, since switch 24 is open, via a delay of n - 8 + 4 = n - 4, to the top of the next lower half-column 35 of four stages via switch 29. Thus in the state shown the configuration of columns needed for the first four columns in Figures 5 and 6 is realized. In the other state of switch Al, half columns 32 and 34 are connected in series and likewise columns 33 and 35 as well as the other two pairs of sub-columns shown in the figure.The state of switches 24, 25, 26 and 27 now reduces the column delay to n - 8 and the state of switches 20,21,22 and 23 routes the output from the bottom of the columns of now eight stages, each with a delay of n - 8, to the top of next adjacent column. Thus the configuration for the first four columns in Figures 2, 3 and 4 is realised.

In Figure 7 the first upper and lower half-columns 36 and 37 of the next quarter of the whole array are shown. The control A2 is the ganged switch control for the next quarter of the whole array corresponding to control Al for the first quarter. However, in addition, double pole switches 38 and 39 are provided at the top of the upper and lower half-columns 36 and 37 respectively controlled by independent inputs B and C respectively. Such independently controlled double pole switches are also provided at the first columns of the third and fourth quarters of the whole array. The input to any upper half of the first column of each quarter can thus be switched to receive input from S or from the bottom of the preceding column or upper half column.The input to any lower half of the first column of each quarter can be switched to receive input from S or from the bottom of the preceding lower half-column or from the bottom of the half-column above. Thus, using the A, B and C controls of each quarter array appropriately all the configurations of Figures 2 to 6 can be realised. Figure 8 shows the whole switching system of the 8 x 16 array, switch control lines being omitted for clarity.

It should be noted that configurations other than those shown in Figures 2 to 6 can be realised. For example, in Figure 3, the second set of eight columns could be switched to the configuration of the second set of eight columns of Figure 6, providing one 8 x 8 operator and four 4 x 4 operators simultaneously.

It has been mentioned above that each of the shift register stages X shown in Figures 2 to 8 inclusive comprises a multiplying device forming the bottom level of nodes of the pyramid. Figure 9 shows a block schematic diagram of one such node. Latch 40 is part of the column of latches which comprise the shift register, receiving pixel values from the preceding latch on line 41 and outputting pixel values to the following latch on line 42. The coefficient by which a pixel value is to be multiplied is stored in latch 43 having been delivered to it on input 44. On a clock pulse at input 45, the contents of latch 40, the pixel value, are stepped into a digital multiplier where they are multiplied by the coefficient and delivered on output 47 to the next layer up the pyramid. Coeffient latch 43 can be set to zero using reset input 48.The use of this multiplying device will be explained in more detail later in relation to specific examples. But in general, the multiplying device allows a pixel value to be given a predetermined weighting or importance in relation to its position in the array before being applied to the upper layers of the pyramid. Also, the multiplying device allows local operators smaller than the base of the array to be implemented merely by setting the coefficients in the unwanted rows and columns to zero. All multiplications at the base of the pyramid are completed in one cycle of the shift registers so that all multiplied pixel values are ready to pass together up to the next layer of the pyramid at the next 'pump' phase.

Figure 10 shows a more detailed plan view of the data processing pyramid. The coefficient multiplying devices X are shown in the columns of shift registers and constitute the bottom layer of nodes of the pyramid. Clusters of nodes comprising squares of four adjacent multiplying nodes, 50, 51, 52 and 53 for example, provide inputs to the next layer of data processing nodes 54, 55, 56 and 57 which, in turn, provide inputs for a data processing node 58 in the next layer. If the base of the pyramid is a 4 x 4 array of nodes, the output of nodes 58 would be the final output of the pyramid. For an 8 x 8 array four nodes 58 would provide input to a final single node, there then being four layers altogether including the bottom layer and the single node top layer.Numbers of nodes in å cluster other than four may be used in any layer, the cluster geometry usually being chosen to utilise all the outputs from that layer.

Figure 11 shows the details of an example of a data processing node used in layers above the base layer of the pyramid. It is an adder/comparator node which can be switched under external control to perform any one of six functions upon the square cluster of four nodes shown in a layer in Figure 10. The six functions are.

e=a+b+c+d ... (1) e = |a| + |b|+ ... (2) e = max(a,b,c,d) ... (3) e = max(|a|,|b|,|c|,|d|) ...(4) e = min(a,b,c,d) ... (5) e =min(IaI,IbI,IcI,IdI)= c... (6) where a, b, c and d are the inputs from the layer of nodes below, and e is the output to the layer of nodes above. The function performed by each of these nodes is fixed for each layer of nodes, but may vary from layer to layer according to how the array has been programmed.

The inputs a,b,c and d are stored in latches 60, 61, 62 and 63 respectively, resettable under the control of input 64. The gangs of switches 65 and 66 either pass the inputs as absolute magnitudes to the following circuits or pass them through units 67 to 70 inclusive which remove the sign of the inputs and provide the magnitude only. In the position of the gangs of switches 71, 72 and 73 shown the adders 74, 75 and 76 produce for e the sum of the four inputs, either absolute value or magnitude, as required by relations 1 or 2 above. Each of the three units 77, 78 and 79 have two inputs, a first output MAX on which the larger of the two inputs is available, and a second output MIN on which the smaller of the two inputs is available.Gauged switches 72 therefore also provide for e the maximum or the minimum of either the absolute values or magnitudes of the four inputs. The ganged switches are controlled by unit 80. A similar control unit (not shown for clarity) is provided for the ganged switches of Figures 7 and 8 which control the arrangement of nodes at the base of the pyramid.

These switch control units might typically contain a read-only memory (ROM) containing all the combinations of switch positions required for any of the picture processing operations envisaged.

Any combination can then be selected at will. The ROM also contains the multiplying coefficients for application to each terminal 44of the multiplying node of Figure 9 for the picture processing operations envisaged.

Three examples will now be given of the kinds of picture processing operation which can all be performed by an 8 x 16 programmable pyramidal systolic picture processing array in accordance with the invention.

The first example is a matched filter devised to detect a bright point source of radiation as imaged onto a picture scanning column by an optical system having a particular point spread function. The base array is switched to the configuration of Figure 2 and the mutliplying coefficients, which express the point spread function, for the base nodes are as follows.

0 1 2 2 -2 -5 -2 2 2 1 0 0 0 0 0 0 -1 -8-15-11 16 37 16-11 -15 -8 -1 0 0 0 0 0 -5 -22 -41 -28 46 100 46 -28 -41 -22 -5 0 0 0 0 0 -1 -8-15-11 16 37 16-11 -15 -8 -1 0 0 0 0 0 0 1 2 2 -2 -5 -2 2 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Each of the adder/comparator nodes are switched to perform the function e = a + b + c + d i.e. relation 1 above so that the output of the array is the result of applying the matched filter. The filter output rises to a sharp maximum as the spread source image comes into exact registration with the filter.

The second example is an octagonal maximum filter which may be required. The octagonal shape is a fairly good approximation to a circle given the rectangular arrangement of the pixels. The coefficients at the base of the array are 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 11 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 whilst the nodes in the upper layers are all be programmed to perform the function e = max(a,b,c,d) i.e. relation 3 above The base array configuration is that of Figure 2, half of the multiplying coefficients being set to zero to remove an unwanted part of the array. In this way the array would output the maximum pixel value lying within its octagonal window.

The third example is of a picture feature detector referred to in the art as the Prewitt Edge Detector. The base array configuration is that of Figure 6. This edge detector works by applying eight finite impulse response linear filters one each to eight copies of the image scanned, and at any one image point taking the largest output from these filters as the edge magnitude at that point.The 8 operators used are: 1 1 1 1 1 1 1 1 -1 1 -1 -1 1 -2 1 1 -2 -1 1 -2 -1 1 -2 -1 -1 -1 -1 1 -1 -1 1 1 -1 1 1 1 -1 -1 -1 -1 -1 1 -1 1 1 1 1 1 1 -2 1 -1 -2 1 -1 -2 1 -1 -2 1 1 1 1 1 1 1 -1 1 i -1 -1 1 The coefficients given to the nodes in the 16 x 8 array of Figure 6 should therefore be: 1 1 1 0 1 1 1 0 1 1 -1 0 1 -1 -1 0 1 -2 1 0 1 -2 -1 0 1 -2 -1 0 1 -2 -1 0 -1 -1 -1 0 1 -1 -1 0 1 1 -1 0 1 1 1 0 0 0 0 0 0 O 0 0 '0 0 0 0 0 0 0 0 -1 -1 -1 0 -1 -1 1 0 -1 1 1 0 1 1 1 0 1 -2 1 0 -1 -2 1 0 -1 -2 1 0 -1 -2 1 0 1 1 1 0 1 1 1 0 -1 1 1 0 -1 -1 1 0 0000000000000000 The input pixel data is supplied to each of the 8 data paths simultaneously. The first two layers of adder/comparator nodesperform the operation e = a + b + c + d ... (1) which gives at the outputs of the second layer of these nodes the outputs from the eight filters.The top two layers of the array then perform the function e = max(a,b,c,d) ... (3) which finds the maximum output from the eight filters, and presents it as the output value at the top of the array.

This example shows how 3 x 3 operators can be implemented with the 4 x 4 base array configuration by setting the appropriate row and column multiplying coefficients to zero.

From reading the present disclosure, other modifications will be apparent to persons skilled in the art. Such modifications may involve other features which are already known in the design, manufacture and use of picture processing apparatus and component parts thereof and which may be used instead of or in addition to features already described herein. Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure of the present application also includes any novel feature or any novel combination of features disclosed herein either explicitly or implicitly or any generalisation thereof, whether or not it relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as does the present invention. The applicants hereby give notice that new claims may be formulated to such features and/or combinations of such features during the prosecution of the present application or of any further application derived therefrom.

Claims (8)

1. Picture processing apparatus for applying a picture processing operator to a picture comprising rows and columns of picture elements, each picture element having a numerically defined intensity, said operator comprising an array of rows and columns of inputs adapted to register with corresponding sub-sets of adjacent picture elements of the picture in turn, the apparatus comprising first means for sampling the picture elements of the picture in turn from top to bottom, column by column, to obtain the numerical intensities, second means for feeding the intensities in turn into the top of the first column of a series of columns of shift register stages corresponding one each to the picture columns, the bottom of each shift register column being connected to feed outgoing intensities into the top of the next column of the series, third means for stepping the intensities through the columns of the series at the rate the picture elements are sampled, the operator input points being connected to a rectangular sub-set of adjacent stages of the shift register columns corresponding to the operator array, and the operator comprising a pyramid of layers of data processsing nodes, a node in a layer being connected to receive the data output by a cluster of adjacent nodes in the next lower layer, to perform a processing operation upon the data, and to output a result to the next higher layer of the pyramid, the top-most layer of the pyramid comprising a single node outputting the result of the operator, characterised in that the columns of shift register stages comprise switching means to separate the shift register stages connected to the pyramid base into a plurality of sub-arrays of stages formed of columns of stages, successive columns of each sub-array being connected bottom to top, shift register stages being added to each column of each sub-array so that the total number of stages in each column equals the number of picture elements in a picture column, and the input of each sub-array being connected to receive the same sequence of sampled picture elements.

2. Picture processing apparatus as claimed in claim 1 characterised in that the nodes in the bottom layer of the pyramid each include a multiplying device comprising means for multiplying a picture element value in a shift register stage by a programmable coefficient and means for outputting the result to the next layer up the pyramid.

3. Picture processing apparatus as claimed in claim 1 or claim 2 characterised in that the data processing nodes in layers above the bottom layer of the pyramid comprise means for switching the nodes to any one of a predetermined set of processing functions.

4. Picture processing apparatus as claimed in claim 3 characterised in that the predetermined set of functions includes means for sel-ecting either the absolute values or the magnitudes of the values input to a node for use in that node.

5. Picture processing apparatus as claimed in claim 3 or claim 4 characterised in that the predetermined set of functions includes means for adding the data values output by the cluster of adjacent nodes in the next lower layer of the pyramid.

6. Picture processing apparatus as claimed in claim 3 or claim 4 characterised in that the predetermined set of functions includes means for sorting the maximum or the minimum data value output by the cluster of adjacent nodes in the next lower layer of the pyramid.

7. Picture processing apparatus as claimed in any one of claims 1 to 6 inclusive characterised in that the cluster comprises a square of four adjacent nodes in the next lower layer of the pyramid.

8. Picture processing apparatus substantially as described with reference to the accompanying drawings.