JPH10134176A - Method and device for image signal processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily vary the number of taps and a conversion ratio in a short time by automatically setting a filter program and a filter coefficient according to filter specification information and performing filter operation. SOLUTION: Pixel data of one horizontal line after being inputted to an input register 24 in order from an image input terminal 10 are supplied to an upper processor element(PE) group 21 in parallel. The PE group 21 performs filter operation processing for respective pixel data that respective element processors 23 have received by using a control instruction from a controller 26 and coefficient data from a coefficient memory 27. The processing result of the upper PE group 21 is inputted to the lower PE group 22 as it is in parallel and filter operation processing is further performed when necessary. The processing result of the PE group 22 is inputted to an output register 25 in parallel and outputted, pixel by pixel, in order. The processing like this is repeated as many times as vertical lines to perform two-dimensional image processing.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image signal processing method and apparatus for performing image signal processing such as pixel number conversion processing and scanning line conversion.

[0002]

2. Description of the Related Art In recent years, digital signal processing of video signals has been performed with the improvement of semiconductor technology and processing speed performance of semiconductors. Recently, image display devices have been replaced with conventional cathode ray tubes, and LCDs (Liquor
id Crystal Display (Liquid Crystal Display) Fixed pixel display devices such as display devices and plasma display devices are becoming widespread.

Recently, the so-called NTSC (Nationa
l Television System Committee) signal, PAL (Phase A
lternation by Line) signals, as well as HDTV (High Definition Televisio
n) signal, VGA (Video Graphics Array) signal, SVG
It is required to be able to display signals in various formats such as A (Super VGA) signal and XVGA (extended VGA) signal.

[0004] In these various formats, the number of pixels to be handled is different. When displaying video signals of various formats having different numbers of pixels, an analog display device such as the above-mentioned cathode ray tube can change the deflection speed of the electron beam according to the number of pixels per scanning line time. I'm done.

However, in the above-mentioned fixed pixel display device, the number of pixels that can be handled is fixed, so that the conventional analog technology as in the case of the above-mentioned cathode ray tube cannot be used. Therefore, in order to display these signals of various formats on the above-described fixed pixel display device, it is essential to convert the number of pixels or the number of scanning lines by digital signal processing.

Here, as a technique for converting the number of pixels of an image as described above, a method of converting the number of pixels using an interpolation filter which has been conventionally used will be described below. The technique of implementing an interpolation filter using, for example, a digital filter has been described in detail in various textbooks in the field of digital signal processing, and will be briefly described here.

First, the enlargement or reduction of an image and the conversion of the sampling frequency (the number of pixels) will be described.

An arithmetic operation for realizing enlargement or reduction of an image and an arithmetic operation for converting a sampling frequency (the number of pixels) of an image (conversion between image standards having different resolutions) are performed in each pixel of the original image. By considering the position as an operation for obtaining pixel data at a position that did not exist at the beginning, it has been found that the same operation of the interpolation filter is sufficient. This will be described with reference to FIG.

FIG. 11A shows a model of an original image. It is assumed that the square frame in FIG. 11A is a part of the screen. It is assumed that the circles in the figure represent the positions of the pixels of the image. That is, the partial original image shown in FIG. 11A is composed of (8 pixels in the horizontal direction) × (6 pixels in the vertical direction). Here, for ease of thinking, the number of pixels is set to be small and easy to handle. Consider a case where this original image is enlarged, for example, by a factor of 10/7. Note that the magnification here is expressed not by area but by length ratio. 10/7 of the original image
Means that the image (the number of pixels) in FIG. 11A is enlarged, and the display image standard does not change.
The arrangement method of the pixels, that is, the pixel interval and the like are shown in FIG.
) And the result of the enlargement is as shown in FIG. That is, in FIG. 11A and FIG. 11B, since the magnification is 10/7 times (10/7 = 1.429 times), the image of FIG. 11) is 1.429 times longer than the image of FIG. 11), and the number of pixels constituting the enlarged screen of FIG. 11B is increased to the square of 1.429.

Therefore, for example, when viewed in the horizontal direction,
The same image, which was 8 pixels in the original image, is enlarged to 8 pixels.
X10 / 7 = 11.429 pixels, and the positional relationship between the enlarged pixels and the image differs. For this reason, the data of each pixel after enlargement (data expressing luminance and color)
Will be different from that of the original image.

FIG. 11 (d) shows the above-described image magnification of 10/7 times only in the horizontal direction. The tip of the arrow in FIG. 11D indicates the position of each pixel, and the upper R _i (i = 1, 2,...) Is the data of the original pixel,
The lower Q _i (i = 1, 2,...) Is the data of the interpolated pixel after the enlargement. That is, the interval of R _i is 1 of the interval of Q _i.
It is 0/7 times. Although FIG. 11D shows only the state of enlargement in the horizontal direction, the vertical direction can be considered in the same manner as FIG. 11D. Here, the description in the vertical direction is omitted.

The value of the data (expressing luminance and color) of each pixel after the enlargement has a positional relationship as shown in FIG. 11D with respect to each pixel position of the original image. 1
For one point, an interpolation filter operation, that is, a convolution operation of an interpolation function may be performed using data values of some original image pixels around the interpolation point to be obtained here. This interpolation calculation will be described later.

Next, let us consider a case where the sampling frequency is increased to, for example, 10/7 while the size of the image is kept as it is. This is a sampling frequency conversion, but in other words, it is the same as converting to an image standard having the same image size and a resolution 10/7 times higher, that is, increasing the number of pixels by 10/7. It is the same as that. In this case, when the original image is shown in FIG. 11A, the result is as shown in FIG. 11C. That is, in one dimension, the number of pixels increases by 1.429 times, and the number of pixels in the same area increases by a square of 1.429.

The calculation operation for converting to an image standard having a large number of pixels may be exactly the same as the calculation operation for image enlargement described above. As can be seen from the relationship between (a) and (b) of FIG. 11 and the relationship between (a) and (c) of FIG. This is because the interpolation filter operation based on the relationship, that is, the convolution operation of the interpolation function may be performed.

Next, the case of image reduction will be described. A case where FIG. 12A similar to FIG. 11A is used as a model of the original image and this original image is reduced to, for example, 10/13 times is considered. Here, since the original image is reduced and the display image standard is not changed, the arrangement of the pixels, that is, the pixel interval is the same as that of FIG. 12A, and the result is shown in FIG. Become like That is, 10/1
Since it is reduced by a factor of 3 (10/13 = 0.769), FIG.
In the image of FIG. 2B, the length of one side of the image of FIG. 12A is reduced to 0.769 times, and the number of pixels constituting the reduced screen is reduced to the square of 0.769. .

Therefore, for example, when viewed in the horizontal direction,
The same image was 8 pixels in the original image, but 8 pixels after reduction.
Since × 10/13 = 6.154 pixels, the positional relationship between the reduced pixels and the image differs. For this reason, the value of the data (expressing luminance and color) of each pixel after reduction is different from that of the original image.

FIG. 12D shows the state of image reduction of 10/13 times only in the horizontal direction. The tip of the arrow in the figure indicates the position of each pixel, and R _i (i =
1, 2,...) Are original pixel data, and the lower Q
_i (i = 1, 2,...) is the reduced interpolated pixel data. That is, the interval of R _i is 10/13 times the interval of Q _i . FIG. 12D shows only the state of enlargement in the horizontal direction, but the vertical direction can be considered in the same manner as in FIG.

The values of the data (expressing luminance and color) of each pixel after the above-described reduction have the positional relationship shown in FIG. 12D with respect to each pixel position of the original image. For each pixel, an interpolation filter operation, that is, a convolution operation of an interpolation function may be performed using data values of some original image pixels around the interpolation point to be obtained. This interpolation calculation will be described later.

Next, let us consider a case where the sampling frequency is increased to, for example, 10/13 times while keeping the size of the image as it is. This is a sampling frequency conversion, in other words, the image size is the same and the resolution is 10/13
This is the same as converting to a twice lower image standard, and the same as increasing the number of pixels by 10/13. In this case, the result when the original image is shown in FIG. 12A is as shown in FIG. That is, in one dimension, the number of pixels is reduced to 0.769, and the number of pixels in the same area is reduced to the square of 0.769.

The calculation operation for converting to the image standard with a small number of pixels may be exactly the same as the calculation operation for image reduction described above. As can be seen from the relationship between FIG. 12A and FIG. 12B and the relationship between FIG. 12A and FIG. This is because an interpolation filter operation based on the relationship, that is, a convolution operation of the interpolation function may be performed.

In both the case of FIG. 11 (d) and the case of FIG. 12 (d), it is necessary to perform an operation of an interpolation filter for obtaining a pixel data value at a position which did not exist in the original image at first. The same arithmetic operation can be used for the arithmetic operation for realizing the enlargement or reduction of the image, the arithmetic operation for converting the sampling frequency (the number of pixels) of the image to a higher (more), or the arithmetic operation for lowering (less) the conversion. Will be. Hereinafter, these are expressed as “pixel number conversion” or “interpolation filter operation”.

Next, an interpolation filter operation for obtaining a pixel data value at a position not existing in the original image, that is, a convolution operation of the interpolation function, will be described with reference to FIGS. 13 and 14 and equations (1) to (4). Will be explained.

In FIG. 13, the horizontal axis represents the same as FIGS. 11 (d) and 12 (d), and schematically shows the positional relationship in the horizontal direction. Also, circles in the figure indicate pixel positions of the original image, and S in the figure indicates sampling intervals of the original image, that is, pixel intervals. Then, the position indicated by the arrow h in the figure is a position to be interpolated (interpolation point). That is, the R _i of circled position indicia shown in FIG. 11 (d) or FIG. 12 (d), the Q _i the position of the arrow in the figure h is watching now
Position. Wherein for one single Q _i, interpolation operation is realized by a convolution operation using R _i around sandwiching the Q _i.

Here, according to the "sampling theorem", the theoretically ideal "interpolation" is generally performed by a sinc function f (x) = sinc () as shown in FIG. x) = sin (x) / x (1) The convolution operation may be performed from the infinite time past to the infinite time future using (1) as an interpolation function. However, it cannot be realized in practice, and a practical problem is how to approximate the sinc function to a simple interpolation function of a finite period.
And as a simple approximation method: 1. Nearest neighbor approximation method ((b) of FIG. 14, equation (2)) 2. Bilinear approximation method ((c) in FIG. 14, equation (3)) A cubic approximation method ((d) in FIG. 14, equation (4)) and the like are known. The approximate interpolation function of the nearest neighbor approximation method is shown in FIG. 14B and equation (2), and the approximate interpolation function of the bilinear approximation method is shown in FIG. 14C and equation (3).
The approximation interpolation function of the cubic approximation method is shown in FIG. 14D and equation (4). FIG.
It is assumed that the horizontal axis of 4 is normalized by the sample interval of the original image.

F (x) = 1 (−0.5 <x ≦ 0.5) f (x) = 0 (−0.5 ≧ x, x <0.5) (2) f (x) = 1− | x | ( | x | ≦ 1) f (x) = 0 (| x |> 1) (3) f (x) = | x | ³ −2 | x | ² +1 (| x | ≦ 1) f ( ^{x) = - | x | 3} +5 | x | 2 -8 | x | +4 (1 <| x | ≦ 2) ·· (4) f (x) = 0 (2 <| x |) after interpolation 1 To calculate the pixels, the convolution operation is performed using the pixel data of the original image using one pixel in the nearest neighbor approximation, two pixels in the bilinear approximation, and four pixels in the cubic approximation. The bilinear approximation method uses a weighted average, and is well known as linear interpolation.

The convolution operation required here is a so-called FI
An R (non-recursive) digital filter may be used, and when the center of the approximate interpolation function is set to the interpolation point, the value obtained by sampling the approximate interpolation function at the sampling points of the nearby original image by a predetermined number of pixels is interpolated. Used as a filter coefficient set.

It should be noted here that the FIR filter calculation is performed based on the difference in the phase (shift) P shown in FIG. 13 showing the relative positional relationship with the original image pixel position for each interpolation point to be interpolated. , The set of filter coefficients will differ.

For example, if the interpolation filter operation is performed by the bilinear approximation method, if P = 0.0, the two weights (filter coefficients) that constitute the filter coefficient set in that case are 1.0 and 1.0. 0.0, which is a coefficient set that passes the pixel data value of the original image at the same position as it is. When P = 0.5, the two weights (filter coefficients) are 0.5 and 0.5. And P = 0.3
, The above two weights (filter coefficients) 0.7 and 0.
The coefficient set is 3. Similarly, in the cubic approximation method, when P = 0.0, the four weights (filter coefficients) are 0.0, 1.0, 0.0, and 0.0, and the original image whose position matches. The coefficient set is such that the data value of the pixel is passed as it is. When P = 0.5, the four weights (filter coefficients) are -0.125, 0.625, 0.625, and -0.125. , And P =
The four weights (filter coefficients) in the case of 0.3 are -0.0
The coefficient set is 63, 0.847, 0.363, and -0.147. These are obtained by substituting P for x in the above equations (2) to (4).

Next, a conventional example of a hardware configuration for implementing an interpolation filter operation, that is, a convolution operation of an interpolation function will be described with reference to FIG. The interpolation filter operation is performed by an FIR digital filter, provided that its filter coefficients must change for each interpolation point. Therefore, it can be realized by performing a product-sum operation with an arithmetic device having a configuration as shown in FIG. In FIG. 15, the cubic approximation method is assumed.

FIG. 15 shows an FIR digital filter for sequentially calculating appropriate values of data at each interpolation position (interpolation filter operation).

In FIG. 15, an input terminal 250 is supplied with time-series input image data that has been horizontally scanned. Further, four registers 254 to 258 represented as M1, M2, M3, and M4 in the drawing are registers capable of storing one pixel data of the original image supplied from the input terminal 250. These registers 254 to 258 are connected in series and operate as a four-stage shift register. To the shift register constituted by these registers 254 to 258, the above-mentioned horizontal-scanned input image data time series is inputted in word units, whereby the shift register always stores continuous four original image pixel data. Will be. The cubic approximation method is a method of obtaining a proper data value of an interpolation point from original image data of a total of four pixels, two pixels in the vicinity of the interpolation point and two pixels on the left and right sides, so the above-described four-stage shift register is required.

The terminal 251 receives a controlled clock or a clock control signal E. This control signal E is a signal for shifting or stopping the four-stage shift register, and is supplied from a controller (not shown).

[0033] terminal 252, from among a plurality of filter coefficient sets, selection signal C to select the filter coefficient set to correspond to the phase P of the interpolation point Q _i to be obtained is input. Since the coefficient memory 253 stores a plurality of filter coefficient sets for each type of the phase P described above, the selection signal is address data.

The multipliers 259 to 262 multiply the filter coefficients FC _{1 to} FC ₄ from the coefficient memory 253 by the pixel data from the registers 254 to 258 of the four-stage shift register, and obtain the respective results. The result of the multiplication is output to the sum adder 263.

The sum adder 263 performs sum addition of the supplied multiplication results, and outputs the addition result from an output terminal 264 as an output image data time series of the interpolation filter operation result.

The configuration shown in FIG. 15 is conventionally realized by hardware. Here, in the case of real-time image processing, since the sampling period is as short as 50 nanoseconds, unlike the case of real-time sound signal processing, it is conventionally realized by a software program. There are few examples, and each part in FIG. 15 is generally realized as a dedicated hardware circuit using a large number of pipeline systems.

Next, the operation of enlarging an image by, for example, 10/7 by cubic approximation in the hardware configuration of FIG. 15 will be described with reference to the tables shown in FIGS. The image magnification of 10/7 is as described with reference to FIG. 11. In the horizontal direction, each interpolation point in the positional relationship of FIG. 11D is considered as shown in FIG. May be used to perform the interpolation filter operation. The table shown in FIG. 16 shows the operation procedure of each unit in FIG. 15 in that case.

The table shown in FIG. 16 is a time chart.
The cycle in the table means time, that is, a clock cycle. The IN column in the table is input data.
Original image pixel data value R _i given to input terminal 250 of
Is shown. C column in the table is a filter phase selection signal C supplied to terminal 252 in FIG. 15, select whether to use a set of filter coefficients of the phase of the coefficient memory 253 throat, for Q _i to be calculated in the cycle to specify the phase P _i. Column E in the table is a shift register control signal supplied to the terminal 251 in FIG. 15, and specifies either a shift (logic "H") or a stop (logic "L") state. In the table of FIG. 16, the control inputs shown in the columns C and E are studied in advance as shown in FIG. 11D depending on what to do, and a predetermined control signal sequence is determined. Will be given to the device. It is assumed that the control in column E is affected by the next clock cycle. M ₁ in the table,
Each column of M ₂ , M ₃ , and M ₄ indicates R _i stored in each of the registers 254 to 258 of the four-stage shift register in FIG. The OUT column in the table is the output terminal 2 in FIG.
64 shows output data Q _i output in the clock cycle. In a hardware device such as that shown in FIG. 15, a latency (a delay that is not essential for realizing a high-speed operation) is usually caused by some kind of pipeline processing in the multiplication and the summation operation. Let's consider it as having no latency.

Here, the coefficient memory 253 stores
Shall ten filter coefficients set shown in the column of 8-bit representation coefficients of the table shown in FIG. 17 is stored, according to the phase P _i given from C column of the table, the four filter coefficients FC ₁ , FC ₂ , FC ₃ , and FC ₄ are output. This is because, in the case of the image magnification of 10/7 times assumed here, there are only ten types of phases necessary for the interpolation operation as shown in FIG. 11D, and these are only used repeatedly. .

The ten types of filter coefficient sets in the table of FIG. 17 correspond to the ten types of phases P for Q _{1 to} Q _{10 in} FIG. 11D. Since there are 10 types of P for the position in the case of equally dividing into ten, in the table of FIG.
The order of _i of Pi is determined in the order of / 10, 1/10, 2/10,... The coefficient obtained by substituting P _i into x in the above equation (4) is a floating-point expression coefficient, which is word-length-limited to 8 bits (here, the maximum amplitude is 128), and an 8-bit expression coefficient Have gained.

In the table of FIG. 16 and the configuration of FIG. 15, first, in the first cycle, R ₁ appears as shown in the IN column. M ₁ at this time, register 254~258, M _2, M
_3, the M _4, the previous data one from R ₁ R _m0, the other previous data R _m1, and before that another R _{m @ 2,}
Further, when the other one is R _m3 , R _m0 ,
It is assumed that R _m1 , R _m2 , and R _m3 are stored.

In the first cycle, the shift control signal in column E is logic "H" and indicates "shift," so that the shift register shifts at the next rising edge of the clock, and
In the cycle, M ₁ , M ₂ , M ₃ , and M ₄ are R ₁ ,
R _m0 , R _m1 and R _m2 are stored. In the second cycle, R ₂ is given as shown in the IN column, and the shift control signal in the E column is logic “H”, that is, “shift”. Therefore, the shift register shifts at the next rising edge of the clock, and in the third cycle, Each register is R ₂ ,
R ₁ , R _m0 and R _m1 . In the third cycle, R ₃ is given from the IN column and the shift control signal is logic “H”.
In the fourth cycle, the registers are R ₃ , R ₂ ,
R ₁ and R _m0 .

In the fourth cycle, R ₄ is given from the IN column, and the positional relationship of Q ₁ with respect to R _m0 , R ₁ , R ₂ , and R ₃ in FIG. , That is, the phase P ₀ is given. Means that this is a _{R a = R m0, R b} = R 1, R c = R 2, R d = P i is i = 0 showing the positional relationship for Q ₁ when the R ₃ in FIG. 13 is there. Therefore, the filter phase selection signal in column C is now P ₀
Therefore, the four filter coefficients FC ₁ , FC ₂ , FC ₃ , and FC ₄ output from the coefficient memory 253 in FIG. 15 are the filter coefficient sets for the phase P _{0 in} the table of FIG.
0, 1.0, 0.0, and 0.0 (0 for 8-bit representation)
And 128, 0 and 0) are obtained.
The product-sum operation for convolution is performed by the product-sum operation configuration as described above, and output data in which the value of R ₁ is directly Q ₁ is obtained at the output terminal 264.

In the fourth cycle, since the shift control signal in column E of FIG. 16 is logic "L" and indicates "stop", the registers 254 to 258 of the shift register are also set in the next fifth cycle. The data of the fourth cycle is held. Then, in the next fifth cycle, I in the table of FIG.
R ₄ is continuously given from the N column, and R _m0 , R ₁ , R ₂ , and R _m in FIG.
Since the positional relationship of Q ₂ with respect to R ₃ , that is, the phase P ₇ is given, a filter coefficient set corresponding to P ₇ in the phase column in the table of FIG. 17 is output from the coefficient memory 253, and the data of Q ₂ The product-sum operation for obtaining the value is performed, and Q ₂ is obtained at the output terminal 264. Here, the phase is P ₇ because R _m0 , R ₁ , R ₂ , R ₂ in FIG.
R in FIG. 13 corresponding to the positional relationship of Q _{2 to 3}
P when _a = R _m0 , R _b = R ₁ , R _c = R ₂ , R _d = R ₃
However, the distance between R ₁ and Q ₂ is 7 / the distance S between R ₁ and R _2.
Because it is 10.

FIG. 11 shows that the shift register did not shift from the fourth cycle to the fifth cycle.
This is because in (d), Q ₁ and Q ₂ are similarly located between R ₁ and R ₂ , and the combination of the four data before and after the original image signal for the interpolation filter operation is the same. FIG.
This is the case when the shift control signal in column E is stopped, that is, becomes logic "L". In this case, the same data is continuously input to the column IN in the next cycle.

In the fifth cycle, the shift control signal in the column E is logic "H", and in the sixth cycle, the registers are R ₄ , R ₃ , R ₂ , and R ₁ respectively. And R ₅ is given by the IN column in the sixth cycle, R ₁ in FIG. 11 (d) is the filter phase selection signal of column C, R _2, R _3,
Given the positional relationship of Q ₃ with respect to R ₄ , that is, the phase P ₄ , a filter coefficient set corresponding to P ₄ in the table of FIG. 17 is output from the coefficient memory, and the product sum for obtaining the data value of Q ₃ The operation is performed, and Q ₃ is obtained at the output terminal 264. Here the phase is in the P ₄ is corresponding to the position relationship between Q ₃ for _{R 1, R 2, R 3} , R 4 in (d) of FIG. 11, R _a = R ₁ in FIG. 13, R _b = R ₂ ,
_{R c = R 3, R d} = P if the R ₄ is, with respect to the spacing S of the R ₂ and R _3, the interval of R ₂ and Q ₃ are 4/10 So. This is because the relative 7/10 P ₇ when the Q _2, is further next 14/10 7/10 is cumulative, subtraction since these 10/10 is equivalent to one minute original image data It is.

That is, each time _i of Q _i increases by 1, P increases by 0.7. However, since an integer can be treated as a data delay, a modulo operation is performed. The increment of this P here will be defined as P _d.

Thereafter, similarly, on the configuration of FIG. 15, Q _i is sequentially obtained by the procedure and control of the table of FIG. 16 determined by (d) of FIG. 11, and the operation between the auxiliary filters is realized.

FIG. 15 shows the implementation of the interpolation filter operation, and the above description does not accurately realize the conversion of the data rate. In other words, the input and output data rates inevitably differ when the number of pixels in an image is converted, but it is usually desirable that each of the input and output data has a steady, constant speed data stream as image data. It is. However, for example, in the conversion example in which the number of pixels is increased in the table of FIG. 16, although the output is constant, the input data string needs to run or stop. Conversely, in the case of conversion in which the number of pixels is reduced, although the input is constant, an output data string is output or not output. This problem is related to the problem of the data flow speed, which is a buffer memory for stabilizing the data flow or intermittently maintaining a constant data speed before or after the configuration as shown in FIG. Or FIFO
It is easily realized by arranging the memory. The configuration of the buffer memory for this purpose is a matter of memory control, and the description is omitted.

[0050]

As described above, the conventional conversion of the number of pixels is performed by using a fixed product-sum operation circuit as shown in FIG.

However, in the case of such a fixed product-sum operation circuit, for example, when it is desired to change the number of taps or the conversion ratio, it is necessary to start over from circuit design, and there is a problem that it takes time and effort. In addition, when it is necessary to perform the pixel number conversion at a plurality of conversion ratios, there is a problem that a plurality of pixel number conversion devices corresponding to each conversion ratio must be prepared.

Therefore, the present invention has been made in view of such a situation. For example, when it is desired to change the number of taps or the conversion ratio, it is possible to change them easily and in a short time. It is an object of the present invention to provide an image signal processing method and apparatus that do not need to prepare a plurality of pixels corresponding to each conversion ratio even when it is necessary to perform pixel number conversion at a ratio.

[0053]

An image signal processing method and apparatus according to the present invention performs a filter operation on a two-dimensional image composed of a plurality of pixels arranged in a first direction and a second direction intersecting the first direction. An image signal processing method and apparatus, which generates a filter program and a filter coefficient required for a filter operation for both first and second directions based on filter specification information, and sequentially and serially inputs them in a pixel order in a first direction. The pixel data thus converted is converted into parallel, and a predetermined filter operation process using a filter program and a filter coefficient for the first direction is performed on each pixel data of the parallel-converted pixel row in the first direction, Subsequent to each pixel data of the pixel column in the first direction, a second
The above-described problem is solved by performing a predetermined filter operation process using a filter program and a filter coefficient for the direction, and sequentially outputting the obtained parallel pixel data in the order of pixels in the second direction.

Here, according to the image signal processing method and apparatus of the present invention, when a two-dimensional image is enlarged or reduced, the two-dimensional enlargement or reduction filter operation according to the enlargement or reduction setting is performed. A necessary filter program and a filter coefficient are generated, and pixel data sequentially input in the pixel order in the first direction or skip information for skipping between the pixel data is generated, and the number of input pixel columns in the second direction and Timing information for controlling the enlargement or reduction ratio of the number of output pixel columns is generated.

That is, according to the present invention, the filter program and the filter coefficient are automatically set based on the filter specification information, and the filter operation is performed based on the automatically set filter program and filter coefficient. . Further, according to the present invention, the two-dimensional image is divided into a first direction and a second direction, and the first direction and the second direction are considered.
Since the filter operation is performed separately for each of the directions, the filter program and the filter coefficient for the filter operation can be designed easily and efficiently. Further, by separating the filter operations in this way, the independence and maintainability of the filter operations in the first direction and the second direction are improved.

[0056]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described below with reference to the drawings.

First, a description will be given of a method of realizing the interpolation filter operation for the pixel number conversion processing by a software program process instead of the hardware configuration as shown in FIG.

As a configuration for performing an interpolation filter operation by the software program, for example, a so-called SIMD (Single Instruction Multip
le Data Stream) controlled linear array type multi-parallel processor 120 is known. This processor 120
Is realized by a so-called DSP (digital signal processor). This processor 120 is roughly divided,
An input SAM (serial access memory) unit 121, a data memory unit 122, and an ALU (arithmetic and logical
(unit) An array unit 123, an output SAM unit 124, and a program control unit 125.

The input SAM section 121, the data memory section 122, the ALU array section 123, and the output SAM section 124
Constitutes a group of element processors which are parallelized in a large number in a linear array (linear array) as a whole, and these many element processors are linked by one common program control unit in the program control unit 125. Control (SIMD
Control). The program control unit 125 includes a program memory and a sequence control circuit for advancing the program. The program control unit 125 generates various control signals connected to each part according to a program written in advance in the program memory, and generates each control signal. Control.

The input SAM unit 121, the data memory unit 122, and the output SAM unit 124 are basically memories, and the "ROW" address decoding for these memories will not be described in detail, but will be described with reference to FIG. Will be described below as being included in the program control unit 125.

A single element of a multi-parallel element processor is a vertically elongated area as shown by hatching in FIG. 1 and is arranged in a horizontal linear array in the figure. That is, each of the vertically elongated element processors shown by oblique lines in FIG. 1 satisfies general configuration requirements as shown in FIG. 2 necessary to be generally called a processor.
The input SAM unit 121 in FIG. 1 corresponds to the input buffer memory (IQ) 130 in FIG. The output SAM unit 124 in FIG. 1 corresponds to the output buffer memory (OQ) 131 in FIG. The data memory unit 122 in FIG. 1 corresponds to the data memory (RF) 132 in FIG. A in FIG. 1 corresponds to the ALU 133 for selecting and reading data from the data memory 132 as needed and performing an operation.
This is the LU array unit 123.

In the configuration shown in FIG. 2, the input data supplied to the input terminal D _IN is temporarily stored in the input buffer memory (IQ) 1.
30 and from there a data memory for storage
(RF) 132 is transferred by the required amount. The latest data currently stored in the data memory 132, data stored in the past, data in the middle of calculation, and the like are selected and read as necessary, and are then calculated by the ALU 133.
(RF) 132 is repeated under program control, and the resulting operation result is stored in a buffer memory.
(OQ) 131 and output at a predetermined speed and format. The configuration in FIG. 2 is a general-purpose processor that can be used for various purposes.

In an ordinary processor, the hardware is generally a word processor, which processes in units of words. However, in one element processor shown by a vertically elongated area shown by hatching in FIG.
1. The data memory unit 122 and the output SAM unit 124 are “columns” of the memory.
Reference numeral 3 denotes a 1-bit ALU, which is essentially a circuit mainly based on a full adder (full adder). Therefore, unlike a normal processor, it is a bit processing processor and performs processing in units of bits. An 8-bit machine or a 16-bit machine in a normal CPU (central processing unit) is a 1-bit machine. Since the bit processing processor has small hardware and can realize a large number of parallel numbers that cannot be normally realized, for an image, the parallel number of the element processor linear array is determined by the number of pixels (H) in one horizontal scanning period of the video signal. ).

Further, the structure of the element processor can be written as shown in FIG. 3 as an example. However, the structure of each cell in FIG. 3 is a very general structure for easy understanding, and a portion where a plurality of the same circuits are arranged is represented by the element circuit.

One element processor of the input SAM unit 121 of FIG. 1 is a plurality of input SAM cells 121B arranged in a vertical direction under the control of the input pointer 121A. Although the input data bus and the input SAM cell 121B are prepared by arranging them vertically in the number of bits (ISB) of the input signal in FIG. 1, they are omitted in FIG. doing. 1. One element processor of the data memory unit 122 in FIG. 1 corresponds to the memory cell 122 in FIG.
A is the number of bits of the capacity of the data memory unit 122 in FIG.
(MB) Although they are prepared and arranged vertically, FIG. 3 omits them and displays only one cell as a representative. As many bits (MB) as the number of bits described above are prepared as necessary as working memory for arithmetic processing. The memory cell 122A of the data memory unit 122 is a three-port memory having two read bit lines and one write bit line, and can simultaneously provide two sets of read addresses and one set of write addresses. It has become. ALU array unit 1
The ALU cell 123A in FIG. 3 corresponds to one element processor 23. Here, the net ALU 123B portion in the ALU cell 123A is a 1-bit ALU, and is a circuit of about a full adder (full adder).
There are also selectors for ALU input selection etc.
(SEL) circuit and the like. Note that F in the ALU cell 123A
The part indicated by F is a flip-flop (1-bit register)
It is. One element processor of the output SAM unit 124 is a plurality of output SAM cells 124B arranged in a row under the control of the output pointer 124A. Although the output data bus and the output SAM cell 124B are provided in a vertical arrangement for the number of output signal bits (OSB) in FIG. 1, they are omitted in FIG. ing.

The input SAM read signal S _IR , the memory write access signal S _WA , the memory read access signals S _RAA and S _RBA , and the output SAM write signal S _{OW shown} in FIG. , And the same circuit elements arranged in the horizontal direction while being connected in the same manner. It is assumed that the word lines of these memory cells have been address decoded in the program control unit 125 of FIG.

Further, in FIG. 3, it is assumed that connection lines, that is, bit lines and pointer signal lines, which vertically pass through the cell, pass while connecting circuit elements arranged in the vertical direction in the same manner. The input data bus passes through the same circuit elements arranged side by side, that is, the input SAM cell 121B, while being similarly connected. The output data buses also pass through the same circuit elements arranged side by side, that is, the output SAM cells 124B while being similarly connected.

Next, the operation of the processor 120 will be described with reference to FIG.
This will be described with reference to FIG.

The input video signal data supplied to the input terminal D _IN is guided to the input SAM section 121 via the input data bus. Input pointer 121A generates a 1-bit signal or input pointer signal S _IP made a logic "H" only one element processor, the input video signal data into the input SAM cell 121B of the specified element processors with a logic "H" Is written. Input SAM specified by pointer
In the cell 121B, transistor Tr ₁ is turned ON, become potential capacitor C ₁ is in accordance with the input video signal data.

[0070] Input Specifies the input pointer signal S _IP is so moved from the left edge of the element processors to each of the input video signal clock in order in the right edge of the element processor, the input video signal data is input SAM cell 121B of the leftmost element processor
The storage location is sequentially moved to the SAM cell of the element processor in the right direction from the right, and the number of element processors arranged horizontally is equal to or more than the number of pixels (H) in one horizontal scanning period of the input video signal. The scanning period can be stored in the input SAM unit 121. Such an input operation is repeated every horizontal scanning period.

The program control unit 125 outputs the data of one horizontal scanning period of the input video signal to the input SAM.
Each time it is stored in the unit 121, the input S
The AM unit 121, the data memory unit 122, the ALU array unit 123, and the output SAM unit 124 are SIMD-controlled as described below to execute processing. This program control is repeated every horizontal scanning period. A program can be constructed by the number of steps obtained by dividing the time of the horizontal scanning period by the instruction cycle period of this processor. Because it is SIMD control,
The following operation is similarly executed in all the element processors at the same time.

The input video signal data accumulated in the input SAM unit 121 for one horizontal scanning period is transferred from the input SAM unit 121 to the data memory unit 122 as necessary in the next one horizontal scanning period, and thereafter, Used for arithmetic processing.

[0073] The first transfer operation from the input SAM unit 121 to the data memory unit 122 accesses to select the input SAM read signal S _IR storage contents of the necessary bits in the input SAM unit 121, the transfer destination of the data Memory unit 12
This is realized by outputting and writing the memory access signal _SWA to two predetermined memory cells. In the input SAM read signal S input SAM cell 121B selected by the _IR, the transistor Tr ₂ is a ON, the capacitor C ₁
A transfer data signal corresponding to the potential of the vertical write bit line is generated on the vertical write bit line. At this time, the signal S _BC is enabled and the data memory unit 122 selected by the memory access signal S _WA
Transistor Tr ₁₁ of a given memory cell is turned ON, become potential capacitor C ₁₁ is corresponding to the transfer data.
This data transfer is performed one bit at a time via a vertical write bit line in one cycle. Input SAM unit 1
Read signal S from each of the 21 input SAM cells 121B
Memory access signal S _WA to each of the memory cells 122A of the _IR and the data memory unit 122 is in the same address space, but are given as being decoded word line in each row decoder.

The data memory unit 122 stores input data written in the past as described above and data in the middle of calculation. Their data or ALU cell 1
By using the data stored in the 1-bit register (FF) in the 23A, the necessary arithmetic processing in the ALU in bit units can be sequentially advanced. For example, to add the data of the memory cell 122A of one bit of the data memory unit 122 and the data of the memory cell 122A of another bit and write the addition result to the memory cell 122A of another bit of the data memory unit 122 again, become that way. Out of the data memory unit 122 of a certain readout signal S _RAA to bits of the memory cells 122A, the transistor Tr ₁₃ of the cell in ON, the issue data stored in the capacitor C ₁₁ to one of the read bit line. At the same time issues a read signal S _RBA also to another bit memory cell 122A, the transistor Tr ₁₄ of the cell in ON, the issue data stored in the capacitor C ₁₁ to the other read bit line. These two read data are caused to cause the selector (SEL) of the ALU array unit 123 to select a predetermined path, to cause the ALU 123B to add,
Output from U123B is to enable the signal S _BCA out to the bit line for writing the operation result data, and a predetermined write memory cells of the data memory unit 122 122
And the transistor Tr ₁₁ of the cell in ON issues a write memory access signal S _WA to A, becomes a potential capacitor C ₁₁ is corresponding to the ALU output data. The arithmetic operation in the ALU cell 123A is designated from a program by the ALU control signal S _ALU-CONT . The result calculated by the ALU cell 123A can be written to the data memory unit 122 again, or can be stored in the 1-bit register (FF) in the ALU cell 123A as needed. In the case of addition, the carry is usually led to the bit register (FF), and the sum is led to the data memory unit 121.

As described above, data is read from the data memory unit 122 in accordance with the program, and the ALU array unit 123 performs a necessary arithmetic operation or logical operation.
The data can be written to a predetermined address of the data memory unit 122 again. This arithmetic processing is all bit processing,
The process proceeds one bit per cycle. For example, in order to execute a logical operation, if it is 8-bit data, the process proceeds one bit at a time, so that it takes eight cycles inevitably. In addition, the addition of 8-bit data necessarily takes 9 cycles. Multiplication between 8-bit data is equivalent to 64 bit additions, so that it is simply realized over 64 cycles.

When the arithmetic processing to be processed is completed within the time of one horizontal scanning period, the output SAM unit outputs the arithmetically processed output data for the horizontal scanning period in the last part of the program during the horizontal scanning period. Need to move to 124. A memory access signal S _RAA is sent to a predetermined memory cell 122A in which data to be output from the data memory unit 122 is stored.
Alternatively, the transistor Tr ₁₃ or the transistor Tr ₁₄ of the cell is _{turned on} by _turning on the S _RBA and the capacitor C ₁₁
Is output to the read bit line.
A control signal to be passed is output to the ALU array unit 123, and an output SAM cell 1 of a predetermined bit is output to the output SAM unit 124.
SAM cell 1 so that data is transferred to 24B.
The write signal S _OW is issued and 24B, the transistor Tr ₁₇ of the cell is turned ON, the potential of the capacitor C ₄ is according to the data. Data is transferred one bit at a time via a vertical bit line. At this time, the ALU may or may not perform some processing when data is moved. The write signal S _OW to each output SAM cell 124B of the output SAM unit 124 and the data memory unit 122
Are in the same address space, but are decoded by respective row decoders and provided as word lines.

As described above, during the time of one horizontal scanning period, the input data stored in the input SAM section 121 is moved to the data memory section 122, necessary arithmetic processing, and output to the output SAM section 124. Up to data movement is controlled and executed by a SIMD control program in units of bits. This program processing is repeated for each horizontal scanning period. Because of the SIMD control, all the element processors operate in conjunction, and the number of pixels for the horizontal scanning period
The same processing is performed for (H).

The output data transferred to the output SAM unit 124 after the completion of the program processing in units of the horizontal scanning period is output from the output SAM unit 124 in the next horizontal scanning period as follows. Output video signal data is output SA
Output terminal D _OUT from M section 124 via output data bus
Led to.

[0079] The output pointer 124A is a 1-bit signal or the output pointer signal S _OP made a logic "H" only one element processor generates, from the output SAM cell 124B of element processors specified in the logical "H" The output data is read out to the output data bus and becomes output video signal data. Transistor Tr ₈ In the output SAM cell 124A designated by the pointer is turned ON, an output signal corresponding to the potential of the capacitor C ₄ is obtained in the output data bus. The logic "H" of the output pointer signal moves from the leftmost element processor to the rightmost element processor every clock of the output video signal, so that the output data is read out from the output SAM cell 124B of the leftmost element processor.
From the right to the SAM cells of the element processors in the right direction. Since the number of element processors arranged horizontally is equal to or more than the number of pixels (H) in one horizontal scanning period of the video signal, one horizontal scanning period of the output video signal is used. Minute data can be output from the output SAM unit 124. Such an output operation is repeated every horizontal scanning period.

Here, 1. An input operation by writing input data to the input SAM unit 121.

2. Movement of input data accumulated in the input SAM unit 121 to the data memory unit 122, necessary arithmetic processing, and movement of output data to the output SAM unit 124 by SIMD control of the program control unit 125.

3. An output operation by reading output data from the output SAM unit 124.

The three operations are pipeline operations in units of one horizontal scanning period of a video signal. If attention is paid to input data in one horizontal scanning period, each operation is performed in one horizontal scanning period. Although performed in a time-shifted manner, the three operations can proceed sequentially and simultaneously in parallel.

In such an image linear array type multi-parallel processor, one element processor is always in charge of any horizontal scanning period at the same pixel position in the horizontal direction. In order to store the data in the data memory unit 122 until the subsequent horizontal scanning period, when the data is transferred from the input SAM unit 121 to the data memory unit 122, the address is shifted for each horizontal scanning period. It is known that each of the element processors can have a predetermined continuous number of pixel data in the vertical direction in the data memory unit 122, thereby realizing the FIR digital filter operation in the vertical direction.

Although not mentioned in the description of FIGS. 1 and 3,
In the element processor row, a connection for communication between processors is prepared between several adjacent processors. Specifically, there is a connection line between the data memory unit 122 and the ALU array unit 123 in FIG.
Where the bit line on the memory cell 122A side of 23A is input / output to the selector SEL in the ALU cell 123A,
The bit line connection can be selected not only from the bit line of the data memory unit 122 in the own element processor but also from the bit lines of the data memory unit 122 of the left and right adjacent processors. Each element processor can access the data memory unit 122 of some adjacent element processors on the left and right. As a result, a predetermined continuous number of pixel data can be handled in the horizontal direction, and the FIR digital filter operation in the horizontal direction can be realized.

However, the access to the data memory unit 122 of some of the adjacent element processors on the left and right
Since the control is the D control, for example, in a cycle in which the data memory unit 122 of the right element processor is accessed, all the element processors are accessing the data memory unit 122 of the right element processor. Of course, this does not hinder the implementation of ordinary FIR digital filters. For data access of a slightly distant element processor without direct interprocessor communication connection, the number of program steps can be increased somewhat by repeating communication between nearby processors.

As described above, the interpolation filter operation for the pixel number conversion processing can be realized by the software program processing by the SIMD-controlled linear array type multi-parallel processor.

Here, the configuration of the image signal processing apparatus of the embodiment of the present invention described above with reference to FIG. 1 is configured in more detail as shown in FIG.

That is, in the image signal processing apparatus of this embodiment, when performing image processing using the linear array type multi-parallel processor, the horizontal and vertical filters are independently executed by two processor element groups. In this way, by adopting a separate filter configuration, efficient horizontal
The vertical filter can be realized by a general one-dimensional filter design technique. In addition, the configuration in which this is executed by two independent processor element groups can enhance the independence, maintainability, and the like of the program and make it easy to use.

The linear array type multi-parallel processor 20 of FIG. 4, which shows FIG. 1 in more detail, includes a plurality of element processors (processor elements: PE) 23.
Each element processor 23 forms two upper and lower processor element groups 21 and 22. The number of element processors 23 of the processor element groups 21 and 22 is at least larger than the number of pixels constituting one horizontal line of the image,
The processor element group 21 is provided for horizontal use, for example, and the processor element group 22 is provided for vertical use, for example. These processor element groups 21 and 21
Each of the two element processors 23 has the functions of the data memory unit 122 and the ALU array unit 123 in FIG.

The controllers 26 and 28 and the coefficient memories 27 and 29 correspond to the program controller 125 shown in FIG.

The controller 26 performs the above-described SIMD control of the processor element group 21 and the coefficient memory 27, and the coefficient memory 27 has a function of supplying data necessary for calculation to the processor element group 21. All the element processors 23 included in the processor element group 21 perform the same arithmetic processing based on the control signal of the controller 26.

The controller 28 and the coefficient memory 29
Has a similar relationship. That is, the controller 28 includes the processor element group 22 and the coefficient memory 29
Is controlled by SIMD, and the coefficient memory 29 has a function of supplying data necessary for calculation to the processor element group 22. All the element processors 23 included in the processor element group 22 perform the same arithmetic processing based on the control signal of the controller 28.

As described above, since the two systems of the controllers 26 and 28 are provided, the processor element groups 21 and
22 can operate independently.

The input register 24 corresponds to the input SAM unit 121 of FIG. 1 and is a memory for performing serial-to-parallel conversion. In the input register 24, image data is sequentially input one pixel at a time, and is read out in parallel when one line is completed. At the time of inputting pixel data to the input register 24, for example, it is also possible to operate as an upsampler that expands the total number of pixels by inserting zeros at arbitrary locations, instead of filling them without gaps.

The output register 25 corresponds to the output SAM unit 124 in FIG. 1 and is a memory for performing parallel-to-serial conversion. The output register 25 inputs pixel data for one line in parallel, and sequentially outputs the pixel data one pixel at a time. When the pixel data is output from the output register 25, for example, it is possible to operate as a downsampler for skipping pixel data at an arbitrary place and reducing the total number of pixels.

Next, the operation of the linear array type multi-parallel processor 20 shown in FIG. 4 will be described.

In FIG. 4, pixel data for one line in the horizontal direction is sequentially input from the image input terminal 10 to the input register 24, and then supplied to the upper processor element group 21 in parallel. The processor element group 21 performs a filter operation on each pixel data received by each element processor 23 using a control command from the controller 26 and coefficient data from the coefficient memory 27. The processing result by the upper processor element group 21 is directly input to the lower processor element group 22 in parallel, and if necessary, further filter operation processing is performed. The processing result by the lower processor element group 22 is input in parallel to the output register 25, and is sequentially output one pixel at a time to obtain a processing result for one line.

As described above, the linear array type multi-parallel processor 20 can process pixels forming one line in the horizontal direction at a time. If this is repeated by the number of lines in the vertical direction, two-dimensional image processing can be performed. Since the specific operation of the processor 20 has already been described above, a detailed description thereof will be omitted here.

Next, the processor element group 21,
The content of the filter operation process by the filter 22 will be described. Since the applicant has already disclosed a method of realizing a horizontal filter and a vertical filter using the linear array type multi-parallel processor 20, it will be briefly described here.

Each of the element processors 23 constituting the processor element groups 21 and 22 can communicate with the element processors 23 located on the left and right sides of each other, so that an operation using a plurality of pixel data on the same line can be performed. it can.
Further, as described above, since the element processor 23 includes a memory for holding pixel data of a plurality of lines as necessary, the element processor 23 can perform an operation across lines.

Here, as an example of the above-mentioned pixel number conversion, a case where the number of pixels is reduced by a factor of 2/3 in both the horizontal and vertical directions is taken as an example. As shown in FIG. 5A, from an image of 9 pixels in the horizontal direction and 6 pixels in the vertical direction, for example, as shown in FIG.
It is like generating an image of 6 pixels in the horizontal direction and 4 pixels in the vertical direction. As described above, the image of FIG.
In order to generate the (B) pixel shown in FIG.
And a process of generating an interpolated pixel Y from the _four original pixels D _{1 to} D _{4 in} FIG. 5A as in equation (a) in FIG.
This is performed for each pixel in the image of FIG. 5A, and the image of FIG. 5B is generated from the obtained interpolated pixels Y.

In the example of the pixel number conversion shown in FIG. 5, if the input pixel data for two lines are aligned, the output pixel can be calculated.
In addition, the number of pixels is converted by 2/3 times in the horizontal and vertical directions. In the horizontal direction, the number of pixels is reduced to 2/3 by using the skip function of the output register 25. On the other hand, in the vertical direction, by alternately repeating “input two lines and output one line” and “input one line and output one line”, the number of output lines is ／ of the number of input lines.
become.

Here, among the two-dimensional filters, the horizontal and vertical separation type filters as described above can be realized by two one-dimensional filters. This separable filter has the advantages of reducing the number of necessary arithmetic units and being able to use many one-dimensional filter design techniques.
It is extremely important in practical use. In brief, for example, a filter (Y = C ₁ D ₁ + C ₂₎ for performing pixel number conversion as shown in FIG.
(D ₂ + C ₃ D ₃ + C ₄ D ₄ ) is (C ₁ = C ₅ C ₇ , C ₂ = C ₆ C)
_7, C ₃ = when _{C 5 C 8, C 4 =} C 6 C 8) and the filter coefficients C ₅ -C ₈ such that there _{is, (H 1 = C 5 D} 1 + C
₆ D ₂ , H ₂ = C ₅ D ₃ + C ₆ D ₄ )
(Y = C ₇ H ₁ + C ₈ H ₂ ).

FIG. 6 shows a program for executing the horizontal and vertical filters as described above independently in the two processor element groups 21 and 22, coefficients, control information for converting the number of pixels, and the like. It can be automatically generated by such a flow. That is, if the above-described program, coefficients, and the like are automatically generated according to the flow shown in FIG. 6, for example, only by determining the specification of the filter, a pixel number conversion device based on the filter specification can be realized. In this case, it is possible to perform various types of filter processing including pixel number conversion simply by exchanging the programs and coefficients generated in the flow of FIG. 6 without changing the circuit.

In FIG. 6, first, a filter design circuit 3 based on a filter specification 30 for designing, for example, the number of taps and a conversion ratio for designing a desired filter.
3 generates horizontal and vertical filter programs 34, 36 and filter coefficients (filter coefficient sets) 35, 37. The horizontal filter program 34 is sent to the controller 26 via the terminal 41 and the terminal 11 in FIG. 4, and the horizontal filter coefficient 35 is sent to the coefficient memory 27 via the terminal 42 and the terminal 12 in FIG. Also,
The vertical filter program 36 is sent to the controller 28 via the terminal 43 and the terminal 13 in FIG. 4, and the vertical filter coefficient 37 is sent to the coefficient memory 29 via the terminal 44 and the terminal 14 in FIG. The filter coefficient sets 35 and 37 may use a general one-dimensional filter design tool, or may be selected from a library. In the case where the scaling of an image involves a change in the number of pixels, the processor element assignment circuit 31 generates processor element skip information 32 for the data input register 24 and the data output register 25, and adjusts the number of pixels in the horizontal direction. The skip information 32 is sent to the terminal 15 of FIG. 4 via the terminal 45 and sent to the input register 24 or the output register 25. In the vertical direction, the timing design circuit 38 generates timing information 39 for controlling the ratio of the number of input lines to the number of output lines, and this is incorporated in the vertical filter program 36. In FIG. 4, the upper and lower processor element groups 2
Although the horizontal and vertical filters are assigned to 1 and 22, respectively, the same processing result can be obtained by performing the opposite assignment.

Next, a specific operation of the processor element assignment circuit 31 will be described.

The processor element assignment circuit 31
Generates skip information 32 for converting the number of pixels in the horizontal direction. The skip information 32 acts to convert the number of horizontal pixels based on the conversion ratio set by the filter specification 30 in the horizontal filter program 34. That is, the ratio of the number of input and output pixels is Ni: No (N
i and No are relatively prime), the conversion ratio R = No /
The operation is divided into enlargement and reduction by Ni.

First, the case where R> 1 (enlargement) will be described.

In the case of R> 1, the skip information to be set in the input register 24 is, for example, normal operation if (i mod Ni) <No with respect to the i-th input register from the left, otherwise, Set skip information such as skip operation. However, (i mod
Ni) represents the remainder of i divided by Ni. For example, 3
When performing the enlargement of / 2 times, the input register 24
The following skip information 32 is sent (however, ○ indicates a normal operation and X indicates a skip operation).

When data such as abcdefghi... Is input from the image input terminal 10 to the input register 24, ab0cd0ef0gh0i. And passed to the processor element group 21 in a form in which the number of data is increased by 3/2 times. At the position where the zero of the data stored in the input register 24 is inserted, an interpolation pixel is later supplemented by the horizontal filter program 34.

Next, the case where R <1 (reduction) will be described.

In the case of R <1, the skip information to be set in the output register 25 is, for example, normal operation if (i mod No) <Ni with respect to the i-th output register from the left. If not, skip information such as performing a skip operation is set. However, (im
od No) represents the remainder of dividing i by No. For example,
To reduce the size by a factor of 2/3, the following skip information 32 is sent to the output register 25 (note that ○).
Represents a normal operation, and x represents a skip operation).

..,... Here, when data abcdefghi... Are input from the processor element group 22, the output register 2
5 output a data string such as abdeg..., And the number of data is reduced to 2/3.

Next, a specific operation of the filter design circuit 33 shown in FIG. 6 will be described.

The filter design circuit 33 obtains a coefficient set for the interpolation operation. This coefficient set is used as a horizontal filter coefficient 35 and a vertical filter coefficient 36. For example, a so-called cubic function can be used as the interpolation function, and an example using the cubic function will be described below.

The cubic function Cub (x) is represented by the following equation (5). Expression (5) is the same expression as expression (4). Here, it is assumed that the horizontal axis of the cubic function shown in Expression (5) is normalized by the sampling interval for sampling the original image into a digital signal.

Cub (x) = | x | ³ −2 | x | ² +1 (when | x | ≦ 1) Cub (x) = − | x | ³ +5 | x | ² −8 | x | +4 ( 1 <| x | ≦ 2) Cub (x) = 0 (when 2 <| x |) (5) In this case, the number of taps of the filter is 4, and as shown in Expression (6), , the interpolation pixel Y is two left and right portions of the original pixels (e.g. the four original pixel D ₀ to D _3), the filter coefficient set C ₀ -C ₃ corresponding to these four original pixels D ₀ to D ₃ By the product-sum operation of.

Y = C ₀ * D ₀ + C ₁ * D ₁ + C ₂ * D ₂ + C ₃ * D ₃ (6) The filter coefficient set is determined as follows.

For example, when the conversion ratio is 3: 2, the positional relationship (phase relationship) between the original pixel D and the interpolated pixel Y is as shown in FIG.

In FIG. 7, the interpolation pixel Y ₀ and the original pixel D ₀ have the same phase, and the interpolation pixel Y ₁ is the original pixel D ₁ ,
It is shifted from D ₂ by １ ／. Since the interpolation pixel Y ₂ thereafter are the more repetitions, it is required two different coefficient sets. For example, interpolated pixel Y for example C (0.0) the coefficient set corresponding to the _0, ..., and C (0, 3), the coefficient set corresponding to the interpolated pixel Y ₁ C (1, 0), ... , C (1,
3) When, C (0,0) = Cub (position of D _-1 viewed from _{Y 0) = Cub (-1)} = 0 C (0,1) = Cub ( a D ₀ as viewed from the Y ₀ Position) = Cub (0) = 0 C (0,2) = Cub (position of D ₁ viewed from Y ₀ ) = Cub (1) = 0 C (0,3) = Cub (D viewed from Y ₀ ) ₂ ) = Cub (2) = 0 C (1,0) = Cub (the position of D ₀ viewed from Y ₁ ) = Cub (−3/2) = − 0.125 C (1,1) = Cub ( position of D ₁ as viewed from the _{Y 1) = Cub (-1/2)} = 0.625 C (1,2) = Cub ( position of D ₂ as viewed from the _{Y 1) = Cub (1/2)} = 0.625 C ( 1, 3) = Cub (position of D ₃ as viewed from the _{Y 1) = Cub (3/2)} = - 0.125 filter design circuit 33 is configured with a program for determining the above coefficients set.

The horizontal filter program 34 and the vertical filter program 36 calculate the interpolation pixel Y using the coefficient set obtained by the filter design circuit 33 as follows.

Y ₀ = C (0,0) × D ₋₁ + C (0,1) × D ₀ + C (0,2) × D ₁ + C (0,3) × D ₂ Y ₁ = C (1 , 0) × D ₀ + C (1,1) × D ₁ + C (1,2) × D ₂ + C (1,3) × D ₃ Y ₂ = C (0,0) × D ₁ + C (0,1 ) × D ₂ + C (0,2) × D ₃ + C (0,3) × D ₄ Y ₃ = C (1,0) × D ₂ + C (1,1) × D ₃ + C (1,2) × Similarly, D ₄ + C (1,3) × D ₅ shows an example where the conversion ratio is 4: 3. In this example, there are three types of phase relationships between the original pixel D and the interpolated pixel Y, and therefore three types of coefficient sets C (0,0),..., C (0,3), C (0,3)
, C (1,3), C (2,0),..., C (2,3) are obtained.

Also in this example, the ratio of the number of input / output pixels is N
When i: No (Ni and No are relatively prime), No type coefficient sets are required.

In the case of 4: 3, the original pixel D and the interpolation pixel Y
The positional relationship (phase relationship) is as shown in FIG.

Here, also in the 4: 3 example, the coefficient sets are set to C (0,0),..., C (0,3) to C (2,
0),..., C (2,3), C (0,0) = Cub (the position of D _-1 as viewed from Y ₀ ) = Cub (-1) = 0 C (0,1) = Cub (position of D ₀ as viewed from Y ₀ ) = Cub (0) = 0 C (0,2) = Cub (position of D ₁ as viewed from Y ₀ ) = Cub (1) = 0 C (0, 3) = Cub (position of D ₂ as viewed from Y ₀ ) = Cub (2) = 0 C (1,0) = Cub (position of D ₀ as viewed from Y ₁ ) = Cub (−4/3) = -0.148 C (1,1) = Cub (position of D ₁ as viewed from the _{Y 1) = Cub (-1/3)} = 0.815 C (1,2) = Cub ( position of D ₂ as viewed from the Y ₁₎ from = Cub (2/3) = 0.407 C (1,3) = Cub ( position of D ₃ as viewed from the _{Y 1) = Cub (5/3)} = -0.074 C (2,0) = Cub (Y 2 position of D ₁ as viewed) = Cub (-5/3) = - 0.074 C (2,1) = Cub ( position of D ₂ viewed from the _{Y 2) = Cub (-2/3)} = 0.407 C (2 , 2) = Cub (position of D ₃ viewed from Y ₂ ) = Cub (1/3) = 0.815 C (2,3) = Cub (position of D ₄ viewed from Y ₂ ) = Cub (4/2 ) = − 0.148 The operation of the timing design circuit 38 in this case is as follows.

The timing design circuit 38 generates timing information 39 for converting the number of pixels (the number of lines) in the vertical direction. The timing information 39 acts to convert the number of input / output scanning lines in the vertical filter program 36 based on the conversion ratio of the filter specification 30. The ratio of the number of input / output lines is Ni: No (N
i and No are relatively prime), the conversion ratio R = No /
The operation is divided into enlargement and reduction by Ni.

First, the case where R> 1 (enlargement) will be described.
That is, the output has more lines than the input.

For example, if (i modNi) <No in the i-th input line, the timing design circuit 38 generates timing information 39 in the order of line input → interpolation processing → line output, and (i mod Ni) ≧
If No, the line output is performed without waiting for the line input. Therefore, the timing information 39 is generated in the order of the interpolation calculation processing and the line output.

For example, when performing 3 / 2-fold enlargement, the following timing signal 39 is generated. The vertical filter program 39 performs processing in this order.

[0131] As described above, the output becomes three lines for every two input lines, and the number of lines increases in the vertical direction by 3/2 times.

Next, R <1 (in the case of reduction) will be described. That is, the output has fewer lines than the input.

In the i-th input line, (i mo
d Ni) <No, line input → interpolation calculation processing →
Generates timing information 39 in the order of line output,
If (imod Ni) ≧ No, the timing information 39 of only the line input is generated in order to perform one more line input.

For example, when performing 、 ２ magnification, the following timing signal 39 is generated. The vertical filter program 39 performs processing in this order.

Line input → interpolation calculation processing → line output → line input → interpolation calculation processing → line output → line input → line input → interpolation calculation processing → line output → line input → interpolation calculation processing → line output → line input → ... As described above, the output becomes 2 lines for every 3 input lines, and the number of lines is increased by 2/3 times in the vertical direction.

Next, the operation of the horizontal filter program 34 will be described. That is, the upper processor element group 2
1, the horizontal filter program 34 is executed as shown in the flowchart of FIG.

In FIG. 9, skip setting is performed in step ST1. That is, the skip operation is set in the input register 24 and the output register 25 according to the skip information 32.

At step ST2, an area is secured.
That is, each element processor stores an area c ₁ for storing coefficient sets C _{0 to} C ₃ for four taps in the local memory.
To ensure the ~c _4.

In step ST3, a coefficient is set. For example, in the case of 3: 2 reduced pixel number conversion, the following is performed. For example, from the left 1, 3, 5, ..., (i + 2)
The th element processor has a coefficient set C (0,0),.
, C (0,3) is stored in the local memory from the coefficient memory. The (i + 1) -th element processor from the left is a coefficient set C (1,0),.
(1,3) is stored from the coefficient memory to the local memory.

In step ST4, data input is performed.
Each element processor stores the data (the data of the original pixel D) input from the input register 24 in the local memory area d.
To be stored.

In step ST5, an interpolation operation is performed. That is, each element processor performs the operation of the following equation (7) using the original pixel D and the coefficient C, and stores the result as an interpolated pixel Y in the area y. Where L ₁ (x),
L ₂ (x), R ₁ (x), and R ₂ (x) (x is a variable) are one left, two left, one right, and one right of the local memory area d of the element processor, respectively. Value (the value of the original pixel D).

Y = L ₁ (D) * C ₀ + D * C ₁ + R ₁ (D) * C ₂ + R ₂ (D) * C ₃ (7) In step ST 6 following step ST 5, Perform output. That is, each element processor transfers the result to the lower processor element group 22.

Then, the process returns to step ST4.

Next, the operation of the vertical filter program 36 will be described. That is, the lower processor element group 22 executes the vertical filter program 36 as shown in the flowchart of FIG.

In FIG. 10, an area is reserved in step ST11. In other words, each element processor stores areas d _{1 to} d ₄ for the original pixels D for four lines in the local memory.
To secure.

In step ST12, each element processor selects and executes the following operations (A) to (C) according to the timing information 39.

[0147] When this step ST12 (A) is selected, the line input is performed in step ST13, the area (d ₀ of the local memory input data (original pixel D) from the upper processor element group 21 ~ d ₃ ). The following procedure is executed to hold the latest four lines of data.

[0148] Copy the value of the original pixel D in the region d ₁ in the region d ₀ at step ST14. In step ST15, it copies the value of the original pixel D region d ₂ in the area d _1. In step ST16 in the area d ₂ copies the value of the original pixel D region d _3. Copying the value of the input data (original pixel D) in step ST17 in the region d _3.

When (B) is selected in step ST12, an interpolation operation is performed in step ST18. That is, the following equation (8) is calculated using the original pixel D and the coefficient C, and the result is stored in the area y as the value of the interpolation pixel Y.
Since the filter coefficients are common to all element processors,
Read directly from coefficient memory.

Y = D ₀ * C ₀ + D ₁ * C ₁ + D ₂ * C ₂ + D ₃ * C ₃ (8) Further, when (C) is selected in step ST12,
Line output is performed in step ST19. That is, each element processor stores the value of the area y (interpolated pixel Y) in the output register 25.

As described above, according to the image signal processing apparatus of the embodiment of the present invention, it is possible to automatically perform all of the pixel number conversion processing, and only to determine the specifications of the filter. A pixel number conversion device based on this can be realized. Further, according to the apparatus of the present embodiment, it is possible to perform filter processing including various kinds of pixel number conversion without changing the circuit simply by exchanging the program or the coefficient, and furthermore, by using the separation type filter, it is possible to efficiently perform the processing. Horizontal and vertical filters can be realized by a general one-dimensional filter design method, and the independence and maintainability of the program can be enhanced by the configuration in which the filter processing is executed by two independent processor elements. Easy to use.

[0152]

According to the present invention, a filter program and a filter coefficient required for the first and second two-way filter operations are generated based on the filter specification information, and are sequentially serially arranged in the order of pixels in the first direction. The input pixel data is converted into parallel, and a predetermined filter operation using a filter program and a filter coefficient for the first direction is performed on each of the parallel-converted pixel data in the pixel row in the first direction. Then, a predetermined filter operation using a filter program and a filter coefficient for the second direction is further performed as necessary on each pixel data of the pixel row in the first direction, and the obtained parallel pixels By serially outputting data in the order of pixels in the second direction, for example, when it is desired to change the number of taps or the conversion ratio, these changes can be easily and quickly made. Is possible, also, there is no need to prepare a plurality corresponding to the respective conversion ratio even when it is necessary to perform the conversion of the number of pixels in a plurality of conversion ratio.

That is, in the image signal processing method and apparatus according to the present invention, the filter program and the filter coefficient can all be automatically set based on the filter specification information. By simply determining the specifications, the program and the filter coefficient can be exchanged, and various types of filtering including pixel number conversion and scanning line conversion can be realized. Further, in the present invention, the two-dimensional image is divided into a first direction and a second direction, and the filter operation is performed separately for each of the first direction and the second direction. Therefore, a filter program and a filter coefficient for a filter operation can be easily and efficiently designed. further,
In the present invention, by separating the filter operations in this way, the independence and maintainability of the filter operations in the first direction and the second direction can be improved, and the filter operations can be easily used.

[Brief description of the drawings]

FIG. 1 is a block circuit diagram showing a basic configuration of a linear array type multi-parallel processor according to an embodiment of the present invention.

FIG. 2 is a block circuit diagram showing a basic configuration of a single element of an element processor of the linear array type multi-parallel processor.

FIG. 3 is a circuit diagram showing a specific structure of an element processor of the linear array type multi-parallel processor.

FIG. 4 is a block circuit diagram showing the linear array type multi-parallel processor according to the embodiment of the present invention in more detail;

FIG. 5 is a diagram used to explain an interpolation image calculation at the time of converting the number of pixels.

FIG. 6 is a diagram showing a flow of automatic design of a program and a filter coefficient in the linear array type multi-parallel processor according to the embodiment of the present invention.

FIG. 7 is a diagram used to explain the positional relationship between an original pixel and an interpolated pixel when converting the number of two pixels.

FIG. 8 is a diagram used to explain the positional relationship between an original pixel and an interpolated pixel at the time of 4: 3 pixel number conversion.

FIG. 9 is a flowchart showing a flow when the horizontal filter program is executed.

FIG. 10 is a flowchart showing a flow when a vertical filter program is executed.

FIG. 11 is a diagram used to explain enlargement of an image and conversion of a sampling frequency (the number of pixels).

FIG. 12 is a diagram used for describing reduction of an image and conversion of a sampling frequency (the number of pixels).

FIG. 13 is a diagram schematically illustrating a positional relationship between pixels in a horizontal direction, which is used for describing an interpolation filter operation for obtaining a pixel data value at a position not existing in the original image.

FIG. 14 is a diagram used to describe an interpolation function approximated to an ideal interpolation function.

FIG. 15 is a block circuit diagram illustrating an example of an image signal processing device having a conventional hardware configuration.

FIG. 16 is a table used to describe the operation of each unit of an image signal processing device having a conventional hardware configuration.

FIG. 17 is a table used for describing coefficients and the like handled by an image signal processing device having a conventional hardware configuration.

[Explanation of symbols]

20,120 Linear array type multi-parallel processor, 1
21 input SAM unit, 122 data memory unit, 123
ALU array section, 124 output SAM section, 125
Program control unit, 130 input buffer memory,
131 output buffer memory, 132 data memory, 133, 123B ALU, 121A input pointer, 121B input SAM cell, 122A memory cell, 123A ALU cell, 124A output pointer, 124B output SAM cell, 21, 22
Processor element group, 23 element processor, 2
4 input registers, 25 output registers, 26, 2
8 controllers, 27, 29 coefficient memory

 ────────────────────────────────────────────────── ─── Continued on the front page (72) Koji Aoyama, Inventor 6-35 Kita Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation

Claims (7)

[Claims] 1. An image signal processing method for performing a filter operation on a two-dimensional image composed of a plurality of pixels arranged in a first direction and a second direction intersecting the first direction. Generating a filter program and a filter coefficient necessary for a filter operation for both the first direction and the second direction, based on the order of pixels in the first direction constituting the two-dimensional image;
The sequentially serially input pixel data is converted into parallel for each pixel column in the first direction, and each pixel data of the parallel converted pixel column in the first direction is converted into the first direction. A predetermined filter operation process using a filter program and a filter coefficient is performed on each of the pixel data of the pixel row in the first direction on which the predetermined operation process has been performed. An image signal processing method comprising: performing a predetermined filter operation process using a filter program and a filter coefficient for the direction of (i), and sequentially outputting the obtained parallel pixel data in order of pixels in the second direction. 2. An enlargement or reduction of the two-dimensional image is set based on filter specification information. When the two-dimensional image is enlarged, an enlargement filter operation in both the first direction and the second direction is performed. To generate skip information for skipping between pixel data sequentially input in the order of pixels in the first direction, the number of input pixel columns in the second direction and output 2. The image signal processing method according to claim 1, wherein timing information for controlling an enlargement ratio of the number of pixel columns is generated. 3. An enlargement or reduction of the two-dimensional image is set based on the filter specification information. When the two-dimensional image is enlarged, a reduction filter operation in both the first direction and the second direction is performed. Generating a filter program and a filter coefficient necessary for generating skip information for skipping each pixel data in the first direction after the predetermined filter operation processing is performed;
2. The image information processing method according to claim 1, wherein timing information for controlling a reduction ratio between the number of input pixel columns and the number of output pixel columns in the second direction is generated. 4. An image signal processing apparatus for performing a filter operation on a two-dimensional image composed of a plurality of pixels arranged in a first direction and a second direction intersecting the first direction, wherein the filter specification information includes A filter program and a coefficient generating means for generating a filter program and a filter coefficient necessary for a filter operation in both the first direction and the second direction, based on the first direction and the second direction. Serial / parallel conversion means for outputting pixel data serially input in the order of pixels in parallel for each pixel column in the first direction; and each pixel of the parallel-converted pixel column in the first direction. A first filter operation unit that performs predetermined filter operation processing on the data using the first direction filter program and the filter coefficient; For each pixel data of the predetermined arithmetic processing is performed pixel column in the first direction by filtering calculation means,
A second filter operation unit for performing a predetermined filter operation process using the filter program and the filter coefficient for the second direction, if necessary, and parallel pixel data obtained from the second filter operation unit. And a parallel / serial conversion means for sequentially outputting serial data in the order of pixels in the second direction. 5. A skip information generating means for generating skip information for skipping between pixel data sequentially input in the order of pixels in the first direction, a number of input pixel columns and an output pixel column in the second direction. Timing information generating means for generating timing information for controlling the number enlargement ratio; when performing the enlargement of the two-dimensional image, the program and the coefficient generating means perform the first direction and the second direction; 5. The image signal processing apparatus according to claim 4, wherein a filter program and a filter coefficient necessary for the expansion filter operation in both directions are generated. 6. A skip information generating means for generating skip information for skipping each pixel data in the first direction after the predetermined filter operation processing has been performed, and an input pixel sequence in the second direction. Information generating means for generating timing information for controlling the number of pixels and the reduction ratio of the number of output pixel columns. When the two-dimensional image is reduced, the program and the coefficient generating means are arranged in the first direction. 5. The image information processing apparatus according to claim 4, wherein a filter program and a filter coefficient required for a reduction filter operation in both directions of the first and second directions are generated. 7. Skip information generating means for generating pixel data sequentially input in the order of pixels in the first direction or skip information for skipping between pixel data, and the number of input pixel columns in the second direction. And timing information generating means for generating timing information for controlling the enlargement or reduction ratio of the number of output pixel columns. When the two-dimensional image is enlarged or reduced, the program and coefficient generation means 5. The image signal processing apparatus according to claim 4, wherein a filter program and a filter coefficient necessary for an enlargement or reduction filter operation in both the first direction and the second direction are generated.