JP2011004193A - Image processing apparatus and method for controlling the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which performs a vicinity pixel reference operation with high flexibility by a small circuit, and to provide a method for controlling the same.SOLUTION: The image processing apparatus processes pixel data to which attribute data is attached by pixel, and has: a plurality of two-dimensional filters 150; assignment means 120, 140 for assigning the pixel data to the two-dimensional filters 150 based on the attribute data; a one-dimensional filter 160 which uses all or part of output from the two-dimensional filters 150 as input; and a selector 170 which selects the output from the two-dimensional filters 150 and output from the one-dimensional filter 160 to be output as the pixel data.

Description

  The present invention relates to an image processing apparatus and a control method thereof, and more particularly, to an image processing apparatus and a control method thereof that perform calculation referring to neighboring pixels for pixel data.

  Various digital processes are applied to the video signal to be displayed on the display for the purpose of improving the image quality. However, the resolution of the input video signal and the number of streams vary depending on the application. Therefore, there is a demand for a flexible image processing apparatus that can perform optimum processing according to the resolution and the number of streams. An example of flexibility required for an image processing apparatus is a tradeoff between processing speed and image quality at the time of execution. As a result, for example, it is possible to perform switching so that processing is performed at high speed when the number of data is large, and processing is performed with more importance on image quality when the amount of data is small. In addition, research on human gaze mechanisms, etc., has made progress in extracting important regions (ROI) in images. By applying processing that emphasizes image quality to ROI and processing that emphasizes speed in other regions, it is possible to realize image processing with high subjective image quality with a small amount of computation. Conventionally, as image quality enhancement processing, two-dimensional neighboring pixel reference calculation that performs processing with reference to neighboring pixels in one frame, or three-dimensional neighboring pixel that performs processing with reference to neighboring pixels in a plurality of frames Reference operations are widely known. The former is, for example, noise removal, edge enhancement, resolution conversion, and the like. The latter is adaptive correction processing using motion detection, IP conversion, and the like, as in Patent Document 1 and Patent Document 2, for example. In these processes, in general, the larger the number of reference pixels and the number of reference frames, the more the image quality can be improved. Since the above processing has a large calculation amount, for example, an application such as a display that requires real-time processing is implemented by hardware. At this time, each process is generally implemented as a separate module.

FIG. 17 is a block diagram illustrating a configuration example of a conventional image processing apparatus. In the figure, 10 is an IP conversion module, 20 is a resolution conversion module, and 30 is an edge enhancement module. In contrast to the above configuration, for example, by inputting pixels in raster order, sending pixels between modules by handshake communication, and performing processing in a pipeline manner, high image quality processing can be realized. Further, for example, by making it possible to set the filter coefficient of the edge enhancement module 30 from the firmware, for example, the edge enhancement degree can be changed, and the processing content can be changed to noise removal processing.
On the other hand, in Patent Document 3, a feedback system is provided in the filter, and the number of times that the pixel data is filtered can be changed. Thereby, the flexibility of the filter processing is improved.

JP 2006-304019 A JP 2006-311061 A Japanese Patent Laid-Open No. 9-297842

  However, in the configurations of Patent Documents 1 and 2, only a few filter coefficients can be changed according to the operation mode, and flexibility is poor. Further, even with the configuration of Patent Document 3, there is a problem that the number of loops and filter coefficients that can be changed according to the operation mode are low in flexibility.

  The present invention has been made in view of the above problems, and provides an image processing apparatus and a control method therefor that perform a flexible neighborhood pixel reference calculation with a small circuit.

  In order to achieve the above object, an image processing apparatus of the present invention is an image processing apparatus that processes pixel data to which attribute data is attached for each pixel, and includes a plurality of two-dimensional neighborhood reference calculation means, and the attribute data. Allocating means for allocating the pixel data to the two-dimensional neighborhood reference computing means, one-dimensional neighborhood reference computing means for receiving all or part of the output from the two-dimensional neighborhood reference computing means, and the two-dimensional neighborhood And selecting means for selecting the output from the reference calculation means and the output from the one-dimensional neighborhood reference calculation means and outputting the selected pixel data as pixel data.

  According to the present invention, it is possible to select various paths for each pixel such as which calculation unit performs processing in which order for a certain pixel in accordance with the operation mode. For example, three-dimensional filter processing referring to pixel data of a plurality of frames, parallel processing and sequential processing of a plurality of two-dimensional filters, two-dimensional filter processing with a large reference area, and the like can be changed by selecting a route. Thereby, it is possible to realize a highly flexible image processing apparatus capable of trade-off between processing speed and image quality according to the operation mode. Furthermore, since the arithmetic unit is shared by a plurality of paths, it can be realized with a small circuit.

1 is a diagram illustrating a configuration example of an image processing apparatus according to Embodiment 1. FIG. 6 is a diagram illustrating a format example of input pixel data in Embodiment 1 or 2. FIG. FIG. 3 is a diagram for explaining a pixel scanning order in the first embodiment. 3 is a diagram illustrating a configuration example of a row delay unit according to Embodiment 1. FIG. FIG. 6 is a diagram for explaining the operation of a row delay unit in the first embodiment. FIG. 6 is a diagram for explaining the operation of a row delay unit in the first embodiment. FIG. 3 is a diagram illustrating a configuration example of a column delay unit in the first embodiment. 3 is a diagram illustrating a configuration example of a calculation unit in Embodiment 1. FIG. FIG. 10 is a diagram illustrating another configuration example of the calculation unit according to the first embodiment. FIG. 3 is a diagram illustrating a path III according to the first embodiment. FIG. 6 is a diagram for explaining another example of the pixel scanning order in the first embodiment. 6 is a diagram illustrating a configuration example of an image processing apparatus according to Embodiment 2. FIG. 6 is a diagram illustrating a configuration example of a calculation unit according to Embodiment 2. FIG. 10 is a diagram illustrating another configuration example of a calculation unit in Embodiment 2. FIG. FIG. 10 is a diagram illustrating a pixel scanning order in a specific example of Embodiment 2. 10 is a diagram illustrating a pixel calculation method in a specific example of Embodiment 2. FIG. 10 is a diagram illustrating a pixel calculation method in a specific example of Embodiment 2. FIG. It is a figure which shows the structural example of the conventional image processing apparatus.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  First Embodiment FIG. 1A is a diagram illustrating a configuration example of an image processing apparatus according to a first embodiment. In the figure, 100 is the image processing apparatus. The image processing apparatus 100 includes submodules 120, 140, 150, 160, and 170 as shown in FIG. 1A. In addition, the reference code which added the code | symbol to the reference code of each submodule is attached | subjected to the signal which connects submodule 120,140,150,160,170. Thereby, in the figure explaining the structure of each following submodule, the same signal is shown with the same reference number and the connection is clarified. An image processing control unit 200 controls the sub-modules 120, 140, 150, 160, and 170 to control the path of pixel data so that the image processing apparatus 100 operates in various modes. The image processing control unit 200 includes, for example, at least a CPU for arithmetic processing, a ROM for storing a program for processing procedures of the CPU, and a RAM for temporary storage used as a work area. Reference numeral 110 denotes an input to the image processing apparatus 100. The input 110 can input a maximum of one pixel in one cycle. The input 110 is input to the input port of the image processing apparatus 100.

  FIG. 1B shows a format example of input pixel data. The desired pixel input pixel data 110b is provided with attribute data 110a for each pixel. The attribute data 110a includes, for example, a 4-bit group field (called gid) 110a-1 and a 1-bit line end field 110a-2 as shown in FIG. 1B. The group field 110a-1 includes frame information set in units of frames and position information set periodically in units of columns or rows. The number of bits of the group field 110a-1 corresponds to the number of two-dimensional neighborhood reference calculation units (submodules) 150 shown below in this example.

  The sub module 120 of the image processing apparatus 100 is a row delay unit. The row delay unit 120 accumulates pixel data, collects the pixel data for the number of rows necessary for the subsequent two-dimensional filter 150, and sends the collected pixel data to the column delay unit 140. At this time, the destination is determined based on the attribute data attached to each pixel data. The submodule 140 is a column delay unit. In the column delay unit 140, pixel data corresponding to the number of columns necessary for the subsequent two-dimensional filter 150 is accumulated and sent to the two-dimensional filter 150 and other column delay units 140. In FIG. 1A, four column delay units 140-1, 140-2, 140-3, and 140-4 are shown, but the present invention is not limited to this. The submodule 150 is a two-dimensional filter that operates as a two-dimensional neighborhood reference computation unit that performs computation by referring to a two-dimensional neighborhood reference pixel centered on the pixel of interest. In this example, the reference pixel of the two-dimensional filter 150 is a (5 × 5) rectangular region, and the number of two-dimensional filters is four. The two-dimensional filter 150 is a pipeline type arithmetic unit that can process one pixel in four cycles. The two-dimensional filter 150 includes a filter coefficient setting unit (not shown) and can control the filter coefficient from firmware or the like. The two-dimensional filter 150 performs a product-sum operation on the (5 × 5) pixel data received from the column delay unit 140 using filter coefficients. The calculation result is sent to the one-dimensional filter 160 or the selector 170. The sub-module 160 is a one-dimensional filter that operates as a one-dimensional neighborhood reference computation unit that performs computation with reference to a one-dimensional neighborhood reference pixel centered on the pixel of interest. In this example, the number of reference data of the one-dimensional filter 160 is four. The one-dimensional filter 160 includes a filter coefficient setting unit (not shown) and can control the filter coefficient from firmware or the like. Such filter coefficients may be loaded from the image processing control unit 200. The one-dimensional filter 160 performs a product-sum operation on the input four pieces of pixel data using filter coefficients. The calculation result is sent to the selector 170. The sub module 170 is a selector. Among the calculation results of the two-dimensional filter 150 or the one-dimensional filter 160, an effective one is selectively output. The selector 170 includes a FIFO (not shown) inside and can adjust the timing. Reference numeral 180 denotes an output from the image processing apparatus 100. The output 180 can output a maximum of one pixel per cycle. The output 180 is output from the output port of the image processing apparatus 100.

  The sub-modules exchange pixel data and attribute data by handshake communication using a valid signal and a stall signal. The image processing control unit 200 sets attribute data 110a for each pixel in the input pixel data 110b in order to determine a processing path according to the processing content to be executed. When the output 180 is fed back to the input 110, the attribute data 110a is set again. The image processing control unit 200 can also set valid data to be selected by the selector and filter coefficients of the two-dimensional filter 150 and the one-dimensional filter 160 according to the processing content.

  <Example of Configuration of Submodules of Embodiment 1 and Example of Path I> In the following, each submodule will be described in detail assuming the following input pixel data.

  (Assumption of Input Pixel Data) The input 110 is assumed to be 4 frames, and the scanning order is as shown in FIG. The resolution of the input image is HD (1920 × 1080). The group fields gid = 1, 2, 4, and 8 in FIG. 1B are allocated to each input pixel in order from the oldest frame in the time direction. For example, during processing of frames 1, 2, 3, and 4, the pixels in frame 1 are gid = 1 (least significant bit), the pixels in frame 2 are gid = 2 (second bit), and the pixels in frame 3 are gid = 4 (third bit), and the pixel in frame 4 is gid = 8 (most significant bit). Accordingly, the pixels are read in the order of frames indicated by 201 in FIG. 2, and are read in the order of the line direction 202 in the frame. Similarly, during the processing of frames 2, 3, 4, and 5, frame 2 has gid = 1, frame 3 has gid = 2, frame 4 has gid = 4, and frame 5 has gid = 8. Also, the line end field is set to “1” for the pixels in the rightmost column of each frame, and the line end field is set to “0” for the other pixels. Therefore, if the line end field is “1”, the process returns to the head of the next line. When the line end field becomes “1” in the last line, the frame 2 is returned to the beginning as shown in 203, and the reading of the frames 2 to 5 is performed. In the path I, the input pixel data passes through the row delay unit 120 and is allocated to the two-dimensional filters 150-1, 150-2, 150-3, and 150-4 according to the attribute data. Perform dimension filtering. Further, each calculation result is processed by the one-dimensional filter 160. As described above, the three-dimensional filter processing with reference to the four frames is performed, and the calculation result for one frame is output.

  (Configuration Example of Row Delay Unit 120) FIG. 3 is a diagram illustrating a configuration example of the row delay unit 120. In the figure, 540 is a FIFO, 541 is a read control unit, 542 is a write control unit, 543 is an SRAM, and 544 is a demultiplexer. An arrow connecting each component represents a data flow. Reference numerals 551, 552, 555, and 556 denote pixel data and attribute data. Reference numeral 553 is pixel data read from the SRAM, 554 is a read address to the SRAM, 557 is pixel data to be written to the SRAM, and 558 is a write address to the SRAM and a write enable. Here, the word width of the SRAM is 4 pixels, and the number of words of the SRAM is 8000.

  An example of the relationship between the input pixel position and the output pixel position under the above conditions is shown in FIG. 4B shows the next cycle of (a), and FIG. 4C shows the next cycle of (b). In the figure, 320 is the position of the input pixel to the row delay unit 120, 321 is the position of the pixel group read from the memory, 322 is the position of the pixel group held in the memory, and 323 is the pixel written in the memory. The position of the group. The black and white numbers in the pixel area indicate SRAM addresses.

  Figure 6 shows an example of memory access timing. Input corresponds to 320, Read corresponds to 321 and Write corresponds to 323. Each numeral represents gid, and (a), (b), and (c) respectively represent the timings of (a), (b), and (c) in FIG.

  For example, as shown in FIG. 4A, when the input pixel 320 to the row delay unit 120 is gid = 2, the read control unit 541 is positioned above the input pixel 320 in the frame of gid = 2. Four pixel groups 321 are read from the SRAM 543. The address 554 is (X) for a frame with gid = 1, (2000 + X) for a frame with gid = 2, (4000 + X) for a frame with gid = 4, and (6000 + X) for a frame with gid = 8. To do. Here, X is the horizontal coordinate of the input pixel 320. X is obtained by counting the pixels for each gid, outputting the pixels whose line end field is “1”, and resetting them to zero. That is, the memory space in the SRAM is equally allocated by 2000 words for each frame. The smallest address in the allocated address space is mapped to the leftmost column of the image, and the smallest addresses are mapped in order in the right direction of the image. The read control unit 541 combines the input pixel 320 and the read pixel group 323 and sends the combined pixel group 555 to the demultiplexer 544. The attribute data of the input pixel 320 is attached to the pixel group 555. Also, four pixel groups 323 excluding the uppermost pixel among the five pixel groups are sent to the light control unit 542. The write control unit 542 writes the pixel data of the pixel group 323 received from the read control unit 541 to the SRAM 543. For example, the input pixel 320 and the read pixel group 321 in FIG. 4A are written into the SRAM 543 at the timing of FIG. The address 558 for writing is obtained in the same manner as for reading. The demultiplexer 544 allocates the pixel data to the column delay unit 140 according to the value of the bit of the group field attached to the received pixel group. That is, if the least significant bit is “1”, the column delay unit 140-1, if the lower second bit is “1”, the column delay unit 140-2, and if the lower third bit is “1”, the column delay. If the most significant bit is “1”, the data is output to the column delay unit 140-4.

  (Configuration Example of Column Delay Unit 140) FIG. 6 is a diagram illustrating a configuration example of one column delay unit 140-1 of four column delay units. The other column delay units 140-2, 140-3, and 140-4 have the same configuration except that the output destination two-dimensional filter 250 is different. In the figure, reference numerals 571, 572, 573, 574, and 575 are registers for five pixels, respectively. The pixel group for five pixels input from the row delay unit 120 is held in the register 571. The pixel group for five pixels held in the register 571 is sent to the register 572 and the two-dimensional filter 150-1. The pixel group for five pixels held in the register 572 is sent to the register 573 and the two-dimensional filter 150-1. The pixel group for five pixels held in the register 573 is sent to the register 574 and the two-dimensional filter 150-1. The pixel group for five pixels held in the register 574 is sent to the register 575 and the two-dimensional filter 150-1. The pixel group for five pixels held in the register 575 is sent to the two-dimensional filter 150-1.

  (Configuration Example of Two-Dimensional Filter 150) FIG. 7 is a diagram illustrating a configuration example of a two-dimensional filter 150-1 that is one of the two-dimensional filters 150. The other two-dimensional filters 150-2, 150-3, and 150-4 have the same configuration. In the figure, 591 is a register for 5 × 5 pixels, 592 is a coefficient setting unit, 593 is a register for holding 5 × 5 coefficients, 594 is a multiplier, 595 is a register for 5 × 5 pixels, and 596 is an adder. Each of the 5 × 5 input pixel groups from the column delay unit 140-1 is multiplied by the product set by the coefficient set by the coefficient setting unit by the multiplier 594 and sent to the register 595. All the values in the register 595 are calculated by the adder 596 and output.

  (Configuration Example of One-dimensional Filter 160) FIG. 8 is a diagram illustrating a configuration example of the one-dimensional filter 160. In the figure, 601 is a register for four pixels, 602 is a coefficient setting unit, 603 is a register for holding four coefficients, 604 is a multiplier, 605 is a register for four pixels, and 606 is an adder. Each input pixel from each two-dimensional filter 150 is input to the register 601. Here, each input pixel is waited. When the necessary input pixels are prepared, the product of the coefficient set by the coefficient setting unit is calculated by the multiplier 604 and sent to the register 605. All the values in the register 605 are calculated by the adder 606 and output to the selector 170.

  (Configuration Example of Selector 170) Selection by the selector 170 is controlled by the image processing control unit 200 corresponding to a desired filter process. In this path I, the selector 170 selects and outputs the calculation result of the one-dimensional filter.

  <Example of Path II According to Configuration Example of Embodiment 1> The scanning order and attribute data of input pixel data are the same as those of path I. The input pixel data is processed independently by the two-dimensional filters 150-1, 150-2, 150-3, and 150-4 through the row delay unit 120. The calculation result of each two-dimensional filter 150 is sent to the selector 170. The selector 170 selects and outputs each calculation result in turn. As described above, the two-dimensional filtering process is performed independently for each of the four frames, and the calculation results for the four frames are output. The calculation result output 180 in the frame of gid = 1 is fed back to the input 110 as gid = 2. The output 180 of the calculation result of gid = 2 is fed back to the input 110 as gid = 4. The output 180 of the calculation result of gid = 4 is fed back to the input 110 as gid = 8. Thereby, sequential processing of a two-dimensional filter is also possible for one frame. Thus, the two-dimensional filter that operates independently can select parallel processing or sequential processing.

  <Example of Path III According to Configuration Example of Embodiment 1> The input 110 is assumed to be one frame, and the scanning order is as shown in FIG. As shown in the drawing, the odd-numbered pixels in the frame are set to gid = 1, and the even-numbered pixels are set to gid = 8. The pixel data of gid = 1 is delayed by five columns by the column delay unit 140-1, and the five pixel groups held in the register 575 are sent to both the two-dimensional filter 150-1 and the column delay unit 140-2. . By further delaying the five pixel groups input to the column delay unit 140-2 by five columns, the (5 × 5) pixel region input to the two-dimensional filter 150-2 is input to the two-dimensional filter 150-1. This is a region delayed by 5 columns from the (5 × 5) pixel region. Similarly, pixel data of gid = 8 is input to the two-dimensional filter 150-3, and the (5 × 5) pixel region is delayed by 5 columns from the pixel region of (5 × 5) input to the two-dimensional filter 150-4. It becomes the area.

  In summary, the pixel areas processed by the two-dimensional filters 150-1, 150-2, 150-3, and 150-4 are as shown in FIG. 9B. In the figure, reference numeral 331 is a reference pixel area of the two-dimensional filter 150-1, 332 is a reference pixel area of the two-dimensional filter 150-2, 333 is a reference pixel area of the two-dimensional filter 150-4, and 334 is a two-dimensional filter 150-3. This is a reference pixel region. The calculation result of each two-dimensional filter 150 is input to the one-dimensional filter 160 and subjected to product-sum calculation. As described above, the two-dimensional filter calculation with reference to the (10 × 10) pixel region can be realized. In the path III, the SRAM 543 in the row delay unit can allocate 4000 words to gid = 1,8. This means that a larger image can be handled in the horizontal direction.

  <Example of Route IV According to Configuration Example of Embodiment 1> As the middle of the above route, for example, two-dimensional filter processing is performed by two-dimensional filters 150-1, 150-2, 150-3 and one-dimensional filter 160 to perform two-dimensional filtering. In 150-4, independent two-dimensional filter processing can be performed in parallel or sequentially.

  <Characteristics of Routes According to Configuration Example of Embodiment 1> The processing by the routes I to IV is summarized. The sequential operation of the path I and the path II and the output throughput in the path III are a maximum of 1/4 (pixel / cycle). The output throughput in the parallel operation of the path II is 1/1 (pixel / cycle) at the maximum. Pathway IV is intermediate between these. On the other hand, in the path I, three-dimensional filter processing with reference to four frames is realized. In the sequential operation of path II, two-dimensional filtering is performed a plurality of times in one frame. In the path III, two-dimensional filter processing with reference to a wide range in one frame is realized. These processes can be expected to have higher image quality than the parallel operation of path II. Pathway IV is intermediate between these.

  <Effects of First Embodiment> As described above, according to the present embodiment, various paths can be selected according to the operation mode, so that highly flexible image processing can be realized.

  <Modification of First Embodiment> Note that the scanning direction of the pixels may not be as described above. For example, scanning may be performed as shown in FIG. In this scanning method, first, each frame is divided into elongated band regions. The height of the band is arbitrary, but it is the same between frames. Within each band region, scanning is first performed in the time direction 1001, scanning is performed in the Y direction, and scanning is performed in the X direction (1002). When one band of each frame is completed, the next band is scanned in the same manner (1003).

  Second Embodiment FIG. 11 is a diagram illustrating a configuration example of an image processing apparatus according to the second embodiment. In the figure, 250 is a two-dimensional neighborhood reference computation unit, 260 is a one-dimensional neighborhood reference computation unit, and other configurations are the same as those in the first embodiment. Components having the same reference numerals as those in FIG. 1A are the same as those in the first embodiment, and duplication of description is avoided. In the first embodiment, the arithmetic units 150 and 160 are product-sum operations. However, in the second embodiment, an arithmetic unit that performs a more general neighborhood reference operation is used. The calculation algorithm of the calculation unit can be changed by loading software or firmware from the image processing control unit 200 in accordance with desired processing.

  (Configuration Example of Two-dimensional Neighborhood Reference Calculation Unit 250 of Embodiment 2) FIG. 12A is a diagram illustrating a configuration example of the two-dimensional neighborhood reference calculation unit 250 according to the second embodiment. In the figure, 597 is an ALU. The ALU 597 includes, for example, a multiplier, an adder, and a comparator. Assuming that the value held in the register 591 is A [k] and the value held in the register 593 is B [k] (k = 0, 1,..., 24), the ALU 597 can be expressed by, for example, the formula (1 ) To C shown in (3) to “3” are output.

k corresponds to the position shown in FIG. Expression (1) is an operation equivalent to that of the first embodiment. Equation (2) calculates the square of the difference between the register 591 and the adjacent element in the X direction. Formula (3) returns “1” when the absolute value of the difference from the adjacent element in the X direction in the register 591 is larger than the threshold, and returns “0” when the difference is smaller.

  (Configuration Example of One-Dimensional Neighborhood Reference Calculation Unit 260 in Embodiment 2) FIG. 13A is a diagram illustrating a configuration example of the one-dimensional neighborhood reference calculation unit 260 in the second embodiment. In the figure, reference numeral 607 denotes an ALU. The ALU 607 includes, for example, a multiplier, an adder, an LUT, and the like. Assuming that the value held in the register 601 is D [k] and the value held in the register 603 is E [k] (k = 0, 1, 2, 3), the ALU 607 can, for example, F shown in (6) is output.

k corresponds to the position shown in FIG. In the formula, LUT (X) is a value of LUT with X as an argument. Expression (4) is an operation equivalent to that of the first embodiment. Equation (5) corrects the value of D [0] based on the sum of D [1], D [2], and D [3]. Equation (6) obtains a linear interpolation value of D [0] and D [1] based on the sum of D [2] and D [3]. With the above configuration, various operations including the product-sum operation similar to those in the first embodiment are enabled, and further flexibility is improved.

  <Specific Application Example of Embodiment 2> For example, when there is a moving object in the captured current frame, a process of highlighting the area with a warning color is performed. Prepare a background frame without moving objects in advance. The background frame and the current frame are input in the scanning order shown in FIG. That is, the pixel data corresponding to the background and the frame 1 are read in the line direction 1402 in the order 1401. When the end of frame 1 is read, the pixel returns to the first pixel of the background (1402, 1403), and the pixel data of the corresponding frame 2 and then frame 3 are read. As shown in FIG. 14, gid = 2, 4, and 8 are repeatedly allocated to the background frame for each column. Thereby, the coordinates (3M, N) in the background frame; M, N is 0 or a positive number is sent to the column delay unit 140-2. The pixel at coordinates (3M + 1, N) is sent to the column delay unit 140-3. The pixel at coordinates (3M + 2, N) is sent to the column delay unit 140-4. Similarly, gid = 3, 5, and 9 are repeatedly allocated to the current frame for each column. As a result, the pixel at the coordinates (3M, N) in the current frame is sent to the column delay units 140-1 and 140-2. The pixel at coordinates (3M + 1, N) is sent to the column delay units 140-1 and 140-3. The pixel at coordinates (3M + 2, N) is sent to the column delay units 140-1 and 140-4.

  FIG. 15 shows an example of a pixel group held by each delay unit 140. In the figure, reference numerals 613, 615, 617, 619, 621, 623, and 625 denote five pixels in the background frame, respectively. Reference numerals 610, 612, 614, 616, 618, 620, 622, and 624 denote 5 pixels in the current frame, respectively. Reference numerals 613 and 614, 615 and 616, 617 and 618, 619 and 620, 621 and 622, and 623 and 624 are pixel groups having the same coordinates. 613, 619, and 625 are gid = 2, 615 and 621 are gid = 4, and 617 and 623 are gid = 8. 614 and 620 are gid = 3, 610, 616 and 622 are gid = 5, and 612, 618 and 624 are gid = 9. In addition, the coordinates of the central pixels 619 and 620 (shaded portions in the figure) are (S, T). At this time, the pixel groups held in the registers 571, 572, 573, 574, and 575 in the column delay unit 140-1 are 624, 622, 620, 618, and 616, respectively. The pixel groups held in the column delay unit 140-2 are 625, 620, 619, 614, and 613, respectively. The pixel groups held in the column delay unit 140-3 are 622, 621, 616, 615, and 610, respectively. The pixel groups held in the column delay unit 140-4 are 624, 623, 618, 617, and 612, respectively. For the pixel group as described above, the two-dimensional neighborhood reference calculation unit 250-2 obtains the sum of squares of the difference between the pixel group 620 and the pixel group 619. The two-dimensional neighborhood reference calculation unit 250-3 obtains the sum of the difference square sum of the pixel group 622 and the pixel group 621 and the difference square sum of the pixel group 616 and the pixel group 615. The two-dimensional neighborhood reference calculation unit 250-4 obtains the sum of the difference square sum of the pixel group 624 and the pixel group 623 and the difference square sum of the pixel group 617 and the pixel group 618. That is, the calculation results of the two-dimensional neighborhood reference calculation units 250-2, 250-3, and 250-4 are the values shown in Expression (7), Expression (8), and Expression (9), respectively.

In the equation, B (x, y) and C (x, y) are the pixel value of the background frame and the pixel value of the current frame, respectively, at the coordinates (x, y). The two-dimensional neighborhood reference calculation unit 250-1 outputs a pixel value of C (S, T). In the one-dimensional neighborhood reference calculation unit 260, the value of Expression (10) is obtained from the sum of the calculation results of the two-dimensional neighborhood reference calculation units 250-2, 250-3, and 250-4, and the two-dimensional neighborhood reference is based on this. The value received from the calculation unit 250-1 is corrected. Expression (10) is the sum of squared differences between the background frame and the current frame in the rectangular area in the vicinity (5 × 5) of the coordinates (S, T), and serves as an index of a change in pixel value. When this value exceeds the threshold value, the coordinates (S, T) are determined to be within the area of the moving object, and the process of converting the output C (S, T) of the two-dimensional neighborhood reference calculation unit 250-1 into a warning color is performed by the LUT. To do.

  Further, when there is an input for one pixel in FIG. 15, the result is as shown in FIG. At this time, the column delay unit 140-1 receives the pixel group 626, and the pixel groups held in the registers in the column delay unit 140-1 are 626, 624, 622, 620, and 618, respectively. The pixel groups held by the registers in the column delay unit 140-2 are 626, 625, 620, 619, and 614, respectively. The pixels held in the column delay units 140-3 and 140-4 are the same as those in FIG. For the pixel groups as described above, the two-dimensional neighborhood reference calculation units 250-2, 250-3, and 250-4 calculate Expression (11), Expression (12), and Expression (9), respectively.

  These sums are given by Equation (13), which is the sum of squared differences between the background frame and the current frame in the rectangular area (5 × 5) in the vicinity of the coordinates (S + 1, T). If this value exceeds the threshold value, the coordinates (S + 1, T) are determined to be within the region of the moving object, and the output C (S + 1, T) of the two-dimensional neighborhood reference calculation unit 250-1 is converted to a warning color. Processing is performed in the LUT.

  (Effects of Specific Example) By sequentially performing the above calculation in the scanning order, the above-described enhancement process of the moving object region can be performed on the entire image.

  <Effects of Second Embodiment> As described above, in the second embodiment, various image processes are possible depending on the scanning order and the attribute data allocation method by using a general-purpose neighborhood reference calculation as the calculation unit. .

  [Other Embodiments] In this embodiment, an example of a specific neighborhood reference calculation is shown. However, by changing the neighborhood reference calculation variously or using various parameters to be used, it is simple. Various desired image processing can be realized by the image processing apparatus having the configuration. Further, in the above-described embodiment, the case where the pixel data is allocated to four cases or the case where the number of attribute data types is four is shown. However, the present invention is not limited to this, and the present invention may be applied to other numbers. The same effect is exhibited and these are also included in the present invention. For example, when the number of allocation destinations is N, the scanning order of the pixels input to the input port is such that all the attribute data referred to by the allocation unit for determining the allocation destination is different within any continuous N inputs. In order. For example, the scanning order of pixels input to the input port is as follows when the number of types of attribute data included in one or more input frames required to output an image of one frame is M It becomes. That is, the scanning order is such that all the attribute data referred to the determination of the allocation destination are different in any continuous M input. In the embodiment, an example of one column delay and four row delays is shown, but the present invention is not limited to this. In general, when either the main scanning direction or the sub-scanning direction is set as the first direction and the other is set as the second direction, the storage unit of the present invention uses the first for delaying the pixels in the first direction. And a second storage unit for delaying pixels in the second direction. The second storage unit is divided into a plurality of areas. Further, the second storage unit has a path for sending pixel data from the first area to the second area of the second storage unit. These are also included in the present invention.

  In addition, the present invention may be applied to a system or an integrated device composed of a plurality of devices (for example, a host computer, an interface device, a printer, etc.) or an apparatus composed of a single device. Another object of the present invention is to supply a storage medium (or recording medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus. Needless to say, this can also be achieved by the computer (or u CPU or MPU) of the system or apparatus reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  The functions of the above-described embodiments are not only realized by executing the program code read by the computer. This includes the case where the operating system (OS) running on the computer performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. Needless to say. Further, the program code read from the storage medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU provided in the function expansion card or function expansion unit performs part or all of the actual processing. It goes without saying that the case where the functions of the above-described embodiments are realized by such processing is also included. When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above.

Claims (17)

An image processing apparatus that processes pixel data to which attribute data is attached for each pixel,
A plurality of two-dimensional neighborhood reference calculation means;
Allocating means for allocating the pixel data to the two-dimensional neighborhood reference calculation means based on the attribute data;
One-dimensional neighborhood reference computing means that receives all or part of the output from the two-dimensional neighborhood reference computing means;
An image processing apparatus comprising: a selection unit that selects an output from the two-dimensional neighborhood reference computation unit and an output from the one-dimensional neighborhood reference computation unit and outputs the selected pixel data. An input port for inputting pixel data to which attribute data is attached for each pixel;
An output port for outputting pixel data output from the selection unit;
Storage means for storing a plurality of pixel data input from the input port and outputting pixel data of neighboring reference pixels including a target pixel to the two-dimensional neighboring reference computing means according to the allocation by the allocating means. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 2, further comprising feedback means from the output port to the input port.   The image processing apparatus according to claim 1, wherein the attribute data includes frame information set in units of frames.   The image processing apparatus according to claim 1, wherein the attribute data includes position information periodically set in units of columns or rows. The attribute data referred to by the allocator for determining the allocation destination is at least one of the frame information and the position information,
6. The image processing apparatus according to claim 4, wherein the number of bits of the attribute data referred to by the allocating unit for determining the allocation destination is the number of the two-dimensional neighborhood reference calculating unit.   The scanning order of the pixels input to the input port is different when all of the attribute data to which the allocating unit refers to the determination of the allocation destination is arbitrary N consecutive inputs when the allocation unit of the allocation unit is N. The image processing apparatus according to claim 1, wherein the scanning order is as described above.   The scanning order of the pixels input to the input port is any continuous M input when the number of types of attribute data included in one or a plurality of input frames necessary for outputting an image of one frame is M. 7. The image processing apparatus according to claim 1, wherein the attribute data referred to by the allocating unit for determining the allocation destination is different in scanning order. When either the main scanning direction or the sub-scanning direction is the first direction and the other is the second direction,
The storage means includes first storage means for delaying pixels in a first direction and second storage means for delaying pixels in a second direction,
The image processing apparatus according to claim 1, wherein the second storage unit is divided into a plurality of regions.   The image processing apparatus according to claim 9, wherein the second storage unit includes a path for sending pixel data from the first area to the second area of the second storage unit.   The image processing apparatus according to claim 1, wherein the one-dimensional neighborhood reference calculation unit includes an LUT.   12. The apparatus according to claim 1, further comprising a control unit that assigns the attribute data to each pixel and controls selection by the selection unit in response to a filtering process performed by the image processing apparatus. The image processing apparatus according to claim 1.   The control means sets an operation algorithm of at least one of the two-dimensional neighborhood reference calculation means and the one-dimensional neighborhood reference calculation means corresponding to the filter processing performed by the image processing apparatus. Item 13. The image processing apparatus according to Item 12. An image processing apparatus for processing pixel data to which attribute data is attached for each pixel, and allocating a plurality of two-dimensional neighborhood reference computing means and allocating the pixel data to the two-dimensional neighborhood reference computing means based on the attribute data Means, a one-dimensional neighborhood reference computing means that receives all or a part of the output from the two-dimensional neighborhood reference computing means, an output from the two-dimensional neighborhood reference computing means and the one-dimensional neighborhood reference computing means A control method of an image processing apparatus having selection means for selecting output and outputting as pixel data,
A step of assigning the attribute data for each pixel in correspondence with the filter processing performed by the image processing apparatus;
A method for controlling the image processing apparatus, wherein the control means includes a step of controlling selection by the selection means in response to a filtering process performed by the image processing apparatus.   The control means calculates at least one of the two-dimensional neighborhood reference computation means and the one-dimensional neighborhood reference computation means corresponding to the filter processing performed by the image processing device prior to the filter processing by the image processing device. The method of controlling an image processing apparatus according to claim 14, further comprising a step of setting an algorithm.   The program for making a computer perform each step of the control method of the image processing apparatus of Claim 14 or 15.   A computer-readable storage medium storing the program according to claim 16.

Images