EP1745433A1

EP1745433A1 - Image processing apparatus and method

Info

Publication number: EP1745433A1
Application number: EP05718810A
Authority: EP
Inventors: Richard P. Kleihorst; Adrianus J. Bink; Harry Broers; Wouter Caarls; Seyed H. Fatemi
Original assignee: Koninklijke Philips Electronics NV
Current assignee: Koninklijke Philips NV
Priority date: 2004-04-29
Filing date: 2005-04-26
Publication date: 2007-01-24
Also published as: CN1950846A; JP2007535066A; WO2005106786A1

Abstract

An image processing apparatus and method identifies a region of interest 22a, for example a line segment, in an image frame. Preferably, the region of interest is defined within a course bounding box 21a, which is used to define a “best-fitting” boundary box 23a. The best-fitting bounding box 23a is then rescanned to an orthogonal grid 25a. The line segment 22a is therefore represented by a two dimensional array, with the entire length of the line segment 22a being in a first dimension, and pixel data to the left and/or right of the line segment 22a being in a second dimension. The orthogonal grid 25 is then processed by the SIMD processor 26 for pixel parallel processing, for example edge finding, enhancement, interpolation or line data. Since the line segment 22a has been rescanned such that its entire length is found in one dimension, the image data is more suited for line-region based processing by the SIMD processor 26. Preferably, the first dimension corresponds to a row in an image and the second dimension to a column in an image. Once the pixel data has been processed by the SIMD processor, scan information stored during the rescanning operation is used to scan the image data back into original image frame.

Description

Image processing apparatus and method

The invention relates to an image processing apparatus and method, and in particular, to an image processing apparatus and method using Single Instruction Multiple Data (SIMD), in which pixels are rescanned in order to make better use of the parallel processing power of the SIMD processor.

SIMD processing is a powerful computing paradigm for applications that exhibit massive parallelism. One such application that adopts the use of SIMD processing is that of image processing. SIMD processors, for example Xetal, perform their operations on each data item (e.g. each pixel in a line for Xetal) whether they are needed or not. In other words, a processing operation is performed on a complete line regardless of whether or not a processing operation is required. Depending on the data distribution, or sparsity, much computation power can therefore be wasted using this technique. More and more image processing algorithms are starting to work on portions of images. For example, in television processing, industrial vision or medical imaging, it is known to work on the edges of images (i.e. line processing). Also, in applications such as image communication or 3D rendering, it is known to work on separate objects within an image (i.e. object processing), thereby reducing the amount of unnecessary processing operations. In other applications, whole frames are processed, and for this the very powerful and efficient SIMD processors could be used. However, for object or line processing, SIMD does not provide an efficient processing method because of the scattering of the objects or the layout of the objects in an image. This means that SIMD processing is not efficient for such processing techniques, because it processes a large proportion of data which is of no interest. Several solutions exist for making efficient use of SIMD computing resources. For example, one method is to load-balance over multiple SIMD processors. Another is to provide algorithms that use special data structures to operate efficiently on sparse structures. For example, such a technique is disclosed in "Massive parallelism for sparse images", Shankar et al, IEEE International Conference on Decision Aiding for Complex Systems, 1991. However, the methods described above suffer from hardware overheads, and have the disadvantage of processing image data in a manner that is not wholly compatible with SIMD processing. The aim of the present invention is to provide an image processing apparatus and method which does not suffer from the disadvantages mentioned above.

According to a first aspect of the present invention, there is provided an image processing apparatus comprising: means for identifying a region of interest in an image signal; means for bounding the image data corresponding to the region of interest into an orthogonal area having a first dimension and a second dimension; and a processor array for processing the image data in the orthogonal area. By rescanning a region to an orthogonal grid, the invention enables a processor array to perform its efficient line or rectangle based processing efficiently by gathering data blocks on lines. According to another aspect of the present invention, there is provided a method of processing an image signal in a processor having an array of parallel processing elements, the method comprising the steps of: identifying a region of interest in the image signal; bounding the region of interest in an orthogonal area having a first dimension and a second dimension; and processing the image data of the orthogonal area in the processor.

For a better understanding of the present invention, and to show more clearly how it may be carried into effect, reference will now be made, by way of example, to the accompanying drawings, in which: Fig. 1 shows the basic components of a known intelligent camera; Fig. 2 shows an image processing method for edges or lines, having a rescan process in accordance with the present invention; Fig. 3 shows an image processing method for objects, having a rescan process in accordance with the present invention; Fig. 4 is a flow diagram detailing the steps of the present invention; Fig. 5 illustrates how object lines are created to memory; and Fig. 6 shows how image data is stored in the memory of Fig. 5.

Fig. 1 shows a block diagram of the main components of a known intelligent camera 1. The intelligent camera 1 comprises a chip 3, having an array of sensors 4, for example a CMOS sensor array, which receive the image to be captured. An on-chip SIMD processing array is integrated onto the same chip 3 as the sensor array 4, and comprises an array of SIMD processing elements (PEs) 5 and a distributed memory 7. The on-chip SIMD processing array is augmented with an off-chip general purpose processor 8. The intelligent camera 1 also comprises other functional units for processing the image signals, such as a Data RAM 9, an Instruction RAM 11, an ILP (Instruction Level Parallelism) processor 13, and an input and output networking unit 15. The general purpose processor 8 provides high level image processing functionality, such as feature extraction, object detection and tracking. The present invention provides an enhanced image processing capability, based on the object detection functions provided by an intelligent camera 1 such as that described in Fig. 1. Fig. 2 shows the method of processing edges and/or lines of an image in accordance with the present invention. An image frame 20 has regions of interest identified therein, for example line segments 22a to 22c. Each line segment 22a to 22c is defined within an orthogonal area, i.e. a course bounding box. For example, Fig. 2 shows a course bounding box 21 defined for the line segment 22a. Next, the course bounding box 21a is used to define a "best-fitting" boundary box 23a. Preferably, the best-fitting boundary box 23a is perpendicular to the edge of the line segment 22a, and having a span of 5 pixels, for example. The best-fitting boundary box 23a is then rescanned to an orthogonal grid 25a. This involves applying a grid to the best-fitting bounding box 23a, as shown in 24a. Then, the respective pixel values of the orthogonal grid 25a are computed using interpolation techniques from the original grid in 24a. As a result, the line segment 22a is therefore "warped" such that it is represented by a two dimensional array, with the entire length of the line segment 22a being in a first dimension, and pixel data to the left and right of the line segment 22a being in a second dimension. During the rescan operation the image processing apparatus maintains scan information in a memory for later use, as will be described below. The orthogonal grid 25 a is then processed by the SIMD processor 26 for pixel parallel processing, for example edge finding, enhancement, interpolation of line data. Since the line segment 22a has been rescanned such that its entire length is found in one dimension, the image data is more suited for line-region based processing by the SIMD processor 26. Preferably, the first dimension corresponds to a row in an image and the second dimension to a column in an image The provision of the orthogonal grid has the advantage of ensuring that each processor will perform a similar instruction on the pixels, hence being SIMD compatible. Thus, according to the invention the instruction flow for line algorithms is typically orthogonal on the line. Once the pixel data has been processed by the SIMD processor, the scan information stored during the rescanning operation is used to rescan the image data into the original grid 27. The original grid 27 is then transformed into a normal line segment 22a by re-warping the line segment from the first dimension back to its original form. The line segment may then be reinserted into the image frame 20, if desired, having been processed by the SIMD processor. In this stage, the inverse scanning is performed to place the processed line back into the original image. Alternatively, if no image is required, for example because only measurements are needed, then the last stages may be omitted. Fig. 3 shows the method of processing objects or blobs in an image in accordance with the present invention. Objects 32a to 32c are identified in an image frame 30. Each object 32a to 32c has an orthogonal area, i.e. a course bounding box, defined in relation therewith. For example, Fig. 3 shows a course bounding box 31a defined for the object 32a. Next, according to one embodiment, the course bounding box 31a is used to define a "best-fitting" bounding box 33a. Although only one best-fitting bounding box.33a is shown in Fig. 3, one or more best-fitting bounding boxes 33a can be used to define an object depending upon the nature of the object 32a. The best-fitting bounding box 33a is determined using known algorithms, or by determining when the object fits best under different rotations. A grid is then applied to the best-fitting bounding box 33a, as shown in 35a, before being rescanned to a two dimensional orthogonal grid 36a. During the rescan operation the image processing apparatus maintains scan information for later use, as will be described below. The orthogonal grid 36a is then processed by the SIMD processor 37 for pixel parallel processing, for example filtering, identification and so on. The image data is processed more efficiently because the object data has been transformed into a two dimensional grid, thereby making it more suitable for processing by the SIMD processor. As a result of the operations described above, the object is effectively re-mapped before processing by the SIMD processor, such that the object lies in fewer rows than before, thereby avoiding the need for the entire frame having to be processed. Once the pixel data has been processed by the SIMD processor, the scan information stored during the rescanning operation is used to rescan the output of the SIMD processor 38a back into an image 39a, which is reinserted into the image frame 30, if desired, having been processed by the SIMD processor. Alternatively, the rescanning steps shown in 38a and 39a can be omitted if an image signal is not required, for example if only numbers or measurements are needed from the object, such as area or color. The invention described above provides an image processing apparatus and method in which more efficient use of SIMD processing is provided when processing objects in an image. It is noted that, in the embodiment of Fig. 3, some of the stages involved in processing an object are only optional, and can therefore be omitted while still retaining a very high processing efficiently in the SIMD processor. For example, since the area defined by the course bounding box 31a is already orthogonal, then the rescanning stages 33a, 35a and 36a (together with the corresponding scan back stage 39a) can be omitted. In other words, while the preferred embodiment of Fig. 3 shows that a best-fitting bounding box 33a is determined, which is then rescanned to an orthogonal box 36a, the invention could also make use of just the orthogonal area defined by the course bounding box 31a. Although this alternative embodiment is not as efficient as the embodiment including all of the stages, it nevertheless still provides an high degree of efficiency, because only the pixel data in the orthogonal course bounding box 3 la is processed, rather than an entire frame. ^■ Fig. 4 summarizes the steps involved in the method of processing an object in an image signal, in accordance with the present invention. First, a course bounding box is determined for a region of interest in the received image signal, step 401. The course bounding box is then used to define a best-fitting bounding box, step 403. The image data in the best-fitting bounding box is then rescanned to an orthogonal grid, step 405. The rescanned image data is processed by a SIMD processor, step 407, in response to instructions received from a program source 408. The processed image data is then scanned back, step 409, using the scanning information 406 stored during the rescanning process. As mentioned above, the steps 403, 405 and 409 are optional, since the SIMD processor can be configured to directly process the pixel data in the orthogonal course bounding box. Fig. 5 shows how object lines are created to a memory 503. A processor 501 performs the recognition and labeling of objects and lines, and general processing of a received image signal. In the second stage 502, the bounding boxes are determined and the rescanning and/or warping performed on the objects and lines, respectively. Output data 502a and control infonnation 502b from the second stage 502 is passed to a memory 503. The memory stores the image data in a manner as converted in the second stage, (and discussed below in Fig. 6). A SIMD processor 504 processes the image data stored in the memory 503. The output of the SIMD processor is either used for measurements 505, or passed to a stage 506 which performs inverse rescanning and/or warping operations. The inverse rescanning and/or warping is performed on the objects and lines using control information 502b received from the second stage 502. The processed objects/lines from the stage 506 are then passed to a combiner 507 which combines the original image signal with the processed image signal to produce a resulting image signal 508. Fig. 6 shows how the image data is stored in the memory 503 of Fig. 5. It can be seen that the processing by the SIMD processor is limited to the rows of the image frame corresponding to the object data and the rows corresponding to the line data (the line data having been converted into straightened line segments, i.e. not the first dimension). As a result, rather than processing the entire frame, the SIMD processor only processes the transformed data that is orthogonal to the line processors. Thus, the invention described above rescans image data into predetermined dimensions corresponding to rows or columns in an image, thereby making the image data more suited for processing by a SIMD array. The invention can be applied to a number of different applications, including: the processing of television images to increase the image quality; performing object recognition in computer vision applications; performing image rendering for computer gaming, education or CAD/CAM; performing object based coding for MPEG4, H263+; performing image processing for medical systems. It is noted that, although the preferred embodiments refer to rescanning to a two dimensional orthogonal grid, it will be appreciated that when processing three dimensional image data such as video data, the grid can be three dimensional rather than two dimensional. It should be noted that the above-mentioned embodiment illustrates rather than limits the invention, and that those skilled in the art will be capable of designing many alternative embodiments without departing from the scope of the invention as defined by the appended claims. In the claims, any reference signs placed in parentheses shall not be construed as limiting the claims. The word "comprising" and "comprises", and the like, does not exclude the presence of elements or steps other than those listed in any claim or the specification as a whole. The singular reference of an element does not exclude the plural reference of such elements and vice-versa. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

CLAIMS:

1. A method of processing an image signal in a processor having an array of parallel processing elements, the method comprising the steps of: identifying a region of interest in the image signal; bounding the region of interest in an orthogonal area having a first dimension and a second dimension; and processing the image data of the orthogonal area in the processor.

2. A method as claimed in claim 1, further comprising the step of transforming line segment data in the image signal into the first dimension.

3. A method as claimed in claim 2, further comprising the step of transforming pixel data to the left and/or right of the line segment data into the second dimension.

4. A method as claimed in claim 2 or 3, further comprising the steps of: - determining a best-fitting bounding box for the line segment data in the image signal; and rescanning the best fitting bounding box to an orthogonal grid prior to the processing step.

5. A method as claimed in claim 1 , further comprising the steps of: determining a best-fitting bounding box for an object in the image signal; and rescanning the best fitting bounding box into an orthogonal grid prior to the processing step.

6. A method as claimed in claims 4 or 5, further comprising the step of storing rescanning information during the rescanning step.

7. A method as claimed in claim 6, further comprising the step of performing an inverse rescanning step on the processed image data, using the information stored during the rescanning step.

8. A method as claimed in claim 7, further comprising the step of re-transforming line segment data from the first dimension back into its pre-transformed dimensions.

9. A method as claimed in any one of the preceding claims, wherein the first dimension corresponds to a row in an image frame.

10. A method as defined in any one of the preceding claims, wherein the second dimension corresponds to a column in an image frame.

11. An image processing apparatus comprising: - means for identifying a region of interest in an image signal; means for bounding the image data corresponding to the region of interest into an orthogonal area having a first dimension and a second dimension; and a processor array for processing the image data in the orthogonal area.

12. An image processing apparatus as claimed in claim 11, further comprising means for transforaiing line segment data in the image signal into the first dimension.

13. An image processing apparatus as claimed in claim 12, further comprising means for transforming pixel data to the left and/or right of the line segment data into the second dimension.

14. An image processing apparatus as claimed in claim 12 or 13, further comprising: means for determining a best-fitting bounding box for the line segment data in the image signal; and means for rescanning the best fitting bounding box to an orthogonal grid.

15. An image processing apparatus as claimed in claim 11 , further comprising: means for determining a best-fitting bounding box for an object in the image signal; and means for rescanning the best fitting bounding box into an orthogonal grid.

16. An image processing apparatus as claimed in any one of claims 11 to 15, further comprising storage means for storing rescanning information provided by the rescanning means.

17. An image processing apparatus as claimed in claim 16, further comprising means for performing an inverse rescanning operation on the processed image data, using the information stored in the storage means.

18. An image processing apparatus as claimed in claim 17, further comprising means for re-transforming line segment data from the first dimension back into its pre- transformed dimensions.

19. An image processing apparatus as claimed in any one of claims 11 to 18, wherein the processor is a SIMD processor.