WO2007116352A2

WO2007116352A2 - Image processing system having a simd processor and a processing unit communicating via a multi-ported memory

Info

Publication number: WO2007116352A2
Application number: PCT/IB2007/051220
Authority: WO
Inventors: Johannes B. Schueler; Richard P. Kleihorst; Alexander A. Danilin
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2006-04-12
Filing date: 2007-04-05
Publication date: 2007-10-18
Also published as: WO2007116352A3

Abstract

The invention relates to an image processing system, and to a wireless camera comprising such an image processing system. The invention relies on the idea that a Single Instruction Multiple Data processor (SIMD) and a processing unit (DSP) should communicate to each other by means of an addressable memory unit (RAM), to which they both can write data objects and from which they both can read data objects. The Single Instruction Multiple Data processor is connected to the memory unit in a special way, i.e. some of the address input lines of the memory unit must be connected to the output lines of the first processing unit. The Single Instruction Multiple Data processor (SIMD) can address the memory unit via its output channels. In this way the streaming mode of the SIMD processor can be transformed into a data mode, which is compatible with the mode of operation of the processing unit (DSP).

Description

Image processing system

FIELD OF THE INVENTION

The invention relates to an image processing system, and to a wireless camera comprising such an image processing system.

BACKGROUND OF THE INVENTION

An image processing system is known from the article "A smart camera for face recognition", Richard Kleihorst et al., International Conference on Image Processing (ICIP), 2004. The article discloses an architecture which combines a parallel pixel processor with a digital signal processor. The parallel pixel processor performs low-level image preprocessing (at pixel level) which is necessary for the face detection part of the face recognition process. The parallel pixel processor works in Single Instruction Multiple Data (SIMD) mode. The face detection part also requires some high-level image processing which is mapped on the high-performance, programmable digital signal processor. The digital signal processor also performs the face recognition part of the face recognition process; it has a Very Long Instruction Word (VLIW) architecture wherein instruction fetch, data fetch and processing are performed in a pipelined fashion. For carrying out the recognition process the two processors are connected in series. The parallel pixel processor performs face detection preprocessing; the digital signal processor performs the actual face detection and recognition. There are some wireless cameras which are equipped with an image processing system of the kind set forth. Typically the derived images are transmitted via a transceiver to a remote device. The image processing system outputs a stream of video data to the transceiver; the transceiver broadcasts the video data to the remote device. The remote device may be a host, another camera or another wireless device. A disadvantage of this system is that it needs a lot of processing and transmitting power and therefore it consumes a lot of energy. SUMMARY OF THE INVENTION

It is an object of the invention to provide image processing that consumes less energy and that is relatively inexpensive. The invention is defined by the independent claims. Advantageous embodiments are defined in the dependent claims. The invention relies on the idea that a Single Instruction Multiple Data

(SIMD) processor and a processing unit (e.g. a DSP) should communicate to each other by means of an addressable memory unit (e.g. a RAM), to which they both can write data objects and from which they both can read data objects. The SIMD processor is connected to the memory unit in a special way, i.e. some of the address input lines of the memory unit must be connected to the output lines of the first processing unit. The SIMD processor can address the memory unit via its output channels. In this way the streaming mode of the SIMD processor can be transformed into a data mode, which is compatible with the mode of operation of the processing unit.

In this manner both the SIMD processor and the processing unit can work at their own speed and they can transport data objects (e.g. images) to each other. The SIMD processor and the processing unit can synchronize their communication for example by reading and writing semaphores from and into the memory unit. Avoiding complex inter- processor synchronization has a positive effect on power consumption and both the SIMD processor and the processing unit are able to work at their respective maximum performance. For example, the SIMD processor can analyze the stream of video data at full speed. If there is a part or region of interest in the video stream the SIMD processor can send it via the memory unit to second processing unit. The processing unit can specifically analyze that part or region of interest at its own speed. If necessary it makes adjustments and sends it back to the SIMD processor, for example to adjust some parameters. The SIMD processor recalculates the image and resends it to the processing unit via the memory unit. If the processing unit has completed its analysis it may output an output object such as an image, part of an image, or data, for example via a transmitter.

In an embodiment of the image processing system according to claim 2 the first data object comprises an image. In an embodiment of the image processing system according to claim 3 the second data object comprises an image.

In an embodiment of the image processing system according to claim 4, the SIMD processor and the processing unit are arranged to synchronize their communication by reading and writing semaphores from and into the memory unit. In an embodiment of the image processing system according to claim 5, the SIMD processor is arranged to perform low-level image preprocessing. In an embodiment of the image processing system according to claim 6, the second processing unit is arranged to perform mid- and high-level image processing. In an embodiment of the image processing system as claimed in claim 7, the processing unit is a Digital Signal Processor (DSP). In an embodiment of the image processing system as claimed in claim 8, the memory unit is a dual-port Random Access Memory (RAM) unit.

In an embodiment of the image processing system as claimed in claim 9, the second processing unit is coupled to a transceiver unit, the processing unit being arranged to send or receive an object to a remote device via the transceiver unit. In an embodiment of the image processing system as claimed in claim 10, the SIMD processor is coupled to a sensor unit, the SIMD processor further being arranged to receive the stream of video data via the sensor unit.

The invention is advantageously applied in wireless camera using a SIMD processor. Where prior art wireless cameras use a lot of energy by broadcasting live video, so that batteries are rather quickly emptied, the invention that combines the SIMD processor with another processor and an intermediate memory allows a wireless camera using an SIMD processor to use less energy, thereby prolonging battery life, by only transmitting events rather than live video. These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 illustrates an example of an image processing system according to the invention;

Fig. 2 illustrates an example of a platform according to the invention;

Fig. 3 illustrates a connection between an SIMD processor and a RAM unit according to the invention;

Fig. 4 illustrates an example of the architecture of the SIMD processor according to the invention. DESCRIPTION OF EMBODIMENTS

Fig. 1 illustrates an example of an image processing system according to the invention. The image processing system comprises a sensor unit (100), a SIMD processor (102), a dual-port memory unit (104), a processing unit (106) and a transceiver unit (108). In operation, the SIMD processor (102) receives a stream of video data via the sensor unit (100). It analyzes the stream and derives at least one data object from the stream. It sends the data object to the processing unit (106) via the dual-port memory unit (104) which temporarily stores the data object. The processing unit (106) reads the data object and analyzes it. The processing unit (106) may send the data object as an output object to a remote device via the transceiver unit (108). It may also send an adjusted data object back to the SIMD processor (102) via the dual-port memory unit (104), together with instructions for further adjustments. The SIMD processor (102) may in turn send an adjusted data object back to the processing unit (106) via the dual-port memory unit (104). This process may be repeated until the processing unit (106) sends the adjusted data object to the transceiver unit (108).

Fig. 2 illustrates an example of a platform according to the invention. This platform is based on three blocks. The first block is an SIMD processing for fast pixel passed operations. The second block is a multi-port RAM unit for storage of objects. The third block is a DSP or normal processor like an ARM, uP, etc. The main problem of the SIMD processor is to leave the streaming mode of operation. By connecting an external RAM to the SIMD it is possible to store objects or events into the RAM unit. The RAM unit is connected to the SMID processor in a special way, which enables leaving the streaming mode of operation. Some of the address input lines of the RAM unit must be connected to output lines of the SIMD processor. The SMID processor can address the RAM unit via the output channels. The data input lines of the RAM unit are also connected to output lines of the SIMD processor and to input lines of the SIMD processor to be able to read back the data.

It is noted that more processors can be connected to the RAM unit. They can perform separate or combined tasks in the memory unit at their own speed. The processors communicate via the memory unit by means of events (for example semaphores).

Fig. 3 illustrates a connection between an SIMD processor and a RAM unit according to the invention. Some of the address input lines of the RAM unit are connected to output lines of the SIMD processor. The SMID processor can address the RAM unit via the output channels. The data input lines of the RAM unit are also connected to output lines of the SIMD processor and to input lines of the SIMD processor to be able to read back the data.

Fig. 4 illustrates an example of the architecture of the SIMD processor according to the invention. The architecture comprises a line memory LINE-MEM, a linear processor array LPA and an output processor P-OUT. The line memory LINE-MEM usually accommodates lines of the image data which are processed by the system. The line memory LINE-MEM receives addresses on its input channel. The linear processor array LPA receives instructions on its input channel. Each processor in the linear processor array LPA shares the same instruction and address (the processing unit works in SIMD mode). Hereinafter examples of the interaction between the SIMD processing unit and the RAM unit will be explained. A first example relates to subsampling of an image, e.g. from resolution 640*480 to resolution 320*240. Subsampling is for instance used to build pyramid trees of images in order to perform image analysis on different resolutions of the same image. In this case, output outl (data) contains low-pass image data as a result of filtering performed by the SIMD processing unit. The output outl comprises the elements [pixelO, pixell, ..., pixel639]. Output out2 contains horizontal addresses: [1,1,2,2,3,3,4,4,...,319,319]. Output out3 contains the row counter divided by two, truncated to obtain an integer number. Output out2 is computed from a column identifier (column-id) to a subsample address according to the formula: out2 = column-id/2, where after out2 is truncated to obtain an integer number.

A second example relates to mirroring of an image, i.e. exchanging vertical and horizontal addresses. Mirroring is for instance used to apply orthogonal transformations (e.g. FFT) in a vertical and horizontal pass on the image. In this case, output outl contains the image data, output out2 contains the value of the row counter and output out3 contains the column counter (column-id). Usually SIMD machines cannot address differently for every pixel from their internal memory; sequential processors have to compute the addresses for every sample separately. By computing the address on the SIMD machine in a parallel way and connecting the address bus of a RAM to the outputs of the SIMD machine, coordinate transformations (if they can be expressed in an SIMD fashion) can be performed on the SIMD connected to the RAM. Typical applications for this image processing system are face recognition, hand recognition and motion detection. In the case of hand recognition, for example, the system may be used as follows:

- a sensor streams video into an SIMD processor; - the SIMD processor analyzes the stream until a hand is detected and places the cropped part into a dual-port memory;

- the SIMD sets a semaphore on a location of the dual-port memory;

- a digital signal processor (DSP) detects the semaphore and reads out the image to analyze it (for example, it checks the number of fingers, rotation, movement); - if the image needs to be reprocessed by the SIMD the DSP will set a semaphore after the image is placed back into memory;

- if no hand is detected a semaphore is set into dual-port memory such that memory can be released;

- otherwise if a hand is detected the DSP can send an event or an image via a transceiver.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and/or by means of a suitably programmed processor. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

CLAIMS:

1. An image processing system comprising: a Single Instruction Multiple Data processor (SIMD, 102) for analyzing a stream of video data and for deriving at least one data object from the stream of video data; a processing unit (106) for analyzing the data object derived from the first processing unit (102); an addressable memory unit (104) coupled to the Single Instruction Multiple Data processor (SIMD, 102) and to the processing unit (106), wherein the Single Instruction Multiple Data processor (SIMD, 102) is arranged to write a first data object into the memory unit (104), and the processing unit (106) is arranged to read the first data object from the memory unit (104), wherein the processing unit (106) is arranged to write a second data object into the memory unit (104), and the Single Instruction Multiple Data processor (SIMD, 102) is arranged to read the second data object from the memory unit (104), wherein the memory unit (104) has a plurality of address input lines and Single Instruction Multiple Data processor (SIMD, 102) has a plurality of output lines, and at least a subset of the address input lines of the memory unit (104) are connected to at least a subset of the output lines of the Single Instruction Multiple Data processor (SIMD, 102).

2. An image processing system according to claim 1, wherein the first data object comprises an image.

3. An image processing system according to claim 1, wherein the second data object comprises an image.

4. An image processing system according to claim 1, wherein the Single

Instruction Multiple Data processor (SIMD, 102) and the processing unit (106) are arranged to synchronize their communication by reading and writing semaphores from and into the memory unit (104).

5. An image processing system according to claim 1, wherein the first Single

Instruction Multiple Data processor (SIMD, 102) is arranged to perform low- level image preprocessing.

6. An image processing system according to claim 1, wherein the processing unit

(106) is arranged to perform mid- and high-level image processing.

7. An image processing system according to claim 1, wherein the processing unit (106) is a Digital Signal Processor (DSP).

8. An image processing system according to claim 1, wherein the memory unit (104) is a dual-port Random Access Memory (RAM) unit.

9. An image processing system according to claim 1, wherein the processing unit (106) is coupled to a transceiver unit (108), the processing unit (106) being arranged to send or receive an object to a remote device via the transceiver unit (108).

10. An image processing system according to claim 1, wherein the Single Instruction Multiple Data processor (SIMD, 102) is coupled to a sensor unit (100), the Single Instruction Multiple Data processor (SIMD, 102) further being arranged to receive the stream of video data via the sensor unit (100).

11. A wireless camera comprising an image processing system according to any preceding claim.