WO2009125372A1

WO2009125372A1 - Image stream processing circuit and a method of performing image processing operations

Info

Publication number: WO2009125372A1
Application number: PCT/IB2009/051511
Authority: WO
Inventors: Tomas Henriksson
Original assignee: Nxp B.V.
Priority date: 2008-04-09
Filing date: 2009-04-09
Publication date: 2009-10-15

Abstract

A first image processing operation produces a stream of images that is used by a second image processing operation. The operations are performed by concurrently executing first and second higher level processes and first and second lower level processes. The first higher level process (20) adapts an image parameter sends calls (32) to the first lower level process (26). The first lower level process (26) is executed each time for a respective one of the images, in response to a respective one of the calls from the first higher level process (20). The second higher level process (22) sends calls to a second lower level process (28). The second lower level process (28) is executed each time for a respective one of the images, in response to a respective one of the calls from the second higher level process (22). The first lower level process (26) and the second lower level process (28) exchange each image via a buffer memory (29, 106) in a plurality of successive transactions (300) that are autonomous from the first higher level process (20) and the second higher level process (22). The first high level process (20) sends signals to the second high level process (22) each for a respective one of the images, each signal indicating (34) the image parameter for the corresponding image. The calls (37a) from the second higher level process (22) to the second lower level process (28) for each of the image are sequenced after reception of the signal for that image by the second higher level process (28).

Description

Image stream processing circuit and a method of performing image processing operations

FIELD OF THE INVENTION

The invention relates to an image stream processing circuit and a method of performing image processing operations on a stream of images.

BACKGROUND OF THE INVENTION

WO2006048826 describes an integrated circuit that comprises a plurality of processing circuits connected by a network. The network is used to pass messages between the processing circuits.

Such a network may be used to process streams of images, for example to perform operations for video compression or decompression, for video editing or for preparing the images for display. When applied to processing of a stream of images, different processing circuits of the integrated circuit may perform different processing operations in series. Often, specialized co-processor circuits are used to perform specific, highly demanding types of computation. In this case images have to be passed between the different processing circuits via the network. Usually, this involves intermediate buffering of the images in a buffer memory.

Considerable flexibility of image processing is also desirable. Thus, it may be desirable to change the size of the images as a stream progresses, for example when a switch is made between different video sources in a stream of images for display on a video screen, or dependent user zoom commands etc. Furthermore, it may be desirable to switch between many different processing operations. This type of flexibility is most easily implemented by using image processing software at a level that specifies operations for images as a whole, to control image processing. Typically, this means that it also has to be possible to buffer the images as a whole between processors that perform image processing operations. This may place a high demand on integrated circuit are, which often forces use of external image memory. SUMMARY OF THE INVENTION

Among others, it is an object to provide for an image stream processing circuit and a method of processing an image stream wherein image processing operations can be controlled by image processing software at a level that handles operations for images as a whole, in combination with a co-processing circuit that performs computation intensive image processing without requiring complete images to be stored.

A method of processing a stream of images according to claim 1 is provided. Herein two higher level processes and two lower level processes are involved in processing a stream of images. All processes are executed concurrently, the lower level processes by a co-processing circuit that is distinct from the processing circuit that executes the higher level processes. The higher level processes communicate with each other at image level, setting image parameters that apply to images as a whole. Each higher level process sends calls to a corresponding lower level process, each call to start processing of an image.

The lower level processes autonomously communicate image data via a buffer memory to process the image, once they have been started for the image by a call from a higher level process. In this way, it is transparent to the high level processes whether the size of the buffer memory is smaller than needed to store a complete image. An on-chip buffer memory of small size may be used in combination with the flexibility of programs that treat images as indivisible entities in the high level processes. The high level processes may change an image parameter, such as a size of the image, among themselves and accurately control the corresponding timing of changes in processing by their lower level processes.

In an embodiment a plurality of co-processor cores is used in the processing circuit that executes the lower level processes and interface circuits are used between these co-processors and the buffer memory. In this embodiment the interface circuits are programmed using buffer management data from the higher level processes, to control transfer of image data to and from the buffer memory. In this way the buffer size can also be made transparent for the co-processor cores, so that standard co-processor cores may be used. BRIEF DESCRIPTION OF THE DRAWINGS

These and other advantageous aspects will become apparent from a description of exemplary embodiments, using the following Figures: Fig. 1 shows an image stream processing circuit Fig. 2 shows processes in an image stream processing circuit Fig. 3 shows communication between processes Fig. 4 shows an embodiment of a co-processor Fig. 5 shows a state diagram of a finite state machine

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Fig. 1 shows an image stream processing circuit, comprising an integrated circuit 10 and an off-chip memory 12. Integrated circuit 10 comprises a programmable processor 100, first and second co-processors 102, 104, an on-chip memory 106, an external interface circuit 108 and an interconnect circuit 109. Interconnect circuit 109 may be a bus or an interconnection network for example. Interconnect circuit 109 couples programmable processor 100, first and second co- processors 102, 104, on-chip memory 106 and external interface circuit 108. Furthermore, external interface circuit 108 is coupled to off-chip memory 106. Programmable processor 100 may comprise one or more processor cores (not show), one or more instruction memories (not shown) and one or more data memories (not shown), alternatively on-chip memory 106 or other memories, coupled to processor 100 via interconnect circuit 109, may be used to supply instructions and/or data. In operation, processor 100 executes programs of instructions that control at least a first image processing task that outputs a stream of images and a second image processing process that inputs the images. The programs of instructions control the image level parameters of the image processing tasks. Thus, for example the programs may set image size as a function of time, adjust filter weights of filter operations that have to be applied to images, adjust basic quantization levels to be used for an image etc. The programs of instructions may control the image level parameters for example in response to switching between different image sources, in response to bandwidth or signal to noise level changes, changes in required aspect ratio etc.

During execution the first and second image processing process activate first and second co-processors 102, 104 respectively, to perform further processes that process information in the images on a piece-by-piece basis of pieces within each image. By way of example, the further processes may include image filtering, image transformation, quantization or inverse quantization etc of imaged data from successive pieces of an image until an entire image has been processed. First and second co-processors 102, 104 exchange image pieces by writing to and reading from a buffer in on-chip memory 106.

In an embodiment, first co-processor 102 may input images of a first further image stream and process these images to produce the pieces of the images in the earlier mentioned stream between first and second co-processors 102, 104. Similarly second co-processor 104 may process pieces of the images of a second further image stream from the images of the stream between first and second coprocessors 102, 104 and second co-processor 104 may output the second further image stream. The images of first and second further image stream may be read from off-chip memory 12 and written to off-chip memory 12 respectively.

Fig. 2 shows processes and elements involved in processing the image stream between first and second co-processors 102, 104. The processes include a first higher level process 20 and a second higher level process 22, executed concurrently by processor 100, for example by one processor core, or by different processor cores. Concurrent execution of processes may be implemented by time division multiplexing or by simultaneous parallel processing. As used herein, concurrent processing means that each concurrent process has an initial starting time before the final ending times of all other concurrent processes.

A message queue 24 is provided for communication between the first and second higher level process 20, 22. The processes furthermore include a first lower level process 26 and a second lower level process 28, executed by first and second co-processor 102, 104 respectively. Arrows show communication between the processes. First and second co-processor 102, 104 communicate pieces of images via a buffer memory 29, which is implemented as an area in on-chip memory 106. Fig. 3 shows part of the communications from the processes in more detail. Lines 3-20, 3-22, 3-24, 3-26, 3-28, 3-29, 3-102, 3-104, 3-106 are shown, to symbolize first higher level process 20, second higher level process 22, queue 24, first lower level process 26, second lower level process 28, buffer memory 29, first co- processor 102, second co-processor 104 and on-chip memory 106, respectively.

First higher level process 20 executes a program wherein it controls image processing of a stream of images. In an embodiment this program may comprise instructions to changes image parameters of the images, such as image size, basic quantization level etc. Furthermore the program comprises instructions that control cycles of application of piece by piece image processing in individual images. After previous cycles wherein images have been processed, first higher level process 20 initiates processing of a new image by configuring communications 30a,b with on- chip memory 106 and/or first co-processor 102 to control the configuration of buffer space in on-chip memory 106 that will be used by first co-processor 102 to process the new image. The amount of configured buffer space is smaller than the buffer space needed for a complete image. The amount may be less than one tenth of the space needed for the entire image for example. Thus the demand for on-chip buffer space is reduced.

First higher level process 20 next sends a higher to lower level communication 32 to first lower level process, commanding it to start producing the new image. Subsequently, first higher level process 20 sends a high to high level communication 34 with a message to queue 24, to enter a message. The message contains information about the size of the new image, the location of the buffer space in on-chip memory 106 and the granularity of image transfer. After this first high level process 20 may switch to a wait state, or perform other tasks.

Like first higher level process 20, second higher level process executes a program wherein it controls image processing of a stream of images. In an embodiment this program may comprise instructions to changes image parameters of the images, such as image size, basic quantization level etc. Second higher level process 22 sends a high to high level communication 35 to queue 24 to request the message from queue 24. When the message has been entered, queue 24 returns the message to second higher level process 22 in a further high to high level communication 36. In response to the message, or to other inputs, second higher level process 22 may adapt processing parameters.

In response to the message from queue 24, second higher level process 22 sends a high to low level communication 37a to second lower level process 28 to command the start of processing. After this second high level process 22 may switch to a wait state, or perform other tasks. Optionally second higher level process 22 may first send a configuring communication 37b to second co-processor 104 to configure the buffer according to information from the message, Alternatively, this information may be included in high to low level communication 38. Subsequently, first and second lower level process 26, 28 perform transactions 300 to exchange a series of image data items, representing pieces of an image, from the new image via the buffer in on-chip memory 106. The individual transactions are autonomous from the first and second higher level process 20, 22, in the sense that after the initial call for the image the lower level processes start each transaction at a time point selected without control from any further signal from the higher level processes. This may involve communicating signals (not shown) to indicate when there is space for writing data in the buffer and when there is data in the buffer for reading. When first lower level process 26 has written all image data from the new image, it sends a low to high level communication 38 to first high level process 20. This may be a signal to switch first high level process 20 from a wait state and/or an interrupt signal. Similarly, when second lower level process 28 has read and processed all image data from the new image, it sends a low to high level communication 39 to second high level process 22. After that, the cycle may repeat with new configuring communications 30a,b from first higher level process 20 and so on.

Fig. 4 shows an embodiment of a co-processor comprising an interface circuit 40 and a processor core 42. Interface circuit 40 is coupled between processor core 42 and interconnect circuit 109, with signal connections for passing commands from processor core 42 to interface circuit, for passing data, or pointers to data, and for handshaking. Processor core 42 has a connection to interconnect circuit 109 bypassing interface circuit 40, at least for receiving commands from higher level processes in another processor (not shown). In operation, interface circuit 40 controls image data transfer to and/or from the buffer in the on-chip memory (not shown) and co-processor processes the image data and issues commands when items of image data are needed or need to be written. In a further embodiment, processor core 42 may be configured to address image data with addresses that are capable of addressing a complete image in memory (e.g. using different addresses for all pixels in the image) and interface circuit 40 translates these addresses into actual buffer addresses, dependent on the address of a piece of the image that is currently stored in the buffer memory, and its location in that buffer memory.

In an embodiment interface circuit 40 may be configured to operate as a cache memory, the interface circuit 40 comprising a cache management circuit, a local memory and an address translation circuit that translates image based addresses (e.g. addresses that distinguish all pixels of a complete image) from processor core into addresses in the local memory that store the image data with the address. In this embodiment the cache management circuit controls fetching or writing of image data between the buffer memory and the local memory. Fig. 5 shows states of a state machine to control communication in a low level process executed by a co-processor. By way of example a state machine for only handling reading is shown. The states include an idle state 50, a prepared to read state 52 and a read accept state 54. In operation the interface circuit 40 is initially in idle state 50. Upon receiving a communication from second higher level process 22, the state machine transitions to the prepared to read state 52. As shown in the state diagram this transition takes place via an intermediate acquire state 51 , wherein the low level process configures the buffer in on-chip memory 106. When this has been completed the transition to the prepared to read state 52 is completed. When there is data in the buffer, the state machine signals the availability of data to processor core 42 and transitions to the read accept state 54. Processor core 42 now processes the data from the buffer. When processor core 42 has received the data from the buffer this is signaled to the state machine. In response the state machine transitions back to the prepared to read state 52 or to the idle state 50, dependent on whether a complete image has been handled by the accumulation of previous transitions to the read accept state 54.

When the state machine transitions back to the idle state 50, it generates a low to high level signal to the higher level process indicating completion of reading of the image. This signal may be an interrupt signal. To make generation of this signal explicit an intermediate finished state 55 wherein the signal from low to high level is generated.

Transitions back and forth between the prepared to read state 52 and the read accept state 54 occur a plurality of times for each image. When the state machine transits back to the prepared to read state 52 it signals to the buffer that the space for the received data is free.

Similar state diagrams apply to interface circuits for writing and for combined reading of input images and writing of output images. In the case of writing the prepared to the prepared to read state 52 is replaced by a "buffer space available" state and the read accept state 54 is replaced by a "data written" state, with transitions between them forth when it is signaled that processor core 42 has written data and back when the buffer indicates that there is free space for a next grain of data. In the case of combined reading of input images and writing of output images, interface circuit 40 may implement both state machines for reading and writing respective buffers, or a combined state machine may be used, wherein a transition sequence is defined between a "prepared to read" state, a "buffer space available" state for one buffer and a "read accept" state and a "data written" state for another buffer.

Although an embodiment has been described with two co-processors, performing two lower level processes communicating via a buffer, it should be appreciated that a larger number of lower level processes may used, executed on the two co-processors or on a larger number of co-processors (not shown) in integrated circuit 10. As another alternative, both the first and second lower level process may be executed by a single co-processor on a time division multiplex basis. Although an embodiment has been described wherein image data is exchanged via an on-chip memory 106 that is coupled to interconnect circuit 109, it should be appreciated that the image data may be exchanged via memory in the interconnect circuit 109, between connections to co-processors 102, 104.

Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

Image stream processing circuit and a method of performing image processing operationsCLAIMS:

1. A method of performing a first and second image processing operation, the second image processing operation operating on a stream of images produced by the first image processing operation, the method comprising: executing a first higher level process (20) under control of a first program of instructions, the first program being configured to adapt an image parameter and to send calls (32) to a first lower level process (26), both during processing of the stream; executing the first lower level process (26) each time for a respective one of the images, in response to a respective one of the calls from the first higher level process (20), to perform the first image processing operation; executing a second higher level process (22) concurrently with the first higher level process (20), under control of a second program of instructions, the second program being configured to send calls to a second lower level process (28); executing the second lower level process (28) each time for a respective one of the images, in response to a respective one of the calls from the second higher level process (22), to perform the second image processing operation, the first lower level process (26) and the second lower level process (28) exchanging each image via a buffer memory (29, 106) in a plurality of successive transactions (300) that are autonomous from the first higher level process (20) and the second higher level process (22); sending signals (34) from the first high level process (20) to the second high level process (22) each for a respective one of the images, each signal indicating (34) the image parameter for the corresponding image; sequencing the calls (37a) from the second higher level process (22) to the second lower level process (28) for each of the image after reception of the signal for that image by the second higher level process (28).

2. A method according to claim 1, comprising: deriving content of buffer management information from the signal processing parameter in the first and second higher level processes (20, 22); - transmitting the buffer management information from the first and second higher level processes (20,22) to the first and second lower level processes (26, 28) respectively; controlling use of the buffer memory (29, 106) under control of the buffer management information in the first and second lower level processes (26, 28).

3. A method according to claim 2, comprising: executing the first and second lower level processes in respective coprocessor cores (42); providing interface circuits (40) between the co-processor cores (42) and the buffer memory (106); storing at least part of the buffer management information of the first and second lower level processes in the interface circuits (40) respectively; controlling access of the co-processor cores (42) to the buffer memory (106) from the interface circuits (40), dependent on the buffer management information in the interface circuits (40).

4. A method according to claim 1, comprising: executing the first and second program with an instruction processing circuit (100) and - executing the first and second lower level process (26, 28) with respective co-processors (102, 104) coupled to the instruction processing circuit (100) via an interconnect circuit (109), the interconnect circuit (109) passing the calls (32, 27a) from the first and second higher level processes to the first and second lower level processes (26, 28).

5. A method according to claim 1, wherein the image parameter comprises information indicating a size of the image.

6. A method according to claim 1, wherein buffer space in the buffer memory (29) is smaller than needed for storing a complete image from the stream, the first and second lower level process (26, 28) exchanging the image using a plurality of transfer communications (300) each plurality of transfer communications (300) following a respective call (32, 37a) from the first and second higher level process (20, 22).

7. An image stream processing circuit for applying at least a first and second image processing operation that produce and consume a stream of images respectively, the processing circuit comprising: a first processing circuit (100) configured to execute a first and second higher level process (20, 22) concurrently, under control of a first and second program of instructions respectively, the first and second program being configured to generate calls (32, 37a) for each image from the stream to a first and second lower level process (26, 28) respectively, the first program, moreover, being configured to adapt an image parameter during processing of the stream; a buffer memory (29, 106); a second processing circuit (102, 104) configured to execute the first and second lower level processes (26, 28), the second processing circuit (102, 104) being coupled to the buffer memory (106) for exchanging the images between the first and second lower level process (26, 28); and wherein the first and second higher level process (20, 22) are configured to send signals (34) from the first high level process (20) to the second high level process (22), each signal (34) for a respective one of the images, each signal (34) indicating the image processing parameter for the corresponding image; and wherein the second higher level process (22) is configured to sequence the call (37a) from the second higher level process (22) to the second lower level process (28) for each of the images after reception of the signal (35) for that image by the second higher level process.

8. An image stream processing circuit according to claim 7, wherein buffer space in the buffer memory space (106) is smaller than needed for storing a complete image from the stream, the first and second lower level (26, 28) process being configured to exchange the complete image using a plurality of transactions (300) following the calls (32, 37a) from the first and second higher level process (20, 22) for the complete image, the transactions (300) between the first and second lower level process (26, 28) being autonomous from the first and second higher level process (20, 22).

9. An image stream processing circuit according to claim 7, wherein the first and second higher level processes (20, 22) are configured to transmit the buffer management information from the first and second higher level processes (20, 22) to the first and second lower level processes (26, 28) respectively; and the first and second lower level processes (26, 28) are configured to control use of the buffer memory under control of the buffer management information.

10. An image stream processing circuit according to claim 9, wherein the second processing circuit (102, 104) comprises: a first and second processor core (42), configured to execute the first and second lower level process (26, 28) respectively, a first and second interface circuits (40) coupled between the buffer memory (106) and the first and second processor core (42) respectively, the first and second interface circuit (40) being configured to control access of the first and second co-processor core (42) to the buffer memory (106) respectively, dependent on buffer management information from the first and second lower level processes (26, 28), stored in the interface circuits (40).

11. An image stream processing circuit according to claim 7, wherein the first processing circuit (100) comprises a programmable processor configured to execute both the first and second program.

12. An image stream processing circuit according to claim 7, wherein the image parameter comprises information indicating a size of the image.