GB2538754A - Single-chip multi-processor communication - Google Patents

Abstract

A heterogeneous multi-core integrated circuit having processors 21 comprising general purpose CPU 21A and second processor 21B, such as a specialized hardware processing engine. The processors are connected by local bus 22, such as Wishbone, CoreConnect, or AXI, and a memory bus on the integrated circuit. The CPU generates an instruction for an atomic operation to be performed by the second processor, the instruction comprising an address of the second processor and a command. The command comprises an action, a descriptor of the action, or a pointer to find the action in an instruction memory 23. The CPU transmits the first instruction over local bus 22 as a distributed signalling mechanism (DSM) signal to the second processors message FIFO buffer. DSM signals are also used for communication among different CPUs, where each CPU is aware of the number of slots available in their message FIFOs. Interrupt requests are replaced with register writes.

Description

Intellectual Property Office Application No. GII1509076.4 RTM Date:18 November 2015 The following term is a registered trade mark and should be read as such wherever it occurs in this document:

AMBA

Intellectual Property Office is an operating name of the Patent Office www.gov.uk/ipo Single-Chip Multi-Processor Communication The invention provides methods and apparatus for improving the efficiency of communication between multiple processors and data-handling engines within an integrated circuit

Background

In modem computing, it is common for data processing and production to be carried out by several processors on a single integrated circuit or chip. Such integrated circuits are often known as multi-core chips and may include multiple general-purpose processors ("CPUs") and specially-designed hardware processing engines, which may each be designed (optimised) for particular tasks such as coding and decoding as well as input and output, whereupon the chip is often called heterogeneous. The use of multiple processors, whether CPUs and/or specialized hardware processing engines, allows parallel processing to be carried out, which improves the speed of processing, but means that communication between the various processors is needed to ensure that the system is being used as efficiently as possible and to avoid problems caused by, for example, data being processed out of order.

The current-art methods of on-chip communication include polling and interrupts. Polling means that a CPU sends signals to other CPUs or appropriate engines and awaits a response that indicates, for example, that the other CPU or engine is ready to receive data. This results in inefficiencies because the signalling CPU must wait for a response, which introduces delay. It also uses processing power that could be more usefully used for other tasks.

An interrupt is a signal sent down a dedicated wire built into an integrated circuit. This means that not only are such dedicated lines necessary, but signals can only be sent to specific places, which may not be desirable if other destinations are required.

Message Signal Interrupts (MSIs) are an alternative type of interrupt. The effect of the NISI is to set an interrupt line/request locally at the controlling processor, so it is the equivalent of an interrupt line without requiring the special dedicated wiring. An MSI comprises a data packet arranged in a specific format containing a small amount of data and an instruction, and is sent between integrated circuits. Such MS's are very strictly defined, which makes it difficult to use them for varied purposes.

In either case, a controlling processor commands a device to perform an action by performing one or more register writes. When the action completes, the device notifies the processor that it requires attention. It does this by asserting an interrupt line or sending an NISI containing an interrupt number, depending on which interrupt method is in use. The CPU detects, by architecture-dependent means, that the interrupt line/request has been set, and performs one or more register reads on the device to determine the reason for the interrupt, and if necessary extract data from the device.

Some less-commonly-used methods of on-chip communication are barriers and hardware semaphores, which comprise a serial link between CPUs These depend on a routing mechanism and are also serial in nature, meaning only one message can be sent at a time.

This introduces delay.

In all cases, memory locks are likely to be required in order to protect data from being changed by another CPU or engine (collectively known as data handlers) before the signal has been dealt with. This may result in interference between data handlers as a locked area of memory cannot be accessed in any way. Both types of signals are also only used between pairs of data handlers; they cannot be forwarded on.

Thus, although it is known for multiple data handlers to be present in an integrated circuit, there is no known efficient way to co-ordinate them. The present invention aims to address this problem.

Overview Accordingly, in a first aspect, the invention provides a heterogeneous multi-core integrated circuit comprising two or more processors, at least one of the processors being a general purpose CPU and at least one of the processors being a specialized hardware processing engine, the processors being connected by a processor local bus on the integrated circuit, wherein the general purpose CPU is configured to: generate a first instruction for an atomic operation to be performed by a second processor, different from the general purpose CPU, the first instruction comprising an address of the second processor and a first command indicating a first action to be executed by the second processor; and transmit the first instruction to the second processor over the processor local bus.

According to a second aspect, the invention provides a method of communication on a heterogeneous multi-core integrated circuit comprising two or more processors, at least one of the processors being a general purpose CPU and at least one of the processors being a specialized hardware processing engine, the processors being connected by a processor local bus on the integrated circuit, wherein the method comprises: generating, by general purpose CPU, a first instruction for an atomic operation to be performed by a second processor, different from the general purpose CPU, the first instruction comprising an address of the second processor and a first command indicating a first action to be executed by the second processor; and transmitting the first instruction to the second processor over the processor local bus.

Preferably, the first command comprises at least one descriptor indicating what the first action comprises. The first command may comprise a pointer to at least one descriptor located in memory indicating what the first action comprises. In one embodiment, the first instruction comprises data on which the first action is to be performed, when executed. The first instruction may comprise a pointer to data located in memory on which the first action is to be performed, when executed.

In a preferred embodiment, the general purpose CPU is configured so that the first instruction cannot be generated until any data required for execution of the first action is available.

The general purpose CPU is preferably further configured to: receive, over the processor local bus, a second instruction from another processor for an atomic operation to be performed by the general purpose CPU, the second instruction comprising an address of the general purpose CPU and a second command indicating a second action to be executed by the general purpose CPU; and execute the second action.

Preferably, the second action is the generation of the first instruction, wherein the second instruction includes the address of the second processor. The second instruction may include the first command for inclusion in the first instruction.

In one embodiment, the heterogeneous multi-core integrated circuit further comprises an input buffer for storing the received second instruction until the second instruction is read by the general purpose CPU and the second action is executed The first instruction may include information indicating the amount of space available in the input buffer for storage of other instructions Preferably, the processor local bus comprises any one of a Wishbone Bus. a CoreConnect bus or an Advanced Microcontroller Bus Architecture, AMBA, Bus, for example a Advance eXtensible Interface, AXI, Bus.

The second processor may be a general purpose CPU or a specialized hardware processing engine.

Preferably, the specialized hardware processing engine is configured to process signals that are formatted in Universal Serial Bus, USB, protocol According to another aspect of the invention, there is provided a method of communication between data handlers using a distributed signalling mechanism (DSM) comprising: a first data handler preparing a DSM signal comprisingan atomic data write over a general-purpose communication bus between a known source and a known destination, the first data handler also adding an identification of a second data handler which is configured such that it will be able to correctly interpret the received DSM signal; the first data handler placing the DSM signal on a general-purpose intra-chip communication medium; the DSM signal being sent to the second data handler as a data write using the conventional mechanism for the medium; and the second data handler receiving the DSM signal and acting on it; wherein one or both of the data handlers are CPUs.

The DSM signal preferably acts as an immediate notification that a process is complete or that a process can be started. This means that there will be rules in place preventing the DSM signal from being prepared until all relevant data is ready.

In this context, an atomic data write is defined as data written to a memory location via an operation that cannot be broken up into smaller parts and therefore cannot be interrupted by other processes. This prevents so-called 'race conditions', whereby it is impossible to predict which of two processes being carried out by two data handlers will be finished first. Since the data write that forms a DSM signal is atomic, once it begins it appears to be instantaneous and is uninterruptable. It is therefore predictable.

This method is beneficial because it provides a method of communication that does not require polling or the use of dedicated interrupt lanes. The fact that the DSM signal carries its own data in an atomic form is beneficial because it means that vital data will not be changed or overwritten but removes the need for a lock that might interfere with other data handlers, thus also allowing multiple data handlers to work on the same data. As a further benefit, this method is easy to scale, meaning that a system implementing this method can be very large, comprising multiple data handlers, without the number of connections required becoming unwieldy as would be the case under any of the current-art systems.

The DSM signal preferably contains a pointer to a descriptor: a piece of stored data that identifies how other data is to be processed and stored, but can contain any data or commands in any combination required for effective communication between data handlers. Examples include: * A command and data on which the command should be carried out * A command and a pointer to a memory location where data is stored * A combination of one or more pointers to descriptors and data * Data on which the data handler should carry out a pre-defined function * A flag indicating success or failure of another function Other formats or data may be included up to the maximum size of the DSM signal, 25 depending on the exact embodiment Preferably, the preparation of a DSM signal is triggered by receiving a DSM signal from a third data handler. This may be through a command contained within the DSM signal itself or as a result of instructions fetched from memory. This is beneficial because it means that the destination and contents of a DSM signal can be programmed rather than needing to be known in advance as part of the main instructions executed by the data handler.

Preferably, the second and third data handlers are different data handlers. This allows chaining of DSM signals, which is beneficial because it means that instead of there being a single controlling data handler that sends a DSM signal to a second data handler and waits for a response before sending a second DSM signal to a third data handler, it is possible for a first data handler to send a single DSM signal and wait for data to be fully processed in a co-ordinated manner by an entire pipeline. This leads to more efficient co-ordination and avoids unnecessary messaging which uses processing power and space in the local communication media. It also improves timing as a DSM signal will not be sent until data has been prepared and all relevant functions have been completed, as aforementioned, but once this is the case it will be sent immediately. This means that all data handlers in a chain will be notified of prepared data and able to access it as promptly as possible.

Preferably, more than one DSM signal may be prepared as a result of a single set of instructions. This is beneficial because it allows a data handler to send messages to multiple other data handlers as may be appropriate and there is no need for repetition, improving efficiency.

Preferably, the method includes preventing an excessive number of DSM signals from being sent to a data handler, comprising the inclusion of 'credits' within the DSM signal to inform other data handlers of the space available in a notional queue This is beneficial as it means that error prevention methods can be put in place to avoid a data handler becoming swamped by an excessive number of DSM signals, which might lead to delays and potentially lost data, which could in turn lead to instructions not being carried out.

Preferably there are different methods depending on the data handlers in question, for example one method for 'intelligent' CPUs sending to 'dumb' engines and a second method where CPUs are communicating with one another.

In a preferred embodiment, a CPU ("DH1") may send a command via DSM signal to a dumb device ("DH2") knowing that it will generate a known non-zero number of responses. DH1 is aware of the size of the input buffer associated with D2 (or at least a part of the input buffer accessible by DH1) as well as the size of its own input buffer and will therefore not issue commands until all necessary buffers have enough space. The awareness of the space in the respective buffers constitutes the 'credits' and therefore in this part of the system the method of sending 'credits' in the DSM signal is not used.

In a continuation of the preferred embodiment, where one CPU ("DH3") is communicating with a second CPU (-DH4") an alternative method is provided whereby DH3 allocates a number of 'slots' in its DSM input buffer that are reserved for DSM signals from DH4 and includes the number of available slots in each DSM signal it sends to DE14.

Assuming that there is a consistent exchange of messages, DH4 will be regularly updated on the number of slots DH3 has available for its DSMs and vice versa. Neither data handler will send a DSM signal if there is no space at its destination.

Preferably, in an embodiment such as that described above, there is further provided a method whereby, where there is little traffic between DEI3 and DH4 as described above, DE13 may send a blank DSM signal carrying only the information on the credits it has available for DH4 and vice versa.

These example methods are described as part of the same embodiment, but could also be used separately.

Advantageously, there may be provided a method whereby an incoming DSM signal may be copied into local memory and replaced with a reference to the location in memory where the contents of the DSM signal can be found. This dramatically increases the number of DSM signals that can be 'stored' in an input buffer.

According to a further aspect of the invention, there is provided a data handler comprising: a data input arranged to fetch or receive data for processing; a processor or engine arranged to execute instructions according to its function and prepare a DSM signal according to a method as described above; optionally, an output for processed data, and an output for DSM signals This arrangement is beneficial because it allows a data handler to be controlled using DSM signals The use of an input buffer is particularly beneficial as it allows the data handler to handle DSM signals in the order in which they were received and the data handler sending the DSM signal does not need to check if it was received; it is guaranteed to be received and eventually handled According to a still further aspect of the invention, there is provided a data handler 30 comprising: a data input arranged to fetch or receive data for processing; a message input arranged to receive a DSM signal that has been produced according to a method as described above; a buffer arranged to store received DSM signals and release them in the order in which they were received; a processor or engine arranged to fetch a DSM signal from the buffer, process the DSM signal as appropriate, depending on its contents, optionally, prepare a further DSM signal according to a method as described above, optionally, an output for processed data, and, optionally, an output for further DSM signals.

Preferably, the 'credits' mentioned above comprise the amount of space left in the input buffer.

According to anotheraspect of the invention, there is provided a system for processing data comprising: two or more data handlers as described above; at least one memory arranged to hold data before it is processed, at least one memory arranged to hold data after it is processed, a bus connected to all the data handlers and arranged to carry DSM signals; a bus connected to all the data handlers and the memories and arranged to carry data; and one or more data outputs.

The memories arranged to hold data before and after it is processed may comprise separate memories or may be different areas of the same memory, or the data may be interleaved within a single area of memory. The data may be processed by multiple data handlers In a preferred embodiment, the bus used to carry DSM signals is arranged according to the Advanced Extensible Interface (AX!) standard This is beneficial because the AXI standard is already widely implemented in integrated circuits, which means that the methods of the invention will be easier to deploy and use. It is also less used for other types of data while at the same time commonly being fast enough and having sufficient bandwidth to allow a significant amount of data to be sent as the atomic write needed for a DSM signal.

The busses arranged to carry data and DSM signals may be separate busses or may be the same bus. However, in a preferred embodiment, the DSM signal and any data output are transmitted over the same bus which has a guaranteed 'flight time' for data written on it. This is beneficial as it means that if the first data handler writes data to local memory and then sends a DSM signal relating to that data, it can guarantee that by the time the DSM signal arrives the data write will be complete. This means that there is no need for the first data handler to wait for a confirmation, reducing latency.

This is a further benefit of the use of an AXI bus: the bus is able to carry data and descriptors as well as DSM signals and, as a result, it is possible to send the DSM signal directly after sending data and/or descriptors to memory as appropriate, thus ensuring that the DSM signal will arrive at its destination after the data and/or descriptors In a preferred embodiment, the system is arranged such that all data handlers and memory appear to be sections of memory and have memory addresses. The addressing system used to send DSM signals to the appropriate data handlers therefore operates as if they were being written to memory. This is beneficial as it takes advantage of existing methods, which makes implementation more straightforward.

Preferably, the system is embodied as an integrated circuit on a single computer chip, but the method of the first aspect of the invention could also be used for signalling between different integrated circuits if a suitable bus is provided.

Short Description of the Drawings

Embodiments of the invention will now be more fully described by way of example, with reference to the drawings, of which: Figure 1 schematically shows movement of data and DSM signals between two data handlers according to an embodiment of the invention; Figure 2 shows a schematic of a system involving four data handlers and an instruction memory; Figure 3 is a flowchart of a process carried out by the system of Figure 2; and Figure 4 schematically shows another embodiment of a system according to the invention.

Detailed Description of the Drawings

Figure 1 shows a simple version of a system according to one embodiment of the invention comprising an integrated circuit having two data handlers [H, 12], such as a CPU and a specialised hardware processing engine, connected by a bus [14] that carries DSM signals. The second data handler [12] is further shown with a DSM signal buffer [13] The first data handler [11] would also have such a buffer, but it is not shown here for clarity. There is also a second bus [15] which is connected to data memory [16] and to the two data handlers [11, 12]. There would also be other components in the system, but for clarity these are not shown A DSM signal is a register write to the address of the second data handler's input FIFO [13], which is programmed as part of the function being carried out by the first data handler [11]. The data being written is not pre-programmed, but is a combination of data pre-programmed in the same way as the address and data generated by the first data handler [1 1], usually but not always some form of status. The exact combination is defined by the needs of the system with regard to the needs of the particular data handlers [11, 12]. Typically, there will be an identification of the command that has completed and the first data handler [11] will also insert other data as appropriate to indicate the status of the completed command.

Both data handlers [11, 12] must have been configured such that they have a common format, such that the second data handler [12] will be able to interpret DSM signals sent by the first data handler [11].

When the DSM signal is sent by the first data handler [11], it is placed on the message bus [14] addressed such that it is written into the second data handler's DSM signal buffer [13]. If there are already other DSM signals in the buffer [13], the new DSM signal will be placed at the back of a notional queue; otherwise it may be fetched from the buffer and handled immediately. The route of the DSM signal is shown by the dashed arrows.

If either data handler [11, 12] requires data for processing, this is fetched from memory [16] via, in this embodiment, a second bus [15]. Because the DSM signal will not be sent until the data is ready, the data is guaranteed to be present and the data handler that needs the data is able to fetch the data without any further delay. In Figure 1, the route of fetched data is shown by the plain arrows and only includes the second data handler [12] although, of course, the first data handler [11] would also be connected to the data bus [15].

Figure 2 shows another embodiment, which is a more complex version of the system comprising an integrated circuit having four data handlers [21], a bus configured to carry DSM signals [22] connected to the data handlers through the paths shown by dashed arrows, and an instruction memory [23] holding instruction sets in the form of descriptors [24], which are fetched along the paths shown by dashed arrows. There will, of course, be other components, but these are not shown for clarity.

Figure 3 is a flowchart showing an example of the process in this embodiment, with reference, also, to Figure 2 At Step 530, the system is booted and configured and descriptors [24] are loaded into memory. Each descriptor [24] contains instructions for data processing and the production of a DSM signal to be sent to an address contained in the descriptor [24]. In Figure 2, these addresses are shown by the lettered boxes within the descriptors [24].

In this embodiment, the descriptors are created at start-up, but they can also be written on 10 the fly in order to provide maximum flexibility. They may also be amended according to, for example, selection of display output standards to be used.

At Step S3], Data Handler A [21A], which is a CPU, carries out a task according to its respective descriptor [24] At Step S32, Data Handler A [21A] prepares a DSM signal to be sent to the address stored in the descriptor [24] that it was executing: in this example, that of Data Handler B [21B]. It will only do this if it knows that there is enough space in Data Handler B's message buffer and also in the message buffers of Data Handlers C and D, since it knows that Data Handler B's descriptor will trigger the production of DSM signals sent to Data Handlers C and D. The DSM signal will also contain instructions for Data Handler B [21B] to execute the appropriate descriptor stored in memory [23], and is placed on the bus [22] to be written to the message buffer associated with Data Handler B [21B] at Step S33, At Step S34, Data Handler B [21B] receives the DSM signal from Data Handler A [21A] and, as instructed, fetches the described descriptor [24] from memory [23]. It then executes the instructions therein, which include an instruction to prepare two DSM signals addressed to Data Handlers C and D [21C, 21D], in the same way as Data Handler A [21A] did. In this way, DSTvl signals can be chained such that the receipt of one DSM signal triggers the sending of another. The DSM signals are sent at Step S36. Data Handler B can be sure that there is space in Data Handler C and D's message buffers, as otherwise Data Handler A would not have sent the original DSM signal that began the chain.

At Step SC37, Data Handler C [21C] receives its DSM signal and fetches its descriptor [24] at Step 5C38. It then executes the instructions in the descriptor [24], which include preparing a DSM signal at Step SC39 to be sent back to Data Handler A [21A] at Step SC310. This means that Data Handler A [21A] will be notified that this limb of the process is complete and the receipt of this DSM signal may trigger a further process, or alternatively Data Handler A [21A] may wait to receive a DSM signal from Data Handler D [21D] indicating that the entire process is complete.

In this example, Data Handler D [21D] had a queue of DSM signals awaiting processing in its message buffer. This means that at Step SD37 the new DSM signal from Data Handler B [21B] was placed in the message buffer to await processing in turn. It is not actually received by Data Handler D [21D] until Step SD38, but Data Handler D [21D] then fetches the descriptor at Step SD39. In this example, Data Handlers C and D [21C, 21D] are both carrying out the same task and so are both directed to the same descriptor [24]. They are carrying it out on different data, which might be carried in their respective DSM signals or alternatively the DSM signals might carry further memory addresses where the data can be found.

Having fetched the descriptor [24], Data Handler D [21D] executes it as if there had been no delay, including preparing a DSM signal addressed to Data Handler A [21A] at Step S310 This is then sent to Data Handler A [21A], notifying it that the entire process is complete.

The total number of potential DSM signals must never exceed the capacity of the message FIFOs to which they are written. In the above case, where the signals are primarily being sent to dumb engines, this is handled by the controlling CPU [21A] not issuing DSM signals to the engines [2]B, C, D] unless it is certain that it has space to receive all the status returns that may be generated as a result of those commands and that the engines [21B, C, D] will have space to receive DSM signals sent to them.

Such a technique could also be used with CPUs organised in a strict master/slave relationship. However, this is not ideal as CPUs are relatively intelligent and may need to communicate asynchronously, with DSM signals not generated in response to particular commands. A sending CPU must know that there is space in the receiving CPU's message FIFO before it sends a DSM signal. As such, a slightly different system is used where there are multiple CPUs.

Figure 4 shows a system with three CPUs [41, 42, 43] on a single chip, each with a message FIFO [44, 45, 46], connected to a bus [47] in a situation where the system has just been started up. The three CPUs [41, 42, 43] are able to communicate with one another through DSM signals as hereinbefore described, but because they are all able to act independently and not only under instructions as is the case for a dumb engine, each CPU [41, 42, 43] needs to be aware of the space available for its messages in the message FIFOs [44, 45, 46] of all the connected CPUs [41, 42, 43].

As with dumb engines that expect to receive DSM signals from multiple sources, each CPU [41, 42, 43] has a number of slots in its message FIFO [44, 45, 46] allocated to each other CPU [41, 42, 43], such that there are essentially multiple FIFOs connected to each CPU [41, 42, 43]. Each CPU [41, 42, 43] is aware of the number of slots in its message FIFO [44, 45, 46] that are actually available for the use of the other CPUs [41, 42, 43], though in Figure 4 this space is shown greyed out.

When the system starts up, each CPU [41, 42, 43] believes that there is only one slot available for it in every other message FIFO [44, 45, 46]. In Figure 4, these slots are shown by the patterning of the FIFOs [44, 45, 46], such that the first CPU [41] has no shading and its slots in the message FIFOs [45, 46] of the other two CPUs [42, 43] also have no shading.

This CPU [41] will be used as an example, but the other two CPUs [42, 43] behave in an identical way.

The first CPU [41] initially reserves space in its message FIFO [44] for both the other CPUs [42, 43] as well as any dumb engines from which it might expect DSM signals. The amount of space for each CPU [42, 43] is determined by allocation at software design time.

The CPU [41] then uses the single slot it believes it has access to [48] in the second CPU's [42] message FIFO [45] to send a DSM signal to the second CPU [42] informing it of the space assigned to it. The second CPU [42] will update its configuration accordingly.

This could be done through configuration registers in each CPU [41, 42, 43] holding the number of slots available in the other CPUs [41, 42, 43] and the connected engines.

Whenever any CPU [4], 42, 43] or engine sends a DSM signal to a CPU [41, 42, 43], it puts the number of slots available for the use of that CPU [41, 42, 43] or its processes into the message and the CPU [41, 42, 43] records them in the configuration registers. This would require no-operation DSM signals to be sent periodically if there were no operations needed, but this would be possible due to the flexible format of the DSM signal.

The same process is followed by the first CPU [41] for the third CPU [43] and the second and third CPUs [42, 43] likewise follow the same process. In this way, all three CPUs [41, 42, 43] can be made aware of the space available for DSM signals in the message FIFOs [44, 45, 46] of all connected CPUs [41, 42, 43] and can therefore ensure that they do not send DSM signals to any CPU [41, 42, 43] that does not have space for it. This reduces the chance of the signal being lost.

If there are multiple CPUs and multiple devices, a single device can be shared between multiple CPUs by splitting the message FIFO in a similar way to a CPU that is expecting to receive DSM signals from multiple other CPUs. Each controlling CPU is then allowed to submit as many descriptors (identified by address/length) as can fit in its allocated space in the engine's message FIFO. It is up to the issuing CPU to ensure that the status address and partial data programmed into the descriptor route any DSM signal produced by the process to an appropriate place.

Although only a few particular embodiments have been described in detail above, it will be appreciated that various changes, modifications and improvements can be made by a person skilled in the art without departing from the scope of the present invention as defined in the claims. For example, hardware aspects may be implemented as software where appropriate and vice versa Furthermore, instructions to implement the method may be provided on a computer readable medium.

For example, from a different perspective, the current art may be considered as follows: * A controlling processor commands a device to perform an action by doing one or more register writes.

* When the action completes, the device notifies the processor that it requires attention. It does this by asserting an interrupt line (old fashioned) or sending an NISI containing an interrupt number (modern). The effect of the MSI is to set an interrupt line/request locally at the controlling processor, so it is the moral equivalent of an interrupt line without the wiring.

* The CPU notices, by architecture dependent means, that the interrupt line/request has been set, and performs one or more register reads on the peripheral (perhaps from an interrupt program) to determine the reason for the interrupt, and if necessary extract data from the 30 device.

Note that there is a simplicity about the interrupt arrangement: since it is effectively a single bit, whether carried by wire or MSI, it does not matter it is is asserted many times. It is simply a call for attention: if multiple calls for attention are needed, they simply keep resetting the same interrupt request. There is no need for extra hardware, but the interrupt program which responds to the request must be aware that there may be multiple events to be processed.

The core of at least some embodiments of the present invention is that the one-bit interrupt request (albeit carried by an MS1) is replaced with a register write. The address to which the write is performed is programmed as part of the device command. The data of the write is an arbitrary combination, defined by the needs of the system, of data pre-programmed in the same way as the address, and data generated by the device -usually some form of status. As a first instance, that will be writing the status to a predefined receiving register on the controlling CPU. Typically, the controlling software will allocate some bits programmed to identify which command has completed, and the device will insert other bits to indicate the status of the completed command.

This means that there is no longer a need for the CPU to perform status reads, which (travelling across chip) may take a while: the status is delivered to the CPU, which can handle it locally.

However, this gain comes at a cost. Status writes are no longer idempotent, unlike interrupt requests: they carry data, which must not be overwritten when Of it is possible, as it is in systems of the sort we consider) the device can send multiple statuses. Therefore, there is provided a storage mechanism, typically but not necessarily a fifo, to receive and hold multiple status writes, be it from one device or many. Such storage will have finite capacity. The controlling software should therefore, by design, condition the devices it controls such that the number of status values (usually the responses to commands) cannot exceed the available storage. As data is removed from the storage, freeing space, more commands (that will eventually generate status writes) can be issued. And the register write operation, whatever it may be, must be atomic. Multiple devices may try to write to the same fifo-like storage: their data must be strictly queued.

This provides a modest gain: status is delivered to the controlling processor instead of it having to go out, expensively in time, to fetch it. However, there is a symmetry: an operation is started by register writes, status is signalled by a register write. By appropriate arrangement of the address and data for the status write, the status write which terminates an operation on one device can become the command write which starts an operation on another device. Since the status write occurs immediately upon command completion, any data generated from that operation is presumably ready for processing. It therefore, there is a pipelined system, where multiple devices take as input a preceding devices output, this mechanism triggers the devices in series at the highest possible speed, while avoiding the possibility of race conditions which would occur if the devices were started together, or the delays if the processor has to respond to an interrupt from the first device before starting the second device, Multiple devices can be arbitrarily chained, with the last device sending its status back to the controlling CPU, reporting the result of the whole chain of operations. And, of course, there exists the possibility of fan-out: the devices could generate multiple register writes. These could be used to start multiple downstream devices, or to report the completion of a first phase to the controller (which can therefore release resources no longer needed) while almost simultaneously starting the second phase on another device.

A particular pattern that is frequently, but not necessarily, used in such systems is that devices are controlled by descriptor blocks in memory, and the command to start a device is the address and length of the descriptor block, which can fit into a single atomic register write. This means that, once the controlling CPU has written the descriptor block, the command to execute the descriptor is a single register write -exactly what we need here. The descriptor can then contained the address and pre-programmed data values for the status (or command to another device) to be sent on command completion. While not necessary, this provides a simple but elegant means of setting up cascaded commands.

Having gone this far with a peripheral oriented system, where a programmable CPU is controlling one or more relatively dumb processing devices, we note that this mechanism also provides for an improved interprocessor communications mechanism should our system have, as many SOCs nowadays do, several moderately programmable general purpose CPUs The same mechanism of a register write by one device which puts the register data into a fifolike memory in the receiver can be used to convey messages from one CPU to another -provided their content does not exceed the amount that can be written atomically, as described above.

As before, the total number of potential register writes must never exceed the capacity of the fifos to which they are written. In the first, device, case, this is handled by the controlling computer not issuing commands to the devices unless it is certain that it has space to receive all the status returns that may be generated as a result of those commands. Such a technique could be used with CPUs organised in a strict master/slave relationship. But the whole point of CPUs is that they are relatively intelligent, and may want to communicate asynchronously, with messages not generated in response to particular commands. The problem is that a sending CPU must know that there is space in the receiving CPU's fifo-like-memory before it sends a message.

This is achieved by all CPUs reserving space for each possible sender, and informing those senders of the current available space from time to time. Such information could be sent by reserving a number of bits in the messages sent to convey a "credit" to potential senders, or by sending special messages to carry the credit. The former mechanism resembles, but is different from, the TCP window in the TCP/IP transport protocol, and is our preferred embodiment.

If there are multiple CPUs and multiple devices, by fitting a device with a command fifolike memory as used for receiving the statuses, and probably by using the memory-based descriptors described above, a single device can be shared between multiple CPUs. The space in the command fifo is split, by allocation at software design time, between the various CPUs.

Each is allowed to submit as many descriptors (identified by address/length) as can fit in its fifo allocation. It is up to that issuer to ensure that the status address and partial data programmed into the descriptor route the status to an appropriate place.

If bought in IP (the USB Mac, in our case) has a "traditional" interrupt request, one can convert it to this scheme by including a unit which will generate a pre-programmed status write when the interrupt line is asserted. A virtue of this is that one does not to decide as SOC design time which CPU to wire the interrupt to -it is decided by software at run time. (This has not been covered in previous discussions, I think).

Thus a single design provides both sophisticated device control and inter-CPU communications. The cost, compared to prior art, is having wide fifos instead of single signal lines or single bit registers. Historically, that cost would have been daunting. In the context of today's SOC, with many millions of gates available, it is now small, and well worth it for the performance gains and the orthogonality of combining device and interprocessor communications into a single model.

Claims (13)

1. Claims 1. A heterogeneous multi-core integrated circuit comprising two or more processors, at least one of the processors being a general purpose CPU and at least one of the processors being a specialized hardware processing engine, the processors being connected by a processor local bus on the integrated circuit, wherein the general purpose CPU is configured to: generate a first instruction for an atomic operation to be performed by a second processor, different from the general purpose CPU, the first instruction comprising an address of the second processor and a first command indicating a first action to be executed by the second processor; and transmit the first instruction to the second processor over the processor local bus.
2. 2. A heterogeneous multi-core integrated circuit according to claim 1, wherein the first command specifies the first action that is to be performed.
3. 3. A heterogeneous multi-core integrated circuit according to claim 1, wherein the first command comprises at least one descriptor indicating what the first action comprises.
4. 4. A heterogeneous multi-core integrated circuit according to claim 1, wherein the first action is pre-programmed in the second processor and is invoked by the first command.
5. 5. A heterogeneous multi-core integrated circuit according to claim 1, wherein the first command comprises a pointer to at least one descriptor located in memory indicating what the first action comprises.
6. 6. A heterogeneous multi-core integrated circuit according to any preceding claim, wherein the first instruction comprises data on which the first action is to be performed, when executed.
7. 7. A heterogeneous multi-core integrated circuit according to any preceding claim, wherein the instruction comprises a pointer to data located in memory on which the first action is to be performed, when executed.
8. 8. A heterogeneous multi-core integrated circuit according to any preceding claim, wherein the general purpose CPU is configured so that the first instruction cannot be generated until any data required for execution of the first action is available.
9. 9 A heterogeneous multi-core integrated circuit according to any preceding claim, wherein the general purpose CPU is further configured to: receive, over the processor local bus, a second instruction from another processor for an atomic operation to be performed by the general purpose CPU, the second instruction comprising an address of the general purpose CPU and a second command indicating a second action to be executed by the general purpose CPU; and execute the second action.
10. 10. A heterogeneous multi-core integrated circuit according to claim 9, wherein the second action is the generation of the first instruction, wherein the second instruction includes the address of the second processor.
11. 11. A heterogeneous multi-core integrated circuit according to claim 10, wherein the second instruction includes the first command for inclusion in the first instruction.
12. 12. A heterogeneous multi-core integrated circuit according to any one of claims 9 to 11, further comprising an input buffer for storing the received second instruction until the second instruction is read by the general purpose CPU and the second action is executed.
13. 13. A heterogeneous multi-core integrated circuit according to claim 12, wherein the first instruction includes information indicating the amount of space available in the input buffer for storage of other instructions 14 A heterogeneous multi-core integrated circuit according to any preceding claim, wherein the processor local bus comprises any one of a Wishbone Bus, a CoreConnect bus or an Advanced Microcontroller Bus Architecture, AMBA, Bus, for example an Advance eXtensible Interface, AXI, Bus.15. A heterogeneous multi-core integrated circuit according to any preceding claim, wherein the second processor is a general purpose CPU.16. A heterogeneous multi-core integrated circuit according to any one of claims 1 to 14, wherein the second processor is a specialized hardware processing engine.17. A heterogeneous multi-core integrated circuit according to any preceding claim, wherein the specialized hardware processing engine is configured to process signals that are formatted in Universal Serial Bus, USB, protocol.18. A method of communication on a heterogeneous multi-core integrated circuit comprising two or more processors, at least one of the processors being a general purpose CPU and at least one of the processors being a specialized hardware processing engine, the processors being connected by a processor local bus on the integrated circuit, wherein the method comprises: generating, by general purpose CPU, a first instruction for an atomic operation to be performed by a second processor, different from the general purpose CPU, the first instruction comprising an address of the second processor and a first command indicating a first action to be executed by the second processor; and transmitting the first instruction to the second processor over the processor local bus.19. A method according to claim 18, wherein the first command specifies the first action that is to be performed.20. A method according to claim 18, wherein the first command comprises at least one descriptor indicating what the first action comprises.21. A method according to claim 18, wherein the first action is pre-programmed in the second processor and is invoiced by the first command.22. A method according to claim 18, wherein the first command comprises a pointer to at least one descriptor located in memory indicating what the first action comprises 23 A method according to any one of claims 18 to 22, wherein the first instruction comprises data on which the first action is to be performed, when executed 24 A method according to any one of claims 18 to 23, wherein the instruction comprises a pointer to data located in memory on which the first action is to be performed, when executed.25. A method according to any one of claims 18 to 24, wherein the general purpose CPU is configured so that the first instruction cannot be generated until any data required for execution of the first action is available.26. A method according to any one of claims 18 to 25, wherein the general purpose CPU is further configured to: receive, over the processor local bus, a second instruction from another processor for an atomic operation to be performed by the general purpose CPU, the second instruction comprising an address of the general purpose CPU and a second command indicating a second action to be executed by the general purpose CPU; and execute the second action.27. A method according to claim 26, wherein the second action is the generation of the first instruction, wherein the second instruction includes the address of the second processor.28 A method according to claim 27, wherein the second instruction includes the first command for inclusion in the first instruction.29. A method according to any one of claims 26 to 28, further comprising an input buffer for storing the received second instruction until the second instruction is read by the general purpose CPU and the second action is executed.30 A method according to claim 29, wherein the first instruction includes information indicating the amount of space available in the input buffer for storage of other instructions.31. A method according to any one of claims 18 to 30, wherein the processor local bus comprises any one of a Wishbone Bus, a CoreConnect bus or an Advanced Microcontroller 20 Bus Architecture, AMBA, Bus, for example an Advance eXtensible Interface, AXI, Bus.32. A method according to any one of claims 18 to 31, wherein the second processor is a general purpose CPU.33. A method according to any one of claims 18 to 32, wherein the second processor is a specialized hardware processing engine.34. A method according to any one of claims 18 to 33, wherein the specialized hardware processing engine is configured to process signals that are formatted in Universal Serial Bus, USB, protocol 35 A heterogeneous multi-core integrated circuit substantially as hereinbefore described with reference to the drawings.36 A method of communication on a heterogeneous multi-core integrated circuit substantially as hereinbefore described with reference to the drawings