GB2515573A - Data processing system and method - Google Patents

Abstract

A data processing system 100 comprises a first processor 116 having access to first configuration data; first configuration data (812 in Figure 8) comprising at least a first configuration program for at least one of configuring and controlling a reconfigurable processor 102; the first processor being adapted to configure the reconfigurable processor using the first configuration data and the reconfigurable processor being arranged to operate according to the first configuration data. A method of configuring a reconfigurable processor within a heterogeneous processing environment comprises: detecting connection of a sensor 104; determining associated identification data; retrieving associated configuration data; and configuring the reconfigurable processor accordingly. A sensor board for coupling to a sensor interface of a reconfigurable processor is also provided. The reconfigurable processor might be a FPGA, the sensor might be a camera CCD sensor, and the system could be used for image processing in a real-time dynamically adaptable environment using adaptive programming. Data may be output to more than one memory 106 using a round robin pattern.

Description

Intellectual Property Office Applicacion Nc,. (lB 1311652.7 RTM Dacc: 0 Dircinbcr 2013 The following terms are registered trade marks and should he rcad as such wherever they occur in this document: Canon Verilog Altera

JTAG

RapidlO WiFi I Wi-Fi Buetooth Texas Instruments PowerVR Imagination Technologies Arm I ARM Cortex

OMAP

Inlelleclual Property Office is an operaling name of the Pateni Office www.ipo.gov.uk

DATA PROCESSING SYSTEM AND METHOD

[0001] Embodiments of the invention relate to data processing systems and methods.

[0002] With any data processing system, the speed of data processing and the volumes of data that can be processed in a given unit of time are limited. The limitations stem at least in part from a combination of the software used to process the data, the architecture upon which the software is executed, the external interfaces to other devices or data processing entities etc. [0003] For example, a typical camera has a COD sensor with associated read-out logic, some processing, for example, jpeg processing, a memory channel and a memory for storing the jpeg data, or RAW data. Each element influences the frame size and the frame rate that can be realised. For example, a Canon EOSSD mark II has a frame rate of 3.9 frames per second, captured using a 21.1 megapixel full-frame CMOS sensor, with the processing being implemented using a DIGIC 4 image processor. While such an arrangement is acceptable, especially for stills or relatively slow moving objects, it is limited. Furthermore, while some cameras have facilities for configuring the post-capture processing, such as, processing RAW data to produce jpeg data, they are insufficiently flexible to achieve more ambitious objectives such as, for example, real-time, pixel level, dynamic range or exposure adjustment.

[0004] Other data processing involves capturing huge amounts of data and then processing the captured data off-line, that is, the data is captured, accumulated and then processed. Typically, the processing is performed by a different entity to the one that captured the data. Again, it there are technical issues that should be addressed during, or immediately after, any data capture such as, for example, image capture, or if there are real-time adjustments that could be made during data capture, they cannot be realised in real-time. The data capture would have to be performed again, which might not always be possible, especially in destructive or dangerous conditions, and especially so for one-time events.

[0005] Still further, present technology relating to data capture and distribution within vehicles relies typically on dedicated bus arrangements, such as, for example, Controller Area Network (CAN) bus and associated electronics such as the CAN transceiver and protocol controller that are all adapted to communicate with one or more Engine Control Units (ECU) to capture and process large amounts of data associated with sensors and actuators within a vehicle. Vehicle controls addressed by ECU5 comprise for example engine management, clutch management, gearbox control and management, differential control, track control systems, braking systems, and fuel system management amongst other things. Furthermore, the foregoing control systems must operate without fault at best, or, at worst, in a fault-tolerant manner with graceful degradation, if any, to avoid catastrophe failures. Still further, on-board diagnostics can be used to provide information about a vehicles condition and performance.

[0006] The above arrangements have limitations in respect of at least one of speed of processing, volume of data capture and processing, rate of data processing, real-time adaptation, inflexibility regarding disparate data capture and processing, inflexibility regarding types of adaptions and sensors to be accommodated, amongst others.

[0007] Accordingly, embodiments of the present invention relate to a system for processing captured data, the system being dynamically configurable and adaptable in real-time during data capture.

[0008] Advantageously, embodiments of the present invention provide a system capable of handling high data rates, high frame rates and/or high bit depths, in a real-time dynamically adaptable environment.

[0009] A still further advantage of embodiments of the present invention is that the embodiments appear to an external processor as fast memory at least insofar as concerns the sensor-subsystem, where all signal processing and sensor control is undertaken by an FPGA.

[0010] Embodiments of the invention are further described hereinafter, by way of example only, with reference to the accompanying drawings, in which: Figure 1 shows a data processing system according to an embodiment; Figure 2 illustrates part of the data processing system; Figure 2A shows schematically connections and data flows between entities of the data processing system: Figure 3 depicts a further aspect of the data processing system; Figure 4 shows a processing architecture according to an embodiment; Figure 5 illustrates a processing architecture according to an embodiment; Figure 6 shows a reconfigurable processor according to an embodiment; Figure 7 depicts a flowchart of communication and data exchange between entities of the system; Figure 8 is a flowchart of system initialisation together with initialisation or configuration data structures; Figure 9 illustrates the configuration data structures in greater detail; Figure 10 shows a data processing system according to another embodiment.

[0011] Referring to figure 1, there is shown an embodiment of a data processing system 100. The data processing system 100 comprises a central reconfigurable processor 102 for receiving data from at least one sensor 104. The central reconfigurable processor 102, such as, a field-programmable gate array (FPGA), is coupled to a memory 106. An FPGA is an integrated circuit or processor that can be configured after manufacture using an associated hardware description language such as, for example, VHDL or Verilog. Such a hardware description language is an embodiment of a configuration program for configuring and controlling such a reconfigurable processor and, in particular, for configuring the hardware of such a configurable processor. The HDL specifies the configuration of configurable logic blocks of such a reconfigurable processor. In the embodiment illustrated, two memory banks 106-1 and 106-2 are used to realise the memory 106. However, other embodiments can use a single memory or two or more memories as illustrated, that is, a plurality of memories. The memory is used for storing at least one of data received from the sensor(s) 104 and processing results following from operations performed by or coordinated by the central reconfigurable processor 102. The processing operations can be performed on or in relation to data that has been sensed by the sensor or data that is currently being sensed by the sensor.

[0012] Embodiments use memory having a 32 bit width to realise the memory 106. This arrangement provides some flexibility in terms of bandwidth according to the width of the data output by one or more than one sensor 104 coupled to the central reconfigurable processor 102. It will be appreciated that having at least two banks of memory effectively doubles the bandwidth of available memory, assuming that the data output by the sensor is 32 bits wide, or quadruples the memory bandwidth assuming that the data output by the sensor is 16 bits wide, and so on. In preferred embodiments, data is accumulated within the FPGA 102 until a sufficient number of bits have been collated to maximise use of the memory data bus. Therefore, if a 32 bit sensor is connected to the FPGA, a single 32 word from the sensor would be sufficient to fill the data bus of the memory and hence such a word would be written to memory. Alternatively, is the sensor output a 16 bit word, accumulated data is not written to memory until two such 16 bit words have been accumulated so that a single 32 bit write operation can be performed. Therefore, embodiments configure the FPGA 102 to have a memory bus width, or data accumulation register or other storage that has a width, corresponding to any connected memory 106.

The size of such a memory bus width or data accumulation register or other storage width is dynamically configurable according to the FAGA programming.

[0013] A benefit of having at least two independent memories such as memories 106-1 and 106-2 is that the memories can be written to using an alternating approach. Data received from a sensor can be written to one of the memories such as memory 106-1 during one sensor read, and data can be written to the other memory such as memory 106-2 during a subsequent sensor read. This has the advantage that processing data written to one memory such as memory 106-1 can commence as soon as it has been written without subsequent immediate memory contention needing to be resolved as a consequence of the next sensor read/write cycle. In essence, data stored in one memory can be processed concurrently with data being written to or stored in the other memory.

By alternating such use of the memories 106-1 and 106-2 each memory can have an associate write/process cycle that runs in anti-phase with the write/process cycle of the other memory.

[0014] In preferred embodiments, the central reconfigurable processor 102 is realised using an FPGA. Embodiments can be realised using, for example, an FPGA from the Altera Cyclone family, such as the Altera Cyclone IV FPGA. Embodiments can be realised using a single sensor or using more than one sensor. In the embodiments described herein reference will be made to a sensor 104 for brevity. However, one skilled in the art will understand that in all instances, unless the context compels otherwise, a reference to a single sensor will not preclude and also encompasses a plurality of sensors. Furthermore, such a plurality of sensors can all be identical, all different or any combination thereof [0015] The central reconfigurable processor 102 is also coupled to a further processing element 108. The further processing element can be realised in the form of a further FPGA, or in the form of further processing units of the central reconfigurable processor 102, especially when the central reconfigurable processor 102 is realised using an FPGA.

Therefore, embodiments can be realised in which the central reconfigurable processor 102 and the further processing element 108 are realised using a single processor, preferably in the form of a single FPGA. Such a single FPGA will have the same functionality as that provided by the two depicted FPGAs.

[0016] In operation, the sensor 104 produces sensor data that is received by the FPGA 102. The received data can be routed to the memory 106. The routed data can represent the raw data received from the sensor 104. Embodiments of use the term raw data to mean data that has not been processed, that is, data in its native form as output by the sensor 104. Embodiments can be realised however in which the sensor data is processed before it is stored in the memory 106. Any such processing can be realised by the central reconfigurable processor 102 or orchestrated by the central reconfigurable processor 102.

[0017] Embodiments can be realised in which the sensor data is conveyed to the central reconfigurable processor via a single data bus 110. The data bus 110 can be n bits wide, where n is greater than or equal to one. Embodiments can be realised in which the sensor data is conveyed to the central reconfigurable processor 102 via more than one data bus.

Again the data bus can be m bits wide, where m is greater than or equal to one. Such a plurality of data buses do not have to have the same bus width, although embodiments can be realised as such. Embodiments can be realised in which one or more of the data buses have different bus widths.

[0018] The central reconfigurable processor 102 is flexibly configurable. In particular, the central reconfigurable processor 102 can be configured to have at least one, and preferably, more than one memory channel for receiving data from the sensor via respective data buses 110-1 to 110-N and routing data to the memory 106, to another entity within the FPGA, or to a still further entity other than the FPGA or to one or more of the foregoing taken jointly and severally in any and all combinations. Embodiments of the memory channel(s) can be reconfigurable. Therefore, the central reconfigurable processor 102 can be configured to provide a one to one relationship between sensor data bus and memory channel. Alternatively, or additionally, embodiments can be realised in which there is a many to one relationship between the sensor data buses and a memory channel and visa-versa. In generally, there can be an n:m relationship between sensor data bus(es) and memory channel(s). Such an arrangement is depicted in figure 2, which shows a view 200 of the functional relationships between one or more than one data bus 110-1 to 110-N and corresponding memory channels 202-1 to 202-N. In the embodiment illustrated, the sensor 104 has m sensor data buses 110-1 to 110-N that are coupled to respective n memory channels 202-1 to 202-N. The memory channels 202-1 to 202-N, or one or more of them, are adapted to output received data to, for example, a memory 106 like one of the memories 106-1 and 106-N described above. Alternatively, or additionally, the memory channels 202-1 to 202-N can be configured to output data to selectable memories of the memory 106 and preferably to both of the memories 106-1 and 106-2. In a preferred embodiment, data can be output from a memory channel such as, for example, the first memory channel 202-1 to both memories 106-1 and 106-2, or a selectable number of a plurality of such memories. The data output to both memories can be the same data, which adds the safety of redundancy, or different data. A time division access technique can be used to such that a memory channel 202-1 to 202-N can route one or more units of data to one memory and then route one or more units of data to another memory. It will be appreciated that a memory controller or arbitrator will be used to control interactions with the memory 106.

[0019] Referring to figure 2A, there is shown a view 200A of the connections and data flows between entities of the system 100. The processors, that is, the DSP 112, the CPU 114 and the general purpose processor 116 have been shown for simplicity as a single entity 202A. The processors, or at least the general purpose processor 116, have access to storage 204A. Preferably, the storage is non-volatile and still more preferably the storage is programmable. A number of external interlaces are provided. Embodiments provide one or more of the following interfaces to the processors; a network interface 120, a display connector 206A, a USB console 208A and a USB connector 210A. An example of the display connector is an HDMI connector. However, embodiments are not limited to such a display port connector. Embodiments can be realised in which some other connector is used, such as, for example, at least one of an audio interlace and a video interlace.

[0020] A power supply 212A is provided to power the various entities of the system 100.

The power supply is arranged to produce at least one, and preferably a plurality, of different supply voltages. Current embodiments provide the following voltages 3.3V, 5V, 1.8V, 2.5V and 1.2V due to the components selected to realise the system 100. Preferably the power supply also actively manages the operations of the system 100 under the influence of, for example, the processor 116 or specific hardware, to dynamically manage the supply or otherwise of power to the different entities of the system 100 according to whether or those entities are scheduled to be used, and optionally according to the extent of their use. Communications with the FPCA are realised using a plurality of buses.

Embodiments provide a first bus, that is, a JTAG bus, that couples to a JTAG port 214A of the FPGA 102 to provide real-time programming of the FPGA 102. Programming of the FGPA 102 can optionally also be realised via a further interface. In an embodiment, the further interlace comprises a USB blaster interface 216A. Programming of FPGAs via a JTAG interlace is well known within the art. However, embodiments of the present invention use the JTAC interface to program and reprogram the FPCA 102 in real-time.

Such real-time, or adaptive programming, allows the functionality of the FPGA to be changed very quickly and to a flexible extent determined via at least one of prevailing hardware and task specific processing. Implementation of such dynamic changes is described in further detail with reference to figures 7 to 9. Although the present embodiment uses JTAG to reconfigure the FPGA 102, embodiments are not limited thereto. Embodiments can be realised in which the FPGA 102 is reconfigured using an alternative bus such as, for example, a PCIe bus, a Serial RapidlO, or a Flash bus or the like.

[0021] The FPGA 102 is arranged to appear to at least one of the processors 202A as memory by arranging for an interface between the FPGA 102 and the at least one processor operate as a memory interface from the processor's perspective. Preferred embodiments therefore realise a second bus between the at least one processor 202A and the FPGA 102 using a bus 218 in conjunction with a corresponding bus controller (not shown). For example, the bus could be a GPMC bus, a PCIe bus, an AXI bus or the like.

The bus could be a memory bus but embodiments are not limited thereto. Additionally, embodiments can be provided in which the bus 218 is a multi-channel bus. Configuring interactions between the processors 202A and the FPGA 102 to operate as memory exchanges between the processors 202A and a notional memory again provides a high speed and flexible way of configuring and/or exchanging at least one of data and code between the processors 202A and the FPGA 102.

[0022] In an embodiment, in anticipation of a sensor being an image sensor, a bus such as, for example, a specific camera bus 220A, is provided for sending image data from the FPGA 102 to at least one of the processors 202A. However, the bus 220A is not limited to being used as a camera or image sensor bus, that is, as a bus for carrying image data, the bus 220A can be used to carry any other data output by the FPGA 102.

[0023] Optionally, the FPGA 102 comprises a still further data interface 222A that provides for streaming data from the sensor(s) or/and memory 106 directly to a connected client at high data rates. The still further data interface 222A is preferably realised as a high-speed 1-lOG or faster serial link.

[0024] A connection is provided to allow at least one of the processors 202A, preferably, the general purpose processor 116, to communicate with, that is, at least write configuration parameters, the sensor(a) 104. The connection 224A is preferably realised using a data link, such as, for example, a serial data link. Such a serial data link can be realised using a Serial Peripheral Interface Bus, which is a synchronous serial data link.

One skilled in the art will appreciate that other buses such as, for example, an 12C bus could alternatively be used. The serial link 224A optionally also presents or is coupled to a connector 235. The connector 235 can receive another connector such, as for example, a connector associated with a peripheral 236A.

[0025] Optionally, an I/O processor 226A is provided to support further sensors such as, for example, a GPS sensor, a wireless connection 228A, such as, for example, WiFi, Bluetooth or an infra-red connection, or an Inertia Measurement Unit system 230A. The IMU 230A can comprise, for example, at least one or more of an accelerometer, preferably a 3D accelerometer, a gyroscope, a magnetometer, preferably a 3D magnetometer, and a pressure sensor, such as, for example, an air pressure sensor. The I/O processor 226A

B

can optionally comprise a bus 238A such as, for example, a 12C or SPI bus to which a compatible device can be coupled via a respective connector 239. Optionally, at least one or more of the processor can access an expansion connector 240. The expansion connector 240 can be used to couple to a further FPGA 102 and accommodate any one or more of the connections and buses associated with the above FPGA 102 taken in any and all combinations jointly and severally. The expansion connector 240 can take the form of one or more of an 12C or SPI bus (from the processor 116 and/or I/O processor 226A), GPIO lines from one of the processors 202A such as, for example, the processor 116, and one or more than one line from the FPGA 102 taken jointly and severally in any and all combinations. Furthermore, such an expansion bus or connector 240 would allow multiple systems such as system 100 described herein to be coupled to one another to allow cooperation between such systems.

[0026] Although the foregoing embodiments have been described with reference to the memory channels 202-1 to 202-N routing data to memories, embodiments are not limited thereto. Referring to figure 3, there is shown a view 300 of an embodiment that can be realised in which the memory channels 202-1 to 202-N are arrange to route data to other entities, either within or external to the central reconfigurable processor 102. as well as or instead as routing to the memories 106-1 and 106-2. It can be appreciated that one of the memory channels, that is, the first memory channel 202-1 is arranged to output data to another entity, which, in the illustrated embodiment is the FPGA 108, but could equally well be some other entity. The same memory channel 202-1 is also adapted to output data to a respective memory 106-1. A further memory channel 202-N is arranged to output data to a processing element 302 for processing according to a predetermined algorithm before storing the processed data in a respective memory 106-2, or outputting the processed data to another entity or to another memory.

[0027] Preferably, all memory channels described herein implement or use DMA to write data to and read data from a respective memory.

[0028] A preferred embodiment additionally comprises a bus interface 232A such as, for example, a peripheral interface such as a PCIe interface. It will be recalled that the bus 218 can be a PCIe interface, serial interface or some other type of interface. Preferably, a shared memory 234A is provided that is common accessible by the FPGA 102 and at least one or more of the processors 202A taken jointly and severally in any and all combinations. Preferred embodiments use a dual-port memory for at least one of substantially simultaneous reading from and writing to the memory 234A. Advantageously.

access to data by, for example, entities other than the FPGA 102 is speeded up using the shared memory because there is no need to transfer data between a memory associated with the FPGA 102 and a memory associated with or accessible by one or more entities other than the FCPA 102, such as, for example, one or more of the DSP 112, the CPU 114, the processor 116, or any other processor or device taken jointly and severally in any and all combinations. Therefore, for example, the FPGA 102 could write data into the memory 234A and provide an indication to one or more of the processors 202A like the processor 116 of the location of the written data.

[0029] Referring to figure 4, there is shown a view 400 of a further configuration of the central reconfigurable processor 102 having a plurality of such processing elements PE1 to PEN adapted to receive data from a memory channel 202. Although the memory channel 202 is shown as serving all processing elements PE1 to PEN, embodiments are not limited thereto. Embodiments can be realised in which the memory channel 202 is a plurality of memory channels, such as, for example, those described with reference to figure 2. In such an embodiment, a memory channel 202-1 to 202-N could serve a respective one or a respective plurality of or all of the processing elements PE1 to PEN. In a preferred embodiment, each processing element PEI to PEN is configured to process corresponding data from a sensor that produces an array or stream of data, for example, such as the above described CMOS sensor. Embodiments can be realised in which there is an m:n relationship between the pixels of such a sensor and the processing elements PE1 to PEN; which gives a 1:1 relationship, a many to one relationship, a one to many relationship or a many to many relationship. The relationships are governed by the values of m and n, which can be suited according to an intended application.

[0030] One skilled in the art will appreciate that the processing elements PE1 to PEN can be advantageously realised when the central reconfigurable processor 102 is an FPCA.

However, the processing capabilities of such FPCA processing elements PE1 to PEN might be insufficient to meet the needs of an application. Therefore, referring back to figure 1, embodiments can be realised in which additional processing capabilities are accessible by the central reconfigurable processor 102. It can be appreciated that the additional processing capabilities can comprise one or more of a digital signal processor 112, a graphics processing unit 114, a microprocessor 116, an application specific or general purpose custom designed processor or coprocessor, a video processor, an audio processor and the like taken jointly or severally in any and all combinations. The processing capabilities of the DSP 112, the CPU 114 and the MPU 116 are accessible via at least one of the above described memory channels 202, 202-1 to 202-N and the DSP 112, the GPU 114 and the MPU 116 represent embodiments of "another entity".

[0031] It will be appreciated that the additional processing units described constitute a heterogeneous processing environment.

[0032] Referring to figure 5, there is shown a view 500 of the processing elements PE1 1 to PEMN arranged to communicate not only with the memory channel 202 but also with neighbouring processing elements. In the embodiment shown, a neighbouring processing element is depicted as being an immediately adjacent processing element, or a selectable one or more of immediately adjacent processing elements. However, embodiments can be realised using greater connectivity, in which one or more processing elements beyond the immediately adjacent processing elements are connected to a given processing element, in addition to or instead of being connected to one or more immediately adjacent processing elements. Also, a simpler connectivity is possible when several processing elements are arranged to create a pipeline-like architecture with a single input and single output, as well as, in addition to or concurrently with, the above network connectivity described above.

[0033] Embodiments can be realised in which the system 100 of figure 1 also comprises at least one of additional memory 118 and a communications adapter 120. The memory 118 can be volatile or non-volatile memory or a combination of volatile and non-volatile memory. The additional memory can be used to store sensor data, both at least one of as natively captured and as processed following capture but pre-storage, and other processing results, such as, for example, the results of processing operations performed by one or more processing elements PE1 to PEN, one or more additional processing elements or data received by or to be transmitted by the communications adapter. An embodiment of a communications adapter 120 would be, for example, a 1-10 gigabit network adapter, although other, higher speed, network adapters could also be used such as, for example, a 10/1 0011 000 Mbps adaptor. One skilled in the art will appreciate that the data unit of captured or read-out data unit could be at least one or more of at least a bit, at least a byte, at least a word, at least a line or other unit of any of the foregoing, or some other data unit such as, for example, in the case where the sensor is an image array groups of pixels. The groups of pixels could be vertically arranged, horizontally arranged, one or more sub-regions of the image array, a complete or partial row of pixels, a complete or partial column of pixels.

[0034] Embodiments can be made in which the above central reconfigurable processor 102 is realised using a member of the Altera Cyclone FPGAs such as, for example, an Altera Cyclone IV FPGA, the above digital signal processor 112 can be realised using a 64-bit Da Vinci DSP available from Texas Instruments, the above graphics processing unit 114 is realised using a PowerVR graphics processor, such as, for example, a Power VR SGX53O processor, available from Imagination Technologies and the above microprocessor 116 can be realised using, for example, a Arm single-core Cortex-A8 and dual core Cortex-AU families of processor. Although embodiments are not limited thereto.

Embodiments can be realised in which such additional processing units are realised using other devices from other manufacturers. Additional processing units can comprise application specific processor in addition to or as alternatives to any of the DSP 112, CPU 114 and processor 116.

[0035] In use, the processor 116 is arranged to program the central processor 102 using a program derived from a suitable programming language such as, for example, VHDL.

Binary programs derived from one or more VHDL programs are typically stored within the additional memory 118 and/or are accessible to the processor 116 to be loaded into the FPGA as described below with reference to figures 8 and 9.

[0036] Referring to figure 6, there is shown a view 600 of an embodiment of the flexibly configurable processor 102, which is preferably realised in the form of the above mentioned FPGA. The processor 102 is configured to have a host controller 602 that is arranged to interact with a corresponding memory channel 1054 (described with reference to figure 10 below) such that the FPGA 102 appears to be a conventional memory device via a bus 218 and associated controllers. The bus 218 could be, for example, a CPMC bus, a PCI bus, a PCIe bus, or some other type of bus, including being a multi-channel bus. The host controller 602 interacts with a memory mapped host bus 603. The memory mapped bus 603 maps 10 ports or registers of entities within the FPGA 102 such that they, or more particularly their registers, and/or memory are addressable in the same way that memory within a memory map is addressable. The GMPC bus can be either synchronous or asynchronous.

[0037] Preferred embodiments use a unified memory-mapped host bus 603. Such a bus can be realised using, for example, an Avalon MM master and Slave arrangement, available from Altera. Such a memory mapped host bus 603 supports flexible data routing between different module using the reconfigurable DMA channels 604 described below under the control of the host controller, which is arranged to specify exact interconnect configurations of the modules inside the FPGA 102 at any instant in time. Having such a memory mapped host bus greatly simplifies the frame store module 608 described below because the frame store module 608 does not need to perform any data flow coordination or switching functions. Although embodiments have been described with reference to using reconfigurable DMA channels 604, embodiments are not limited thereto.

Embodiments can be realised in which some other type of data routing or data switching is used such as, for example, a packet-switching interconnect like Altera's Qsys network on chip interconnect.

[0038] In the embodiment shown in figure 6, the memory mapped bus 603 is arranged to have within its memory map a sensor controller 606 (for interacting with at least one of an attached sensor 607 and one or more other modules such as, for example, a frame store 608), a data store or frame store 608 (for storing into memory 618 or redirecting to other modules native or raw data received from the sensor 607 and/or processed data associated with the sensor 607), a profiler 610 (for performing an analysis of the data such as, for example, calculating profiles, histograms, centre(s) of gravity of data etc.), a signal processing module 612 (for performing signal processing operations on data from at least one of the frame store 608 and memory 106/618) and a dynamic range image reconstruction module 614 (for influencing the dynamic range of the values constituting the data acquired from the sensor 607, and/or stored in memory 618. The memory mapped bus 603 is dynamically configurable to include any configuration of modules such as, for example, modules at least one or more of modules 606, 608, 610, 612, 614 or any other module(s) taken jointly and severally in any and all combinations.

[0039] The host controller 602 manages interactions of the other entities within the FPGA 102 with the remainder of the system via one or more DMA memory channels 604, in preference to programmed 10 or interrupt driven 10. Embodiments use a single DMA channel or, preferably, multiple DMA channels 604-1 to 604-N, where N > 1. Preferred embodiments use four DMA channels.

[0040] A DMA access multiplexor 616 is used to connect one or more than one of the modules 606 to 614 or any other module to one or more DMA channels 604 to support access to data stored in an external memory 618, such as, for example, memory 106, via those memory channel(s) or memory controllers 620 as described above, including with reference to figures 3 to 5. The memory controller 620 has bus arbitration. Embodiments use round robin scheduling for multi-access to the memory 618.

[0041] The sensor control 606 is coupled to a respective sensor 607. The sensor 607 can be changed and therefore the interface between the sensor control 606 and any given sensor 607 is sensor specific. The interlace between the sensor control and the memory-mapped host bus 603 is generic.

[0042] The FPGA firmware implements low-level sensor control and also performs most of the data processing tasks in relation to data associated with the sensor or sensors, which ensures that a significant reduction in the data flow between the FPGA 102 and the general purpose processor 116 is realised and, in turn, between the system 100 and a client computer 1012 (described with reference to figure 10) [0043] The central data acquisition module is the frame store 608, which is arranged to receive data from the sensor control module 606 and saves the data to memory, preferably memory buffers, that are accessed by one of the DMA channels. Therefore, it will be appreciated that the frame store 608 is a species of the genus acquisition control module; the latter controlling receiving and storing of sensor data. It will be appreciated that the FPGA 102 is connected to memory 106, which cannot be accessed by, or at least is not shared with, the general purpose processor 116. The frame store 608 can perform data processing tasks on a data unit by data unit basis such as, for example, on a pixel-by-pixel basis. Although the temi pixel-by-pixel basis is used to describe the units of data processing, embodiments are not limited thereto. Embodiments can be realised in which other units of data can be processed. Reference, however, will be made to pixel-by-pixel basis for convenience without intending to exclude processing on some other data unit-by-data unit basis. For example, the frame store 608 can perform pixel-by-pixel operation such as, for example, pixel-by-pixel arithmetic eg summation, image correction such as, for example, dark image correction having previously stored and retained access to an initial dark image. It will be appreciated that dark image correction is a species of the genus of processing a current image with reference to a previous image and/or visa versa. The frame store 608 can also direct data, received by the frame store or processed by the frame store 608, to at least one of the other FPGA 102 entities such as, for example, at least one or more of the profiler 610, the high dynamic range reconstructor 614 or other FPGA entities. Additionally, the frame store 608 can output data from the FPGA 102 via the host controller 602 and its associated bus 218.

[0044] Within the FPGA 102. operations are controlled by the host controller 602 in response to commands received from the FPGA driver 104 via the GPMC 1054.

[0045] Referring to figure 7, there is shown a view 700 of an embodiment of a sensor interface. The sensor interface comprises a connector 701 for accommodating a number of general purpose, bidirectional FPGA input/output lines 702. Also provided is a communication channel 704. Preferably, the communication channel 704 is a serial communication channel. However, embodiments are not limited thereto. Embodiments can be realised in which a number of serial lines are used, or in which a parallel communication channel is realised via several communication lines. The connector 701 is arranged to receive a complementary sensor board connector 706 that is carried by a respective sensor board 708. The sensor board 708 carries a sensor board communication channel 710 that is coupled to the above communication channel 704.

The communication channels 704 and 710 are used by the processor 116 to determine the type of sensor carried by the sensor board. The determination is based on sensor data 712 being provided from the sensor board to the processor 116, preferably via a bus such as, for example, the SPI/12C bus 224A. An embodiment uses programmable storage 714 to store the sensor data 712. The sensor data comprises sensor identification data 716 that can be read by the processor 116 and used to configure the FPGA 102 according to the nature of the data anticipated as being received over the parallel 10 lines 702/110 and anticipated subsequent processing, if any, prior to it being stored or output from the FPGA 102 for further processing. Embodiments can be realised in which the sensor data 712 additionally or alternatively comprises parameter data 718 for influencing the operation of the sensor 607. For example, the parameter data 718 might include at least one of sampling speeds, frame rate, read out rates, sensor readout speed, frame rate, integration time, sensor modes, region of interest, pixel or other data unit binning, sensor gain, sensor offset, high dynamic range mode select, dual or multiple amplifier control/read out taken jointly and severally in any and all permutations. Also shown in figure 7 is a view 750 of a protocol governing the exchanges that take place between a sensor board and the processor 116 and FPGA 102 during initialisation and operation. Upon connecting a sensor board's connector 706 to the interface connector 701, the processor 116 detects that coupling at step 752. The processor 116 requests sensor data 714 from the sensor board 708 at step 754 via the communication channels 704 and 710. The sensor data 714 is provided to the processor 116 at step 756, which, in response to the sensor data 714, requests from memory 118, FPGA configuration data for configuring the FPGA 102 according to the type of sensor detected. The FPGA configuration data is sent from memory 118 to the processor 116, at step 762, which then oversees configuration of the FPGA 102 using that data at step 764. The dynamic configuration of the FPGA 102 in response to a detected sensor or detected sensors is described below with reference to figures 8 and 9. Once the FPGA 102 has been configured in response to the sensor data 714, the FPGA 102 operates to control the sensor 607 at step 766 and to receive data therefrom at step 768. The received data is written via DMA channels 604 to memory at step 770. The foregoing is undertaken for all sensors connected to the system 100. One or more than one sensor can be connected to the system 100 at any onetime. The ERGA 102 would be configured to accommodate such one or more than one sensor.

[0046] The sensor board 708 can accommodate any one or more of a plurality of sensors such as, for example, diode arrays, CMOS sensors, CCD sensors, accelerometers, position sensors, angular displacement sensors, chemical sensors, speed sensors, fuel level sensors, body configuration and attitude sensors etc. [0047] Referring in greater detail to the configuration of the FPGA, figure 8 shows a flowchart 800 of system configuration. A power-up or reset occurs at step 802. The reset is applicable to the system as a whole and particularly involves re-booting or powering up general purpose processor 116. which acts as an overall orchestrator of operations and configuration. A boot strap operating system is loaded in the processor 116 at step 804.

System interrogation by the processor 116 is performed at step 806 to identify the present hardware configuration, in particular to determine the type of sensor(s) 104 connected to the system, that is, the sensor(s) interfaced with the FPGA 102. As indicated above with regard to figure 7, each sensor board contains data that is provided to the processor 116 that retrieves corresponding configuration data for at least the FPGA 102 at step 808. The configuration data is retrieved from a configuration data structure 810 stored in memory 118 or otherwise accessible by the processor 116. The configuration data structure 810 contains one or more instances of configuration data 812 to 816. In the present example, N instances of configuration data are provided. Each instance of configuration data 812 to 816 corresponds to a configuration of at least the FPGA in response to the detected attached sensors. The appropriate instance of configuration data is indexed using the sensor data 712 via respective indices 818 to 822. Having received the appropriate configuration data at step 808, the FPGA is programmed at step 824 by the processor, and system, or at least FPGA, configuration ends at step 826. Although N instances of configuration data are shown, other numbers of configuration data could be used such as one or more than one instance of configuration data. Configuration data comprises at least one or more of code for the FPGA, code for one or more of the processors, drivers for one or more elements of the system, and code for the DSP taken jointly and severally in any and all permutations.

[0048] Referring to figure 9, there is shown a view 900 of the configuration data structures 810 in further detail. The configuration data is organised in a directory structure having a root directory 902 and respective subdirectories each corresponding to a particular system configuration and each containing respective configuration data. It can be appreciated that a first index 818 into the first subdirectory contains a number of further subdirectories 904 to 910. A first further subdirectory 904 contains configuration code for the FPGA. A second further subdirectory 906 contains drivers for one or more hardware elements of the system such as, for example, 12C, 10 processor, image sensor driver and one or more drivers for respective modules within a configured FPGA taken jointly and severally in any and all combinations.. A third further subdirectory 908 contains algorithms for execution by one or more, taken jointly and severally in any and all combinations, of the FPGA 102, the DSP 112, the general purpose processor 116, the graphics processor 114 and the I/O processor 226A or any other processing module. The algorithms are stored within respective still further subdirectories 912 to 918.

[0049] Referring to figure 10, there is shown an overview of a software architecture 1000 that uses the above data processing system. The software architecture 1000 comprises three parts; namely, a main application server 1002, which runs on the processor 116, FPGA firmware 1004, which configures or runs on the FPGA 102, and a network client 1006. which is run on a computer connected to the data processing system 100 via a network 1008.

[0050] On start-up, the server application 1002 is arranged to configure itself and the FPGA on the basis of the content of a configuration data or configuration file 1010, described above with reference to figures 8 and 9. Once the main server application 1002 has finished configuring itself and the FPGA 102, it enters an idle mode. In the idle mode, the processor 116 awaits command requests from the network client 1006 running on an associated computer 1012. However, examples can be realised in which the main server application commences operations without awaiting such commands.

[0051] The network client 1006 relies upon an API 1014 that implements a protocol for issuing commands to the main server application 1002. Exchanges with the network client 1006 are handled by a command manager 1016, which is realised in software that can be executed by at least one of the processors, preferably the general purpose processor 116.

The command manager 1016 is arranged to implement network communications with the network client 1006 and parse and give effect to received commands. Effect is given to received commands in accordance with a predetermined, or accessible, command list 1018. Each command is a sequence of functions that are implemented by a software implemented utilities library 1020. The utilities library 1020 comprises a plurality of utilities 1022 to 1034. An advantage of providing a utility library is that it supports changing the functionality of the system 100 without having to undertake at least one of re-compilation and re-configuration of at least the FPGA 102, as well as, or instead of, other entities of the system that can be re-compiled or re-configured. A sensor configuration utility 1022 is responsible for configuring a connected sensor to allow an associated sensor to operate as desired. Embodiments of sensor configuration settings comprises at least one or more of analogue signal settings (such as gain, offset, signal conditioning DACs), operating modes (such as rolling image capture and snap-shot shutter, high dynamic range), ROl, integration time etc. A system configuration utility 1024 is provided to orchestrate overall configuration of the system 100. An analogue-to-digital input-output utility 1026 is provided to effect A-to-D conversion. A signal processing utility 1028 is provided that contains a set of signal processing functions that can be executed on processor 116 or delegated to at least one or more of the FPGA 102, the DSP 112 or the GPU 114 taken jointly and severally in any and all combinations. A mathematic library 1030 is provided to provide access to predictable or stable mathematical operations that can be performed in relation to presented data. An image acquisition utility 1032 that assists in image acquisition by configuring, for example, the at least one or more of the frame store 608 and FPGA modules such as, for example, at least one or more of the processing elements PE1 to PEN and the memory channels. One or more than one sensor utility 1034 is provided for orchestrating interacting with attached sensors such as the IMU sensors. For example, an accelerometer utility is illustrated.

[0052] The command list 1018 can be dynamic, that is, it is reconfigurable. The command list 1018 is created by a configuration manager 1056. The configuration manager 1056 is responsible for overall configuration of the system 100. Embodiments of the configuration manager 1056 are arranged at least to process commands received from the command manager 1016, to retrieve configuration data from the configuration file 1010 and give effect to that configuration data, to read hardware information from the storage associated with a or more than one, sensor, loading drivers and configuring the FPGA 102.

Embodiments can be realised in which the commands received from the command manager 1016 are configuration-related commands.

[0053] Many of the hardware elements of the system 100 require associated drivers.

Therefore, the software architecture 1000 also supports driver initialisation of a driver layer 1036 within that architecture 1000. Embodiments provide at least one or more of the following drivers. Peripheral drivers 1038 are provided to support interactions with respective peripherals. Embodiments are provided in which the peripheral drivers comprise at least one or more of a SRI bus driver, an 12C driver, a GRIO driver, an input/output processor driver, an EEPROM driver, an accelerometer driver, a DAC driver, a ADC driver, respective IMU drivers. In embodiments that are arranged to provide image processing, a frame store driver 1040 is provided for controlling image acquisition and image processing, as well as an image sensor driver 1042 for providing appropriate settings for controlling the initialisation and operation of a respective image sensor.

Finally, an FPGA driver 1044 is provided for initialising and controlling at least one of the configuration and operation of the FPGA 102. The driver 1044 also provides functions to provide a communication path to the FPGA 102 such as, for example, functions that support writing to a specific address of the memory map.

[0054] The above described software architecture 1002 executes within the above described hardware environment. Figure 1000 also further schematically depicts that hardware environment 1046. The hardware comprises the general purpose processor 116, a system DMA module 1048, an SDRAM controller 1050, one or more than one peripheral interface 1052, a general purpose memory controller 1054 and optionally a camera parallel or serial interlace or bus 220A. The system DMA module 104 is realised, for example, using a Texas Instruments OMAP 35x processor, which provides 32 logical channels. The peripheral interface 1052 depends upon the peripherals to be used with the system 100 and, as indicated above in figure 2A, it is connected to the I/O processor 226A.

Optionally, the peripheral interface 1052 can interface directly with peripherals. The peripheral interface 1052 also provides the above described JTAG interface(s). The general purpose memory controller 1054 is used to realise the above-described GPMC bus 218.

[0055] The configuration of the ERGA 102, or 1004. has been described above in detail with respect to figure 6.

[0056] Embodiments find particular application within the context of image processing.

Numerous image processing techniques can be implemented on the system 100 such as, for example, the image processing techniques disclosed in "High dynamic range colour imaging using complementary metal-oxide semiconductor (CMOS) sensors with non-destructive readout", by Anton Kachatkou and Roelof van Silfhout, lOP Publishing, Meas.

Sci. Technol. 20 (2009) 104010 (7pp), which is incorporated herein by reference for all purposes and included as an appendix of this application.

All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and/or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and/or steps are mutually exclusive. The embodiments are not restricted to the details of any foregoing embodiments. The embodiments extend to any novel one, or any novel combination, of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one, or any novel combination, of the steps of any method or process so disclosed.

Claims (38)

1. CLAIMS1. A data processing system (100) comprising a first processor (116) having access to first configuration data; first configuration data (812) comprising at least a first configuration program for at least one of configuring and controlling a reconfigurable processor (102); the first processor being adapted to configure the reconfigurable processor using the first configuration data and the reconfigurable processor being arranged to operate according to the first configuration data (812).
2. 2. A data processing system (100) as claimed in claim 1, comprising second configuration data (814) comprising at least a second configuration program for configuring and controlling the reconfigurable processor (102); the first processor (116) being adapted to re-configure the reconfigurable processor (102) from a first configuration state corresponding to the first configuration data (812) to a second configuration state corresponding to the second configuration data (814).
3. 3. A data processing system (100) as claimed any preceding claim, wherein at least one of the first configuration data (812) and the second configuration data (814) comprises at least one executable program for the first processor.
4. 4. A data processing system (100) as claimed in any preceding claim, further comprising at least one of a second processor (112) and a third processor (114); at least one of the second (112) and third processor (114) being different to the first processor (116) and wherein at least one of the first and second configuration data comprises at least one of second processor configuration data (908) and third processor configuration data (918).
5. 5. A data processing system (100) as claimed in any preceding claim, wherein any of the configuration data comprises at least one algorithm for execution by a respective one of the processors and the reconfigurable processor (112).
6. 6. A data processing system (100) as claimed in any preceding claim, wherein the reconfigurable processor (102) is adapted to receive data from one or more than one sensor.
7. 7. A data processing system (100) as claimed in any preceding claim, further comprising one or more than one memory (106) arranged to store data output from the reconfigurable processor (102).
8. 8. A data processing system as claimed in claim (100), wherein the reconfigurable processor (102) is adapted to process the received data prior to outputting the processed data to the one or more than one memory.
9. 9. A data processing system as claimed in either of claims 7 and 8, wherein the reconfigurable processor (102) is adapted to output data to selectable ones of the one or more than one memory (106) according to a predetermined pattern.
10. 10. A data processing system (100) as claimed in claim 9, wherein the predetermined pattern is a round robin pattern
11. 11. A data processing system (100) as claimed in any preceding claim, wherein the first configuration data (812) is arranged to configure the reconfigurable processor (102) to have a host controller (602); the host controller (602) being arranged to provide a memory interconnect such that at least the first processor (116) communicates or interacts with the reconfigurable processor (102) as if communicating with or interacting with a memory.
12. 12. A data processing system (100) as claimed in any preceding claim, further comprising a memory mapped host bus (603) for hosting, in a memory mapped manner, at least one or more than one operable entity (604..620).
13. 13. A data processing system (100) as claimed in claim 12, wherein the at least one or more than one operable entity (604.620) comprises a sensor control (606) for controlling communication with at least one sensor (607).
14. 14. A data processing system (100) as claimed in claim 13, wherein the at least one or more than one operable entity (604.620) comprises a frame store (608) for storing data at least received from or associated with data the at least one sensor (607).
15. 15. A data processing system (100) as claimed in claim any of claims 12 to 14, wherein the at least one or more than one operable entity (604.620) comprises a profiler (610) for performing an analysis of data such as at least one of calculating a profile, a histogram and a centre of gravity of the data.
16. 16. A data processing system (100) as claimed in any of claims 12 to 15, wherein the at least one or more than one operable entity (604.620) comprises at least a signal processing module (612) adapted to perform signal processing operations on data presented to or accessible to the at least one signal processing module (612).
17. 17. A data processing system (100) as claimed in any of claims 12th 16, wherein the at least one or more than one operable entity (604..620) comprises a dynamic range processor (614) adapted to process data presented to or accessible to the dynamic range processor according to a respective dynamic range of said data.
18. 18. A data processing system (100) as claimed in claim 17. wherein the respective dynamic range is at least one of prescribed or adaptable.
19. 19. A data processing system (100) as claimed in any of claims 12 to 18, wherein the one or more than one operable entity (604.620) comprises one or more than one memory access channel (604) for outputting data to a memory (618) via a memory manager (616 and 620).
20. 20. A data processing system (100) as claimed in claim 19, wherein the memory manager comprises at least one of a memory access multiplexor (616) and a memory controller (620).
21. 21. A data processing system (100) as claimed in any preceding claim, further comprising an input-output interface (226A) adapted to cooperate with one or more than one external entity (228A, 230A).
22. 22. A data processing system (100) as claimed in claim 21, wherein the one or more than one external entity (228A, 230A) comprises at least one of a positioning system, a communication system, an inertial measurement unit, a magnetometer.
23. 23. A data processing system (100) as claimed in any preceding claim, wherein the reconfigurable processor (102) is adapted to have at least one network controller (222A) for supporting communications over a network.
24. 24. A data processing system (100) as claimed in any preceding claim, wherein the reconfigurable processor is arranged to have a further programming interface (216A) for receiving the first configuration data (812) or other configuration data (814).
25. 25. A data processing system (100) as claimed in any preceding claim, further comprising at least one sensor interface via which at least first sensor data associated with at least a first sensor is received; the at least first sensor data being associated with the first configuration data.
26. 26. A data processing system (100) as claimed in claim 25, comprising a plurality of configuration data; each associated with a respective sensor and being selectable by the first processor to configure the reconfigurable processor in response to connection of a sensor to a sensor interface.
27. 27. A sensor interface for communicating with a sensor (607); the sensor comprising storage for storing identification data associated with the sensor (607); the identification data associated with the sensor having respective configuration data for adapting the configuration of a reconfigurable processor (102) according to the sensor or the reconfigurable processor (102) of a system (100) as claimed in any preceding claim; the sensor interface being adapted to read the identification data.
28. 28. A method of configuring a reconfigurable processor (102) within a heterogeneous processing environment; the method comprising the steps of a. Detecting connection of a sensor (607) to the reconfigurable processor (102); b. Determining identification data (714) associated with the sensor (607); c. Retrieving configuration data (812) associated with the identification data (714) of the sensor (607); the configuration data comprising at least a first configuration program for at least one of configuring and controlling a reconfigurable processor (102); d. Configuring the reconfigurable processor (102) according to the first configuration data (812).
29. 29. A method as claimed in claim 28, further comprising operating the reconfigurable processor (102) configured according to the first configuration data (812) to obtain data from the sensor (607).
30. 30. A method as claimed in claim 29, further comprising at least one of processing the obtained data and outputting the obtained data or outputting the processed obtained data to a selectable memory (106).
31. 31. A method as claimed in any of claims 28 to 30 wherein said retrieving configuration data (812) associated with the identification data (714) of the sensor (607); the configuration data comprising at least a first program for data processing.
32. 32. A sensor board for coupling to a sensor interface (606 and 701) of a reconfigurable processor (102); the sensor board comprising at least a sensor (604) adapted to output data via respective 10 to the sensor interface; the sensor board further comprising storage (714) for storing identification data (712) associated with the sensor (607); the identification data (712) having associated configuration data adapted to configured the reconfigurable processor (102) for operation with the sensor (607).
33. 33. A sensor board as claimed in claim 32, wherein the storage (714) comprises parameter data (718) associated with operating parameters of the sensor (607).
34. 34. A heterogeneous processing environment comprising a plurality of heterogeneous processors and a data processing system as claimed in any of claims 1 to 27.
35. 35. A data processing system substantially as described herein with reference to and/or as illustrated in the accompany drawings.
36. 36. A method substantially as described herein with reference to and/or as illustrated in the accompany drawings.
37. 37. A computer program substantially as described herein with reference to and/or as illustrated in the accompany drawings.
38. 38. Machine readable storage substantially as described herein with reference to and/or as illustrated in the accompany drawings.