CN115061975B

CN115061975B - Firmware platform, network on chip and neuromorphic chip based on FPGA

Info

Publication number: CN115061975B
Application number: CN202210935285.2A
Authority: CN
Inventors: 卡斯滕·尼尔森; 武晨曦; 罗伯托·卡塔内奥; 张�林; 薛凯; 柯政; 任宇迪
Original assignee: Chengdu Shizhi Technology Co ltd; Shenzhen Shizhi Technology Co ltd
Current assignee: Chengdu Shizhi Technology Co ltd; Shenzhen Shizhi Technology Co ltd
Priority date: 2022-08-05
Filing date: 2022-08-05
Publication date: 2023-01-10
Anticipated expiration: 2042-08-05
Also published as: CN115061975A

Abstract

The invention discloses a firmware platform, an on-chip network and a neuromorphic chip based on an FPGA (field programmable gate array). In order to solve the technical problem that the existing firmware platform is not suitable for event-driven neuromorphic hardware, the invention designs an FPGA-based firmware platform which is used for communicating the neuromorphic hardware with other components; the firmware platform is capable of transmitting event type data including an event generated address and an event generated timestamp. The unified, modularized and flexible firmware platform can perform event-based communication, has universality and flexibility, and reduces the development cost of chips. The invention is suitable for the field of neuromorphic chips.

Description

Firmware platform, network on chip and neuromorphic chip based on FPGA

Technical Field

The invention relates to a firmware platform, a network on chip and a neuromorphic chip based on an FPGA (field programmable gate array), in particular to the firmware platform, the network on chip and the neuromorphic chip based on the FPGA, which can transmit event type data.

Background

The neuromorphic chip (also called a brain-like chip) is a highly specialized chip, has event-based characteristics, and is suitable for the fields of edge, internet of things and the like. The data set is composed of a spatio-temporal event stream, which is a time sequence composed of discrete/sparse events, and the generation of any event is independent, asynchronous, and does not follow any fixed clock frequency.

Although neuromorphic chips are proprietary, they typically share communication protocols and some basic requirements, and to ensure the rapid implementation of new product systems, a uniform, modular, flexible firmware platform is required and customized to retain the unique advantages of a neuromorphic chip.

However, existing firmware systems are universally von neumann architectures that are synchronously clocked, which are not suitable for handling asynchronous and sparse event streams. If the event is read and processed by adopting the clock with fixed frequency, the delay is larger when the clock frequency is lower, and the power consumption is larger when the clock frequency is higher. Meanwhile, if a plurality of devices generate event streams simultaneously (parallelism), the von neumann architecture is adopted for processing, so that the communication bandwidth is greatly reduced, and the delay is increased.

Network on chip (SoC) is widely used in SoC, and recently, it is gradually used in FPGA. Some documents show that routers in a network on chip can adopt an asynchronous circuit design, but the design idea and how to adapt to an event-driven neuromorphic chip are not given.

Therefore, there is a need for a firmware platform and network on chip suitable for neuromorphic chips, capable of event-based communication, and having universality and flexibility.

Disclosure of Invention

In order to solve or alleviate some or all of the technical problems, the invention is realized by the following technical scheme:

an FPGA-based firmware platform for neuromorphic hardware to communicate with other components; the firmware platform is capable of transmitting event type data including an event generated address and an event generated timestamp.

In some embodiments, the firmware platform comprises: at least one network module coupled with neuromorphic hardware or other component; the network on chip is an interconnection system coupled with the network modules and used for communication among the network modules.

In some embodiments, the network on chip comprises at least one node, each node corresponding to a network module; each node comprises a network interface and a router; the network interface is used for exchanging data with the corresponding network module and the router; the router is used for routing data among routing networks.

In some embodiments, the network on chip supports control flow or/and data flow; the data stream includes events or/and any other type of data.

In some embodiments, the routing network is based on deadlock free wormhole routing algorithms to maintain order between events.

In certain embodiments, the routing network communicates data in the form of data packets; the router includes: an input selection for determining valid input ports; a routing control for selecting, based on a packet received from an active input port, all output ports to which the packet may be sent; and the output selection is used for selecting one target port from all the output ports so as to output the data packet.

In some embodiments, the network interface comprises a transmit path and a receive path; the sending path is used for converting the data format of the network module into a data packet format supported by a routing network; the receiving path is used for converting the data packet format supported by the routing network into the data format of the corresponding network module.

In some embodiments, the network module has at least one data path to interact with the network on chip, or/and the data packets include event packets or/and configuration packets or/and any other type of data packets.

In some embodiments, the transmission path includes a transmission module and a transmission assistance module, the transmission assistance module being configured to pre-process input data from the network module; the sending module is coupled between the sending auxiliary module and the router, creates a data packet based on the input data or the preprocessed data, and sends the created data packet to the corresponding router; the receiving path comprises a receiving module and a receiving auxiliary module, and the receiving module analyzes the data packet from the router to obtain analyzed data; the receiving auxiliary module is coupled between the receiving module and the corresponding network module and used for processing the analyzed data; and based on the type of the analyzed data, sending the analyzed data or the data processed by the receiving auxiliary module to a corresponding network module or a sending module in a return sending path.

In some embodiments, the preprocessing of the transmission assistance module includes length conversion, or/and data path selection, or/and arbitration, or/and merging of two or more types of data; the receiving auxiliary module processes the parsed data, including length conversion, or/and data path selection, or/and arbitration.

In some embodiments, if the analyzed data is of a data stream type, the analyzed data or the data processed by the receiving auxiliary module is sent to the corresponding network module, and if the analyzed data is of a control stream type, the analyzed data or the data processed by the receiving auxiliary module is returned to the sending module in the sending path.

In some embodiments, the network interfaces or/and routing networks are asynchronous, communicating using a handshake protocol.

In certain embodiments, the handshake protocol is a four-phase bonded data handshake protocol or a two-phase bonded data handshake protocol.

In certain embodiments, the network on chip has multicast and on-line reconfiguration functionality.

In certain embodiments, the network-on-chip is a two-dimensional network or a three-dimensional network; the two-dimensional network is a mesh or butterfly network.

A network on chip capable of transmitting event type data, the event type data comprising an event generated address and an event generated timestamp; the network on chip comprises at least one network node, each node comprises a network interface and a router; the router is used for routing data packets among routing networks; the data packets comprise event packets or/and configuration packets or/and data packets of any other data type; the network interface is coupled with the router and used for converting data formats, converting data packets into other data formats or converting other data formats into data packets.

In some embodiments, the data type includes the control flow or/and data flow.

In some embodiments, the routing network is based on a deadlock free wormhole routing algorithm to maintain the order between events.

In some embodiments, the network interface comprises a transmit path and a receive path; the sending path comprises a sending module, a receiving module and a sending module, wherein the sending module is used for creating a data packet based on the input data of the network interface and sending the created data packet to a corresponding router; the receiving path comprises a receiving module, and the receiving module analyzes the data packet from the router, obtains the analyzed data, and directly or processes and outputs the analyzed data.

In some embodiments, the transmit path further comprises a transmit assist module for pre-processing input data from the network module; the preprocessing comprises length conversion, or/and data path selection, or/and arbitration, or/and combination of two types of data and more types of data.

In some embodiments, if the analyzed data is of a data stream type, the analyzed data or the processed data is output, and if the analyzed data is of a control stream type, the analyzed data or the processed data is returned to the sending module in the sending path.

In some embodiments, the network interfaces or/and routing networks are asynchronous, communicating using a handshake protocol; the network on chip is a two-dimensional network or a three-dimensional network, and the two-dimensional network is a mesh network or a butterfly network.

A neuromorphic chip communicates with other devices based on the FPGA-based firmware platform or the network-on-chip.

In some embodiments, the neuromorphic chip is tested or/and debugged or/and configured based on the firmware platform or the on-chip network to obtain the optimal configuration parameters of the neuromorphic chip.

A neuromorphic chip having the same configuration parameters as the neuromorphic chip having the optimal configuration parameters.

Some or all embodiments of the invention have the following beneficial technical effects:

1) The network-on-chip can transmit events, is used for communication between a chip and other components or modules, and utilizes the rapid integration of a neuromorphic system;

2) The network on chip has the functions of multicast and online reconfiguration, and is convenient for test and monitoring;

3) The network on chip maintains the sequence among the events based on the deadlock-free wormhole routing algorithm, has strong stability and reliability, can customize a data structure (such as customizing a data result aiming at a data packet which is sensitive to the event and cannot be cracked), can flexibly convert a data format, and can realize the maximum throughput;

4) The firmware platform based on the FPGA can improve the flexibility of the neuromorphic chip;

5) The firmware platform supports event-driven type neuromorphic hardware, and can improve the portability of the FPGA on hardware board levels with different neuromorphic types; meanwhile, the firmware platform can transmit AER events and data in any other format/type;

6) The firmware platform adopts the modularized framework, and a user can repeatedly use the existing network module, thereby reducing the workload, shortening the research and development period and reducing the chip development cost.

Further advantages will be further described in the preferred embodiments.

The technical solutions/features disclosed above are intended to be summarized in the detailed description, and thus the ranges may not be completely the same. The technical features disclosed in this section, together with technical features disclosed in the subsequent detailed description and parts of the drawings not explicitly described in the specification, disclose further aspects in a mutually rational combination.

The technical scheme combined by all the technical features disclosed at any position of the invention is used for supporting the generalization of the technical scheme, the modification of the patent document and the disclosure of the technical scheme.

Drawings

FIG. 1 is an application of a firmware platform in a preferred embodiment of the present invention;

FIG. 2 is a block diagram of a firmware platform in a preferred embodiment of the invention;

FIG. 3 is a schematic diagram of a network on chip in an embodiment of the invention;

FIG. 4 is a block diagram of a node in a preferred embodiment of the present invention;

FIG. 5 is a block diagram of a network on chip in a first preferred embodiment of the present invention;

FIG. 6 is a block diagram of a network on chip in a first preferred embodiment of the invention;

FIG. 7 is a block diagram of a router in accordance with a preferred embodiment of the present invention;

fig. 8 is a block diagram of a network interface NI in a preferred embodiment of the invention;

fig. 9 is a schematic diagram of a firmware platform suitable for use with the neuromorphic chip.

Detailed Description

Since various alternatives cannot be exhaustively described, the following will clearly and completely describe the main points in the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Other technical solutions and details not disclosed in detail below are generally regarded as technical objects or technical features that are conventionally achieved in the art by means of conventional means, and are not described in detail herein.

In the present invention, "/" at any position indicates a logical "or" unless it is a division meaning. The ordinal numbers such as "first," second, "etc., in any position of the invention are used merely as distinguishing labels in description and do not imply an absolute sequence, either temporally or spatially, or that the terms in such a sequence, and hence the term" similar terms in any other ordinal relation, are necessarily different.

The present invention may be described in terms of various elements combined into various embodiments, which may be combined into various methods, articles of manufacture. In the present invention, even if the points are described only when introducing the method/product scheme, it means that the corresponding product/method scheme explicitly includes the technical features.

The presence or inclusion of a step, module, feature in any location in the disclosure does not imply that such presence is the only exclusive presence, and those skilled in the art are fully enabled to derive other embodiments based on the teachings herein, along with other techniques. The embodiments disclosed herein are generally for the purpose of disclosing preferred embodiments, but this does not imply that the opposite embodiment to the preferred embodiment is excluded/excluded from the present invention, and it is intended to cover the present invention as long as such opposite embodiment solves at least some technical problem of the present invention. Based on the point described in the embodiments of the present invention, those skilled in the art can completely apply the means of substitution, deletion, addition, combination, and order change to some technical features to obtain a technical solution still following the concept of the present invention. Such a configuration without departing from the technical idea of the present invention is also within the scope of the present invention.

Firmware platform (also known as unified firmware): bear the weight of the hardware that FPGA runs, promoted the flexibility of FPGA hardware through the modularization to and FPGA's portability on different hardware board levels.

A Network On Chip (NOC) is the core of a firmware platform, which controls communications between loads mounted to the firmware platform and is configurable.

Address Event Representation (AER): the method is used for communication between neuromorphic chips or modules inside the chips, and comprises an event generation address (such as pixel coordinates which are triggered to generate an event) and an event generation time stamp. The AER protocol can establish virtual connections between neurons, and is beneficial to efficient hardware implementation of a neuromorphic chip. In some cases, the AER signal may be converted to SAER (Serial Address Event Representation) to improve transmission efficiency. Reference is made in particular to chinese patent No. cn202210277193.X (CN 114372019B).

A neuromorphic chip: the event-driven circuit has the characteristic of event driving, and the event is driven to be calculated or processed when occurring, so that the ultrahigh real-time performance and the ultralow power consumption are realized on a hardware circuit. Depending on the type, neuromorphic chips are classified as neuromorphic chips based on analog or digital or data hybrid circuits.

Spiking Neural Network (SNN): one of the event-driven neuromorphic chips is a third-generation artificial neural network, and has the advantages of abundant space-time dynamics characteristics, various coding mechanisms, event-driven characteristics, low calculation cost and low power consumption. Compared with an artificial neural network ANN, the SNN is more bionic and advanced, and the SNN-based brain-like computing (brain-embedded computing) or neuromorphic computing (neuromorphic computing) has better performance and computing cost than the traditional AI chip.

Fig. 1 shows an application of a firmware platform according to a preferred embodiment of the present invention, and a Computer (PC) interacts with a neuromorphic chip based on the FPGA firmware platform. In the figure, the FPGA firmware platform may receive or send data with the chip or/and the memory or/and the USB control module, for example, the FPGA firmware platform receives data from the chip and transfers the data to the USB control module, and transmits the data to the PC through the USB. The figure is merely an illustration of which devices a particular FPGA firmware platform communicates with, and the invention is not limited in this regard.

Fig. 2 is a block diagram of a firmware platform including a network on chip NOC and at least one network module according to a preferred embodiment of the present invention. The network on chip NOC is an interconnection system, and the network on chip has a plurality of network nodes which can be connected with each other, so that the information transmission can be rapidly, flexibly and safely carried out among the network modules. A network module (also referred to as module in the present invention) is a modular design, and is an execution node of a firmware platform, which performs tasks or communicates with an external load. The network modules are interconnected through a Network On Chip (NOC). Users only need to mount the loads on the firmware platform, utilize the network on chip NOC and define the network on chip, and the interaction among the loads can be easily realized.

In some embodiments, the network module may be a USB network module, a first neuromorphic chip network module, a second neuromorphic chip network module, an SPI network module, a JTAG network module, an I2C network module, a memory network module, or the like. The neuromorphic chip may be a general brain-like chip, or/and a brain-like chip for processing two-dimensional or more signals such as visual signals, or/and a brain-like chip for processing one-dimensional signals such as auditory signals, etc., which are not limited in the present invention.

In some embodiments, each network module may interact with the network on chip through a variety of data paths (channels), each having a different communication mechanism. Illustratively, the first network module communicates the second type of data to the third network module and the fourth network module through the second channel by configuring the network on chip such that the first network module communicates the first type of data to the second network module through the first channel.

In some embodiments, the network modules and the network on chip may communicate via data streams (streams) or control streams (flows), although the invention is not limited thereto.

In some embodiments, a data stream (stream) includes two basic data types: AER events and RAW (RAW block data), to which the invention is not limited. Each AER event includes the address and time at which the event occurred, is an inherent property of the neuromorphic system, and is arbitrarily long and arbitrarily divisible. RAW data, unlike AERs, includes all other formats that can be transmitted as complete packets, such as SPI, JTAG, I2C, etc. The RAW data length can be set to any multiple of bytes, which provides great flexibility for the network module, and the RAW includes a packet header, an extension packet header (optional), data (data), and the like.

The control flow (flow) includes a network-on-chip configuration for configuring attributes such as routing paths and target network interfaces or/and a network module configuration for setting the attributes of the target network module. Determining and configuring a target network module for receiving data based on the network-on-chip configuration or/and the network module configuration, and controlling operations such as chip reset, internal signal enable and the like.

Fig. 3 shows the communication between the PC and the sensor, and the network module 3 (equivalent to SPI module) receives data from the sensor, transmits the data to the network module 1 (equivalent to USB module) through the network on chip, and transmits the data to the PC via the USB protocol.

The modular framework decouples the work of data lanes (data pipeline), communication modules, chip drivers and algorithm design, and allows the application system to be implemented most quickly with minimal effort. For example, a network on chip NOC suitable for a neuromorphic chip is designed and a network interface (also referred to as an interface for short in the present invention) suitable for the neuromorphic chip is defined, some network modules (such as an IO network module, a driver network module, a USB network module, a bluetooth network module, a readout network module, a pulse generator network module, an algorithm network module, etc.) are gradually developed in system implementation, and a unified modular design facilitates fast integration of the chips and has strong reusability, and only an AER data protocol or a RAW data protocol needs to be implemented on a network interface NI. When a new system or a new chip needs to be designed subsequently, the existing communication module and the algorithm module can be massively multiplexed, the module and the defined interface do not need to be redesigned, and only the corresponding adaptive module needs to be developed aiming at the difference, so that the workload is reduced, and the research and development period is shortened.

Fig. 4 is a schematic diagram of a network on chip according to an embodiment of the invention, through which signals can be conveniently and quickly transferred between any network modules. The network on chip comprises a plurality of nodes, any node in the network on chip NOC comprises an interface and a router which are coupled with each other, and fig. 5 is a block diagram of the node in a preferred embodiment of the invention.

The interface NI is equivalent to a bridge between a route (transmitting data based on a routing protocol) and a network module (transmitting data based on a network module protocol) and a terminal to ensure consistency and stability of data.

Routers (routers) are used to route data between routing networks, in some embodiments, in the form of packets. The present invention employs asynchronous routing to better communicate event data.

Network on chip packets (NOC packets) include a number of types, and in a preferred embodiment, 4 types of packets: AER data packet, RAW data packet, configuration data packet, wherein the configuration includes network-on-chip configuration and network module configuration. In addition, the type of packet may be identified using identification information, which in some embodiments may be represented using certain bits in the packet fragment.

Fig. 5 is a structural diagram of a network on chip according to a first preferred embodiment of the present invention, where the network on chip NOC adopts a mesh structure (also called mesh), and the routing protocol is a deadlock-free wormhole routing algorithm (dead-free wormhole routing algorithm), the protocol not only maintains the order inside a single packet, but also maintains the order of packets between any source and any destination, for example, from a specific source to a specific destination, the total ratio of packets sent first is compared with those sent first and then the packets arrive first, thereby improving the stability and the security of transmission. The deadlock-free wormhole routing algorithm is incorporated into the present application by way of full reference: xiaong D, luo W. An effective adaptive decode-free routing for torus networks [ J ]. IEEE Transactions on Parallel and Distributed Systems, 2011, 23 (5): 800-808.

However, for small systems on chip, the mesh Network consumes relatively high hardware resources, and a more efficient Butterfly Network (Butterfly Network) or cross-bar (crossbar) and corresponding topology may be used.

In some embodiments, the network-on-chip is a two-dimensional (2D) network-on-chip or a three-dimensional (3D) network-on-chip, which is not limited in this respect.

In other preferred embodiments, the network on chip delivers the signals by multicast (multicasting), and the network on chip supports online reconfiguration of the network connection to facilitate monitoring and testing.

In a modified embodiment, the network on chip includes a filter (filter), for example, to filter the signal transmitted to the network on chip by the network module, so that the signal filtering can be realized quickly and effectively.

Fig. 6 is a structural diagram of a network on chip according to a second preferred embodiment of the present invention, which employs a mesh structure, including 9 nodes, where any node includes an interface and a corresponding router, and information is transmitted via a route to implement connection between a source interface and a destination interface. The routing network is a 3 x 3 network, and each router includes 9 pairs of ports (9 input ports and 9 output ports), where one pair of ports is connected to a corresponding interface (e.g., interface 1), and the remaining 8 pairs of ports are connected to the remaining routers. Alternatively, the routing network may have any number of routers or may take any other configuration, such as a ring (torus) configuration, but the invention is not limited thereto. In some embodiments, the router may process multiple data streams (streams) simultaneously in parallel.

The router is equivalent to an arbiter that receives data from 9 different input ports and arbitrates to determine which of the 9 output ports to output.

Fig. 7 is a schematic diagram of a router in a preferred embodiment of the present invention, wherein when a packet is received from any input port, the router parses (also called decodes) the packet header of the packet and calculates its output port. To find the optimal path, in a preferred embodiment, the output ports of the router are computed for a particular input using a deadlock free wormhole routing algorithm.

The router includes input selection, routing control, and output selection. An input selection for determining valid input ports; a routing control for selecting all output ports to which a data packet may be transmitted, based on a header of the data packet received from an active input port; and the output selection is used for selecting a target port from all possible output ports to output the data packet.

In a refinement, the router further comprises a buffering unit for buffering incoming or/and outgoing data packets.

The interface NI of the present invention includes a transmission path for a network module to transmit data to the network on chip NOC or a router, and a reception path for the network module to receive data from the network on chip NOC or the router. The network on chip NOC transmits data packets internally, and the network modules communicate with the load, and the data type and protocol used by the network on chip NOC are different from those of the network on chip NOC, so that the interface NI is required to be used as a bridge between the two protocols to convert the data format, so as to realize stable and effective transmission.

The transmission path includes a transmission module, creates a packet based on the data from the network module, and determines a transmission destination of the packet, for example, transmits the created packet to the router R8 corresponding to the eighth interface, and routes the packet to a target router (such as the router R6) via the router R8. In some cases, the sending module checks before sending the data packet. In some cases, the sending module communicates in a multicast manner to improve efficiency.

In a further embodiment, the transmit path further comprises a transmit auxiliary module, which is coupled between the input of the interface NI and the transmit module, for preprocessing the input data. For example, length conversion is performed to convert input from the network module into a data length required by the router, or/and an output path is selected according to a data type (e.g., data is transmitted through a first path and is controlled to be transmitted through a second path), or/and arbitration is performed when a plurality of types of data are input (e.g., when AER event and RAW data are input simultaneously, a data type with higher priority is arbitrated to be output), or/and a plurality of types of data are combined when a plurality of types of data are input, and the plurality of types of data are combined.

Corresponding to the sending path, the receiving path includes a receiving module, which parses the data packet received from the router to obtain parsed data, and the parsed data format is suitable for the network module. The packets received from the router may include AER packets and RAW packets. In some embodiments, the data received from the router may include a configuration data packet, wherein the configuration data may be a network-on-chip configuration or/and a network module configuration, and if the configuration data is a network-on-chip configuration, the configuration data is coupled to a sending module in a sending path to configure a sending destination. Since the configuration data is a control flow, it is different from a data flow (AER event and RAW data).

In a modified embodiment, the receive path further comprises a receive auxiliary module coupled between the output of the interface NI and the receive module. The receive assist module may be operable to perform a length conversion to convert an input from the router to a data length required by the network module or/and to perform an arbitration to output a higher priority data type.

In a preferred embodiment, the interface NI of the present invention is an asynchronous interface, communicating using a handshake protocol. The handshake protocol is a four-phase binding data handshake protocol, a two-phase binding data handshake protocol, or other handshake protocols, which is not limited in the present invention.

Fig. 8 is a block diagram of a network interface NI in a preferred embodiment of the invention. In the transmission path, the transmission auxiliary module includes a first byte conversion unit and a transmission control unit. The first byte conversion unit is used for performing length conversion, for example, performing byte conversion on input AER data, and converting the input AER data from 32 bits (4 bytes) to 16 bits (2 bytes). And the sending control unit is used for combining the converted AER event and the RAW data and then outputting the combined AER event and RAW data, or arbitrating the converted AER event and RAW data and then outputting the arbitrated AER event and RAW data.

Alternatively, the first byte conversion unit is not necessary, and for example, the input data in the transmission path does not need to be converted in length, the transmission control unit directly outputs the input data. Alternatively, the transmission control unit may combine the different types of input data and output the combined data, or may arbitrate a plurality of different types of input data and output the arbitrated data.

In the receiving path, the receiving auxiliary module comprises a second byte conversion unit and a receiving control unit. A second byte conversion unit, configured to perform length conversion, for example, perform byte conversion on the length of the parsed packet, such as converting from 16 bits (2 bytes) to 32 bits (4 bytes). And the receiving control unit is used for combining the converted AER event and the RAW data and then outputting the combined AER event and RAW data, or arbitrating the converted AER event and RAW data and then outputting the arbitrated AER event and RAW data. Alternatively, byte conversion is not necessary in the receive path.

In fig. 8, the port on the left side of the interface NI is coupled to the network module, and the inputs from the network module to the interface include aerosio (for transporting AER packets), channel _ DI (for transporting channel ID of AER packets), RAW _ DI (for transporting RAW packets), and RAW _ channel _ DI (for transporting channel ID of RAW packets). The outputs of the interface to the network module include config _ DO, AER _ DO (the network module receives the AER packet), AER _ source _ DO (the network module ID corresponding to the source of the current AER packet), RAW _ DO (for the network module to receive the RAW packet), and RAW _ source _ DO (the target network module ID corresponding to the source of the current RAW packet).

In fig. 8, the port on the right side of the interface NI is coupled to the relevant router, the signal DO indicates that the data packet output by the interface NI is transmitted to the relevant router, and the signal DI indicates that the data packet received by the router is transmitted to the target network module via the interface NI.

The invention relates to a firmware platform suitable for a neuromorphic chip and a corresponding network on chip, wherein in a preferred embodiment, at least one neuromorphic chip is tested, debugged, configured and the like based on the firmware platform or the network on chip so as to obtain the optimal configuration parameters (such as weight value, time constant, attenuation coefficient, network topology and the like) of the neuromorphic chip.

FIG. 9 is a diagram of a firmware platform suitable for neuromorphic chips, where a unified, modular, flexible firmware platform is portable, for example, suitable for use with a first neuromorphic chip and a second neuromorphic chip, where the first neuromorphic chip and the second neuromorphic chip are different chips. In some embodiments, the optimal configuration parameter deployment of the first neuromorphic chip maps to a first class of commercial neuromorphic chips, and the optimal configuration parameter deployment of the second neuromorphic chip maps to a second class of commercial neuromorphic chips, i.e., some neuromorphic chips have the same configuration parameters as the aforementioned neuromorphic chips (neuromorphic chips coupled to the firmware platform). In some preferred embodiments, the memory is used to store the configuration parameters of the neuromorphic chip coupled to the firmware platform and the mapping tool is used to map the configuration parameter deployment into a large batch of engineering or commercial chips, the memory may be a flash, cache, register, or the like.

The neuromorphic chip may be a general brain-like chip, or/and a brain-like chip for processing two-dimensional or more signals such as visual signals, or/and a brain-like chip for processing one-dimensional signals such as auditory signals, etc., which are not limited in the present invention. In a preferred embodiment, the neuromorphic chip of the present invention is based on a spiking neural network, see in particular international patent No. PCT/CN2021/088143 (application number) or chinese patent No. CN202180004244.5.

The firmware platform is different from a firmware platform in the prior art, supports event-driven type neural morphology hardware, and can improve the transportability of the FPGA on different neural morphology hardware board levels; the firmware platform can not only transmit AER events, but also transmit data in any other format/type, and has great flexibility; the firmware platform simplifies the implementation mode of the user algorithm on hardware, and the user can repeatedly use the existing network module, thereby reducing the workload, shortening the research and development period and reducing the chip development cost; the firmware platform of the invention decouples the work of a data pipeline, a communication module, a chip driver, an algorithm design and the like by adopting a modularized framework, and allows an application system to realize the fastest implementation with the minimum workload; the firmware platform is easy to modify, and the network-on-chip or the network module can be modified by using the configuration data; the firmware platform of the invention can easily modify and maintain the data sequence from any source network module to any target network module, and has strong stability and reliability.

The network on chip of the invention can transmit events, and the NI interface is suitable for a neuromorphic chip. Different from the network on chip in the prior art, the network on chip of the invention is different from the communication network in the chip, and is used for communication between the chip and other components or modules. The network on chip can quickly integrate new neural form hardware, has the functions of multicast and online reconfiguration, maintains the sequence among events based on a deadlock-free wormhole routing algorithm, can customize a data structure (for example, customizing a data result aiming at a data packet which is sensitive to the event and cannot be cracked), can flexibly convert a data format, and can realize the maximum throughput.

While the present invention has been described with reference to particular features and embodiments thereof, various modifications, combinations, and substitutions may be made thereto without departing from the invention. The scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification, and it is intended that the method, means, and method may be practiced in association with, inter-dependent on, inter-operative with, or after one or more other products, methods.

Therefore, the specification and drawings should be considered simply as a description of some embodiments of the technical solutions defined by the appended claims, and therefore the appended claims should be interpreted according to the principles of maximum reasonable interpretation and are intended to cover all modifications, variations, combinations, or equivalents within the scope of the disclosure as possible, while avoiding an unreasonable interpretation.

To achieve better technical results or for certain applications, a person skilled in the art may make further improvements on the technical solution based on the present invention. However, even if the partial improvement/design is inventive or/and advanced, the technical idea of the present invention is covered by the technical features defined in the claims, and the technical solution is also within the protection scope of the present invention.

Several technical features mentioned in the attached claims may have alternative technical features or may be rearranged with respect to the order of certain technical processes, materials organization, etc. Those skilled in the art can easily understand the alternative means, or change the sequence of the technical process and the material organization sequence, and then adopt substantially the same means to solve substantially the same technical problems to achieve substantially the same technical effects, so that even if the means or/and the sequence are explicitly defined in the claims, the modifications, changes and substitutions shall fall within the protection scope of the claims according to the equivalent principle.

The method steps or modules described in connection with the embodiments disclosed herein may be embodied in hardware, software, or a combination of both, and the steps and components of the embodiments have been described in a functional generic manner in the foregoing description for the sake of clarity in describing the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application or design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention as claimed.

Claims

1. An FPGA-based firmware platform, characterized in that:

the firmware platform is used for the neuromorphic hardware to communicate with other components;

the firmware platform is capable of transmitting event type data, the event type data including an event generated address and an event generated timestamp;

the firmware platform comprises at least one network module and a network on chip;

the network module is coupled with neuromorphic hardware or other components;

the network on chip is an interconnection system, coupled to the network modules, and configured to perform communication between the network modules.

2. The firmware platform of claim 1, wherein:

the network on chip comprises at least one node, and each node corresponds to one network module;

each node comprises a network interface and a router;

the network interface is used for exchanging data with the corresponding network module and the router; the router is used for routing data among routing networks.

3. The firmware platform of claim 2, wherein:

the network on chip supports control flow or/and data flow;

the data stream includes events or/and any other type of data.

4. The firmware platform of claim 2, wherein:

the routing network is based on a deadlock free wormhole routing algorithm to maintain the order between events.

5. The firmware platform of claim 2, wherein:

the routing network transmits data in the form of data packets;

the router includes: an input selection for determining valid input ports; a routing control for selecting, based on a packet received from an active input port, all output ports to which the packet is to be sent; and the output selection is used for selecting one target port from all the output ports so as to output the data packet.

6. The firmware platform of claim 2, wherein:

the network interface comprises a sending path and a receiving path;

the sending path is used for converting the data format of the network module into a data packet format supported by a routing network;

the receiving path is used for converting the data packet format supported by the routing network into the data format of the corresponding network module.

7. The firmware platform of claim 6, wherein:

the network module is provided with at least one data path to interact with the network on chip, or/and the data packet comprises an event packet or/and a configuration packet.

8. The firmware platform of claim 6 or 7, wherein:

the sending path comprises a sending module and a sending auxiliary module, and the sending auxiliary module is used for preprocessing input data from the network module; the sending module is coupled between the sending auxiliary module and the router, creates a data packet based on the input data or the preprocessed data, and sends the created data packet to the corresponding router;

the receiving path comprises a receiving module and a receiving auxiliary module, and the receiving module analyzes the data packet from the router to obtain analyzed data; the receiving auxiliary module is coupled between the receiving module and the corresponding network module and used for processing the analyzed data; and based on the type of the analyzed data, sending the analyzed data or the data processed by the receiving auxiliary module to a corresponding network module or a sending module in a return sending path.

9. The firmware platform of claim 8, wherein:

the preprocessing of the transmission auxiliary module comprises length conversion, or/and data path selection, or/and arbitration, or/and combination of two or more types of data;

the receiving auxiliary module processes the parsed data, including length conversion, or/and data path selection, or/and arbitration.

10. The firmware platform of claim 8, wherein:

if the analyzed data is of a data stream type, the analyzed data or the data processed by the receiving auxiliary module is sent to the corresponding network module, and if the analyzed data is of a control stream type, the analyzed data or the data processed by the receiving auxiliary module is returned to the sending module in the sending path.

11. The firmware platform of any one of claims 2 to 7, wherein:

network interfaces and/or routing networks are asynchronous and communicate using a handshake protocol.

12. The firmware platform of claim 11, wherein:

the handshake protocol is a four-phase binding data handshake protocol or a two-phase binding data handshake protocol.

13. The firmware platform of any one of claims 2 to 7, wherein:

the network on chip has multicast and on-line reconfiguration functions.

14. The firmware platform according to any one of claims 2 to 7, wherein:

the network on chip is a two-dimensional network or a three-dimensional network;

the two-dimensional network is a mesh or butterfly network.

15. A network on chip, characterized by:

the network on chip is capable of transmitting event type data, the event type data including an event generated address and an event generated timestamp;

the network on chip comprises at least one network node, each node comprises a network interface and a router;

the router is used for routing data packets among routing networks; the data packet comprises an event packet or/and a configuration packet;

the network interface is coupled to the router and used for converting data formats, converting data packets into other data formats or converting other data formats into data packets.

16. The network on chip of claim 15, wherein:

the network on chip supports control flow or/and data flow.

17. The network on chip of claim 15, wherein:

18. The network on chip of claim 15, wherein:

the network interface comprises a sending path and a receiving path;

the sending path comprises a sending module, a receiving module and a sending module, wherein the sending module is used for creating a data packet based on the input data of the network interface and sending the created data packet to a corresponding router;

the receiving path comprises a receiving module, and the receiving module analyzes the data packet from the router, obtains the analyzed data, and directly or processes and outputs the analyzed data.

19. The network on chip of claim 18, wherein:

the transmission path further comprises a transmission auxiliary module, and the transmission auxiliary module is used for preprocessing input data from the network module;

the preprocessing comprises length conversion, or/and data path selection, or/and arbitration, or/and combination of two types of data and more types of data.

20. The network on chip of claim 18, wherein:

and if the analyzed data is of a control flow type, returning the analyzed data or the processed data to the sending module in the sending path.

21. The network on chip of any of claims 15 to 20, wherein:

the network interface or/and the routing network are asynchronous and communicate by using a handshake protocol;

the network on chip is a two-dimensional network or a three-dimensional network, and the two-dimensional network is a mesh network or a butterfly network.

22. A neuromorphic chip characterized by:

communicating with other devices based on a firmware platform of any one of claims 1 to 14 or a network on chip of any one of claims 15 to 21.

23. The neuromorphic chip of claim 22 wherein:

and testing or/and debugging or/and configuring the neuromorphic chip based on the firmware platform or the on-chip network so as to obtain the optimal configuration parameters of the neuromorphic chip.

24. A neuromorphic chip having the same configuration parameters as the neuromorphic chip of claim 23.