GB2583957A - Associative memory apparatus - Google Patents

Abstract

An associative memory apparatus 100 includes an array of synapse elements 106, partitioned into at least one synapse row group, and an array of concept neurons 107. Each synapse row group is controlled by an individual concept neuron. Each synapse element has a pre-synapse memory and a post-synapse memory (fig. 8A). Input data is formatted and then encoded using an integer number of hash function pairs 103,104. A corresponding integer number of synapse elements of an active synapse row group are selected and configured in response to the formatted and encoded input data. The corresponding concept neuron is subsequently configured based on logic states of the pre-synapse and post-synapse memories of each selected synapse element. The associative memory apparatus may implement a number of modes of operation in order to learn and recognise concepts or sequences of concepts and generate sequences.

Description

ASSOCIATIVE MEMORY APPARATUS

FIELD OF THE INVENTION

This invention relates to apparatus for implementing associative memories in modular and scalable electronic hardware, in particular associative memory apparatus that employ polychronization techniques and sparse vectors in order to recognise and generate spatiotemporal data dependent on, and representative of, previously learnt example spatiotemporal data.

BACKGROUND TO THE INVENTION

An associative memory is a content-addressable structure that creates mappings between specific input representations (based on input informational content rather than an explicit address) and specific output representations and data is retrieved from memory by the presentation of a known input causing the return of its associated outputs.

There are two types of associative memories, namely hetero-associative memory that make associations between paired patterns, such as text and images, and auto-associative memories, which are used to retrieve a previously stored pattern that most closely resembles the current pattern based on fractional parts of the pattern.

Associative memories can be implemented in one of several ways, for example using Content Addressable Memories (CAMs) or artificial neural networks (ANN). Content Addressable Memories (also referred to as an associative memory, associative storage, or an associative array) arc memory architectures that arc generally used to perform fast lookups and searches in networking applications (for example, Ethernet address lookup, data compression, pattern-recognition, cache tags, high-bandwidth address filtering in switches, bridges and routers). Contents are addressed based on an input search symbol, key or tag rather than an address as in conventional semiconductor memory hardware, such as Static Random Access Memory (SRAM).

CAMs can be implemented using conventional SRAM semiconductor memory. However, to achieve single cycle search operations each memory location has additional comparison logic to compare a location's contents to the input search tag. There are two basic forms of CAM, namely binary and ternary. Binary CAMs support storage and searching of binary bits, zero or one (0, 1). Ternary CAMs (TCAMs) support partial matching by storing and allowing don't care bits as part of the comparison (0, I, X). As such, TCAMs have more complex circuitry and are therefore larger, consume more power and are more expensive than their binary counterparts.

In typical use, a CAM has only one or a small number of matches and most words mismatch. Since mismatches dominate, most match-lines transition during pre-charging and evaluation, which leads to a high power consumption. In addition, the search-lines, which broadcast the data to the CAM cells are highly capacitive and are therefore another cause of high power dissipation. As such, CAMs have several drawbacks. They have greater silicon area per bit, dissipate more power, have a higher latency and are more expensive.

Neural associative memories (NAMs) can be implemented by using either feed forward or recurrent artificial neural networks. Such associative artificial neural networks are used to associate one set of vectors with another set of vectors, say input and output patterns. The aim of an associative memory is, to produce the associated output pattern whenever one of the input patterns is applied to the artificial neural network. Examples of NAMs include Hopfield models, linear associator models, bidirectional associative memory (BAM) models and convolutional neural networks.

However. NAMs suffer from the following problems; the capacity of the network is restricted; depending on the number and properties of the patterns to be stored, some of the exemplar may not be the stable states, some spurious stable states different than the exemplars may arise by themselves the converged stable state may be other than the one closest to the applied pattern. Another disadvantage is that there is a loss in accuracy during neural network inference.

In addition, artificial neural network require large datasets to learn new concepts or make associations, often through complex time consuming, computationally expensive and extensive iterative training. This includes performing large numbers of multiplications and additions) that require the storage of a large number of parameters, such as weights. When learning new and related data, such systems are prone to poor learning, and/or catastrophic interference. If the machine learning is not supervised then large labelled datasets are required for unsupervised learning. The latter are not easily available or require significant effort to implement. Consequently, this is a major disadvantage when developing new Artificial Intelligence (Al) applications or Robotics and Autonomous System (RAS) based products. Another disadvantage is that that ANNs do not scale well. In addition, complex associations requiring a hierarchical approach are slow.

Despite recent advances in areas such as vision and natural language processing, deep learning techniques do not provide a satisfactory solution for learning new concepts and associations rapidly from a few data examples. To alleviate over-fitting problems, data augmentation and regularization techniques are employed when using a few examples in low data regimes, but do not solve it. Furthermore, learning is still slow and based on large datasets, requiring many weight updates using stochastic gradient descent, for example.

Consequently, current machine and associative learning methods are inefficient and do not scale well when considering complex system problems. Many Natural Language Processing (NLP) systems base their analysis on statistical approach by creating language models based on simple word count statistics. Hence they require large datasets to extract any useful information. Another disadvantage is that a large number of parameters have to be calculated and stored.

Ideally, associative learning and machine learning needs to be achieved using only one (one-shot) or a few examples and in real time. Allied to this is zero-shot learning where the goal is to recognize and or associate objects or concepts whose instances may not have been encountered during training. The associative memory of the present invention learns or is trained using one or a few example data together with any available metadata, which can include context data, related properties and characteristics of a concept. The said example data (input data) can be structured data, unstructured data, spatio data, temporal data or spatiotemporal data. The data can he derived from real time sources (real time datastreams), such as audio or non real time sources (non-real time datastreams), such as data that represents discrete events recorded over time, for example medical records. For continuous analogue signals, discrete sampled digital data is derived using Analogue to Digital Convertors (ADC) means. The time between consecutive sequential data samples does not have to be the same. Embodiments of the present invention may process input data including, but is not limited to, audio (music. MIDI, manuscript and voice), video, datastreams, sequence data, image, text, sensor, control, actuator, biological, Hoinformatics, chemical, physical, search, game and other data. Another goal of the present invention is to reduce the both the hardware and computational complexity and hence have a dense computation / area figure. One way to achieve this is not to use weights and multipliers as used in conventional neural networks and convolutional neural networks.

The present invention describes associative memory apparatus to overcome the limitations of the aforementioned content addressable memory and artificial neural network based associative memories and related machine learning and inference methods.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention there is therefore provided an associative memory apparatus configured to perform data processing, the associative memory apparatus comprising: control logic circuitry configured to control associative memory modules and execute control tasks to perform different modes of operation on input data; wherein the associative memory synapse array module comprises a plurality of synapse elements organised into one or a plurality of synapse row groups, each synapse row group being controlled by an individual concept neuron of a connected associative memory concept neuron array module, wherein, S synapse elements of an active synapse row group are selected and configured in response to each formatted and encoded input data, where S is an integer and equal to the number of hash function pairs and the corresponding concept neuron is subsequently configured based on the mode of operation and the logic states of the pre synapse memory and post synapse memory of each selected synapse element.

The present invention is based on modular and scalable electronic hardware (digital and mixed signal) to allow the implementation of associative memory architectures for a wide range of applications. In certain embodiments of the invention, a number of associative memory units 100 can be instantiated and combined to form hierarchical associative memory structures required for more complex, multi-dimensional associations. In another embodiment, each associative memory 100 is an addressable core and so can be used to implement different architectures and be interconnected and communicate using Network on Chip (NoC) technology means. The different architectures include, but are not limited to, finite state machines (FSMs), tiled arrays, torus networks, fractal networks, butterfly fat-trees, hypercube networks. In another embodiment, data is transferred between associative memory units 100 using the Address Event Representation (AER) Protocol. The latter protocol is an asynchronous handshaking protocol used to transmit signals between neuromorphic systems. This includes asynchronous Network on Chip (NoC) technology means, such as Globally Asynchronous Locally Synchronous (GALS). With this technique it is possible to remove the global clock and replace it with an asynchronous communication scheme (also referred to herein as an asynchronous timing scheme). Each block consists of an asynchronous wrapper and a synchronous module. The synchronous module handles all computations and the asynchronous wrapper handles all communication with other GALS blocks.The apparatus allows for one-to-many mappings as well as many-to-one mappings or associations. This includes data fusion by mapping data from several related sources to the same associations. Also, the output or part of the output from an associative memory may be feedback to the input of the associative memory in order to implement. complex finite state machines for learning and generating data sequences.

As the skilled person will appreciate, the associative memory described herein may be distributed between a plurality of coupled components in communication with one another. Computer control code and/or data to implement embodiments of the invention may be provided on a physical (non-transitory) data carrier such as a disc, programmed memory, for example non-volatile memory such as Flash, or in firmware. Code and/or data to implement embodiments of the invention may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as C, or code for a hardware description language such as SystemVerilog, Verilog, Verilog-AMS, V1DL/VHDL-AMS or SystemC/SystemC-AMS.

Data input (input data) to the associative memory apparatus 100 is processed and formatted into discrete, self contained units of data called concept data or a concept. The terms concept data and concept refers to any form of data that represents key drivers of a system, including, but not limited to systems states, events, system variables, difference values when transitionin2 from one state to another, memory addresses, objectives, degree of membership or fuzzy values, attributes. spatiotemporal data, numerical data, alphanumeric data coefficients, data signatures or fingerprints, abstract concepts, characteristics and policies. Each concept data or concept may comprise one or a plurality of properties or attributes, each attribute can comprise data and metadata if it is available as part of the input data (and can include context data if it is available as part of the input data) that reflects a characteristic of an entity represented by the concept. If a label (label data) is not available to identify input data then one is generated autonomously by the control logic 109 and will be associated with that particular input data. This will be achieved using a random number generator (not shown).

In order to be able to perform any useful tasks and or gain insights into situations, data relevant to a situation needs to be analysed and processed so that associations can be generated, learnt and subsequent patterns correctly recognised and generated. Learning and mapping between input data and concepts (concept neurons (CN)) can be performed in either supervised or unsupervised mode. In general, the associative memory of the present invention is a machine learning and inference engine and therefore required to perform data processing in the form of filtering, classification, recognition, inference, and prediction functions, which can relate to both precise and fuzzy data. The implementation of the associative memory in parallel hardware circuitry allows several of these data processing tasks to be performed simultaneously and in real time. The associative memory needs to perform these data processing functions autonomously and adapt to situations when employed in applications that. operate in unknown environments; for example, in Robotics and Autonomous System (RAS) applications. An aspect of the present invention is that the same associative memory hardware can implement and perform both machine learning and inference functions. Therefore, the same associative memory can be trained to learn and identify complex multichannel associations, including rules for making multi-criteria decision making.

As such, the associative memory implements and performs several modes of operation to achieve its data processing tasks. A list of these modes of operation are shown in table 1 and described in detail later.

All the modes of operation outlined in table I are based around performing the same sequence of processing steps. In certain modes, some steps are not implemented, are repeated several times or performed in parallel. The generic sequence of processing steps is shown in table 2.

MODES OF OPERATION

LEARN A SINGLE CONCEPT

RECOGNISE A CONCEPT

LEARN ONE OR MORE ATTRIBUTES TOGETHER WITH A CONCEPT

IDENTIFY CONCEPT(S) WITH ONE OR MORE ATTRIBUTES

LEARN ONE OR MORE ATTRIBUTES ASSOCIATED WITH A KNOWN CONCEPT

LEARN AND LINK CONSECUTIVE CONCEPTS THAT FORM A SEQUENCE

IDENTIFY A SEQUENCE FROM A SEQUENCE OF CONCEPTS

GENERATE A SEQUENCE OF CONCEPTS

Table 1.

* ANALYSIS PROCESSING PHASE ^ SYNAPSE PROCESSING --^ PHASE CONCEPT NEURON 4--- 4-- --^

PROCESSING PHASE

HASHING & PRE SYNAPSE PROCESSING READ PRE SYNAPSE MEMORIES LEARNINC1, ANALYSIS &

OUTPUT PHASE

RECOGNITION PHASE

Table 2.

The associative memory 100 comprises a plurality of individual modules referred to herein as either associative memory <name> module or <name> module or <name>, where <name> is the name of the individual module. For example, associative memory synapse array module, synapse array module or synapse array. From a high level perspective, the associative memory operates as follows. Input data is first processed and formatted to produce concept data and any related attribute data (including queries). This data is then hashed S limes using S different hash function, where S is an integer. Two hashes are performed in parallel, a synapse column hash and a synapse row hash. These are explained in more detail later. However, the output from the two hash functions allows any synapse element in the associative memory synapse array module to be selected and its pre synapse memory to be set/configured. Hashing the concept or attribute data and then selecting and setting/configuring the corresponding synapse element S times, where S is an integer, is referred to as the Hashing & Pre Synapse Processing phase. In another embodiment of the invention differential hash coding and synapse row order coding, analogous to polychronization and Rank Order Coding (ROC) is employed. This further increases the memory capacity and reduces false positive errors. Once the concept or attribute data has been hashed S times and the corresponding pre synapse memory set the second phase of the Synapse Processing phase is performed. This is the Read Pre Synapse Memories phase, where each concept neuron (CM) reads the connected synapse row(s) and counts the number of synapses that have been set on each synapse row in a synapse row group (SRG). A synapse row with set synapse elements is referred to as an active synapse row. As the number of synapse columns in an array can be large, for example 1024, any synapse element that has not been set is automatically placed into bypass mode, where its input is connected to its output. Consequently, the set pre synapse memories are effectively concatenated and form a shift register. As such, only the set pre synapse memories need to be accessed and so reduces processing time and power dissipation. Once the Read Pre Synapse Memories phase is complete the first phase of the concept neuron Processing phase is implemented. This is the Recognition phase. In this phase all the synapse row count values in a given synapse row group (R synapse rows per concept neuron, where R is an integer) are added together and compared to a previously stored (also referred to as learnt or configured) threshold value. If the value is equal to or greater than the threshold the concept neuron sets its Recognised flag / bit and outputs the associated concept identification data or bit. If the value is less than the threshold value the Recognised bit is not set. IT none of the concept neurons recognise the concept or attribute then the concept neuron that is currently in the Prepared To Learn state (PTL) configures the corresponding post synapse memories to learn the current settings and sets the threshold value to the number of set pre synapse memories. In other embodiments of the invention more complex recognition schemes are performed. Once learning has been performed the concept neurons clear the set pre synapse memories in readiness for the next input data or Analysis processing phase.

In different modes of operation, the Synapse Processing and Concept Neuron Processing phases may he repeated several times if there is a plurality of attribute data, for example. As explained later, each concept neuron also has a programmable logic function allowing it to perform logic functions on each successive processed attribute. For example, assume the concept neurons in a concept neuron array each represent a different disease and each concept neuron has learnt many attributes (medical conditions, biomarkers) associated with the corresponding disease. A search query could be find all diseases with attributes X, Y and Z. Here, the attributes would be processed in turn (and in parallel by the concept neurons) and at each stage if recognised a concept neuron would set the recognised flag and perform a logic And with the next consecutive search attribute result. If on completion, any concept neuron's recognised flag is still set then those diseases meet the conditions of the search query (X And Y And Z) and are output. In an embodiment of the invention, more complex searches can be performed as the associative memory autonomously learns many relationships in parallel.

The associative memory of the present invention is based around an associative memory synapse array module and an associative memory concept neuron an-ay module. The synapse array consists of plurality of synapse elements comprising C synapse columns and R synapse rows, where both C and R are integers greater than or equal to one. Individual synapse columns and synapse rows are selectable allowing any synapse element to be selected. The number of synapse columns C is determined by the number of output address bits taken from a synapse column hash function. For example, ten output bits taken from the synapse column hash function would enable addressing/selecting up to (2A10) 1024 synapse columns. Associated with each synapse column hash function is a corresponding synapse row hash function (also referred to as a timeslot hash function). Likewise, a number of synapse row hash function output bits are used to generate a timeslot value or synapse row value. For example, four bits can be used to represent up to 16 synapse rows. The number of synapse rows that are addressable by a synapse row hash function is referred to as a synapse row group (SRG), where SRG is an integer. The number of synapse row hash function output bits is small compared to the number of synapse columns. In addition, the number of hash functions S and the number of synapse rows R does not have to be the same. In general, S is larger than or equal to R. A concept neuron (CN) is connected to a synapse row group. The number of concept neurons in a concept neuron array is N, where N is an integer. Consequently, the number of synapse elements in a synapse array is equal to C synapse columns x (N x SRG).

Allocated (3025) Next Free Concept (3024) concept neuron State 0 0 IDLE 0 1 Prepared To Learn (PTL) 1 0 ALLOCATED or LEARNT 1 N/A

Table 3.

-10 -As shown in table 3, a concept neuron has several operational states that are determined by the state of the Allocated status flag 3025 and the next Free Concept status flag 3024. At reset or power up, all but the first concept neuron are put into an idle state. The first concept neuron in a concept neuron array is placed into a Prepared To Learn (PTL) state. Once a concept neuron has learnt a concept it is put into an Allocated state (also referred to as the learnt state). This state transition then triggers the next concept neuron in the array to be put into a Prepared To Learn state in order to learn a new concept. A concept neuron and the connected synapse row group are considered active if the concept neuron is in the PTL state or the Allocated state. Once in the Allocated state, each time a concept neuron recognises a concept or associated attribute an internal counter, referred to as the occurrence counter is incremented by a predefined integer amount. On reaching a maximum value, any further increments are effectively ignored and the value of the occurrence counter remains or is clamped to the maximum occurrence counter value. In addition, a concept neuron's occurrence counter is decremented periodically by a predefined integer amount, usually one and less than or equal to the increment value. If the occurrence counter value is deeremented and becomes zero, the corresponding concept neuron is put into the PTL state, resets the synapse memories (forgets) in its connected synapse row group and outputs a signal to indicate the state transition. This latter signal is propagated to all the other concept neurons in the corresponding concept neuron array and if another concept neuron is in the PTL state then it is subsequently placed into the idle state ensuring that only one concept neuron is in the PTL state.

Both the concept neurons and synapse elements perform several functions or tasks depending on the mode of operation. A synapse row group is controlled by its connected concept neuron. Likewise, a concept neuron array is controlled by the connected processing and format module. Input data is processed and formatted by the associative memory processing and formatting module 102 into a data block referred to as formatted data. Formatted data comprises any combination of concept data, attribute data and query data. Concept data, attribute data and query data are generated by using Natural Language Processing (NLP) techniques implemented by the processing and formatting module 102. Formatted data is input to the both the synapse column hash function module and the synapse row hash function module. Depending on the application and input data type a sequence of processing events is performed by the Processing and Format module in conjunction with the control logic. Each concept data and or attribute data is presented in turn to the hash functions to generate S hash pairs for each concept and or attribute. Together, the synapse column hash function and synapse row hash function will generate a pair of values that arc used to select synapse elements in SRGs whose concept neurons are in either the allocated state or PTL state. Idle concept neurons and hence their respective SRGs do not respond to commands from the Processing and Format module. Selected synapse elements set their pre synapse memory to logic 1, if positive logic is employed. This process occurs S times using the S different synapse column and synapse row hash functions. As such, participating SRG will subsequently have up to S synapses set. There is a small probability that the same synapse element may be selected more than once. In such a case, the corresponding pre synapse memory is set to logic 1 multiple times or once set inhibits any more updates. As the value of S is small compared to C then the concept data and or attribute data is represented internally as a sparse vector consisting of up to S bits. The associative memory should have a large capacity and cater for many to one associations. If only one synapse row per concept neuron is employed then the number of unique synapse element encodings is C! / S!(C -S)! For example, if C = 1024 and S = 16 then the number of encodings is 6.2 x 10^34. Has there are a plurality of synapse rows then this figures increases accordingly.

In order to output results, recognised concepts are converted to their respective labels (label data) and output to peripheral processing apparatus, such as a Graphical User Interface (GUI) or Human Machine Interface (HMI). One way to achieve this is to have a memory (Concept memory 129) where each location or locations is addressed by a concept neuron. That is, there is a one-to-one mapping between concept neurons and each Concept Memory location. In a preferred embodiment of the invention, concept label data is stored as a sequence of concepts in the associative memory. This approach then uses the same circuit structures rather than implementing separate memory structures or technologies. In addition, as concepts can be used multiple times it ultimately reduces the level of circuitry. hi this latter approach, the first concept in the sequence is selected by performing a similar hashing and recognition operation when recognising input data. Each concept neuron is allocated a sparse output vector (referred to as concept neuron identification data) which is hashed and used as input into another related associative memory that in turn is used to generate answers based on learnt labels. For example, an identified medical condition might be pneumonia whose label is the sequence of letter (concepts) p-n-e-u-m-o-n-i-a. It could also be the sequence of phonemes used to pronounce the label. When learning and generating sequences that represent output labels the processing and formatting module 102 takes the form of a concept -12 -data combiner (as described later with reference to figure 16), where selected concept neuron identification data from a plurality of consecutive concept neuron identification data are combined using arithmetic and or logic functions to generate a vector (attribute data) that is then used by the encoder 130 to generate S synapse element addresses as before and subsequently used to recognise the next concept in the label sequence.

A feature of the present invention is that the performance of the associative memory is tuneable. Performance is tuneable by selecting the number of hash functions, synapse column and synapse row address sizes, the inter-hash encoding scheme and memory partitions. Also, the associative memory of the present invention does not use multipliers or weight memories as with conventional neural networks and so uses significantly less hardware resources. Another feature of the present invention is that associations can be learnt and retrieved in real time. They can also be learnt using one of a small number of examples to implement so called one-shot and zero-shot learning. In addition the apparatus can learn and generate spatiotemporal data sequences. The approach adopted by the present invention ensures negligible accuracy loss during inference operations.

In an embodiment of the invention, the Associative Memory apparatus 100 can be represented as an Intellectual Properly (IP) core for instantiating one or a plurality of the cores within a semiconductor hardware circuitry / device, such as an Application Specific Integrated Circuit (ASIC), System on Chip (SoC), or Field Programmable Gate Array (FPGA) devices. An Intellectual Property (IP) core for an associative memory is written in a hardware description language, such as SystemVerilog/S ystemVerilog-AMS, Verilog/Verilog-AMS, VHDL/VHDL-AMS or SystemC/SysmtemC-AMS. The latter are computer readable code (data and instructions) that can be used by an Electronic Design Automation (EDA) tool chain to generate representations of the associative memory 100 in semiconductor hardware circuitry. In such embodiments, interfaces are provided on the said devices allowing them to be directly interconnected to form larger associative and hierarchical associative memories and communicate with processing devices 140, such as a central processing unit (CPU), a microcontroller, a graphic processing unit (GPU) and the like.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be further described, by way of example, with reference to the accompanying figures in which: -13 -Figure 1 shows a high level block diagram of an embodiment of the associative memory; Figure 2 shows waveforms for selected signals and associative modules when performing the recognise mode of operation; Figure 3 shows waveforms for selected signals and associative modules when performing the learn mode of operation; Figure 4 shows waveforms for selected signals and associative modifies when recognising multiple attributes associated with a concept; Figure 5 shows a high level block diagram of the synapse element array connected to the concept neuron array; Figure 6 shows details of a synapse row group (SRG) and related concept neuron; Figure 7 shows inter concept neuron status communications; Figure 8A shows a detailed block diagram of a synapse element; Figure 8B shows a partial truth table for generating pre synapse memory A data and write signals; Figure 8C shows a partial truth table for generating pre synapse memory B data and write signals; Figure 8D shows a partial truth table for generating post synapse memory data and write signals; Figure 8E shows a truth table for generating the synapse element output multiplexer select signal; Figure 9A shows a block diagram for the inter-hash encoding is a scheme based on differential hash encoding; -14 -Figure 9B shows the values at various points in the four stage pipeline of a differential hash encoder using S = 8 hash values; Figure 10 shows a block diagram of a concept neuron; Figure 11 shows a block diagram of the Concept neuron control and status logic; Figure 12 shows an example hierarchical associative memory architecture; Figure 13 shows an embodiment of the processing and formatting module for implementing rules; Figure 14 shows an example fuzzy membership function; Figure 15 shows an embodiment of the processing and formatting module for implementing fuzzy rules; Figure 16 shows an embodiment of the processing and formatting module for learning, recognising and generating temporal data.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The following description is of preferred embodiments by way of example only and without limitation to the combination of features necessary for carrying the invention into effect. Associative memory apparatus are disclosed for learning data associations, especially spatiotemporal data associations based on one or a few examples. The associative memory can be configured to operate in several modes to implement related data processing functions, such as classification, recognition, situation analysis and inference. Several of these functions are performed simultaneously using the same hardware blocks, which makes for efficient hardware implementation.

Associative memory is of critical importance for machine learning, information representation, signal processing and a wide range of related applications. Consequently, the associative memory of the present invention can be employed in many artificial intelligence (Al), networking and autonomous system applications.

-15 -Figure 1 shows a high level block diagram of the associative memory 100 of the present invention. The associative memory 100 comprises nine main modules or associative memory modules that are used to perform data processing on input data. Input data is transferred via the communication bus 110 from peripheral sources or other associative memories 100 under the control of the associative memory input interface module or input interface module 101. Data can be transferred serially or in parallel and or over multiple clock cycles. Data transfers can be either big endian or little endian. Control signals (not shown) control the transfer flow and latching of the data in internal register within the input interface module 101. The input interface module 101 can be configured to receive datastreams (temporal data) or non-temporal data and spatio data and is application dependent. They also allow data from several input sources to be multiplexed on the communication bus 110. Received data is subsequently transferred to the associative memory processing and formatting (P&F) module or processing and formatting module 102 via bus 111. Depending on the application, the processing and formatting module 102 can implement different programming or signal processing hardware and algorithms. For example, in voice recognition an analogue signal is filtered and sampled before being converted to digital samples. The digital samples can be further processed using Fast Fourier Transforms (FFTs), normalisation, Connectionist Temporal Classification and Mel-frequency cepstral coefficients (MFCCs), which extracts and maps raw voice information into the frequency domain. For text applications, Natural Language Processing techniques, such as parsers and tokenisers (shingling) are implemented and the resulting data formatted into search queries for example. The processing and formatting module 102 can generate a plurality of related concepts for each input data block. For example, an input signal or data sample could be low pass filtered, bandpass filtered and high pass filtered where concepts are generated for each filter output. For long data strings, for example gene sequences, signatures can be generated using modulo hashing. In image processing applications, the signal processing algorithms include, but are not limited to segmentation, histograms, fuzzy histograms, spatial filters, demosaicing, correlation, convolution, wavelets, recognition and compression.

In certain applications, data from different sources are used to represent the same entity or concept. For example, in voice recognition applications, spectral analysis of the same spoken words will be different for male and female voices and different accents. They will also be subject to different noise and distortions. This scenario would represent a many-to-one mapping or association. However, certain characteristics of the latter mentioned spectral -16 -analysis will be similar. In an embodiment of the invention, processed data is "fuzzified." Fuzzification can be performed in one or more dimensions. The term fuzzification used herein refers to using a reduced resolution for a sampled signal or concept. For example, only using the thirteen most significant bits output from a sixteen bit analogue to digital convertor (ADC). In the frequency domain, fuzzification would mean allowing a group of frequencies to be represented by the same value. A 2-dimensional fuzzification example would be in the frequency domain where both the amplitude and frequency have reduced resolutions. In another embodiment of the invention, fuzzification can be performed at different resolutions either sequentially or preferably in parallel in order to produce a confidence level in the mapping or association.

Formatted data (any combination of concept data, attribute data and query data) output 112 from the processing and formatting module 102 is transferred to an associative memory encoder module or encoder module 130. An encoder module 130 can take one of several forms that perform dimension reduction and are used to implement an encoding scheme for selecting synapse elements. In preferred embodiments of the invention the encoder module 130 uses hash functions, namely the synapse column hash functions module 103 and the synapse row hash functions module 104. As shown in figure 1, formatted data (any combination of concept data, attribute data and query data) from the processing and formatting module 102 is transferred to the associative memory synapse column hash functions module (or synapse column hash functions module) 103 and the associative memory synapse row hash functions module (synapse row hash functions module) 104 via interface bus 112. Both the synapse column hash functions module 103 and the synapse row hash functions module 104 each comprise or implements S. where S is an integer, independent hash functions or hashes. A synapse column hash function is linked or coupled with a synapse row hash function and they operate on the same data and preferably in parallel, the coupled hash functions being referred to as a hash function pair.

For illustrative purposes, several module inputs, outputs and internal blocks are shown consisting of A to S instances, implying 19 instances. In embodiments of the invention the number of instances is an integer and is user definable that can be less than, equal to or greater than 19.

-17 -The output 113 from the synapse column hash functions module 103 are S addresses 113A -113S and a synapse column Hash Valid (SCH VLD) signal 113T, individually and collectively shown as 113. The output 113 represents a form of formatted and encoded input data. Each address is formed by taking M bits, where M is an integer, from a particular synapse column hash function. The S addresses are multiplexed on the M-bit output 113. This allows the addressing of up to 2AM synapse columns. Selection of a synapse column is performed by the synapse column Select module 105, which converts the M bit synapse column hash function output 113 into 2AM synapse column enable signals (SC ENB) shown as SC ENBO -SC ENB(C-1), individually and collectively shown as 115, where C is an integer and represent the number of synapse element columns in a synapse array. These are individual one hot select lines 115 and connect directly to the synapse array module 106.

The output 114 from the synapse row hash functions module 104 are S addresses 114A -114S and a Synapse Row Hash Valid (SRH VLD) signal 114T, individually and collectively shown as 114. The output 114 represents a form of formatted and encoded input data. Output 114 connect to the concept neuron array 107. Each address is formed by taking P bits, where P is an integer, from a particular synapse row hash function. This allows the addressing of up to 2AP synapse rows. The S addresses are multiplexed on the P-bit output 114. The number of synapse rows R, where R is an integer, that are addressable by a synapse row hash function is referred to as a synapse row group (SRG) 200. Each synapse row group connects directly to an individual concept neuron 300 in the concept neuron array 107. The number of synapse columns is generally much larger than the number of synapse rows in order to implement sparse vectors.

Example hash functions used to implement both the synapse column hash function and the synapse row hash function include, but are not limited to, universal hash functions (Zorbrist hashing), non-cryptographic hashing (Fowler-Noll-Vo, SpookyHash, MurmurHash, Pearson hashing, Fast-Hash), orthogonal hashing, modulo hashing, rolling hashes, semantic hashing, keyed cryptographic hash functions and unkeyed cryptographic hash functions (Blake, FSB, MD4, MD5, SHA-1, SHA-256) and the like. The S different hash functions or hashes can be generated by using the same hash function and using S different seed values. In addition, hashing can be implemented using combinatorial logic gates by and-ing and or-ing combinations of input data bits (also known as a Sigma-Pi function) to form an address. The hash functions should have properties of unifoim randomness and good local dispersion.

-18 -Formatted data 112 is hashed S times, where S is an integer and each hash function preferably performs a different hash function on the formatted data. Likewise, the formatted data can be partitioned (including interleaving of hits) or segmented into S sub-blocks and each sub-block is hash independently. The latter approach is preferred when the input data is a large data block or hypervector for example. In an embodiment of the invention, input strings can be processed and formatted to form search queries, which are analysed by the associative memory to calculate and output the nearest matches.

Several prior art systems employ multiple hashing schemes, such as Bloom Filters. However, a Bloom Filter only identifies set membership and cannot store and retrieve processed data. Also, in order to delete a stored set member elaborate extensions, such as counting Bloom Filters are required. The present invention is a more sophisticated architecture (for example, employing polychronization/inter-hash encoding and decoding to reduce overall memory requirements) and overcomes these and other limitations. Also, unlike Bloom Filters, the present invention can perform learning, classification, data retrieval and generation of both spatial and temporal data.

The associative memory synapse array module 106 consists of plurality of synapse elements 400 organised as an array of synapse columns and synapse rows allowing any synapse element to be selected. The synapse array 106 is connected to the synapse column Select module 105 via bus 115 and the associative memory concept neuron array module (or concept neuron array) 107 via N separate buses 116A....116N individually and collectively referred to as 116, where N is an integer and equal to the number of concept neurons 300 in a concept neuron array107. Each bus 116 is further sub-divided into R buses, where R is the number of synapse rows in a synapse row group that a concept neuron controls. An individual synapse element 400 has two memory elements, namely a pre synapse memory 402 and a post synapse memory 404. The pre and post synapse memories are set and reset depending on the mode of operation and input data. A synapse row group 200 is controlled by an individual concept neuron 300 in a concept neuron array 107. Likewise, a concept neuron array 107 is controlled by the corresponding processing and formatting module 102. Results from concept neuron processing phase are transferred to the associative memory output interface module (or output interface module) 108 via bus 117. Depending on the mode of operation, the Output Interface outputs the results as either a vector with one or more bit sets, where each bit corresponds to a set concept neuron and can subsequently be used to address a concept -19 -memory 129 or as a unique vector with many bits set that represents a set concept neuron (referred to as a concept neuron Identification data). If more than one concept neuron is set, then they are read and output in turn by the control logic 109. Concept memory 129 contains labels for the corresponding concepts and is loaded when a concept or attribute is first learnt. The output 118 (concept neuron Identification data) from the Output Interface can be transferred to other associative memories 100 and or be input to the same associative memory 100 to form a finite state machine. In an embodiment of the invention the output 118 only changes value if a new event occurs. Otherwise, the value remains the same.

Control of the individual modules is by associative memory control logic module means 109. The associative memory control logic module (or control logic) 109 implements all the control circuitry to perform all module, memory and register initialisations. The control logic 109 is configured to control all the associative memory modules and execute control tasks or algorithms that implement the different modes of operation (as shown in table 1 and table 2) autonomously based on the received input data. Consequently, the control logic 109 autonomously configures the associative memory modules to implement the different modes of operation and perform all data processing functions or tasks. Clocking for all modules is derived from system clock (SYS_CLOCK) 128. Those familiar with the art will realise that the control implementation and partitioning can take several forms to achieve the same results. For example, the control could be centralised or distributed and embodiments of the invention can employ different control schemes to achieve the same overall functionality, processing and results. For clarity and illustrative purposes the overall control logic 109 is shown in Figure 1 as a separate module with local slave control logic implemented in selected modules. Control buses 121, 122, 124, 125, 126 and 127 control modules 101, 102, 105, 106, 107 and 108 respectively. Control bus 123 controls both the synapse column hash functions module 103 and the synapse row hash functions module 104. Each of the control buses 121, 122, 123, 124, 125, 126 and 127 comprises a group of individual signals, such as clocking signals, reset, enables, data buses and control buses relevant to the functionality of the connected module. External or peripheral processors can access each internal memory, control registers, status registers and control logic 109 of the associative memory unit via the external processor interface bus 119 and External Interface control logic (EICL) 120. The External Interface control logic 120 enables processors to perform read and write memory accesses to all internal memory locations. Access to the synapse memories 402 and 404 is performed using the synapse column Select module 105 and concept neuron array 107. As -20 -such, the External Interface control logic (EICL) 120 of an associative memory 100 has select/enable inputs allowing it to be addressed by an external processor. In addition, the External Interface control logic (ETCL) 120 contains a Node Address register (not shown) which can be configured to uniquely identify each associative memory 100 (referred to as <concept node>) when more than one associative memories 100 are interconnected to form larger associative memory network.

OP-CODE (126C) CONCEPT NEURON OP-CODE (126C) CONCEPT NEURON OPERATION

OPERATION

0000 NULL 1000 AND 0001 RECOGNISE MODE 1001 OR LEARN MODE 1010 EXOR 0011 RESERVED 1011 RESERVED RESERVED 1100 RESERVED 0101 RESERVED 1101 RESERVED RESERVED 1110 RESERVED 0111 RESET 1111 END

Table 4.

As mentioned previously, the selection and implementation of the modes of operation defined in table 1 and table 2 depend on the input data. The processing and formatting module 102 in conjunction with the control logic 109 analyse the input data from the input interface in order to generate concept data, any attribute data and control signals to process the received input data. The control signals include the concept neuron Mode Valid signal 126B and the concept neuron Mode bus 126C. These signals are used to control the concept neuron array 107. The generated concept neuron operation codes (op-codes) are transferred between the control logic and the concept neuron array 107 using concept neuron Mode bus 126C. Table 4 shows exemplary concept neuron operation codes.

Figure 2, figure 3 and figure 4 show waveforms for selected signals and associative modules when performing certain modes of operation. Figure 2 shows selected waveforms for the recognise mode where concept neurons analyse the set synapse elements and determine if the concept is recognised by any of the concept neurons. if so, the recognised flag 333 is set and the result output on bus 117. The mode of operation is terminated using the "END" concept -21 -neuron op-code on bus 126C. The Concept Neuron Mode Bus 126C is validated using Concept Neuron Mode Valid signal 126B. Figure 3 shows selected waveforms for the learn mode where a selected concept neuron reads the set synapse elements in its synapse row group (using SSRD signal 501C), determines the total value and sets the threshold value accordingly. Hence, a concept neuron is configured based on the mode of operation and the logic states of the pre synapse memory and post synapse memory of each selected synapse element. Figure 4 shows selected waveforms for the multiple attribute recognise mode where concept neurons analyse each input attribute in turn and maintain the results of the previous phase. Figure 4 shows the waveforms for a concept neuron that has learnt and is associated with three attributes. The concept neuron logic function is the "AND" op-code in this case. More detailed explanations of the modes of operation are described later.

Figure 5 shows a high level block diagram of the synapse array 106 and the connected concept neuron array 107. The synapse array 106 consists of a plurality of synapse row group modules 200. Each synapse row group module 200 connects to a concept neuron 300 in the concept neuron array 107 using R control buses 316, where R is an integer and represents the number of synapse rows in a synapse row group and is referred to individually and collectively as control bus 316. Each control bus 316 comprises several control and data signals used to control and access synapses in a synapse row. Each synapse row group module 200 is connected to the synapse column Select module output 115. The number of synapse columns C and hence synapse column Select signals that form output 115 is determined by the number of output address bits taken from a synapse column hash function. For example, eleven output bits taken from the synapse column hash function would enable addressing/selecting up to (2^11) 2048 synapse columns. The individual Synapse Column Enable (SC ENB) signals that form output 115. The individual concept neuron outputs that form bus 117 are shown as signals CNO -CN(N-1), where N is an integer. These connect to the concept memory 129 via bus 126.

Figure 6 shows in more detail a high level block diagram of a synapse row group 200 and a corresponding concept neuron 300. A synapse row group 200 comprises R synapse rows, where R is an integer and is equal to 2AP and P represents the number of output bits taken from the synapse row Hash function. The input to each synapse element in column SCO is logic 0 (not shown). The output 413 last synapse element of a synapse row is connected to the synapse row control interface 502 in a concept neuron 300. -22 -

A concept neuron 300 communicates with other concept neurons in a concept neuron array 107 using two interfaces, where each interface comprises an input signal and output signal. The two interfaces are the Next Free Concept (NFC) interface and the Inter Concept Recognised (ICR) interface. The Next Free Concept interface is used to indicate and transfer the operating state of a Concept neuron, that is, Idle, Prepared to Learn (PTL) or Allocated. The Inter Concept Recognised interface is used to inform other concept neurons that the current concept has been recognised. A Next Free Concept (NFC) interface comprises a Next Free Concept Input (NEC IP) 314 and a Next Free Concept Output (NEC OP) 315. The Inter Concept Recognised (ICR) interface comprises an Inter Concept Recognised Input (ICR IP) 312 and an Inter Concept Recognised Output (ICR_OP) 313.

Figure 7 shows how concept neurons 300 are interconnected using the Next Free Concept (NEC) interface and the Inter Concept Recognised (ICR) interface. The Next Free Concept Output (NFC OP) 315 of a concept neuron is connected to the Next Free Concept Input (NFC_IP) 314 of the next concept neuron in the concept neuron array 107, unless it is the last concept neuron. In this case, the Next Free Concept Output (NFC OP) 315 of the last Concept neuron connects to the Next Free Concept Input (NFC IP) 314 of the first concept neuron in the concept neuron array.

The Next Free Concept (NFC) interface is used to indicate that a concept neuron is in the Prepared to Learn (PTL) state and that once a concept has been learnt it transitions to the Allocated state. The latter state transition is used to propagate a signal to the next concept neuron. If the next concept neuron is in an Idle state it will then be placed into the Prepared to Learn (PTL) state. If it is in the Allocated state, it will transfer the signal to its Next Free Concept Output (NFC OP) 315 and so propagate to the next concept neuron. When a concept neuron recognises a concept it sets its recognised flag 3026, which is output on the Inter Concept Recognised Output (ICR_OP) 313. This is daisy chained via internal or gates 3027 to all other concept neurons in a concept neuron array. Note, more than one concept neuron can recognise a concept. If none of the concept neurons recognise a concept then the concept neuron that is in the Prepared to Learn (PTL) state will learn the new concept, transition to the Allocated state and output a signal on its Next Free Concept Output (NFC OP) 315 to indicate the state transition.

-23 -A detailed block diagram of a Synapse element 400 is shown in figure 8. A synapse element uses two memories, namely the pre synapse memory 402 and the post synapse memory 404. The post synapse memory stores the learnt value of a concept or attribute. The pre synapse memory 402 has two 1-bit memories referred to as pre synapse memory A 402A and pre synapse memory B 402B while the post synapse memory 404 is a 1-bit memory and each can store either logic 0 (false) or logic 1 (true) values. In a synchronous example (also referred to herein as a synchronous timing scheme), the memories 402 and 404 are clocked using the synapse clock (SCLOCK) 125A, which is derived from the system clock 128. In addition, each memory can be reset either by a master reset used at initialisation or power up or by the connected concept neuron 300. The resets arc active low is this example. The pre synapse memory reset signal 414 is derived from an And gate 408 which has a master reset input signal (M RESET) 125B and a concept neuron Pre Reset (PRS RESET) 501F. Likewise, the post synapse memory reset signal 415 is derived from an And gate 409 which has a master reset input signal (M RESET) 125B and a concept neuron Post Reset (PUS _RESET) 501G.

The memory technology used to implement both the pre synapse memory 402 and the post synapse memory 404 can be volatile or non-volatile semiconductor memories and includes both digital and analogue memory technologies. They can also be synchronous and asynchronous memory technologies. These include, but are not limited to Random Access Memory (RAM), Read Only Memory (ROM), SRAM, Dynamic RAM (DRAM), Double Data Rate (DDR) SDRAM, Electrically Erasable Programmable Read Only Memory (EEPROM), Magneto-resistive RAM (M-RAM), Ferroelectric RAM (F-RAM), Phase change Random Access Memory (P-RAM), memistors and Resistive Random Access Memory (ReRAM).

Pre synapse memory A and pre synapse memory B are used for different purposes. If selected and the corresponding concept neuron is in the PTL state or Allocated state, both memories are loaded in parallel during the Hashing & Pre Synapse Processing phase. pre synapse memory A is used to form a shift register that is accessed when the connected concept neuron performs the Read Pre Synapse Processing phase using the synapse row Read (SRRD) signal 501C. During this latter phase data on the synapse element Input 413 is loaded into pre synapse memory A on the next clock cycle. This process is repeated until all set synapse memories have been read. However, this process of loading and shifting input data ultimately resets the set pre synapse memories A. Therefore, pre synapse memory B is used to retain the -24 -original value so it can be used in the subsequent concept neuron Processing phase. When in recognition mode of operation, if a pre synapse memory is set, but the corresponding post synapse memory has not learnt a value (is false or logic 0) then the output 413 from the multiplexer 406 is a logic 0. This will be read by the concept neuron and be interpreted as a not recognised concept or attribute. A concept or attribute can only be recognised if it has been previously learnt and its post synapse memories have been set. These then validate the setting of any corresponding pre synapse memories.

Both the pre synapse memory 402 and the post synapse memory 404 are configured based on the value of the input data and the mode of operation. In addition to the synapse clock and reset signals the pre synapse memory 402 and the post synapse memory 404 have Write inputs and Data inputs used to configure the memory.

Pre synapse memory 402 has two Write signals 416A, 416B and two Data input signals 417A, 417B used to control pre synapse memory A 402A and pre synapse memory B 402B. Post synapse memory 404 is configured using the Write signal 418 and the Data signal 419.

The logic values of the Write inputs 416A, 416B, 418 and Data inputs 417A, 417B, 419 are generated by their respective pre synapse memory Logic 401 and post synapse memory Logic 403. Both the pre synapse memory Logic 401 and post synapse memory Logic 403 use combinatorial logic to decode their respective input states and generate write outputs 416A, 416B, 418 and data outputs 417A, 417B, 419 that correspond to the functional requirements for implementing the selected mode of operation.

Both the pre synapse memory A 402A and pre synapse memory B 402B are loaded/set when the corresponding Concept neuron is in the PTL state or Allocated state, the concept neuron operation code is "recognise mode" and both the synapse column Enable (SC_ENB) 115 and the synapse row Enable 501D are logic 1 or true. pre synapse memory A 402A operates in shift register mode and stores the SLIP input value 412 when the corresponding Concept neuron is in the PTL state or Allocated state and the SRRD signal 501C is active (logic 1). A partial truth table for the pre synapse memory Logic 401 for generating Data In 416A, and Write 417A for pre synapse memory A 402A is shown in figure 8B. Likewise, a partial truth table for the pre synapse memory Logic 401 for generating Data_In 416B and Write 417B for pre synapse memory B 402B is shown in figure 8C.

-25 -A partial truth table for the post synapse memory Logic 403 showing the logic conditions for setting the Data in 418 and write signal 419 for the post synapse memory is shown in figure 8D Synapse elements 400 are concatenated to form a synapse row. A synapse element output (SEOP) 413 is connected to the next synapse element's input (SNP) 412, unless it is the first synapse element which has its input 412 connected to logic 0 or the last synapse element in a row which is connected directly to the corresponding synapse row interface 500 of the connected concept neuron 300. Any synapse element that has not been set during the Hashing & Pre Synapse Processing phase is automatically placed into bypass mode, where its input 412 is routed to its output 413 via the synapse element output multiplexer 406. The output of the synapse dement output multiplexer 406 is buffered using buffer 407 to form output signal 413. The buffering / sense amplifier depends on the fan in / fan out requirements of the selected device technology. By bypassing the unused synapse elements, the set pre synapse memories are effectively concatenated and form a shift register. The multiplexer select logic 405 controls synapse element output multiplexer 406 using signal 420 and selects between the next synapse element's input (SEIP) 412 or the output of the pre synapse memory 410B. Figure 8E shows an example truth table for generating the synapse element output multiplexer select signal 420 for synapse element output multiplexer 406.

If volatile memory technologies are used to implement the pre synapse memory 402 and or the post synapse memory 404 then the memories and concept neuron parameters, for example threshold values, can be configured at power up or initialisation by inputting known data and sequentially selecting concept neurons and performing the Synapse Processing and Concept Neuron Processing phases (see table 2). This is effectively in-band configuration using existing circuitry and reduces the need for separate address and decoding logic. However, this mode is controlled via the ECIL interface 120.

In an embodiment of the invention synapse elements 400 and their pre synapse memory 402 and or the post synapse memory 404 are selected and configured sequentially based on encoding or hashing the formatted data S times to produce the synapse column select value and the corresponding synapse row select value. In the subsequent Read Pre Synapse memories phase synapses row in a synapse row group that have synapses set are read in parallel. As each sequential hashing is independent there is no relationship between set -26 -synapse elements. When a concept neuron reads each synapse row there is a potential to read a false positive setting as a synapse may have been set due to learning a related attribute. However, this latter method can he employed for relatively low capacity applications.

One way to reduce potential false positive errors is to implement a more complex synapse element select encoding scheme. In embodiments of the invention, inter-hash encoding is employed to implement the synapse element select encoding scheme. The term inter-hash encoding refers to encoding schemes where synapse column addresses are generated using arithmetic and or logic functions that take two or more of the S hashes output from the synapse column hash function 103 and produce S synapse column addresses. In a preferred embodiment of the invention a form of inter-hash encoding is a scheme referred to as differential hash encoding. A block diagram for the proposed differential hash encoding scheme is shown in figure 9A. Differential hash encoding is implemented by taking consecutive hashes, subtracting their value to produce an absolute value, which is also modulo C and adding the result to a running total, which is also modulo C, where C is the number of synapse columns. This is performed S times on successive hashes. The inter-hash encoding is used together with synapse row order coding, which access each selected synapse row sequentially. The differential hash encoding links selected synapses not just in terms of relative distance, but also the synapse row order in which they are read by the corresponding concept neuron. Figure 9A show a high level block diagram of the differential hash encoding scheme implemented in an embodiment of the synapse column hashing function module 103. The synapse row values are calculated as previously described using the synapse row hash function module 104. However, each generated synapse row value is now stored in the synapse row sequence FIFO 3028 of the concept neuron's control and status module 302.

Registers 103D, 103E, 103H and 103K form a four stage pipeline register. These registers together with the synapse column Hash Generators 103B are controlled locally by control logic 103A. This includes clocking, reset and enable signal, which are not shown for clarity reasons. These registers are also reset to zero at the start of each hashing sequence for an input data value. Input data from the processing and formatting module 102 is hashed S limes, where S is an integer and represent the number of independent hash generators or hashes, by the synapse column hash generators 103B. The P bits from a generated hash are used to represent a synapse column address. These addresses are output on bus 103L from the synapse column hash generators 103B and loaded into register 103C. All S hashes are -27 -generated and output in succession. These are then clocked through the four stage pipeline register stage, where arithmetic operations are performed at different stages. Though arithmetic operations are performed in this example, in other embodiments other operations, such as logical operation can be performed and on more than two values. In this example data values stored in registers 103C and 103D are input to the subtraction logic 103E on buses 103M and 103N respectively. The results of the subtraction are input the absolute value logic 103F, which converts minus values into positive values. The output from the absolute value logic 103E in then input to the Modulo C logic 103G, which restricts the input value to the synapse column address range. The output of Modulo C logic 103G is stored in register 10311 and output on bus 1030. This latter output is input into Adder 1031 together with the output of register 103K, which maintains a running total in order to provide a relative synapse element displacement. The control clocks, processes and output the pipeline registers so that S synapse column addresses are output on bus 113 and transferred to the synapse column Select module 105. Once complete the registers are reset ready for the next input data. To help explain the above mentioned process, figure 9B shows the values at various points in the four stage pipeline using S = 8 hash values. Once all S synapse elements have been selected and set using the above method the Hashing & Pre Synapse Processing phase is complete.

The next phase is the Read Pre Synapse Memories phase. In this embodiment, a synapse row is selected and read by a concept neuron based on the synapse row values previously loaded into the synapse row sequence FIFO 3028. A running total of set synapses is maintained during the synapse row reading process. As S synapse elements have been set then if a logic 0 is read before S synapse elements have been read then the concept or attribute is deemed to be unrecognised by that particular concept neuron and further synapse row reading is halted and the synapses reset ready for the next phase. If all S synapses return a logic 1 then the concept neuron has recognised the concept or attribute and continues to the next phase.

A concept neuron 300 implements the Synapse Processing phase and the Concept Neuron Processing phase as shown in table 2. Embodiments of the invention enable the setting and reading of synapse elements to be performed in different ways. In mode A, synapse rows with set synapse elements are read in parallel. In mode B (inter-hash encoding), synapse rows are read sequentially based on the synapse row value read from the synapse row sequence FIFO 3028. However, the overall function is the same in that selected synapses are read, a -28 -running total maintained, which is subsequently compared to a previously learnt and stored threshold value.

Figure 10 shows a block diagram of a concept neuron 300. A concept neuron 300 is controlled locally by the concept neuron control and status module 302. A more detailed block diagram of the concept neuron control and status module 302 is provided in figure 11, where the concept neuron control logic 3021 implements all the control circuitry to perform the different modes of operation tasks as shown in tables 1 and 2. A concept neuron has R synapse row Interface modules 500, where R is an integer and represents the number of synapse rows in a synapse row group or the number of synapse rows a concept neuron is connected to. The R synapse row Interface modules are shown as 301A... 301R in figure 10 and individually and collectively referred to as 301 when viewed from a concept neuron perspective or 500 from a synapse row Interface module perspective.

For illustrative purposes, several module inputs, outputs and internal blocks are shown consisting of A to R instances, implying 18 instances. In embodiments of the invention the number of instances is an integer and is user definable that can be less than, equal to or greater than 18.

Depending on the mode of operation, the synapse row Interface modules 500 read their respective synapse row either in parallel or sequentially based on the values output from the synapse row sequence FIFO 3028. A synapse row Interface module 500 and the threshold register 307 are controlled locally by the concept neuron control and status module 302 via control bus 311. The control bus 311 consists of control signals, such as clocks, resets, enables (not shown) to control each synapse row Interface module 500 and the threshold register 307 in order to perform the tasks needed to implement the different modes of operation of table 1. A synapse row Interface module 500 connects with a synapse row using synapse row control bus 501 (also shown as synapse row control buses 316A... 316R for the R number of modules and individually and collectively referred to as synapse row control bus 316). A synapse row control bus 501 consists of the following nine signals: synapse clock 125A, Allocated (ALLOC.) status bit 501A, Next Free Concept (NFC) status bit 501B, synapse row Read (SSRD) 501C, synapse row Enable (SR ENB) 501D (which is derived from the synapse row hash function and represents the selected synapse row in a synapse row group), synapse column Enable (SC ENB) 115 (which is derived from the synapse column -29 -hash function, output from the synapse column select logic 105 and represents the selected synapse column in a synapse array 106), Pre Synapse Reset (PRS RESET) 501F, Post Synapse Reset (POS RESET) 501G, the Learn signal 501H (which is active when the received OP-Code 126C is 0010 (LEARN_MODE)) and the synapse element Output (SEOP) 413 of the last synapse element in the synapse row. Logic circuitry to implement the synapse row control bus signals 501 is provided by the synapse row control interface 502. A synapse row control interface 502 operates as a slave controller to the master controller provided by the concept neuron Control and Status logic 302. Rather than logic levels as above, in an embodiment of the invention the synapse column Enable (SC ENB) 115 and synapse row Enable 501D can be timed pulses or spikes.

In mode A. a synapse row Interface module 500 begins to read a synapse row if the value on the synapse element output (SEOP) 413 of the last synapse (that is, the synapse element connected to the synapse row Interface module 500) in the synapse row is a logic 1. If the value is logic 0 this implies that no synapses have been set and all synapses in the synapse row are in bypass mode. A synapse row Interface module 500 will read a synapse row every clock cycle until the input (synapse element output (SEOP) 413 of the last synapse in the synapse row) becomes logic 0. The read counter 503 is incremented if the input value is logic 1. All counters 503 are reset at the start of a pre synapse memory Read phase. Once all participating synapse row Interface modules 500 have completed their synapse row reading phase the count values are output on bus 506 and input to adder 306. The count values 503, 506 can be added sequentially using an adder with a registered output and selecting the participating counters (outputs) 503, (506) in turn or each count value can be added in parallel. The sum of all the participating counters is then compared to a previously set threshold value 307. The threshold value 307 is the smallest sum value for any previously learnt concept or attribute. If a concept had been previously learnt and the sum value was X and a new attribute is about to be learnt with a sum value Y and Y is less than X then the threshold value 307 will be set to the value of Y. The result of the threshold test is output on signal 308 and input to the concept neuron recognition logic 305. The concept neuron Recognition Logic 305 outputs the recognition result on output 309, which is input to the concept neuron control and status module 302 and the concept neuron function logic 304. As shown in figure 11, the result of the recognition process is stored in the recognised register 3026 and subsequently logically OR-ed with input signal 312 using OR gate 3027. The concept neuron Recognition Logic 305 can implement recognition schemes of varying -30 -degrees of complexity. In its simplest form the input signal 308 is transferred to the output signal 309, either directly or in buffered form. In embodiments of the invention, more complex recognition schemes can be implemented, such as storing which rows where activated and read during the Synapse Processing phase, storing the count values for individual synapse rows or performing thresholding on individual synapse row count values and storing the corresponding bit patterns in memory. In the latter recognition scheme, the count value 503 is compared to a previously set threshold value 504 using comparator 505. The result 507 is input to the concept neuron recognition logic 305. Synapse row recognition counts 503 and or comparison values 507 are stored in the recognition memory 3051. Any suitable method can be used to implement a recognition memory 3051. For example, each recognition memory can be implemented using finite state machines (FSMs), correlation matrix memories (CMM), volatile memory, non-volatile memory, CAMs, Bloom Filters or a combination of these methods. These latter methods enable a concept neuron to be configured based on the mode of operation and the logic states of the pre synapse memory and post synapse memory of each selected synapse element.

Figure 11 shows a more detailed diagram of the concept neuron control & status logic 302. When in the Allocated state (register 3025 is set to logic 1), each time a concept neuron recognises a concept or associated attribute an internal counter, referred to as the occurrence counter 3023 is incremented by a predefined integer amount. In addition, a concept neuron's occurrence counter 3023 is decremented periodically by a predefined integer amount, usually one and less than or equal to the increment value. If the occurrence counter value is decremented and becomes zero, the corresponding concept neuron is put into the PTL state, resets the synapses 400 (using signals PRS RESET 501F and POS RESET 501G) in its connected synapse row group and outputs a signal 315 to indicate the state transition. This latter signal 315 is propagated to all the other concept neurons in the corresponding concept neuron array and if another concept neuron is in the PTL state then it is subsequently placed into the idle state ensuring that only one concept neuron is in the PTL state. If no other concept neuron is in the PTL state and the signal is returned to the originating concept neuron on input 314 then the learn status logic 3022 ignores the signal as it is already in the PTL slate.

The associative memory apparatus 100 can be used in a standalone configuration or combined with other associative memories 100 to form larger more complex associative memories. The -31 -associative memories 100 can be combined in many different ways to form different associative memory architectures. The associative memories 100 that are used to form a larger associative memory do not necessarily have to he identical. They can each perform the same functions, but have different input interfaces, processing and formatting modules, hash functions and synapse arrays for example. In such architectures the control logic 109 of each associative memory 100 operates autonomously based on the received input data In another embodiment, lower level associative memories 100 are controlled by the control logic 109 of a master associative memory 100. Figure 12 shows one such associative memory architecture. The architecture is hierarchical consisting of three levels. All the associative memories 100 are connected to an external processor 140 via a control/data bus 119. The control/data bus 119 allows the external processor 140 to communicate with each associative memory 100 in order to configure, set control parameters, read status registers and gain access to all their internal memories. In general, an associative memory at a particular level receives inputs from the next below it. For example, an associative memory 100 configured for operation at level of hierarchy 2 would receive input data from associative memories at level 1. However, this is not always the case and an output from an associative memory 100 at a hither level in the hierarchy can be fed back as an input to associative memories 100 that are configured at a lower level in the hierarchy. This is shown in figure 26. As an output from an associative memory 100 can be fed back to the input then this allows one or more associative memories 100 to implement finite state machines. Depending on the application and configuration each associative memory can be performing learning, recognition and generation operations in parallel and on different input datastreams. When configured in a multi-level hierarchy, the learning process can be considered a deep learning process.

In an embodiment of the invention, finite state machines (FSMs) of control logic 109 can be implemented using at least modules 101 to 106, wherein registered outputs from module 106 are input to module 101.

The processing and formatting module 102 can take one of several forms depending on the application. In an embodiment of the invention the processing and formatting module 102 comprises an array of comparators used to implement rules of the form; IT (input l<> input 2) and (input l<> input 2) or (input l<> input 2) then Concept A, where <> can be less than (<) or equal to (=) or greater than (>). The associative memory 100 can implement and perform rules where the antecedent has multiple parts and the consequence has one or more -32 -parts. In such an embodiment, the processing and formatting module has two or more inputs 111 and outputs a bit vector 112 than contains the results of each individual comparison. A comparator, individually and collectively referred to as 1021 and shown as 1021A -1021K in figure 13 has two inputs and implements the less than, equal to and greater than functions. The comparator results are output and stored in the corresponding bit locations of the output register 1022. The processing and formatting module 102 is controlled by the control logic module 109 using signals 122. The output vector will be hashed as before and considered an attribute of an associated concept. If several actions are associated with a certain rule then the different concepts would learn that particular attribute, that is, a one-to-many mapping.

In another embodiment of the invention the processing and formatting module 102 implements an array of fuzzy membership functions for implementing fuzzy rules and hence fuzzy systems. Figure 14 shows example fuzzy membership functions for each linguistic fuzzy set value (Large negative LN, Small Negative SN, Zero ZE. Small Positive SP and Large Positive LP) for an example input signal. Figure 15 shows a generic block diagram for the processing and formatting module 102 for implementing fuzzy membership functions. The processing and formatting module comprises Q membership function modules 1023, where Q is an integer and equal to the number of fuzzy linguistic terms. A membership function module, individually and collectively referred to as 1023 and shown as 1023A -1021M in figure 15 has an input 111 and two outputs, a Membership Function Valid (MFV) bit that is set if the input signal falls within in the limits of the membership function and a M-Bit Value output that represents the degree of membership and is a M-bit value, where M is an integer. The outputs from the membership function modules 1023 are stored in the output register 1024. The processing and formatting module 102 is controlled by the control logic module 109 using signals 122. The output vector will be hashed as before and considered an attribute of associated concept.

The associative memory 100 of the present invention can also be used to learn, recognise and generate data sequences, such as voice or music datastreams. In such embodiments, a concept neuron represents a concept in a sequence, such as a musical note, chord or a phoneme for example. The information stored in the synapses of a SRG for a concept neuron represents the results of generating the next state link using the current. and a plurality of previous concept states.

-33 -Temporal or sequence learning, recognition and generation requires knowledge of previous concepts of a sequence in order to generate a link to the next concept in the sequence. The analysis and associative linking of consecutive concepts is performed using the same hardware used for spatial learning and recognition. As such, the same hardware can be employed to perform temporal linking or separate parallel hardware can be employed for the task.

A link to select the next concept in a sequence is generated by combining selected values from a plurality of consecutive concept data in a sequence. To terminate a sequence an End of Sequence (EOS) concept is appended to the sequence data. On recognising an EOS concept sequence generation is terminated. A link to the next concept in a sequence is represented by one or more attribute data, that is, the processing and formatting module 102 takes input data and generates formatted data 112 that is of type attribute data. This is similar to the mode of operation for spatial data as previously explained: rather than one concept many concepts are used.

In such an embodiment of the invention the processing and formatting module 102 takes the form of a concept data combiner, where selected concept neuron identification data from a plurality of consecutive concept neuron identification data are combined using arithmetic and or logic functions to generate a vector (attribute data) that is then used by the encoder 130 to generate S synapse element addresses as before. Figure 16 shows a block diagram for the concept data combiner. Concept neuron identification data is input on bus 111 and stored in next concept register 1026. Concept neuron identification data can be generated by the same associative memory and feedback to the input or be generated by a separate associative memory. The output from the next concept register 1026 is input to the concept neuron pipeline register 1025. It is also used to select a concept neuron as the next concept in a sequence and for learning the associated attribute data. The concept neuron pipeline register 1025 serves as a W stage pipeline/shift register, where W is an integer. The concept neuron pipeline register 1025 stores the previous W concept neuron identification data in time order with register 1025A storing the oldest value. The concept neuron pipeline register 1025 provides a sliding window of W consecutive concept neuron identification data from a sequence. The concept neuron pipeline register 1025 comprises a plurality of concatenated concept neuron identification data registers, individually and collectively 1025 and shown as 1025A -1025W. Multiplexer 1027 is used to select concept neuron identification data from -34 -any of die concept neuron identification data registers 1025 and transfer it to the concept data combiner 1028. The concept data combiner control logic 1029 provides local control of the modules 1025, 1026, 1027 and 1028 and is itself controlled by the control logic module 109 via signals 122. The concept data combiner control logic 1029 selects outputs from concept neuron identification data registers 1025, transfers them to the concept data combiner 1028 via multiplexer 1027, where arithmetic and or logic operations are performed on the current input and the results of previous operations (registered) in order to generate attribute data. The attribute data is output on bus 112. The attribute data is then hashed S times to generate S synapse element addresses for the selected concept neuron, which will be the next concept in the sequence. The latter procedures are those previously described for learning attribute data, waveforms for which are shown in figure 2.

In order to recognise or generate a sequence the next concept in a sequence must be uniquely identified. As an attribute data may be recognised by more than one concept neuron, when learning a sequence more than one attribute data can be generated and learnt by a concept neuron. When searching for the next concept in a sequence, concept neurons can be tested using an increasing number of generated attribute data until only one concept neuron remains. If a concept neuron is associated with all the attributes (as shown in figure 4 for three attributes) then this concept neuron is considered the correct next concept in the sequence. The concept data combiner control logic 1029 performs this task and generates a plurality of attribute data using different combinations of concept neuron identification data registers and performing different arithmetic and or logic operations on the selected data. For consistency, the same selected concept neuron identification data registers and operations are performed for the same attribute in a group of generated attributes. For example, if three attributes are generated for a next concept then attribute 1 would use selection A and operation group A, attribute 2 would use selection B and operation group B and attribute 3 would use selection C and operation group C. In addition, the hashing process used by encoder 130 is different and employs a rolling hash, where the start and end of the hashing process is triggered when a predefined hash result occurs when processing a datastream or sequence. The rolling hashes generate trigger signal as part of control bus 123, which are used by the control logic 109 to coordinate the processing of sequence data.

-35 -Though the present invention has been described with reference to digital logic circuitry, those familiar with the art will realise that many aspects of the present invention can be implemented in analogue circuitry. For example, the pre synapse memory 402, the post synapse memory 404 and signal processing functions, such as filters, fuzzy variables and threshold functions related to the processing and formatting module 102 and concept neurons 300. Therefore, embodiments of the invention can be implemented in mixed signal or analogue device technologies that incorporate analogue circuitry for implementing analogue and or mixed signal associative memory modules and related sub-functions.

The present description illustrates the principles of the present invention. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described herein, embody the principles of the invention and are included within the spirit and scope.

Moreover, all statements herein reciting principles, aspects and embodiments of the invention, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents in the future, i.e., any elements developed that perform the same function, regardless of structure.

While the present invention has been illustrated and described in detail in the drawings and foregoing description, the same is to be considered as illustrative and not restrictive in character, it being understood that only exemplary embodiments have been shown and described and do not limit the scope of the invention in any manner. It can be appreciated that any of the features described herein may be used with any embodiment. The illustrative embodiments are not exclusive of each other or of other embodiment not recited herein. Accordingly, the invention also provides embodiments that comprise combinations of one or more of the illustrative embodiments described above. Modifications and variations of the invention as herein set forth can be made without departing from the spirit and scope thereof, and, therefore, only such limitations should be imposed as are indicated by the appended claims.

Claims (25)

1. -36 -CLAIMS1. Associative memory apparatus configured to perform data processing, the associative memory apparatus comprising: control logic circuitry configured to control associative memory modules and execute control tasks to perform different modes of operation on input data; wherein the associative memory synapse array module comprises a plurality of synapse elements organised into one or a plurality of synapse row groups, each synapse row group being controlled by an individual concept neuron of a connected associative memory concept neuron array module, wherein, S synapse elements of an active synapse row group are selected and configured in response to each formatted and encoded input data, where S is an integer and equal to the number of hash function pairs and the corresponding concept neuron is subsequently configured based on the mode of operation and the logic states of the pre synapse memory and post synapse memory of each selected synapse element.
2. 2. The associative memory of claim 1, wherein unselected synapse elements of an active synapse row group are put into bypass mode.
3. 3. The associative memory of claim 1, wherein a concept neuron that is in an Idle state is subsequently put into the Prepared To Learn (PTL) state in response to another concept neuron in a concept neuron array performing a state transition from Prepared To Learn state to an Allocated state.
4. 4. The associative memory of claim 1, wherein a concept neuron that is in the Allocated state increments the value of its occurrence counter by a predefined integer amount when it recognises a formatted and encoded input data, the occurrence counter being clamped to a maximum amount.
5. -37 - 5. The associative memory of claim 4. wherein a concept neuron that is in the Allocated state periodically decrements the value of its occurrence counter by a predefined integer amount and if the occurrence count value subsequently equals zero the concept neuron resets configured synapse elements in its connected synapse row group, puts the concept neuron into the Prepared To Learn (PTL) state and transmits the state transition to other concept neurons in the concept neuron at-ray.
6. 6. The associative memory of claim 1, wherein the encoding scheme for selecting synapse elements is performed using inter-hash encoding.
7. 7. The associative memory of claim 6, wherein a concept neuron reads each selected synapse dement in turn, maintains a running total of set synapse elements for each active synapse row, calculates the total number of set synapse elements and if the concept neuron is in the Prepared To Learn (PTL) state or learn mode, sets the threshold register to the total number value, resets the set pre synapse memories and then transitions to the Allocated state.
8. 8. The associative memory of claim 6, wherein a concept neuron reads each selected synapse element in turn, maintains a running total of set synapse elements for each selected synapse row, calculates the total number of set synapse elements and if the concept neuron is in the Allocated state and recognise mode, sets the recognised register to the true state and resets the pre synapse memories.
9. 9. The associative memory of claim 1, wherein the encoding scheme for selecting synapse elements is performed by sequentially hashing formatted data S times and selecting the addressed synapse elements, where S is an integer equal to the number hash function pairs.
10. 10. The associative memory of claim 9, wherein a concept neuron reads each active synapse row in parallel once all S synapse elements have been set, maintains a running total of set synapse elements for each synapse row, calculates the total number of set synapse elements and if the concept neuron is in the Prepared To Learn (PTL) state and learn mode sets the threshold register to the total number value, resets the set pre synapse memories and then transitions to the Allocated state.
11. -38 - 1 1. The associative memory of claim 9, wherein a concept neuron reads each active synapse row in parallel, maintains a running total of set synapse elements for each synapse row, calculates the total number of set synapse elements and if the concept neuron is in the Allocated state and recognise mode sets the recognised register to the true state and resets the pre synapse memories.
12. 12. The associative memory of any preceding claim, wherein a concept neuron further comprises a configurable logic function block that performs a selected logic function on consecutive analysis processing phases, where the input to the logic function are the current recognised state and the stored result of the previous analysis processing phase.
13. 13. The associative memory of any preceding claim, wherein a concept neuron further comprises concept neuron recognition logic, wherein the concept neuron recognition logic implements one of a plurality of recognition schemes, the plurality of recognition schemes consisting of storing which rows where activated and read during the Synapse Processing phase, storing the count values for individual synapse rows or performing thresholding on individual synapse row count values and storing the corresponding bit patterns in memory.
14. 14. The associative memory of any preceding claim, wherein a concept neuron is coupled with one or a plurality of consecutive locations in a concept memory, the concept memory storing label data associated with a particular concept neuron.
15. 15. The associative memory of any preceding claim, wherein label data associated with a concept neuron is processed and partitioned into a sequence of concepts and stored in an associative memory, each consecutive pair of concept from the partitioned sequence of concepts of the label data being linked with attribute data generated from selected concept neuron identification data using a concept data combiner.
16. 16. The associative memory of any preceding claim, wherein the associative memory processing and formatting module 102 comprises circuitry to perform signal processing algorithms on input data and generates formatted data, where the formatted data comprises any combination of concept data, attribute data and query data.-39 -
17. 17. The associative memory as claimed in claim 16, wherein the associative memory processing and formatting module 102 further comprises circuitry that can implement one of a plurality of different functions, the plurality of different functions consisting of an array of comparators for implementing rules, an array of fuzzy membership functions for implementing fuzzy rules or a concept data combiner of processing temporal data
18. 18. The associative memory as claimed in claim 17, wherein the concept data combiner of an associative memory processing and formatting module 102 further comprises logic to repeatedly generate different formatted data outputs from the same input in order to identify a single concept neuron as the next concept in a sequence.
19. 19. The associative memory of any preceding claim, wherein the control logic is distributed amongst the associative memory modules.
20. 20. The associative memory as claimed in claim 19, wherein the control logic operates autonomously in response to the input data.
21. 21. The associative memory as claimed in claim 20, wherein the control logic further comprises an External Interface Control Logic (EICL) interface for communicating with external processor means.
22. 22. The associative memory of any preceding claim, wherein a plurality of associative memory units (100) are interconnected using Network on Chip (NoC) means to form a plurality of associative memory architectures, the plurality of associative memory architectures including finite state machines, hierarchical, tiled arrays, torus, butterfly fat-tree, fractal and hypercube architectures.
23. 23. The associative memory of any preceding claim, wherein associative memory modules or associative memory module functions are implemented in analogue circuitry and the overall device is implemented in a mixed signal technology.
24. 24. The associative memory of any preceding claim, wherein one or a plurality of associative memory modules or associative memory module functions are implemented -40 -using one of a plurality of timing schemes, the plurality of timing schemes including synchronous timing schemes, asynchronous timing schemes or a combination of synchronous and asynchronous timing schemes.
25. 25. A non-transitory computer readable storage medium comprising computer readable data, wherein the computer readable data form part of an Electronic Design Automation (EDA) tool chain for use in a System-on-chip (SOC) design process or an Application Specific Integrated Circuit (ASIC) design process or a Field Programmable Gate Array (FPGA) design process wherein, the Electronic Design Automation (EDA) tool chain is employed to generate a representation of the associative memory apparatus.