EP3803699A1 - Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations - Google Patents

Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations

Info

Publication number
EP3803699A1
EP3803699A1 EP19728962.2A EP19728962A EP3803699A1 EP 3803699 A1 EP3803699 A1 EP 3803699A1 EP 19728962 A EP19728962 A EP 19728962A EP 3803699 A1 EP3803699 A1 EP 3803699A1
Authority
EP
European Patent Office
Prior art keywords
neural network
activity
patterns
artificial neural
input
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP19728962.2A
Other languages
German (de)
English (en)
Inventor
Henry Markram
Ran Levi
Kathryn Pamela Hess Bellwald
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
INAIT SA
Original Assignee
INAIT SA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/004,635 external-priority patent/US20190378007A1/en
Priority claimed from US16/004,757 external-priority patent/US11893471B2/en
Priority claimed from US16/004,671 external-priority patent/US11972343B2/en
Priority claimed from US16/004,837 external-priority patent/US11663478B2/en
Priority claimed from US16/004,796 external-priority patent/US20190378000A1/en
Application filed by INAIT SA filed Critical INAIT SA
Publication of EP3803699A1 publication Critical patent/EP3803699A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/008—Artificial life, i.e. computing arrangements simulating life based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/01—Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound

Definitions

  • This specification relates to the characterization of activity in a recurrent artificial neural network.
  • the characterization of activity can be used, e.g., in the identification of decision moments, as well as in encoding/decoding signals in contexts such as transmission, encryption, and data storage. It also relates to encoding and decoding information, and to systems and techniques that use the encoded information in various contexts.
  • the encoded information can represent the activity in a neural network, e.g., a recurrent neural network.
  • Artificial neural networks are devices that are inspired by the structure and functional aspects of networks of biological neurons.
  • artificial neural networks mimic the information encoding and other processing capabilities of networks of biological neurons using a system of interconnected constructs called nodes.
  • the arrangement and strength of connections between nodes in an artificial neural network determines the results of information processing or information storage by the artificial neural network.
  • Neural networks can be trained to produce a desired signal flow within the network and achieve desired information processing or information storage results. In general, training a neural network will change the arrangement and/or strength of connections between nodes during a learning phase. A neural network can be considered trained when sufficiently appropriate processing results are achieved by the neural network for given sets of inputs.
  • Non-linear data processing does not satisfy the superposition principle, i.e., the variables that are to be determined cannot be written as a linear sum of independent components.
  • Examples of contexts in which non-linear data processing is useful include pattern and sequence recognition, speech processing, novelty detection and sequential decision making, complex system modelling, and systems and techniques in a variety of other contexts. Encoding and decoding both convert information from one form or representation into another.
  • the different representations can provide different features that are more or less useful in different applications. For example, some forms or representations of information (e.g., natural language) may be easier for a human to understand. Other forms or representations may be smaller in size (e.g.,“compressed”) and easier to transmit or store. Still other forms or representations may intentionally obscure the information content (e.g., information can be cryptographically encoded).
  • an encoding or decoding process will generally follow a predefined set of rules or an algorithm that establishes the correspondence between the information in the different forms or representations.
  • encoding process that yields a binary code may assign a role or meaning to individual bits according to their position in a binary sequence or vector.
  • This specification describes technologies relating to the characterization of activity in an artificial neural network.
  • a method can include characterizing activity in an artificial neural network.
  • the method is performed by data processing apparatus and can include identifying clique patterns of activity of the artificial neural network.
  • the clique patterns of activity can enclose cavities.
  • the method can include defining a plurality of windows of time during which the activity of the artificial neural network is responsive to an input into the artificial neural network.
  • the clique patterns of activity can be identified in each of the pluralities of windows of time.
  • the method can include identifying a first window of time within the plurality of windows of time based on a distinguishable likelihood of the clique patterns of activity occurring during the first window. Identifying clique patterns can include identifying directed cliques of activity. Tower dimensional directed cliques that are present in higher dimensional directed cliques can be discarded or ignored.
  • the method can include classifying the clique patterns into categories and
  • Classifying the clique patterns can include classifying the clique patterns according to a number of points within each clique pattern.
  • the method can include outputting a binary sequence of zeros and ones from the recurrent artificial neural network. Each digit in the sequence can represent whether or not a respective pattern of activity is present in the artificial neural network.
  • the method can include structuring the artificial neural network by reading the digits output from the artificial neural network, and evolving the structure of the artificial neural network.
  • the structure of the artificial neural network can be evolved by iteratively changing the structure, characterizing the complexity of patterns of activity in the changed structure, and using the characterization of the complexity of the pattern as an indication of whether the changed structure is desirable.
  • the artificial neural network can be a recurrent artificial neural network.
  • the method can include identifying decision moments in the recurrent artificial neural network based on the determination of the complexity of patterns of activity in the recurrent artificial neural network.
  • the identification of decision moments can include determining a timing of activity having a complexity that is distinguishable from other activity that is responsive to the input and identifying the decision moments based on the timing of the activity that has the distinguishable complexity.
  • the method can include inputting a data stream into the recurrent artificial neural network and identifying the clique patterns of activity during the input of the data stream.
  • the method can include estimating whether the activity is responsive to the input into the artificial neural network.
  • the estimating can include estimating that relatively simpler patterns of activity relatively soon after the input event are responsive the input but that relatively more complex patterns of activity relatively soon after the input event are not responsive the input and estimating that relatively more complex patterns of activity relatively later after the input event are responsive the input but that relatively simpler patterns of activity relatively later after the input event are not responsive the input.
  • a system can include one or more computers operable to perform operations.
  • the operations can include characterizing activity in an artificial neural network and comprise identifying clique patterns of activity of the artificial neural network, wherein the clique patterns of activity enclose cavities.
  • the operations can include defining a plurality of windows of time during which the activity of the artificial neural network is responsive to an input into the artificial neural network.
  • the clique patterns of activity can be identified in each of the pluralities of windows of time.
  • the operations can include identifying a first window of time within the plurality of windows of time based on a distinguishable likelihood of the clique patterns of activity occurring during the first window. Identifying clique patterns can include discarding or ignoring lower dimensional directed cliques that are present in higher dimensional directed cliques.
  • the operations can include structuring the artificial neural network, including reading the digits output from the artificial neural network and evolving the structure of the artificial neural network.
  • the structure of the artificial neural network can be evolved by iteratively changing the structure,
  • the artificial neural network can be a recurrent artificial neural network.
  • the operations can include identifying decision moments in the recurrent artificial neural network based on the determination of the complexity of patterns of activity in the recurrent artificial neural network the identification of decision moments can include determining a timing of activity having a complexity that is distinguishable from other activity that is responsive to the input, and identifying the decision moments based on the timing of the activity that has the distinguishable complexity.
  • the operations can include inputting a data stream into the recurrent artificial neural network and identifying the clique patterns of activity during the input of the data stream.
  • the operations can include estimating whether the activity is responsive to the input into the artificial neural network.
  • the estimating can include estimating that relatively simpler patterns of activity relatively soon after the input event are responsive the input but that relatively more complex patterns of activity relatively soon after the input event are not responsive the input and estimating that relatively more complex patterns of activity relatively later after the input event are responsive the input but that relatively simpler patterns of activity relatively later after the input event are not responsive the input.
  • a method for identifying decision moments in a neural network includes determining a complexity of patterns of activity in the recurrent artificial neural network, wherein the activity is responsive to input into the recurrent artificial neural network, determining a timing of activity having a complexity that is distinguishable from other activity that is responsive to the input, and identifying the decision moments based on the timing of the activity that has the distinguishable complexity.
  • a method for characterizing activity in a recurrent artificial neural network includes identifying pre-defined clique patterns of activity of the recurrent artificial neural network. The method is performed by data processing apparatus.
  • a method can include outputting a binary sequence of zeros and ones from a recurrent artificial neural network, wherein each digit in the sequence represents whether or not a particular group of nodes in the recurrent artificial neural network displays a respective pattern of activity.
  • a method of structuring a recurrent artificial neural network can include characterizing a complexity of patterns of activity that can arise in the recurrent artificial neural network that comprises a structured collection of nodes and links between the nodes, and evolving the structure of the recurrent artificial neural network to increase the complexity of the patterns of activity.
  • This method of structuring can also be used, e.g., a part of a method of training the recurrent artificial neural network.
  • decision moments in artificial recurrent neural networks can be identified based on characteristics of the dynamic properties of the neural network during information processing. Rather than waiting for the artificial neural network to reach a pre-defined end of a logic sequence, decision moments in artificial neural networks can be identified based on characteristics of the functional states of the artificial neural network in the midst of information processing.
  • the characteristics of the dynamic properties of recurrent artificial neural networks during information processing can be used in a variety of signalling operations, including signal transmission, encoding, encryption, and storage.
  • the characteristics of the activity in a recurrent artificial neural network during information processing reflects the input and can be considered to be an encoded form of the input (i.e., the“output” of a recurrent artificial neural network in encoding processes).
  • These characteristics can be, e.g., transmitted to a remote receiver which could decode the transmitted characteristics to reconstitute the input or a portion of the input.
  • activity in different groups of nodes a recurrent artificial neural network can be represented as a binary sequence of zeros and ones, each digit indicating whether or not that activity comports with a pattern. Since the activity can be the output of a recurrent artificial neural network in some contexts, the output of a recurrent artificial neural network can be represented as a vector of binary digits and compatible with digital data processing.
  • such characterization of the dynamic properties of a recurrent artificial neural network can be used prior to and/or during training to increase the likelihood of complex patterns of activity arising during information processing.
  • the links between nodes in a recurrent neural network can be intentionally evolved to increase the complexity of activity patterns.
  • the links between nodes in a recurrent artificial neural network can be intentionally evolved to increase the likelihood that, e.g., clique patterns and directed clique patterns of activity arise during information processing. This can reduce the time and effort needed to train a recurrent artificial neural network.
  • such characterization of the dynamic properties of a recurrent artificial neural network can be used to determine a degree of completeness in the training of a recurrent neural network.
  • a recurrent artificial neural network that displays particular types of ordering in the activity e. g, clique patterns and directed clique patterns
  • the extent of training can be quantified by quantifying the degree of ordering of the activity in the recurrent artificial neural network.
  • a method for identifying decision moments in a neural network includes determining a complexity of patterns of activity in the recurrent artificial neural network, wherein the activity is responsive to input into the recurrent artificial neural network, determining a timing of activity having a complexity that is distinguishable from other activity that is responsive to the input, and identifying the decision moments based on the timing of the activity that has the distinguishable complexity.
  • a method for characterizing activity in a recurrent artificial neural network includes identifying clique patterns of activity of the recurrent artificial neural network. The method is performed by data processing apparatus.
  • a method can include outputting a binary sequence of zeros and ones from a recurrent artificial neural network, wherein each digit in the sequence represents whether or not a particular group of nodes in the recurrent artificial neural network displays a respective pattern of activity.
  • a method of structuring a recurrent artificial neural network can include characterizing a complexity of patterns of activity that can arise in the recurrent artificial neural network that comprises a structured collection of nodes and links between the nodes, and evolving the structure of the recurrent artificial neural network to increase the complexity of the patterns of activity.
  • This method of structuring can also be used, e.g., a part of a method of training the recurrent artificial neural network.
  • Other embodiments of these aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.
  • decision moments in artificial recurrent neural networks can be identified based on characteristics of the dynamic properties of the neural network during information processing. Rather than waiting for the artificial neural network to reach a pre-defined end of a logic sequence, decision moments in artificial neural networks can be identified based on characteristics of the functional states of the artificial neural network in the midst of information processing.
  • the characteristics of the dynamic properties of recurrent artificial neural networks during information processing can be used in a variety of signalling operations, including signal transmission, encoding, encryption, and storage.
  • the characteristics of the activity in a recurrent artificial neural network during information processing reflects the input and can be considered to be an encoded form of the input (i.e., the“output” of a recurrent artificial neural network in encoding processes).
  • These characteristics can be, e.g., transmitted to a remote receiver which could decode the transmitted characteristics to reconstitute the input or a portion of the input.
  • activity in different groups of nodes a recurrent artificial neural network can be represented as a binary sequence of zeros and ones, each digit indicating whether or not that activity comports with a pattern. Since the activity can be the output of a recurrent artificial neural network in some contexts, the output of a recurrent artificial neural network can be represented as a vector of binary digits and compatible with digital data processing.
  • such characterization of the dynamic properties of a recurrent artificial neural network can be used prior to and/or during training to increase the likelihood of complex patterns of activity arising during information processing.
  • the links between nodes in a recurrent neural network can be intentionally evolved to increase the complexity of activity patterns.
  • the links between nodes in a recurrent artificial neural network can be intentionally evolved to increase the likelihood that, e.g., clique patterns and directed clique patterns of activity arise during information processing. This can reduce the time and effort needed to train a recurrent artificial neural network.
  • such characterization of the dynamic properties of a recurrent artificial neural network can be used to determine a degree of completeness in the training of a recurrent neural network.
  • a recurrent artificial neural network that displays particular types of ordering in the activity e. g, clique patterns and directed clique patterns
  • the extent of training can be quantified by quantifying the degree of ordering of the activity in the recurrent artificial neural network.
  • a device includes a neural network trained to produce, in response to a first input, an approximation of a first representation of topological structures in patterns of activity arising in a source neural network in response to the first input, in response to a second input, an approximation of a second representation of topological structures in patterns of activity arising in the source neural network in response to the second input, and, in response to a third input, an approximation of a third
  • the topological structures can all encompass two or more nodes in the source neural network and one or more edges between the nodes.
  • the topological structures can include simplices.
  • the topological structures can enclose cavities.
  • Each of the first representation, the second representation, and the third representation can represent topological structures that arise in the source neural network exclusively at times during which the patterns of activity have a complexity that is distinguishable from the complexity of other activity that is responsive to the respective of the inputs.
  • the device can further include a processor coupled to receive the approximations of the representations produced by the neural network device and process the received approximations.
  • the processor can include e a second neural network that has been trained to process representations produced by the neural network.
  • Each of the first representation, the second representation, and the third representation can include multi- valued, non-binary digits.
  • Each of the first representation, the second representation, and the third representation can represent the occurrence of the topological structures without specifying where the patterns of activity arise in the source neural network.
  • the device can include a smart phone.
  • the source neural network can be a recurrent neural network.
  • a device in another implementation, includes a neural network coupled to input representations of topological structures in patterns of activity arising in a source neural network in response to a plurality of different inputs.
  • the neural network is trained to process the representations and produce a responsive output.
  • the topological structures can all encompass two or more nodes in the source neural network and one or more edges between the nodes.
  • the topological structures can include simplices.
  • the representations of topological structures can represent topological structures that arise in the source neural network exclusively at times during which the patterns of activity have a complexity that is distinguishable from the complexity of other activity that is responsive to the respective of the inputs.
  • the device can include a neural network trained to produce, in response to a plurality of different inputs, respective approximations of the representations of topological structures in the patterns of activity arising in the source neural network in response to the different inputs.
  • the representations of topological structures can include multi-valued, non-binary digits.
  • the representations of topological structures can represent the occurrence of the topological structures without specifying where the patterns of activity arise in the source neural network.
  • the source neural network can be a recurrent neural network.
  • a method is implemented by a neural network device and includes inputting a representation of topological structures in patterns of activity in a source neural network, wherein the activity is responsive to an input into the source neural network, processing the representation, and outputting a result of the processing of the representation.
  • the processing is consistent with a training of the neural network to process different such representations of topological structures in patterns of activity in the source neural network.
  • the topological structures can all encompass two or more nodes in the source neural network and one or more edges between the nodes.
  • the topological structures can include simplices.
  • the topological structures can enclose cavities.
  • the representations of topological structures can represent topological structures that arise in the source neural network exclusively at times during which the patterns of activity have a complexity that is distinguishable from the complexity of other activity that is responsive to the respective of the inputs.
  • the representations of topological structures can include multi-valued, non-binary digits.
  • the representations of topological structures can represent the occurrence of the topological structures without specifying where the patterns of activity arise in the source neural network.
  • the source neural network can be a recurrent neural network.
  • a device in one implementation, includes a neural network coupled to input representations of topological structures in patterns of activity arising in a source neural network in response to a plurality of different inputs.
  • the neural network is trained to process the representations and produce a responsive output.
  • the topological structures all encompass two or more nodes in the source neural network and one or more edges between the nodes.
  • the device can include an actuator coupled to receive the responsive output from the neural network and act upon a real or virtual environment, a sensor coupled to measure a characteristic of the environment, and a teacher module configured to interpret the measurements received from the sensor and provide a reward and/or a regret to the neural network.
  • the topological structures can include simplices.
  • the topological structures can enclose cavities.
  • the representations of topological structures can represent topological structures that arise in the source neural network exclusively at times during which the patterns of activity have a complexity that is distinguishable from the complexity of other activity that is responsive to the respective of the inputs.
  • the device can include a second neural network trained to produce, in response to a plurality of different inputs, respective approximations of the representations of topological structures in the patterns of activity arising in the source neural network in response to the different inputs.
  • a device can also include an actuator coupled to receive the responsive output from the neural network and act upon a real or virtual environment and a sensor coupled to measure a characteristic of the environment.
  • the second neural network can be trained to produce the respective approximations at least in part in response to the measured characteristic of the environment.
  • the device can also include a teacher module configured to interpret the measurements received from the sensor and provide a reward and/or a regret to the neural network.
  • the representations of topological structures can include multi-valued, non-binary digits.
  • the representations of topological structures can represent the occurrence of the topological structures without specifying where the patterns of activity arise in the source neural network.
  • the device can be a smart phone.
  • the source neural network can be a recurrent neural network.
  • a method that is implemented by one or more data processing devices can include receiving a training set that includes a plurality of representations of topological structures in patterns of activity in a source neural network and training a neural network using the representations either as an input to the neural network or as a target answer vector.
  • the activity is responsive to an input into the source neural network.
  • the topological structures all encompass two or more nodes in the source neural network and one or more edges between the nodes.
  • the training set can include a plurality of input vectors each corresponding to a respective of the representations.
  • Training the neural network can include training the neural network using each of the plurality of representations as a target answer vector.
  • Training the neural network can include training the neural network using each of the plurality of representations as an input.
  • the training set can include a plurality of reward or regret values.
  • Training the neural network can include reinforcement learning.
  • the topological structures can include simplices.
  • representations of topological structures can represent topological structures that arise in the source neural network exclusively at times during which the patterns of activity have a complexity that is distinguishable from the complexity of other activity that is responsive to the respective of the inputs.
  • the representations of topological structures can include multi valued, non-binary digits.
  • the representations of topological structures can represent the occurrence of the topological structures without specifying where the patterns of activity arise in the source neural network.
  • the source neural network can be a recurrent neural network.
  • FIG. 1 is a schematic illustration of the structure of recurrent artificial neural network device.
  • FIGS. 2 and 3 are schematic illustrations of the function of recurrent artificial neural network device in different windows of time.
  • FIG. 4 is a flowchart of a process for identifying decision moments in a recurrent artificial neural network based on characterization of the activity in the network.
  • FIG. 5 is a schematic illustration of patterns of activity that can be identified and used for identifying decision moments in a recurrent artificial neural network.
  • FIG. 6 is a schematic illustration of patterns of activity that can be identified and used for identifying decision moments in a recurrent artificial neural network.
  • FIG. 7 is a schematic illustration of patterns of activity that can be identified and used for identifying decision moments in a recurrent artificial neural network.
  • FIG. 8 is a schematic illustration of a data table that can be used in a determination of the complexity or degree of ordering in the activity patterns in a recurrent artificial neural network device.
  • FIG. 9 is a schematic illustration of a determination of the timing of activity patterns that have a distinguishable complexity.
  • FIG. 10 is a flowchart of a process for encoding signals using a recurrent artificial neural network based on characterization of the activity in the network.
  • FIG. 11 is a flowchart of a process for decoding signals using a recurrent artificial neural network based on characterization of the activity in the network.
  • FIGS. 12, 13, and 14 are schematic illustrations of an binary form or representation of topological structures.
  • FIGS. 15 and 16 schematically illustrate an example of how the presence or absence of features that correspond to different bits are not independent of one another.
  • FIGS. 17, 18, 19, 20 are schematic illustrations of the use of representations of the occurrence of topological structures in the activity in a neural network in four different classification systems.
  • FIGS. 21, 22 are schematic illustration of edge devices that include a local artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
  • FIG. 23 is a schematic representation of a system in which local neural networks can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
  • FIGS. 24, 25, 26, 27 are schematic illustrations of the use of representations of the occurrence of topological structures in the activity in a neural network in four different systems.
  • FIG. 28 is a schematic illustration of a system 0 that includes an artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
  • FIG. 1 is a schematic illustration of the structure of a recurrent artificial neural network device 100.
  • Recurrent artificial neural network device 100 is a device that mimics the information encoding and other processing capabilities of networks of biological neurons using a system of interconnected nodes.
  • Recurrent artificial neural network device 100 can be implemented in hardware, in software, or in combinations thereof.
  • the illustration of recurrent artificial neural network device 100 includes a plurality of nodes 101, 102, ... , 107 that are interconnected by a plurality of structural links 110.
  • Nodes 101, 102, ... , 107 are discrete information processing constructs that are analogous to neurons in biological networks.
  • Nodes 101, 102, ... , 107 generally process one or more input signals received over one or more of links 110 to produce one or more output signals that are output over one or more of links 110.
  • nodes 101, 102, ... , 107 can be artificial neurons that weight and sum multiple input signals, pass the sum through one or more non-linear activation functions, and output one or more output signals.
  • Nodes 101, 102, ... , 107 can operate as accumulators.
  • nodes 101, 102, ... , 107 can operate in accordance with an integrate-and-fire model in which one or more signals accumulate in a first node until a threshold is reached. After the threshold is reached, the first node fires by transmitting an output signal to a connected, second node along one or more of links 110.
  • the second node 101, 102, ... , 107 accumulates the received signal and, if a threshold is reached, then the second node 101, 102, ... , 107 transmits yet another output signal to a further connected node.
  • Structural links 110 are connections that are capable of transmitting signals between nodes 101, 102, ... , 107.
  • all structural links 110 are treated herein as identical bidirectional links that convey a signal from every first of nodes 101, 102, ... , 107 to every second of nodes 101, 102, ... , 107 in identically the same manner as a signal is conveyed from the second to the first.
  • this is not necessarily the case.
  • structural links 110 can be unidirectional links that convey a signal from a first of nodes 101, 102, ... , 107 to a second of nodes 101, 102, ... , 107 without conveying signals from the second to the first.
  • structural links 110 can have diverse properties other than or in addition to directionality.
  • different structural links 110 can carry signals of different magnitudes— resulting in a different strengths of interconnection between respective of nodes 101, 102, ... , 107.
  • different structural links 110 can carry different types of signal (e.g., inhibitory and/or excitatory signals).
  • structural links 110 can be modelled on the links between soma in biological systems and reflect at least a portion of the enormous morphological, chemical, and other diversity of such links.
  • recurrent artificial neural network device 100 is a clique network (or subnetwork) in that every node 101, 102, ... , 107 is connected to every other node 101, 102, ... , 107. This is not necessarily the case. Rather, in some
  • each node 101, 102, ... , 107 can be connected to a proper subset of nodes 101, 102, ... , 107 (by identical links or diverse links, as the case may be).
  • recurrent artificial neural network device 100 is illustrated with only seven nodes.
  • real-world neural network devices will include significantly larger numbers of nodes.
  • neural network devices can include hundreds of thousands, millions, or even billions of nodes.
  • recurrent neural network device 100 can be a fraction of a larger recurrent artificial neural network (i.e., a subnetwork).
  • accumulation and signal transmission processes require the passage of time in the real world.
  • soma of a neuron integrates input received over time
  • signal transmission from neuron to neuron requires times that are determined by, e.g., the signal transmission velocity and the nature and length of the links between neurons.
  • the state of a biological neural network device is dynamic and changes over time.
  • time is artificial and represented using mathematical constructs. For example, rather than requiring a real world passage of time for signals to transmit from node to node, such signals can be represented in terms of artificial units that are generally unrelated to the real world passage of time— as measured in computer clock cycles or otherwise. Nevertheless, the state of an artificial recurrent neural network device can be described as“dynamic” in that it changes with respect to these artificial units.
  • time units are referred to herein as“time” units. Nevertheless, it is to be understood that these units are artificial and generally do not correspond to the real world passage of time.
  • FIGS. 2 and 3 are schematic illustrations of the function of recurrent artificial neural network device 100 in different windows of time. Because the state of device 100 is dynamic, the functioning of device 100 can be represented using the signal transmission activity that occurs within a window. Such a functional illustration generally shows activity in only a fraction of links 110. In particular, since in general not every link 110 conveys a signal within a particular window, not every link 110 is illustrated as actively contributing to the functioning of the device 100 in these illustrations.
  • an active link 110 is illustrated as a relatively thick solid line connecting a pair of nodes 101, 102, ... , 107.
  • inactive links 110 are illustrated as dashed lines. This is for the sake of illustration only. In other words, the structural connections formed by links 110 exist whether or not links 110 are active.
  • the direction of the activity is also schematically illustrated.
  • the relatively thick solid lines that illustrate active of links 110 also include arrow heads that denote the direction of signal transmission along the link during the relevant window.
  • the direction of signal transmission in a single window does not conclusively constrain the link to being a unidirectional link having the indicated directionality.
  • a link in a first functional illustration for a first window of time, a link can be active in a first direction.
  • a link can be active in the opposite direction.
  • the directionality of signal transmission will conclusively indicate the directionality of the link.
  • feedforward neural network devices In feedforward neural network devices, information moves exclusively in a single direction (i.e., forward) to an output layer of nodes that is at the end of the network. Feedforward neural network devices indicate that a“decision” has been reached and that information processing is complete by the propagation of the signals through the network to the output layer.
  • the connections between nodes form cycles and the activity of the network dynamically progresses without a readily identifiable decision.
  • the first node can transmit a signal to the second node, which in response can transmit a signal to the third.
  • the third node can transmit a signal back to the first.
  • the signals received by the first node can be responsive— at least in part— to the signals transmitted from that same node.
  • FIGS. 2 and 3 illustrate this in a network that is only slightly larger than a three-node recurrent neural network.
  • the functional illustrations shown in FIG. 2 can be illustrative of activity within a first window and FIG. 3 can be illustrative of activity within a second, immediately following.
  • a collection of signal transmission activity appears to originate in node 104 and progress in a generally clockwise direction through device 100 during the first window.
  • the second window at least some of the signal transmission activity generally appears to return to node 104. Even in such a simplistic illustration, signal transmission does not proceed in a manner that yields a clearly identifiable output or end.
  • a recurrent neural network of, e.g., thousands of nodes or more
  • signal propagation can occur over a huge number of paths and that these signals lack a clearly identifiable“output” location or time.
  • the network may by design return to a quiescent state in which only background or even no signal transmission activity occurs, the quiescent state itself does not indicate the results of information processing.
  • the recurrent neural network always returns to the quiescent state regardless of the input.
  • The“output” or the result of the information processing is thus encoded in the activity that occurs within the recurrent neural network in response to a particular input.
  • FIG. 4 is a flowchart of a process 400 for identifying decision moments in a recurrent artificial neural network based on characterization of the activity in the network.
  • a decision moment is a point of time when the activity in a recurrent artificial neural network is indicative of the results of information processing by the network in response to an input.
  • Process 400 can be performed by a system of one or more data processing apparatus that perform operations in accordance with the logic of one or more sets of machine-readable instructions.
  • process 400 can be performed by that same system of one or more computers that executes software for implementing the recurrent artificial neural network used in process 400.
  • the system performing process 400 receives a notification that a signal has been input into the recurrent artificial neural network at 405.
  • the input of the signal is a discrete injection event in which, e.g., information is injected into one or more nodes and/or one or more links of the neural network.
  • the input of the signal is a stream of information that is injected into the one or more nodes and/or links of the neural network over a period of time.
  • the notification indicates that the artificial neural network is actively processing information and not, e.g., in a quiescent state.
  • the notification is received from the neural network itself, e.g., such as when the neural network exits an identifiable quiescent state.
  • the system performing process 400 divides the responsive activity in the network into a collection of windows at 410.
  • the windows can subdivide the time between injection and a return to a quiescent state into a number of periods during which the activity displays variable complexities.
  • the duration of the injection (and optionally the time to return to a quiescent state after injection is complete) can be subdivided into windows during which the activity displays variable complexities.
  • the windows all have the same duration, but this is not necessarily the case. Rather, in some implementations, the windows can have different durations. For example, in some implementations, duration can increase as time since a discrete injection event has occurred increases.
  • the windows can be a successive series of separate windows. In other implementations, the windows overlap in time so that one window begins before a previous window ends. In some cases, the windows can be a moving window that moves in time.
  • different durations of windows are defined for different determinations of the complexity of activity. For example, for activity patterns that define activity occurring between relatively larger numbers of nodes, the windows can have a relatively longer duration than windows that are defined for activity patterns that define activity occurring between relatively smaller numbers of nodes. For example, in the context of patterns 500 of activity (FIG. 5), a window that is defined for identifying activity that comports with pattern 530 can be longer than a window that is defined for identifying activity that comports with pattern 505.
  • the system performing process 400 identifies patterns in the activity in the network in the different windows at 415.
  • patterns in the activity can be identified by treating a functional graph as a topological space with nodes as points.
  • the activity patterns that are identified are cliques, e.g., directed cliques, in a functional graph of the network.
  • the system performing process 400 determines the complexity of the activity patterns in different windows at 420.
  • the complexity can be a measure of the likelihood that an ordered pattern of activity arises within a window.
  • activity patterns that arise randomly would be relatively simple.
  • activity patterns that show non- random order are relatively complex.
  • the complexity of an activity pattern can be measured using, e.g., the simplex counts or the Betti numbers of the activity pattern.
  • the system performing process 400 determines the timing of activity patterns having a distinguishable complexity at 425.
  • a particular activity pattern can be distinguishable based on a complexity that deviates upwards or deviates downward, e.g., from a fixed or a variable baseline.
  • the timing of activity patterns that indicate particularly high levels or particularly low levels of non-random order in the activity can be determined.
  • signal input is a discrete injection event
  • deviations e.g., from a stable baseline or from a curve that is characteristic of the neural network’s average response to a variety of different discrete injection events can be used to determine the timing of distinguishable complex activity patterns.
  • deviations e.g., from a stable baseline or from a curve that is characteristic of the neural network’s average response to a variety of different discrete injection events can be used to determine the timing of distinguishable complex activity patterns.
  • the system performing process 400 times the reading of the output from the neural network based on the timing of distinguishably complex activity patterns at 430.
  • the output of the neural network can be read at the same time that distinguishable complex activity patterns arise.
  • the observed activity patterns themselves can also be taken as the output of the recurrent artificial neural network.
  • FIG. 5 is an illustration of patterns 500 of activity that can be identified and used for identifying decision moments in a recurrent artificial neural network.
  • patterns 500 can be identified at 415 in process 400 (FIG. 4).
  • Patterns 500 are illustrations of activity within a recurrent artificial neural network.
  • a functional graph is treated as a topological space with nodes as points.
  • Activity in nodes and links that comports with patterns 500 can be recognized as ordered regardless of the identity of the particular nodes and/or links that participate in the activity.
  • first pattern 505 can represent the activity between nodes 101, 104, 105 in FIG. 2, with point 0 in pattern 505 as node 104, point 1 as node 105, and point 2 as node 101.
  • first pattern 505 can also represent the activity between nodes 104, 105, 106 in FIG.
  • patterns 500 are all directed cliques or directed simplices. In such patterns, activity originates from a source node that transmits signals to every other node in the pattern. In patterns 500, such source nodes are designated as point 0 whereas the other nodes are designated as points 1, 2, .... Further, in directed cliques or simplices, one of the nodes acts a sink and receives signals transmitted from every other node in the pattern. In patterns 500, such sink nodes are designated as the highest numbered point in the pattern. For example, in pattern 505, the sink node is designated as point 2. In pattern 510, the sink node is designated as point 3. In pattern 515, the sink node is designated as point 3, and so on. The activity represented by patterns 500 is thus ordered in a
  • Each of patterns 500 has a different number of points and reflects ordered activity in a different number of nodes.
  • pattern 505 is a 2D-simplex and reflects activity in three nodes
  • pattern 510 is a 3D-simplex and reflects activity in four nodes, and so on.
  • the degree of ordering and the complexity of the activity For example, for a large collection of nodes that have a certain level of random activity within a window, some of that activity may comport with pattern 505 out of happenstance. However, it is progressively more unlikely that random activity will comport with the respective of patterns 510, 515, 520....
  • the presence of activity that comports with pattern 530 indicates a relatively higher degree of ordering and complexity in the activity that the presence of activity that comports with pattern 505.
  • duration windows can be defined for different determinations of the complexity of activity. For example, when activity that comports with pattern 530 is to be identified, longer duration windows can be used than when activity that comports with pattern 505 is to be identified.
  • FIG. 6 is an illustration of patterns 600 of activity that can be identified and used for identifying decision moments in a recurrent artificial neural network.
  • patterns 600 can be identified at 415 in process 400 (FIG. 4).
  • patterns 600 are illustrations of activity within a recurrent artificial neural network. However, patterns 600 depart from the strict ordering of patterns 500 in that patterns 600 are not all directed cliques or directed simplices. In particular, patterns 605, 610 have a lower directionality than pattern 515. Indeed, pattern 605 lacks a sink node altogether. Nevertheless, patterns 605, 610 indicate degree of ordered activity that exceeds that expected through random happenstance and can be used to determine the complexity of activity in a recurrent artificial neural network.
  • FIG. 7 is an illustration of patterns 700 of activity that can be identified and used for identifying decision moments in a recurrent artificial neural network. For example, patterns 700 can be identified at 415 in process 400 (FIG. 4).
  • Patterns 700 are groups of directed cliques or directed simplices of the same dimension (i.e., have the same number of points) that define patterns involving more points than the individual cliques or simplices and enclose cavities within the group of directed simplices.
  • pattern 705 includes six different three point, 2-dimensions patterns 505 that together define a homology class of degree two
  • pattern 710 includes eight different three point, 2-dimensions patterns 505 that together define a second homology class of degree two.
  • Each of the three point, 2-dimensions patterns 505 in patterns 705, 710 can be thought of as enclosing a respective cavity.
  • the n lh Betti number associated with a directed graph provides a count of such homology classes within a topological representation.
  • patterns 700 illustrates a relatively high degree of ordering of the activity within a network that is unlikely to arise by random happenstance. Patterns 700 can be used to characterize the complexity of that activity.
  • the patterns of activity that are identified are discarded or otherwise ignored during the identification of decision moments.
  • activity that comports with the five point, 4-dimensional simplex pattern 515 inherently includes activity that comports with the four point, 3 -dimensional and three point, 2-dimension simplex patterns 510, 505.
  • points 0, 2, 3, 4 and points 1, 2, 3, 4 in 4-dimensional simplex pattern 515 of FIG. 5 both comport with 3-dimensional simplex pattern 510.
  • patterns that include fewer points— and hence are of a lower dimension— can be discarded or otherwise ignored during the identification of decision moments.
  • FIG. 8 is a schematic illustration of a data table 800 that can be used in such a determination.
  • Data table 800 can be used to determine the complexity of the activity patterns in isolation or in conjunction with other activities.
  • data table 800 can be used at 420 in process 400 (FIG. 4).
  • table 800 includes a number count of pattern occurrences during a window“N,” where the number counts of activity that matches patterns of different dimensions are presented in different rows.
  • row 805 includes a number count (i.e.,“2032”) of the occurrences of activity that matches one or more three point, 2-dimensional patterns
  • row 810 includes a number count (i.e., “877”) of the occurrences of activity that matches one or more four point, 3-dimensional patterns. Since an occurrence of the patterns indicates that the activity has an order which is non-random, the number counts also provide generalized characterization of the overall complexity of the activity patterns.
  • a table that is analogous to table 800 can be formed for each window that is defined, e.g., at 410 in process 400 (FIG. 4).
  • table 800 includes a separate row and a separate entry for every type of activity pattern, this is not necessarily the case.
  • one of more counts can be omitted (e.g., counts of simpler patterns) can be omitted from table 800 and from a determination of the complexity.
  • a single row or entry can include counts of occurrences of multiple activity patterns.
  • FIG. 8 presents the number count in a table 800, this is not necessarily the case.
  • the number count can be presented as a vector (e.g., ⁇ 2032, 877, 133, 66, 48, ...>).
  • the counts can be expressed in binary and can be compatible with digital data processing infrastructure.
  • number counts of the occurrences of the patterns can be weighted or combined to determine the degree or complexity of the ordering, e.g., at 420 in process 400 (FIG. 4).
  • the Euler characteristic can provide an approximation of the complexity of the activity and is given by: EQUATION 1 where Sn is number of occurrences of a pattern of n points (i.e., a pattern of dimensionality n- 1).
  • the patterns can be, e.g., the directed clique patterns 500 (FIG. 5).
  • pattern occurrences can be weighted based on the weights of the links that are active.
  • the strength of connection between nodes in an artificial neural network can vary, e.g., as a consequence of how active the connection was during training.
  • An occurrence of a pattern of activity along a collection of relatively stronger links can be weighted differently from the occurrence of that same pattern of activity along a collection of relatively weaker links.
  • the sum of the weight of the links that are active can be used to weight the occurrence.
  • the Euler characteristic or other measure of complexity can be normalized by the total number of patterns that are matched within a particular window and/or the total number of patterns that it is possible for a given network to form given its structure.
  • An example of a normalization with regard to the total number of patterns that it is possible for a network to form is given below in Equations 2, 3.
  • occurrences of higher dimension patterns involving larger numbers of nodes can be weighted more heavily than occurrences of lower dimension patterns involving smaller numbers of nodes. For example, the probability of forming directed cliques decreases rapidly with increasing dimension. In particular, to form an n- clique from n+l nodes, one needs (n+l)n/2 edges all oriented correctly. This probability can be reflected in the weighting.
  • both the dimension and the directionality of patterns can be used to weight occurrences of patterns and determine the complexity of the activity. For example, with reference to FIG. 6, occurrences of five point, 4-dimensional pattern 515 can be weighted more heavily than occurrences of five point, 4-dimensional patterns 605, 610 in accordance with the differences in directionality of those patterns.
  • An example of the use of both the directionality and the dimension of patterns to determine the degree of ordering or complexity of the activity can be given by
  • Sx actlve indicates the number of active occurrences of a pattern of n points and ERN is the calculation for an equivalent random network, i.e., a network with the same number of nodes randomly connected.
  • SC is given by
  • Equation 3 Sx sllent indicates the number of occurrences of a pattern of n points when the recurrent artificial neural network is silent and can be thought of as embodying the total number of patterns that it is possible for the network to form.
  • the patterns can be, e.g., the directed clique patterns 500 (FIG. 5).
  • FIG. 9 is a schematic illustration of a determination of the timing of activity patterns that have a distinguishable complexity.
  • the determination illustrated in FIG. 9 can be performed in isolation or in conjunction with other activities. For example, the determination can be performed at 425 in process 400 (FIG. 4).
  • FIG. 9 includes a graph 905 and a graph 910.
  • Graph 905 illustrates occurrences of patterns as a function of time along the x-axis.
  • individual occurrences are illustrated schematically as vertical lines 906, 907, 908, 909.
  • Each row of occurrences can be instances where activity matches a respective pattern or class of pattern.
  • the top row of occurrences can be instances where activity matches pattern 505 (FIG. 5)
  • the second row of occurrences can be instances where activity matches pattern 510 (FIG. 5)
  • the third row of occurrences can be instances where activity matches pattern 515 (FIG. 5), and so on.
  • Graph 905 also includes dashed rectangles 915, 920, 925 that schematically delineate different windows of time when the activity patterns have a distinguishable complexity. As shown, the likelihood that activity in the recurrent artificial neural network matches a pattern indicative of complexity is higher during the windows delineated by dashed rectangles 915, 920, 925 than outside those windows.
  • Graph 910 illustrates the complexity associated with these occurrences as a function of time along the x-axis.
  • Graph 910 includes a first peak 930 in complexity that coincides with the window delineated by dashed rectangle 915 and a second peak 935 in complexity that coincides with the window delineated by dashed rectangles 920, 925.
  • the complexity illustrated by peaks 930, 925 is distinguishable from what can be considered to be a baseline level 940 of complexity.
  • the times at which the output of a recurrent artificial neural network is to be read coincide with the occurrences of activity patterns that have a distinguishable complexity.
  • the output of a recurrent artificial neural network can be read at peaks 930, 925, i.e., during the windows delineated by dashed rectangles 915, 920, 925.
  • the identification of distinguishable levels of complexity in a recurrent artificial neural network is particularly beneficial when the input is a stream of data.
  • data streams include, e.g., video or audio data.
  • data streams have a beginning, it is generally desirable to process information in the data stream that does not have a pre-defined relationship with the beginning of the data stream.
  • a neural network could perform object recognition such as, e.g., recognizing bicyclists in the vicinity of an automobile.
  • object recognition such as, e.g., recognizing bicyclists in the vicinity of an automobile.
  • Such a neural networks should be able to recognizing bicyclists regardless of when those bicyclists appear in the video stream, i.e., without regard to time since the beginning of the video.
  • any patterns of activity in the neural network will generally display a low or quiescent level of complexity. These low or quiescent level of complexity are displayed regardless of the continuous (or nearly continuous) input of streaming data into the neural network device.
  • the complexity of the activity will become distinguishable and indicate the time at which an object is recognized in the video stream.
  • the timing of a distinguishable level of complexity of the activity can also act as a yes/no output as to whether the data in the data stream satisfies certain criteria.
  • the activity patterns that have a distinguishable complexity.
  • the identity and activity of the nodes that participate in activity that comports with the activity patterns can be considered the output of the recurrent artificial neural network.
  • the identified activity patterns can thus illustrate the result of processing by the neural network, as well as the timing when this decision is to be read.
  • the content of the decision can be expressed in a variety of different forms.
  • the content of the decision can be expressed as a binary vector or matrix of ones and zeros. Each digit can indicate whether or not a pattern of activity is present or not, e.g., for a pre-defined group of nodes and/or a predefined duration.
  • the content of the decision is expressed in binary and can be compatible with traditional digital data processing
  • FIG. 10 is a flowchart of a process 1000 for encoding signals using a recurrent artificial neural network based on characterization of the activity in the network.
  • Signals can be encoded in a variety of different contexts such as, e.g., transmission, encryption, and data storage.
  • Process 1000 can be performed by a system of one or more data processing apparatus that perform operations in accordance with the logic of one or more sets of machine-readable instructions.
  • process 1000 can be performed by that same system of one or more computers that executes software for implementing the recurrent artificial neural network used in process 1000.
  • process 1000 can be performed by the same data processing apparatus that performs process 400.
  • process 1000 can be performed by, e.g., the encoder in a signal transmission system or the encoder of a data storage system.
  • the system performing process 1000 inputs a signal into a recurrent artificial neural network at 1005.
  • the input of the signal is a discrete injection event.
  • the input signal is streamed into the recurrent artificial neural network.
  • the system performing process 1000 identifies one or more decision moments in the recurrent artificial neural network at 1010. For example, the system can identify one or more decision moments by performing process 400 (FIG. 4).
  • the system performing process 1000 reads the output of the recurrent artificial neural network at 1015.
  • the content of the output of the recurrent artificial neural network is the activity in the neural network that matches the patterns used to identify the decision point(s).
  • individual“reader nodes” can be added to a neural network to identify occurrences of a particular pattern of activity at a particular collection of nodes and hence to read the output of the recurrent artificial neural network at 1015.
  • the reader nodes can fire if and only if the activity at a particular collection of nodes satisfies timing (and possibly magnitude, as well) criteria. For example, in order to read an occurrence of pattern 505 (FIG. 5) at nodes 104, 105, 106 (FIGS, 2, 3), a reader node could be connected to nodes 104, 105, 106 (or the links 110 between them). The reader node would itself only become active if a pattern of activity involving nodes 104, 105, 106 (or their links) occurred.
  • the use of such reader nodes would eliminate the need to define windows of time for the recurrent artificial neural network as a whole.
  • individual reader nodes can be connected to different nodes and/or numbers of nodes (or the links between them).
  • the individual reader nodes can be set to have tailored responses (e.g., different decay times in an integrate-and-fire model) to identify different activity patterns.
  • the system performing process 1000 transmits or stores the output of the recurrent artificial neural network at 1020.
  • the particular action performed at 1020 can reflect the context in which process 1000 is being used. For example, in contexts where secure or compressed communications are desired, the system performing process 1000 can transmit the output of the recurrent neural network to a receiver that has access to the same or to a similar recurrent neural network. As another example, in contexts where secure or compressed data storage is desired, the system performing process 1000 can record the output of the recurrent neural network in one or more machine-readable data storage devices for later access.
  • the complete output of the recurrent neural network is not transmitted or stored.
  • the content of the output of the recurrent neural network is the activity in the neural network that matches the patterns indicative of complexity in the activity
  • only activity that matches relatively more complex or higher dimensional activity may be transmitted or stored.
  • patterns 500 FIG. 5
  • only activity that matches patterns 515, 520, 525, and 530 is transmitted or stored whereas activity that matches patterns 505, 510 is ignored or discarded. In this way, a lossy process allows the volume of data that is transmitted or stored to be reduced at the cost of the completeness of the information being encoded.
  • FIG. 11 is a flowchart of a process 1100 for decoding signals using a recurrent artificial neural network based on characterization of the activity in the network.
  • Signals can be decoded in a variety of different contexts such as, e.g., signal reception, decryption, and reading data from storage.
  • Process 1100 can be performed by a system of one or more data processing apparatus that perform operations in accordance with the logic of one or more sets of machine-readable instructions.
  • process 1100 can be performed by that same system of one or more computers that executes software for implementing the recurrent artificial neural network used in process 1100.
  • process 1100 can be performed by the same data processing apparatus that performs process 400 and/or process 1000.
  • process 1100 can be performed by, e.g., the decoder in a signal reception system or the decoder of a data storage system.
  • the system performing process 1100 receives at least a portion of the output of a recurrent artificial neural network at 1105.
  • the particular action performed at 1105 can reflect the context in which process 1100 is being used.
  • the system performing process 1000 can receive a transmitted signal that includes the output of the recurrent artificial neural network or read a machine -readable data storage device that stores the output of the recurrent artificial neural network.
  • the system performing process 1100 reconstructs the input of the recurrent artificial neural network from the received output at 1110. Reconstruction can proceed in a variety of different ways. For example, in some implementations, a second artificial neural network (recurrent or not) can be trained to reconstruct the input into the recurrent neural network from the output received at 1105.
  • a decoder that has been trained using machine learning (including but not limited to deep learning) can be trained to reconstruct the input into the recurrent neural network from the output received at 1105.
  • machine learning including but not limited to deep learning
  • input into the same recurrent artificial neural network or into a similar recurrent artificial neural network can be iteratively permuted until the output of that recurrent artificial neural network matches, to some degree, the output received at 1105.
  • process 1100 can include receiving user input specifying an extent to which the input is to be reconstructed and, in response, adjust the reconstruction at 1110 accordingly.
  • the user input could specify that a complete reconstruction is not needed.
  • the system performing process 1100 adjusts the reconstruction.
  • the content of the output of the recurrent neural network is the activity in the neural network that matches the patterns indicative of complexity in the activity
  • only the output that characterizes activity that matches relatively more complex or higher dimensional activity would be used to reconstruct the input.
  • patterns 500 FIG. 5
  • only activity that matches patterns 515, 520, 525, and 530 could be used to reconstruct the input, whereas activity that matches patterns 505, 510 could be ignored or discarded. In this way, a lossy reconstruction can proceed in selected circumstances.
  • processes 1000, 1100 can be used for peer-to-peer encrypted communications.
  • both the sender (i.e., the encoder) and the receiver (i.e., the decoder) can be provided with the same recurrent artificial neural network.
  • the shared recurrent artificial neural network can be tailored to ensure that a third party cannot reverse-engineer it and decrypt the signal, including:
  • the decision moment time points can be used as keys to decrypt the signal.
  • processes 1000, 1100 are presented in terms of encoding and decoding a single recurrent artificial neural network, processes 1000, 1100 can also be applied in systems and processes that rely upon multiple recurrent artificial neural networks. These recurrent artificial neural networks can operate either in parallel or in series.
  • the output of a first recurrent artificial neural network can serve as the input of a second recurrent artificial neural network.
  • the resultant output of the second recurrent artificial neural network is a twice encoded (or twice encrypted) version of the input into the first recurrent artificial neural network.
  • Such a series arrangement of recurrent artificial neural networks can be useful in circumstances where different parties have different levels of access to information, e.g., in medical record systems where patient identity information may not be accessible to a party that will be using and have access to the remainder of the medical record.
  • the same information can be input into multiple, different recurrent artificial neural networks.
  • the different outputs of those neural networks can be used, e.g., to ensure that the input can be reconstructed with high fidelity.
  • activity within a recurrent artificial neural network should match a pattern indicative of ordering, this is not necessarily the case. Rather, in some implementations, activity within a recurrent artificial neural network can comport with a pattern without necessarily displaying activity that matches the pattern. For example, an increase in the likelihood that a recurrent neural network is to display activity that would match a pattern can be treated as non-random ordering of the activity. As yet another example, in some implementations, different groups of patterns can be tailored for use in characterizing the activity in different recurrent artificial neural networks. The patterns can be tailored, e.g., according to the effectiveness of the patterns in
  • Effectiveness can be quantified, e.g., based on the size of a table or vector that represents the occurrence counts of different patterns.
  • the patterns used to characterize the activity in a recurrent artificial neural networks can consider the strength of a connection between nodes.
  • the patterns described previously herein treat all signal transmission activity between two nodes in a binary manner, i.e., either the activity exists or it doesn’t. This is not necessarily the case. Rather, in some implementations, comporting with a pattern can require activity of a certain level or strength of connection to be taken as indicative of ordered complexity in the activity of a recurrent artificial neural network.
  • the content of the output of the recurrent artificial neural network can include activity patterns that occur outside windows of time in which the activity in a neural network has a distinguishable level of complexity.
  • the output of the recurrent artificial neural network that is read at 1015 and transmitted or stored at 1020 (FIG. 10) can include information encoding activity patterns that occur, e.g., outside dashed rectangles 915, 920, 925 in graph 905 (FIG. 9).
  • the output of the recurrent artificial neural network could characterize only the highest dimensional patterns of activity, regardless of when those patterns of activity occur.
  • the output of the recurrent artificial neural network could characterize only patterns of activity that enclose cavities, regardless of when those patterns of activity occur.
  • FIGS. 12, 13, and 14 are schematic illustrations of a binary form or representation 1200 of topological structures such as, e.g., patterns of activity in a neural network.
  • the topological structures illustrated in FIGS. 12, 13, and 14 all include the same information, namely, an indication the presence or absence of features in a graph.
  • the features can be, e.g., activity in a neural network device.
  • the activity is identified based on or during periods of time in which the activity in the neural network has a complexity that is distinguishable from other activity that is responsive to an input.
  • binary representation 1200 includes bits 1205, 1207, 1211, 1293, 1294, 1297 and an additional, arbitrary number of bits (represented by the ellipses
  • bits 1205, 1207, 1211, 1293, 1294, 1297 ... are illustrated as discrete rectangular shapes that are either filled or unfilled to indicate the binary value of the bit.
  • representation 1200 superficially appears to be either a one- dimensional vector of bits (FIGS. 12, 13) or a two-dimensional matrix of bits (FIG. 14).
  • representation 1200 differs from a vector, from a matrix, or from other ordered collection of bits in that the same information can be encoded regardless of the order of the bits— i.e., regardless of the location of individual bits within the collection.
  • bit 1294, 1297 ... can represent the presence or absence of a topological feature— regardless of the location of that feature in the graph.
  • a bit such as bit 1207 can indicate the presence of a topological feature that comports with pattern 505 (FIG. 5), regardless of whether that activity occurs between nodes 104, 105, 101 or between nodes 105, 101, 102.
  • pattern 505 FIG. 5
  • representation 1200 may only provide an isomorphic topological reconstruction of the graph.
  • the source graph 1205, 1207, 1211, 1293, 1294, 1297, ... does indeed encode information such as, e.g., the location of a feature in the graph.
  • the source graph can be reconstructed using representation 1200.
  • such an encoding is not necessarily present.
  • bit 1205 appears before bit 1207, which appears before bit 1211 at the start of representation 1200.
  • bit 1211 appears before bit 1211 at the start of representation 1200.
  • FIGS. 2 and 3 the order of bits 1205, 1207, and 1211 within representation 1200— and the position of bits 1205, 1207, and 1211 relative to other bits within
  • binary representation 1200 has changed. Nevertheless, binary representation 1200 remains the same— as does the set of rules or algorithm that defines the process for encoding information in binary representation 1200. So long as the correspondence between the bit and the feature is known, the location of the bits in the representation 1200 is irrelevant.
  • each bit 1205, 1207, 1211, 1293, 1294, 1297 ... individually represents the presence or absence of a feature in a graph.
  • a graph is a set of nodes and a set of edges between those nodes.
  • the nodes can correspond to objects. Examples of objects can include, e.g., artificial neurons in a neural network, individuals in a social network, or the like. Edges can correspond to some relation between the objects. Examples of relations include, e.g., a structural connection or activity along the connection. In the context of a neural network, artificial neurons can be related by a structural connection between neurons or by transmission of information along a structural connection.
  • edges can thus characterize relatively long-lived structural characteristics of the set of nodes or relatively transient activity characteristics that occur within a defined time frame. Further, edges can either be directed or bidirectional. Directed edges indicate directionality of the relation between the objects. For example, the transmission of information from a first neuron to a second neuron can be represented by a directed edge that denotes the direction of
  • a relational connection may indicate that second user is to receive information from the first but not that the first is to receive information from the second.
  • a graph can be expressed as a set of unit intervals [0, 1] where 0 and 1 are identified with respective nodes that are connected by an edge.
  • bits 1205, 1207, 1211, 1293, 1294, 1297 can be, e.g., a node, a set of nodes, a set of sets of nodes, a set of edges, a set of sets of edges, and/or additional hierarchically-more-complex features (e.g., a set of sets of sets of nodes).
  • Bits 1205, 1207, 1211, 1293, 1294, 1297 generally represent the presence or absence of features that are at different hierarchical levels. For example, bit 1205 may represent the presence or absence of a node, whereas bit 1205 may represent the presence or absence of a set of nodes.
  • bits 1205, 1207, 1211, 1293, 1294, 1297 may represent features in a graph that have a threshold level of some characteristic.
  • bits 1205, 1207, 1211, 1293, 1294, 1297 can represent not only that there is activity in a set of edges, but also that this activity is weighted either above or below a threshold level.
  • the weights can, e.g., embody the training of a neural network device to a particular purpose or can be an innate characteristic of the edges.
  • FIGS. 5, 6, and 8 above illustrate features whose presence or absence can be represented by bits 1205, 1207, 1211, 1293, 1294, 1297 ...
  • the directed simplices in collections 500, 600, 700 treat functional or structural graphs as a topological space with nodes as points. Structure or activity involving one or more nodes and links that comports with simplices in collection 500, 600, 700 can be represented in a bit regardless of the identity of the particular nodes and/or links that participate in the activity.
  • 4-dimensional simplex pattern 515 inherently includes structure or activity that comports with the four point, 3 -dimensional and three point, 2-dimension simplex patterns 510, 505.
  • points 0, 2, 3, 4 and points 1, 2, 3, 4 in 4- dimensional simplex pattern 515 of FIG. 5 both comport with 3-dimensional simplex pattern 510.
  • simplex patterns that include fewer points— and hence are of a lower dimension— can be discarded or otherwise ignored.
  • bits 1205, 1207, 1211, 1293, 1294, 1297 may not be independent of one another.
  • bits 1205, 1207, 1211, 1293, 1294, 1297 represent the presence or absence of OD-simplices that each reflect the existence or activity of a single node, then bits 1205, 1207, 1211, 1293, 1294, 1297 are independent of one another.
  • bits 1205, 1207, 1211, 1293, 1294, 1297 represent the presence or absence of higher-dimensional simplices that each reflect the existence or activity of multiple nodes, then the information encoded by the presence or absence of each individual feature may not be independent of the presence or absence of the other features.
  • FIG. 15 schematically illustrates an example of how the presence or absence of features that correspond to different bits are not independent of one another.
  • a subgraph 1500 that includes four nodes 1505, 1510, 1515, 1520 and six directed edges 1525, 1530, 1535, 1540, 1545, 1550 is illustrated.
  • edge 1525 is directed from node 1525 to node 1510
  • edge 1530 is directed from node 1515 to node 1505
  • edge 1535 is directed from node 1520 to node 1505
  • edge 1540 is directed from node 1520 to node 1510
  • edge 1545 is directed from node 1515 to node 1510
  • edge 1550 is directed from node 1515 to node 1520.
  • a single bit in representation 1200 may indicate the presence of a directed 3D-simplex.
  • a bit could indicate the presence of the 3D-simplex formed by nodes 1505, 1510, 1515, 1520 and edges 1525, 1530, 1535, 1540, 1545, 1550.
  • a second bit in representation 1200 e.g., filled bit 1293 in FIGS.
  • bit 1293 may indicate the presence of a directed 2D-simplex.
  • a bit could indicate the presence of the 2D-simplex formed by nodes 1515, 1505, 1510 and edges 1525, 1530, 1545.
  • the information encoded by bit 1293 is
  • the information encoded by bit 1293 may also be redundant with the information encoded by still further bit.
  • the information encoded by bit 1293 would be redundant with both a third bit and a fourth bit that indicates the presence of additional directed 2D-simplices. Examples of those simplices are formed by nodes 1515, 1520, 1510 and edges 1540, 1545, 1550 and nodes 1520, 1505, 1510 and edges 1525, 1535,
  • FIG. 16 schematically illustrates another example of how the presence or absence of features that correspond to different bits are not independent of one another.
  • a subgraph 1600 that includes four nodes 1605, 1610, 1615, 1620 and five directed edges 1625, 1630, 1635, 1640, 1645 is illustrated.
  • Nodes 1505, 1510, 1515, 1520 and edges 1625, 1630, 1635, 1640, 1645 generally correspond to nodes 1505, 1510, 1515, 1520 and edges 1525, 1530, 1535, 1540, 1545 in subgraph 1500 (FIG. 15).
  • nodes 1615, 1620 are not connected by an edge.
  • a single bit in representation 1200 may indicate the absence of a directed 3D-simplex, such as, e.g., the directed 3D-simplex that encompasses nodes 1605, 1610, 1615, 1620.
  • a second bit in representation 1200 (e.g., filled bit 1293 in FIGS. 12, 13, 14) may indicate the presence of a 2D-simplex.
  • An example directed 2D-simplex is formed by nodes 1615, 1605, 1610 and edges 1625, 1630, 1645.
  • This combination of a filled bit 1293 and an unfilled bit 1205 provides information indicative of presence or absence of other features (and the state of other bits) that may or may not be present in representation 1200.
  • the combination of the absence of a directed 3D-simplex and the presence of a directed 2D-simplex indicates that at least one edge is absent from either:
  • a representation 1200 that includes a collection of bits that corresponds only to, e.g., the presence or absence of 3D-simplices is possible.
  • the encoding of information is fault tolerant and provides “graceful degradation” of the encoded information.
  • the loss of a particular bit (or group of bits) may increase the uncertainty as to the presence or absence of a feature.
  • estimates of the likelihood that a feature is present or absent will still be possible from the other bits that indicate the presence or absence of adjacent features.
  • the patterns of activity in a neural network can be encoded in a representation 1200 (FIGS. 12, 13, and 14).
  • the patterns of activity in a neural network are a result of a number of characteristics of the neural network such as, e.g., the structural connections between nodes of the neural network, the weights between nodes, as well as a whole host of possible other parameters.
  • the neural network could have been trained prior to the encoding of the patterns of activity in representation 1200.
  • the responsive pattern of activity can be thought of a“representation” or an
  • representation 1200 can appear to be a straightforward-appearing collection of (in some cases, binary) digits, each of the digits can encode the relationship or correspondence between a particular input and relevant activity in the neural network.
  • FIGS. 17, 18, 19, 20 are schematic illustrations of the use of representations of the occurrence of topological structures in the activity in a neural network in four different classification systems 1700, 1800, 1900, 2000.
  • Classification systems 1700, 1800 each classify representations of the patterns of activity in a neural network as part of the classification of input.
  • Classification systems 1900, 2000 each classify approximations of representations of the patterns of activity in a neural network as part of the classification of input.
  • the patterns of activity that are represented occur in and are read from a source neural network device 1705 that is part of the classification system 1700, 1800.
  • classification systems 1900, 2000 the patterns of activity that are approximately represented occur in a source neural network device that is not part of the classification system 1700, 1800.
  • classification system 1700 includes a source neural network 1705 and a linear classifier 1710.
  • Source neural network 1705 is a neural network device that is configured to receive an input and present representations of the occurrence of topological structures in the activity within source neural network 1705.
  • source neural network 1705 includes an input layer 1715 that receives the input.
  • this is not necessarily the case.
  • some or all of the input can be injected into different layers and/or edges or nodes throughout source neural network 1705.
  • Source neural network 1705 can be any of variety of different types of neural network.
  • source neural network 1705 is a recurrent neural network such as, e.g., a recurrent neural network that is modelled on a biological system.
  • source neural network 1705 can model a degree of the morphological, chemical, and other characteristics of a biological system.
  • source neural network 1705 is
  • classification system 1700 will generally be a dispersed system in which a remote classifier 1710 communicates with source neural network 1705, e.g., via a data communications network.
  • source neural network 1705 can be untrained and the activity that is represented can be the innate activity of source neural network 1705. In other implementations, source neural network 1705 can be trained and the activity that is represented can embody this training.
  • the representations read from source neural network 1705 can be representations such as representation 1200 (FIGS. 12, 13, 14).
  • the representations can be read from source neural network 1705 in a number of ways.
  • source neural network 1705 includes“reader nodes” that read patterns of activity between other nodes within source neural network 1705.
  • the activity within source neural network 1705 are read by a data processing component of that is programmed to monitor source neural network 1705 for relatively highly-ordered patterns of activity.
  • source neural network 1705 can include an output layer from which representation 1200 can be read, e.g., when source neural network 1705 is
  • Linear classifier 1710 is a device that classifies an object— namely, representations of the patterns of activity in source neural network 1705— based on a linear combination of the object’s characteristics.
  • Linear classifier 1710 includes an input 1720 and an output 1725.
  • Input 1720 is coupled to receive representations of the patterns of activity in source neural network 1705.
  • the representations of the patterns of activity in source neural network 1705 is a feature vector that represents the characteristics of the input into source neural network 1705 that are used by linear classifier 1710 to classify that input.
  • Linear classifier 1710 can receive the representations of the patterns of activity in source neural network 1705 in a variety of ways.
  • the representations of the patterns of activity can be received as discrete events or as a continuous stream over a real time or non- real time communication channel.
  • Output 1725 is coupled to output the classification result from linear classifier 1710.
  • output 1725 is schematically illustrated as a parallel port with multiple channels. This is not necessarily the case.
  • output 1725 can output classification result over a serial port or a port with combined parallel and serial capabilities.
  • linear classifier 1710 can be implemented on one or more computing devices with relatively limited computational performance.
  • linear classifier 1710 can be implemented on a personal computer or a mobile computing device such as a smart phone or tablet.
  • classification system 1800 includes source neural network 1705 and a neural network classifier 1810.
  • Neural network classifier 1810 is a neural network device classifies an object— namely, representations of the patterns of activity in source neural network 1705— based on a non-linear combination of the object’s characteristics.
  • neural network classifier 1810 is a feedforward network that includes an input layer 1820 and an output layer 1825.
  • neural network classifier 1810 can receive the representations of the patterns of activity in source neural network 1705 in a variety of ways. For example, the representations of the patterns of activity can be received as discrete events or as a continuous stream over a real time or non- real time communication channel.
  • neural network classifier 1810 can perform inferences on one or more computing devices with relatively limited computational performance.
  • neural network classifier 1810 can be implemented on a personal computer or a mobile computing device such as a smart phone or tablet, e.g., in a Neural Processing Unit of such a device.
  • classification system 1800 will generally be a dispersed system in which a remote neural network classifier 1810 communicates with source neural network 1705, e.g., via a data communications network.
  • neural network classifier 1810 can be, e.g., a deep neural network such as a convolutional neural network that includes convolutional layers, pooling layers, and fully-connected layers.
  • Convolutional layers can generate feature maps, e.g., using linear convolutional filters and/or nonlinear activation functions. Pooling layers reduce the number of parameters and control overfitting.
  • the computations performed by the different layers in image classifier 1820 can be defined in different ways in different implementations of image classifier 1820.
  • classification system 1900 includes source approximator 1905 and a linear classifier 1710.
  • source approximator 1905 is a relatively simple neural network that is trained to receive input— either at an input layer 1915 or elsewhere— and output a vector that approximates a representation of topological structures that arise in the patterns of activity in a relatively more complex neural network.
  • source approximator 1905 can be trained to approximate a recurrent source neural network such as, e.g., a recurrent neural network that is modelled on a biological system and includes a degree of the morphological, chemical, and other characteristics of a biological system.
  • source approximator 1905 includes an input layer 1915 and an output layer 1920.
  • Input layer 1915 is couplable to receive the input data.
  • Output layer 1920 is coupled to output an approximation of a representation of the activity within a neural network device for receipt by input 1720 of linear classifier.
  • output layer 1920 can output an approximation 1200’ of representation 1200 (FIGS. 12, 13, 14).
  • the representation 1200 schematically illustrated in FIGS. 17, 18 and the approximation 1200’ of representation 1200 schematically illustrated in FIGS. 19, 20 are identical. This is for the sake of convenience only. In general, approximation 1200’ will differ from representation 1200 in at least some ways. Despite those differences, linear classifier 1710 can classify approximation 1200’.
  • source approximator 1905 can perform inferences on one or more computing devices with relatively limited computational performance.
  • source approximator 1905 can be implemented on a personal computer or a mobile computing device such as a smart phone or tablet, e.g., in a Neural Processing Unit of such a device.
  • classification system 1900 will generally be housed in a single housing, e.g., with source approximator 1905 and linear classifier 1710 implemented on either the same data processing devices or on data processing devices coupled by a hardwired connection.
  • classification system 2000 includes source approximator 1905 and a neural network classifier 1810.
  • Output layer 1920 of source approximator 1905 is coupled to output an approximation 1200’ of a representation of the activity within a neural network device for receipt by input 1820 of neural network classifier 1810.
  • neural network classifier 1810 can classify approximation 1200’.
  • classification system 1900 will generally be housed in a single housing, e.g., with source approximator 1905 and neural network classifier 1810 implemented on either the same data processing devices or on data processing devices coupled by a hardwired connection.
  • FIG. 21 is a schematic illustration of an edge device 2100 that includes a local artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
  • a local artificial neural network can be, e.g., an artificial neural network that is executed entirely on one or more local processors that do not require a communications network to exchange data.
  • the local processors will be connected by hardwire connections.
  • the local processors can be housed in a single housing, such as a single personal computer or a single handheld, mobile device. In some instances, the local processors can be under the control of and accessible by a single individual or a limited number of individuals. In effect, by using a representation of the occurrence of topological structures in a more complex source neural network to train (e.g., using a supervised learning or reinforcement learning technique) a simpler and/or less highly trained but more exclusive second neural network, even individuals with limited computing resources and limited numbers of training samples can train a neural network as desired. Storage requirements and computational complexity during training are reduced and resources like battery lifespan are spared.
  • edge device 2100 is schematically illustrated as a security-camera device that includes an optical imaging system 2110, image processing electronics 2115, a source approximator 2120, a representation classifier 2125, and a communications controller and interface 2130.
  • Optical imaging system 2110 can include, e.g., one or more lenses (or even a pinhole) and a CCD device.
  • Image processing electronics 2115 can read the output of optical imaging system 2110 and in general can perform basic image processing functions.
  • Communications controller and interface 2130 is a device that is configured to control the flow of information to and from device 2100. As discussed further below, among the operations that
  • communications controller and interface 2130 can perform are the transmission of images of interest to other devices and the receipt of training information from other devices.
  • Communications controller and interface 2130 can thus include both a data transmitter and a receiver that can communicate over, e.g., a data port 2135.
  • Data port 2135 can be a wired port, a wireless port, an optical port, or the like.
  • Source approximator 2120 is relatively simple neural network that is trained to output a vector that approximates a representation of topological structures that arise in the patterns of activity in a relatively more complex neural network. For example, source
  • approximator2l20 can be trained to approximate a recurrent source neural network such as, e.g., a recurrent neural network that is modelled on a biological system and includes a degree of the morphological, chemical, and other characteristics of a biological system.
  • a recurrent source neural network such as, e.g., a recurrent neural network that is modelled on a biological system and includes a degree of the morphological, chemical, and other characteristics of a biological system.
  • Representation classifier 2125 is either a linear classifier or a neural network classifier that is coupled to receive an approximation of a representation of the patterns of activity in a source neural network from source approximator 2120 and output a classification result.
  • Representation classifier 2125 can be, e.g., a deep neural network such as a convolutional neural network that includes convolutional layers, pooling layers, and fully-connected layers.
  • Convolutional layers can generate feature maps, e.g., using linear convolutional filters and/or nonlinear activation functions. Pooling layers reduce the number of parameters and control overfitting.
  • the computations performed by the different layers in representation classifier 2125 can be defined in different ways in different implementations of representation classifier 2125.
  • optical imaging system 2110 can generate raw digital images.
  • Image processing electronics 2115 can read the raw images and will generally perform at least some basic image processing functions.
  • Source approximator 2120 can receive images from image processing electronics 2115 and perform inference operations to output a vector that approximates a representation of topological structures that arise in the patterns of activity in a relatively more complex neural network. This
  • approximation vector is input into representation classifier 2125 which determines whether the approximation vector satisfies one or more sets of classification criteria. Examples include facial recognition and other machine vision operations. In the event that
  • communications controller and interface 2130 can transmit the image itself, the
  • the training set can include raw or processed image data
  • Such a training set can be used to retrain source approximator 2120, e.g., using a supervised learning or reinforcement learning technique.
  • the representations are used as the target answer vectors and represent the desired result of source approximator 2120 processing the raw or processed image data.
  • the training set can include representations of topological structures that arise in the patterns of activity in a relatively more complex neural network and the desired classification of those representations of topological structures.
  • Such a training set can be used to retrain a neural network representation classifier 2125, e.g., using a supervised learning or reinforcement learning technique.
  • the desired classification are used as the target answer vectors and represent the desired result of representation classifier 2125 processing the representations of topological structures.
  • inference operations at device 2100 can be facilely adapted to changing circumstances and objectives without large sets of training data and time- and computing power-intensive iterative training.
  • FIG. 22 is a schematic illustration of a second edge device 2200 that includes a local artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
  • second edge device 2200 is schematically illustrated as a mobile computing device such as smart phone or a tablet.
  • Device 2200 includes an optical imaging system (e.g., on the backside of device 2200, not shown), image processing electronics 2215, a representation classifier 2225, a communications controller and interface 2230, and a data port 2235. These components can have characteristics and perform actions that correspond to those of optical imaging system 2110, image processing electronics 2115, representation classifier 2125, communications controller and interface 2130, and data port 2135 in device 2100 (FIG. 21).
  • the illustrated implementation of device 2200 additionally includes one or more additional sensors 2240 and a multi-input source approximator 2245.
  • Sensor(s) 2240 can sense one of more characteristics of the environment surrounding device 2200 or of device 2200 itself.
  • sensor 2240 can be an accelerometer that senses the acceleration to which device 2200 is subject.
  • sensor 2240 can be an acoustic sensor such as a microphone that senses noise in the environment of device 2200.
  • Still further examples of sensor 2240 include chemical sensors (e.g.,“artificial noses” and the like), humidity sensors, radiation sensors, and the like.
  • sensor 2240 is coupled to processing electronics that can read the output of sensor 2240 (or other information such as, e.g., a contact list or map) and perform basic processing functions.
  • processing electronics can read the output of sensor 2240 (or other information such as, e.g., a contact list or map) and perform basic processing functions.
  • sensor 2240 can thus have different“modalities” in that the physical sensed physical parameter changes from sensor to sensor.
  • Multi-input source approximator 2245 is a relatively simple neural network that is trained to output a vector that approximates a representation of topological structures that arise in the patterns of activity in a relatively more complex neural network.
  • multi-input source approximator 2245 can be trained to approximate a recurrent source neural network such as, e.g., a recurrent neural network that is modelled on a biological system and includes a degree of the morphological, chemical, and other characteristics of a biological system.
  • multi-input source approximator 2245 is coupled to receive raw or processed sensor data from multiple sensors and return an approximation of a representation of topological structures that arise in the patterns of activity in a relatively more complex neural network based on that data.
  • multi-input source approximator 2245 can receive processed image data from image processing electronics 2215 as well as, e.g., acoustic, acceleration, chemical, or other data from one or more sensors 2240.
  • Multi-input source approximator 2245 can be, e.g., a deep neural network such as a convolutional neural network that includes convolutional layers, pooling layers, and fully- connected layers.
  • the computations performed by the different layers in multi-input source approximator 2245 can be dedicated to a single type of sensor data or to sensor data of multiple modalities.
  • multi-input source approximator 2245 is to return the approximation based on raw or processed sensor data from multiple sensors. For example, processed image data from image processing electronics 2215 and acoustic data from a microphone sensor 2240 can be used by multi-input source approximator 2245 to approximate a representation of topological structures that would arise in the patterns of activity in a relatively more complex neural network that received the same data. At times, it may be desirable to change the classification process at device 2200. In these cases, communications controller and interface 2230 can receive a training set.
  • the training set can include raw or processed image, sounds, chemical or other data and representations of topological structures that arise in the patterns of activity in a relatively more complex neural network.
  • Such a training set can be used to retrain multi input source approximator 2245, e.g., using a supervised learning or reinforcement learning technique.
  • the representations are used as the target answer vectors and represent the desired result of multi-input source approximator 2245 processing the raw or processed image or sensor data.
  • the training set can include representations of topological structures that arise in the patterns of activity in a relatively more complex neural network and the desired classification of those representations of topological structures.
  • Such a training set can be used to retrain a neural network representation classifier 2225, e.g., using a supervised learning or reinforcement learning technique.
  • the desired classification are used as the target answer vectors and represent the desired result of representation classifier 2225 processing the representations of topological structures.
  • inference operations at device 2200 can be facilely adapted to changing circumstances and objectives without large sets of training data and time- and computing power-intensive iterative training.
  • FIG. 23 is a schematic representation of a system 2300 in which local neural networks can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
  • the target neural networks are implemented on relatively simple, less expensive data processing systems whereas the source neural network can implemented on a relatively complex, more expensive data processing system.
  • System 2300 includes a variety of devices 2305 with local neural networks, a telephone base station 2310, a wireless access point 2315, a server system 2320, and one or more data communications networks 2325.
  • Local neural networks devices 2305 are devices that are configured to process data using computationally-less-intensive target neural networks. As illustrated, local neural networks devices 2305 can be implemented as mobile computing devices, cameras, automobiles, or any of a legion of other appliances, fixtures, and mobile components, as well as different makes and models of devices within each category. Different local neural networks devices 2305 can belong to different owners. In some implementations, access to the data processing functionality of local neural networks devices 2305 will generally be restricted to these owners and/or the owner’s designates.
  • Local neural networks devices 2305 can each include one or more source
  • the relatively more complex neural network can be a recurrent source neural network such as, e.g., a recurrent neural network that is modelled on a biological system and includes a degree of the morphological, chemical, and other characteristics of a biological system.
  • local neural networks devices 2305 can also be programmed to re-train the source approximators using representations of topological structures that arise in the patterns of activity in a relatively more complex neural network as the target answer vectors.
  • local neural networks devices 2305 can be programmed to perform one or more iterative training techniques (e.g., gradient descent or stochastic gradient descent).
  • the source approximators in local neural networks devices 2305 are trainable by, e.g., a dedicated training system or by a training system that is installed on a personal computer that can interact with the local neural networks devices 2305 to train source approximators.
  • Each local neural networks device 2305 includes one or more wireless or wired data communication components.
  • each local neural networks device 2305 includes at least one wireless data communication components such as a mobile phone transceiver, a wireless transceiver, or both.
  • the mobile phone transceivers are able to exchange data with phone base station 2310.
  • the wireless transceivers are able to exchange data with a wireless access point 2315.
  • Each local neural networks device 2305 may also be able to exchange data with peer mobile computing devices.
  • Phone base station 2310 and wireless access point 2315 are connected for data communication with one or more data communication networks 2325 and can exchange information with a server system 2320 over the network(s).
  • Local neural networks devices 2305 are thus generally also in data communication with server system 2320. However, this is not necessarily the case. For example, in implementations where local neural networks devices 2305 are trained by other data processing devices, local neural networks devices 2305 need only be in data communication with these other data processing devices at least once.
  • Server system 2320 is a system of one or more data processing devices that is programmed to perform data processing activities in accordance with one or more sets of machine-readable instructions.
  • the activities can include serving training sets to training systems for mobile computing devices 2305.
  • the training systems can be internal to mobile local neural networks devices 2305 themselves or on one or more other data processing devices.
  • the training sets can include representations of the occurrence of topological structures corresponding to activity in a source neural network and corresponding input data.
  • server system 2320 also includes the source neural network. However, this is not necessarily the case and server system 2320 may receive the training sets from yet another system of data processing device(s) that implement the source neural network.
  • server system 2320 can serve the training set to trainers that train mobile computing devices 2305.
  • the source approximators in target local neural networks devices 2305 can be trained using the training set so that the target neural networks approximate the operations of the source neural network.
  • FIGS. 24, 25, 26, 27 are schematic illustrations of the use of representations of the occurrence of topological structures in the activity in a neural network in four different systems 2400, 2500, 2600, 2700.
  • Systems 2400, 2500, 2600, 2700 can be configured to perform any of a number of different operations.
  • systems 2400, 2500, 2600, 2700 can perform object localization operations, object detection operations, object segmentation operations, object detection operations, prediction operations, action selection operations, or the like.
  • Object localization operations locate an object within an image. For example, a bounding box can be constructed around an object. In some cases, object localization can be combined with object recognition, in which the localized object is labeled with an appropriate designation.
  • Object detection operations classify image pixels as either belonging to a particular class (e.g., belonging to an object interest) or not.
  • object detection is performed by grouping pixels and forming bounding boxes around the pixel groups. The bounding box should be a tight fit around the object.
  • Object segmentation generally assigns class labels to each image pixel. Thus, rather than a bounding box, object segmentation proceeds on a pixel-by-pixel basis and generally requires that only a single label be assigned to each pixel.
  • Prediction operations seek to draw conclusions that are outside the range of a observed data. Although prediction operations can seek to forecast future occurrences (e.g., based on information about the past and current state), prediction operations can also seek to draw conclusions about the past and current state based on incomplete information on those states.
  • Action selection operations seek to choose an action based on a set of conditions.
  • Action selection operations have traditionally be broken down into different approaches such a symbol-based systems (classical planning), distributed solutions, and reactive or dynamic planning.
  • Classification systems 2400, 2500 each perform a desired operation on
  • Systems 2600, 2700 each perform a desired operation on approximations of representations of the patterns of activity in a neural network.
  • systems 2400, 2500 the patterns of activity that are represented occur in and are read from a source neural network device 1705 that is part of the system 2400, 2500.
  • system 2400 includes a source neural network 1705 and a linear processor 2410.
  • Linear processor 2410 is a device that performs operations based on a linear combination of the characteristics of representations of the patterns of activity in a neural network (or approximations of such representations).
  • the operations can be, e.g., object localization operations, object detection operations, object segmentation operations, object detection operations, prediction operations, action selection operations, or the like.
  • Linear processor 2410 includes an input 2420 and an output 2425.
  • Input 2420 is coupled to receive representations of the patterns of activity in source neural network 1705.
  • Linear processor 2410 can receive the representations of the patterns of activity in source neural network 1705 in a variety of ways.
  • the representations of the patterns of activity can be received as discrete events or as a continuous stream over a real time or non- real time communication channel.
  • Output 2525 is coupled to output the processing result from linear processor 2410.
  • linear processor 2410 can be implemented on one or more computing devices with relatively limited computational performance.
  • linear processor 2410 can be implemented on a personal computer or a mobile computing device such as a smart phone or tablet.
  • system 2400 includes a source neural network 1705 and a linear processor 2410.
  • Linear processor 2410 is a device that performs operations based on a linear combination of the characteristics of representations of the patterns of activity in a neural network (or approximations of such representations).
  • the operations can be, e.g., object localization operations, object detection operations, object segmentation operations,
  • Linear processor 2410 includes an input 2420 and an output 2425.
  • Input 2420 is coupled to receive representations of the patterns of activity in source neural network 1705.
  • Linear processor 2410 can receive the representations of the patterns of activity in source neural network 1705 in a variety of ways.
  • the representations of the patterns of activity can be received as discrete events or as a continuous stream over a real time or non- real time communication channel.
  • Output 2525 is coupled to output the processing result from linear processor 2410.
  • linear processor 2410 can be implemented on one or more computing devices with relatively limited computational performance.
  • linear processor 2410 can be implemented on a personal computer or a mobile computing device such as a smart phone or tablet.
  • classification system 2500 includes source neural network 1705 and a neural network 2510.
  • Neural network 2510 is a neural network device that is configured to perform operations based on a non-linear combination of the characteristics of
  • neural network 2510 is a feedforward network that includes an input layer 2520 and an output layer 2525. As with linear processor 2410, neural network 2510 can receive the representations of the patterns of activity in source neural network 1705 in a variety of ways.
  • neural network 2510 can perform inferences on one or more computing devices with relatively limited computational performance.
  • neural network 2510 can be implemented on a personal computer or a mobile computing device such as a smart phone or tablet, e.g., in a Neural Processing Unit of such a device.
  • system 2500 will generally be a dispersed system in which a remote neural network 2510 communicates with source neural network 1705, e.g., via a data communications network.
  • neural network 2510 can be, e.g., a deep neural network such as a convolutional neural network.
  • system 2600 includes source approximator 1905 and a linear processor 2410. Despite any differences between approximation 1200’ representation 1200, processor
  • system 2700 includes source approximator 1905 and a neural network 2510. Despite any differences between approximation 1200’ and representation 1200, neural network 2510 can still perform operations on approximation 1200’.
  • systems 2600, 2700 can be implemented on an edge device, such as, e.g., edge devices 2100, 2200 (FIGS. 21, 22).
  • systems 2600, 2700 can be implemented as part of a system in which local neural networks can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network, such as system 2300 (FIG. 23).
  • FIG. 28 is a schematic illustration of a reinforcement learning system 2800 that includes an artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
  • Reinforcement learning is a type of machine learning in which an artificial neural network learns from feedback regarding the consequences of actions taken in response to the artificial neural network’s decisions.
  • a reinforcement learning system moves from one state in the environment to another by performing actions and receiving information characterizing the new state and a reward and/or regret that characterizes the success (or lack of success) of the action.
  • Reinforcement learning seek to maximize the total reward (or minimize the regret) through a learning process.
  • the artificial neural network in reinforcement learning system 2800 is a deep neural network 2805 (or other deep learning architecture) that is trained using a reinforcement learning approach.
  • deep neural network 2805 can be a local artificial neural network (such as neural network 2510 (FIGS.
  • deep neural network 2805 can be implemented on a system of networked devices.
  • reinforcement learning system 2800 includes an actuator 2810, one or more sensors 2815, and a teacher module 2820. In some implementations, reinforcement learning system 2800 also includes one or more sources 2825 of additional data.
  • Actuator 2810 is a device that controls a mechanism or a system that interacts with an environment 2830.
  • actuator 2810 controls a physical mechanism or system (e.g., the steering of an automobile or the positioning of a robot).
  • actuator 2810 can control a virtual mechanism or system (e.g., a virtual game board or an investment portfolio).
  • environment 2830 may also be physical or virtual.
  • Sensor(s) 2815 are devices that measure characteristics of the environment 2830. At least some of the measurements with characterize interactions between the controlled mechanism or system and other aspects of the environment 2830. For example, when actuator 2810 steers an automobile, sensor(s) 2815 may measure one or more of the speed, direction, and acceleration of the automobile, the proximity of the automobile to other features, and the response of other features to the automobile. As another example, when actuator 2810 controls an investment portfolio, sensor(s) 2815 may measure the value and risk associated with the portfolio.
  • both source approximator 1905 and teacher module 2820 are coupled to receive at least some of the measurements made by sensor(s) 2815.
  • source approximator 1905 can receive measurement data at input layer 1915 and output an approximation 1200’ of a representation of topological structures that arise in the patterns of activity in a source neural network.
  • Teacher module 2820 is a device that is configured to interpret the measurements received from sensor(s) 2815 and provide a reward and/or a regret to deep neural network 2805. Rewards are positive and indicate successful control of the mechanism or system. Regrets are negative and indicate unsuccessful or less than optimal control. In general, teacher module 2820 also provides a characterization of the measurements along with the reward/regret for reinforcement learning. In general, the characterization of the
  • teacher module 2820 may read approximations 1200’ output from source approximator 1905 and pair the read approximations 1200’ with corresponding reward/regret values.
  • reinforcement learning does not occur in system 2800 in real time or during active control of actuator 2810 by deep neural network 2805. Rather, training feedback can be collected by teacher module 2820 and used for reinforcement training when deep neural network 2805 is not actively instructing actuator 2810.
  • teacher module 2820 can be remote from deep neural network 2805 and only in intermittent data communication with deep neural network 2805. Regardless of whether reinforcement learning is intermittent or continuous, deep neural network 2805 can be evolved, e.g., to optimize reward and/or reduce regret using the information received from teacher module 2820.
  • system 2800 also includes one or more sources 2825 of additional data.
  • Source approximator 1905 can also receive data from data sources 2825 at input layer 1915. In these instances, approximation 1200’ will result from processing both sensor data and the data from data sources 2825.
  • the data collected by one reinforcement learning system is the data collected by one reinforcement learning system
  • the characterization of the measurements along with the reward/regret values can be provided by teacher module 2820 to a data exchange system that collects such data from a variety of reinforcement learning systems and redistributes it among them. Further, as discussed above, the characterization of the measurements can be an approximation of a representation of topological structures that arise in the patterns of activity in a source neural network, such as approximation 1200’.
  • reinforcement learning system 2800 The particular operations that are performed by reinforcement learning system 2800 will of course depend on the particular operational context. For example, in contexts where source approximator 1905, deep neural network 2805, actuator 2810, and sensors 2815 are part of an automobile, deep neural network 2805 can perform object localization and/or detection operations while steering the automobile.
  • reward/regret values and approximations 1200’ that characterize the state of the environment when object localizations and/or detection operations were performed can be provided to the data exchange system.
  • the data exchange system can then distribute the reward/regret values and approximations 1200’ to other reinforcement learning systems 2800 associated with other vehicles for reinforcement learning at those other vehicles.
  • reinforcement learning can be used to improve object localization and/or detection operations at a second vehicle using the reward/regret values and approximations 1200’.
  • the operations that are learned at other vehicles need not be identical to the operations that are performed by deep neural network 2805.
  • reward/regret values that are based on travel time and approximations 1200’ that result from the input of sensor data characterizing, e.g., an unexpectedly wet road at a location identified by a GPS data source 2825 can be can be used for route planning operations at another vehicle.
  • Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
  • Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus.
  • program instructions can be encoded on an
  • a computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them.
  • a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal.
  • the computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).
  • the operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.
  • the term“data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing
  • the apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application- specific integrated circuit).
  • the apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them.
  • the apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.
  • a computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment.
  • a computer program may, but need not, correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
  • the processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output.
  • the processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
  • processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer.
  • a processor will receive instructions and data from a read-only memory or a random access memory or both.
  • the essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data.
  • a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks.
  • mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks.
  • a computer need not have such devices.
  • a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.
  • PDA personal digital assistant
  • GPS Global Positioning System
  • USB universal serial bus
  • Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices;
  • magnetic disks e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
  • the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
  • a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • a keyboard and a pointing device e.g., a mouse or a trackball
  • Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
  • a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to
  • representation 1200 is a binary representation in which each bit individually represents the presence or absence of a feature in a graph
  • representations of information are possible.
  • a vector or matrix of multi-valued, non-binary digits can be used to represent, e.g., the presence or absence of features and possibly other characteristics of those features.
  • An example of such a characteristic is the weight of edges with the activity that constitutes the features.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Robotics (AREA)
  • Image Analysis (AREA)
  • Multi Processors (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Mobile Radio Communication Systems (AREA)

Abstract

La présente invention concerne des procédés, des systèmes et un appareil, incluant des programmes informatiques codés sur un support de stockage informatique, destinés à caractériser l'activité dans un réseau neuronal artificiel récurrent et à coder et à décoder des informations. Selon un aspect, un procédé peut comprendre la caractérisation de l'activité dans un réseau neuronal artificiel. Le procédé est mis en œuvre par un appareil de traitement de données et peut comprendre l'identification de motifs de clique d'activité du réseau neuronal artificiel. Les motifs de clique d'activité peuvent renfermer des cavités.
EP19728962.2A 2018-06-11 2019-06-05 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations Pending EP3803699A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US16/004,635 US20190378007A1 (en) 2018-06-11 2018-06-11 Characterizing activity in a recurrent artificial neural network
US16/004,757 US11893471B2 (en) 2018-06-11 2018-06-11 Encoding and decoding information and artificial neural networks
US16/004,671 US11972343B2 (en) 2018-06-11 2018-06-11 Encoding and decoding information
US16/004,837 US11663478B2 (en) 2018-06-11 2018-06-11 Characterizing activity in a recurrent artificial neural network
US16/004,796 US20190378000A1 (en) 2018-06-11 2018-06-11 Characterizing activity in a recurrent artificial neural network
PCT/EP2019/064593 WO2019238483A1 (fr) 2018-06-11 2019-06-05 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations

Publications (1)

Publication Number Publication Date
EP3803699A1 true EP3803699A1 (fr) 2021-04-14

Family

ID=66776339

Family Applications (5)

Application Number Title Priority Date Filing Date
EP19728962.2A Pending EP3803699A1 (fr) 2018-06-11 2019-06-05 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations
EP19728989.5A Pending EP3803705A1 (fr) 2018-06-11 2019-06-06 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations
EP19728990.3A Pending EP3803706A1 (fr) 2018-06-11 2019-06-06 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations
EP19728992.9A Pending EP3803707A1 (fr) 2018-06-11 2019-06-06 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations
EP19728993.7A Pending EP3803708A1 (fr) 2018-06-11 2019-06-06 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations

Family Applications After (4)

Application Number Title Priority Date Filing Date
EP19728989.5A Pending EP3803705A1 (fr) 2018-06-11 2019-06-06 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations
EP19728990.3A Pending EP3803706A1 (fr) 2018-06-11 2019-06-06 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations
EP19728992.9A Pending EP3803707A1 (fr) 2018-06-11 2019-06-06 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations
EP19728993.7A Pending EP3803708A1 (fr) 2018-06-11 2019-06-06 Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations

Country Status (5)

Country Link
EP (5) EP3803699A1 (fr)
KR (5) KR102497238B1 (fr)
CN (5) CN112567387A (fr)
TW (1) TWI822792B (fr)
WO (5) WO2019238483A1 (fr)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11569978B2 (en) 2019-03-18 2023-01-31 Inait Sa Encrypting and decrypting information
US11580401B2 (en) 2019-12-11 2023-02-14 Inait Sa Distance metrics and clustering in recurrent neural networks
US11615285B2 (en) 2017-01-06 2023-03-28 Ecole Polytechnique Federale De Lausanne (Epfl) Generating and identifying functional subnetworks within structural networks
US11651210B2 (en) 2019-12-11 2023-05-16 Inait Sa Interpreting and improving the processing results of recurrent neural networks
US11652603B2 (en) 2019-03-18 2023-05-16 Inait Sa Homomorphic encryption
US11797827B2 (en) 2019-12-11 2023-10-24 Inait Sa Input into a neural network
US11816553B2 (en) 2019-12-11 2023-11-14 Inait Sa Output from a recurrent neural network
US11893471B2 (en) 2018-06-11 2024-02-06 Inait Sa Encoding and decoding information and artificial neural networks
US11972343B2 (en) 2018-06-11 2024-04-30 Inait Sa Encoding and decoding information

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11663478B2 (en) 2018-06-11 2023-05-30 Inait Sa Characterizing activity in a recurrent artificial neural network
US11610134B2 (en) * 2019-07-08 2023-03-21 Vianai Systems, Inc. Techniques for defining and executing program code specifying neural network architectures
TWI769466B (zh) * 2020-06-17 2022-07-01 台達電子工業股份有限公司 類神經網路系統及其操作方法
CN112073217B (zh) * 2020-08-07 2023-03-24 之江实验室 一种多网络结构差异向量化方法及装置
CN113219358B (zh) * 2021-04-29 2024-08-20 东软睿驰汽车技术（沈阳）有限公司 电池包健康状态计算方法、系统及电子设备
TWI769875B (zh) * 2021-06-24 2022-07-01 國立中央大學 深度學習網路裝置、其使用的記憶體存取方法與非揮發性儲存媒介
CN113626721B (zh) * 2021-10-12 2022-01-25 中国科学院自动化研究所 基于遗憾探索的推荐方法、装置、电子设备与存储介质

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7958071B2 (en) * 2007-04-19 2011-06-07 Hewlett-Packard Development Company, L.P. Computational nodes and computational-node networks that include dynamical-nanodevice connections
WO2011095342A1 (fr) * 2010-02-05 2011-08-11 Ecole Polytechnique Federale De Lausanne Epfl-Sri Organisation de réseaux neuronaux
US8170971B1 (en) * 2011-09-28 2012-05-01 Ava, Inc. Systems and methods for providing recommendations based on collaborative and/or content-based nodal interrelationships
US20180053114A1 (en) * 2014-10-23 2018-02-22 Brighterion, Inc. Artificial intelligence for context classifier
CN106156845A (zh) * 2015-03-23 2016-11-23 日本电气株式会社 一种用于构建神经网络的方法和设备
CN105095966B (zh) * 2015-07-16 2018-08-21 北京灵汐科技有限公司 人工神经网络和脉冲神经网络的混合计算系统
US20180005111A1 (en) * 2016-06-30 2018-01-04 International Business Machines Corporation Generalized Sigmoids and Activation Function Learning
US11544539B2 (en) * 2016-09-29 2023-01-03 Tsinghua University Hardware neural network conversion method, computing device, compiling method and neural network software and hardware collaboration system
US10748060B2 (en) * 2016-10-14 2020-08-18 Intel Corporation Pre-synaptic learning using delayed causal updates
CN107247989B (zh) * 2017-06-15 2020-11-24 北京图森智途科技有限公司 一种实时的计算机视觉处理方法及装置
CN107423814A (zh) * 2017-07-31 2017-12-01 南昌航空大学 一种采用深度卷积神经网络建立动态网络模型的方法

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11615285B2 (en) 2017-01-06 2023-03-28 Ecole Polytechnique Federale De Lausanne (Epfl) Generating and identifying functional subnetworks within structural networks
US11893471B2 (en) 2018-06-11 2024-02-06 Inait Sa Encoding and decoding information and artificial neural networks
US11972343B2 (en) 2018-06-11 2024-04-30 Inait Sa Encoding and decoding information
US11569978B2 (en) 2019-03-18 2023-01-31 Inait Sa Encrypting and decrypting information
US11652603B2 (en) 2019-03-18 2023-05-16 Inait Sa Homomorphic encryption
US12113891B2 (en) 2019-03-18 2024-10-08 Inait Sa Encrypting and decrypting information
US11580401B2 (en) 2019-12-11 2023-02-14 Inait Sa Distance metrics and clustering in recurrent neural networks
US11651210B2 (en) 2019-12-11 2023-05-16 Inait Sa Interpreting and improving the processing results of recurrent neural networks
US11797827B2 (en) 2019-12-11 2023-10-24 Inait Sa Input into a neural network
US11816553B2 (en) 2019-12-11 2023-11-14 Inait Sa Output from a recurrent neural network
US12020157B2 (en) 2019-12-11 2024-06-25 Inait Sa Interpreting and improving the processing results of recurrent neural networks

Also Published As

Publication number Publication date
TWI822792B (zh) 2023-11-21
CN112567388A (zh) 2021-03-26
CN112585621A (zh) 2021-03-30
KR102475411B1 (ko) 2022-12-07
KR102497238B1 (ko) 2023-02-07
EP3803706A1 (fr) 2021-04-14
KR20210008417A (ko) 2021-01-21
WO2019238523A1 (fr) 2019-12-19
KR20210010894A (ko) 2021-01-28
KR20210008419A (ko) 2021-01-21
CN112567390A (zh) 2021-03-26
EP3803708A1 (fr) 2021-04-14
KR102526132B1 (ko) 2023-04-26
EP3803707A1 (fr) 2021-04-14
KR102465409B1 (ko) 2022-11-09
KR20210008858A (ko) 2021-01-25
KR102488042B1 (ko) 2023-01-12
WO2019238513A1 (fr) 2019-12-19
CN112567387A (zh) 2021-03-26
WO2019238483A1 (fr) 2019-12-19
EP3803705A1 (fr) 2021-04-14
WO2019238512A1 (fr) 2019-12-19
KR20210008418A (ko) 2021-01-21
WO2019238522A1 (fr) 2019-12-19
CN112567389A (zh) 2021-03-26
TW202001693A (zh) 2020-01-01

Similar Documents

Publication Publication Date Title
US11972343B2 (en) Encoding and decoding information
US20240176985A1 (en) Encoding and decoding information and artificial neural networks
US11663478B2 (en) Characterizing activity in a recurrent artificial neural network
EP3803699A1 (fr) Caractérisation de l'activité dans un réseau neuronal artificiel récurrent et codage et décodage d'informations
US20190378000A1 (en) Characterizing activity in a recurrent artificial neural network
US20190378007A1 (en) Characterizing activity in a recurrent artificial neural network
CN105447498B (zh) 配置有神经网络的客户端设备、系统和服务器系统
Liu et al. Heterogeneous sensor data fusion by deep multimodal encoding
Liu et al. Deep fusion of heterogeneous sensor data
Jin et al. Improving the Performance of Deep Learning Model‐Based Classification by the Analysis of Local Probability
CN112766464A (zh) 基于时空聚合的灵活动态网络链路预测方法、系统及应用
Jeong Performance of Neural Computing Techniques in Communication Networks
Kabir BRB based deep learning approach with application in sensor data streams
Sánchez-Marono et al. Self-organizing maps and functional networks for local dynamic modeling.
Alhalabi Ensembles of Pruned Deep Neural Networks for Accurate and Privacy Preservation in IoT Applications
WO2023164166A1 (fr) Couches de bruit conditionnelles permettant de générer des exemples contradictoires
Polepalli Scalable Digital Architecture of a Liquid State Machine
Kang et al. Rp-Gcn: A Recurrence Plot Based Graph Convolutional Network for Time Series Classification

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20201203

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20230118