TWI822792B - Method of characterizing activity in an artificial nerual network, and system comprising one or more computers operable to perform said method - Google Patents

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for characterizing activity in a recurrent artificial neural network and encoding and decoding information. In one aspect, a method can include characterizing activity in an artificial neural network. The method is performed by data processing apparatus and can include identifying clique patterns of activity of the artificial neural network. The clique patterns of activity can enclose cavities.

Description

Methods for characterizing activity in artificial neural networks and systems including one or more computers capable of executing the methods

The present disclosure relates to activity characterization in recurrent artificial neural networks. This disclosure also relates to the encoding and decoding of information, as well as systems and techniques for using encoded information in a variety of environments.

The present disclosure relates to activity characterization in recurrent artificial neural networks. Activity characterization can be applied, for example, to the identification of decision moments, and to the encoding/decoding of messages in environments such as transmission, encryption, data storage, etc. This disclosure also relates to the encoding and decoding of information, as well as systems and techniques for using encoded information in a variety of environments. This encoded information can represent activity in a neural network, such as a recurrent neural network.

Artificial neural networks are devices inspired by the structural and functional aspects of biological neuronal networks. In particular, artificial neural networks use a system of interconnected structures called nodes to simulate the information encoding and other processing capabilities of biological neuronal networks. The connection distribution and connection strength between nodes in an artificial neural network determine the results of information processing or information storage of the artificial neural network.

Neural networks can be trained to generate ideal signal flows within the network and implement Achieve ideal information processing or information storage results. Typically during the learning phase, training a neural network will change the distribution and/or strength of connections between nodes. A neural network is considered to have been trained when it is able to output sufficiently suitable processing results for a specific set of inputs.

Artificial neural networks can be used in a variety of devices to perform nonlinear data processing and analysis. Nonlinear data processing does not satisfy the superposition principle, that is, the variable to be determined cannot be written as a linear sum of independent components. Nonlinear data processing is useful in specific contexts, such as pattern and sequence recognition, speech processing, novelty detection and sequential decision-making, complex systems modeling, and systems and techniques in a variety of other contexts.

Encoding and decoding both convert information from one form or representation to another. Different presentation methods provide different features that are more or less useful in different applications. For example, some forms or information presentation methods (such as natural language) can be easier for humans to understand. Some other forms or presentations may be smaller in size (eg, compressed) and more convenient for transmission or storage. Other forms or presentation methods can intentionally obscure information content (for example, information can be encrypted and encoded).

Regardless of the specific application method, the encoding or decoding process will usually follow a preset set of rules or algorithms, and this set of rules or algorithms can establish correspondences between information in different forms or presentation methods. For example, an encoding process that produces a binary (ie, binary) code may assign a role or meaning to each bit based on its position in a binary sequence or vector.

This disclosure describes techniques for characterization of activity in artificial neural networks.

For example, in one embodiment, a method may include characterizing activity in an artificial neural network. The method is performed by a data processing device and may include identifying clique patterns of activity in the artificial neural network. The active group patterns may surround multiple cavities.

This and other implementations may include one or more of the following features. The method may include defining windows of time, and the activity of the artificial neural network being responsive to an input to the artificial neural network during the time windows. The group patterns of activity can be identified in each of the time windows. The method may include identifying the first time window based on a distinguishable likelihood of the clique pattern of activity occurring during the first time window of the time windows. Identifying such clique patterns may include identifying multiple directed cliques of activity. Lower-dimensional directed groups existing in higher-dimensional directed groups can be discarded or ignored.

The method may include distinguishing the group modes into multiple categories, and characterizing the activity based on the number of occurrences of the group modes in each category. Distinguishing the clique modes may include distinguishing the clique modes based on a number of points in each clique mode. The method may include outputting from the recurrent artificial neural network a binary sequence consisting of the number zero and the number one. Each number in the binary sequence can indicate whether a corresponding activity pattern exists in the artificial neural network. The method may include constructing the artificial neural network by reading numbers output by the artificial neural network and evolving a structure of the artificial neural network. Whether the altered structure is ideal can be determined by iteratively altering the structure of the artificial neural network, characterizing the complexity of the pattern of activity in the altered structure, and using the characterization of the complexity of the pattern. An indicator to evolve the structure.

The artificial neural network may be a recursive artificial neural network. The method may include identifying multiple decision moments in the recurrent artificial neural network based on the complexity of identifying patterns of activity in the recurrent artificial neural network. Identifying the decision moments may include: identifying a point in time of an activity that has a distinguishable complexity compared to other activities in response to the input; and based on the distinguishable complexity of the activity. This time point in the activity is used to identify these decision-making moments. The method may include: inputting a data stream into the recurrent artificial neural network; and identifying the clique patterns of activity when inputting the data stream. The method may include evaluating whether the activity is responsive to the input to the artificial neural network. Evaluating whether the activity is responsive to the input to the artificial neural network may include evaluating whether a pattern of activity relatively early and relatively simple after the input event is responsive to the input, and whether a pattern of activity relatively early and relatively simple after the input event is responsive to the input. A complex activity pattern is not responsive to the input; and a relatively late and relatively complex activity pattern is evaluated to be responsive to the input, while a relatively early and relatively complex activity pattern is not responsive to the input event. to this input.

In another implementation, a system may include one or more computers that may perform multiple operations. The operations may include characterizing activity in the artificial neural network and identifying clique patterns of activity in the artificial neural network, where the clique patterns of activity surround cavities. The operations may include defining time windows during which the activity of the artificial neural network is responsive to an input to the artificial neural network. The group patterns of activity can be identified within the time windows. The operations may include identifying a first time window of the time windows based on a distinguishable probability of the clique pattern of activity, and the clique pattern of activity occurs in the first time window. Identifying such clique patterns may include discarding or ignoring lower dimensional directed cliques that exist within higher dimensional directed cliques. The operations may include constructing the artificial neural network, and constructing The artificial neural network includes: reading the numbers output by the artificial neural network; and evolving a structure of the artificial neural network. By iteratively changing the structure of the artificial neural network, characterizing the complexity of the activity patterns in the structure, and using the characterization of the complexity of the patterns to indicate whether the changed structure ideal to evolve the structure. The artificial neural network may be a recursive artificial neural network. The method may include identifying a plurality of decision moments in the recurrent artificial neural network based on identifying the complexity of activity patterns in the recurrent artificial neural network. Identifying the decision moments may include: identifying a point in time of an activity that has a distinguishable complexity compared to other activities that respond to the input; and based on the activity having the distinguishable complexity point in time to identify these decision-making moments. The operations may include inputting a data stream into the recurrent artificial neural network and identifying clique patterns of activity when inputting the data stream. The operations may include evaluating whether the activity is responsive to the input to the artificial neural network. Evaluating whether the activity is responsive to the input to the artificial neural network may include evaluating a pattern of activity that is relatively early and relatively simple after the time of the input in response to the input, and that is relatively early and relatively early after the time of the input. A relatively complex activity pattern is not responsive to the input; and a relatively late and relatively complex activity pattern is evaluated relatively late after the time of the input, whereas a relatively early and relatively complex activity pattern is evaluated relatively early after the time of the input. Active mode does not respond to this input.

As another example, a method of identifying multiple decision moments in an artificial neural network includes identifying the complexity of patterns of activity in the recurrent artificial neural network, where the activity is responsive to the artificial neural network's An input identifies a point in time of an activity; identifies a point in time of an activity that has a distinguishable complexity compared to other activities that respond to the input; and the activity based on the activity that has the distinguishable complexity point in time to identify these decision-making moments.

As another example, a method for characterizing activity in a recurrent artificial neural network The method involves identifying predefined multiple clique patterns of activity in recurrent artificial neural networks. The method is executed by a data processing device. As another example, the method may include outputting from a recurrent artificial neural network a binary sequence consisting of the number zero and the number one, wherein each number in the binary sequence represents the recurrent artificial neural network Whether a specific node group in a node displays a corresponding activity mode.

As another example, a method of constructing a recurrent artificial neural network may include characterizing the complexity of one of multiple patterns of activity that may occur in the recurrent artificial neural network. The network consists of a structured set of nodes and the connections between nodes; and the structure of the recurrent artificial neural network is evolved to increase the complexity of multiple modes of activity. For example, the construction method can also be applied as part of a method for training the recurrent artificial neural network.

Other examples of the above embodiments include corresponding systems, devices, and computer programs for executing steps of the method written on a computer storage device.

Certain embodiments of the disclosure may be implemented to achieve one or more of the advantages described below. For example, traditional data processing devices (such as digital computers and other computers) are programmed to follow a predefined logical sequence when processing information. It is relatively easy to recognize when a computer operation produces a result. That is, the completion of the logical sequence embodied in programming indicates that the information processing has been completed and the computer has "made a decision." The results may be stored in a relatively permanent form (eg, a memory device, a set of buffers, etc.) at the output of the computer's data processor and may be accessed for a variety of purposes.

Instead, as described in this article, based on the dynamics of neural networks during information processing Characteristics of characteristics to identify decision moments in recurrent artificial neural networks. Decision-making moments in an artificial neural network can be identified based on the characteristics of the functional states of the artificial neural network while performing information processing, rather than by waiting for the artificial neural network to reach a predefined end in a logical sequence.

In addition, the dynamic properties of recurrent artificial neural networks (including activity characteristics consistent with clique mode and directed clique mode) during information processing can be used in various signaling operations. , including signal transmission, encoding, encryption and storage. Particularly during information processing, the characteristics of activity in a RANN reflect the input and can be viewed as an encoded form of the input (i.e., the "output" of the RANN during the encoding process) . For example, these features can be sent to a remote receiver, which can decode the features to reconstruct the input or a portion of the input.

In addition, in some cases, activities in different node groups in a recurrent artificial neural network (for example, activities consistent with clique patterns and directed clique patterns) can be expressed as "0" and "1" A binary sequence in which each number indicates whether the activity conforms to a pattern. Since activity can in some cases be the output of a recurrent artificial neural network, the output of a recurrent artificial neural network can be represented as a vector of binary numbers and is compatible with digital data processing.

In addition, in some cases, this characterization of the dynamics of recurrent artificial neural networks can be used before and/or during training to increase the probability of the emergence of complex activity patterns during information processing. For example, the connections between nodes in a recurrent neural network can be intentionally evolved before or during training to increase the complexity of the activity pattern. For example, the connections between nodes in a recursive artificial neural network can be intentionally evolved to increase the probability of clique and directed clique patterns appearing during information processing. In this way, the time required to train recurrent artificial neural networks can be reduced and energy.

As another example, this characterization of the dynamics of a recurrent artificial neural network can be used to confirm how well the recurrent neural network is trained. For example, a recurrent artificial neural network that displays a specific type of ordering in its activities (e.g., clique pattern and directed clique pattern) can be considered to be better trained than a recursive artificial neural network that does not display that particular type of ordering. deeper. Indeed, in some cases the extent of training can be quantified by quantifying the degree of ordering of activities in a recurrent artificial neural network.

For example, methods for identifying multiple decision moments in a neural network include: identifying the complexity of patterns of activity in a recurrent artificial neural network in response to inputs to the recurrent artificial neural network; identifying a point in time of the activity having a distinguishable complexity compared to other activities responsive to the input; and identifying the decision moments based on the point in time of the activity having a distinguishable complexity.

As another example, a method for characterizing activity in a recurrent artificial neural network includes identifying clique patterns of activity in the recurrent artificial neural network. The method is executed by the data processing device.

As another example, a method may include outputting a binary sequence of zeros and ones from a recurrent artificial neural network, where each number in the sequence represents whether a particular group of nodes in the recurrent artificial neural network displays a corresponding pattern of activity .

As another example, a method of constructing a recurrent artificial neural network may include characterizing the complexity of patterns of activity that may occur in a recurrent artificial neural network that contains multiple a structured collection of nodes and the links between the nodes; and evolving the structure of a recurrent artificial neural network to increase the complexity of patterns of activity. this construction The method may also be used as part of a method for training a recurrent artificial neural network, for example.

Other examples of the implementations include corresponding systems, devices, and computer programs configured to perform the steps of a method encoded on a computer storage device.

One or more of the following advantages may be realized by implementing specific embodiments of the present disclosure. For example, traditional data processing devices, such as digital and other types of computers, are programmed to follow predefined logical sequences when processing information. Therefore, the moment when the computer calculates the result is relatively easy to identify. That is, the completion of the logical sequence contained in the programming indicates when information processing is complete and indicates that the computer has "made a decision." The results calculated by the computer can be stored in a relatively long-lived form at the output of the computer data processor (eg, a storage device, a set of buffers, etc.) and accessed for various purposes.

Conversely, as described in this article, decision moments in recursive artificial neural networks can be identified based on characteristics of the neural network's dynamic properties during information processing. Decision-making moments in an artificial neural network can be identified based on characteristics of the functional state of the artificial neural network when information processing is performed, rather than by waiting for the artificial neural network to reach a predefined end of a logical sequence.

In addition, the dynamic characteristics of recurrent artificial neural networks during information processing (including activity characteristics consistent with clique mode and directed clique mode) can be used in various signaling operations, including signal transmission and encoding. , encryption and storage. Particularly during information processing, the characteristics of activity in a RANN reflect the input and can be viewed as an encoded form of the input (i.e., the "output" of the RANN during the encoding process) . For example, these features can be sent to a remote receiver, which can decode the features to reconstruct the input or a portion of the input.

In addition, in some cases, activities in different node groups in a recurrent artificial neural network (for example, activities consistent with clique patterns and directed clique patterns) can be expressed as "0" and "1" A binary sequence in which each number indicates whether the activity conforms to a pattern. Since activity can in some cases be the output of a recurrent artificial neural network, the output of a recurrent artificial neural network can be represented as a vector of binary numbers and is compatible with digital data processing.

In addition, in some cases, this characterization of the dynamics of recurrent artificial neural networks can be used before and/or during training to increase the probability of the emergence of complex activity patterns during information processing. For example, the connections between nodes in a recurrent neural network can be intentionally evolved before or during training to increase the complexity of the activity pattern. For example, the connections between nodes in a recursive artificial neural network can be intentionally evolved to increase the probability of clique and directed clique patterns appearing during information processing. In this way, the time and effort required to train recurrent artificial neural networks can be reduced.

As another example, this characterization of the dynamics of a recurrent artificial neural network can be used to confirm how well the recurrent neural network is trained. For example, a recurrent artificial neural network that displays a specific type of ordering in its activities (e.g., clique pattern and directed clique pattern) can be considered to be better trained than a recursive artificial neural network that does not display that particular type of ordering. deeper. Indeed, in some cases the extent of training can be quantified by quantifying the degree of ordering of activities in a recurrent artificial neural network.

As another example, in one embodiment, a device includes a neural network trained to generate a first representation of topology in a pattern of activity in response to a first input. An approximation of , where the activity occurs in response to the first input in a source neural network. The neural network is also trained An approximation of a second representation of a topology in a pattern that produces an activity in the source neural network in response to the second input. The neural network is further trained to generate an approximation of a third representation of topology in a pattern of activity in response to a third input, wherein the activity is in the source neural network in response to the third input And appear.

This and other implementations may include one or more of the following features. The topology may consist entirely of two or more nodes in the source neural network and one or more edges between the nodes. Topology can contain simplices. Topology can surround cavities. Each of the first representation, the second representation, and the third representation may represent a topology that occurs in the source neural network, and the topology occurs only when the patterns of activity have characteristics that are distinguishable from responses to the respective inputs to the complexity of other activities. The device may also include a processor coupled to receive an approximation of the representation generated by the neural network device and to process the received approximation. The processor may include a second neural network, and the second neural network has been trained to process the representation generated by the neural network. Each of the first representation, the second representation, and the third representation may include multi-valued and non-binary numbers. Each of the first representation, the second representation, and the third representation may represent the occurrence of a topological structure without specifying where in the source neural network the activity pattern occurs. The device may include a smartphone. The source neural network may be a recurrent neural network.

In another embodiment, a device includes a neural network coupled to an input representation of a topology in a pattern of activity that occurs in a source neural network in response to a plurality of different inputs . The neural network is trained to process the representations and produce corresponding outputs.

This and other implementations may include one or more of the following features. The topology may consist entirely of two or more nodes in the source neural network and one or more edges between the nodes. Topology can contain monoliths. Such representations of topology may represent topologies that occur in the source neural network only when patterns of activity have a complexity that is distinguishable from other activities in response to respective inputs. The device may also include a neural network trained to generate multiple representations of the multiple topologies in patterns of activity in response to multiple different inputs, the activity being represented in the source neural network appear in response to these different inputs. Such representations may contain multi-valued and non-binary numbers. These representations can represent the occurrence of topological structures without specifying where in the source neural network the pattern of activity occurs. The source neural network may be a recurrent neural network.

In another embodiment, a method is implemented by a neural network device and includes: inputting a representation of a topology of a pattern of activity in a source neural network, wherein the activity is responsive to the input to the source neural network; Process the representation; and output the processing results of the representation. This processing is consistent with the training of the neural network to handle different representations of the topology in the activity patterns in the source neural network.

This and other implementations may include one or more of the following features. The topology may consist entirely of two or more nodes in the source neural network and one or more edges between the nodes. Topology can contain monoliths. Topology can surround cavities. Such representations of topology may represent topologies that occur in the source neural network only when patterns of activity have a complexity that is distinguishable from other activities in response to respective inputs. Such representations may contain multi-valued and non-binary numbers. These representations can represent the occurrence of topological structures without specifying where in the source neural network the pattern of activity occurs. The source neural network may be a recurrent neural network.

As another example, in one embodiment, a device includes a neural network coupled to an input representation of a topology in a pattern of activity that occurs in a source neural network in response to a plurality of different inputs. appear on the Internet. The neural network is trained to process the representations and produce corresponding outputs.

This and other implementations may include one or more of the following features. The topology may consist entirely of two or more nodes in the source neural network and one or more edges between the nodes. The device may include an actuator coupled to receive response output from the neural network and act on a real or virtual environment; a sensor coupled to measure characteristics of the environment; and a teacher module , which is configured to interpret measurements received from the sensors and provide rewards and/or regrets to the neural network. Topology can contain monoliths. Topology can surround cavities. Such representations of topology may represent topologies that occur in the source neural network only when patterns of activity have a complexity that is distinguishable from other activities in response to respective inputs. The device may include a second neural network trained to generate approximations of each of the plurality of representations of the plurality of topologies in the pattern of activity in the source neural network in response to the plurality of different inputs. The path appears in response to these different inputs. The device may also include an actuator coupled to receive response output from the neural network and act on a real or virtual environment, and a sensor coupled to measure characteristics of the environment. The second neural network can be trained to produce a corresponding approximation in response, at least in part, to the measured environmental characteristics. The device may also include a teacher module configured to interpret measurements received from the sensors and provide rewards and/or regrets to the neural network. Such representations of topology may contain multi-valued and non-binary numbers. These representations can represent the occurrence of topological structures without specifying where in the source neural network the pattern of activity occurs. The device can be a smart phone. The source neural network can be A return neural network.

In another embodiment, a method implemented by one or more data processing devices may include: receiving a training set containing a plurality of topologies in a plurality of patterns of activity in a source neural network. A plurality of representations; and training a neural network by using the representations as an input to the neural network or as a target answer vector. The activity is responsive to an input to the source neural network.

This and other implementations may include one or more of the following features. The topology may consist entirely of two or more nodes in the source neural network and one or more edges between the nodes. The training set can contain multiple input vectors, one for each representation. Training the neural network may include training the neural network by treating each of the representations as a target answer vector. Training the neural network may include training the neural network by taking each of the representations as input. The training set can contain multiple reward or regret values. Training the neural network may include reinforcement learning. Topology can contain monoliths. Such representations of topology may represent topologies that occur in the source neural network only when patterns of activity have a complexity that is distinguishable from other activities in response to respective inputs. Such representations of topology may contain multi-valued and non-binary numbers. Such representations of topology may represent the occurrence of topology without specifying where in the source neural network the pattern of activity occurs. The source neural network may be a recurrent neural network.

The details of one or more implementations described in the disclosure are set forth in the drawings and the description below. Other features, embodiments, and advantages of the present invention will become apparent by reference to the specification, drawings, and claims.

As follows:

100:Recursive Artificial Neural Network Device

101, 102, 103, 104, 105, 106, 107: nodes

110:Link

400:Process

405, 410, 415, 420, 425, 430: steps

500:Mode

505, 510, 515, 520, 525, 530: Mode

600:Mode

605, 610: Mode

700:Mode

705, 710: Mode

800:Data table

805, 810: column

905: Chart

906, 907, 908, 909: vertical lines

910: Chart

915, 920, 925: dashed rectangle

930:peak

935:peak

940:bottom line

1000:Process

1005, 1010, 1015, 1020: steps

1100:Process

1105, 1110: steps

1200: indicates

1200’: Approximately

1205, 1207, 1211, 1293, 1294, 1297: bits

1500:Sub-picture

1505, 1510, 1515, 1520: nodes

1525, 1530, 1535, 1540, 1545, 1550: side

1600:Sub-picture

1605, 1610, 1615, 1620: nodes

1625, 1630, 1635, 1640, 1645: side

1700:Classification system

1705: Source Neural Network Device

1710: Linear classifier

1715:Input layer

1720:Input

1725:Output

1800:Classification system

1810: Neural Network Classifier

1820:Input layer

1825:Output layer

1900:Classification system

1905: Source Approximator

1915:Input layer

1920:Output layer

2000: Classification system

2100:Edge device

2110: Optical imaging system

2115:Image processing electronic devices

2120: Source Approximator

2125: represents the classifier

2130: Communication Controllers and Interfaces

2135:Data port

2200: Edge device

2215:Image processing electronic devices

2225: Indicates the classifier

2230: Communication Controllers and Interfaces

2235:Data port

2240: Sensor

2245:Multiple input source approximators

2300:System

2305:Regional Neural Network Device

2310:Telephone base station

2315:Wireless access point

2320:Server system

2325:Data communication network

2400:System

2410:Linear Processor

2420:Input

2425:Output

2500:System

2510:Neural Network

2520:Input layer

2525:Output layer

2600:System

2700:System

2800: Reinforcement Learning Systems

2805:Deep Neural Networks

2810: Actuator

2815: Sensor

2820:Teacher Module

2825:Source

2830:Environment

Figure 1 illustrates a schematic diagram of the structure of a recursive artificial neural network device.

Figures 2 and 3 illustrate schematic diagrams of the functions of the recurrent artificial neural network device in different time windows.

Figure 4 illustrates a flowchart of a process for identifying decision moments in a recurrent artificial neural network based on characterization of activity in the network.

Figure 5 illustrates a schematic diagram of an activity pattern that can be identified and used to identify decision moments in a recurrent artificial neural network.

Figure 6 illustrates a schematic diagram of an activity pattern that can be identified and used to identify decision moments in a recurrent artificial neural network.

Figure 7 illustrates a schematic diagram of an activity pattern that can be identified and used to identify decision moments in a recurrent artificial neural network.

FIG. 8 illustrates a schematic diagram of a data table that can be used to confirm the complexity of an activity pattern or the degree of ordering in an activity pattern in a recurrent artificial neural network device.

Figure 9 illustrates a schematic diagram identifying time points for activity patterns with distinguishable complexity.

Figure 10 illustrates a flowchart of a process for encoding signals using a recurrent artificial neural network based on characterization of activity in the network.

Figure 11 illustrates a flowchart of a process for decoding signals using a recurrent artificial neural network based on characterization of activity in the network.

Figures 12, 13 and 14 illustrate schematic diagrams of binary forms or representations of topological structures.

Figures 15 and 16 illustrate the existence of features corresponding to different bits. A schematic diagram of how (presence) or absence (absence) are not independent of each other.

Figures 17, 18, 19 and 20 illustrate schematic representations of the occurrence of topological structures in activity in neural networks using four different classification systems.

Figures 21 and 22 illustrate schematic diagrams of edge devices including regional artificial neural networks that can emerge through the use of topologies that correspond to activity in the source neural network. Means to train.

Figure 23 illustrates a schematic diagram of a system in which a regional neural network can be trained by using representations that correspond to the occurrence of topological structures of activity in the source neural network.

Figures 24, 25, 26 and 27 illustrate diagrams using representations of the emergence of topological structures in activity in neural networks in four different systems.

Figure 28 illustrates a schematic diagram of a system including an artificial neural network that can be trained by using representations that correspond to the occurrence of topological structures of activity in the source neural network.

In the drawings, the same reference symbols represent the same components.

Figure 1 illustrates a schematic diagram of the structure of a recursive artificial neural network device 100. The recursive artificial neural network device 100 is a device that uses a system of interconnected nodes to simulate information encoding and other processing capabilities in a network composed of biological neurons. The recursive artificial neural network device 100 may be implemented in hardware, software, or a combination thereof.

The illustration of a recurrent artificial neural network device 100 includes a plurality of nodes 101 , 102 , . . . , 107 interconnected by a plurality of structural links 110 . Nodes 101, 102, ..., 107 are similar to the The discrete information processing structure of neurons in the Internet of Things. Nodes 101 , 102 , . . . , 107 typically process one or more input signals received on one or more of the links 110 to produce one or more outputs output on one or more of the links 110 signal. For example, in some embodiments, nodes 101, 102, ..., 107 may be artificial neurons that weight and sum multiple input signals through one or more nonlinear activation functions. ) transmits the required sum and outputs one or more output signals.

Nodes 101, 102, ..., 107 can be used as an accumulator. For example, nodes 101, 102, ..., 107 may operate according to an integrated-and-fire model, where one or more signals are accumulated in the first node until a threshold is reached. After reaching the threshold, the first node triggers by sending an output signal along one or more links 110 to the connected second node. Then, the second node 101, 102, ..., 107 accumulates the received signal, and if the accumulated signal reaches the threshold, the second node 101, 102, ..., 107 sends another output signal to another connected node.

Structural link 110 is a link capable of transmitting signals between nodes 101, 102, ..., 107. For convenience, all structural links 110 may be considered herein as the same bidirectional link carrying a signal from a first of nodes 101, 102, ..., 107 to nodes 101, 102 ,..., the second node in 107 in the same manner as the method of transmitting the signal of the second node to the first node. However, not all structural links 110 need to be considered bidirectional links. For example, part or all of structural link 110 may be a one-way link that carries a signal from a first of nodes 101, 102, . . . , 107 to nodes 101, 102, .. ., the second node in 107 without transmitting the signal from the second node to the first node.

As another example, in some embodiments, structural link 110 may have a structure other than attributes other than directionality. For example, in some embodiments, different structural connections 110 may carry signals of different sizes, which results in different connection strengths between corresponding nodes among the nodes 101, 102, ..., 107. As another example, different structural connections 110 may carry different types of signals (eg, inhibitory signals and/or excitatory signals). Indeed, in some embodiments, structural connections 110 may model connections between somatic cells in biological systems and reflect at least a portion of their morphological, chemical, and other diversity.

In the embodiment shown, the recurrent artificial neural network device 100 is a clique network (or a sub-network) in which each node 101, 102, ..., 107 is connected to each Other nodes 101, 102, ..., 107. However, this is not a limitation, that is, in some implementations, each node 101, 102, ..., 107 may be connected to an appropriate subset of nodes 101, 102, ..., 107 (perhaps via The same link or a different link, as the case may be).

For clarity of illustration, a recurrent artificial neural network device 100 with only seven nodes is shown here. Typically real-world neural network devices will contain many more nodes. For example, in some implementations, a neural network device may contain hundreds of thousands, millions, or even billions of nodes. Therefore, the recurrent artificial neural network device 100 can be a part (ie, a sub-network) of a larger recurrent artificial neural network.

In biological neural network devices, the process of accumulation and signal transmission requires the passage of real-world time. For example, the soma of a neuron integrates input received over time, and the transmission of signals from neuron to neuron requires time determined by, for example, the speed of signal transmission and the nature and length of the connections between neurons. Therefore, the state of a biological neural network device is dynamic and changes over time.

In artificial recurrent neural network devices, time is artificial and mathematical results are used represented by structure. For example, when a signal passes from one node to another, its time can be expressed in artificial units that generally have nothing to do with the passage of time in the real world, such as computer clock cycles or other units. However, because the artificial recurrent neural network device changes relative to these artificial units, its state can be described as "dynamic."

It should be noted that, for ease of explanation, these man-made units are referred to in this disclosure as "time" units. However, it should be understood that these units are artificial and generally do not correspond to the passage of time in the real world.

Figures 2 and 3 illustrate schematic diagrams of functions of the recurrent artificial neural network device 100 in different time windows. Because the state of the recurrent artificial neural network device 100 is dynamic, the signaling activity that occurs within a time window can be used to represent the functionality of the recurrent artificial neural network device 100 . Such functional depictions typically show activity in only a portion of link 110 . Specifically, not every link 110 is shown in these illustrations as contributing to the functionality of the recurrent artificial neural network device 100 because typically not every link 110 transmits signals within a specific time window.

In Figures 2 and 3, an active link 110 is shown as a relatively thick solid line connecting a pair of nodes 101, 102, ..., 107. In contrast, inactive links 110 are shown as dashed lines. This presentation is for illustration only. In other words, the structural link formed by link 110 exists regardless of whether link 110 is active or not. However, this representation highlights the activity and functionality of the recurrent artificial neural network device 100 .

In addition to schematically showing the presence of activity along a link, the direction of the activity is also schematically shown. Specifically, the relatively thick solid line used to show that link 110 is active also contains arrows that represent the direction of signal transmission along the link during the relevant time window. Generally In other words, the direction of signal transmission in a single time window cannot limit a link to a unidirectional link with the same directivity as the direction of transmission. Conversely, in a first functional graph of a first time window, a link can be active in a first direction, and in a second functional graph of a second time window, the link can be active in the opposite direction. active in the direction. However, in some cases, such as in RANN devices 100 that specifically include unidirectional connections, the directionality of signal transmission will ultimately dictate the directionality of the connection.

In a feedforward neural network device, messages only move in a single direction (i.e., forward) to the output layer of nodes located at the end of the network. In a feedforward neural network device, the propagation of signals through the network to the output layer indicates that a "decision" has been made and information processing has been completed.

In contrast, in a recurrent neural network, connections between nodes form loops, and activity in the network occurs dynamically without easily identifiable decisions. For example, even in a recurrent neural network with three nodes, a first node can send a signal to a second node, and the second node can respond by sending a signal to a third node. The third node can also respond by sending the signal back to the first node. The signal received by the first node may be at least partially responsive to a signal sent from the same node.

The schematic functional diagrams in Figures 2 and 3 illustrate this through a network that is only slightly larger than a three-node recurrent neural network. The functional diagram shown in Figure 2 may illustrate activity within a first time window, while Figure 3 may illustrate activity within a second time window immediately following the first time window. As shown, the set of signaling activity originates from node 104 and proceeds in a generally clockwise direction through the recursive artificial neural network device 100 during the first time window. During the second time window, at least some signaling activity is returned to node 104. Even in this simple illustration, the signal transmission does not occur in a way that produces a clearly identifiable output or end.

When considering a recurrent neural network with, for example, thousands or more nodes, it can be seen that signal propagation can occur over a large number of paths, and these signals lack a clearly identifiable "output" location or time. Although networks can be designed to return to a quiescent state with only background activity or even no signaling activity, the quiescent state itself does not represent the result of information processing. Recurrent neural networks always return to a resting state regardless of the input. Thus, the "output" or result of information processing is encoded into the activity that occurs within a recurrent neural network in response to a specific input.

Figure 4 illustrates a flow diagram of a process 400 for identifying decision moments in a recurrent artificial neural network based on characterization of activity in the network. A decision moment is the point in time when the activity in a recursive artificial neural network directs the network to respond to the input information processing results. Process 400 may be performed by a system of one or more data processing devices that perform operations based on the logic of one or more sets of machine-readable instructions. For example, process 400 may be executed by a system that is the same system as one or more computers executing software for implementing the recurrent artificial neural network used in process 400.

The system executing process 400 receives notification (labeled 405) that a signal has been input to the recurrent artificial neural network. In some cases, the input of the signal is a discrete injection event, for example, in which information is injected into one or more nodes and/or one or more parts of the neural network. Multiple links in progress. In other cases, the input of the signal is a stream of information injected into one or more nodes and/or connections of the neural network over a period of time. This notification indicates that the artificial neural network is actively processing information and is not, for example, in a quiescent state. In certain situations (for example: when the neural network exits a recognizable quiescent state), notifications can be received from the neural network itself.

The system executing process 400 divides the response activities in the network into time window sets (labeled 410). When a discrete event is injected, the time window can subdivide the time from injection to return to quiescence into periods in which the activity exhibits variable complexity. When the injection is an information stream, the duration of the injection (and, optionally, the time required to return to the quiescent state after the injection is completed) can be subdivided into multiple time windows, and within these time windows During this period, the activity exhibits variable complexity. Various methods of identifying activity complexity are discussed further below.

In some embodiments, the durations of all time windows are the same. However, this is not a limitation. That is, in some embodiments, the durations of the time windows may be different. For example, in some embodiments, the duration of the time window may increase with the time at which discrete injection events occur.

In some embodiments, the time windows may be a series of consecutive and independent time windows. In certain other implementations, the time windows overlap in time such that one time window begins before the previous time window ends. In some cases, these time windows may be time windows that can be moved at any time.

In some embodiments, different durations of the time window are defined for different validation methods of activity complexity. For example, a time window may be relatively longer for a pattern that defines activity that occurs between a relatively larger number of nodes than for a pattern that defines activity that occurs between a relatively smaller number of nodes. duration. For example, in pattern 500 (as shown in Figure 5), the time window defined for identifying activities consistent with pattern 530 may be longer than the time window defined for identifying activities consistent with pattern 505. .

The system executing process 400 identifies activities in the network within different time windows. mode (labeled 415). As discussed further below, patterns of activity can be identified by viewing the functional graph as a topological space and the nodes as points. In some embodiments, the identified patterns of activity are cliques in a functional graph of the network, such as directed cliques.

The system executing process 400 identifies the complexity of the pattern of activity in different time windows (labeled 420). Complexity can be a measure of the probability that an ordered pattern of activity occurs within a time window. Therefore, randomly occurring patterns of activity will be relatively simple. On the other hand, activity patterns that occur in a non-random order will be relatively complex. For example, in some embodiments, the complexity of an activity pattern can be measured using, for example, simplex counts or Betti numbers of the activity pattern.

The system executing process 400 identifies a point in time (labeled 425) that has an activity pattern of distinguishable complexity. Specific activity patterns may be distinguished based on the complexity of upward or downward deviations (eg, relative to a fixed or variable baseline). In other words, time points can be identified that display a non-randomly ordered pattern of activity in a particularly high or low degree of activity.

For example, when the signal input is a discrete injection event, such as deviation from a stable baseline or deviation from a curve characteristic of the neural network's average response to various discrete injection events can be used to determine the distinguishable complex The time point of active mode. As another example, when the signal input is an information stream, large changes in complexity during the transmission of the information stream can be used to determine the time points at which distinguishable patterns of complex activity occur.

The system executing process 400 schedules the reading of the output of the neural network based on time points of distinguishable complex activity patterns (labeled 430). For example, in some embodiments, the output of a neural network can be read while distinguishable complex patterns of activity occur. In certain embodiments, when a deviation in complexity exhibits a relatively high degree of non-random ordering in activities, the observed The activity pattern itself can also be used as the output of a recurrent artificial neural network.

Figure 5 illustrates a schematic diagram of a pattern 500 of activity that can be identified and used to identify decision moments in a recurrent artificial neural network. For example, pattern 500 may be identified at step 415 in flow 400 shown in FIG. 4 .

Schema 500 is a schematic diagram of activity within a recursive artificial neural network. During application of mode 500, the functional graph is viewed as a topological space, and the nodes of the functional graph are viewed as points. Activities within nodes and links consistent with pattern 500 may be identified as ordered regardless of the identity of the specific nodes and/or links participating in the activity. For example, when point 0 in the first pattern 505 is taken to be node 104, point 1 is taken to be node 105, and point 2 is taken to be node 101, the first pattern 505 can represent the nodes in Figure 2 Activities between 101, 104 and 105. For another example, when point 0 in the first pattern 505 is regarded as node 106, point 1 is regarded as node 104, and point 2 is regarded as node 105, the first pattern 505 can also represent the node in Figure 3 Activities between nodes 104, 105 and 106. The order of activities in the directed group is also specified. For example, in pattern 505, activity between points 1 and 2 occurs after activity between points 0 and 1.

In the embodiment shown, the patterns 500 are all directed groups or directed monomers. In this pattern, activity originates from a source node that sends signals to every other node in the pattern. In pattern 500, such source node is designated as point 0, while other nodes are designated as points 1, 2, . . . In addition, in a directed clique or a single clique, one of the nodes acts as a sink node and receives signals sent from every other node in the pattern. In pattern 500, such a sink node is designated as the highest numbered point in the pattern. For example, in pattern 505, the sink node is designated point 2. In mode 510, the sink node is designated point 3. In mode 515, the sink node is designated as point 4, and so on. Therefore, the activities represented by pattern 500 are ordered in a distinguishable manner.

Each pattern 500 has a different number of points and reflects ordered activity in a different number of nodes. For example, pattern 505 is a two-dimensional (2D) cell and reflects activity in three nodes, pattern 510 is a three-dimensional (3D) cell and reflects activity in four nodes, and so on. . As the number of points in the pattern increases, so does the degree of sequencing and complexity of the activities. For example, for a large set of nodes that have a certain degree of random activity within a time window, some of the activity may coincide with pattern 505. However, the probability that the random activities are consistent with patterns 510, 515, 520, ... respectively, will gradually decrease. The presence of activities consistent with pattern 530 represents a higher degree of ordering and complexity of the activities than when there are activities consistent with pattern 505 .

As mentioned above, in some embodiments, time windows with different durations may be defined due to different confirmation methods of activity complexity. For example, when activities consistent with pattern 530 are to be identified, a time window with a longer duration may be used than when activities consistent with pattern 505 are identified.

Figure 6 illustrates a schematic diagram of a pattern 600 of activity that can be identified and used to identify decision moments in a recurrent artificial neural network. For example, pattern 600 may be identified at step 415 in flow 400 of FIG. 4 .

Similar to pattern 500, pattern 600 is a schematic diagram of activity within a recursive artificial neural network. However, pattern 600 differs from the strict ordering of pattern 500 in that pattern 600 is not entirely a directed group or directed monomer. Specifically, modes 605 and 610 have lower directivity than mode 515. In addition to this, Mode 605 has no sink nodes at all. However, patterns 605 and 610 show a degree of orderly activity that exceeds that expected through random chance events, and can be used to confirm the complexity of activity in recurrent artificial neural networks.

Figure 7 illustrates a schematic diagram of a pattern 700 of activity that can be identified and used to identify decision moments in a recurrent artificial neural network. For example, pattern 700 may be identified at step 415 in flow 400 of FIG. 4 .

Pattern 700 is a collection of cliques or monomers of the same dimension (i.e., with the same number of points), which defines a pattern involving more points than a single clique or monomer, and which is in a clique or monomer. A collection of bodies encloses a cavity.

For example, pattern 705 includes six different three-point and two-dimensional patterns 505 that together define a homology class of degree two; and pattern 710 includes eight different three-point and two-dimensional patterns. Dimensional pattern 505, which together define first, second and second order homology categories. Each three-point, two-dimensional pattern 505 among patterns 705 and 710 may be considered to surround a corresponding cavity. The counting of such homologous classes within the topological representation can be performed by the "n"th Betti number associated with the directed graph.

The activity represented by patterns such as pattern 700 shows a relatively high degree of ordering of activity within the network, which is unlikely to be caused by random chance events. Pattern 700 can be used to characterize the complexity of an activity.

In some embodiments, in identifying decision moments, only certain patterns of activity may be identified, and/or certain portions of the identified patterns of activity may be discarded or otherwise ignored. For example, as shown in Figure 5, activities consistent with the five-point, four-dimensional monolithic pattern 515 inherently include activities consistent with the four-point, three-dimensional activity 510 and the three-point, two-dimensional one-sided pattern 505 . For example, points 0, 2, 3, and 4 and points 1, 2, 3, and 4 of the four-dimensional single body model 515 in Figure 5 are all consistent with the three-dimensional single body model 510. In some implementations, patterns that contain fewer points and therefore have lower dimensions may be discarded or otherwise ignored when identifying decision moments.

As another example, only certain activity patterns need to be identified. For example, in some embodiments, only images with an odd number of points (e.g., three, five, seven, ...) or an even number of dimensions (two-dimensional, four-dimensional, six-dimensional, ...) mode.

The complexity or ordering of patterns in a recurrent artificial neural network device in different time windows can be identified in various ways. FIG. 8 illustrates a schematic diagram of a data table 800 that can be used to confirm the complexity of an activity pattern or the degree of ordering in an activity pattern in a recurrent artificial neural network device. The data sheet 800 can be used to confirm the complexity of the activity pattern independently or in conjunction with other activities. For example, data table 800 may be used at step 420 in process 400 of FIG. 4 .

More specifically, the data table 800 contains a count of the number of occurrences of the pattern during time window "N", where each column presents a count of the number of activities matching the pattern of different dimensions. For example, in the example shown, column 805 contains the number of activity occurrences that match one or more three-point and two-dimensional patterns (i.e., “2032”), while column 810 contains the number of activity occurrences that match one or more four-point and three-dimensional patterns. The number of occurrences of pattern-matching activity (i.e., "877"). Since the occurrence of a pattern indicates a non-random ordering of activities, quantity counts also provide a general characterization of the overall complexity of activity patterns. A table similar to data table 800 may be made for each time window defined, for example, in step 410 of flow 400 of FIG. 4 .

Although the data table 800 contains separate rows and columns for each type of activity mode, this is not a limitation. For example, one or more counts (eg, counts for simpler modes) may be omitted in the data table 800 and when determining complexity. As another example, in some implementations, a single row or column may contain a count of occurrences of multiple activity patterns.

Although Figure 8 illustrates quantity counts in data sheet 800, this is not limited to system. For example, a numeric count can be represented as a vector (eg: <2032,877,133,66,48,...>). Regardless of how the count is presented, in some embodiments the count may be represented in binary form and be compatible with the digital data processing infrastructure.

In some embodiments, the occurrences of patterns may be weighted or combined to determine the degree or complexity of the ordering, for example, at step 420 in flow 400 of FIG. 4 . For example, the Euler characteristic can provide an approximation of the activity complexity through the following formula: S ₀ - S ₁ + S ₂ - S ₃ +… <Formula 1> Among them, "S _n " has The number of occurrences of a pattern of "n" points (i.e., a pattern with dimension "n-1"). For example, the pattern may be the directed clique pattern 500 in Figure 5 .

As another example of how the number of occurrences of a pattern may be weighted to determine the degree or complexity of the ordering, in some embodiments, the occurrences of a pattern may be weighted based on the weight of active links. Specifically, as mentioned previously, the strength of connections between nodes in an artificial neural network can vary, e.g., the strength represents how active a connection is during training. When activity occurs along a relatively strong set of links, it may be weighted differently than when the same pattern of activity occurs along a relatively weak set of links. For example, in some implementations, events may be weighted using the sum of the weights of active links.

In some embodiments, the Eurasian index or other complexity metric may be measured by the total number of patterns matched within a specific time window and/or the number of patterns that a specific network may form given its structure. The total number is normalized. Equations 2 and 3 below provide examples of normalization based on the total number of patterns that may be formed by the network.

In some embodiments, when higher dimensions involving a larger number of nodes occur When a pattern occurs, its weight can be higher than when a lower-dimensional pattern involving a smaller number of nodes occurs. For example, the probability of forming a directed clique decreases rapidly as the dimension increases. Specifically, in order to form an "n"-dimensional clique among "n+1" nodes, "(n+1)n/2" edges need to be correctly oriented. Such probabilities can be reflected in the weights.

In some embodiments, both the dimensionality and directionality of patterns can be used to weight the occurrence of patterns and identify the complexity of activities. For example, referring to Figure 6, in view of the difference in directionality between modes 515, 605 and 610, when the five-point and four-dimensional mode 515 appears, its weight is comparable to that of the five-point and four-dimensional mode 605 and 610. high.

當中，「S_x ^active」表示具有「n」個點的模式的活動出現次數，且「ERN」是等效隨機網路(即，具有相同數量的節點並且隨機地連接的網路)的計算。除此之外，「SC」可由下式所獲得：

當中，「S_x ^silent」表示當遞迴人工神經網路靜止時，具有「n」個點的模式的出現次數，並且可被認為體現了網路可能形成的模式的總數。在第2式及第3式中，模式可為例如第5圖中有向團的模式500。 An example of confirming the ordering or complexity of activities through the directionality and dimensions of the pattern is as follows:

where "S _x ^active " represents the number of active occurrences of a pattern with "n" points, and "ERN" is the calculation of an equivalent random network (i.e., a network with the same number of nodes connected randomly). In addition, "SC" can be obtained by the following formula:

Among them, "S _x ^silent " represents the number of occurrences of a pattern with "n" points when the recurrent artificial neural network is at rest, and can be considered to reflect the total number of patterns that the network may form. In Equations 2 and 3, the pattern may be, for example, the pattern 500 of the directed clique in Figure 5 .

Figure 9 illustrates a schematic diagram identifying time points for activity patterns with distinguishable complexity. The embodiment of identifying time points with activity patterns of distinguishable complexity shown in Figure 9 can be performed alone or in combination with other activities. For example, the identification of time points with activity patterns of distinguishable complexity may be performed at step 425 in flow 400 of FIG. 4 .

Figure 9 includes chart 905 and chart 910. The occurrence of patterns is shown in graph 905 as a function of time along the "x" axis. Specifically, each occurrence is schematically shown as vertical lines 906, 907, 908, 909. The occurrence of each column matches the activity to an instance of the corresponding pattern or pattern category. For example, the appearance in the top column may be an instance of the activity matching pattern 505 in Figure 5, the appearance in the second column may be an instance of the activity matching pattern 510 in Figure 5, and the appearance in the third column may be an instance of the activity matching pattern 510 in Figure 5. 5 instance of pattern 515, and so on.

Diagram 905 also contains dashed rectangles 915, 920, 925, which schematically depict different time windows when active patterns have distinguishable complexity. As shown, activity in a recurrent artificial neural network has a higher probability of matching patterns representing complexity during the time windows depicted by dashed rectangles 915, 920, 925 than outside such time windows. window period.

The complexity associated with these occurrences is shown in graph 910 as a function of time along the "x" axis. Graph 910 includes a first peak 930 of complexity that coincides with the time window depicted by dashed rectangle 915, and a second peak 935 of complexity that coincides with the time window depicted by dashed rectangles 920, 925. As shown, the complexity represented by peaks 930, 935 is distinguishable relative to a bottom line of complexity 940.

In certain embodiments, the time at which the output of a recurrent artificial neural network is read coincides with the occurrence of activity patterns of distinguishable complexity. For example, in the illustrative context of Figure 9, the output of a recurrent artificial neural network may be read at peaks 930, 935 (i.e., during the time windows depicted by dashed rectangles 915, 920, 925) .

It is particularly useful to identify distinguishable complexity in recurrent artificial neural networks when the input to the recurrent artificial neural network is a data stream. Examples of data streams include video or audio data, for example. Although the data flow has a beginning, usually processing data flow is not related to the beginning of the data flow. Information that establishes relationships is ideal. For example, neural networks can perform object recognition, such as identifying a cyclist near a car. The neural network should be able to recognize the cyclist no matter when the cyclist appears in the video stream, that is, regardless of the time since the video started. Continuing with this example, when a stream of data is fed into an object recognition neural network, any pattern of activity in the neural network will typically exhibit low or static complexity. Regardless of whether streaming data is fed continuously (or nearly continuously) into a neural network device, the neural network will exhibit these low or static levels of complexity. However, when an object of interest appears in the video stream, the complexity of the activity becomes distinguishable and displays the time at which the object was recognized in the video stream. Therefore, the time point of the distinguishable complexity of the activity can also be used as a "Yes/No" output as to whether the data in the data stream meets certain criteria.

In certain embodiments, activity patterns with distinguishable complexity may provide content of the output of an artificial neural network in addition to time points. Specifically, the identities and activities of nodes participating in activities consistent with activity patterns can be viewed as the output of a recursive artificial neural network. The identified activity patterns thus represent the processing results of the neural network, as well as the point in time at which that decision was read.

The content of decisions can be expressed in a variety of different forms. For example, in certain embodiments and as described in further detail below, the content of a decision may be represented as a binary vector or a matrix formed of ones and zeros. For example, each number may represent whether an activity pattern exists for a preset node group and/or a preset duration. In this embodiment, the content of the decision is expressed in binary form and is compatible with traditional digital data processing infrastructure.

Figure 10 illustrates a flow chart of a process 1000 for encoding signals using recurrent artificial neural networks based on characterization of activity in a network. Signals can be found in a variety of Encoding in context, such as transmission, encryption and data storage. Process 1000 may be performed by a system having one or more data processing devices that perform operations based on the logic of one or more sets of machine-readable instructions. For example, process 1000 may be performed by the same system that executes one or more computers that execute software for implementing the recurrent artificial neural network used in process 1000. In some cases, process 1000 may be performed by the same data processing device that performs process 400. In some cases, process 1000 may be performed by, for example, an encoder in a signal transmission system or an encoder in a data storage system.

The system executing process 1000 inputs the signal into the recurrent artificial neural network (labeled 1005). In some cases, the signal input is a discrete injection event. In other cases, the input signal is streamed into a recurrent artificial neural network.

The system executing process 1000 identifies one or more decision moments (labeled 1010) in a recurrent artificial neural network. For example, the system may identify one or more decision moments by executing process 400 of FIG. 4 .

The system executing process 1000 reads the output of the recursive artificial neural network (labeled 1015). As discussed above, in some embodiments, the output of a recurrent artificial neural network is the activity in the neural network that matches the pattern used to identify the decision point.

In some embodiments, individual "reader nodes" can be added to the neural network to identify the occurrence of specific activity patterns at specific sets of nodes and thus read the recursive artificial neural network at step 1015 output. A reader node can fire if and only if activity at a particular set of nodes meets time (and possibly even size) criteria. For example, to read the occurrence of pattern 505 of Figure 5 at nodes 104, 105, 106 in Figures 2 and 3, a reader node can be connected to nodes 104, 105, 106 (or the links between nodes 110). If an activity pattern involving nodes 104, 105, 106 (or their links) occurs, then only the reader The node itself becomes active.

By using this reader node, the need to define time windows for the entire recurrent artificial neural network is eliminated. In particular, each reader node may be connected to a different node and/or to several nodes (or connections between them). Each reader node can be set up to have a tailored response (eg, different decay times in an integrated coalescence triggering model) to recognize different activity patterns. The system executing process 1000 sends or stores the output of the recursive artificial neural network (labeled 1020). The specific actions performed in step 1020 may reflect the context in which process 1000 is being used. For example, in situations where secure or compressed communication is desired, a system performing process 1000 may send the output of a recurrent neural network to a receiver that has access to the same or similar recurrent neural network. As another example, in scenarios where secure or compressed data storage is desired, a system executing process 1000 may record the output of a recurrent neural network in one or more machine-readable data storage devices for later access.

In some implementations, the complete output of the recurrent neural network may not be sent or stored. For example, in one embodiment, the output content of the recurrent neural network is an activity in the neural network that matches a pattern representing activity complexity, then only the activity that matches a relatively more complex or higher-dimensional activity can be sent or stored. Activity. As an example, referring to pattern 500 in Figure 5, in some embodiments, only activities matching patterns 515, 520, 525, and 530 are sent or stored, while activities matching patterns 505 and 510 are ignored or discarded. In this way, the lossy process allows the amount of data to be transmitted or stored to be reduced at the expense of the integrity of the encoded information.

Figure 11 illustrates a flow chart of a process 1100 for decoding signals using a recurrent artificial neural network based on characterization of activity in the network. Signals can be decoded in a variety of different contexts, such as signal reception, decryption, and reading data from storage. Process 1100 can be implemented by A system execution of one or more data processing devices that performs operations based on the logic of one or more sets of machine-readable instructions. For example, process 1100 may be performed by the same system that executes one or more computers that execute software for implementing the recurrent artificial neural network used in process 1100 . In some cases, process 1100 may be performed by the same data processing device that performs process 400 and/or process 1000. In some cases, process 1100 may be performed by, for example, a decoder in a signal transmission system or a decoder in a data storage system.

The system executing process 1100 receives at least a portion of the output of the recurrent artificial neural network (labeled 1105). The specific actions performed at 1105 may reflect the context in which process 1100 is being used. For example, a system executing process 1000 may receive a transmission signal containing an output of a recurrent artificial neural network, or read a machine-readable data storage device storing an output of a recurrent artificial neural network.

The system executing process 1100 reconstructs the input of the recursive artificial neural network from the received output (labeled 1110). Reconstruction can be done in a variety of different ways. For example, in some implementations, a second artificial neural network (recurrent or non-recurrent) may be trained to reconstruct the input into the recurrent neural network from the output received at step 1105 .

As another example, in some embodiments, a decoder that has been trained using machine learning, including but not limited to deep learning, can be trained to reconstruct the input into the recurrent neural network from the output received at step 1105.

As another example, in some embodiments, inputs to the same recurrent artificial neural network or similar recurrent artificial neural networks may be iteratively permuted until the output of the recurrent artificial neural network is at some level matches the output received at step 1105.

In some embodiments, process 1100 may include receiving user input that causes The user input indicates the extent to which the input is to be reconstructed, and the reconstruction may be adjusted accordingly at step 1110 in response. For example, user input may specify that a complete rebuild is not required. A system adjustment rebuild of process 1100 is performed in response. For example, in one embodiment, the output of the recurrent neural network is the activities in the neural network that match a pattern indicative of activity complexity, and only activities that match relatively more complex or higher-dimensional activities are characterized. The output of will be used to reconstruct the input. As an example, referring to pattern 500 of Figure 5, in some implementations, only activities matching patterns 515, 520, 525, and 530 may be used to reconstruct input, while activities matching patterns 505 and 510 may be ignored or Give up. In this way, lossy reconstruction can be performed under selected circumstances.

In some embodiments, process 1000 and process 1100 may be used for peer-to-peer encrypted communications. Specifically, both the transmitter (i.e., encoder) and the receiver (i.e., decoder) can be equipped with the same recurrent artificial neural network. Several ways to formulate a shared RANN to ensure that third parties cannot reverse engineer it and decrypt the signal could include: the structure of the RANN; the functional setup of the RANN, including nodes state and edge weights; the size (or dimensions) of the pattern; and the fraction of the pattern in each dimension. These parameters can be viewed as multiple layers, which together ensure the security of the transmission. Additionally, in some implementations, the point in time of the decision moment can be used as a key to decrypt the signal.

Although process 1000 and process 1100 are shown for encoding and decoding a single recurrent artificial neural network, process 1000 and process 1100 may also be applied to systems and processes that rely on multiple recurrent artificial neural networks. These recurrent artificial neural networks can run in parallel or in series.

As an example of running in series, the output of a first recurrent artificial neural network can be used as the input of a second recurrent artificial neural network. The resulting output of the second recurrent artificial neural network is a twice-encoded (or twice-encrypted) version of the input to the first recurrent artificial neural network. This tandem arrangement of recursive artificial neural networks is useful in situations where parties have different levels of access to the information. For example, in a medical records system, the patient's identity information may not be accessible to a certain party. Access by a party who uses and has access to other parts of the medical record.

As an example of parallel operation, the same information can be fed into multiple different recurrent artificial neural networks. For example, the different outputs of the neural networks can be used to ensure that the input can be reconstructed with high fidelity.

Various modifications may be made to the many embodiments described above. For example, although applications typically indicate that activity within a recurrent artificial neural network should match patterns representing ordering, this is not a limitation. Conversely, in some embodiments, activity within a recurrent artificial neural network may be consistent with a pattern, but does not necessarily display activity that matches the pattern. For example, the increased probability that a recurrent neural network displays activity that matches a pattern can be viewed as a non-random ordering of activities.

As another example, in certain embodiments, different sets of patterns may be formulated to characterize activity in different recurrent artificial neural networks. For example, the model may be formulated based on its effectiveness in characterizing the activity of different recurrent artificial neural networks. The effectiveness may be quantified by, for example, based on the size of a table or vector showing counts of occurrences of different patterns.

As another example, in some embodiments, patterns used to characterize activity in a recurrent artificial neural network may consider the strength of connections between nodes. In other words, the pattern described previously handles all information between two nodes in a binary manner (i.e., activity is present or absent). No. transmission activities. However, this situation is not a limitation. Conversely, in some embodiments, activity with a certain level or strength of connections may need to be viewed as exhibiting ordered complexity in the activity of a recurrent artificial neural network in order to be consistent with a pattern.

As another example, the content of the output of a recurrent artificial neural network may contain patterns of activity that occur outside a time window, and wherein the activity in the neural network has a distinguishable complexity. For example, referring to Figure 10, the output of the recurrent artificial neural network read in step 1015 and sent or stored in step 1020 may include patterns of information encoding activity that occur in, for example, Figure 9 outside the dashed rectangles 915, 920, and 925 in diagram 905. For example, the output of a recurrent artificial neural network can be characterized only for the highest dimensional activity patterns, regardless of when these activity patterns occur. As another example, the output of a recurrent artificial neural network can be characterized only for the activity patterns surrounding the cavity, regardless of when these activity patterns occur.

Figures 12, 13 and 14 illustrate schematic diagrams of binary forms or representations 1200 of topological structures (eg, activity patterns in neural networks). The topologies shown in Figures 12, 13 and 14 all contain the same information, namely the presence or absence of features in a graph. The feature may be, for example, activity in a neural network device. In some embodiments, the activity is identified based on a period in which the activity in the neural network has a complexity that is distinguishable compared to other activity in response to the input.

As shown in Figure 12, binary representation 1200 includes bits 1205, 1207, 1211, 1293, 1294, 1297 and any additional number of bits (represented by "..."). For ease of illustration, bits 1205, 1207, 1211, 1293, 1294, 1297, ... are shown as discrete rectangles, and the binary value of the bit is represented by whether the rectangle is filled or not. In Figures 12, 13 and 14, the binary representation 1200 appears to be a one-dimensional vector composed of bits (Figures 12 and 13 Figure) or a two-dimensional matrix (Figure 14). However, representation 1200 differs from a vector, matrix, or other ordered collection of bits in that the same information can be encoded regardless of the order of the bits (that is, regardless of the position of each bit within the collection).

For example, in some embodiments, each individual bit 1205, 1207, 1211, 1293, 1294, 1297, ... may indicate whether a topological feature is present regardless of its location in the diagram. . As an example, as shown in Figure 2, a bit such as bit 1207 may indicate the presence of a topological feature consistent with pattern 505 in Figure 5, regardless of whether the activity is between nodes 104, 105, and 101 or between nodes 104, 105, and 101. Occurs between 105, 101 and 102. Although each individual bit 1205, 1207, 1211, 1293, 1294, 1297, ... may be associated with a particular feature, the location of that feature in the diagram need not be determined, for example, by representing the corresponding location of the bits in 1200 is encoded. In other words, in some embodiments, representation 1200 may simply provide isomorphic topological reconstruction of the graph.

On the other hand, in other embodiments, the positions of the respective bits 1205, 1207, 1211, 1293, 1294, 1297, . . . can indeed encode information such as feature positions in the figure. In these implementations, representation 1200 may be used to reconstruct a source graph. However, such an encoding does not necessarily need to exist.

Since a bit can represent the presence or absence of a topological feature regardless of its position in the diagram, as shown in Figure 12, bit 1205 appears at the beginning of representation 1200 and before bit 1207, while bit 1207 appears before bit 1211. In Figures 13 and 14, the order of bits 1205, 1207 and 1211 within representation 1200 has been changed relative to the position of other bits within representation 1200. However, the binary representation 1200 remains the same, and the set of rules or algorithms that define the process of encoding information in the binary representation 1200 remains the same. As long as the correspondence between bits and features The system is known, then the position of the bit in the representation 1200 is irrelevant.

More specifically, each bit 1205, 1207, 1211, 1293, 1294, 1297, ... respectively represents the presence or absence of a feature in the image. A graph is formed by a set of nodes and a set of edges between these nodes. Nodes can correspond to objects. Examples of such objects may include, for example, artificial neurons in neural networks, individuals in social networks, etc. Edges can correspond to some relationship between objects. Examples of relationships include, for example, structural links or activities along links. In the context of neural networks, artificial neurons can be related to each other through structural connections between neurons or by transmitting information along structural connections. In the context of a social network, each person may be related through links of "friends" or other relationships, or by transmitting information along such links (for example, posting). Thus, edges can characterize relatively long-lasting structural features in a node set or relatively transient activity features that occur within a limited time frame. In addition, edges can be directed or bidirectional. Directed edges represent the directionality of relationships between objects. For example, the transmission of information from a first neuron to a second neuron can be represented by a directed edge indicating the direction of transmission. As another example, in a social network, a relationship link may indicate that a second user will receive information from a first user, but not that the first user will receive information from a second user. In topological terms, a graph can be represented as a set of unit intervals "[0,1]", where "0" and "1" are represented by corresponding nodes connected by an edge.

Features whose presence or absence is represented by bits 1205, 1207, 1211, 1293, 1294, and 1297 may be, for example, a node, a group of nodes, one of a plurality of groups of nodes, a group of edges, a group of a plurality of groups of edges, and / Or other hierarchically more complex features (for example: a set of nodes within multiple sets of nodes within multiple sets of nodes). Bits 1205, 1207, 1211, 1293, 1294, and 1297 generally represent the presence or absence of features at different levels. For example, bit 1205 may represent the presence or absence of a single node, and bit 1205 may represent the presence or absence of a group of nodes.

In some embodiments, bits 1205, 1207, 1211, 1293, 1294, and 1297 may represent features in the graph that have threshold criteria for certain features. For example, bits 1205, 1207, 1211, 1293, 1294, and 1297 may indicate not only the presence of activity in a set of edges, but also whether the weight of that activity is above or below a threshold criterion. For example, the weights may reflect the training of the neural network device for a specific purpose, or may be intrinsic characteristics of the edges.

Figures 5, 6 and 8 above illustrate features whose presence or absence can be represented by bits 1205, 1207, 1211, 1293, 1294, 1297, ....

The directed monomers in the pattern 500, 600, and 700 sets treat functional or structural diagrams as topological spaces with nodes as points. Regardless of the identity of the specific nodes and/or links participating in the activity, bits may be used to represent structures or activities involving one or more nodes and links consistent with the entities in the pattern 500, 600, and 700 sets.

In some embodiments, only certain patterns of structure or activity may be identified, and/or certain portions of the identified patterns of structure or activity may be discarded or ignored. For example, as shown in Figure 5, structures or activities consistent with the five-point, four-dimensional cell pattern 515 inherently include those consistent with the four-point, three-dimensional cell pattern 510 and the three-point, two-dimensional cell pattern 505. structure or activity. For example, points 0, 2, 3, and 4 and points 1, 2, 3, and 4 in the four-dimensional single body model 515 in Figure 5 are all consistent with the three-dimensional single body model 510. In some embodiments, monolithic patterns that contain fewer points and therefore have lower dimensions may be discarded or ignored.

As another example, only certain patterns of structure or activity need to be identified. For example, in some embodiments, only patterns with an odd number of points (eg, three, five, seven, etc.) or an even number of dimensions (eg, two, four, six, etc.) are used.

Referring again to Figure 12, Figure 13 and Figure 14, from bits 1205, 1207, 1211, 1293, 1294, 1297, etc. to indicate the presence or absence of features may not be independent of each other. Specifically, if bits 1205, 1207, 1211, 1293, 1294, and 1297 represent the presence or absence of zero-dimensional (0-D) cells, and each zero-dimensional cell reflects the value of a single node, existence or activity, bits 1205, 1207, 1211, 1293, 1294, and 1297 are independent of each other. However, if bits 1205, 1207, 1211, 1293, 1294, 1297 represent the presence or absence of higher-dimensional cells, and each of these higher-dimensional cells reflects the presence or activity of multiple nodes, then by The presence or absence of each individual feature encodes information that is independent of the presence or absence of other features.

Figure 15 illustrates a schematic diagram of how the presence or absence of features corresponding to different bits is not independent of each other. Specifically, a subgraph 1500 is shown that includes four nodes 1505, 1510, 1515, and 1520 and six directed edges 1525, 1530, 1535, 1540, 1545, and 1550. More specifically, edge 1525 points from node 1505 to node 1510, edge 1530 points from node 1515 to node 1505, edge 1535 points from node 1520 to node 1505, edge 1540 points from node 1520 to node 1510, and edge 1545 points from node 1515 to node 1510. , edge 1550 points from node 1515 to node 1520.

A single bit in representation 1200 (eg, filled bit 1207 in FIGS. 12, 13, and 14) may indicate the presence of a directed three-dimensional cell. For example, such bits may represent the existence of a three-dimensional entity formed by nodes 1505, 1510, 1515, and 1520 and edges 1525, 1530, 1535, 1540, 1545, and 1550. The second bit in representation 1200 (eg, the filled bit 1293 in Figures 12, 13, and 14) may indicate the presence of a directed two-dimensional monomer. For example, such bits may represent the existence of a two-dimensional entity formed by nodes 1515, 1505, and 1510 and edges 1525, 1530, and 1545. In this simple example, the information encoded by bit 1293 is completely redundant to the information encoded by bit 1207.

It should be noted that the information encoded by bit 1293 may also be redundant to the information encoded by subsequent bits. For example, the information encoded by bit 1293 is redundant with the third and fourth bits indicating the presence of additional directional two-dimensional cells. Examples of such cells are formed by nodes 1515, 1520, 1510 and edges 1540, 1545, 1550 and nodes 1520, 1505, 1510 and edges 1525, 1535, 1540.

Figure 16 illustrates another schematic diagram of how the presence or absence of features corresponding to different bits are not independent of each other. Specifically, a subgraph 1600 is shown that includes four nodes 1605, 1610, 1615, and 1620 and five directed edges 1625, 1630, 1635, 1640, and 1645. Nodes 1505, 1510, 1515, and 1520 and edges 1625, 1630, 1635, 1640, and 1645 generally correspond to nodes 1505, 1510, 1515, and 1520 and edges 1525, 1530, 1535, and 1540 in the subgraph 1500 of Figure 15. and 1545. However, subgraph 1600, in contrast to subgraph 1500 (whose nodes 1515 and 1520 are connected by edge 1550), has nodes 1615 and 1620 that are not connected by edges.

A single bit in representation 1200 (eg, unfilled bit 1205 in FIGS. 12, 13, and 14) may represent the absence of a directed three-dimensional entity, e.g., the directed three-dimensional The cell contains nodes 1605, 1610, 1615 and 1620. The second bit in representation 1200 (eg, filled bit 1293 in Figures 12, 13, and 14) may indicate the presence of a two-dimensional monomer. For example, a two-dimensional unit is formed by nodes 1615, 1605, and 1610 and edges 1625, 1630, and 1645. The combination of filled bits 1293 and unfilled bits 1205 provides information indicating the presence (and the state of other bits) of other features that may or may not be present in representation 1200 . Specifically, the combination of the absence of a directed three-dimensional monomer and the existence of a directed two-dimensional monomer indicates that at least one edge does not exist in: (1) the possible directed two-dimensional monomer formed by nodes 1615, 1620, and 1610 monomer, or (2) Possible directed two-dimensional monomer formed by nodes 1620, 1605, and 1610. Therefore, the state of the bit indicating the presence or absence of any of these possible cells is not independent of the state of bit 1205 and bit 1293.

Although the examples have been described in terms of features with varying numbers of nodes and hierarchical relationships, this is not a limitation. For example, it may be the case that the representation 1200 includes a set of bits, and the set of bits only corresponds to, for example, the presence or absence of a three-dimensional unit.

Certain properties arise when individual bits are used to represent the presence or absence of features in the image. For example, the encoding of the information is error-tolerant and provides "moderate degradation" of the encoded information. Specifically, the loss of specific bits (or groups of bytes) may increase uncertainty as to the presence or absence of a feature. However, the probability of the presence or absence of a feature can still be evaluated from other bits that represent the presence or absence of adjacent features.

Likewise, as the number of bits increases, the certainty about the presence or absence of a feature also increases.

As another example, as mentioned above, the ordering or arrangement of bits is irrelevant to the isomorphic reconstruction of the graph represented by the bits. All that is needed is a known correspondence between bits and specific nodes/structures in the graph.

In certain embodiments, patterns of activity in neural networks may be encoded in representation 1200 in Figures 12, 13, and 14. In general, patterns of activity in a neural network are the result of many characteristics of the neural network, such as the structural connections between the nodes of the neural network, the weights between nodes, and possibly other parameters throughout the host. For example, in some embodiments, a neural network may be trained prior to encoding of activity patterns in representation 1200.

However, regardless of whether the neural network has been trained or not, for a given input, the response The corresponding pattern of activity can be thought of as a "representation" or "summary" of the input within the neural network. Therefore, although the representation 1200 appears to be a straightforward collection of numbers (and in some cases binary numbers), each number may encode a relationship or correspondence between a specific input and the associated activity in the neural network.

Figures 17, 18, 19 and 20 illustrate schematic representations of the occurrence of topological structures in activity in neural networks using four different classification systems 1700, 1800, 1900 and 2000. Classification systems 1700 and 1800 classify representations of activity patterns in neural networks as part of the input classification. Classification systems 1900 and 2000 each classify approximations of representations of activity patterns in neural networks as part of the input classification. In classification systems 1700 and 1800, the activity patterns represented occur in, and are read from, source neural network devices 1705 that are part of classification systems 1700 and 1800. In contrast, in classification systems 1900 and 2000 , the activity patterns that are approximately represented occur in source neural network devices that do not belong to classification systems 1700 and 1800 , and the activity patterns that are approximately represented are from the source neural network devices as classification systems 1900 and 1800 . A portion of 2000 is read in the source approximator 1905.

In additional detail, as shown in Figure 17, classification system 1700 includes a source neural network 1705 and a linear classifier 1710. Source neural network 1705 is a neural network device configured to receive an input in source neural network 1705 and render the appearance of an active topology. In the embodiment shown, source neural network 1705 includes an input layer 1715 that receives input. However, this situation is not a limitation. For example, in some implementations, some or all of the inputs may be injected into different levels and/or edges or nodes in the source neural network 1705.

Source neural network 1705 can be various different types of neural networks. Typically, the source neural network 1705 is a recurrent neural network, such as a recurrent neural network modeled on a biological system. In some cases, source neural network 1705 may simulate the extent of morphological characteristics, chemical characteristics, and other characteristics of biological systems. Typically, the source neural network 1705 is implemented on one or more computing devices with a relatively high degree of computing performance, such as a supercomputer. In this case, the classification system 1700 is typically a decentralized system, and in the classification system 1700 the linear classifier 1710 communicates with the source neural network 1705 through, for example, a data communication network.

In some embodiments, the source neural network 1705 may be untrained, and the activity it represents may be intrinsic to the source neural network 1705. In other embodiments, the source neural network 1705 may be trained, and the activity it represents may reflect that training.

The representation read from the source neural network 1705 may be such as the representation 1200 in Figures 12, 13, and 14. Representations can be read from the source neural network 1705 in a variety of ways. For example, in the example shown, source neural network 1705 includes "reader nodes" that read activity patterns among other nodes within source neural network 1705. In other embodiments, activity within the source neural network 1705 is read by a data processing element programmed to monitor patterns of activity in the source neural network 1705 that have a relatively high degree of order. In other embodiments, the source neural network 1705 may include an output layer, for example, when the source neural network 1705 is implemented as a feedforward neural network, the representation 1200 may be read from the output layer.

Linear classifier 1710 is a device that classifies objects based on a linear combination of features of the objects (ie, representations of activity patterns in source neural network 1705). Linear classifier 1710 contains input 1720 and output 1725. Input 1720 is coupled to receive a representation of the pattern of activity in source neural network 1705 . In other words, the representation of the activity pattern in the source neural network 1705 is a feature vector that represents the characteristics of the input of the source neural network 1705 and the source neural network 1705 is used by the linear classifier 1710 to classify the input. . Linear classifier 1710 can receive in various ways Representation of activity patterns in source neural network 1705. For example, representations of activity patterns may be received as discrete events or as a continuous stream over a real-time or non-real-time communication channel.

Output 1725 is coupled to output classification results from linear classifier 1710 . In the embodiment shown, output 1725 is schematically shown as a parallel port with multiple channels. However, this situation is not a limitation. For example, the output 1725 may output the classification results through a serial port or a port with combined parallel and serial capabilities.

In some implementations, linear classifier 1710 may be implemented on one or more computing devices with relatively limited computing capabilities. For example, linear classifier 1710 may be implemented on a personal computer or mobile computing device such as a smartphone or tablet.

Referring to Figure 18, the classification system 1800 includes a source neural network 1705 and a neural network classifier 1810. Neural network classifier 1810 is a neural network device that classifies objects based on a non-linear combination of characteristics of the objects (ie, representations of activity patterns in source neural network 1705). In the embodiment shown, neural network classifier 1810 is a feed-forward network including an input layer 1820 and an output layer 1825 . Neural network classifier 1810, like linear classifier 1710, may receive representations of activity patterns in source neural network 1705 in various ways. For example, representations of activity patterns may be received as discrete events or as a continuous stream over a real-time or non-real-time communication channel.

In certain implementations, neural network classifier 1810 may perform inferences on one or more computing devices with relatively limited computing capabilities. For example, neural network classifier 1810 may be implemented on a personal computer or mobile computing device such as a smartphone or tablet, for example, in a neural processing unit of such a device. Similar to the classification system 1700, the classification system 1800 is generally a decentralized system, and in the classification system 1800, the remote neural network classifier 1810 communicates with the source neural network 1705 through, for example, a data communication network.

In some embodiments, the neural network classifier 1810 can be a deep neural network, such as a convolutional neural network, which includes convolutional layers and pooling layers. and fully-connected layers. The convolutional layer may generate feature maps by, for example, using linear convolution filters and/or nonlinear activation functions. Pooling layers can reduce the number of parameters and control overfitting. The computations performed by different layers in neural network classifier 1810 may be defined differently in different implementations of neural network classifier 1810.

Referring to Figure 19, classification system 1900 includes source approximator 1905 and linear classifier 1710. As described in detail below, the source approximator 1905 is a relatively simple neural network that is trained to receive an input vector at the input layer 1915 or elsewhere, and output a vector that is approximated in a relatively more complex neural network A representation of the topology that occurs in the pattern of activity in a road. For example, the source approximator 1905 may be trained to approximate a recurrent source neural network, such as a recurrent neural network modeled on a biological system and incorporating some degree of morphological, chemical, and other characteristics of the biological system. road. In the embodiment shown, source approximator 1905 includes an input layer 1915 and an output layer 1920. Input layer 1915 may be coupled to receive input data. The output layer 1920 is coupled to output an approximation of the activity representation within the neural network device for receipt by the input 1720 of the linear classifier. For example, the output layer 1920 may output an approximation of 1200' representing 1200 in Figures 12, 13, and 14. On the other hand, the representation 1200 shown in Figures 17 and 18 is the same as the approximation 1200' of the representation 1200 shown in Figures 19 and 20. This is for illustration only. Generally speaking, approximating 1200’ will differ in at least some aspects from expressing 1200. Despite these differences, linear classifier 1710 can classify approximately 1200'.

In general, the source approximator 1905 can operate on a system with relatively limited computational performance. or perform inference on multiple computing devices. For example, source approximator 1905 may be implemented on a personal computer or mobile computing device such as a smartphone or tablet, such as in a neural processing unit of such a device. In contrast to classification systems 1700 and 1800, classification system 1900, including, for example, source approximator 1905 and linear classifier 1710, is typically housed in a single housing, with linear classifier 1710 implemented on the same data processing device or through hard-wiring. The connection is coupled to the data processing device.

Referring to Figure 20, classification system 2000 includes source approximator 1905 and neural network classifier 1810. The output layer 1920 of the source approximator 1905 is coupled to output an approximation 1200&apos; of the activity representation within the neural network device for receipt by the input 1820 of the neural network. Neural network classifier 1810 may classify approximation 1200' despite any differences between approximation 1200' and representation 1200. Like the similar classification system 1900, the classification system 2000, for example, including the source approximator 1905 and the neural network classifier 1810, is typically housed in a single housing, with the neural network classifier 1810 being implemented on the same data processing device. or to a data processing device coupled through a hardwired connection.

Figure 21 illustrates a schematic diagram of an edge device 2100 including a regional artificial neural network that can be trained by using representations that correspond to the occurrence of topological structures of activity in a source neural network. In this case, the local artificial neural network may be, for example, an artificial neural network that executes entirely on one or more local processors, which does not require a communication network to exchange data. Typically, the local processor will be hardwired. In some cases, the local processor may be housed in a single housing, such as a single personal computer or a single handheld mobile device. In some cases, a local processor may be controlled and accessed by a single person or a limited number of persons. In fact, by using representations of the occurrence of topological structures in more complex source neural networks, training (For example: using supervised learning or reinforcement learning techniques) A second neural network that is simpler and/or less trained but more unique. People with limited computing resources and training samples can also train neural networks as needed. In this way, storage requirements and computational complexity during training are reduced, and resources such as battery life are saved.

In the embodiment shown, edge device 2100 is schematically represented as a security camera device that includes an optical imaging system 2110, image processing electronics 2115, a source approximator 2120, a representation classifier 2125, and a communications controller and Interface 2130.

Optical imaging system 2110 may include, for example, one or more lenses (or even pinholes) and charge-coupled device (CCD) devices. Image processing electronics 2115 may read the output of the optical imaging system 2110 and generally perform basic image processing functions. Communications controller and interface 2130 are devices configured to control the flow of information to and from edge device 2100 . As described in detail below, communications controller and interface 2130 may perform operations to transmit images of interest to other devices and receive training information from other devices. Accordingly, communications controller and interface 2130 may include data senders and receivers that may communicate through data port 2135, for example. The data port 2135 can be a wired port, a wireless port, an optical port, etc.

Source approximator 2120 is a relatively simple neural network that is trained to output vectors that approximate representations of topological structures that occur in activity patterns in relatively more complex neural networks. For example, the source approximator 2120 may be trained to approximate a recurrent source neural network, such as a recurrent neural network modeled on a biological system and incorporating some degree of morphological characteristics, chemical characteristics, and other characteristics of the biological system. .

Representation classifier 2125 is a linear classifier or neural network classifier that is coupled to receive an approximation of the representation of the activity pattern in the source neural network from source approximator 2120 and output Classification results. The representation classifier 2125 may be a deep neural network, such as a convolutional neural network including convolutional layers, pooling layers, and fully connected layers. Convolutional layers may generate feature maps by, for example, using linear convolution filters and/or nonlinear activation functions. Pooling layers reduce the number of parameters and control overfitting. The computations performed by different layers in presentation classifier 2125 may be defined differently in different implementations of presentation classifier 2125.

In certain embodiments, optical imaging system 2110 may operate to generate raw digital images. Image processing electronics 2115 can read the raw image and will typically perform at least some basic image processing functions. Source approximator 2120 may receive images from image processing electronics 2115 and perform inference operations to output vectors that approximate representations of topological structures that would occur in activity patterns in relatively more complex neural networks. The approximated vector is input into the representation classifier 2125, and the representation classifier 2125 determines whether the approximated vector satisfies one or more sets of classification criteria. Examples include facial recognition and other machine vision operations. In the event that the representation classifier 2125 confirms that the approximate vector satisfies a set of classification criteria, the representation classifier 2125 may instruct the communications controller and interface 2130 to send information about the image. For example, communications controller and interface 2130 may send the image itself, a classification of the image, and/or other information about the image.

At some point it may be necessary to change the classification process. In these cases, communications controller and interface 2130 may receive the training set. In some embodiments, the training set may include raw or processed image data as well as representations of topological structures that occur in activity patterns in relatively more complex neural networks. Such a training set may be used to retrain the source approximator 2120, for example, by using supervised learning or reinforcement learning techniques. Specifically, these representations are used as target answer vectors and represent expected results of source approximator 2120 processing raw or processed image data.

In other embodiments, the training set may be included in a relatively more complex neural network A representation of the topology that occurs in the activity patterns in , and the desired classification of such representations of the topology. Such a training set may be used to retrain the neural network's representation classifier 2125, for example, by using supervised learning or reinforcement learning techniques. In particular, the desired classification is used as the target answer vector, and the desired result of the representation classifier 2125 represents a representation of the processing topology.

Regardless of whether the source approximator 2120 or the representation classifier 2125 is retrained, the inference operations of the edge device 2100 can be easily adapted to changing environments and targets without requiring large amounts of training data and time-consuming and computationally energy-intensive iterations. Training.

Figure 22 illustrates a schematic diagram of an edge device 2200 including a regional artificial neural network that can be trained by using representations that correspond to the occurrence of topological structures of activity in a source neural network. In the illustrated embodiment, edge device 2200 is schematically shown as a mobile computing device such as a smartphone or tablet. Edge device 2200 includes an optical imaging system (eg, on the back of edge device 2200, not shown), image processing electronics 2215, presentation classifier 2225, communications controller and interface 2230, and data port 2235. These elements may have characteristics and perform actions corresponding to optical imaging system 2110, image processing electronics 2115, presentation classifier 2125, communications controller and interface 2130, and data port 2135 in edge device 2100 in Figure 21.

The illustrated implementation of edge device 2200 additionally includes one or more additional sensors 2240 and a multi-input source approximator 2245. One or more sensors 2240 may sense one or more characteristics of the edge device 2200 itself or the environment surrounding the edge device 2200 . For example, in some embodiments, sensor 2240 may be an accelerometer that senses acceleration experienced by edge device 2200 . As another example, in some embodiments, sensor 2240 may be an acoustic sensor, such as a microphone that senses noise in the environment of edge device 2200 . Sensor 2240 other Other examples include chemical sensors (such as "artificial noses", etc.), humidity sensors, radiation sensors, etc. In some cases, the sensor 2240 is coupled to processing electronics that can read the output of the sensor 2240 (or other information, such as a contact list or map) and perform basic processing functions. Therefore, since the physical parameters actually sensed by various sensors are different, different implementations of the sensor 2240 may have different "modalities."

Multiple input source approximators 2245 are relatively simple neural networks that are trained to output vectors that approximate representations of topological structures that occur in activity patterns in relatively more complex neural networks. For example, the multi-input source approximator 2245 can be trained to approximate a recurrent source neural network, such as a recurrent neural network modeled on a biological system and incorporating some degree of morphological characteristics, chemical characteristics, and other characteristics of the biological system. road.

Unlike the source approximator 2120, the multi-input source approximator 2245 is coupled to receive raw or processed sensor data from multiple sensors and return activity in a relatively more complex neural network based on the data. An approximation of the representation of the topology occurring in the pattern. For example, multi-input source approximator 2245 may receive processed image data from image processing electronics 2215 as well as acoustic data, acceleration data, chemical data, or other data from one or more sensors 2240 . The multi-input source approximator 2245 may be a deep neural network such as a convolutional neural network, which includes convolutional layers, pooling layers, and fully connected layers. The computations performed by different layers in the multi-input source approximator 2245 may be specialized for a single type of sensor data or for multiple modalities of sensor data.

Regardless of the specific organization of the multi-input source approximator 2245, the multi-input source approximator 2245 will return an approximation based on raw or processed sensor data from multiple sensors. For example, the processed image data from the image processing electronic device 2215 is combined with the image data from the microphone sensor. Acoustic data from detector 2240 may be used by multi-input source approximator 2245 to approximate representations of topological structures that would occur in activity patterns in relatively more complex neural networks that receive the same data.

At some point it may be necessary to change the classification process. In these cases, communications controller and interface 2230 may receive the training set. In some embodiments, the training set may include raw or processed image data as well as representations of topological structures that occur in activity patterns in relatively more complex neural networks. Such a training set may be used to retrain the multiple input source approximator 2245, for example, by using supervised learning or reinforcement learning techniques. Specifically, these representations are used as target answer vectors and represent the expected results of the multi-input source approximator 2245 processing raw or processed image data.

In other embodiments, the training set may contain representations of topologies that occur in activity patterns in relatively more complex neural networks and desired classifications of those representations of topologies. Such a training set may be used to retrain the neural network representation classifier 2225, for example, using supervised learning or reinforcement learning techniques. Specifically, the desired classification is used as the target answer vector, and the desired outcome of the representation representing the topology processed by the representation classifier 2225 is represented.

Regardless of whether the multi-input source approximator 2245 or the representation classifier 2225 is retrained, the inference operations of the edge device 2200 can be easily adapted to changing environments and goals without requiring large amounts of training data and time-consuming and computationally energy-intensive Iterative training.

Figure 23 illustrates a schematic diagram of a system 2300 in which a regional neural network can be trained by using representations that correspond to the occurrence of topological structures in the source neural network. The target neural network can be implemented on a relatively simple and less expensive data processing system, while the source neural network can be implemented on a relatively complex and more expensive data processing system.

System 2300 includes various regional neural network devices having regional neural networks 2305, telephone base station 2310, wireless access point 2315, server system 2320, and one or more data communication networks 2325.

Regional neural network device 2305 is a device configured to process data using a computationally less expensive target neural network. As shown, regional neural network device 2305 may be implemented as any of a mobile computing device, a camera, a car or any other device, a fixed device and a mobile device, as well as different makes and models of devices within each category. Different regional neural network devices 2305 may belong to different owners. In certain embodiments, access to the data processing functions of the regional neural network device 2305 will generally be limited to these owners and/or designees of the owners.

Each regional neural network device 2305 may include one or more source approximators that are trained to output vectors that approximate representations of topological structures that occur in activity patterns in relatively more complex neural networks. For example, a relatively more complex neural network may be a recurrent neural network, for example, a recurrent neural network that is modeled on a biological system and includes a certain degree of morphological characteristics, chemical characteristics, and other characteristics of the biological system. .

In some embodiments, in addition to using source approximators to process data, the regional neural network device 2305 can be programmed to use representations of topologies that occur in activity patterns in relatively more complex neurons as target answer vectors. to retrain the source approximator. For example, regional neural network device 2305 may be programmed to perform one or more iterative training techniques (eg, gradient descent or stochastic gradient descent). In other embodiments, the source approximators in the regional neural network device 2305 can be trained by, for example, a dedicated training system or a training system installed on a personal computer that can interact with the regional neural network device 2305 to train the sources. approximator.

Each regional neural network device 2305 includes one or more wireless or wired data communication components. In the illustrated embodiment, each regional neural network device 2305 includes at least one Wireless data communication components, such as mobile phone transceivers, wireless transceivers, or both. The mobile phone transceiver is capable of exchanging data with the phone base station 2310. The wireless transceiver can exchange data with the wireless access point 2315. Each regional neural network device 2305 is also capable of exchanging data with peer mobile computing devices.

The telephone base station 2310 is connected to the wireless access point 2315 for data communication with one or more data communication networks 2325, and can exchange information with the server system 2320 through the network. Therefore, the regional neural network device 2305 typically also communicates with the server system 2320. However, this situation is not a limitation. For example, in embodiments where the regional neural network device 2305 is trained by other data processing devices, the regional neural network device 2305 only needs to perform at least one data communication with these other data processing devices.

Server system 2320 is a system of one or more data processing devices programmed to perform data processing activities in accordance with one or more sets of machine-readable instructions. Data processing activities may include providing training sets to a training system of regional neural network device 2305. As discussed above, the training system may be within the mobile regional neural network device 2305 itself or on one or more other data processing devices. The training set may contain representations of occurrences of topological structures that correspond to activity in the source neural network, as well as corresponding input data.

In some embodiments, server system 2320 also includes a source neural network. However, this is not a limitation, and server system 2320 may receive the training set from another data processing device system implementing the source neural network.

In operation, after server system 2320 receives a training set (from a source neural network in server system 2320 itself or elsewhere), server system 2320 may provide the training set to training area neural network device 2305 trainer. A training set can be used to train the target The source approximator in the regional neural network device 2305 enables the target neural network to approximate the operation of the source neural network.

Figures 24, 25, 26 and 27 illustrate representations of the emergence of topological structures in activity in neural networks using four different systems, namely systems 2400, 2500, 2600 and 2700. schematic diagram. Systems 2400, 2500, 2600, and 2700 may be configured to perform any of a number of different operations. For example, systems 2400, 2500, 2600 and 2700 can perform object positioning operations, object detection operations, object segmentation operations, object detection operations, prediction operations, action selection operations, etc.

Object positioning operations position objects within an image. For example, a bounding box can be constructed around the object. In some cases, object positioning can be combined with object recognition, where localized objects are tagged with appropriate names.

Object detection operations classify image pixels as belonging to a specific class (eg, belonging to an object of interest) or not. Generally speaking, object detection is performed by grouping pixels and forming a bounding box around the group of pixels. The bounding box should tightly surround the object.

In general, object segmentation assigns a class label to each image pixel. Therefore, with the exception of bounding boxes, object segmentation is performed on a pixel-by-pixel basis, and typically only a single label is assigned to each pixel.

Predictive operations attempt to draw conclusions beyond the range of observed data. Although predictive operations attempt to predict future events (e.g., based on information about past and current states), predictive operations may also seek to draw conclusions about past and current states based on incomplete information about those states.

The action selection operation attempts to select an action based on a set of conditions. Traditionally, actions Choice operations are broken down into different methods, such as symbol-based systems (classical planning), decentralized solutions, and passive or dynamic programming.

Classification systems 2400 and 2500 each perform desired operations on representations of activity patterns in neural networks. Systems 2600 and 2700 each perform desired operations on approximations of representations of activity patterns in neural networks. In systems 2400 and 2500, the represented activity patterns occur in a source neural network device 1705 that is part of systems 2400 and 2500, and the represented activity is read from the source neural network. In contrast, in systems 2600 and 2700, activity patterns that are approximately represented occur in source neural network devices that are not part of systems 2400 and 2500. However, approximations of these representations of activity patterns are read from source approximators 1905 that are part of systems 2600 and 2700.

In addition, as shown in Figure 24, the system 2400 includes a source neural network 1705 and a linear processor 2410. Linear processor 2410 is a device that performs operations based on linear combinations of features of representations of activity patterns in neural networks (or approximations of these representations). The operation may be, for example, object positioning operation, object detection operation, object segmentation operation, prediction operation, action selection operation, etc.

Linear processor 2410 includes input 2420 and output 2425. Input 2420 is coupled to receive a representation of the pattern of activity in source neural network 1705 . Linear processor 2410 may receive representations of activity patterns in source neural network 1705 through various means. For example, representations of activity patterns may be received as discrete events or as a continuous stream over a real-time or non-real-time communication channel. Output 2425 is coupled to output processing results from linear processor 2410. In certain implementations, linear processor 2410 may be implemented on one or more computing devices with relatively limited computing capabilities. For example, linear processor 2410 may be implemented on a personal computer or a computer such as a smart phone or tablet. on a mobile computing device.

Referring to Figure 25, the classification system 2500 includes a source neural network 1705 and a neural network 2510. Neural network 2510 is a neural network device configured to perform operations based on non-linear combinations of characteristics of representations of activity patterns in neural networks (or approximations of these representations). The operations may be, for example, object positioning operations, object detection operations, object segmentation operations, prediction operations, action selection operations, etc. In the embodiment shown, neural network 2510 is a feed-forward network including an input layer 2520 and an output layer 2525. Like linear processor 2410, neural network 2510 may receive representations of activity patterns in source neural network 1705 through various means.

In certain implementations, neural network 2510 may perform inference on one or more computing devices with relatively limited computing capabilities. For example, neural network 2510 may be implemented on a personal computer or mobile computing device such as a smartphone or tablet, such as in a neural processing unit of such a device. Similar to system 2400, system 2500 is generally a decentralized system, and in system 2500 remote neural network 2510 may communicate with source neural network 1705, such as via a data communications network. In some embodiments, for example, the neural network 2510 may be a deep neural network, such as a convolutional neural network.

Referring to Figure 26, system 2600 includes source approximator 1905 and linear processor 2410. Notwithstanding any differences between the approximation 1200' and the representation 1200, the processor 2410 may perform operations on the approximation 1200'.

Referring to Figure 27, system 2700 includes source approximator 1905 and neural network 2510. Notwithstanding any differences between approximating 1200' and representing 1200, neural network 2510 may still perform operations on approximating 1200'.

In some embodiments, systems 2600 and 2700 may be implemented on edge devices, For example, edge devices 2100 and 2200 in Figures 21 and 22. In some embodiments, systems 2600 and 2700 may be implemented as part of a system in which a topology corresponding to activity in the source neural network may be used (eg, system 2300 in Figure 23). representation to train regional neural networks.

Figure 28 illustrates a schematic diagram of a reinforcement learning system 2800 that includes an artificial neural network that can be trained by using representations that correspond to the occurrence of topological structures in the source neural network. Reinforcement learning is a type of machine learning in which an artificial neural network learns from feedback about the consequences of actions taken in response to the artificial neural network's decisions. Reinforcement learning systems move from one state in the environment to another new state by performing actions and receiving information that characterizes the new state and rewards and/or regrets that signify the success (or failure) of the action. Reinforcement learning aims to maximize the total reward (or minimize regret) through the learning process.

In the embodiment shown, the artificial neural network in reinforcement learning system 2800 is a deep neural network 2805 (or other deep learning architecture) trained using reinforcement learning methods. In some embodiments, the deep neural network 2805 can be a regional artificial neural network (for example, the neural network 2510 in Figures 25 and 27), and is implemented in a car, an airplane, a robot, or other devices. superior. This is not a limitation, and in other implementations, the deep neural network 2805 can be implemented on a system of connected devices.

In addition to the source approximator 1905 and the deep neural network 2805, the reinforcement learning system 2800 also includes an actuator 2810, one or more sensors 2815, and a teacher module 2820. In some embodiments, reinforcement learning system 2800 also includes one or more sources of additional data 2825.

Actuator 2810 is a device for controlling a mechanism or system that interacts with environment 2830 . In some embodiments, actuators 2810 control an entity's mechanism or system (eg, the steering of a car or the positioning of a robot). In other embodiments, the actuator 2810 may control a virtual mechanism or system (eg, a virtual game board or investment portfolio). For this reason, environment 2830 may also be physical or virtual.

Sensor 2815 is a device that measures characteristics of environment 2830. At least a portion of the measurements made by the sensor may characterize the interaction between the controlled mechanism or system and other aspects of the environment 2830 . For example, when actuators 2810 operate the car, one or more sensors 2815 may measure the car's speed, the car's direction, the car's acceleration, the car's proximity to other features, and the response of other features to the car. one or more of. As another example, when actuator 2810 controls a portfolio, sensor 2815 can measure the value and risk associated with the portfolio.

Generally speaking, source approximator 1905 and teacher module 2820 are coupled to receive at least some measurements made by sensor 2815. For example, the source approximator 1905 may receive measurement data at the input layer 1915 and output an approximation 1200&apos; of a representation of the topology occurring in the activity patterns in the source neural network.

Teacher module 2820 is a device configured to interpret measurements received from sensor 2815 and provide rewards and/or regrets to deep neural network 2805 . Rewards are positive and represent successful control of a mechanism or system. Regret is negative and indicates unsuccessful or suboptimal control. In general, the teacher module 2820 also provides characterization of measurements and reward/regret for reinforcement learning. In general, a characterization of a measurement is an approximation (e.g.: approximation 1200′) of a representation of the topology present in the activity patterns in the source neural network. For example, the teacher module 2820 can read the approximate 1200' output from the source approximator 1905 and combine the read approximate 1200' with the corresponding reward. /regret the pairing.

In many embodiments, reinforcement learning does not occur instantaneously in system 2800, or occurs during active control of actuators 2810 of deep neural network 2805. Conversely, training feedback may be collected by the teacher module 2820 and used to reinforce training when the deep neural network 2805 is not actively instructing the actuator 2810. For example, in some embodiments, the teacher module 2820 can be remote from the deep neural network 2805 and only intermittently communicate with the deep neural network 2805. Whether reinforcement learning is intermittent or continuous, the deep neural network 2805 can be evolved to optimize rewards and/or reduce regret, for example, using information received from the teacher module 2820.

In some embodiments, system 2800 also includes one or more data sources 2825 of additional data. Source approximator 1905 may also receive data from data source 2825 at input layer 1915 . In these cases, approximately 1200' will be generated by processing the sensor data as well as the data from data source 2825.

In some embodiments, data collected by a reinforcement learning system 2800 can be used for training or reinforcement learning of other systems, including other reinforcement learning systems. For example, characterization of measurements along with reward/regret values may be provided by the teacher module 2820 to a data exchange system that collects and redistributes these data from various reinforcement learning systems. In addition to this, as mentioned above, the characterization of the measurements may be an approximation, e.g. approximation 1200', of a representation of the topology present in the activity pattern in the source neural network.

The specific operations performed by reinforcement learning system 2800 will depend on the specific operating context. For example, in the case where source approximator 1905, deep neural network 2805, actuator 2810, and sensor 2815 are part of a car, deep neural network 2805 can perform object location operations and/or while steering the car. Object detection operation.

In embodiments where the data collected by the reinforcement learning system 2800 is used for training of other systems or reinforcement learning, the reward/regret values that characterize the state of the environment when performing object location operations and/or object detection operations and approximately 1200' Can be provided to the data exchange system. The data exchange system may then assign reward/regret values and approx. 1200' to other reinforcement learning systems 2800 associated with other vehicles in order to perform reinforcement learning on those other vehicles. For example, reinforcement learning can be used to improve object location operations and/or object detection operations at the second vehicle using reward/regret values and approximation 1200'.

However, the operations learned at other vehicles need not be the same operations performed by the deep neural network 2805. For example, reward/regret values based on travel time and characterization of inputs from sensor data (e.g., unexpected wetness in a location identified by Global Positioning System (GPS) data source 2825 The resulting approximate 1200' of road) can be used for another vehicle's route planning operation.

Embodiments of the subject matter and operations described in this disclosure may be implemented in digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or any of them. A combination of one or more. Embodiments of the subject matter described in this disclosure may be implemented as one or more computer programs, that is, one or more modules consisting of computer program instructions encoded on a computer storage medium for use by data. The processing device performs or is used to control operations of the data processing device. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. . Computer storage media may be or may be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of the foregoing. Other than that Additionally, although a computer storage medium is not a propagated signal, a computer storage medium may be the source or destination of computer program instructions encoded in artificially generated encoding of a propagated signal. Computer storage media may also be or be contained in one or more separate physical components or media (eg, multiple optical disks, magnetic disks, or other storage devices).

The operations described in this disclosure may be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

The term "data processing device" includes all types of devices, equipment, and machines for processing data, including, for example, programmable processors, computers, system-on-a-chip (SoC), or combinations of the foregoing. one or a combination thereof. The aforementioned device may include a dedicated logic circuit, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). In addition to hardware, the device may also contain code that creates an execution environment for the computer program in question, for example, code that constitutes the processor's firmware, a protocol stack, a database management system, an operating system , a cross-platform runtime environment, a virtual machine, or a combination of one or more thereof. Devices and execution environments can implement various computing model infrastructures, such as web serving, distributed computing, and grid computing infrastructures.

A computer program (also called a program, software, software application, script, or code) may be written in any form of programming language, including compiled or translated languages, declarative or procedural languages, and may be deployed in any form, including as a stand-alone Programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (for example, one or more scripts stored in a markup language file), in a A single file for a discussed program, or stored in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at a site or distributed across multiple sites and interconnected by a communications network.

The programs and logic flows described in this disclosure may be executed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. Programs and logic flows may also be executed by special purpose logic circuits, and devices may be implemented as special purpose logic circuits, such as field programmable logic gate arrays or special purpose integrated circuits.

By way of example, processors suitable for the execution of a computer program include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Generally speaking, the processor will receive instructions and data from read-only memory or random access storage, or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to, and send and receive data to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a Personal Digital Assistant (PDA), a mobile music or video player, a game console, a GPS receiver, or a portable Storage devices (such as Universal Serial Bus (USB) flash drives) and other devices. Devices suitable for storing computer program instructions and data include all forms of non-volatile storage, media and memory devices, including, for example, semiconductor memory devices (e.g., Erasable Programmable Read-Only Memory). Memory, EPROM), electronics Electrically-Erasable Programmable Read-Only Memory (EEPROM) and flash memory), magnetic disks (such as internal hard drives or removable disks), magneto-optical disks and CD-ROMs -ROM) and Digital Versatile Video Disc (DVD-ROM). The processor and memory may be supplemented by or incorporated into special purpose logic circuitry.

In order to provide interaction with users, embodiments of the subject matter described in the present disclosure may be implemented in a display device having a display device (such as a cathode ray tube (CRT) display or a liquid crystal display (LCD)). On a computer, information is displayed to the user, a keyboard, and a pointing device (such as a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the feedback received from the user can be in any form. Input, including acoustic, speech or tactile input. In addition, the computer can interact with the user by sending files to and receiving files from the device used by the user, such as by sending web pages to in response to requests received from a web browser. The web browser on the user's client device.

Although this disclosure contains many specific implementation details, these implementation details should not be construed as any limitation on the scope of the invention or claims, but rather as descriptions of features of particular embodiments of particular inventions. Certain features that are described in the context of separate embodiments of this disclosure can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. In addition, although the features described above may be described as operating in certain combinations, and may even be claimed as such, one or more elements from the claimed combination may in some cases be excised from the combination. multiple features, and therefore the claimed combination may be a subcombination or a subcombination changes.

Similarly, although operations are depicted in the drawings in a specific order, it should not be understood that such operations must be performed in the specific order shown, or sequentially, or that all operations shown must be performed to achieve desirable results. . In some cases, multiplexing and parallel processing can be used. In addition, the respective separation of various system components in the above embodiments should not be understood as being required in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product Or integrated into multiple software products.

Thus, specific implementations of the subject matter have been described. Other embodiments are also covered by the following patent applications. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the procedures depicted in the drawings do not necessarily require the specific order shown, or sequential order, to achieve desirable results. In certain embodiments, multiplexing and parallel processing may be employed.

This disclosure has described numerous embodiments. However, it is to be understood that various modifications may be made to the various embodiments of the present disclosure. For example, although representation 1200 is a binary representation in which each bit individually represents the presence or absence of a feature in the image, other kinds of information representations are possible. For example, a vector or matrix of multi-valued and non-binary numbers may be used to represent, for example, the presence or absence of features and possibly other characteristics of those features. An example of such a property is the weight of the active edges that make up the feature.

In summary, other embodiments are also within the scope of the following patent applications.

100‧‧‧Recurrent Artificial Neural Network Device

101, 102, 103, 104, 105, 106, 107‧‧‧Nodes

110‧‧‧Link

Claims (20)

A method of characterizing a signal transmission activity in a recursive artificial neural network or a probability showing the signal transmission activity, the method is executed by a data processing device, and the method includes: converting the recursive artificial neural network Characterizing the signaling activity in the network or the probability showing the signaling activity includes: identifying multiple group patterns of the signaling activity or the probability of the signaling activity in the recurrent artificial neural network ( clique patterns, wherein the clique patterns surround a plurality of cavities; and output from the recurrent artificial neural network is a binary sequence consisting of zeros and ones, wherein each binary sequence in the binary sequence The binary numbers represent whether there is a corresponding pattern in the recursive artificial neural network, and the binary numbers represent the existence or absence of patterns in different dimensions and have different numbers of nodes; and when a digital data processing device receives and Processing the binary sequence consisting of zeros and ones, including: in response to the loss of a first binary digit in the binary sequence, evaluating that a pattern corresponding to the first binary digit still exists in the binary sequence a probability of other binary numbers. The method of claim 1, wherein the method further includes defining multiple time windows (windows of time), and the signal transmission activity or the probability of the signal transmission activity in the recurrent artificial neural network is within the time window during which to respond to this recursion An input to an artificial neural network in which the clique patterns are identified in each of the time windows. The method of claim 2, wherein the method further includes identifying the first time window based on a distinguishable likelihood of the clique patterns occurring during a first time window in the time windows. The method of claim 1, wherein identifying the clique patterns includes identifying multiple directed cliques. The method of claim 4, wherein identifying the directed groups includes discarding or ignoring lower-dimensional directed groups existing in higher-dimensional directed groups. The method as described in claim 1, further comprising: distinguishing the group patterns into multiple categories; and characterizing the signaling activity or the probability of the signaling activity based on the number of occurrences of the group patterns in each category. . The method of claim 6, wherein distinguishing the group modes includes distinguishing the group modes based on a number of points in each group mode. The method of claim 1, wherein each binary number in the binary sequence indicates whether a corresponding pattern exists in the recurrent artificial neural network, regardless of whether the corresponding pattern is in one of the recurrent artificial neural networks. What is the location. The method described in claim 1 further includes: constructing the recursive artificial neural network, including: reading the binary numbers output by the recursive artificial neural network; and Evolving a structure of the recursive artificial neural network, wherein evolving the structure of the recursive artificial neural network includes: iteratively changing the structure; characterizing the complexity of patterns in the changed structure; and Use the characterization of the complexity of the pattern as an indicator of whether the structure of the change is desirable. The method of claim 1, further comprising: identifying a plurality of decision moments in the recursive artificial neural network based on the complexity of the pattern identified in the recursive artificial neural network, wherein , the decision moment refers to the time point when the signal transmission activity or the probability of the signal transmission activity in the recurrent artificial neural network indicates that the recurrent artificial neural network responds to the input information processing result, and the identification of the Such decision times include: a point in time when the signaling activity or the probability of the signaling activity is confirmed, and the signaling activity or the probability of the signaling activity is compared with other signaling activities or the signaling in response to the input The probability of the signaling activity has a distinguishable complexity; and the decision moments are identified based on the point in time of the signaling activity or the probability of the signaling activity with the distinguishable complexity. The method of claim 10, further comprising inputting a data stream into the recurrent artificial neural network; and identifying the clique patterns when inputting the data stream. The method of claim 1, further comprising evaluating whether the signaling activity or the probability of the signaling activity is responsive to an input of the recurrent artificial neural network, and evaluating whether the activity is responsive to the recurrent artificial neural network The input to the network includes: evaluating that relatively early and relatively simple patterns after the input event are responsive to the input and that relatively early and relatively complex patterns after the input event are not responsive to the input; and evaluating Patterns that are relatively late and relatively complex after the input event respond to the input, while patterns that are relatively early and complex after the input event do not respond to the input. A system that includes one or more computers that perform operations that characterize a signaling activity in a recursive artificial neural network or a probability that exhibits that signaling activity, including: identifying the signaling activity clique patterns of the signaling activity or the probability of the signaling activity in a recursive artificial neural network, wherein the clique patterns surround cavities; and from the recursive artificial neural network The neural network outputs a binary sequence composed of zeros and ones, where each binary number in the binary sequence represents whether there is a corresponding pattern in the recursive artificial neural network, and where the binary numbers Representing the existence or absence of patterns in different dimensions and having different numbers of nodes; and receiving and processing the binary sequence composed of zeros and ones in a digital data processing device, including: In response to the loss of a first binary digit in the binary sequence, a probability that a pattern corresponding to the first binary digit still exists in other binary digits in the binary sequence is evaluated. The system of request 13, wherein the operations further include defining a plurality of time windows (windows of time), and the probability of the signal transmission or signal transmission activity in the recurrent artificial neural network is within these time windows. A period in which the clique patterns are identified in the time windows in response to an input to the recurrent artificial neural network. The system of claim 14, wherein the operations further include identifying a first time window of the time windows based on a distinguishable likelihood of the clique patterns of activity, and the clique patterns are Occurs within this first time window. The system of claim 14, wherein identifying the clique patterns includes discarding or ignoring lower-dimensional directed cliques that exist within higher-dimensional directed cliques. The system of claim 13, wherein the operations further include: constructing the recursive artificial neural network, including: reading the binary numbers output by the recursive artificial neural network; and evolving (evolving). A structure of the recursive artificial neural network, wherein evolving the structure of the recursive artificial neural network includes: iteratively changing the structure; characterizing the complexity of the patterns in the structure will target the complexity of the pattern This characterization of the degree is used to indicate whether the modified structure is ideal. The system of claim 13, wherein the operations further include: identifying multiple decision moments in the recursive artificial neural network based on the complexity of identifying patterns in the recursive artificial neural network. , one of the decision moments is the time point when the signaling activity or the probability of the signaling activity in the recurrent artificial neural network indicates that the network responds to the input information processing results, and identifying such decision moments includes: Identifying a point in time in which the signaling activity or the probability of the signaling activity has a complexity that is distinguishable compared to other activities in response to the input; and based on The decision moments are identified at the point in time of the signaling activity of the distinguishable complexity or the probability of the signaling activity. The system of claim 18, wherein the operations further include inputting a data stream to the recurrent artificial neural network; and identifying the clique patterns when inputting the data stream. The system of claim 13, wherein the operations further include evaluating whether the activity is responsive to one of the inputs of the recurrent artificial neural network, and evaluating whether the signaling activity or the probability of the signaling activity is responsive to the The input to the recurrent artificial neural network includes: evaluating that relatively early and relatively simple patterns after the time of the input respond to the input, and that relatively early and relatively complex patterns after the time of the input do not respond. to that input; and to evaluate relatively late and relatively complex pattern responses after the moment of that input to the input, and the relatively early and relatively complex pattern after the moment of the input does not respond to the input.