KR102465409B1 - Characterization of Activities in Recurrent Artificial Neural Networks and Information Encoding and Decoding - Google Patents

Abstract

The methods, systems, and apparatuses include computer programs encoded on a computer storage medium, the computer programs for characterizing activity in a recurrent artificial neural network and for encoding and decoding information. In one aspect, a device may include a neural network, wherein the neural network is trained: in response to a first input, a second set of topological structures in patterns of activity occurring in a source neural network in response to the first input. an approximation of a first representation, in response to a second input, an approximation of a second representation of topological structures in patterns of activity occurring in the source neural network in response to the second input, and in response to a third input, the approximating a third representation of topological structures in patterns of activity occurring in the source neural network in response to a third input.

Description

Characterization of Activities in Recurrent Artificial Neural Networks and Information Encoding and Decoding

This disclosure relates to characterizing activity in recurrent artificial neural networks.

Characterizing activity can be used, for example, in the identification of decision moments, as well as in encoding/decoding signals in environments such as transmission, encryption, and data storage. It also relates to encoding and decoding information, and also to systems and techniques that use the encoded information in various environments. The encoded information may represent activity in a neural network, for example, a recurrent neural network.

Artificial neural networks are devices inspired by the structural and functional aspects of a network of biological neurons. In particular, artificial neural networks use a system of interconnected structures called nodes to mimic the information encoding and other processing capabilities of networks of biological neurons. The arrangement and strength of connections between nodes in an artificial neural network determines the results of information storage or information processing by the artificial neural network.

Neural networks may be trained to produce a desired signal flow within the network and to achieve desired information processing or information storage results. In general, training a neural network will change the arrangement and/or strength of connections between nodes during the learning phase. A neural network can be considered trained when sufficiently appropriate processing results are achieved by the neural networks for a given set of inputs.

Artificial neural networks can be used in a variety of different devices to perform non-linear data processing and analysis. Non-linear data processing does not satisfy the superposition principle, ie, the determined variables cannot be written as a linear sum of independent components. Examples of environments where non-linear data processing is useful are systems and techniques in pattern and sequence recognition, speech processing, novelty detection and sequential decision making, complex system modeling, and a variety of other environments. includes

Both encoding and decoding transform information from one aspect or representation to another. Different representations may provide different features that are more or less useful in different applications. For example, some aspects or representations of information (eg, natural language) may be easier for a human to understand. Other aspects or representations may be smaller in size (eg, compressed) and may be easier to transmit or store. Still other aspects or representations may intentionally obscure the information content (eg, the information may be cryptographically encoded).

Irrespective of the particular application, the encoding or decoding process will generally follow an algorithm or set of predefined rules that establish correspondences between information in different appearances or representations. For example, an encoding process that yields a binary code may assign roles or meanings to individual bits according to individual bits in a binary sequence or vector.

The present application seeks to provide techniques related to characterizing activity in artificial neural networks.

BACKGROUND This disclosure relates to encoding and decoding information, and to systems and techniques for using the encoded information in various environments. For example, in one implementation, a device comprises a neural network, the neural network being trained and topological structure in patterns of activity occurring in a source neural network in response to, and in response to, a first input. approximation of a first representation of, in response to a second input, an approximation of a second representation of topological structures in patterns of activity occurring in the source neural network in response to the second input, and in response to a third input , computes an approximation of a third representation of topological structures in patterns of activity occurring in the source neural network in response to the third input.

These and other implementations may include one or more of the following features. All of the topological structures may include two or more nodes in the source neural network and one or more edges between the nodes. The topological structures may include simplexes. The topological structures surround cavities. Each of the first, second, and third representations represents topological structures occurring in the source neural network at points in time where the patterns of activity have a complexity distinguishable from the complexity of another activity responsive to each of the inputs. can be expressed exclusively. The device may further comprise a processor coupled to receive approximations of the representations calculated by the neural network and process the received approximations. The processor may comprise a second neural network trained to process the representations produced by the neural network. Each of the first, second, and third expressions may include multi-valued, non-binary digits. Each of the first, second, and third representations may represent the occurrence of the topological structures without prescribing where the patterns of activity occur within the source neural network. The device may include a smartphone. The source neural network may be a recurrent neural network.

In another implementation, a device includes a neural network coupled to input representations of topological structures in patterns of activity occurring in a source neural network in response to a plurality of different inputs. The neural network is trained to process the representations and produce a response output.

These and other implementations include one or more of the following features. All of the topological structures may include two or more nodes in the source neural network and one or more edges between the nodes. The topological structures may include simplexes. The representations of topological structures at points in time where the patterns of activity have a complexity distinguishable from the complexity of another activity responding to each of the inputs may exclusively represent topological structures occurring in the source neural network. The device may comprise a neural network trained in response to a plurality of different inputs to produce respective approximations of representations of topological structures in patterns of activity occurring in a source neural network in response to the different inputs. have. Representations of the topological structures may include multi-valued, non-binary digits. The representations of topological structures may represent the occurrence of the topological structures without prescribing where the patterns of activity occur within the source neural network. The source neural network may be a recurrent neural network.

In another implementation, a method is implemented by a neural network device, the method comprising inputting a representation of topological structures in patterns of activity in a source neural network, wherein the activity is at input to the source neural network. responding, input steps, processing the expression, and outputting a result of processing the expression. The processing step coincides with training the neural network to process different such representations of topological structures in patterns of activity in the source neural network.

These and other implementations may include one or more of the following features. All of the topological structures may include two or more nodes in the source neural network and one or more edges between the nodes. The topological structure may include simplexes. The topological structures may surround cavities. The representations of topological structures at points in time where the patterns of activity have a complexity distinguishable from the complexity of another activity responding to each of the inputs may exclusively represent topological structures occurring in the source neural network. Representations of the topological structures may include multi-valued, non-binary digits. The representations of topological structures may represent the occurrence of the topological structures without prescribing where the patterns of activity occur within the source neural network. The source neural network may be a recurrent neural network.

The details of one or more implementations described herein are set forth in the accompanying drawings and description below. Other features, aspects, and advantages of the present subject matter will become apparent from the description, drawings and claims.

The effects of the present invention are individually indicated in the corresponding parts of the specification.

1 schematically shows the structure of a recurrent artificial neural network device.
2 and 3 schematically show the functionality of a cyclic artificial neural network device in different time windows.
4 is a flow diagram of a process for identifying decision moments in a recurrent artificial neural network based on characterization of activity in the network.
5 schematically illustrates patterns of activity that can be identified within a recurrent artificial neural network and that can be used to identify decision moments.
6 schematically illustrates patterns of activity that can be identified within a recurrent artificial neural network and that can be used to identify decision moments.
7 schematically illustrates patterns of activity that can be identified within a recurrent artificial neural network and that can be used to identify decision moments.
8 is a schematic illustration of a data table that may be used in determining the complexity or degree of ordering in activity patterns in a recurrent artificial neural network device.
9 is a schematic illustration of determining the timing of activity patterns with distinguishable complexity.
10 is a flow diagram of a process for encoding signals using a recurrent artificial neural network based on characterization of activity within the network.
11 is a flow diagram of a process for decoding signals using a recurrent artificial neural network based on characterization of activity in the network.
12, 13 and 14 are schematic illustrations of binary representations or representations of topological structures.
15 and 16 schematically show an example of how the presence or absence of features corresponding to different bits is not independent of each other.
17, 18, 19, 20 are schematic diagrams of using representations of the occurrence of topological structures in activity in neural networks in four different classification systems.
21 and 22 are schematic diagrams of edge devices comprising a local artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
23 is a schematic illustration of a system in which local artificial neural networks can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
Figures 24, 25, 26, 27 are schematic diagrams of using representations of the occurrence of topological structures in activity in neural networks in four different classification systems.
28 is a schematic illustration of a system including artificial neural networks that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
Like reference numbers in the various drawings indicate like elements.

1 is a schematic diagram of the structure of a recurrent artificial neural network device 100 . The recurrent artificial neural network device 100 is a device that mimics the information encoding and other processing capabilities of networks of biological neurons using a system of interconnected nodes. The recurrent artificial neural network device 100 may be implemented in hardware, software, or a combination thereof.

The above example of a recurrent artificial neural network device 100 includes a plurality of nodes 101 , 102 , ... , 107 interconnected by a plurality of structural links. Nodes 101 , 102 , ... , 107 are separate information processing structures similar to neurons in biological networks. In general, nodes 101 , 102 , ... , 107 process one or more input signals received over one or more links 110 , and output one or more output signals output over one or more links 110 . produce them For example, in some implementations, nodes 101 , 102 , ... , 107 weight multiple input signals, passing the sum through one or more non-linear activation functions, and outputs one or more output signals.

Nodes 101 , 102 , ... , 107 may act as accumulators. For example, nodes 101 , 102 , ... , 107 may operate according to an integrate-and-fire model in which one or more signals accumulate in a first node until a threshold is reached. can After reaching the threshold, the first node fires a connected second node by sending an output signal along one or more links 110 . In turn, the second node 101 , 102 , ... , 107 accumulates the received signal and when a threshold is reached the second node 101 , 102 , ... , 107 adds another output signal transmitted to the connected node of

Structured links 110 are connections capable of transmitting signals between nodes 101 , 102 , ... , 107 . For convenience, all of the structured links 110 are provided herein from every first node of nodes 101, 102, ... , 107 to every second node of nodes 101, 102, ... , 107. , are treated as being the same bidirectional link carrying signals in the very same way that signals are carried from the second node to the first node. However, this is not necessarily the case. For example, some or all of the structural links 110 may be from a first one of the nodes 101 , 102 , ... , 107 to a second one of the nodes 101 , 102 , ... , 107 . They may be unidirectional links that carry signals to and from the second node but not signals from the second node to the first node.

As another example, in some implementations, structural links 110 may have various properties that are not or in addition to directional. For example, in some implementations, different structured links 110 may carry signals of different magnitudes - of interconnections with different strengths between each of the nodes 101 , 102 , ... , 107 . brings results As another example, different structural links 110 may carry different types of signals (eg, inhibitory and/or excitatory signals). Indeed, in some implementations, structural links 110 may be modeled on links between somatic cells in biological systems and may reflect at least some of the enormous morphological, chemical and other diversity of such links.

In the above illustrated implementation, the recurrent artificial neural network device 100 is a subpopulation in that every node 101 , 102 , ... , 107 is connected to every other node 101 , 102 , ... , 107 . (clique) A network (or subnetwork). This is not necessarily the case. Rather, in some implementations, each node 101 , 102 , ... , 107 is a suitable sub of nodes 101 , 102 , ... , 107 (by the same links or various links as the case may be). can be connected to a set.

For clarity of illustration, the recurrent artificial neural network device 100 is shown with only 7 nodes. In general, real-world neural network devices will contain a much larger number of nodes. For example, in some implementations, neural network devices may include hundreds of thousands, millions, or even billions of nodes. Thus, the recurrent neural network device 100 may be part (ie, a subnetwork) of a larger recurrent artificial neural network.

In biological neural network devices, the accumulation process and the signal transmission process require the passage of time in the real world. For example, the soma of a nerve integrates received inputs over time, and signal transmission from nerve to nerve depends on, for example, the rate of signal transmission and the nature and length of the links between the nerves. times determined by Thus, the state of a biological neural network device is dynamic and changes over time.

In artificial recurrent neural network devices, time is represented using artificial and mathematical structures. For example, rather than signals requiring real-world time lapse for transmission from node to node, such signals are typically measured in computer clock cycles in terms of artificial units independent of real-world time lapse. may be expressed as or otherwise. Nevertheless, the state of an artificial recurrent neural network can be described as "dynamic" in that the state changes with respect to these artificial units.

Note that, for convenience, these artificial units are referred to herein as “time” units. Nevertheless, it should be understood that these units are artificial and generally do not correspond to the passage of time in the real world.

2 and 3 are schematic diagrams of the functionality of the cyclic artificial neural network device 100 in different time windows. Because the state of the device 100 is dynamic, the functioning of the device 100 can be expressed using signaling activity that occurs within a window. It is common for such a functional city to show activity on only some of the links 110 . In particular, not all links 110 are shown in these figures as activities contributing to the functioning of the device 100 as not all links 110 carry signals within a particular window.

2 and 3 , the active link 110 is shown as a relatively thick continuous line connecting the pair of nodes 101 , 102 , ... , 107 . In contrast, the inactive link 110 is shown with dashed lines. This is for illustrative purposes only. In other words, the structural connections formed by the links 110 exist whether the links 110 are active or not. However, this formalism emphasizes the functioning and activity of the device 100 .

In addition to schematically illustrating the existence of an activity along a link, the direction of that activity is also schematically illustrated. In particular, the relatively thick dashed lines depicting activity of links 110 also include arrowheads indicating the direction of signal transmission along the link during the associated window. In general, the direction of signal transmission within a single window does not deterministically force the link to be a unidirectional link with the indicated directionality. Rather, in a first functional illustration for a first time window, the link may be active in a first direction. In a second functional illustration for a second time window, the link may be active in the opposite direction. However, in some cases, such as, for example, in a cyclic artificial neural network device 100 comprising only unidirectional links, the directionality of the signal transmission will decisively indicate the directionality of that link.

In feedforward neural network devices, information travels in only one direction (ie forward only) to the output layer of nodes at the end of the network. Feedforward neural network devices indicate that a “decision” has been reached and that information processing is complete by propagation of signals through the network to the output layer.

In contrast, in a recurrent neural network, connections between nodes form cycles and the activity of the network proceeds dynamically without immediately discernible decisions. For example, even in a three-node recurrent neural network, a first node may send a signal to a second node, which in response may send a signal to a third node. In response, the third node may send a signal back to the first node. The signals received by the first node may respond -at least in part - to signals transmitted from the same node.

2 and 3, which are schematic functional diagrams, show this in a network that is only slightly larger than a three-node recurrent neural network. These functional illustrations shown in FIG. 2 may be an example of activity within a first window and FIG. 3 may be an example of activity within a second window immediately following. As shown, the collection of signaling activity originates from node 104 and appears to proceed generally clockwise through device 100 during the first window. In the second window, at least a portion of the signaling activity is generally seen going back to node 104 . Even in such extremely simple cities, signal transmission does not proceed in a way that yields a clearly identifiable output or termination.

For example, when a recurrent neural network of thousands or more nodes is considered, it is clear that signal propagation can occur through a very large number of paths and that these signals lack a clearly identifiable "output" location or time. can be recognized Although by design the network may return to a quiescent state in which background-only signal transmission or even no signal transmission occurs, the quiescent state itself does not represent the results of information processing. does not The circulatory neural network always returns to an inactive state regardless of input. The “output” or result of information processing is thus encoded within the activity occurring within the recurrent neural network in response to a particular input.

4 is a flow diagram of a process 400 for identifying decision moments within a recurrent artificial neural network based on activity characterization within that network. A decision moment is a point in time when activity in a recurrent artificial neural network indicates the results of information processing by the network in response to an input. Process 400 may be performed by a system of one or more data processing devices that perform operations according to logic of one or more sets of machine-readable instructions. For example, the processor 400 may be performed by the same system of a set of one or more computers running software for implementing the recurrent artificial neural network used in the process 400 .

The system performing process 400 receives at 405 a notification that a signal has been input into the recurrent artificial neural network. In some cases, the signal input is a separate injection event, eg, in which information is injected into one or more nodes and/or one or more links of the neural network. In other cases, the signal input is a stream of information injected into one or more nodes and/or links of the neural network over a period of time. The notification indicates that the artificial neural network is actively processing a notification, eg, not in an inactive state. In some cases, the notification is received from the neural network itself, eg, when the neural network is in an identifiable state of inactivity.

At 410, the system performing process 400 divides the response activity in the network into a collection of windows. If injection is a separate event, the windows may subdivide the time between injection and return to an inactive state into several intervals in which the activity displays variable complexities. If the injection is a stream of information, the duration of the injection (and optionally the time after injection until return to inactivity is complete) is subdivided into windows in which the activity displays variable complexities. can be Various approaches to determining the complexity of an activity are described further below.

In some implementations, the windows all have the same duration, although not necessarily. Rather, in some implementations, the windows may have different durations. For example, in some implementations, the duration increases with increasing time after the discrete injection event occurs.

In some implementations, the windows may be a continuous series of separate windows. In other implementations, the windows overlap in time, so that one window starts before the previous window ends. In some cases, the windows may be moving windows that move in time.

In some implementations, different durations of windows are defined for different determinations of the complexity of the activity. For example, for activity patterns that confine activity that occurs among a relatively large number of nodes, the windows are defined for activity patterns that confine activity that occurs among a relatively smaller number of nodes. It may have a longer duration than windows. For example, in the context of patterns of activity 500 ( FIG. 5 ), a window defined to identify an activity conforming to the pattern 530 is a window defined to identify an activity conforming to the pattern 505 . could be longer than

The system performing process 400 identifies patterns in activity in the network in different windows at 415 . As described further below, patterns in activity can be identified by treating a functional graph as being a topological space with nodes as points. In some implementations, the identified activity patterns are subpopulations in a functional graph of the network, eg, directed cliques.

At 420 the system performing process 400 determines the complexity of activity patterns in different windows. The complexity may be a measure of the likelihood that an ordered pattern of activity appears within a window. So randomly appearing activity patterns can be relatively simple. On the other hand, activity patterns exhibiting non-random ordering are relatively complex. For example, in some implementations, the complexity of an activity pattern may be measured using, for example, simple counts or Betti numbers of the activity pattern.

At 425 the system performing process 400 determines the timing of activity patterns of distinguishable complexity. A particular activity pattern may be distinguishable, for example, based on complexity that deviates upwards or downwards from a fixed or variable baseline. In other words, the timing of activity patterns indicating particularly high levels or particularly low patterns of non-random order within the activity may be determined.

For example, if the signal input is a distinct injection event, deviations from, for example, a stable baseline or from a curve characteristic of the average response of the neural network to a variety of different distinct injection events are distinguishable It can be used to determine the timing of complex activity patterns. As another example, where information is input within a stream, large changes in complexity during streaming can be used to determine the timing of distinguishable complex activity patterns.

At 430 the system performing process 400 timings an output reading from the neural network based on the timing of distinguishable complex activity patterns. For example, in some implementations, the output of a neural network may be read simultaneously that the distinguishable complex patterns of activity have occurred. In some implementations where the complexity variance indicates a relatively high non-random order within the activity, the observed activity patterns themselves may also be taken as an output of the recurrent artificial neural network.

5 is an illustration of patterns 500 of activity that may be identified within a recurrent artificial neural network and that may be used to identify decision moments. For example, patterns 500 can be identified at 415 in process 400 ( FIG. 4 ).

Patterns 500 are examples of activity in a recurrent artificial neural network. During application of patterns 500 , the functional graph is treated as a topological space with nodes as points. Activity within nodes and links that fit the patterns 500 may be recognized as ordered, regardless of the identity of the particular nodes and/or links involved in the activity. For example, the first pattern 505 may represent activity among the nodes 101 , 104 , 105 in FIG. 2 , where point 0 in the pattern 505 is the node 104 , Point 1 has a node 105, and point 2 is a node 101. As another example, the first pattern 505 may also represent activity among the nodes 104 , 105 , 106 in FIG. 3 , where point 0 in the pattern 505 is the node 106 . , point 1 is a node 104 , and point 2 is a node 105 . The sequence of activities within the directional subgroup is also defined. For example, in the pattern 5050, activity between points 1 and 2 occurs after activity between points 0 and 1.

In the illustrated implementation above, the patterns 500 are modally directional subpopulations or directional simplexes. In such patterns, activity originates from a source node that sends signals to all other nodes in the pattern. In patterns 500 , such a source node is denoted as point 0 while other nodes are denoted as points 1, 2, ... . Also, in directional subpopulations or simplexes, one of the nodes acts like a sink and receives signals transmitted from all other nodes in the pattern. In patterns 500 , such sink nodes are indicated by the highest numbered pattern in the pattern. For example, in pattern 505, the sink node is indicated by point 2. In pattern 510, the sink node is indicated by point 3. In pattern 515, the sink node is indicated by point 3, and so on. The activity represented by the patterns 500 is thus ordered in a distinguishable manner.

Each of the patterns 500 has a different number of points and reflects an ordered activity at a different number of nodes. For example, pattern 505 is 2D-simplex and reflects behavior in three nodes, pattern 510 is 3D-simplex and reflects behavior in four nodes, and so on. As the number of points in the pattern increases, the degree of ordering and complexity of the behavior also increases. For example, for a large collection of nodes with a certain level of random activity within a window, some of that activity may fit into the pattern 505 deviate by chance. However, it is increasingly unlikely that a random activity will fit each of the patterns 510, 515, 520, ... The presence of an activity conforming to the pattern 530 is indicative of a relatively higher degree and complexity of ordering within that activity, which is the presence of an activity conforming to the pattern 505 .

As previously described, in some implementations, different duration windows may be defined for different determinations of the complexity of the activity. For example, when an activity that fits the pattern 530 is about to be identified, longer duration windows can be used than when an activity that fits the pattern 505 is about to be identified.

6 is an illustration of patterns 600 of activity that may be identified within a recurrent artificial neural network and that may be used to identify decision moments. For example, patterns 600 can be identified at 415 in process 400 ( FIG. 4 ).

Similar to patterns 500 , patterns 600 are examples of activity within a recurrent artificial neural network. However, the patterns 600 differ from the strict ordering of the patterns 500 in the sense that not all of the patterns 600 are directional subpopulations or directional simplexes. In particular, the patterns 605 and 610 have lower directivity than the pattern of reference numeral 515 . In practice, pattern 605 has no sink nodes at all. Nevertheless, patterns 605, 610 indicate a degree of ordered activity that exceeds that expected through random chance and can be used to determine the complexity of activity within a recurrent artificial neural network.

7 is an illustration of patterns 700 of activity that may be identified within a recurrent artificial neural network and that may be used to identify decision moments. For example, patterns 700 can be identified at 415 in process 400 ( FIG. 4 ).

Patterns 700 define patterns that contain more points than individual subpopulations or simplexes and are directivity of the same dimension (ie, having the same number of points) surrounding cavities within a group of directivity simplexes. These are small groups or groups of directional simplexes. By way of example, the pattern at 705 includes six different three-point, two-dimensional patterns 505 that together define a homologous class of degree 2, whereas the pattern at 710 includes It contains eight different three-point, two-dimensional patterns 505 that together define a second homologous class of degree 2. Each of the three-point, two-dimensional patterns 505 in patterns 705 and 710 can be thought of as enclosing a respective cavity. The nth Betty number associated with the topographic graph gives the count of such homologous classes in the topological representation.

Activity depicted by patterns, such as patterns at reference number 700, shows a relatively high degree of ordering of activity within the network that is unlikely to occur by random chance. Patterns 700 can be used to characterize the complexity of the activity.

In some implementations, only some patterns of activity are identified and/or some of the identified patterns of activity are discarded or otherwise ignored during identification of decision moments. For example, referring to FIG. 5 , an activity suitable for a 5-point, 4-dimensional simplex pattern 515 is suitable for 4-point, 3-dimensional and 3-point, 2-dimensional simplex patterns 510. 505 Activities originally included. For example, points 0, 2, 3, 4 and points 1, 2, 3, 4 in the 4-dimensional simplex pattern 515 of FIG. 5 fit into the 3-dimensional simplex pattern 510 do. In some implementations, patterns containing a smaller number of points - and thus lower order - may be discarded during identification of decision moments and otherwise ignored.

As another example, only some patterns of activity need to be identified. For example, in some implementations only patterns with odd points (3, 5, 7, ...) or even even dimensions (2, 4, 6, ...) identify decision moments is used in

The complexity or degree of ordering in activity patterns within a recursive artificial neural network device in different windows may be determined in a variety of different ways. 8 is a schematic illustration of a data table 800 that may be used in such a determination. Data table 800 may be used to determine the complexity of the activity patterns separately or in conjunction with other activities. For example, data table 800 may be used at 420 in reference numeral 400 (FIG. 4).

In further detail, table 800 includes numeric counts of pattern occurrences during window "N", wherein numeric counts of activity matching patterns of different dimensions are in different rows. . For example, in the example shown above, the row 805 contains a numeric count (ie, “2032”) of occurrences of an activity that conform to one or more three-point, two-dimensional patterns, while reference numeral 805 . A row of 810 includes a numeric count (ie, “8772”) of occurrences of the activity matching one or more four-point, three-dimensional patterns. Since the occurrence of patterns indicates that the activity has a non-random order, the numeric counts also provide a generalized characterization of the overall complexity of the activity patterns. A table similar to the table at 800 may be formed, for example, for each window defined at 410 in process 400 ( FIG. 4 ).

Although table 800 contains separate rows and separate entries for all types of activity patterns, this is not necessarily the case. For example, one or more counts (eg, counts of simpler patterns) may be omitted from table 800 and from determining complexity. As another example, in some implementations, a single row or entry may include counts of occurrences of multiple activity patterns.

Although FIG. 8 presents numeric counts in table 800, this is not always necessary. For example, the numeric count may be represented as a vector (eg, <2032, 877, 133, 66, 48, ...>). Regardless of how the counts are presented, in some implementations, the counts may be represented in binary and compatible with the digital data processing infrastructure.

In some implementations, the numeric counts of the occurrences of the patterns may be weighted or combined to determine the degree or complexity of the ordering, for example at 420 in process 400 ( FIG. 4 ). For example, the Euler property can provide an approximation of the complexity of the activity and is given as:

[Equation 1]

where S _n is the number of occurrences of the pattern of n points (ie, the pattern of order n-1). The patterns may be, for example, directional subpopulation patterns 500 ( FIG. 5 ).

As another example of how numeric counts of occurrences of the patterns may be weighted or combined to determine the degree or complexity of the ordering, in some implementations, based on the weights of active links, pattern occurrences are Weights may be assigned. In further detail, as previously described, the strength of a connection between nodes in an artificial neural network may change as a result of, for example, how active the connection was during training. The occurrence of a pattern of behavior along a collection of relatively stronger links may be weighted differently than occurrence of the same pattern of activity along a collection of relatively weaker links. For example, in some implementations, the sum of the weights of active links may be used to weight the occurrence.

In some implementations, the Euler property or other measure of complexity may be normalized by the total number of patterns that enable a given network to form its own structure given and/or the total number of matching patterns within a particular window. can The normalization with respect to the total number of patterns that the network enables to form is given in Equations 2 and 3 below.

In some implementations, occurrences of higher dimensional patterns involving a greater number of nodes may be weighted more heavily than occurrences of lower dimensional patterns involving a smaller number of nodes. For example, the likelihood of forming directional subgroups decreases sharply as the dimension increases. In particular, to form an n-subpopulation from n+1 nodes, we need (n+1)n/2 edges that all come from correctly. This probability can be reflected in weighting.

In some implementations, both dimension and directionality of patterns can be used to weight occurrences of patterns and determine complexity of an activity. For example, referring to FIG. 6 , the occurrences of the 5-point, 4-dimensional pattern 515 are more dependent on differences in the directionality of the 5-point, 4-dimensional patterns 605 , 610 than the occurrences of the 5-point, 4-dimensional patterns 605 , 610 . Thus, it can be weighted more heavily.

An example of using both the dimension and directionality of patterns to determine the degree of complexity or sequencing of an action can be given as

[Equation 2]

where S _x ^active denotes the number of active occurrences of the pattern of n points and ERN is a calculation for an equivalent random network, that is, a network with the same number of randomly connected nodes. Also, SC is given as

[Equation 3]

where S _x ^silent denotes the number of occurrences of the pattern of n points when the recurrent artificial neural network is silent and the patterns it is capable of forming can be thought of as embodying the total number. In Equations 2 and 3, the patterns may be, for example, directional subpopulation patterns 500 ( FIG. 5 ).

9 is a schematic illustration of determining the timing of activity patterns of distinguishable complexity. The determination shown in FIG. 9 may be performed separately or together with other activities. For example, the determination may be performed at 425 of process 400 ( FIG. 4 ).

9 includes a graph 905 and a graph 910 . The graph at 905 shows the occurrence of the patterns as a function of time along the x-axis. In particular, the individual occurrences are schematically illustrated by vertical lines 906, 907, 908, 909. Each row of occurrences may be instances in which an activity matches a respective pattern or class of patterns. For example, occurrences of the top row may be instances in which the behavior matches patterns 505 ( FIG. 5 ), and occurrences in the second row may include instances where the behavior matches patterns 510 ( FIG. 5 ). and occurrences of the third row may be instances in which the action conforms to the patterns of reference numeral 515 ( FIG. 5 ), and so on.

The graph at 905 also includes dashed-line rectangles 915 , 920 , 925 that schematically indicate different time windows when the activity patterns have distinguishable complexity. As can be seen, the likelihood that activity within a recurrent artificial neural network will conform to a pattern indicative of complexity is higher during those windows than outside the windows indicated by dashed-line rectangles 915 , 920 , 925 .

The graph at 910 shows the complexity associated with these occurrences as a function of time along the x-axis. The graph at reference numeral 910 shows a first peak 930 of complexity consistent with the window represented by dashed rectangle 915 and a second peak 930 of complexity consistent with the window represented by dashed rectangles 920 and 925 . peak 935 . As can be seen, the complexity shown by peaks 930 , 925 is distinguishable from what may be considered to be a baseline level of complexity 940 .

In some implementations, the point in time at which the output of the recurrent artificial neural network is about to be read coincides with occurrences of activity patterns of distinguishable complexity. For example, in the exemplary environment of FIG. 9 , the output of the recurrent artificial neural network may be read at peaks 930 , 925 , ie, during the window represented by dashed-line rectangles 915 , 920 , 925 .

Identification of distinguishable levels of complexity in a recurrent artificial neural network is particularly beneficial when the input is a stream of data. Examples of data streams include, for example, video or audio data. Although data streams have a beginning portion, it is generally desirable to process information in a data stream that does not have a pre-defined relationship with the beginning of the data stream. As an example, the neural network may perform object recognition, eg, recognizing cyclists in the vicinity of a car. Such neural networks should be able to recognize cyclists regardless of when they appear within the video stream, ie without taking into account the time since the beginning of the video. Continuing with this example, when a data stream is input into an object recognition neural network, any patterns of activity within the neural network will typically display a low or inactive level of complexity. This low or inactive level of complexity is displayed irrespective of the continuous (or near continuous) input of the data stream to the neural network device. However, when an object of interest appears in the video stream, the complexity of the activity will become distinguishable and will indicate when an object is recognized within the video stream. Thus, the timing of the distinguishable level of complexity of the activity may also act as a yes/no output as to whether or not the data in the data stream meets a particular criterion.

In some implementations, the timing of the output of the recurrent artificial neural network as well as the content of the neural network is given by activity patterns of distinguishable complexity. In particular, the identity and activity of nodes participating in an activity suitable for the activity patterns may be regarded as an output of the recurrent artificial neural network. The identified activity patterns thus exemplify the outcome of processing by the neural network and the timing when this determination is about to be read.

The content of the crystal can be represented in a variety of different aspects. For example, in some implementations and as described in further detail below, the content of the crystal may be represented as a binary vector or matrix of ones and zeros. Each digit may indicate, for example, whether a pattern of activity exists or does not exist for a predefined group of nodes and/or a predefined duration. In such implementations, the content of the crystal is represented as a binary number and is compatible with traditional digital data processing infrastructure.

FIG. 10 is a flow diagram of a process 1000 for encoding signals using a recurrent artificial neural network based on characterization of activity within the network. Signals may be encoded in a variety of different environments, such as, for example, in transmission, encryption, and data storage. Process 1000 may be performed by a system of one or more data processing apparatuses that perform operations according to logic of one or more sets of machine-readable instructions. For example, process 1000 may be performed by the same system of one or more computers running software for implementing the recurrent artificial neural network used in process 1000 . In some instances, process 1000 may be performed by the same data processing apparatus performing the process 400 . In some instances, process 1000 may be performed, for example, by an encoder in a signal transmission system or an encoder in a data storage system.

The system performing process 1000 inputs a signal into the recurrent artificial neural network at 1005 . In some cases, the signal input is a separate injection event. In other cases, the input signal is streamed to the recurrent artificial neural network.

The system performing process 1000 identifies one or more decision moments in the recurrent artificial neural network at 1010 . For example, the system may identify one or more decision moments by performing process 400 ( FIG. 4 ).

The system performing process 1000 reads the output of the recurrent artificial neural network at 1015 . As described above, the content of the output of the recurrent artificial neural network is activity in the neural network that conforms to the patterns used to identify the decision point(s).

In some implementations, at 1015 individual “reader nodes” are added to the neural network to identify occurrences of a particular pattern of activity in a particular collection of nodes and thus to read the output of the recurrent artificial neural network. can be The reader nodes can fire if and only if activity in a particular collection of nodes meets a timing (and possibly size) criterion. For example, to read the occurrence of pattern 500 ( FIG. 5 ) at nodes 104 , 105 , 106 ( FIGS. 2 , 3 ), the reader node may use nodes 104 , 105 , 106 (or between them). may be connected to links 110 of The reader node will only become active on its own if an activity pattern involving nodes 104, 105, 106 (or links thereof) occurs.

Using such reader nodes would eliminate the need to globally define windows of time for the recurrent artificial neural network. In particular, individual reader nodes may be connected to different nodes and/or numbers of nodes (or links between them). The individual reader nodes may be set with tailored responses (eg, different decay times in an integrate-and-fire model) to identify different activity patterns.

The system performing process 1000 transmits or stores the output of the recurrent artificial neural network at 1020 . The special action performed at 1020 may reflect an environment in which the process of reference numeral 1000 is being used. For example, in environments where secure or compressed communications are desired, the system performing process 1000 may send the output of the recurrent neural network to a receiver with access to the same or similar recurrent neural network. have. As another example, in environments where secure or compressed data storage is desired, the system performing process 1000 may output the output of the recurrent neural network into one or more machine-readable data storage devices for later accesses. can be recorded

In some implementations, the complete output of the recurrent neural network is not transmitted or stored. For example, in implementations where the content of the output of the recurrent neural network is the activity in the neural network that matches patterns indicative of complexity within the activity, only an activity corresponding to a relatively more complex or higher-dimensional activity may be transmitted or stored. As an example, with reference to the patterns of reference number 500 ( FIG. 5 ), in some implementations only activities matching the patterns of reference numbers 515 , 520 , 525 , and 530 are transmitted or stored, while reference number 505 , Activities matching the patterns of 510 are ignored or discarded. In this way, the lossy process enables the volume of data transmitted or stored to be reduced at the cost of the completeness of the information being encoded.

11 is a flow diagram of a process 1100 for decoding signals using a recurrent artificial neural network based on a characterization of the activity therein. Signals may be decoded in a variety of different environments, such as, for example, receiving a signal, decoding, and reading data from storage. Process 1100 may be performed by a system of one or more data processing apparatuses that perform operations according to logic of one or more sets of machine-readable instructions. For example, process 1100 may be performed by the same system of one or more computers running software for implementing the recurrent artificial neural network used in process 1100 . In some instances, process 1100 may be performed by the same data processing apparatus performing the process 400 and/or the process 1000 . In some instances, process 1100 may be performed, for example, by a decoder in a signal receiving system or a decoder in a data storage system.

The system performing process 1100 receives at least a portion of an output of the recurrent artificial neural network at 1105 . The particular action to be performed at 1105 may reflect the environment in which process 1100 is being used. For example, the system performing process 1100 may receive a transmitted signal comprising an output of the recurrent artificial neural network or may include a machine-readable data storage device that stores the output of the recurrent artificial neural network. can read

The system performing process 1100 reconstructs the input of the recurrent artificial neural network from the output received at 1110 . The rebuild may proceed in a variety of different ways. For example, in some implementations, a second artificial neural network (circular or non-cyclic) may be trained to reconstruct an input to the recurrent neural network from the output received at 1105 .

As another example, in some implementations, a decoder trained using machine learning (including but not limited to deep learning) reconstructs an input to the recurrent neural network from an output received at 1105 . can be trained to

As another example, in some implementations, input to the same recurrent artificial neural network or to a similar recurrent artificial neural network may be repeated until the output of the recurrent artificial neural network matches the output received at 1105 to some extent. can be changed.

In some implementations, process 1100 may include receiving a user input, the user input defining the degree to which the input is reconstructed and, in response, adjusted at 1110 to reconstruct accordingly. For example, the user input may specify that a complete rebuild is not required. In response, the system performing process 1100 coordinates the rebuild. For example, in implementations where the content of the output of the recurrent neural network is the activity in a neural network that conforms to patterns indicative of complexity within the activity, characterizing an activity corresponding to a relatively more complex or higher-dimensional activity Only the output will be used to reconstruct the input. As an example, referring to the patterns at 500 ( FIG. 5 ), in some implementations only activities matching the patterns 515 , 520 , 525 , and 530 may be used to reconstruct the input. , whereas an activity matching the patterns 506 and 510 may be ignored or discarded. In this way, a costly rebuild can proceed in selected circumstances.

In some implementations, processes 1000 and 1100 may be used for peer-to-peer encrypted communications. In particular, both the transmitter (ie the encoder) and the receiver (ie the decoder) may be provided with the same recursive artificial neural network. There are several ways in which the shared recurrent artificial neural network can be tailored to ensure that third parties cannot reverse engineer and decipher the signal, including:

- the structure of the circulatory artificial neural network,

- functional settings of the recurrent artificial neural network, including node states and edge weights

- the size (or dimension) of the patterns, and

- Fragments of patterns within each dimension.

These parameters can be thought of as of multiple layers which together ensure transmission security. Also, in some implementations, the decision instantaneous time points may be used as keys to decrypt the signal.

Although the processes 1000 and 1100 are presented in terms of encoding and decoding a single recurrent artificial neural network, the processes 1000 and 1100 are also shown in systems and processes that rely on multiple recurrent artificial neural networks. can be applied. This recurrent artificial neural network can operate either in parallel or in series.

As an example of serial operation, an output of a first recurrent artificial neural network may be used as an input of a second recurrent artificial neural network. The resulting output of the second recurrent artificial neural network is a twice encoded (or twice encrypted) version of the input to the first recurrent artificial neural network. Such a cascading arrangement of a recurrent artificial neural network is such that, in environments where different parties have different levels of access to information, for example, patient identity information may be using the remainder of the medical record and provide access to the remainder. May be useful in medical records systems that may not be accessible to the party with

As an example of parallel operation, the same information may be input to multiple different recurrent artificial neural networks. The different outputs of the neural networks can be used, for example, to ensure that the input can be reconstructed with high fidelity.

Although many implementations have been described, various modifications may be made. For example, although it is generally implied that the application must conform to a pattern indicating that activity within a recurrent artificial neural network is ordered, this is not necessarily the case. Rather, in some implementations, activity within a recurrent artificial neural network may conform to a pattern, but not necessarily display activity conforming to the pattern. For example, an increase in the likelihood that a recurrent neural network will display an activity matching a pattern may be treated as a non-random ordering of the activity.

As another example, in some implementations, different groups of patterns may be tailored for use in characterizing the activity in different recurrent artificial neural networks. The patterns can be tailored, for example, according to their effectiveness in characterizing the activity of different recurrent artificial neural networks. Validity may be quantified, for example, based on the magnitude of a table or vector representing counts of occurrences of different patterns.

As another example, in some implementations, patterns to be used to characterize activity in recurrent artificial neural networks may take into account the strength of a connection between nodes. In other words, the patterns previously described herein represent all signaling activity between two nodes in a binary fashion, i.e. either with or without the activity. This is not always the case. Rather, in some implementations, conforming to a pattern may require that the activity of a connection of some level or strength be taken as indicating an ordered complexity within the activity of a recurrent artificial neural network.

As another example, the content of the output of the recurrent artificial neural network may include activity patterns occurring outside time windows where activity in the neural network has a distinguishable level of complexity. For example, the output of the recurrent artificial neural network read at 1015 and transmitted or stored at 1020 ( FIG. 10 ) is, for example, the dashed-line rectangles 915 , 920 , 925 in the graph 905 ( FIG. 9 ). ) may include information encoding activity patterns occurring externally. As an example, the output of the recurrent artificial neural network may characterize only the highest order patterns of activity, regardless of when the patterns of activity occur.

12, 13 and 14 are schematic illustrations of a binary representation or representation 1200 of topological structures, such as, for example, patterns of activity in a neural network. The topological structures shown in FIGS. 12 , 13 and 14 all contain the same information, ie an indication of the presence or absence of features in the graph. The characteristics may be, for example, activity in a neural network device. In some implementations, the activity is identified based on or during a time interval in which the activity in the neural network has a complexity distinguishable from other activity responsive to input.

As can be seen, the binary representation of reference numeral 1200 includes the bits 1205, 1207, 1211, 1293, 1294, 1297 and an additional arbitrary number of bits (represented by the ellipsis "..."). For didactic purposes, the bits 1205, 1207, 1211, 1293, 1294, 1297 ... are shown as individual rectangular shapes, either filled or unfilled to indicate the binary value within those bits. do. In the above schematic drawings, the representation of reference numeral 1200 appears ostensibly to be either a one-dimensional vector of bits ( FIGS. 12 and 13 ) or a two-dimensional matrix of bits ( FIG. 14 ). However, the representation of reference numeral 1200 can be encoded as a vector, a matrix, or other ordered collection of bits, in which the same information is encoded independently of the order of the bits, ie regardless of the position of the individual bits within the collection. It is different in that there is

For example, in some implementations, each individual bit of reference numerals 1205, 1207, 1211, 1293, 1294, 1297 ... may represent the presence or absence of a topological feature independent of its location in the graph. have. By way of example, referring to FIG. 2 , a bit such as the bit 1207 indicates the existence of a topological feature suitable for the pattern of 505 ( FIG. 5 ), between or between the nodes 104 , 105 , 106 . It can be indicated whether or not activity occurs between the nodes numbered 105, 101, and 102. So, although each individual bit of 1205, 1207, 1211, 1293, 1294, 1297 ... may be associated with a particular feature, the location of that feature in the graph is, for example, 1200's. It need not be encoded by the corresponding position of the bit in the representation. In other words, in some implementations, the representation of reference numeral 1200 merely provides an isomorphic topological reconstruction of the graph.

As an aside, in other implementations, it is possible that the position of the individual bits of reference numbers 1205, 1207, 1211, 1293, 1294, 1297 ... actually encodes information such as, for example, the position of a feature in the graph. do. In these implementations, the source graph may be reconstructed using the representation at 1200 . However, such an encoding is not necessarily presented.

Considering the ability of a bit to express the presence or absence of a topological feature irrespective of the location of that feature in the graph, in FIG. 1, the bit 1205 appears before the bit 1207, and the bit 1207 is It appears before the bit of reference number 1211 at the beginning of the representation of reference number 1200. In contrast, in FIGS. 2 and 3 , the order of bits 1205, 1207, and 1211 in the representation of reference numeral 1200 - and the sequence of bits 1205, 1207, and 1211 in relation to other bits in the representation of reference numeral 1200 The positions of the bits have changed. Nevertheless, the binary representation 1200 remains the same, as is the case with the set of algorithms or rules that define the process for encoding the information in the binary representation 1200 . As long as the correspondence between the bit and the feature is known, the position of the bits in the representation 1200 is irrelevant.

In further details, each bit of reference numerals 1205, 1207, 1211, 1293, 1294, 1297 ... individually represents the presence or absence of a feature in the graph. A graph is a set of nodes and a set of edges between those nodes. The nodes may correspond to objects. Examples of objects may include, for example, artificial neurons in a neural network, individuals in a social network, and the like. Edges may correspond to some relationship between objects. Examples of relationships include, for example, structural connections or activities along those connections. In the context of a neural network, artificial neurons may be related by structural connections between neurons or by transmission of information along structural connections. In the context of a social network, individuals may be related by "friends" or other relevant connections, or by transmission of information (eg, postings) along such connections. Edges may thus be characterized as relatively long-lived structural properties of a set of nodes or relatively transient activity properties occurring within a finite time frame. Also, the edges may be either directional or bidirectional. Directed edges indicate the directionality of the relationship between the objects. For example, information transmission from a first neuron to a second neuron may be represented by a directional edge indicating the direction of transmission. As another example, in a social network, a relational connection may indicate that a second user will receive information from the first user but the first user will not receive information from the second user. In topological terms, the graph can be represented as a set of unit intervals [0, 1], where 0 and 1 are identified as respective nodes connected by edges.

Said characteristics, in which the presence or absence is indicated by the bits 1205, 1207, 1211, 1293, 1294, 1297, are, for example, a node, a set of nodes, a set of sets of nodes, a set of edges, a set of edges. a set of sets, and/or additional hierarchically-more-complex features (eg, a set of sets of sets of nodes). The bits 1205, 1207, 1211, 1293, 1294, 1297 generally represent the presence or absence at different hierarchical levels. For example, the bit 1205 may represent the presence or absence of a node, while the bit 1205 may represent the presence or absence of a set of nodes.

In some implementations, the bits 1205 , 1207 , 1211 , 1293 , 1294 , 1297 may represent features in a graph having a threshold level of some characteristic. For example, the bits 1205 , 1207 , 1211 , 1293 , 1294 , 1297 may indicate not only that there is activity in the set of edges, but that this behavior is weighted either above or below a threshold level. The weights may specify for a special purpose, for example, training of a neural network device, or may be an intrinsic characteristic of the edges.

5, 6 and 8 above show features in which presence or absence can be represented by the bits 1205, 1207, 1211, 1293, 1294, 1297...

The directional simplexes in the collections 500, 600 and 700 treat functional or structural graphs as a topological space with nodes as points. A structure or activity comprising one or more nodes and links suitable for simplexes in the collections of reference numerals 500, 600 and 700 is represented in some bit, regardless of the identity of the particular nodes and/or links participating in that activity. can be

In some implementations, only some patterns of structure or activity are identified and/or some of the identified patterns of structure or activity are discarded or otherwise ignored. For example, referring to FIG. 5 , a structure or activity suitable for a five-point, four-dimensional simplex pattern 515 is a four-point, three-dimensional and three-point, two-dimensional simplex pattern 510 , 505 . It originally contains a structure or activity suitable for For example, points 0, 2, 3, 4 and points 1, 2, 3, 4 in the four-dimensional simplex patterns 515 of FIG. 5 are both three-dimensional simplex pattern 510 . suitable for In some implementations, simplex patterns containing a smaller number of points - and thus of lower dimensionality - may be discarded or otherwise ignored.

As another example, only some patterns of structure or activity need to be identified. For example, in some implementations only patterns with odd points (3, 5, 7, ...) or even dimensions (2, 4, 6, ...) are used.

12 , 13 , 14 , the features whose presence or absence is represented by the bits 1205 , 1207 , 1211 , 1293 , 1294 , 1297 ... may not be independent of each other. As an explanation, if the bits of reference numerals 1205, 1207, 1211, 1293, 1294, 1297 each represent the presence or absence of zero-dimensional-simplexes reflecting the presence or activity of a single node, reference numerals 1205, 1207, Bits 1211, 1293, 1294, and 1297 are independent of each other. However, if the bits of reference numerals 1205, 1207, 1211, 1293, 1294, and 1297 each represent the presence or absence of higher-dimensional simplexes reflecting the presence or activity of multiple nodes, the presence of each individual feature Or the information encoded by the absence may not be independent of the presence or absence of other features.

15 schematically shows an example of how the presence or absence of features corresponding to different bits is not independent of each other. In particular, a subgraph 1500 comprising four nodes 1505 , 1510 , 1515 , 1520 and six directional edges 1525 , 1530 , 1535 , 1540 , 1545 , 1550 is illustrated. In particular, the edge of reference number 1525 is directed from the node of reference number 1525 to the node of reference number 1510,

The edge of reference number 1530 points from the node of reference number 1515 to the node of reference number 1505, the edge of reference number 1535 is directed from the node of reference number 1520 to the node of reference number 1505, and the edge of reference number 1540 is the node of reference number 1520 From the node to the node 1510, the edge 1545 points from the node 1515 to the node 1510, and the edge 1550 points from the node 1515 to the node 1520.

A single bit in representation 1200 (eg, filled bit 1207 in FIGS. 12 , 13 , 14 ) may indicate the presence of a directional three-dimensional-simplex. For example, such a bit may indicate the presence of a three-dimensional-simplex formed by nodes 1505, 1510, 1515, 1520 and edges 1525, 1530, 1535, 1540, 1545, 1550. A second bit in representation 1200 (eg, filled bit 1293 in FIGS. 12 , 13 , 14 ) may indicate the presence of a directional two-dimensional-simplex. For example, such a bit may indicate the presence of a two-dimensional-simplex formed by nodes 1515, 1505, 1510 and edges 1525, 1530, 1545. In this simple example, the information encoded by the bits 1293 completely overlaps the information encoded by the bits 1207 .

Note that information encoded by the bits of reference numeral 1293 may also overlap with information encoded by other additional bits. For example, the information encoded by the bit 1293 would overlap the third bit and the fourth bit indicating the presence of additional directional two-dimensional-simplexes. Examples of these simplexes are at the nodes 1515, 1520, 1510 and the edges 1540, 1545, 1550 and the nodes 1520, 1505, 1510 and the edges 1525, 1535, 1540. is formed by

16 schematically shows another example of how the presence or absence of features corresponding to different bits are not independent of each other. In particular, a subgraph 1600 is shown comprising four nodes 1605 , 1610 , 1615 , 1620 and five directional edges 1625 , 1630 , 1635 , 1640 , 1645 . Nodes 1605, 1610, 1615, 1620 and edges 1625, 1630, 1635, 1640, 1645 represent nodes 1505, 1510, 1515, 1520 and edges 1525, 1530, 1535, 1540, 1545 is generally equivalent. However, in contrast to the subgraph 1500 in which the nodes 1515 and 1520 are connected by the edge 1550, the nodes 1615 and 1620 are not connected by the edge.

A single bit in the representation of reference number 1200 (eg the unfilled bit of reference number 1205 in FIGS. 12, 13, 14 ) is, for example, a directional 3D-simple surrounding nodes 1605 , 1610 , 1615 , 1620 . It can indicate the absence of a directional 3D-simplex like Rex. The second bit in the representation of reference number 1200 (eg, the filled bit of reference number 1293 in FIGS. 12, 13, 14 ) may indicate the presence of a 2D-simplex. The example directional 2D-simplex is formed by the nodes 1615, 1605, 1610 and the edges 1625, 1630, 1645. This combination of filled bits 1293 and unfilled bits 1205 provides information indicating the presence or absence of other features (and states of other bits) that may or may not be present within the representation of reference numeral 1200 . do. In particular, the combination of the absence of a directional 3D-simplex and the presence of a directional 2D-simplex indicates from any of the following that at least one edge is absent:

a) a possible directional 2D-simplex formed by nodes 1615, 1620, 1610 or

b) A possible directional 2D-simplex formed by nodes 1620, 1605, 1610.

Thus, the state of the bit representing the presence or absence of any one of these possible simplexes is not independent of the state of the bits 1205 and 1293 .

Although these examples have been described in terms of different numbers of nodes and features with hierarchical relationships, this is not necessarily the case. For example, a representation 1200 comprising a collection of bits corresponding only to the presence or absence of a 3D-simplex is possible.

The use of individual bits to represent the presence or absence of features in a graph results in certain properties. For example, encoding information is fault tolerant and provides a “graceful degradation” of the encoded information. In particular, loss of a particular bit (or group of bits) can increase uncertainty regarding the presence or absence of a feature. However, eliminating the possibility that a feature is present or absent would still be possible from other bits indicating the presence or absence of an adjacent feature.

Similarly, as the number of bits increases, the certainty about the presence or absence of a feature also increases.

As another example, as described above, the ordering or arrangement of the bits is independent of the isomorphic reconstruction of the graph represented by the bits. All that is needed is a known correspondence between the bits and particular nodes/structures in the graph.

In some implementations, patterns of activity within a neural network may be encoded within the representation of reference numeral 1200 ( FIGS. 12 , 13 and 14 ). In general, patterns of activity in a neural network are the result of several properties of the neural network, such as, for example, the structural connections between the nodes of the neural network, weights between nodes and a whole host of possible other parameters. to be. For example, in some implementations, the neural network may be trained prior to encoding the patterns of activity in the representation 1200 .

However, regardless of whether the neural network is trained or untrained, for a given outcome, the response pattern of an activity can be thought of as a "representation" or "abstraction" of its input within that neural network. . Thus, although the representation of reference number 1200 can be seen as a linearly-visible collection of (in some cases binary) digits, each of those digits encodes a relationship or correspondence between a particular input and an associated activity within the neural network. can do.

17 , 18 , 19 , 20 are schematic diagrams using representations of the occurrence of topological structures in activity in a neural network in four different classification systems 1700 , 1800 , 1900 , 2000 . Each of the classification systems 1700 and 1800 classifies representations of patterns of activity in a neural network as part of an input classification. Each of the classification systems 1900 and 2000 classifies approximations of representations of patterns of activity in the neural network as part of the input classification. In common systems 1700, 1800, the patterns of activity to be represented originate from and are read from a source neural network device 1705 that is part of the classification system 1700, 1800. In contrast, in the classification systems 1900, 2000, the closely represented patterns of activity occur in a source neural network device that is not part of the classification system 1700, 1800. Nevertheless, an approximation of the representation of such patterns of activity is read from an approximator 1905 which is part of the classification systems of reference 1900, 2000.

In further detail, returning to FIG. 17 , the classification system 1700 includes a source neural network 1705 and a linear classifier 1710 . The source neural network 1705 is a neural network device configured to receive an input and present representations of occurrences of topological structures in activity within the source neural network 1705 . In the illustrated implementation, the source neural network 1705 includes an input layer 1715 that receives the input. However, this is not necessarily the case. For example, in some implementations, some or all of the input may be injected into different layers and/or edges or nodes throughout the source neural network 1705 .

The source neural network 1705 can be any of a variety of different types of neural networks. In general, the source neural network 1705 is a cyclic neural network, such as, for example, a cyclic neural network modeled on a biological system. In some cases, the source neural network 1705 can model a degree of morphological, chemical, and other properties of a biological system. In general, the source neural network 1705 is implemented on one or more computing devices with a relatively high level of computational power, eg, a supercomputer. In such a case, the classification system 1700 will typically be a distributed system in which the remote classifier 1710 communicates with the source neural network 1705 via, for example, a data communication network.

In some implementations, the source neural network 1705 may not be trained and the expressed activity may be an intrinsic activity of the source neural network 1705 . In other implementations, the source neural network 1705 may be trained and the represented activity may implement this training.

The representations read from the source neural network 1705 may be representations such as the representation 1200 ( FIGS. 12 , 13 , 14 ). The representations can be read from the source neural network 1705 in many ways. For example, in the example shown above, the source neural network 1705 includes “reader nodes” that read patterns of activity among other nodes in the source neural network 1705 . In other implementations, activity in the source neural network 1705 is read by a data processing component programmed to monitor the source neural network 1705 for relatively high-order patterns of activity. In yet other implementations, the source neural network 1705 can include an output layer, eg, when the source neural network 1705 is implemented as a feed-forward neural network, the representation 1200 from the output layer is can be read

The linear classifier 1710 is a device that classifies an object - ie, representations of patterns of activity in the source neural network 1705 - based on a linear combination of properties of the object. The linear classifier 1710 includes an input 1720 and an output 1725 . An input 1720 is coupled to receive representations of patterns of activity in the source neural network 1705 . In other words, representations of patterns of activity in source neural network 1705 are feature vectors representing characteristics of an input to source neural network 1705 , which source neural network 1705 is linear to classify that input. The source neural network used by the classifier 1710 . The linear classifier 1710 can receive representations of patterns of activity within the source neural network 1705 in various ways. For example, the representations of patterns of activity may be received as separate events or as a continuous stream over a real-time or non-real-time communication channel.

An output stage 1725 is coupled to output the classification result from the linear classifier 1710 . In the implementation shown above, output 1725 is schematically shown as a parallel port with multiple channels. This is not necessarily the case. For example, the output stage 1725 may output the classification result through a serial port or a port with combined parallel and serial capabilities.

In some implementations, the linear classifier 1710 may be implemented on one or more computing devices with relatively limited computational power. For example, the linear classifier 1710 may be implemented on a personal computer or mobile computing device such as a smartphone or tablet.

In FIG. 18 , a classification system 1800 includes a source neural network 1705 and a neural network classifier 1810 . The neural network classifier 1810 is a neural network device that classifies an object - ie, representations of patterns of activity in the source neural network 1705 - based on a non-linear combination of properties of the object. In the implementation shown above, the neural network classifier 1810 is a feedforward network comprising an input layer 1820 and an output layer 1825 . With the linear classifier 1710 , the neural network classifier 1810 can receive representations of patterns of activity in the source neural network 1705 in various ways. For example, representations of patterns of activity may be received as separate events or as a continuous stream over a real-time or non-real-time communication channel.

In some implementations, the neural network classifier 1810 can perform inferences on one or more computing devices with relatively limited computational power. For example, the neural network classifier 1810 may be implemented on a mobile computing device in a personal computer or a neural processing unit of a mobile computing device such as a smartphone or tablet. Similar to the classification system 1700 , the classification system 1800 will generally be a distributed system in which the neural network classifier 1810 communicates with the source neural network 1705 via, for example, a data communication network.

In some implementations, the neural network classifier 1810 is configured with a convolutional neural network including, for example, convolutional layers, pooling layers, and fully-connected layers. It may be the same deep neural network. The convolution layer may generate feature maps using, for example, linear convolution filters and/or non-linear activation functions. Pooling layers reduce the number of parameters and control overfitting. The calculations performed by different layers in the image classifier 1820 may be defined in different ways in different implementations of the image classifier 1820 .

In FIG. 19 , a classification system 1900 includes a source approximator 1905 and a linear classifier 1710 . As explained further below, the source approximator 1905 is a relatively simple neural network that receives an input - either in the input layer 1915 or elsewhere - and a relatively more complex neural network. Outputs a vector approximating the representation of topological structures occurring in the patterns of my activity. For example, the source approximator 1905 is a circulating source neural network, such as, for example, a cyclic neural network that is modeled on a biological system and includes degrees of morphological, chemical, and other properties of the biological system. can be trained to approximate . In the implementation shown above, the source approximator 1905 includes an input layer 1915 and an output layer 1920 . The input layer 1915 is combinable to receive input data. The output layer 1920 is combined to output an approximation of the representation of activity within the neural network device for reception by the input 1720 of the linear classifier. For example, the output layer 1920 may output an approximation 1200' of the representation 1200 (FIGS. 12, 13, 14). Incidentally, the representation of reference numeral 1200 schematically shown in FIGS. 17 and 18 and the approximation 1200' of the representation 1200 schematically shown in FIGS. 19 and 20 coincide. This is for the side only. In general, the approximation of reference numeral 1200' will differ from the representation of reference numeral 1200 in at least some ways. Notwithstanding these differences, the linear classifier 1710 can classify an approximation of 1200'.

In general, the source approximator 1905 can perform inference on one or more computing devices with relatively limited computational power. For example, the source approximator 1905 may be implemented on a mobile computing device in a personal computer or a neural processing unit of a mobile computing device such as a smartphone or tablet. In general and compared to sorting systems 1700, 1800, the sorting system 1900 is usually combined within a single housing, for example in one of the same data processing devices or by hardwired connection. It will be housed with a source approximator 1905 and a linear classifier 1710 implemented in data processing devices.

In FIG. 20 , a classification system 2000 includes a source approximator 1905 and a neural network classifier 1810 . The output layer 1920 of the source approximator 1905 is combined to output an approximation 1200 ′ of the representation of activity in the neural network device for reception by the input 1820 of the neural network classifier 1810 . Notwithstanding the differences between approximation 1200' and representation 1200, neural network classifier 180 can classify approximation 1200'. In general and similar to the classification system 1900, the classification system 2000 is a data processing device that is usually combined within a single housing, for example, either in one of the same data processing devices or by a hardwired connection. It will be housed together with a source approximator 1905 and a neural network classifier 1810 implemented on them.

21 is a schematic illustration of an edge device 2100 comprising a local artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network. In this environment, the local artificial neural network may be, for example, an artificial neural network running entirely on one or more local processors that does not require a communication network to exchange data. Typically, the local processors will be connected by hardwired connections. In some examples, the local processors may be housed within a single housing, such as a single personal computer or single handheld mobile device. In some instances, the local processors may be under control and accessed by a single individual or a limited number of individuals. In practice, a simpler and/or less highly trained but more complex source neuron to train a more exclusive second neural network (e.g., using supervised or reinforcement learning techniques). By using the representation of occurrences of topological structures in a network, even individuals with limited computing resources and a limited number of training samples can train a neural network as desired. Storage requirements and computational complexity during training are reduced and resources such as battery life are saved.

In the implementations shown above, the edge device 2100 includes an optical imaging system 2110 , image processing electronics 2115 , a source approximator 2120 , a representation classifier 2125 , and a communication controller and interface 2130 . It is schematically shown as a security-camera device comprising:

The optical imaging system 2110 may include, for example, one or more lenses (or even pinholes) and a CCD device. Image processing electronics 2115 may read the output of optical imaging system 2110 and may generally perform basic image processing functions. Communication controller and interface 2130 is a device configured to control the flow of information to and from the device 2100 . As described further below, operations that the communication controller and interface 2130 may perform are to send images of interest to and receive training information from other devices. Communication controller and interface 2130 may thus include, for example, both a data transmitter and a receiver capable of communicating via data port 2135 . The data port 2135 may be a wired port, a wireless port, an optical port, or the like.

The source approximator 2120 is a relatively simple neural network that is trained to output a vector approximating the representation of topological structures appearing in patterns of activity in a relatively more complex neural network. For example, the source approximator 2120 may be used to approximate a circulating source neural network, such as, for example, a cyclic neural network modeled on a biological system and including degrees of morphological, chemical, and other properties of the biological system. can be trained.

The representation classifier 2135 is either a neural network classifier or a linear classifier coupled to receive an approximation of a representation of patterns of activity in the source neural network from the source approximator 2121 and output a classification result. Representation classifier 2125 may be, for example, a deep neural network, such as a convolutional neural network that includes convolutional layers, pooling layers, and fully-connected layers. The convolution layer may generate feature maps using, for example, linear convolution filters and/or non-linear activation functions. Pooling layers reduce the number of parameters and control overfitting. The calculations performed by different layers in the representation classifier 2125 may be defined in different ways in different implementations of the representation classifier 2125 .

In some implementations, in operation, the optical imaging system 2110 can generate raw digital images. Image processing electronics 2115 can read raw images and will normally perform at least some basic image processing functions. A source approximator 2120 may receive images from the image processing electronics 2115 and infer to output a vector approximating a representation of topological structures occurring within patterns of activity in a relatively more complex neural network. operations can be performed. This approximation vector is input to an expression classifier 2125 that determines whether the approximation vector satisfies one or more sets of classification criteria. Examples include facial recognition and other machine vision operations. When expression classifier 2125 determines that the approximation vector satisfies a set of classification criteria, expression classifier 2125 may instruct communication controller and interface 2130 to send information about the images. For example, the communication controller and interface 2130 may transmit the image itself, the classification, and/or other information about the images.

Occasionally, it may be desirable to change the sorting process. In this case, the communication controller and interface 2130 may receive the training set. In some implementations, the training set may include raw or processed image data and representations of topological structures occurring in patterns of activity within a relatively more complex neural network. Such a training set may be used to retrain the source approximator 2120 using, for example, supervised learning or reinforcement learning techniques. In particular, the representations are used as a target answer vector and represent the desired result of the source approximator 2120 processing the raw or processed image data.

In other implementations, the training set may include representations of topological structures occurring in patterns of activity within a relatively more complex neural network and a desired classification of such representations of topological structures. Such a training set may be used to retrain the neural network representation classifier 2125 using, for example, supervised learning or reinforcement learning techniques. In particular, the desired classification is used as a target answer vector and represents the desired result of the expression classifier 2125 processing the representations of topological structures.

Irrespective of whether the source approximator 2120 or the expression classifier 2125 is retrained, the inference operations at the device 2100 can be performed on large sets of training data and environments without time- and computing power-intensive iterative training. It can be easily adapted to change purposes and purposes.

22 is a schematic illustration of a second edge device 2200 comprising a local artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in the source neural network. In the implementation shown above, the second edge device 2200 is schematically depicted as a mobile computing device, such as a smartphone or tablet. Device 2200 includes an optical imaging system (not shown), image processing electronics 2215 , expression classifier 2225 , communication controller and interface 2230 (eg, on the back side of device 2200 ). include These components may have characteristics and include an optical imaging system 2110 , image processing electronics 2115 , a representation classifier 2125 , a communication controller and interface 2130 , and a data port of the device 2100 ( FIG. 21 ). It is possible to perform actions corresponding to the actions of (2135).

The illustrated implementations of device 2200 further include one or more additional sensors 2240 and a multiple-input source approximator 2245 . The sensor(s) 2240 may sense one or more characteristics of the environment surrounding the device 2200 or of the device 2200 itself. For example, in some implementations, the sensor 2240 can be an accelerometer that senses the acceleration that the device 2200 will experience. As another example, in some implementations, the sensor 2240 can be an auditory sensor, such as a microphone, that senses noise in the vicinity of the device 2200 . Still other examples of sensor 2240 include chemical sensors (eg, “artificial nose” and the like), humidity sensors, radiation sensors, and the like. In some cases, the sensor 2240 is coupled to processing electronics capable of reading the output of the sensor 2240 (or other information such as, for example, a contact list or map) and performing basic processing functions. Different implementations of sensor 2240 may thus have different “modality” in that the physical sensed physical parameter varies from sensor to sensor.

The multiple-input source approximator 2245 is a relatively simple neural network that is trained to output a vector that approximates the representation of topological structures appearing in patterns of activity within a relatively more complex neural network. For example, the multi-input source approximator 2245 may be, for example, a cyclic neural network modeled on a biological system and including degrees of morphological, chemical, and other properties of the biological system. It can be trained to approximate the source neural network.

Unlike the source approximator 2120 , the multi-input source approximator 2245 is coupled to receive raw or processed sensor data from multiple sensors and based on that data activity in a relatively more complex neural network. Returns an approximation of the representation of topological structures occurring in the patterns of For example, the multi-input source approximator 2245 can be used, for example, from the image processing electronics 2215 as well as audio, acceleration, chemical, or other data from the one or more sensors 2240 . The processed image data may be received. The multi-input source approximator 2245 may be, for example, a deep neural network, such as a convolutional neural network that includes convolutional layers, pooling layers, and fully-connected layers. The calculations performed by the different layers in the multi-input source approximator 2245 may be dedicated to a single type of sensor data or to multiple modalities of sensor data.

Irrespective of the particular organization of the multiple-input source approximator 2245, the multiple-input source approximator 2245 returns an approximation based on raw or processed sensor data from multiple sensors. For example, the processed image data from the image processing electronics 2215 and the auditory data from the microphone sensor 2240 are topologically occurring in patterns of activity within a relatively more complex neural network that received the same data. may be used by the multi-input source approximator 2245 to approximate the representation of structures.

Occasionally, it may be desirable to change the sorting process in device 2200 . In this case, the communication controller and interface 2230 may receive the training set. In some implementations, the training set may include raw or processed image data, chemical data or other data, and representations of topological structures occurring in patterns of activity within a relatively more complex neural network. Such a training set may be used to retrain the multi-input source approximator 2245 using, for example, supervised learning or reinforcement learning techniques. In particular, the expressions are used as target answer vectors and represent the desired result of the multi-input source approximator 2245 processing the raw or processed image data or sensor data.

In other implementations, the training set may include representations of topological structures occurring in patterns of activity within a relatively more complex neural network and a desired classification of such representations of topological structures. Such a training set may be used to retrain the neural network representation classifier 2225 using, for example, supervised learning or reinforcement learning techniques. In particular, the desired classification is used as a target answer vector and represents the desired result of the expression classifier 2225 processing the representations of topological structures.

Irrespective of whether the multi-input source approximator 2245 or the representation classifier 2225 is retrained, the inferential operations at the device 2200 are iterative and time- and compute-power intensive and with large sets of training data. Can be easily adapted to change environments and objectives without training.

23 is a schematic illustration of a system 2300 in which local artificial neural networks can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network. Target neural networks are relatively simple and can be implemented on less expensive data processing systems, while the source neural network is relatively complex and can be implemented on more expensive data processing systems.

The system 2300 includes various devices with local neural networks 2305 , a telephone base station 2310 , a wireless access point 2315 , a server system 2320 , and one or more data communication networks 2325 .

Local neural network devices 2305 are devices configured to process data using computationally-less-intensive target neural networks. As shown, local neural network devices 2305 can be any one of mobile computing devices, cameras, automobiles, or a number of other appliances, equipment, and mobile components and devices of different make and model within each category. can be implemented. Different local neural network devices 2305 may belong to different owners. In some implementations, access to the data processing function of the local neural network devices 2305 will generally be limited to these owners and/or those designated by the owner.

Each of the local neural network devices 2305 includes one or more source approximators that are trained to output a vector approximating a representation of topological structures occurring within patterns of a relatively more complex neural network. For example, the relatively more complex neural network may be, for example, a circulating source neural network, such as a cyclic neural network modeled on a biological system and including a degree of morphological, chemical, and other properties of the biological system. have.

In some implementations, in addition to processing data using source approximators, local neural network devices 2305 can target representations of topological structures occurring in patterns of activity within a relatively more complex neural network. It can also be programmed to re-train the source approximators for use as vectors. For example, the local neural network devices 2305 may be programmed to perform one or more iterative training techniques (eg, gradient descent or stochastic gradient descent). have. In other implementations, source approximators in local neural network devices 2305 may interact with the local neural network devices 2305 to train source approximators or, for example, by a dedicated training system. can be trained by a training system installed on a personal computer.

Each local neural network device 2305 includes one or more wireless or wired data communication components. In the implementation shown above, each local neural network device 2305 includes at least one wireless data communication components, such as a mobile phone transceiver, a wireless transceiver, or both. The mobile phone transceivers may exchange data with a phone base station 2310 . The wireless transceivers may exchange data with a wireless access point 2315 . Each local neural network device 2305 may also exchange data with peer mobile computing devices.

Telephone base station 2310 and wireless access point 2315 are coupled for data communication with one or more data communication networks 2325 and can exchange information with server system 2320 via the network(s). So the local neural network devices 2305 may also be normally in data communication with the server system 2320 . However, this is not necessarily the case. For example, in implementations where local neural network devices 2305 are trained by other data processing devices, local neural network devices 2305 only need to communicate data with these other data processing devices at least once. there is only this

Server system 2320 is a system of one or more data processing devices programmed to perform data processing activities in accordance with one or more sets of machine-readable instructions. The activities may include serving training sets to training systems for mobile computing devices 2305 . As described above, the training systems may be internal to the mobile local neural network devices 2305 itself or may be on one or more other data processing devices. The training sets may include corresponding data and representations of occurrences of topological structures corresponding to activity in a source neural network.

In some implementations, server system 2320 also includes a source neural network. However, this is not necessarily the case and server system 2320 may receive training sets from another system of data processing device(s) implementing the source neural network.

In operation, after server system 2320 receives a training set (from a source neural network found in subsystem 2320 itself or elsewhere), server system 2320 is 2305) may serve the training set. Source approximators in target local neural network devices 2305 may be trained using the training set, thereby allowing the target neural networks to approximate operations of the source neural network.

24 , 25 , 26 , 27 are schematic diagrams of using representations of the occurrence of topological structures in activity in a neural network in four different systems 2400 , 2500 , 2600 , 2700 . Systems 2400 , 2500 , 2600 , 2700 may be configured to perform any one of several different operations. For example, systems 2400 , 2500 , 2600 , 2700 may perform object localization operations, object detection operations, object segmentation operations, object detection operations, prediction operations, action selection operations, etc. can

Object localization operations locate an object in an image. For example, a bounding box may be built around an object. In some cases, object localization may be combined with object recognition, in which the localized object is labeled with an appropriate indicia.

Object detection operations classify image pixels as either belonging to a particular class (eg, belonging to an object of interest) or not. In general, object detection is performed by grouping pixels and forming bounding boxes around the pixel groups. The bounding box must fit snugly around the object.

Object segmentation usually assigns class labels to each image pixel. So, it is common for object segmentation rather than bounding boxes to proceed on a per-pixel basis and require only a single label to be assigned to each pixel.

Predictive operations attempt to draw conclusions that are outside the bounds of the observed data. Although predictive operations may attempt to predict occurrences in the future (eg, based on information about past and present states), predictive operations may attempt to predict occurrences in the past and present states based on incomplete information about those states. We also try to draw conclusions about

Action selection actions attempt to select an action based on a set of conditions. Behavior selection operations are traditionally decomposed into different approaches such as symbol-based systems (traditional planning), distributed solutions, and reactive or dynamic planning.

Each of the classification systems 2400 and 2500 performs a desired action on representations of patterns of activity in the neural network. Each of the systems 2600 and 2700 performs a desired operation with respect to those approximating representations of patterns of activity in the neural network. In the systems 2400, 2500, the patterns of activity represented occur in and read from the source neural network device 1705 that is part of the systems 2400, 2500. In contrast, in the systems 2400, 2500, the approximated and expressed patterns of activity occur within the source neural network device that is not part of the systems 2400, 2500. Nevertheless, an approximation of the representation of those patterns of activity is read from the approximator 1905 which is part of the systems 2400, 2500.

In further detail, turning to FIG. 24 , the system 2400 includes a source neural network 1705 and a linear processor 2410 . The linear processor 2410 is a device that performs operations based on a linear combination of characteristics (or approximations of such representations) of representations of patterns of activity in a neural network. The operations may be, for example, object localization operations, object detection operations, object segmentation operations, object detection operations, prediction operations, action selection operations, and the like.

The linear processor 2410 includes an input 2420 and an output 2425 . Input 2420 is coupled to receive representations of patterns of activity in source neural network 1705 . The linear processor 2410 may receive representations of patterns of activity within the source neural network 1705 in various ways. For example, representations of patterns of activity may be received as separate events or as a continuous stream over a real-time or non-real-time communication channel. An output stage 2525 is coupled to output a classification result from the linear classifier 2410 . In some implementations, the linear processor 2410 may be implemented on one or more computing devices with relatively limited computational power. For example, linear processor 2410 may be implemented on a personal computer or on a mobile computing device such as a smartphone or tablet.

24 , the system 2400 includes a source neural network 1705 and a linear processor 2410 . The linear processor 2410 is a device that performs operations based on a linear combination of characteristics (or approximations of such representations) of representations of patterns of activity in a neural network. The operations may be, for example, object localization operations, object detection operations, object segmentation operations, prediction operations, action selection operations, and the like.

The linear processor 2410 includes an input 2420 and an output 2425 . Input 2420 is coupled to receive representations of patterns of activity in source neural network 1705 . The linear processor 2410 may receive representations of patterns of activity within the source neural network 1705 in various ways. For example, representations of patterns of activity may be received as separate events or as a continuous stream over a real-time or non-real-time communication channel. An output stage 2525 is coupled to output a classification result from the linear classifier 2410 . In some implementations, the linear processor 2410 may be implemented on one or more computing devices with relatively limited computational power. For example, linear processor 2410 may be implemented on a personal computer or on a mobile computing device such as a smartphone or tablet.

In FIG. 25 , the classification system 2500 includes a source neural network 1705 and a neural network 2510 . Neural network 2510 is a neural network device configured to perform operations based on a linear combination of characteristics (or approximations of such representations) of representations of patterns of activity within the neural network. The operations may be, for example, object localization operations, object detection operations, object segmentation operations, prediction operations, action selection operations, and the like. In the implementation shown above, neural network 2510 is a feedforward network comprising an input layer 2520 and an output layer 2525 . Like the linear processor at 2410 , neural network 2510 may receive representations of patterns of activity within source neural network 1705 in a variety of ways.

In some implementations, neural network 2510 can perform inferences on one or more computing devices with relatively limited computational power. For example, neural network 2510 may be implemented on a mobile computing device within a neural processing unit, such as a personal computer or mobile computing device such as a smartphone or tablet. Similar to the system at 2400, the system at 2500 will generally be a distributed system in which the remote neural network 2510 communicates with the source neural network 1705 via, for example, a data communication network. In some implementations, neural network 2500 may be, for example, a deep neural network, such as a convolutional neural network.

In FIG. 26 , the system 2600 includes a source approximator 1905 and a linear processor 2410 . Notwithstanding the differences between the approximation 1200' and the representation 1200, the processor 2410 may still perform operations with respect to the approximation 1200'.

In FIG. 27 , the system 2700 includes a source approximator 1905 and a neural network 2510 . Notwithstanding the differences between the approximation 1200 ′ and the representation 1200 , the neural network 2510 may still perform operations with respect to the approximation 1200 ′.

In some implementations, the systems 2600 and 2700 may be implemented on an edge device, such as the edge devices 2100 and 2200 ( FIGS. 21 , 22 ), for example. In some implementations, systems 2600 and 2700 can be trained using representations of occurrences of topological structures corresponding to activity in a source neural network, such as system 2300 ( FIG. 23 ). It can be implemented as part of an existing system.

28 is a schematic illustration of a system including artificial neural networks that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network. Reinforcement learning is a type of machine learning in which an artificial neural network learns from feedback about the results of actions taken in response to the artificial neural network's decisions. A reinforcement learning system moves from one state to another in the environment. A reinforcement learning system moves from one state to another in the environment by performing actions and receiving information characterizing the new state and rewards and/or regrets characterizing the success (or no success) of that action. Reinforcement learning attempts to maximize the overall reward (or minimize regret) throughout the learning process.

In the implementation shown above, the artificial neural network in the reinforcement learning system 2800 is a deep neural network 2805 (or other deep learning structure) that uses a reinforcement learning approach. In some implementations, deep neural network 2805 can be a local artificial neural network (such as neural network 2510 ( FIGS. 25 , 27 )), and, for example, a car, airplane, robot, or other device. It can be implemented locally on However, this is not necessarily the case and in other implementations the deep neural network 2805 may be implemented on a system of networked devices.

In addition to the source approximator 1905 and the deep neural network 2805 , the reinforcement learning system 2800 includes an actuator 2810 , one or more sensors 2815 and a teacher module 2820 . In some implementations, the reinforcement learning system 2800 also includes one or more sources 2825 of additional data.

Actuator 2810 is a device that controls a mechanism or system that interacts with environment 2830 . In some implementations, actuator 2810 controls a physical mechanism or system (eg, steering of a car or positioning of a robot). In other implementations, the actuator 2810 may control a virtual mechanism or system (eg, a virtual game board or investment portfolio). Thus, the environment 2830 may also be physical or virtual.

Sensor(s) 2815 are devices that measure characteristics of the environment 2830 . At least some of the measurements characterize interactions between the controlled mechanism or system and other aspects of the environment 2830 . As the actuator 2810 steers the vehicle, the sensor(s) 2815 measures one or more of the vehicle's speed, direction, and acceleration, the vehicle's proximity to other characteristics, and the response of the other characteristics to the vehicle. can do. As another example, when the actuator 2810 controls an investment portfolio, the sensor(s) 2815 may measure the value and risk associated with that portfolio.

In general, both the source approximator 1905 and the teacher module 2820 are coupled to receive at least some of the measurements made by the sensor(s) 2815 . For example, the source approximator 1905 may receive measurement data at the input layer 1915 and output an approximation 1200' of the representation of topological structures occurring in patterns of activity within the source neural network. have.

The teacher module 2820 interprets the measurements received from the sensor(s) 2815 and provides a reward and/or regret to the deep neural network 2805 . The reward is positive and indicates successful control of the mechanism or system. Regret is negative and indicates no success or less than optimal control. In general, the teacher module 2820 also provides characterization of the measures along with the reward/remorse for reinforcement learning. In general, the characterization of the measurements is an approximation (such as the approximation of 1200') of the representation of topological structures occurring within patterns of activity in the source neural network. For example, the teacher module 2820 can read the approximations 1200' output from the source approximator 1905 and pair the read approximations 1200' with corresponding reward/regret values.

In many implementations, reinforcement learning does not occur in the system 2800 in real time or during active control of the actuator 2810 by the deep neural network 2805 . Rather, training feedback is collected by the teacher module 2820 and can be used for reinforcement training when the deep neural network 2805 is not actively issuing commands to the actuators 2810 . For example, in some implementations, the teacher module 2820 may be remote from the deep neural network 2805 and may be in intermittent data communication with the deep neural network 2805 . Whether reinforcement learning is intermittent or continuous, the deep neural network 2805 can be developed using information received from the teacher module 2820 to optimize rewards and/or reduce regret.

In some implementations, system 2800 also includes one or more sources 2825 of additional data. The source approximator 1905 can also receive data from the data sources 2825 at the input layer 1915 . In such instances, the approximation 1200 ′ would result from processing both sensor data and data from data sources 2825 .

In some implementations, data collected by one reinforcement learning system 2800 may be used for training or reinforcement learning of other systems, including other reinforcement learning systems. For example, characterizing the measures, along with the reward/repentance values, may include the teacher module as a data exchange system that collects such data from various reinforcement learning systems and redistributes the data among the reinforcement learning systems. (2820). Also, as described above, characterizing the measurements may be an approximation of the representation of topological structures occurring within patterns of activity in the source neural network, such as the approximation of reference numeral 1200'.

The particular operations performed by reinforcement learning system 2800 will of course depend on that particular operating environment. For example, in environments where the source approximator 1905 , the deep neural network 2805 , the actuator 2810 , and the sensors 2815 are part of an automobile, the deep neural network 2805 may control the automobile while steering the object. Localization and/or detection operations may be performed.

In implementations where data collected by reinforcement learning system 2800 is used for reinforcement learning or training of other systems, reward/repentance values and the state of the environment when object localization and/or detection operations were performed. Approximations 1200' that characterize ? may be provided to the data exchange system. The data exchange system can then distribute its reward/regret values and approximations 1200' to other reinforcement learning systems 2800 associated with other vehicles for reinforcement learning in those other vehicles. . For example, reinforcement learning may be used to improve object localization and/or detection operations in the second vehicle using the reward/repentance values and approximations 1200 ′.

However, the actions learned in other vehicles need not coincide with the actions performed by the deep neural network 2805 . For example, compensation/regret values based on travel time and approximations 1200 ′ resulting from input of sensor data characterizing an unexpected wet road at a location identified by, for example, GPS data source 2825 , for example. can be used for route planning in other vehicles.

Embodiments of the patent objects and operations described herein may be in digital electronic circuitry, or in computer software, firmware, or hardware including the structures disclosed herein and their equivalents, or in combinations of one or more thereof. can be implemented. Embodiments of the patented subject matter described herein are one or more computer programs encoded on a computer storage medium, ie, one or more modules of computer program instructions, for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. can be implemented as Alternatively or additionally, the program instructions are an artificially generated propagated signal, eg, a machine-generated signal, generated for encoding information for transmission to a receiver device suitable for execution by a data processing device. It can be encoded on electrical, optical, or electromagnetic field signals. A computer storage medium may be contained within or be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. Moreover, if the computer storage medium is not a propagated signal, the computer storage medium may be the source or destination of computer program instructions encoded within an artificially generated propagated signal. The computer storage medium may also be included in or be one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

The term "data processing apparatus" includes all types of apparatus, devices, and machines, including by way of example a programmable processor, computer, system on chip, or many or combinations of the foregoing. The device may include a special purpose logic circuit, for example, a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The device may include, in addition to hardware, code that creates an execution environment for the computer program in question, for example processor firmware, protocol stack, database management system, operating system, cross-platform runtime environment, virtual machine, or one or more of them. Combinations may also be included. The device and execution environment may realize a variety of different computing model infrastructures, such as web services, distributed computing and grid computing infrastructure.

A computer program (also known as a program, software, software application, script, or code) may be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and the computer program stands alone It can be deployed as a program or in any form, including modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program can, but need not, correspond to a file in a file system. A program may be in part of a file holding another program or data (eg, one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple consolidated files. ) (eg, files that store portions of one or more modules, subprograms, or code). A computer program may be deployed for execution on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a communication network.

The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by a special purpose logic circuit, for example a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and the apparatus also implements as the special purpose logic circuit. can be

Processors suitable for computer program execution include, by way of example, both general purpose microprocessors and special purpose microprocessors, and one or more processors of any type of digital computer. Generally, the processor will receive instructions and data from either read-only memory or random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with the instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, eg, a magnetic disk, a magneto-optical disk, an optical disk, or one or more mass storage devices for receiving and transferring data. will be operatively coupled with However, the computer need not be equipped with such devices. Moreover, a computer may be located within another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., to name a few) for example, in a Universal Serial Bus (USB) flash drive). Devices suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disk; and all aspects of non-volatile memory, media and memory devices, including CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or integrated into, special purpose logic circuitry.

In order to provide interaction with a user, embodiments of the patent subject matter described herein may be implemented on a computer, which computer may include a display device for displaying information to the user, such as a CRT ( a cathode ray tube) or liquid crystal display (LCD) monitor, and a keyboard and pointing device that allows a user to provide input to the computer, such as a mouse or a trackball. Other types of devices may likewise be used to provide interaction with a user; For example, the feedback provided to the user may be any form of sensory feedback, eg, visual feedback, auditory feedback, or tactile feedback; And input from the user may be received in any form, including audible, verbal, or tactile input. Additionally, for example, by sending web pages to the web browser in response to requests received from a web browser on the user's client device, the computer can send documents to and receive documents from the device the user is using. You can interact with that user.

Although this specification contains many specific implementation details, these should not be construed as limitations as to the scope of the invention or as may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. do. Certain features described herein in the context of separate embodiments may also be implemented in combination within a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features are described above as acting in certain combinations and may even be initially declared as such, one or more features from a declared combination may in some cases be deleted from the combination, and the declared combination is It may relate to a subcombination or variation of a subcombination.

Similarly, although acts are shown in a particular order in the drawings, this should not be construed as requiring that such acts be performed in the particular order or sequentially shown or that all shown acts must be performed to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as requiring such separation in all embodiments, and the program components and systems described above may generally be integrated together in a single software product or multiple It should be understood that they may be packaged into software products.

Thus, particular implementations of the subject matter of this patent have been described. Other implementations are within the scope of the following claims. In some cases, the acts recited in the claims may be performed in a different order and still achieve desired results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

Several embodiments have been described. Nevertheless, it will be understood that various modifications may be made. For example, the representation of reference numeral 1200 is a binary representation, where each bit individually represents the presence or absence of a feature in the graph, although other representations of information are possible. A matrix of non-binary digits, eg, a vector or multi-valued, may be used to represent, for example, the presence or absence of features and possibly other characteristics of those features. An example of such a characteristic is the weight of the edges with activity constituting said characteristic.

Accordingly, other embodiments are within the scope of the following claims.

Claims (27)

A device comprising a neural network, comprising:
The neural network is trained to:
in response to a first input, a first representation of the presence or absence of each of a plurality of topological patterns of signal transmission activity that would have occurred between nodes along links in a source recurrent neural network in response to the first input;
the source circulatory neural network is different from the neural network, and
wherein the first representation approximates only the presence or absence of each of the respective topological patterns of signaling activity that would have occurred in the source recurrent neural network in response to the first input;
in response to a second input, a second representation of the presence or absence of each of a plurality of topological patterns of signal transmission activity that would have occurred between nodes along the links in the source recurrent neural network in response to the second input , a second representation approximating only the presence or absence of each of the respective topological patterns of signaling activity that would have occurred in the source recurrent neural network in response to the second input; and
in response to a third input, a third representation of the presence or absence of each of a plurality of topological patterns of signal transmission activity that would have occurred between nodes along the links in the source recurrent neural network in response to the third input , wherein the third representation approximates only the presence or absence of each of the respective topological patterns of signaling activity that would have occurred in the source recurrent neural network in response to the third input. According to claim 1,
wherein all of the topological patterns include three or more nodes in the source recurrent neural network and three or more edges between the nodes. According to claim 1,
wherein the topological patterns comprise directional simplexes. According to claim 1,
wherein the topological patterns surround cavities. According to claim 1,
Each of the first representation, the second representation, and the third representation at the points in time the patterns of signaling activity have a complexity distinguishable from the complexity of another signaling activity responsive to each of the inputs is in the source recurrent neural network. A device that exclusively represents topological patterns that would have occurred. According to claim 1,
and a processor coupled to receive the representations produced by the neural network and process the received representations. 7. The method of claim 6,
wherein the processor comprises a second neural network trained to process the representations produced by the neural network. According to claim 1,
wherein each of the first expression, the second expression, and the third expression includes multi-valued, non-binary digits. According to claim 1,
Each of the first expression, the second expression, and the third expression includes:
representing the presence or absence of the topological patterns without prescribing where the patterns of signaling activity occur within the source circulatory neural network. According to claim 1,
The device comprises a smartphone. delete As a device,
a neural network configured to be coupled to input and process representations of the presence or absence of topological patterns of signal transmission activity that would have occurred between nodes along links in a source recurrent neural network in which the nodes act as accumulators;
The processed topological patterns and input of signal transmission activity occur in response to a plurality of different inputs, the neural network being trained to process the representations and produce response outputs. 13. The method of claim 12,
wherein all of the topological patterns include three or more nodes in the source recurrent neural network and three or more edges between the nodes. 13. The method of claim 12,
wherein the topological patterns comprise directional simplexes. 13. The method of claim 12,
Representations of the presence or absence topological patterns at points in time at which topological patterns of signaling activity have a complexity distinguishable from the complexity of other signaling activity that will respond to each of the inputs may be present or absent in the source recurrent neural network. A device that exclusively represents the presence or absence topological patterns to be performed. The method of claim 12, wherein the device comprises:
a neural network trained to produce, in response to a plurality of different inputs, respective approximations of representations of the presence or absence of topological patterns in patterns of signal transmission activity that would have occurred in a source recurrent neural network in response to the different inputs. Further comprising, the device. 13. The method of claim 12,
wherein the representations of the presence or absence topological patterns include multi-valued, non-binary digits. 13. The method of claim 12,
wherein the representations of the presence or absence topological patterns represent the presence or absence of the topological patterns without specifying where the patterns of signal transmission activity occurred within the source recurrent neural network. delete A method implemented by a neural network device, the method comprising:
inputting a representation of the presence or absence of topological patterns of signal transmission activity that would have occurred between nodes along links in a source recurrent neural network, wherein the signal transmission activity is responsive to input to the source recurrent neural network. phosphorus, input step;
processing the representation in the source recurrent neural network that would have occurred between nodes along links in the source recurrent neural network in response to a plurality of different inputs to the source recurrent neural network. processing, consistent with training the neural network to process different representations of the presence or absence of topological patterns of signal transmission activity; and
outputting a result of the expression processing. 21. The method of claim 20,
wherein all of the topological patterns include three or more nodes in the source recurrent neural network and three or more edges between the nodes. 21. The method of claim 20,
wherein the topological pattern comprises directional simplexes. 21. The method of claim 20,
wherein the topological patterns surround cavities. 21. The method of claim 20,
The representations of topological patterns at the points in time at which the patterns of signaling activity have a complexity distinguishable from the complexity of other signaling activity responding to each of the inputs exclusively represent topological patterns that would have occurred in the source recurrent neural network. how to express. 21. The method of claim 20,
The representations of the topological patterns include multi-valued, non-binary digits. 21. The method of claim 20,
wherein said representations of topological patterns represent the presence or absence of said topological patterns without specifying where said patterns of signaling activity would have occurred within said source circulatory neural network. delete

Images