KR20210008417A - Characterization of activities and information encoding and decoding in recurrent artificial neural networks - Google Patents

Abstract

The methods, systems, and devices include computer programs encoded on a computer storage medium, the computer programs for characterizing activity in a recurrent artificial neural network and for encoding and decoding information. In one aspect, the method includes outputting digits from a recurrent artificial neural network, where each digit is determined whether or not an activity within a particular group of nodes in the recurrent artificial neural network is suitable for a respective pattern of activity. Express whether or not.

Description

Characterization of activities and information encoding and decoding in recurrent artificial neural networks

This specification is directed to characterizing activity within a recurrent artificial neural network.

Characterizing the activity may be used in encoding/decoding signals in environments such as transmission, encryption, and data storage, as well as in identification, for example, for decision moments. It relates also to encoding and decoding information, and to systems and technologies that use the encoded information in various environments. The encoded information may indicate activity in a neural network, for example, a recurrent neural network.

Artificial neural networks are devices inspired by the structural and functional features of the network of biological neurons. In particular, artificial neural networks mimic the information encoding and other processing capabilities of a network of biological neurons using a system of interconnected structures called nodes. The arrangement and strength of connections between nodes in an artificial neural network determine the results of information storage or information processing by that artificial neural network.

Neural networks can be trained to produce the desired signal flow within the network and to achieve the desired information processing or information storage results. In general, training a neural network will change the strength and/or arrangement of connections between nodes during the learning phase. The neural network can be considered trained when sufficiently appropriate processing results are achieved by neural networks for given input sets.

Artificial neural networks can be used in a variety of different devices to perform non-linear data processing and analysis. Non-linear data processing does not satisfy the superposition, that is, the determined variables cannot be written as a linear sum of independent components. Examples of environments in which non-linear data processing is useful are pattern and sequence recognition, speech processing, novelty detection and sequential decision making, complex system modeling, and systems and technologies in various other environments. Includes.

Both encoding and decoding transform information from one shape or presentation to another. Different representations may provide different features that are more or less useful in different applications. For example, some aspects or expressions of information (eg, natural language) may be easier for humans to understand. Other figures or representations may be smaller in size (eg, compressed) and may be easier to transmit or store. Other features or representations can deliberately obscure the information content (for example, the information can be cryptographically encoded).

Regardless of the particular application, the encoding or decoding process will generally follow an algorithm or set of predefined rules that establish a correspondence between information in different shapes or representations. For example, an encoding process that produces a binary code can assign roles or meanings to individual bits according to individual bits in a binary sequence or vector.

This application seeks to provide techniques related to characterizing activity in artificial neural networks.

This application describes techniques related to characterizing activity in artificial neural networks.

As another example, a method for identifying decision moments in a recurrent artificial neural network comprises determining the complexity of patterns of activity within the recurrent artificial neural network, the activity responsive to input into the recurrent artificial neural network, Determining, determining a timing of an activity with a complexity distinguishable from the complexity of another activity responsive to the input, and identifying a decision moment based on the timing of the activity having a distinguishable complexity.

As another example, a method of characterizing activity in a circulating artificial neural network includes identifying subpopulation patterns of activity of the circulating artificial neural network. The method is performed by a data processing device.

As another example, the method may include outputting a binary sequence of zeros and ones (1) from a recurrent artificial neural network, wherein each digit in the sequence is the number of nodes in the recurrent artificial neural network. It indicates whether or not special groups display their respective patterns of activity.

As another example, a method of structuring a recurrent artificial neural network includes characterizing the complexity of patterns of activity that may occur in a recurrent artificial neural network comprising a structured collection of nodes and links between the nodes, and It may include deploying the structure of the recurrent artificial neural network to increase the complexity of the patterns. This method of structuring can also be used, for example, as part of a method of training the circulating artificial neural network.

Other embodiments of these aspects include corresponding systems, apparatus, and computer programs, which computer programs are configured to perform the actions of the above methods and are encoded on computer storage devices.

The particular embodiments of the subject matter described in this specification may be implemented to realize one or more of the following advantages. For example, traditional data processing devices such as digital computers and other computers are programmed to follow a pre-defined sequence of logic when processing information. The moment when the computer reaches the result is thus relatively easy to identify. That is, the completion of the logic sequence embodied in the above programming indicates when the information processing is complete and the computer "reached a decision". The result can be maintained in a relatively long-lived form at the output of the computer's data processor by, for example, a memory device, a set of buffers, etc., and can be accessed for a variety of purposes.

In contrast, as described herein, decision moments within artificial recurrent neural networks can be identified based on the characteristics of the dynamic properties of the neural network during information processing. Rather than waiting for the artificial neural network to reach a pre-defined end of a logical sequence, decision moments within the artificial neural networks can be identified based on the characteristics of the functional states of the artificial neural network during information processing.

In addition, the characteristics of dynamic properties of recurrent artificial neural networks during information processing include:-including characteristics such as activity suitable for subgroup patterns and directional subgroup patterns-various signaling operations including signal transmission, encoding, encryption, and storage. Can be used in the field. In particular, characteristics of the activity in the recurrent artificial neural network during information processing reflect the input and can be considered to be the encoded appearance of the input (ie, the “output” of the recurrent artificial neural network in encoding processes). These characteristics may be transmitted, for example, to a remote receiver, which may be able to decode the transmitted characteristics to reconstruct the input or part of the input.

Moreover, in some cases, the activity in different groups of nodes in the recurrent artificial neural network (e.g., activity suitable for subpopulation patterns and directional subpopulation patterns) is a binary sequence of zeros (0) and one (1). Can be expressed, and each digit of the binary sequence indicates whether the activity fits the pattern or not. Since the activity may be the output of the recurrent artificial neural network in some circumstances, the output of the recurrent artificial neural network can be represented as a vector of binary digits and is compatible with digital data processing.

Further, in some cases, such characterization of the dynamic properties of the recurrent artificial neural network may be used prior to and/or during training to increase the likelihood of complex patterns of activity occurring during information processing. For example, prior to or during training, links between nodes in a recurrent neural network can be intentionally deployed to increase the complexity of activity patterns. This can reduce the time and effort required to train a recurrent artificial neural network.

As another example, such characterization of the dynamic properties of the recurrent artificial neural network can be used to determine the degree of perfection in the training of the recurrent neural network. For example, a recurrent artificial neural network that displays special types of ordering within an activity (e.g., subgroup patterns and directional subgroup patterns) trains more highly than a recurrent artificial neural network that does not display such ordering. Can be considered to be. Indeed, in some cases, the degree of training can be quantified by quantifying the degree of ordering of activities within the recurrent artificial neural network.

Details of one or more implementations described herein are set forth in the accompanying drawings and description below. Other features, aspects, and advantages of the subject matter of the invention will become apparent from the description, drawings and claims.

The effects of the invention are specified individually in the corresponding parts of this specification.

1 schematically shows the structure of a circulatory artificial neural network device.
2 and 3 schematically illustrate the functioning of a recurrent artificial neural network device in different time windows.
4 is a flowchart of a process for identifying decision moments in a recurrent artificial neural network based on the characterization of the activity in the network.
Figure 5 schematically depicts patterns of activity that can be identified within a recurrent artificial neural network and used to identify decision moments.
6 schematically depicts patterns of activity that can be identified within a recurrent artificial neural network and used to identify decision moments.
7 schematically depicts patterns of activity that can be identified within a recurrent artificial neural network and used to identify decision moments.
8 is a schematic illustration of a data table that can be used in determining the complexity or degree of ordering in activity patterns in a recurrent artificial neural network device.
9 is a schematic illustration of determining the timing of activity patterns with distinguishable complexity.
10 is a flow diagram of a process of encoding signals using a recurrent artificial neural network based on characterization of activity in the network.
11 is a flow diagram of a process of decoding signals using a recurrent artificial neural network based on characterization of activity in the network.
12, 13, and 14 are schematic examples of binary representations or representations of topological structures.
15 and 16 schematically show an example of how the presence or absence of features corresponding to different bits is not independent of each other.
17, 18, 19, 20 are schematic illustrations of using a representation of the occurrence of topological structures in activity within a neural network in four different classification systems.
21 and 22 are schematic illustrations of edge devices comprising a local artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in the source neural network.
23 is a schematic illustration of a system in which local artificial neural networks can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
24, 25, 26, 27 are schematic illustrations of using a representation of the occurrence of topological structures in activity in a neural network in four different classification systems.
28 is a schematic illustration of a system comprising artificial neural networks that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network.
Similar reference numbers in the various drawings indicate similar elements.

1 is a schematic diagram of the structure of a recurrent artificial neural network device 100. The recurrent artificial neural network device 100 is a device that mimics information encoding and other processing capabilities of networks of biological neurons using a system of interconnected nodes. The recurrent artificial neural network device 100 may be implemented in hardware, software, or a combination thereof.

The above example of a recurrent artificial neural network device 100 includes a plurality of nodes 101, 102, ..., 107 interconnected by a plurality of structural links. Nodes 101, 102, ..., 107 are discrete information processing structures similar to neurons in biological networks. In general, nodes 101, 102, ..., 107 process one or more input signals received through one or more links 110, and one or more output signals output through one or more links 110 Yields. For example, in some implementations, nodes 101, 102, ..., 107 weight multiple input signals, passing the sum through one or more non-linear active functions, and Output one or more output signals.

Nodes 101, 102, ..., 107 can act as accumulators. For example, nodes 101, 102, ..., 107 will operate according to an integral-and-fire model in which one or more signals accumulate within the first node until a threshold is reached. I can. After reaching the threshold, the first node fires the connected second node by sending an output signal along one or more links 110. In turn, the second node (101, 102, ..., 107) accumulates the received signal, and when the threshold is reached, the second node (101, 102, ..., 107) adds another output signal. To the connected node of.

Structural links 110 are connections capable of transmitting signals between nodes 101, 102, ..., 107. For convenience, all structural links 110 are here from all first of nodes 101, 102, ..., 107 to all second of nodes 101, 102, ..., 107. However, it is treated as being the same bidirectional link carrying signals in the very same way signals are carried from the second node to the first node. However, this is not necessarily the case. For example, some or all of the structural links 110 are from the first of the nodes 101, 102, ..., 107 to the second of the nodes 101, 102, ..., 107 They may be unidirectional links that carry signals to but not from the second node to the first node.

As another example, in some implementations, structural links 110 may have various properties that are not directional or in addition to directionality. For example, in some implementations, different structural links 110 may carry signals of different sizes-the nodes 101, 102, ..., 107 of interconnections with different strengths between each of them. It brings results. As another example, different structural links 110 may carry different types of signals (eg, inhibitory and/or excitatory signals). Indeed, in some implementations, structural links 110 may be modeled on links between somatic cells in biological systems and may reflect at least some of the enormous morphological, chemical and other diversity of such links.

In the above illustrated implementation, the recurrent artificial neural network device 100 is a small group in that all nodes 101, 102, ..., 107 are connected to all other nodes 101, 102, ..., 107. (clique) network (or subnetwork). This is not necessarily the case. Rather, in some implementations, each node 101, 102, ..., 107 is a suitable sub of nodes 101, 102, ..., 107 (by the same links or various links as the case may be). Can be connected to the set.

For clarity of illustration, the recurrent artificial neural network device 100 is shown with only 7 nodes. In general, neural network devices in the real world will contain a very large number of nodes. For example, in some implementations, neural network devices may include hundreds of thousands, millions, or even billions of nodes. Thus, the recurrent neural network device 100 may be part of a larger recurrent artificial neural network (ie, a subnetwork).

In biological neural network devices, the accumulation process and signal transmission process require the passage of time in the real world. For example, the soma of the nerve integrates the inputs received over time, and the transmission of signals from the nerve to the nerve, for example, the speed of signal transmission and the nature and length of the links between the nerves. It requires the times determined by So, the state of a biological neural network device is dynamic and changes over time.

In artificial recurrent neural network devices, time is represented using artificial and mathematical structures. For example, rather than requiring real-world time lapse for signals to be transmitted from node to node, such signals are typically measured in computer clock cycles in terms of artificial units that are not related to real-world time lapse. It can be expressed as if it was done or otherwise. Nevertheless, the state of the artificial circulatory neural network can be described as being "dynamic" in that its state changes with respect to these artificial units.

Note that for convenience these artificial units are referred to herein as "time" units. Nevertheless, it should be understood that these units are artificial and generally do not correspond to the passage of time in the real world.

2 and 3 are schematic illustrations of the functionality of the recurrent artificial neural network device 100 within different time windows. Since the state of the device 100 is dynamic, the functional action of the device 100 can be expressed using a signal transmission activity occurring within a window. It is common for such functional cities to show activity only in some of the links 110. In particular, not all links 110 are shown in these figures as an activity contributing to the functional functioning of the device 100, as not all links 110 generally carry signals within a particular window.

In the figures in Figures 2 and 3, the active link 110 is shown as a relatively thick continuous line connecting a pair of nodes 101, 102, ..., 107. In contrast, the inactive link 110 is shown as intermittent lines. This is for illustrative purposes only. In other words, the structural connections formed by the links 110 exist whether the links 110 are active or not. However, this formalism emphasizes the functioning and activity of device 100.

In addition to schematically showing the existence of an activity along a link, the direction of the activity is also schematically shown. In particular, the relatively thick interruptions showing the activity of links 110 also include arrowheads indicating the direction of signal transmission along the link during the associated window. In general, the direction of signal transmission within a single window is not conclusively enforced that the link is a unidirectional link having the indicated directionality. Rather, in the first functional city for the first time window, the link may be active in the first direction. In the second functional city for the second time window, the link may be active in the opposite direction. However, in some cases, such as, for example, in recurrent artificial neural network device 100 that includes only unidirectional links, the directionality of the signal transmission will definitively indicate the directionality of that link.

In feedforward neural network devices, information travels only in a single direction (ie forwards only) to the output layer of nodes at the end of the network. Feedforward neural network devices indicate that a "decision" has been reached and that information processing is complete by propagation of signals to the output layer through the network.

In contrast, in a recurrent neural network, the connections between nodes form cycles and the activity of the network proceeds dynamically without immediately identifiable decisions. For example, even in a three-node recurrent neural network, a first node may transmit a signal to a second node, and that second node may transmit a signal to a third node in response. In response, the third node may reversely transmit a signal to the first node. The signals received by the first node may respond-at least partially-to signals transmitted from the same node.

Figures 2 and 3, which are schematic functional illustrations, show this in a network that is slightly larger than a three-node recurrent neural network. These functional illustrations shown in FIG. 2 may be an example of an activity in a first window and FIG. 3 may be an example of an activity in a second window immediately following. As shown, the collection of signal transmission activity originates at node 104 and appears to proceed generally clockwise through device 100 during the first window. In the second window, at least some of the signal transmission activity is generally seen as going back to node 104. Even in such an extremely simple city, signal transmission does not proceed in a way that yields clearly identifiable outputs or terminations.

For example, when a circulatory neural network of thousands or more nodes is considered, signal propagation can take place over a very large number of paths, and these signals lack a clearly identifiable "output" location or time. Can be recognized. Although by design the network may return to a quiescent state in which only background signal transmission occurs or even no signal transmission occurs, the inactivity state itself does not represent the results of information processing. Does not. The circulatory neural network always returns to an inactive state regardless of input. The "output" or result of information processing is thus encoded within the activity occurring within the recurrent neural network in response to a particular input.

4 is a flow diagram of a process 400 for identifying decision moments within a recurrent artificial neural network based on activity characterization within that network. The moment of determination is the point at which an activity in a recurrent artificial neural network indicates the results of information processing by the network in response to an input. Process 400 may be performed by a system of one or more data processing devices that perform operations according to the logic of one or more sets of machine-readable instructions. For example, the processor 400 may be performed by the same system of a set of one or more computers running software for implementing the recurrent artificial neural network used in the process 400.

The system performing process 400 receives a notification at 405 that a signal has been input into the recurrent artificial neural network. In some cases, the signal input is, for example, a separate injection event in which information is injected into one or more nodes and/or one or more links of the neural network. In other cases, the signal input is a stream of information injected into one or more nodes and/or links of the neural network over a period of time. The notification indicates that the artificial neural network is actively processing the alert and, for example, is not in an inactive state. In some cases, the notification is received from the neural network itself, for example when the neural network is in an identifiable inactive state.

At 410, the system execution process 400 divides the response activity within the network into a collection of windows. In the case where the injection is a separate event, the windows can subdivide the time between injection and return to the inactive state into several intervals in which the activity displays variable complexity. If the injection is a stream of information, the duration of the injection (and optionally the time until the return to the inactive state after injection is complete) is subdivided into windows in which the activity displays variable complexity. Can be. Various approaches to determining the complexity of an activity are described further below.

In some implementations, the windows all have the same duration, but not necessarily. Rather, in some implementations, the windows can have different durations. For example, in some implementations, the duration increases as the time after a separate injection event occurs increases.

In some implementations, the windows can be a contiguous series of separate windows. In other implementations, the windows overlap in time, so that one window starts before the previous window ends. In some cases, the windows may be moving windows that move in time.

In some implementations, different durations of windows are defined for different determinations of the complexity of the activity. For example, for activity patterns that limit activity occurring between a relatively large number of nodes, the windows are defined for activity patterns that limit activity occurring between a relatively small number of nodes. It can have a longer duration than windows. For example, in the environment of patterns of reference number 500 of the activity (Fig. 5), a window defined to identify an activity suitable for the pattern of reference number 530 is a window defined to identify an activity suitable to the pattern of reference number 505. Can be longer than

The system performing process 400 identifies patterns in activity within the network in different windows at 415. As described further below, patterns within an activity can be identified by treating the functional graph as being a topological space with nodes as points. In some implementations, the identified activity patterns are subpopulations in the functional graph of the network, eg, directed cliques.

At 420, the system performing process 400 determines the complexity of activity patterns in different windows. The complexity may be a measure of the likelihood that an ordered pattern of activity appears within a window. So, activity patterns that appear randomly may be relatively simple. On the other hand, activity patterns showing non-random order are relatively complex. For example, in some implementations, the complexity of an activity pattern can be measured using, for example, simple counts or Betti numbers of the activity pattern.

At 425, the system performing process 400 determines the timing of activity patterns with distinguishable complexity. A particular pattern of activity may be distinguishable, for example, based on the complexity of deviating upwards or deviating downwards from a fixed or variable baseline. In other words, the timing of activity patterns indicating specially high levels or specially low patterns in a non-random order within the activity may be determined.

For example, if the signal input is a separate injection event, for example, deviations from a stable baseline or from a curve that is a characteristic of the average response of the neural network to various different separate injection events It can be used to determine the timing of complex activity patterns. As another example, in the case where the information is input within a stream, large changes in complexity during streaming can be used to determine the timing of distinguishable complex activity patterns.

At 430, the system execution process 400 determines the timing of reading output from the neural network based on the timing of distinguishable complex activity patterns. For example, in some implementations, the output of the neural network can be read at the same time that the distinguishable complex patterns of activity have occurred. In some implementations in which the complexity deviation indicates a relatively high non-random order within the activity, the observed activity patterns themselves may also be taken as an output of the recurrent artificial neural network.

5 is an illustration of patterns 500 of activity that may be identified within a recurrent artificial neural network and used to identify decision moments. For example, patterns 500 can be identified at 415 in process 400 (FIG. 4 ).

Patterns 500 are examples of activity in a recurrent artificial neural network. During application of the patterns 500, the functional graph is treated as being a topological space with nodes as points. Activities within nodes and links that conform to patterns 500 may be recognized as being ordered regardless of the identity of particular nodes and/or links involved in the activity. For example, the first pattern 505 may represent an activity between nodes of reference numerals 101, 104, and 105 in FIG. 2, in which point 0 in the pattern 505 is a node of reference numeral 104, Point 1 is provided with a node of 105, and point 2 is provided with a node of 101. As another example, the first pattern 505 may also represent activity between nodes of reference numerals 104, 105, and 106 in FIG. 3, wherein point 0 in the pattern 505 is a node of reference numeral 106 , Point 1 is a node of reference numeral 104, and point 2 is a node of reference number 105. The sequence of activities within the directional subgroup is also defined. For example, in the pattern of reference numeral 5050, an activity between points 1 and 2 occurs after the activity between points 0 and 1.

In the illustrated implementation, the patterns 500 are mode directional subpopulations or directional simplexes. In such patterns, activity comes from a source node sending signals to all other nodes in the pattern. In patterns 500, such a source node is denoted as point 0 while the other nodes are denoted as points 1, 2, .... Also, in directional subgroups or simplexes, one of the nodes behaves like a sink and receives signals transmitted from all other nodes in the pattern. In patterns 500, such sink nodes are indicated by the highest number pattern in the pattern. For example, in pattern 505, the sink node is indicated by point 2. In pattern 510, the sink node is indicated by point 3. In pattern 515, the sink node is indicated by point 3, and so on thereafter. The activities represented by the patterns 500 are thus ordered in a distinguishable manner.

Each of the patterns 500 has a different number of points and reflects an ordered activity in a different number of nodes. For example, pattern 505 is a 2D-simplex and reflects the behavior within three nodes, pattern 510 is a 3D-simplex and reflects the behavior within four nodes, and so on. As the number of points in the pattern increases, the degree of ordering and the complexity of the action also increase. For example, for a large collection of nodes with a certain level of random activity within a window, some of the activity may fit into a pattern 505 that accidentally deviates from it. However, it is increasingly hopeless that a random activity will fit each of the patterns (510, 515, 520, ...). The presence of an activity suitable for pattern 530 indicates a relatively higher degree and complexity of ordering within that activity, which is the presence of an activity suitable for pattern 505.

As previously described, in some implementations, different duration windows can be defined for different decisions about the complexity of the activity. For example, when an activity suitable for pattern 530 is about to be identified, longer duration windows may be used than when an activity suitable for pattern 505 is about to be identified.

6 is an illustration of patterns of activity 600 that can be identified within a recurrent artificial neural network and used to identify decision moments. For example, patterns 600 can be identified at 415 in process 400 (FIG. 4 ).

Similar to patterns 500, patterns 600 are examples of activity in a recurrent artificial neural network. However, the patterns 600 differ from the strict ordering of the patterns 500 in the sense that not all of the patterns 600 are directional subpopulations or directional simplexes. In particular, the patterns 605 and 610 have a lower directionality than the pattern of reference numeral 515. In practice, pattern 605 has no sink nodes at all. Nevertheless, the patterns 605 and 610 indicate an ordered degree of activity that exceeds what would be expected through random contingency and can be used to determine the complexity of the activity within the recurrent artificial neural network.

7 is an illustration of patterns of activity 700 that may be identified within a recurrent artificial neural network and used to identify decision moments. For example, patterns 700 may be identified at 415 in process 400 (FIG. 4 ).

The patterns 700 define patterns containing more points than individual subgroups or simplexes, and directivity of the same dimension (i.e., having the same number of points) surrounding cavities within a group of directional simplexes. These are small groups or groups of directional simplex. By way of example, the pattern of reference numeral 705 includes six different three-point, two-dimensional patterns 505 that together define a homology class of degree 2, while the pattern of reference numeral 710 is Eight different three-point, two-dimensional patterns 505 that together define a second homology class of (degree) 2. Each of the three-point, two-dimensional patterns 505 in the patterns 705 and 710 may be considered to surround their respective cavity. The nth Betty number associated with the topographical graph provides a count of those homologous classes in the topological representation.

The activity illustrated by patterns, such as the patterns of reference numeral 700, shows a relatively high degree of ordering of activities in the network that are unlikely to occur by random chance. Patterns 700 can be used to characterize the complexity of the activity.

In some implementations, only some patterns of activity are identified while identifying decision moments and/or some of the patterns of identified activity are discarded or otherwise ignored. For example, referring to FIG. 5, an activity suitable for a 5-point, 4-dimensional simplex pattern 515 is suitable for 4-point, 3-dimensional and 3-point, 2-dimensional simplex patterns 510. 505. Includes the original activity. For example, points 0, 2, 3, 4 and points 1, 2, 3, 4 both in the 4-dimensional simplex pattern 515 of FIG. 5 are suitable for the 3-dimensional simplex pattern 510 Do. In some implementations, patterns containing a smaller number of points-and therefore of a lower order-may be discarded during the identification of decision moments and may otherwise be ignored.

As another example, only some patterns of activity need to be identified. For example, in some implementations only patterns with odd points (3, 5, 7, ...) or even even dimensions (2, 4, 6, ...) identify decision moments. Used in

The complexity or degree of ordering in the activity patterns in the recurrent artificial neural network device in different windows can be determined in a variety of different ways. 8 is a schematic illustration of a data table 800 that can be used in such a determination. Data table 800 may be used to determine the complexity of the activity patterns separately or in conjunction with other activities. For example, the data table 800 may be used at 420 in reference numeral 400 (FIG. 4).

In further detail, table 800 includes a numerical count of pattern occurrences during a window "N", where numerical counts of activities matching patterns of different dimensions are in different rows. . For example, in the example shown above, the row of reference number 805 contains a numeric count of occurrences of activity matching one or more three point, two-dimensional patterns (i.e., "2032"), while the reference number The row of 810 contains a numeric count of occurrences of an activity matching one or more four point, three-dimensional patterns (ie, "8772"). Since the occurrence of patterns indicates that the activity has a non-random order, the numeric counts also provide a generalized characterization of the overall complexity of the activity patterns. A table similar to the table of reference numeral 800 may be formed, for example, for each window defined in 410 in process 400 (FIG. 4).

Although table 800 contains separate rows and separate entries for all types of activity patterns, this is not necessarily the case. For example, one or more counts (eg, counts of simpler patterns) may be omitted from table 800 and from determining complexity. As another example, in some implementations, a single row or entry may contain counts of occurrences of multiple activity patterns.

Although FIG. 8 presents the numeric counts in table 800, this is not always necessary. For example, the number count can be expressed as a vector (eg, <2032, 877, 133, 66, 48, ...>). Regardless of how the count is presented, in some implementations, the counts may be expressed in binary and compatible with the digital data processing infrastructure.

In some implementations, for example, the numeric counts of occurrences of the patterns may be weighted or combined to determine the degree or complexity of the ordering at 420 in process 400 (FIG. 4). For example, the Euler property can provide an approximation of the complexity of the activity and is given by:

[Equation 1]

Where S _n is the number of occurrences of a pattern of n points (ie, a pattern of order n-1). The patterns can be, for example, directional subpopulation patterns 500 (FIG. 5).

As another example of how the numerical counts of occurrences of the patterns can be weighted or combined to determine the degree or complexity of the sequencing, in some implementations, pattern occurrences based on the weights of active links are Weight can be assigned. In further detail, as previously described, the strength of the connections between nodes in the artificial neural network can vary, for example, as a result of how active the connection was during training. The occurrence of a pattern of behavior following a collection of relatively stronger links may be weighted differently from occurrence of the same pattern of activity following a collection of relatively weaker links. For example, in some implementations, the sum of weights of active links can be used to weight the occurrence.

In some implementations, the Euler property or other measure of complexity will be normalized by the total number of patterns that enable a given network to form its own structure and/or the total number of matching patterns within a particular window. I can. The normalization with respect to the total number of patterns that enables the network to form is given in Equations 2 and 3 below.

In some implementations, occurrences of higher dimensional patterns involving a larger number of nodes may be weighted more heavily than occurrences of lower dimensional patterns involving a smaller number of nodes. For example, the likelihood of forming directional subgroups decreases sharply as the dimension increases. In particular, in order to form an n-subgroup from n+1 nodes, (n+1)n/2 edges, all of which originate correctly, are required. This probability can be reflected in weighting.

In some implementations, both the dimensionality and directionality of the patterns can be used to weight occurrences of the patterns and determine the complexity of the activity. For example, referring to FIG. 6, occurrences of a 5-point, 4-dimensional pattern 515 are more sensitive to differences in the directionality of the 5-point, 4-dimensional patterns 605 and 610 than occurrences of the 5-point, 4-dimensional pattern 515. Therefore, it can be weighted more heavily

An example of using both the dimension and direction of patterns to determine the degree of complexity or ordering of an action can be given as follows.

[Equation 2]

Here, S _x ^active represents the number of active occurrences of the pattern of n points, and ERN is a calculation for an equivalent random network, that is, a network with the same number of randomly connected nodes. In addition, SC is given as

[Equation 3]

Here, S _x ^silent indicates the number of occurrences of a pattern of n points when the recurrent artificial neural network is quiet and the network can be considered as implementing the total number of possible patterns. In Equations 2 and 3, the patterns may be, for example, directional small group patterns 500 (FIG. 5).

9 is a schematic illustration of determining the timing of activity patterns of distinguishable complexity. The determination shown in FIG. 9 may be performed separately or in conjunction with other activities. For example, the determination may be performed at 425 of process 400 (FIG. 4).

9 includes a graph of reference numeral 905 and a graph of reference numeral 910. The graph of reference numeral 905 shows the occurrence of patterns as a function of time along the x-axis. In particular, the individual occurrences are schematically shown by vertical lines 906, 907, 908, 909. Each row of occurrences can be instances where the activity matches its own pattern or class of patterns. For example, occurrences of the top row may be cases where the behavior matches the patterns of reference number 505 (Fig. 5), and the occurrences of the second row match the patterns of reference number 510 (Fig. 5). The occurrences of the third row may be cases in which the behavior matches the patterns of reference numeral 515 (FIG. 5), and the same applies hereinafter.

The graph of reference numeral 905 also includes interruption line rectangles 915, 920, 925 that schematically represent different time windows when the activity patterns have distinctive complexity. As can be seen, the likelihood that the activity in the recurrent artificial neural network will conform to the pattern indicative of complexity is higher during those windows than outside the windows indicated by the interruption line rectangles 915, 920, 925.

The graph 910 shows the complexity associated with these occurrences as a function of time along the x-axis. The graph of reference numeral 910 shows a first peak 930, which is the complexity corresponding to the window represented by the chopper rectangle 915, and the second peak, 930, which is the complexity corresponding to the window represented by the chopper rectangles 920, 925. Includes peak 935. As can be seen, the complexity illustrated by peaks 930 and 925 is distinct from what can be considered to be the baseline level of complexity 940.

In some implementations, the point at which the output of the recurrent artificial neural network is about to be read coincides with occurrences of activity patterns of distinguishable complexity. For example, in the example environment of FIG. 9, the output of the recurrent artificial neural network may be read at peaks 930, 925, that is, during the window represented by the crest rectangles 915, 920, 925.

Identification of distinguishable levels of complexity within a recurrent artificial neural network is particularly beneficial when the input is a stream of data. Examples of data streams include, for example, video or audio data. Although the data streams have a starting part, it is generally desirable to process information in the data stream that does not have a pre-defined relationship with the beginning of the data stream. As an example, the neural network could perform object recognition, for example, recognizing cyclists in the vicinity of a car. Such neural networks must be able to recognize the cyclist regardless of when the cyclist appears within the video stream, i.e. without taking into account the time after the beginning of the video. Continuing with this example, when a data stream is input into an object recognition neural network, any patterns of activity in the neural network will generally display a low or inactive level of complexity. This low or inactive level of complexity is displayed independent of the continuous (or nearly continuous) input of the data stream to the neural network device. However, when an object of interest appears in the video stream, the complexity of the activity will become distinguishable and will indicate the point in time at which the object is recognized within the video stream. Thus, the timing of the discriminable level of complexity of the activity can also act as a yes/no output as to whether or not the data in the data stream meets certain criteria.

In some implementations, the timing of the output of the recurrent artificial neural network as well as the content of the neural network is given by activity patterns of distinguishable complexity. In particular, the identities and activities of nodes participating in an activity suitable for the activity patterns may be regarded as an output of the recurrent artificial neural network. The identified activity patterns thus exemplify the result of processing by the neural network and the timing when this decision is about to be read.

The content of the decision can be expressed in a variety of different aspects. For example, in some implementations and as described in further detail below, the content of the decision may be represented as a binary vector or a matrix of 1s and 0s. Each digit may indicate, for example, whether a pattern of activity exists or does not exist for a predefined group of nodes and/or a predefined duration. In such implementations, the content of the decision is represented in binary and is compatible with traditional digital data processing infrastructure.

10 is a flow diagram of a process 1000 for encoding signals using the network based on characterization of activity in a recurrent artificial neural network. Signals can be encoded in a variety of different environments, such as, for example, transmission, encryption, and data storage. Process 1000 may be performed by a system of one or more data processing devices that perform operations according to the logic of one or more sets of machine-readable instructions. For example, process 1000 may be performed by the same system of one or more computers running software for implementing the recurrent artificial neural network used in process 1000. In some instances, process 1000 may be performed by the same data processing apparatus that performs the process at 400. In some instances, process 1000 may be performed by an encoder in a signal transmission system or an encoder in a data storage system, for example.

The system performing process 1000 inputs the signal at 1005 into the circulating artificial neural network. In some cases, the signal input is a separate injection event. In other cases, the input signal is streamed to the recurrent artificial neural network.

The system performing process 1000 identifies at 1010 one or more decision moments within the recurrent artificial neural network. For example, the system can identify one or more decision moments by performing process 400 (FIG. 4).

The system performing process 1000 reads the output of the recurrent artificial neural network at 1015. As explained above, the content of the output of the recurrent artificial neural network is an activity in the neural network that matches the patterns used to identify the decision point(s).

In some implementations, at 1015, individual "reader nodes" are added to the neural network to identify occurrences of a particular pattern of activity in a particular collection of nodes, so to read the output of the recurrent artificial neural network. Can be. The reader nodes can fire if and only if the activity in a particular collection of nodes meets the timing (and possibly size) criterion. For example, in order to read the occurrence of the pattern 500 (Fig. 5) in nodes 104, 105, 106 (Fig. 2, Fig. 3), the reader node is the node 104, 105, 106 (or between the nodes) May be connected to the links 110 of the. The reader node will only itself become active if an activity pattern involving nodes 104, 105, 106 (or their links) occurs.

The use of such reader nodes would eliminate the need to entirely limit the windows of time for the recurrent artificial neural network. In particular, individual reader nodes may be connected to different nodes and/or number of nodes (or links between them). The individual reader nodes may be set with tailored responses (eg, different decay times in an integral-and-fire model) to identify different activity patterns.

The system performing process 1000 transmits or stores the output of the recurrent artificial neural network at 1020. The special action performed in 1020 may reflect the environment in which the process of reference number 1000 is being used. For example, in environments where secure or compressed communications are desired, the system performing process 1000 may transmit the output of the recurrent neural network to a receiver with access to the same or similar recurrent neural network. have. As another example, in environments where secure or compressed data storage is desired, the system performing process 1000 may direct the output of the recurrent neural network into one or more machine-readable data storage devices for later accesses. Can be recorded.

In some implementations, the complete output of the recurrent neural network is not transmitted or stored. For example, in implementations where the content of the output of the circulatory neural network is the activity in the neural network that matches patterns representing the complexity in the activity, only the activity that matches a relatively more complex or higher dimensional activity It can be transmitted or stored. As an example, referring to the patterns of reference number 500 (Fig. 5), in some implementations only activities matching the patterns of reference numbers 515, 520, 525, and 530 are transmitted or stored, whereas reference number 505, Activities matching the patterns of 510 are ignored or discarded. In this way, a lossy process enables the volume of data to be transmitted or stored to be reduced at the cost of the integrity of the information being encoded.

11 is a flow diagram of a process 1100 for decoding signals using the network based on characterization of activity in a recurrent artificial neural network. Signals can be decoded in a variety of different environments, such as, for example, receiving, decoding, and reading data from storage. Process 1100 may be performed by a system of one or more data processing devices that perform operations according to the logic of one or more sets of machine-readable instructions. For example, process 1100 may be performed by the same system of one or more computers running software to implement the recurrent artificial neural network used in process 1100. In some instances, process 1100 may be performed by the same data processing device performing process 400 and/or 1000. In some instances, process 1100 may be performed by, for example, a decoder in a signal receiving system or a decoder in a data storage system.

The system performing process 1100 receives at 1105 at least a portion of the output of the circulating artificial neural network. The particular action to be performed at 1105 may reflect the environment in which process 1100 is being used. For example, the system performing process 1100 may receive a transmitted signal comprising the output of the recurrent artificial neural network or may have a machine-readable data storage device that stores the output of the recurrent artificial neural network. Can be read.

The system performing process 1100 reconstructs the input of the recurrent artificial neural network from the output received at 1110. Rebuilding can proceed in a number of different ways. For example, in some implementations, a second artificial neural network (circulation or not) may be trained to reconstruct an input to the circulatory neural network from an output received at 1105.

As another example, in some implementations, a decoder trained using machine learning (including but not limited to deep learning) reconstructs the input to the recurrent neural network from the output received at 1105. Can be trained to do.

As another example, in some implementations, input to the same recurrent artificial neural network or to a similar recurrent artificial neural network is repeated until the output of that recurrent artificial neural network matches the output received at 1105 to some extent. can be changed.

In some implementations, process 1100 may include receiving user input, the user input defining a degree to which the input is reconstructed and, in response, 1110 adjusts the reconstruction accordingly. For example, the user input may stipulate that a complete rebuild is not required. In response, the system performing process 1100 coordinates the rebuild. For example, in implementations where the content of the output of the recurrent neural network is the activity in the neural network that matches patterns representing the complexity in the activity, characterizing the activity corresponding to a relatively more complex or higher dimensional activity Only the output will be used to reconstruct the input. As an example, referring to the patterns of reference number 500 (FIG. 5), in some implementations only activities matching the patterns of reference numbers 515, 520, 525, and 530 may be used to reconstruct the input. On the other hand, activities matching the patterns of reference numerals 506 and 510 may be ignored or discarded. In this way, a lossy rebuild can proceed in selected circumstances.

In some implementations, processes 1000, 1100 may be used for peer-to-peer encrypted communications. In particular, the same recurrent artificial neural network may be provided to both the transmitter (ie, the encoder) and the receiver (ie, the decoder). There are several ways in which the shared recurrent artificial neural network can be tailored to ensure that a third party cannot reverse engineer and decipher the signal, including:

-The structure of the circulatory artificial neural network,

-Functional settings of the recurrent artificial neural network, including node states and edge weights

-The size (or dimension) of the patterns, and

-Fragments of patterns within each dimension.

These parameters can be thought of as of multiple layers that together ensure transmission security. Also, in some implementations, the moment of determination time points may be used as keys to decrypt the signal.

Although the processes of reference numbers 1000 and 1100 are presented in terms of encoding and decoding a single recurrent artificial neural network, the processes of reference numbers 1000 and 1100 are also used in systems and processes that rely on multiple recurrent artificial neural networks. Can be applied. These recurrent artificial neural networks can operate in either parallel or series.

As an example of a serial operation, the output of the first recurrent artificial neural network can be used as an input of the second recurrent artificial neural network. The output resulting from the second recurrent artificial neural network is a twice encoded (or twice encrypted) version of the input to the first recurrent artificial neural network. Such a serial arrangement of recurrent artificial neural networks would allow, in environments where different parties have different levels of access to information, e.g., patient identity information would be using the remainder of the medical record and have access to the rest. May be useful in medical record systems that may not be accessible to the equipped party

As an example of parallel operation, the same information can be input to a number of different recurrent artificial neural networks. Different outputs of the neural networks can be used, for example, to ensure that the input can be reconstructed with high fidelity.

Although many implementations have been described, various modifications can be made. For example, although this generally means that the application must conform to a pattern indicating the ordering of activities in the recurrent artificial neural network, this is not necessarily the case. Rather, in some implementations, activity within a recurrent artificial neural network may fit a pattern, but not necessarily display activity that matches the pattern. For example, an increase in the likelihood that a circulatory neural network will display an activity matching a pattern can be treated as a non-random ordering of that activity.

As another example, in some implementations, different groups of patterns can be tailored for use in characterizing the activity in different recurrent artificial neural networks. The patterns can be tailored according to their validity, for example, in characterizing the activity of different recurrent artificial neural networks. Validity can be quantified, for example, based on the size of a vector or a table representing occurrence counts of different patterns.

As another example, in some implementations, the patterns that will be used to characterize activity in recurrent artificial neural networks may take into account the strength of the connection between nodes. In other words, the patterns previously described herein represent all signal transmission activity between two nodes in a binary manner, ie, either with or without the activity. This is not always the case. Rather, in some implementations, conforming to the pattern may require that the activity of a connection of any level or strength be taken as indicating an ordered complexity within the activity of the recurrent artificial neural network.

As another example, the content of the output of the recurrent artificial neural network may include activity patterns occurring outside time windows having a level of complexity in which an activity in the neural network can be distinguished. For example, the output of the recurrent artificial neural network that is read at 1015 and transmitted or stored at 1020 (Fig. 10) is, for example, the line rectangles 915, 920, and 925 in the graph of reference numeral 905 (Fig. 9). ) It may include information encoding activity patterns occurring outside. As an example, the output of the recurrent artificial neural network may be able to characterize only the highest order patterns of activity, irrespective of when the patterns of activity occur.

12, 13 and 14 are schematic examples of a binary representation or representation 1200 of topological structures such as patterns of activity in a neural network, for example. The topological structures shown in FIGS. 12, 13 and 14 all contain the same information, that is, an indication of the presence or absence of features in the graph. The features can be, for example, activity within a neural network device. In some implementations, the activity is identified based on or during a time interval that the activity in the neural network has a complexity distinguishable from other activities in response to the input.

As can be seen, the binary representation of reference numeral 1200 includes bits of reference numerals 1205, 1207, 1211, 1293, 1294, 1297 and an additional arbitrary number of bits (represented by an abbreviation "..."). For didactic purposes, the bits of references 1205, 1207, 1211, 1293, 1294, 1297 ... are shown as individual rectangular figures, either filled or unfilled to indicate the binary value within that bit. do. In the above schematic illustrations, the representation of reference numeral 1200 appears to be either a one-dimensional vector of bits (Figs. 12 and 13) or a two-dimensional matrix of bits (Fig. 14) on the surface. However, the representation of reference numeral 1200 is that the same information can be encoded regardless of the order of bits, that is, regardless of the position of individual bits in the collection, with a vector, a matrix, or another ordered collection of bits. It is different in that there is.

For example, in some implementations, each individual bit of reference numerals 1205, 1207, 1211, 1293, 1294, 1297... can represent the presence or absence of a topological feature, regardless of the location of the feature in the graph. have. As an example, referring to FIG. 2, a bit such as a bit of reference number 1207 indicates the presence of a topological feature suitable for a pattern of reference number 505 (FIG. 5), or between nodes of reference numbers 104, 105, and 106. It can be displayed irrespective of whether or not an activity occurs between nodes of numbers 105, 101, and 102. So, even though each individual bit of reference numerals 1205, 1207, 1211, 1293, 1294, 1297 ... can be associated with a particular characteristic, the position of that characteristic in the graph is, for example, reference numeral 1200. It need not be encoded by the corresponding position of the bit in the representation. In other words, in some implementations, the representation 1200 only provides for an isomorphic topological reconstruction of the graph.

Incidentally, in other implementations, it is possible that the position of individual bits of reference numerals 1205, 1207, 1211, 1293, 1294, 1297 ... actually encodes information such as, for example, the position of a feature in the graph. Do. In these implementations, the source graph can be reconstructed using a representation of 1200. However, such an encoding is not necessarily presented.

Considering the ability of the bit to express the presence or absence of a topological feature regardless of the location of the feature in the graph, the bit of reference number 1205 in FIG. 1 appears before the bit of reference number 1207, and the bit of reference number 1207 is At the beginning of the representation of reference numeral 1200, it appears before the bit of reference numeral 1211. In contrast, in Figs. 2 and 3, the order of the bits of reference numerals 1205, 1207, and 1211 in the representation of reference numeral 1200-and of the reference numerals 1205, 1207, and 1211 for other bits in the expression of reference numeral 1200 The positions of the bits have changed. Nevertheless, the binary representation 1200 remains the same, as is the case with the set of rules or algorithms that define the process for encoding the information in the binary representation 1200. As long as the correspondence between the bit and the feature is known, the position of the bits in the representation 1200 is irrelevant.

In further detail, each bit of reference numerals 1205, 1207, 1211, 1293, 1294, 1297... individually indicates the presence or absence of a feature in the graph. The graph is a set of nodes and a set of edges between the nodes. The nodes may correspond to objects. Examples of objects may include, for example, artificial neurons in a neural network, individuals in a social network, and the like. The edges may correspond to some relationship between objects. Examples of relationships include, for example, structural connections or activities along those connections. In the environment of neural networks, artificial neurons may be related by structural connections between neurons or by transmission of information along structural connections. In the context of a social network, individuals may be related by “friends” or other related connections, or by transmission of information along such connections (eg, postings). The edges may thus be characterized as relatively long-lived structural characteristics of a set of nodes or relatively transient activity characteristics occurring within a limited time frame. Further, the edges can be either directional or bidirectional. Directed edges indicate the directionality of the relationship between the objects. For example, information transmission from a first neuron to a second neuron may be represented by a directional edge indicating the direction of transmission. As another example, in a social network, a relational connection may indicate that a second user will receive information from a first user but that first user will not receive information from a second user. In topological terms, the graph can be represented as a set of unit intervals [0, 1], where 0 and 1 are identified as respective nodes connected by an edge.

The above features in which presence or absence is indicated by the bits of reference numerals 1205, 1207, 1211, 1293, 1294, 1297 are, for example, a node, a set of nodes, a set of nodes, a set of edges, a set of edges. It may be a set of sets, and/or additional hierarchically-more-complex features (eg, a set of sets of sets of nodes). The bits of reference numerals 1205, 1207, 1211, 1293, 1294, and 1297 generally represent presence or absence in different hierarchical levels. For example, a bit of reference numeral 1205 may represent the presence or absence of a node, while a bit of reference numeral 1205 may represent the presence or absence of a set of nodes.

In some implementations, bits 1205, 1207, 1211, 1293, 1294, and 1297 may represent features in a graph with a threshold level of some feature. For example, bits of reference numerals 1205, 1207, 1211, 1293, 1294, 1297 may indicate that not only that there is an activity in the set of edges, but that this behavior is weighted either above or below the threshold level. The weights may, for example, specify for a special purpose the training of the neural network device or may be an original characteristic of the edges.

5, 6 and 8 above show features in which presence or absence can be represented by bits of reference numerals 1205, 1207, 1211, 1293, 1294, 1297...

The directional simplexes in the collections of reference numerals 500, 600, 700 treat functional or structural graphs as a topological space with nodes as points. A structure or activity comprising one or more nodes and links suitable for simplexes in the collections of reference numerals 500, 600, 700 is expressed in any bit regardless of the identity of the particular nodes and/or links participating in the activity. Can be.

In some implementations, only some patterns of structure or activity are identified and/or some of the patterns of identified structure or activity are discarded or otherwise ignored. For example, referring to FIG. 5, a structure or activity suitable for a 5-point, 4-dimensional simplex pattern 515 is a 4-point, 3-dimensional and 3-point, 2-dimensional simplex pattern 510, 505. It originally includes structures or activities suitable for For example, points 0, 2, 3, 4 and points 1, 2, 3, 4 in the 4-dimensional simplex patterns 515 of FIG. 5 are both a 3-dimensional simplex pattern 510 Suitable for In some implementations, simplex patterns containing a smaller number of points-and thus a lower dimension-may be discarded or otherwise ignored.

As another example, only some patterns of structure or activity need to be identified. For example, in some implementations only patterns with odd points (3, 5, 7, ...) or even dimensions (2, 4, 6, ...) are used.

Returning to FIGS. 12, 13, and 14, features in which presence or absence is represented by bits of reference numerals 1205, 1207, 1211, 1293, 1294, 1297, ... may not be independent of each other. By way of explanation, if the bits of reference numbers 1205, 1207, 1211, 1293, 1294, 1297 each represent the presence or absence of zero-dimensional-simplexes reflecting the presence or activity of a single node, reference numbers 1205, 1207, The bits of 1211, 1293, 1294 and 1297 are independent of each other. However, if the bits of reference numbers 1205, 1207, 1211, 1293, 1294, and 1297 each represent the presence or absence of higher-dimensional simplexes that reflect the presence or activity of multiple nodes, the presence of each individual feature Alternatively, the information encoded by the absence may not be independent of the presence or absence of other features.

15 schematically shows an example of how the presence or absence of features corresponding to different bits are not independent of each other. In particular, a subgraph 1500 is illustrated including four nodes 1505, 1510, 1515, 1520 and six directional edges 1525, 1530, 1535, 1540, 1545, 1550. In particular, the edge of reference number 1525 goes from the node of reference number 1525 to the node of reference number 1510,

The edge of reference number 1530 goes from the node of reference number 1515 to the node of reference number 1505, the edge of reference number 1535 goes from the node of reference number 1520 to the node of reference number 1505, and the edge of reference number 1540 is of reference number 1520. From the node to the node of reference number 1510, the edge of reference number 1545 is from the node of reference number 1515 to the node of reference number 1510, and the edge of reference number 1550 from the node of reference number 1515 to the node of reference number 1520.

A single bit in representation 1200 (e.g., filled bit 1207 in FIGS. 12, 13, 14) may indicate the presence of a directional three-dimensional-simplex. For example, such a bit may indicate the presence of a three-dimensional-simplex formed by nodes 1505, 1510, 1515, 1520 and edges 1525, 1530, 1535, 1540, 1545, 1550. A second bit in representation 1200 (eg, filled bit 1293 in FIGS. 12, 13, 14) may indicate the presence of a directional two-dimensional-simplex. For example, such a bit may indicate the presence of a two-dimensional-simplex formed by nodes 1515, 1505, 1510 and edges 1525, 1530, 1545. In this simple example, the information encoded by the bits of reference numeral 1293 completely overlaps with the information encoded by the bits of reference numeral 1207.

Note that the information encoded by the bits of reference numeral 1293 may also overlap with the information encoded by other additional bits. For example, the information encoded by the bit 1293 would be redundant with the third and fourth bits indicating the presence of additional directional two-dimensional-simplexes. Examples of these simplexes include nodes of reference numerals 1515, 1520, 1510 and edges of reference numerals 1540, 1545, 1550 and nodes of reference numerals 1520, 1505, 1510 and edges of reference numerals 1525, 1535, 1540. Formed by

16 schematically shows another example of how the presence or absence of features corresponding to different bits are not independent of each other. In particular, a subgraph 1600 is shown comprising four nodes 1605, 1610, 1615, 1620 and five directional edges 1625, 1630, 1635, 1640, 1645. Nodes 1605, 1610, 1615, 1620 and edges 1625, 1630, 1635, 1640, 1645 are nodes 1505, 1510, 1515, 1520 and edges 1525, 1530, 1535, 1540 in the subgraph of (Fig. 15). Corresponds largely to 1545. However, when the nodes 1515 and 1520 are compared with the subgraph 1500 connected by the edge 1550, the nodes 1615 and 1620 are not connected by the edge.

A single bit in the representation of reference number 1200 (e.g., the unfilled bit of reference number 1205 in FIGS. 12, 13, 14) is, for example, a directional 3D-simple surrounding nodes 1605, 1610, 1615, 1620 It can indicate the absence of a directional 3D-simplex like Lex. The second bit in the representation of reference numeral 1200 (eg, the filled bits of reference numeral 1293 in FIGS. 12, 13, and 14) may indicate the presence of the 2D-simplex. The example directional 2D-simplex is formed by nodes 1615, 1605, 1610 and edges 1625, 1630, 1645. This combination of filled bit 1293 and unfilled bit 1205 provides information indicating the presence or absence of other features (and states of other bits) that may or may not be present in the representation of reference numeral 1200. do. In particular, the combination of the absence of a directional 3D-simple and the presence of a directional 2D-simple indicates from any of the following that at least one edge is absent:

a) possible directional 2D-simplex formed by nodes 1615, 1620, 1610 or

b) Possible directional 2D-simplex formed by nodes 1620, 1605, 1610.

Thus, the state of the bit representing the presence or absence of any one of these possible simplexes is not independent of the state of the bits 1205 and 1293.

Although these examples have been described in terms of different numbers of nodes and features with hierarchical relationships, this is not necessarily the case. For example, a representation 1200 comprising a collection of bits corresponding only to the presence or absence of a 3D-simplex is possible.

Using individual bits to represent the presence or absence of features in the graph results in certain properties. For example, encoding information is fault tolerant and provides "graceful degradation" of the encoded information. In particular, the loss of a particular bit (or group of bits) can increase uncertainty about the presence or absence of a feature. However, it would still be possible from other bits indicating the presence or absence of adjacent features to eliminate the possibility that a feature is present or absent.

Similarly, as the number of bits increases, so does the certainty about the presence or absence of a feature.

As another example, as described above, the ordering or arrangement of bits is independent of the isomorphic reconstruction of the graph represented by the bits. All that is needed is a known correspondence between the bits and particular nodes/structures in the graph.

In some implementations, patterns of activity in a neural network may be encoded within the representation of 1200 (FIGS. 12, 13 and 14 ). In general, patterns of activity within a neural network are the result of several properties of the neural network, such as, for example, structural connections between nodes of the neural network, weights between nodes and the total host of other possible parameters. to be. For example, in some implementations, the neural network may be trained prior to encoding patterns of activity in the representation of 1200.

However, regardless of whether the neural network is untrained or trained, for a given outcome, the response pattern of an activity can be thought of as a "representation" or "abstraction" of its inputs within that neural network. . So, although the expression of reference 1200 may appear to be a linearly-visible collection of (binary in some cases) digits, each of those digits encodes a relationship or correspondence between a particular input and related activity in the neural network. can do.

17, 18, 19, 20 are schematic illustrations using a representation of the occurrence of topological structures in activity in a neural network in four different classification systems 1700, 1800, 1900, 2000. Each of the classification systems 1700 and 1800 classifies expressions of patterns of activity in the neural network as part of the input classification. Each of the classification systems of reference numerals 1900 and 2000 classifies approximations of expressions of patterns of activity in a neural network as part of the input classification. In the human system of reference numerals 1700, 1800, patterns of activity to be represented occur at and are read from the source neural network device 1705, which is part of the classification system 1700, 1800. In contrast, in the classification systems 1900, 2000, patterns of activity that are approximated occur in the source neural network device that is not part of the classification system 1700, 1800. Nevertheless, an approximation of the representation of such patterns of activity is read from the approximator 1905, which is part of the classification systems of reference numbers 1900 and 2000.

In further detail, turning to FIG. 17, the classification system 1700 includes a source neural network 1705 and a linear classifier 1710. Source neural network 1705 is a neural network device configured to receive input and present representations of the occurrence of topological structures in activity within source neural network 1705. In the illustrated implementation, source neural network 1705 includes an input layer 1715 that receives inputs. However, this is not necessarily the case. For example, in some implementations, some or all of the input may be injected into different layers and/or edges or nodes throughout the source neural network 1705.

Source neural network 1705 can be any of a variety of different types of neural networks. In general, the source neural network 1705 is a circulatory neural network, such as a circulatory neural network modeled on a biological system. In some cases, the source neural network 1705 can model the degree of morphological, chemical, and other properties of the biological system. In general, the source neural network 1705 is implemented on one or more computing devices, such as a supercomputer, with a relatively high level of computational power. In such a case, the classification system 1700 will typically be a distributed system in which the remote classifier 1710 communicates with the source neural network 1705 via, for example, a data communication network.

In some implementations, the source neural network 1705 may be untrained and the expressed activity may be an intrinsic activity of the source neural network 1705. In other implementations, the source neural network 1705 can be trained and the represented activity can implement this training.

Expressions read from the source neural network 1705 may be expressions such as expressions of 1200 (Figs. 12, 13, and 14). The expressions can be read from the source neural network 1705 in many ways. For example, in the example shown above, source neural network 1705 includes “leader nodes”, which reader node reads patterns of activity among other nodes in source neural network 1705. In other implementations, activity in the source neural network 1705 is read by a data processing component programmed to monitor the source neural network 1705 for relatively high-order patterns of activity. In still other implementations, the source neural network 1705 may include an output layer, e.g., when the source neural network 1705 is implemented as a feed-forward neural network, the representation 1200 from the output layer is Can be read.

Linear classifier 1710 is a device that classifies an object-that is, representations of patterns of activity in the source neural network 1705-based on a linear combination of properties of that object. The linear classifier 1710 includes an input stage 1720 and an output stage 1725. Input 1720 is coupled to receive representations of patterns of activity in source neural network 1705. In other words, the expressions of the patterns of activity in the source neural network 1705 are feature vectors expressing the characteristics of the input to the source neural network 1705, and the source neural network 1705 is linear to classify the input. It is the source neural network used by classifier 1710. Linear classifier 1710 can receive representations of patterns of activity in source neural network 1705 in various ways. For example, the representations of patterns of activity may be received as separate events or as a continuous stream over a real-time or non-real-time communication channel.

The output stage 1725 is coupled to output the classification result from the linear classifier 1710. In the illustrated implementation, output stage 1725 is schematically shown as a parallel port with multiple channels. This is not necessarily the case. For example, the output stage 1725 can output the classification result through a serial port or a port with combined parallel and serial capabilities.

In some implementations, linear classifier 1710 can be implemented on one or more computing devices with relatively limited computational power. For example, the linear classifier 1710 may be implemented on a personal computer or a mobile computing device such as a smartphone or tablet.

In FIG. 18, the classification system 1800 includes a source neural network 1705 and a neural network classifier 1810. The neural network classifier 1810 is a neural network device that classifies an object-that is, representations of patterns of activity in the source neural network 1705-based on a non-linear combination of properties of that object. In the illustrated implementation, the neural network classifier 1810 is a feedforward network comprising an input layer 1820 and an output layer 1825. With linear classifier 1710, neural network classifier 1810 can receive representations of patterns of activity in source neural network 1705 in various ways. For example, representations of patterns of activity may be received as separate events or as a continuous stream over a real-time or non-real-time communication channel.

In some implementations, neural network classifier 1810 can perform inferences on one or more computing devices with relatively limited computational power. For example, neural network classifier 1810 may be implemented on a mobile computing device in a neural processing unit of a personal computer or mobile computing device such as a smartphone or tablet. Similar to classification system 1700, classification system 1800 will generally be a distributed system in which neural network classifier 1810 communicates with source neural network 1705, for example, via a data communication network.

In some implementations, the neural network classifier 1810 includes, for example, a convolutional neural network comprising convolutional layers, pooling layers, and fully-connected layers. It may be the same deep neural network. The convolutional layer may generate feature maps using, for example, linear convolution filters and/or nonlinear activation functions. The pooling layers reduce the number of parameters and control overfitting. Calculations performed by different layers in image classifier 1820 may be defined in different ways in different implementations of image classifier 1820.

In FIG. 19, the classification system 1900 includes a source approximator 1905 and a linear classifier 1710. As explained further below, the source approximator 1905 is a relatively simple neural network, which-either at the input layer 1915 or elsewhere-receives input and is a relatively more complex neural network. I output a vector that approximates the representation of the topological structures occurring in my activity patterns. For example, source approximator 1905 is a circulatory source neural network, such as, for example, a circulatory neural network that is modeled on a biological system and includes degrees of morphological, chemical, and other properties of the biological system. Can be trained to approximate In the illustrated implementation, the source approximator 1905 includes an input layer 1915 and an output layer 1920. The input layer 1915 is combinable to receive input data. The output layer 1920 is combined to output an approximation of the representation of the activity within the neural network device for reception by the input 1720 of the linear classifier. For example, the output layer 1920 may output an approximation 1200' of the expression of reference numeral 1200 (FIGS. 12, 13, and 14). As a side note, the expression 1200 schematically illustrated in FIGS. 17 and 18 and the approximation 1200' of the expression 1200 schematically illustrated in FIGS. 19 and 20 coincide. This is for convenience only. In general, the approximation of reference numeral 1200' will differ from the representation of reference numeral 1200 in at least some way. Despite these differences, the linear classifier 1710 is capable of classifying an approximation of 1200'.

In general, source approximator 1905 may perform inference on one or more computing devices with relatively limited computational power. For example, the source approximator 1905 may be implemented on a mobile computing device in a personal computer or a neural processing unit of a mobile computing device such as a smartphone or tablet. In general and compared to the classification systems 1700, 1800, the classification system of reference 1900 is usually combined in a single housing, for example in either of the same data processing devices or by a hardwire connection. It will be housed with a source approximator 1905 and a linear classifier 1710 implemented in data processing devices.

In FIG. 20, the classification system 2000 includes a source approximator 1905 and a neural network classifier 1810. The output layer 1920 of the source approximator 1905 is combined to output an approximation 1200' of a representation of the activity in the neural network device for reception by the input 1820 of the neural network classifier 1810. Despite the differences between the approximation 1200' and the representation 1200, the neural network classifier 180 can classify the approximation 1200'. In general and similar to the classification system 1900, the classification system of reference numeral 2000 is usually a data processing device combined in a single housing, for example in any of the same data processing devices or by a hardwire connection. It will be housed with a source approximator 1905 and a neural network classifier 1810 implemented on the field.

21 is a schematic illustration of an edge device 2100 comprising a local artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in the source neural network. In this environment, the local artificial neural network may be, for example, an artificial neural network running entirely on one or more local processors that do not require a communication network to exchange data. Typically, the local processors will be connected by hardwire connections. In some examples, the local processors may be housed in a single housing, such as a single personal computer or a single handheld mobile device. In some instances, the local processors may be under control and accessed by a single individual or a limited number of individuals. In fact, more complex source neurons to train a simpler and/or less highly trained but more exclusive second neural network (e.g., using supervised learning or reinforcement learning techniques). By using the representation of occurrences of topological structures in the network, individuals with limited computing resources and limited number of training samples can also train the neural network as desired. Storage requirements and computational complexity during training are reduced and resources such as battery life are saved.

In the implementations shown above, the edge device 2100 includes an optical imaging system 2110, an image processing electronics 2115, a source approximator 2120, a representation classifier 2125, and a communication controller and interface 2130. It is schematically shown as a security-camera device comprising:

Optical imaging system 2110 may include, for example, one or more lenses (or even pinholes) and a CCD device. Image processing electronics 2115 can read the output of optical imaging system 2110 and can generally perform basic image processing functions. Communication controller and interface 2130 is a device configured to control the flow of information to and from a device of reference numeral 2100. As described further below, the operations that the communication controller and interface 2130 may perform are to send images of interest to other devices and to receive training information from other devices. The communication controller and interface 2130 may thus include both a data transmitter and a receiver capable of communicating via, for example, data port 2135. The data port 2135 may be a wired port, a wireless port, an optical port, or the like.

Source approximator 2120 is a relatively simple neural network that is trained to output a vector that approximates a representation of topological structures that appear in patterns of activity in a relatively more complex neural network. For example, source approximator 2120 to approximate a circulating source neural network, such as, for example, a circulatory neural network that is modeled on a biological system and includes degrees of morphological, chemical, and other properties of the biological system. Can be trained.

The representation classifier 2135 is either a neural network classifier or a linear classifier coupled to receive an approximation of the representation of patterns of activity in the source neural network from the source approximator 2121 and output a classification result. Representation classifier 2125 may be, for example, a deep neural network, such as a convolutional neural network comprising convolutional layers, pooling layers, and fully-connected layers. The convolutional layer may generate feature maps using, for example, linear convolution filters and/or nonlinear activation functions. The pooling layers reduce the number of parameters and control overfitting. Calculations performed by different layers in the presentation classifier 2125 may be defined in different ways in different implementations of the presentation classifier 2125.

In some implementations, in operation, the optical imaging system 2110 can generate raw digital images. Image processing electronics 2115 can read raw images and will normally perform at least some basic image processing functions. Source approximator 2120 may receive images from image processing electronics 2115 and infer to output a vector that approximates the representation of topological structures occurring within patterns of activity in a relatively more complex neural network. Can perform operations. This approximation vector is input to a representation classifier 2125 which determines whether the approximation vector satisfies one or more sets of classification criteria. Examples include facial recognition and other machine vision operations. When the expression classifier 2125 determines that the approximation vector satisfies the set of classification criteria, the expression classifier 2125 may instruct the communication controller and interface 2130 to transmit information about the images. For example, the communication controller and interface 2130 may transmit the image itself, the classification, and/or other information regarding the images.

Sometimes, it may be desirable to change the classification process. In this case, the communication controller and interface 2130 may receive the training set. In some implementations, the training set may include representations of topological structures occurring in raw or processed image data and patterns of activity in a relatively more complex neural network. Such a training set may be used to retrain the source approximator 2120 using, for example, supervised learning or reinforcement learning techniques. Specifically, the expressions are used as a target answer vector and represent the desired result of the source approximator 2120 processing the raw or processed image data.

In other implementations, the training set may include representations of topological structures occurring in patterns of activity in a relatively more complex neural network and a desired classification of such representations of topological structures. Such a training set can be used to retrain the neural network representation classifier 2125, for example, using supervised learning or reinforcement learning techniques. In particular, the desired classification is used as a target answer vector and represents the desired result of a representation classifier 2125 that processes the representations of topological structures.

Regardless of whether the source approximator 2120 or the representation classifier 2125 is retrained, the inferencing operations at the device 2100 are large sets of training data and time-and computing power-the environment without intensive iterative training. It can be easily adapted to change the fields and objectives.

22 is a schematic illustration of a second edge device 2200 comprising a local artificial neural network that can be trained using representations of the occurrence of topological structures corresponding to activity in the source neural network. In the above illustrated implementation, the second edge device 2200 is schematically illustrated as a mobile computing device such as a smartphone or tablet. Device 2200 includes an optical imaging system (not shown), image processing electronics 2215, representation classifier 2225, communication controller and interface 2230 (e.g., on the back of device 2200). Include. These components may have characteristics and include optical imaging system 2110, image processing electronics 2115, representation classifier 2125, communication controller and interface 2130, and data port of device 2100 (FIG. 21). Can perform actions corresponding to the actions of (2135).

The illustrated implementations of device 2200 further include one or more additional sensors 2240 and a multi-input source approximator 2245. The sensor(s) 2240 may sense one or more characteristics of the environment surrounding the device 2200 or of the device 2200 itself. For example, in some implementations, sensor 2240 can be an accelerometer that senses the acceleration that device 2200 will undergo. As another example, in some implementations, sensor 2240 may be an audible sensor, such as a microphone, that senses noise around device 2200. Still other examples of sensor 2240 include chemical sensors (eg, “artificial nose” and the like), humidity sensors, radiation sensors, and the like. In some cases, sensor 2240 is coupled to processing electronics capable of reading the output of sensor 2240 (or other information such as, for example, a contact list or map) and performing basic processing functions. Different implementations of sensor 2240 may thus have different "modalities" in that the physical sensed physical parameter varies from sensor to sensor.

The multi-input source approximator 2245 is a relatively simple neural network that is trained to output a vector that approximates the representation of topological structures that appear in patterns of activity in a relatively more complex neural network. For example, the multi-input source approximator 2245 may be modeled on a biological system, for example, and circulatory, such as a circulatory neural network that includes degrees of morphological, chemical, and other properties of the biological system. It can be trained to approximate the source neural network.

Unlike source approximator 2120, multi-input source approximator 2245 is combined to receive raw or processed sensor data from multiple sensors, and based on that data, activity in a relatively more complex neural network. Returns an approximation of the representation of topological structures occurring in the patterns of. For example, the multi-input source approximator 2245 can be used from image processing electronics 2215 as well as audible, accelerated, chemical, or other data, for example, from one or more sensors 2240. Receive processed image data. The multi-input source approximator 2245 may be, for example, a deep neural network, such as a convolutional neural network comprising convolutional layers, pooling layers, and fully-connected layers. Calculations performed by different layers in the multi-input source approximator 2245 may be dedicated to a single type of sensor data or to multiple forms of sensor data.

Regardless of the particular organization of the multi-input source approximator 2245, the multi-input source approximator 2245 returns an approximation based on raw or processed sensor data from multiple sensors. For example, processed image data from image processing electronics 2215 and audible data from microphone sensor 2240 can be phased in patterns of activity in a relatively more complex neural network that received the same data. It can be used by the multi-input source approximator 2245 to approximate the representation of structures.

Occasionally, it may be desirable to change the classification process in device 2200. In this case, the communication controller and interface 2230 may receive the training set. In some implementations, the training set may include representations of topological structures occurring in raw or processed image data, chemical data or other data, and patterns of activity in a relatively more complex neural network. Such a training set can be used to retrain the multi-input source approximator 2245 using, for example, supervised learning or reinforcement learning techniques. In particular, the expressions are used as target answer vectors and represent the desired results of the multi-input source approximator 2245 processing the raw or processed image data or sensor data.

In other implementations, the training set may include representations of topological structures occurring in patterns of activity in a relatively more complex neural network and a desired classification of such representations of topological structures. Such a training set may be used to retrain the neural network representation classifier 2225 using, for example, supervised learning or reinforcement learning techniques. In particular, the desired classification is used as a target answer vector and represents the desired result of a representation classifier 2225 processing the representations of topological structures.

Regardless of whether the multi-input source approximator 2245 or the representation classifier 2225 is retrained, inferencing operations at device 2200 can be used with large sets of training data and time-and computing power-intensive iterative. It can be easily adapted to change environments and objectives without training.

23 is a schematic illustration of a system 2300 in which local artificial neural networks may be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network. Target neural networks are relatively simple and are implemented on less expensive data processing systems, while the source neural network is relatively complex and can be implemented on more expensive data processing systems.

The system 2300 includes various devices 2305 with local neural networks, a telephone base station 2310, a wireless access point 2315, a server system 2320, and one or more data communication networks 2325.

Local neural network devices 2305 are devices configured to process data using computationally-less-intensive target neural networks. As shown, local neural network devices 2305 are mobile computing devices, cameras, automobiles, or any one of a number of other appliances, facilities, and mobile components, and devices of different make and models in each category. Can be implemented. Different local neural network devices 2305 may belong to different owners. In some implementations, access to the data processing function of local neural network devices 2305 will generally be limited to those owners and/or those designated by that owner.

Each of the local neural network devices 2305 includes one or more source approximators that are trained to output a vector that approximates a representation of topological structures occurring within patterns of a relatively more complex neural network. For example, the relatively more complex neural network can be, for example, a circulating source neural network, such as a circulatory neural network that is modeled on a biological system and contains degrees of morphological, chemical, and other properties of the biological system. have.

In some implementations, in addition to processing data using source approximators, local neural network devices 2305 can target representations of topological structures occurring in patterns of activity within a relatively more complex neural network. It can also be programmed to re-train the source approximators using as vectors. For example, local neural network devices 2305 can be programmed to perform one or more iterative training techniques (e.g., gradient descent or stochastic gradient descent). have. In other implementations, source approximators in local neural network devices 2305 can interact with the local neural network devices 2305 to train source approximators, for example, or by a dedicated training system. Can be trained by a training system installed on a personal computer.

Each local neural network device 2305 includes one or more wireless or wired data communication components. In the illustrated implementation, each local neural network device 2305 includes at least one wireless data communication components, such as a mobile telephone transceiver, a wireless transceiver, or both. The mobile telephone transceivers can exchange data with the telephone base station 2310. The wireless transceivers may exchange data with the wireless access point 2315. Each local neural network device 2305 can also exchange data with peer mobile computing devices.

Telephone base station 2310 and wireless access point 2315 are connected for data communication with one or more data communication networks 2325 and can exchange information with server system 2320 through the network(s). So the local neural network devices 2305 may also be in data communication with the server system 2320 normally. However, this is not necessarily the case. For example, in implementations where local neural network devices 2305 are trained by other data processing devices, local neural network devices 2305 only need to communicate data with these other data processing devices at least once. There is only this.

Server system 2320 is a system of one or more data processing devices programmed to perform data processing activities according to one or more sets of machine-readable instructions. The activities may include serving training sets to training systems for mobile computing devices 2305. As described above, the training systems may be internal to mobile local neural network devices 2305 themselves, or may be on one or more other data processing devices. The training sets may include corresponding data and representations of the occurrence of topological structures corresponding to activity in the source neural network.

In some implementations, server system 2320 also includes a source neural network. However, this is not necessarily the case and server system 2320 may receive training sets from another system of the data processing device(s) implementing the source neural network.

In operation, after the server system 2320 receives the training set (from the source neural network found on the sub system 2320 itself or elsewhere), the server system 2320 is 2305) can be served with the training set to trainers. Source approximators in target local neural network devices 2305 may be trained using the training set, so that the target neural networks approximate the operations of the source neural network.

24, 25, 26, 27 are schematic illustrations of using a representation of the occurrence of topological structures in activity in a neural network in four different systems 2400, 2500, 2600, 2700. Systems 2400, 2500, 2600, 2700 may be configured to perform any of several different operations. For example, the systems 2400, 2500, 2600, 2700 may perform object localization operations, object detection operations, object segmentation operations, object detection operations, prediction operations, behavior selection operations, etc. I can.

Object localization actions determine the position of an object in the image. For example, a bounding box can be built around an object. In some cases, object localization can be combined with object recognition, where objects localized in that object recognition are labeled with appropriate indications.

Object detection operations classify image pixels as either belonging to a particular class (eg, belonging to an object of interest) or not. In general, object detection is performed by grouping pixels and forming bounding boxes around the pixel groups. The bounding box should fit tightly around the object.

Object segmentation usually assigns class labels to each image pixel. So, it is common that object segmentation, not a bounding box, proceeds on a per-pixel basis, and only requires a single label to be assigned to each pixel.

Predictive actions try to draw conclusions that are outside the range of the observed data. Although predictive actions may attempt to predict occurrences in the future (e.g., based on information about past and present states), predictive actions are based on incomplete information about those states. I also try to draw conclusions about it.

Action selection actions attempt to select an action based on a set of conditions. Behavior selection operations are traditionally decomposed into different approaches such as symbol-based systems (traditional planning), distributed solutions, and reactive or dynamic planning.

Each of the classification systems of reference numerals 2400 and 2500 performs a desired operation on expressions of patterns of activity in the neural network. Each of the systems of reference numerals 2600 and 2700 performs a desired operation on those approximating representations of patterns of activity in the neural network. In systems of reference numbers 2400 and 2500, the patterns of activity represented occur within and are read from the source neural network device 1705, which is part of the systems at reference numbers 2400 and 2500. In contrast, in systems of reference numerals 2400 and 2500, patterns of activity represented by approximations occur within the source neural network device that are not part of the systems of reference numerals 2400 and 2500. Nevertheless, an approximation of the representation of those patterns of activity is read from the approximator 1905, which is part of the systems of reference numbers 2400 and 2500.

In further detail, turning to FIG. 24, the system at reference numeral 2400 includes a source neural network 1705 and a linear processor 2410. The linear processor 2410 is a device that performs operations based on a linear combination of properties (or approximations of such representations) of representations of patterns of activity in a neural network. The operations may be, for example, object localization operations, object detection operations, object segmentation operations, object detection operations, prediction operations, action selection operations, and the like.

The linear processor 2410 includes an input stage 2420 and an output stage 2425. The input 2420 is coupled to receive representations of patterns of activity in the source neural network 1705. Linear processor 2410 can receive representations of patterns of activity in source neural network 1705 in various ways. For example, representations of patterns of activity may be received as separate events or as a continuous stream over a real-time or non-real-time communication channel. The output stage 2525 is coupled to output the classification result from the linear classifier 2410. In some implementations, the linear processor 2410 can be implemented on one or more computing devices with relatively limited computational power. For example, the linear processor 2410 may be implemented on a personal computer or on a mobile computing device such as a smartphone or tablet.

Turning to FIG. 24, the system at reference numeral 2400 includes a source neural network 1705 and a linear processor 2410. The linear processor 2410 is a device that performs operations based on a linear combination of properties (or approximations of such representations) of representations of patterns of activity in a neural network. The operations may be, for example, object localization operations, object detection operations, object segmentation operations, prediction operations, behavior selection operations, and the like.

The linear processor 2410 includes an input stage 2420 and an output stage 2425. The input 2420 is coupled to receive representations of patterns of activity in the source neural network 1705. Linear processor 2410 can receive representations of patterns of activity in source neural network 1705 in various ways. For example, representations of patterns of activity may be received as separate events or as a continuous stream over a real-time or non-real-time communication channel. The output stage 2525 is coupled to output the classification result from the linear classifier 2410. In some implementations, the linear processor 2410 can be implemented on one or more computing devices with relatively limited computational power. For example, the linear processor 2410 may be implemented on a personal computer or on a mobile computing device such as a smartphone or tablet.

In FIG. 25, the classification system 2500 includes a source neural network 1705 and a neural network 2510. Neural network 2510 is a neural network device configured to perform operations based on a linear combination of characteristics (or approximations of such expressions) of representations of patterns of activity within the neural network. The operations may be, for example, object localization operations, object detection operations, object segmentation operations, prediction operations, behavior selection operations, and the like. In the illustrated implementation, the neural network 2510 is a feedforward network comprising an input layer 2520 and an output layer 2525. Like the linear processor of reference numeral 2410, neural network 2510 can receive representations of patterns of activity in source neural network 1705 in various ways.

In some implementations, neural network 2510 can perform inferences on one or more computing devices with relatively limited computational power. For example, neural network 2510 may be implemented on a mobile computing device in a neural processing unit such as a personal computer or a mobile computing device such as a smartphone or tablet. Similar to the system at 2400, the system at 2500 will generally be a distributed system in which the remote neural network 2510 communicates with the source neural network 1705, for example via a data communication network. In some implementations, neural network 2500 can be a deep neural network, such as, for example, a convolutional neural network.

In Fig. 26, the system at 2600 includes a source approximator 1905 and a linear processor 2410. Despite the differences between the approximation 1200 ′ and the representation 1200, the processor 2410 may still perform operations on the approximation 1200 ′.

In FIG. 27, the system at 2700 includes a source approximator 1905 and a neural network 2510. Despite the differences between the approximation 1200 ′ and the representation 1200, the neural network 2510 can still perform operations on the approximation 1200 ′.

In some implementations, systems 2600 and 2700 may be implemented on an edge device, such as, for example, edge devices 2100 and 2200 (FIGS. 21 and 22 ). In some implementations, systems 2600, 2700 can be trained using representations of the occurrence of topological structures corresponding to activity in the source neural network, such as system 2300 (FIG. 23 ). It can be implemented as part of an existing system.

28 is a schematic illustration of a system comprising artificial neural networks that can be trained using representations of the occurrence of topological structures corresponding to activity in a source neural network. Reinforcement learning is a type of machine learning in which artificial neural networks learn from feedback about the results of actions taken in response to decisions of the artificial neural network. The reinforcement learning system moves from one state in the environment to another. The reinforcement learning system moves from one state in the environment to another, by performing actions and receiving information that characterizes that new state, and rewards and/or regrets that characterize the success (or no success) of that action. Reinforcement learning attempts to maximize the overall reward (or minimize regret) through the learning process.

In the illustrated implementation, the artificial neural network in the reinforcement learning system 2800 is a deep neural network 2805 (or other deep learning structure) using a reinforcement learning approach. In some implementations, deep neural network 2805 may be a local artificial neural network (such as neural network 2510 (FIGS. 25, 27 )), and, for example, a car, airplane, robot, or other device. It can be implemented locally on the. However, this is not necessarily the case and in other implementations the deep neural network 2805 may be implemented on a system of networked devices.

In addition to the source approximator 1905 and the deep neural network 2805, the reinforcement learning system 2800 includes an actuator 2810, one or more sensors 2815 and a teacher module 2820. In some implementations, the reinforcement learning system 2800 also includes one or more sources 2825 of additional data.

Actuator 2810 is a device that controls a mechanism or system that interacts with environment 2830. In some implementations, the actuator 2810 controls a physical mechanism or system (eg, steering of a vehicle or positioning of a robot). In other implementations, the actuator 2810 can control a virtual mechanism or system (eg, a virtual game board or investment portfolio). Thus, the environment at 2830 may also be physical or virtual.

Sensor(s) 2815 are devices that measure characteristics of the environment 2830. At least some of the measurements characterize the interactions between a controlled mechanism or system and other aspects of the environment 2830. As the actuator 2810 steers the vehicle, the sensor(s) 2815 measures the vehicle's proximity to one or more of the vehicle's speed, direction, and acceleration, the vehicle's proximity to other features, and the response of other features to the vehicle. can do. As another example, when actuator 2810 controls an investment portfolio, sensor(s) 2815 may measure the value and risk associated with the portfolio.

In general, both the source approximator 1905 and the teacher module 2820 are combined to receive at least some of the measurements made by the sensor(s) 2815. For example, the source approximator 1905 may receive measurement data from the input layer 1915 and output an approximation 1200' of the representation of topological structures occurring in patterns of activity in the source neural network. have.

Teacher module 2820 interprets measurements received from sensor(s) 2815 and provides reward and/or regret to deep neural network 2805. The reward is positive and indicates successful control over the mechanism or system. Regret is negative and indicates unsuccessful or less than optimal control. In general, teacher module 2820 also provides characterization of the measures along with the reward/regret for reinforcement learning. In general, the characterization of the measurements is an approximation (such as an approximation of 1200') of the representation of topological structures occurring within patterns of activity in the source neural network. For example, the teacher module 2820 may read the approximations 1200' output from the source approximator 1905 and pair the read approximations 1200' with corresponding reward/regret values.

In many implementations, reinforcement learning does not occur in the system at 2800 in real time or during active control of the actuator 2810 by the deep neural network 2805. Rather, training feedback is collected by the teacher module 2820 and can be used for reinforced training when the deep neural network 2805 is not actively commanding the actuator 2810. For example, in some implementations, the teacher module 2820 may be remote from the deep neural network 2805 and may be in intermittent data communication with the deep neural network 2805. Regardless of whether reinforcement learning is intermittent or continuous, the deep neural network 2805 can be developed to optimize rewards and/or reduce regret using the information received from the teacher module 2820.

In some implementations, system 2800 also includes one or more sources 2825 of additional data. Source approximator 1905 may also receive data from data sources 2825 at input layer 1915. In these instances, the approximation 1200 ′ will be the result from processing both sensor data and data from data sources 2825.

In some implementations, the data collected by one reinforcement learning system 2800 can be used for reinforcement learning or trayang of other systems, including other reinforcement learning systems. For example, characterizing the measurements along with the reward/regret values is a data exchange system that collects such data from various reinforcement learning systems and redistributes the data among the reinforcement learning systems, which is the teacher module. May be provided by (2820). Further, as described above, characterizing the measurements may be an approximation of the representation of topological structures occurring within patterns of activity in the source neural network, such as an approximation of 1200'.

The special operations performed by the reinforcement learning system 2800 will, of course, depend on that particular operating environment. For example, in environments where source approximator 1905, deep neural network 2805, actuator 2810, and sensors 2815 are part of a vehicle, deep neural network 2805 can adjust the vehicle Localization and/or detection operations may be performed.

In implementations where the data collected by the reinforcement learning system 2800 is used for training or reinforcement learning of other systems, reward/regret values and the state of the environment when object localization and/or detection operations are performed. Approximations 1200 ′ that characterize can be provided to the data exchange system. The data exchange system can then distribute its reward/regret values and approximations 1200' to other reinforcement learning systems 2800 associated with other vehicles, for reinforcement learning in those other vehicles. . For example, reinforced learning can be used to improve object localization and/or detection operations in a second vehicle using the compensation/regret values and approximations 1200'.

However, the operations learned in other vehicles need not match the operations performed by the deep neural network 2805. For example, compensation/regret values based on travel time and approximations 1200' resulting from input of sensor data characterizing an unexpected wet road at a location identified, for example by GPS data source 2825. Can be used for route planning in other vehicles.

Embodiments of the subject matters and operations described herein are digital electronic circuits, or computer software, firmware, hardware including structures disclosed herein and equivalents of the structures, or combinations of one or more of them. Can be implemented. The patented embodiments described herein include one or more computer programs encoded on a computer storage medium, i.e., one or more modules of computer program instructions for execution by a data processing device or for controlling the operation of a data processing device. Can be implemented as Alternatively or additionally, the program instructions may be artificially generated propagated signals, e.g., machine-generated, generated to encode information for transmission to a suitable receiver device for execution by the data processing device. It can be encoded on electrical, optical, or electromagnetic field signals. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or included therein. Moreover, if the computer storage medium is not a propagated signal, the computer storage medium may be a source or destination of computer program instructions encoded within an artificially generated propagated signal. The computer storage medium may also be or be contained within one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

The term “data processing apparatus” includes all types of apparatus, devices, and machines, and includes, for example, a programmable processor, computer, system on a chip, or many or combinations of the foregoing. The device may include a special purpose logic circuit, for example, a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In addition to hardware, the device may include code that creates an execution environment for the computer program in question, for example, processor firmware, protocol stack, database management system, operating system, cross-platform runtime environment, virtual machine, or one or more of them. Combinations may also be included. The device and execution environment can realize a variety of different computing model infrastructures such as web services, distributed computing, and grid computing infrastructure.

Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted language, declarative or procedural language, and It may be arranged as a program or in any shape, including modules, components, subroutines, objects, or other units suitable for use in a computing environment. Computer programs can respond to files in the file system, but they don't have to. A program may be in a portion of a file that holds another program or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or a number of consolidated files. Files (eg, one or more modules, subprograms, or files that store portions of code). The computer program may be arranged to run on one computer or on a number of computers located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit, for example a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and the device is also implemented as the special purpose logic circuit. Can be.

Processors suitable for executing a computer program include, by way of example, both general purpose microprocessors and special purpose microprocessors, and one or more processors of any type of digital computer. In general, the processor will receive instructions and data from read-only memory or random access memory or both. Essential elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices for storing data, e.g., a magnetic disk, a magnetic optical disk, an optical disk, or one or more mass storage devices for receiving and transferring data. Will be operatively combined with the field. However, the computer need not be equipped with such devices. Moreover, the computer can be used in other devices, for example, mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, global positioning system (GPS) receivers, or portable storage devices (e.g. For example, it can be embedded within a Universal Serial Bus (USB) flash drive). Devices suitable for storing computer program instructions and data include, for example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; Magnetic disks such as internal hard disks or removable disks; Magneto-optical disk; And all aspects of non-volatile memory, media and memory devices including CD ROM and DVD-ROM disks. The processor and memory may be supplemented by special purpose logic circuitry, or may be incorporated within the special purpose logic circuitry.

In order to provide an interaction with a user, the patented embodiments described herein can be implemented on a computer, which computer is a display device for displaying information to a user, e.g. CRT ( cathode ray tube) or liquid crystal display (LCD) monitor, and a keyboard and pointing device, for example a mouse or a trackball, that allow a user to provide input to the computer. Other types of devices can likewise be used to provide interaction with the user; For example, the feedback provided to the user may be sensory feedback, eg, visual feedback, auditory feedback, or any of the tactile feedback; In addition, the input from the user may be received in an arbitrary form including an audible, voice, or tactile input. In addition, for example, by sending web pages to the web browser in response to requests received from the web browser on the user's client device, the computer transmits documents to the device used by the user and receives documents from the device. You can interact with that user.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope or claims of the invention, but rather as descriptions of features specific to particular embodiments of the particular invention. do. Certain features described in this specification in the context of separate embodiments may also be implemented in combination within a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Moreover, although features are described above as acting in certain combinations and may even be declared so initially, one or more features from a declared combination may in some cases be deleted from the combination, and the declared combination It may be about subcombinations or mutations in subcombinations.

Similarly, although the operations are shown in a particular order in the figures, this should not be construed as that such operations are performed in the particular order or sequentially shown to achieve the desired results, or that all illustrated operations must be performed. In certain environments, multitasking and parallel processing can be advantageous. Moreover, separation of the various components in the embodiments described above should not be interpreted as requiring such separation in all embodiments, and the program components and systems described above are generally integrated together within a single software product or It should be understood that it can be packaged as software products.

So, special implementations of the subject of this patent have been described. Other implementations are within the scope of the claims that follow. In some cases, the actions described above in the claims may be performed in a different order and still achieve the desired results. In addition, the processes shown in the accompanying figures do not necessarily require that particular order or sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

Several embodiments have been described. Nevertheless, it will be understood that various modifications can be made. For example, although the expression of reference numeral 1200 is a binary representation, and each bit in the binary representation individually expresses the presence or absence of a feature in the graph, other representations of information are possible. For example, a matrix of non-binary digits, which is a vector or multi-valued, can be used, for example, to represent the presence or absence of features and possibly other properties of those features. An example of such a characteristic is the weight of the edges with activities that make up the characteristic.

Accordingly, other embodiments are within the scope of the claims that follow.

Claims (20)

A method comprising the step of outputting digits from a recurrent artificial neural network,
Each digit represents whether or not an activity within a particular group of nodes in the recurrent artificial neural network is suitable for a respective pattern of activity. The method of claim 1,
Further comprising the step of determining whether or not each particular group of nodes fits a respective pattern of activity by identifying clique patterns of activity of the recurrent artificial neural network. The method of claim 2,
The method further comprises defining a plurality of time windows,
During those time windows the activity of the recurrent artificial neural network responds to input to the recurrent artificial neural network,
Subpopulation patterns of activity are identified in each of the plurality of time windows, and
Each digit represents whether a particular group of nodes is suitable for respective subpopulation patterns of activity within a first of the windows or not. The method of claim 3,
The method further comprises identifying a first time window within the plurality of time windows,
Wherein the identifying is based on a distinguishable increased likelihood of subpopulation patterns identified during the first window. The method of claim 1,
Each pattern of activity surrounds cavities, how. The method of claim 2,
The method, wherein identifying subgroups comprises discarding or ignoring lower dimensional subgroups existing within higher dimensional subgroups. The method of claim 1,
Each of the digits is output from a reader node coupled to a collection of nodes in the recurrent artificial neural network,
Wherein the reader nodes indicate whether the activity of the nodes in the collection is suitable for a particular pattern of activity. The method of claim 1,
The method further comprises transmitting or storing only a portion of the read outputs, wherein transmitting or storing only a portion of the read outputs comprises:
Transmitting or storing outputs associated with patterns of activity with a relatively higher complexity; And
Discarding or ignoring outputs associated with patterns of activity of relatively lower complexity. The method of claim 1,
Structuring the artificial neural network, the structuring step:
Reading a digit output from the artificial neural network, and
Including the step of developing the structure of the artificial neural network,
The steps of developing the structure of the artificial neural network include:
Repeatedly changing the structure,
Characterizing the complexity of the patterns of activity within the altered structure, and
Using the characterization of the complexity of the pattern as an indication of whether the modified structure is desirable. The method of claim 1,
The digits are multi-valued, non-binary digits,
Wherein each of the values represents a weight assigned to edges in patterns of corresponding activity. An encoder or decoder comprising one or more computers operable to perform operations,
The operations include an operation of outputting digits from a recurrent artificial neural network,
Each digit represents whether or not an activity within a particular group of nodes in the recurrent artificial neural network is suitable for a respective pattern of activity. The method of claim 11,
The operations further include determining whether or not each particular group of nodes fits a respective pattern of activity by identifying subpopulation patterns of activity of the recurrent artificial neural network. The method of claim 12,
The operations further include defining a plurality of time windows,
During those time windows the activity of the recurrent artificial neural network responds to input to the recurrent artificial neural network,
Subpopulation patterns of activity are identified in each of the plurality of time windows, and
Each digit represents whether a particular group of nodes is suitable for respective subpopulation patterns of activity within a first of the windows or not. The method of claim 13,
The operations further include identifying a first time window within the plurality of time windows,
The identifying operation is based on a distinguishable increased likelihood of subpopulation patterns identified during the first window. The method of claim 11,
Each pattern of activity surrounds the cavities, an encoder or a decoder. The method of claim 11,
Each of the digits is output from a reader node coupled to a collection of nodes in the recurrent artificial neural network,
Reader nodes, encoders or decoders that indicate whether the activity of the nodes in the collection is suitable for a particular pattern of activity. The method of claim 11,
Further comprising the operation of transmitting or storing only a portion of the read outputs, the operation of transmitting or storing only a portion of the read outputs:
Transmitting or storing outputs associated with patterns of activity of relatively higher complexity; And
An encoder or decoder comprising discarding or ignoring outputs associated with patterns of activity of relatively lower complexity. The method of claim 11,
The above actions are:
Structuring the circulatory artificial neural network, the structuring operation:
Reading a binary output from the recurrent artificial neural network, and
And deploying the structure of the circulatory artificial neural network,
The operation of developing the structure of the circulating artificial neural network is:
The operation of repeatedly changing the structure,
Behavior that characterizes the complexity of patterns of activity within the altered structure, and
Using characterization of the complexity of the pattern as an indication of whether the modified structure is desirable. The method of claim 11,
The digits are multi-valued, non-binary digits,
Each of the values represents a weight assigned to edges in patterns of corresponding activity. The method of claim 11,
The encoder or decoder is a video encoder or decoder.

Images