CN112106069A

CN112106069A - Streaming data tensor analysis using blind source separation

Info

Publication number: CN112106069A
Application number: CN201980031659.4A
Authority: CN
Inventors: Y·奥维考
Original assignee: HRL Laboratories LLC
Current assignee: HRL Laboratories LLC
Priority date: 2018-06-13
Filing date: 2019-03-11
Publication date: 2020-12-18
Also published as: WO2019240856A1

Abstract

A system for controlling a device based on streaming data analysis using blind source separation is described. The system updates a set of parallel processing pipelines for streaming two-dimensional (2D) tensor slices of tensor data in different orientations, wherein the streaming tensor data includes incomplete sensor data. When updating the parallel processing pipeline, the system replaces the first tensor slice with the new tensor slice, resulting in an updated set of tensor slices in different orientations. At each time step, a loop of unmixing, transfer matching, and tensor factor weight calculations is performed on the updated set of tensor slices. Tensor factor weight computation is used for sensor data reconstruction, and based on sensor data reconstruction, hidden sensor data is extracted. Upon identifying an object in the extracted hidden sensor data, the device is caused to perform a maneuver to avoid collision with the object.

Description

Streaming data tensor analysis using blind source separation

Cross Reference to Related Applications

This application is a continuation-in-part application, U.S. application No.16/034,780, entitled "Independent Component Analysis of Compounds for Sensor Data Fusion and Reconstruction," filed in the U.S. at 13.7.2018, which is a non-provisional patent application, U.S. application No.62/558,094, filed in the U.S. at 13.9.2017, entitled "Independent Component Analysis of Compounds for Sensor Data Fusion and Reconstruction," which is incorporated herein by reference in its entirety.

This application is also a continuation-in-part application, U.S. application Ser. No.16/127,927, entitled "Mixed Domain Source Separation for Sensor Array Processing", filed in the United states at 11.9.2018, which is a non-provisional patent application, U.S. application Ser. No.62/624,054, filed in the United states at 30.1.2018, entitled "Mixed Domain Source Separation for Sensor Array Processing", which is incorporated herein by reference in its entirety.

This application is also a non-provisional application, U.S. provisional application No.62/684,364, entitled "Streaming Data Analysis Using blade Source Separation," filed in the United states at 13.6.2018, the entire contents of which are incorporated herein by reference.

Background

(1) Field of the invention

The present invention relates to a system for revealing hidden structures in data, and more particularly, to a system for revealing hidden structures in streaming data using tensor decomposition.

(2) Background of the invention

Tensor rank decomposition is the generalization of matrix singular value decomposition into tensors. A tensor is a generalization of a matrix to higher dimensions, in other words, a tensor is a multidimensional table of data values. The current state of the art for tensor decomposition is based on a method of least squares fitting data to a model. Examples include: a PARALLEL FACtor analysis (PARAFAC) described by Kiers et al in "PARAFAC 2-Part I.A Direct Fitting Algorithm for the PARAFAC2 Model", Journal of Chemometrics,13,275-294,1999, and an alternating least squares method (ALS) described by N.Sidipoulos et al in "software composition for Signal Processing and machine learning", IEEE Trans.on Signal Processing, Vol.65, No.13,2017. Each of the foregoing references is incorporated by reference as if fully set forth herein. These methods do not scale to higher dimensional tensors and do not handle sparse data well because gradients cannot be computed accurately when the data is sparse.

U.S. application No.16/034,780, which is incorporated by reference herein as if fully set forth herein, describes Independent Component Analysis (ICAT) of tensors, which is a processing method for tensors, both for sensor data fusion and reconstruction. A tensor is a generalization of a matrix to higher dimensions, in other words, a tensor is a multidimensional table of data values. ICAT is a unique method for decomposing the tensor into the sum of simpler component tensors formed from basis vectors (basis vectors) that reveal hidden patterns in the data.

Many big data analysis problems, such as malicious activity detection on a data network, require real-time analysis of streaming data. Therefore, there remains a need for a system that uses modifications of ICAT (which enable the system to process streaming data) to reveal hidden patterns in the data.

Disclosure of Invention

The present invention relates to a system for revealing hidden structures in data, and more particularly, to a system for revealing hidden structures in streaming data using tensor decomposition. The system includes one or more processors and a non-transitory computer-readable medium encoded with executable instructions such that, when executed, the one or more processors perform a plurality of operations. At each time step, the system updates a set of parallel processing pipelines for streaming two-dimensional (2D) tensor slices of tensor data in different orientations, wherein the streaming tensor data includes incomplete sensor data. In updating the set of parallel processing pipelines, replacing the first tensor slice with a new tensor slice, resulting in an updated set of tensor slices in different orientations. At each time step, the system performs a loop of unmixing, transfer matching, and tensor factor weight calculations on the updated set of tensor slices. The tensor factor weight computation is used for sensor data reconstruction, and hidden sensor data is extracted based on the sensor data reconstruction. Upon identifying an object in the extracted hidden sensor data, the system causes the device to perform a maneuver to avoid collision with the object.

In another aspect, the system processes the tensor slice into a unmixed output while performing a loop of unmixing, transfer matching, and tensor factor weight calculations; converting the unmixed output back to a tensor slice and decomposing the tensor slice into mode factors using matrix decomposition; repeating the operations of processing the tensor slices and converting the unmixed output until a mode factor is determined for all tensor modes; assigning a mode factor common to two or more unmixes to a tensor factor by matching the mode factor; determining a tensor factor weight value using the assigned mode factor; and combining the tensor factors using the tensor factor weight values for the sensor data reconstruction.

In another aspect, the tensor factor weight values are determined by building a linear system of equations using sensor data and solving the tensor weight factors.

In another aspect, the rate of unmixing is increased by multiplying the newly sampled stream transmission tensor data by the previous unmixing matrix.

In another aspect, while updating the set of parallel processing pipelines, the system measures a tensor data matrix at time t over a sliding time window; running a blind source separation algorithm on the tensor data matrix to obtain a solution mixing matrix; generating an estimate of a tensor pattern factor for time t by multiplying the tensor data matrix by the unmixing matrix; measuring a new tensor data matrix at time t + 1; initializing a current solution by multiplying the new tensor data matrix by the solution mixing matrix; running a blind source separation algorithm on the current solution to obtain a new solution mixing matrix; and generating a new estimate of the tensor modal factor for time t +1 by multiplying the new tensor data matrix by the new unmixing matrix.

In another aspect, the blind source separation algorithm is Independent Component Analysis (ICA).

Finally, the present invention also includes a computer program product and a computer-implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors such that, when the instructions are executed, the one or more processors perform the operations listed herein. Alternatively, a computer-implemented method includes acts that cause a computer to execute such instructions and perform the resulting operations.

Drawings

The objects, features and advantages of the present invention will become apparent from the following detailed description of the various aspects of the invention, when taken in conjunction with the following drawings, in which:

FIG. 1 is a block diagram depicting components of a system for revealing hidden structures in streaming data, in accordance with some embodiments of the present disclosure;

FIG. 2 is an illustrative diagram of a computer program product in accordance with some embodiments of the present disclosure;

figure 3 is an illustration of factoring a tensor canonical multivariate decomposition (CPD), according to some embodiments of the present disclosure;

figure 4 is an illustrative diagram of step 1 of the tensor Independent Component Analysis (ICAT) algorithm to extract tensor pattern factors in accordance with some embodiments of the present disclosure;

FIG. 5 is an illustrative diagram of step 2 of the ICAT algorithm, which step 2 addresses the ICA permutation ambiguity, assigns a pattern factor to the correct tensor factor and computes tensor factor weights, in accordance with some embodiments of the present disclosure;

fig. 6 is an illustration of online ICAT tensor slice sampling of streaming tensor data, in accordance with some embodiments of the present disclosure;

figure 7 is an illustration of tensor slice temporal sampling for streaming ICAT tensor decomposition, in accordance with some embodiments of the present disclosure;

FIG. 8 is a flow diagram illustrating a streaming data tensor analysis in accordance with some embodiments of the present disclosure; and

fig. 9 is a flow chart providing an illustration of controlling a device using hidden data extracted using ICAT according to some embodiments of the present disclosure.

Detailed Description

The present invention relates to a system for revealing hidden structures in data, and more particularly, to a system for revealing hidden structures in streaming data using tensor decomposition. The following description is presented to enable any person skilled in the art to make and use the invention and is incorporated in the context of a particular application. Various modifications and uses of the various applications will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to a wide variety of aspects. Thus, the present invention is not intended to be limited to the aspects shown, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without limitation to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.

The reader is directed to all papers and documents which are filed concurrently with this specification and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference. All the features disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

Furthermore, any element in the claims that does not explicitly recite "means" or "step" for performing a particular function is not to be construed as an "means" or "step" clause specified in 35 u.s.c. section 112, clause 6. In particular, the use of "step … …" or "action … …" in the claims herein is not intended to refer to the provisions of section 6, section 112, 35 u.s.c.

(1) Main aspects of the invention

Various embodiments of the present invention include three "primary" aspects. A first aspect is a system for revealing hidden structures in streaming data. The system typically takes the form of the operating software of a computer system or the form of a "hard-coded" instruction set. The system may be incorporated into a wide variety of devices that provide different functions. The second main aspect is a method, usually in the form of software, run using a data processing system (computer). A third broad aspect is a computer program product. The computer program product generally represents computer readable instructions stored on a non-transitory computer readable medium such as an optical storage device (e.g., a Compact Disc (CD) or a Digital Versatile Disc (DVD)) or a magnetic storage device (e.g., a floppy disk or a magnetic tape). Other non-limiting examples of computer readable media include hard disks, Read Only Memories (ROMs), and flash-type memories. These aspects will be described in more detail below.

FIG. 1 provides a block diagram depicting an example of the system of the present invention (i.e., computer system 100). The computer system 100 is configured to perform calculations, processes, operations, and/or functions associated with a program or algorithm. In one aspect, certain processes and steps discussed herein are implemented as a series of instructions (e.g., a software program) residing in a computer readable memory unit and executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform particular actions and exhibit particular behaviors, such as those described herein.

Computer system 100 may include an address/data bus 102 configured to communicate information. In addition, one or more data processing units, such as a processor 104 (or multiple processors), are coupled with the address/data bus 102. The processor 104 is configured to process information and instructions. In one aspect, the processor 104 is a microprocessor. Alternatively, the processor 104 may be a different type of processor, such as a parallel processor, an Application Specific Integrated Circuit (ASIC), a Programmable Logic Array (PLA), a Complex Programmable Logic Device (CPLD), or a Field Programmable Gate Array (FPGA).

Computer system 100 is configured to utilize one or more data storage units. The computer system 100 may include a volatile memory unit 106 (e.g., random access memory ("RAM"), static RAM, dynamic RAM, etc.) coupled to the address/data bus 102, wherein the volatile memory unit 106 is configured to store information and instructions for the processor 104. The computer system 100 may also include a non-volatile memory unit 108 (e.g., read only memory ("ROM"), programmable ROM ("PROM"), erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory, etc.) coupled to the address/data bus 102, wherein the non-volatile memory unit 108 is configured to store static information and instructions for the processor 104. Alternatively, the computer system 100 may execute instructions retrieved from an online data storage unit, such as in "cloud" computing. In one aspect, computer system 100 may also include one or more interfaces (such as interface 110) coupled to address/data bus 102. The one or more interfaces are configured to enable computer system 100 to interface with other electronic devices and computer systems. The communication interfaces implemented by the one or more interfaces may include wired communication techniques (e.g., serial cable, modem, network adapter, etc.) and/or wireless communication techniques (e.g., wireless modem, wireless network adapter, etc.).

In one aspect, computer system 100 may include an input device 112 coupled to address/data bus 102, wherein input device 112 is configured to communicate information and command selections to processor 100. According to one aspect, the input device 112 is an alphanumeric input device (such as a keyboard) that may include alphanumeric and/or function keys. Alternatively, the input device 112 may be an input device other than an alphanumeric input device. In one aspect, the computer system 100 may include a cursor control device 114 coupled with the address/data bus 102, wherein the cursor control device 114 is configured to communicate user input information and/or command selections to the processor 100. In one aspect, cursor control device 114 is implemented using a device such as a mouse, trackball, track pad, optical tracking device, or touch screen. Nonetheless, in one aspect, cursor control device 114 is directed and/or activated via input from input device 112, such as in response to using special keys and key sequence commands associated with input device 112. In an alternative aspect, cursor control device 114 is configured to be guided or directed by voice commands.

In one aspect, computer system 100 may also include one or more optional computer usable data storage devices (such as storage device 116) coupled to address/data bus 102. Storage device 116 is configured to store information and/or computer-executable instructions. In one aspect, storage device 116 is a storage device such as a magnetic or optical disk drive (e.g., hard disk drive ("HDD"), floppy disk, compact disk read only memory ("CD-ROM"), digital versatile disk ("DVD")). According to one aspect, a display device 118 is coupled with the address/data bus 102, wherein the display device 118 is configured to display video and/or graphics. In one aspect, the display device 118 may include a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a field emission display ("FED"), a plasma display, or any other display device suitable for displaying video and/or graphical images, as well as alphanumeric characters recognizable to a user.

The computer system 100 presented herein is an example computing environment in accordance with an aspect. However, non-limiting examples of computer system 100 are not strictly limited to a computer system. For example, one aspect provides that computer system 100 represents a type of data processing analysis that can be used in accordance with various aspects described herein. Other computing systems may also be implemented. Indeed, the spirit and scope of the present technology is not limited to any single data processing environment. Thus, in one aspect, one or more operations of various aspects of the present technology are controlled or implemented using computer-executable instructions, such as program modules, executed by a computer. In one implementation, such program modules include routines, programs, objects, components, and/or data structures that are configured to perform particular tasks or implement particular abstract data types. In addition, one aspect provides for implementing one or more aspects of the technology through the use of one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer storage media including memory-storage devices.

FIG. 2 depicts an illustrative diagram of a computer program product (i.e., a storage device) embodying the present invention. The computer program product is depicted as a floppy disk 200 or an optical disk 202 such as a CD or DVD. However, as previously mentioned, the computer program product generally represents computer readable instructions stored on any compatible non-transitory computer readable medium. The term "instructions" as used in relation to the present invention generally indicates a set of operations to be performed on a computer and may represent a fragment of an entire program or a single separable software module. Non-limiting examples of "instructions" include computer program code (source code or object code) and "hard-coded" electronic devices (i.e., computer operations encoded into a computer chip). "instructions" are stored on any non-transitory computer readable medium, such as on a floppy disk, CD-ROM, and flash drive or in the memory of a computer. In either case, the instructions are encoded on a non-transitory computer readable medium.

(2) Details of various embodiments

Tensor Independent Component Analysis (ICAT) is a processing method for tensors, both for sensor data fusion and reconstruction. Non-limiting examples of applications include pattern detection across multimodal datasets, such as a combination of sensors and social network link data to find groups of actors involved in malicious activity. A tensor is a generalization of a matrix to higher dimensions, in other words, a tensor is a multidimensional table of data values. ICAT, described in U.S. application Ser. No.16/034,780, is a unique method for decomposing the tensor into a sum of simpler component tensors formed from basis vectors that reveal hidden patterns in the data.

ICAT is very efficient in terms of processing speed and memory usage. The processing speed is faster than the existing method because there is no need to solve the alternating least squares fit problem over the entire tensor as in the current method. ICAT also has higher memory efficiency because the entire tensor does not need to be fit in memory. The memory requirements scale linearly with the tensor dimension, rather than exponentially, making it feasible to process tensors with more than three dimensions. These properties enable ICAT to perform efficient fusion and reconstruction of multiple sensor data streams if the dimensions of the tensor are used to represent different sensors.

The current state of the art of tensor decomposition includes methods based on least squares fitting of data to a model. These methods are much slower than ICAT, require more memory, and cannot be extended to higher dimensional tensors. These methods also do not handle sparse data well because gradients cannot be calculated accurately when the data is sparse. Finally, these methods are not intended to process streaming data. Initial experimental studies have shown that ICAT is accelerated more than fifty times (50X) faster than current methods.

Independent Component Analysis (ICA) has not previously been used for tensor resolution due to the uncertain order and scaling of ICA outputs. ICA blindly separates the signal mixture into pure components based on the statistical independence of the components, but since the order and scaling of the output components is freely variable, the usual way of using ICA prevents the combination of the correct ICA outputs into the individual tensor factors.

For tensor dimensions, the ICAT algorithm uses a unique sequence of ICA solution mixing stages, where each pair of stages has a common dimension. This enables linking or matching of the ICA components to the correct combination of individual tensor factors. This linking eliminates the need to try every possible combination of ICA outputs to find the correct combination. Once the correct combination has been determined, a simple system of linear equations can be solved to determine the correct scaling of the individual tensor factors.

An important requirement for sensing systems involving multiple sensors is the ability to extract weak signals from a mixture with other environmental signals and to fill in lost data. The system described herein meets this requirement using a sensor fusion framework that models the relationship between sensor signals and different contexts based on using tensor representations. The system analyzes the tensor using ICAT. The dimensions (or patterns) of the tensor are used to represent both sensor data and context conditions, such as time of day, geographic location, signals from other sensors, and the like. The tensor element values represent the relationship between the signal and the context. Tensor decomposition can then reveal hidden structures in the relationships, which in turn can be used to extract weak signals and predict or fill in lost sensor data. In a system according to an embodiment of the present disclosure, ICAT is extended to handle streaming data.

Tensor decomposition has been successfully used in many applications involving multidimensional data, such as for movie recommendation systems (e.g., Netflix challenge), other sensor fusion applications, chemometrics and social network activity analysis, and so forth. Existing tensor resolution methods operate by fitting a multi-linear model to the measured data using a mean square error fit metric and some form of gradient descent, such as Nonlinear Least Squares (NLS). The ICAT method is the first method to decompose the tensor using completely different metrics based on maximizing the statistical independence of the tensor pattern factors. Using ICA enables ICAT to extract weak signals in the interference because, unlike least squares error measurements, statistical independence measurements are not sensitive to the relative amplitude of the signal components. ICAT is also much faster than the prior art because the small relative effect of weak signals on the gradient slows down the prior art method. The fact that ICAT is non-iterative also reduces computation time. ICAT has greatly reduced memory requirements because only the measured portion of the tensor needs to be represented during computation rather than the full tensor. This is because ICAT calculates the tensor pattern factor vector directly from the measured data without loading the full tensor into memory prior to operating on it as in prior methods.

ICAT is based on a standard canonical multivariate decomposition (CPD) form of the tensor decomposition shown in fig. 3. In particular, fig. 3 illustrates factoring the tensor canonical multivariate decomposition (CPD), revealing the tensor structure used by ICAP for denoising, data completion, and signal extraction. Similar to the Singular Value Decomposition (SVD) of the matrix, CPD decomposes the tensor 300 into a weighted sum of R tensor factors 302, 304 and 306, each of which is given by the outer product of D tensor mode factors or vectors, where D is the tensor order or dimension and R is the rank of the tensor. Smaller R represents more structure in the data because for a D-order tensor with N elements per mode, it is compared to N^DOne parameter, CPD representation has only RDN parameters. The decomposition is guaranteed to be unique if some less strict (mil) condition is met with respect to the tensor, which is not the case with matrix decomposition.

Unlike prior approaches, ICAT uses statistical independence to decompose tensors into CPD representations. ICAT includes FIGS. 4 and 4The two main steps shown in fig. 5, wherein a tensor with three modes or dimensions is used for simplicity of illustration. Fig. 4 depicts step 1 of the ICAT algorithm, which step 1 extracts the tensor mode factor. In this example, for a third order tensor (element 300), a three-dimensional (3D) slab (slab) of the tensor (a slice with thickness greater than 1) is sampled and integrated along one mode to form a two-dimensional (2D) tensor slice. In the reshaping and marginalizing block (element 406), the 2D slices are vectorized or converted to a one-dimensional (1D) signal (element 400) by concatenation and used as input to the ICA (element 402) to be unmixed into a tensor mode factor. The R unmixed outputs (element 404) of the ICA are reformatted or reshaped back into 2D slices by dividing the 1D signal into rows and stacking the rows, summed (integrated) along the k-dimension (marginalize), and normalized to determine b_n(j) A mode factor. The 2D slices are then marginalized and normalized along the j dimension to determine c_n(k) A mode factor. The same process is repeated using mixed slices orthogonal to the first set to extract a_n(i) And c_n(k) A mode factor. Although 3D slabs and tensors are used as examples herein, the method can be used for slabs and tensors of any dimension. The slab may also be a sparse subsample of the tensor, as long as at least R slices are measured for each tensor mode and each slice has enough data samples for ICA to converge.

To reiterate, in a first step, the 2D horizontal slices in the second and third modes of the tensor 300 are converted or reshaped into a 1D vector 400 and used as a signal mixture input to the ICA 402. Each of the R unmixed outputs 404 of ICA is then converted or reshaped back to a 2D slice format. The nth output of ICA is then a matrix of rank 1, which is the outer product of the factors of tensor mode n. Referring to element 406, "reshaping" is the conversion of a 2D matrix to a 1D vector (or vice versa). "marginalizing" is the integration of a 2D slice or matrix along one dimension to form a 1D vector. Equation 410 represents reshaping the ICA output into a matrix, and equation 412 represents estimating the mode factor b by marginalizing the matrix along k_r(j) And equation 414 represents estimating the mode factor c by marginalizing the matrix along j_r(k) In that respect By integrating (marginalizing) the rank-1 matrix along each dimension and normalizing the resulting vector by its maximum value, the matrix can be divided into factors 408 (b) for the second and third modes_nAnd c_n). These two mode factors 408 are automatically correctly assigned to tensor factors because they are both part of the same unmixing operation.

Next, a factor (a) of the first mode needs to be determined_n). The first and third mode factors may be determined in the same manner as before but by using a vertical slice of the tensor as the mixture input of the ICA instead of the horizontal slice. However, since the order of ICA output is uncertain and the factor of the first mode is determined in a separate unmixing operation, the first mode factor a of unmixing is still required_nThe correct tensor factor is assigned. A naive approach is to search for a that minimizes the reconstruction error_nThe best order of the mode factors, but this will involve R! This is a rapid surge and does not scale well with increasing rank R. Element 416 results in the transfer pattern matching and tensor factor weight computation step (fig. 5).

ICAT uses the transfer matching method described in U.S. application No.16/034,780 to convert a_nThe mode factor is assigned to the correct tensor factor. Fig. 5 depicts step 2 of the ICAT algorithm, which step 2 solves the ICA permutation ambiguity, assigns the mode factor to the correct tensor factor and computes the tensor factor weight λ r as shown in fig. 5. The solution is to use c common to both unmixing operations_nMode factor to find the correct a_nAnd (4) mode allocation. The algorithm searches for c between the two

unmixes

502 and 504 for each of the R tensor mode factors_nBest match for pattern factor 500. Then, c matched with_nAssociated a_n506 to c with a match_nAnd their association b_nIs measured by the tensor factor. This only requires RD (D-1) times instead of R! The sub-vector matching operation, where R and D are the rank and order of the tensor, respectively, greatly reduces the computational complexity for high rank tensors. Once the pattern factors are correctly assigned, the pattern factors can beThe tensor weight factor λ r is calculated by building a system of linear equations using the CPD representation and a subset of the measured tensor values. The linear equation can then be solved for λ r using standard methods (such as the matrix pseudo-inverse), as described in U.S. application No.16/034,780, which is incorporated herein by reference as if fully set forth herein. The subset of tensor values may be randomly selected or selected using a value weighted sampling method.

Many important applications of tensor resolution require continuous online updating of solutions (solutions) of streaming data, such as traffic on a computer network. The ICAT algorithm, as described above and in U.S. application Ser. No.16/034,780, is designed to analyze the static tensor. To efficiently process streaming data, the system described herein is a pipelined version of ICAT for streaming tensor data, where the mixed slices are stored in the pipeline and shifted in time, as shown in fig. 6, which depicts online ICAT tensor slice sampling of the streaming tensor data. A data pipeline is a set of data processing elements connected in series, where the output of one element is the input of the next element. The elements of the pipeline are typically executed in parallel or in a time sliced manner.

A set of parallel pipelines 600 for 2D slices 602 of the tensor in different orientations are updated at respective time steps. At each time step, the first slice is replaced with a new sample of data, depending on the orientation of the slice. Slices that have been in the pipeline are shifted by one increment and the last slice in the pipeline is disengaged and replaced by the next oldest slice. Fig. 7 illustrates this data shifting, using a third order tensor as an example. FIG. 7 depicts a newly sampled slice 700, an existing slice 702 shifted down the pipeline, and an oldest sliced out 704. At various time increments, the ICAT 604 (fig. 6) uses the updated set of tensor slices in different orientations to update the tensor pattern factor and the streaming tensor data decomposition 606 (fig. 6) at various time steps using the processes in fig. 4 and 5. Streaming ICAT 604 processes streaming data using a continuous loop of ICA solution mixing, transfer matching, and pattern factor weight calculations as tensor samples move through the pipeline.

If the ICA algorithm being used is one that iteratively converges on the solution (such as one that uses gradient descent), the update rate of the ICA solution mix of the factors may be increased in the current ICA cycle by multiplying the most recently sampled data by the previous solution mix matrix. This preprocessing will partially unmix the data, which will reduce the unmixing time in the current loop, since the previous solution is a good initialization close to the current solution. Initializing with an approximately correct solution will not help the execution time of non-iterative ICA methods (such as JADE) that do not have an initialization step and always compute the solution "from scratch". This iterative ICA can be used in streaming mode ICAT using the following steps, which are also shown in the flow chart of fig. 8:

1. at time t, a tensor data matrix Y is formed from the sampled slices of the data tensor D^(t)(element 800).

2. For Y^(t)Operating ICA to generate a solution mixing matrix B^(t)(element 802).

3. Will Y^(t)Multiplying by B^(t)To generate an ICA output matrix G^(t)To G^(t)Reshaping and marginalizing, performing transfer mode matching, and calculating a tensor factor weight λ r to create a CPD representation of D for time t (element 804). Methods for calculating tensor factor weights are described in detail in U.S. application No.16/034,780. Using tensor factor weight calculations for sensor data reconstruction is equivalent to calculating the CPD tensor representation in fig. 3, where streaming tensor data may be composed of streaming sensor data.

4. At time t +1, a new data matrix Y formed by slices of tensor D is measured^(t+1)(element 806).

5. Will Y^(t+1)Multiplying by the previous unmixing matrix B^(t)To initialize the probe for Y^(t+1)The current ICA solution (element 808).

6. For Y^(t+1)Operating ICA to generate a new solution mixing matrix B^(t+1)(element 810).

7. Will Y^(t+1)Multiplying by B^(t+1)To generate an ICA output matrix G^(t+1)To G^(t+1)Reshaping and marginalizing, performing transfer mode matching, and calculating a tensor factor weight λ r to create a CPD representation of D for time t +1 (element 812).

8. T +1 is set (element 814).

9. Go to step 4 and repeat for the next sliding window of data (element 816).

Note that since ICA is insensitive to the order of data in Y's rows, so long as the order is consistent across the rows, the position of data elements in Y's rows may be changed even though the position of the data elements has changed due to the streaming of data^(t+1)Upper use of B^(t)。

ICAT can be applied to any sensing application involving the fusion of multiple sensor data streams. For example, it is desirable to use it for fusion of multiple sensors used in vehicles, including denoising of data, extraction of useful features, and reconstruction of missing data. For example, the extracted missing or hidden data may be the detection and identification of objects such as vehicles, pedestrians, and traffic signs under different weather conditions (e.g., rain, snow, fog) and lighting conditions (e.g., low light, high light). The extracted hidden data may then be used to cause automatic operations related to controlling components of the autonomously driven vehicle. Another application is the analysis of traffic on a computer network to detect anomalies and computer attacks.

Fig. 9 is a flow chart providing an illustration of controlling an apparatus 900 using hidden data extracted using ICAT according to some embodiments of the present disclosure. Non-limiting examples of devices 900 that may be controlled via processor 104 include a motor vehicle or a motor vehicle component (electrical, non-electrical, mechanical), such as a brake, steering mechanism, suspension, or safety device (e.g., airbag, seat belt tensioner, etc.). Further, the vehicle may be an Unmanned Aerial Vehicle (UAV), an autonomous ground vehicle, or a manually operated vehicle controlled by a pilot or a remote operator. For example, upon object detection (e.g., based on hidden data) and recognition, the system may cause the autonomously driven vehicle to perform a driving operation/maneuver (such as a steering or another command) in accordance with the driving parameters in accordance with the recognized object. For example, if the system identifies a cyclist, another vehicle or a pedestrian, the system described herein may steer/operate the vehicle to avoid collision with the cyclist or the vehicle (or any other object that should be avoided while traveling). The system may cause the autonomous driving vehicle to apply a functional movement response (such as a braking operation followed by a steering operation) to redirect the vehicle away from the object to avoid a collision.

Other suitable responses may include one or more of the following: steering, throttle operation to accelerate or decelerate, or to decide to keep heading and speed constant. The response may be adapted to avoid collisions, increase driving speed, or increase efficiency. Other device types may also be controlled, as will be appreciated by those skilled in the art. Thus, given the particular object detected and the environment in which the system is implemented, there are many automatic actions that can be initiated by an autonomous driving vehicle. For example, the method may be applied to border safety (e.g., detection of smugglers at night); intelligence, surveillance and reconnaissance (ISR); an unmanned aerial vehicle; autonomous driving vehicles, and perception and safety in autonomous systems (e.g., detecting human-robot interaction in a manufacturing environment).

Another application of the invention described herein is the fusion of multiple body-mounted sensors for human activities and conditions. Non-limiting examples of sensors include blood pressure sensors, pulse sensors, Electromyography (EMG) sensors, temperature sensors, electroencephalogram (EEG) sensors, accelerometers, gyroscopes, pedometers, and pressure sensors. For example, activity detected from the extracted hidden data (such as a decrease in walking speed) in combination with a biometric measurement (such as a heart rate at certain times of the day) may be used to infer the health condition of the human. For example, a reduction in movement combined with detected hypertension may indicate a potential stroke. Based on the detection, a text message, email, or audible alert may be sent to the human via a smart watch, smart phone, or other mobile device. For example, the message/alarm may provide instructions to the human to go to the hospital or to rest (e.g., sit, lie down). In this example, the device 900 controlled by the processor 104 obtaining the extracted hidden data according to an embodiment of the present disclosure is a mobile device (smart watch, smart phone, mobile phone) or a personal computer comprising a display, and the display presents instructions for at least one action for the user to perform.

Finally, while the invention has been described in terms of various embodiments, those skilled in the art will readily recognize that the invention can have other applications in other environments. It should be noted that many embodiments and implementations are possible. Furthermore, the following claims are in no way intended to limit the scope of the present invention to the specific embodiments described above. Additionally, any term "means for … …" is intended to induce an element and a device-plus-function interpretation of the claims, and any element not specifically used with the term "means for … …" should not be interpreted as a device-plus-function element, even if the claims otherwise include the term "means. Further, although specific method steps have been set forth in a particular order, these method steps may be performed in any desired order and are within the scope of the present invention.

Claims

1. A system for controlling a device based on streaming data analysis using blind source separation, the system comprising:

one or more processors and a non-transitory computer-readable medium encoded with executable instructions such that, when executed, the one or more processors:

updating, at each time step, a set of parallel processing pipelines for two-dimensional (2D) tensor slices of streaming tensor data in different orientations, wherein the streaming tensor data includes incomplete sensor data,

wherein, when updating the set of parallel processing pipelines, replacing the first tensor slice with a new tensor slice, thereby obtaining an updated set of tensor slices at different orientations;

performing a loop of unmixing, transfer matching, and tensor factor weight calculations for the updated set of tensor slices at each time step;

using the tensor factor weight calculation for sensor data reconstruction;

extracting hidden sensor data based on the sensor data reconstruction; and

upon identifying an object in the extracted hidden sensor data, causing the device to perform a maneuver to avoid collision with the object.

2. The system of claim 1, wherein in performing the loop of unmixing, transfer matching, and tensor factor weight calculations, the one or more processors perform the following:

processing the tensor slice into a unmixed output;

converting the unmixed output back to a tensor slice and decomposing the tensor slice into mode factors using matrix decomposition;

repeating the operations of processing the tensor slice and converting the unmixed output until a mode factor is determined for all tensor modes;

assigning a mode factor common to two or more unmixes to a tensor factor by matching the mode factor;

determining a tensor factor weight value using the assigned mode factor;

combining the tensor factors using the tensor factor weight values for the sensor data reconstruction.

3. The system of claim 2, wherein the tensor factor weight value is determined by building a linear system of equations using sensor data and solving the tensor weight factor.

4. The system of claim 1, wherein a rate of unmixing is increased by multiplying newly sampled streaming tensor data by a previous unmixing matrix.

5. The system of claim 1, wherein, in updating the set of parallel processing pipelines, the one or more processors further perform the following:

measuring a tensor data matrix at time t over a sliding time window;

running a blind source separation algorithm on the tensor data matrix to obtain a solution mixing matrix;

generating an estimate of a tensor pattern factor for time t by multiplying the tensor data matrix by the unmixing matrix;

measuring a new tensor data matrix at time t + 1;

initializing a current solution by multiplying the new tensor data matrix by the solution mixing matrix;

running a blind source separation algorithm on the current solution to obtain a new solution mixing matrix; and

generating a new estimate of the tensor modal factor for time t +1 by multiplying the new tensor data matrix by the new unmixing matrix.

6. The system according to claim 5, wherein the blind source separation algorithm is Independent Component Analysis (ICA).

7. A computer-implemented method of controlling a device based on streaming data analysis using blind source separation, the method comprising acts of:

causing one or more processors to execute instructions encoded on a non-transitory computer-readable medium such that when the instructions are executed the one or more processors perform the following:

using the tensor factor weight calculation for sensor data reconstruction;

extracting hidden sensor data based on the sensor data reconstruction; and

8. The method of claim 7, wherein in performing the loop of unmixing, transfer matching, and tensor factor weight calculations, the one or more processors perform the following:

processing the tensor slice into a unmixed output;

repeating the operations of processing the tensor slices and converting the unmixed output until a mode factor is determined for all tensor modes;

determining a tensor factor weight value using the assigned mode factor;

9. The method of claim 8, wherein the tensor factor weight value is determined by building a linear system of equations using sensor data and solving the tensor weight factor.

10. The method of claim 7, wherein a rate of unmixing is increased by multiplying the newly sampled streaming tensor data by a previous unmixing matrix.

11. The method of claim 7, wherein, in updating the set of parallel processing pipelines, the one or more processors further perform the following:

measuring a tensor data matrix at time t over a sliding time window;

measuring a new tensor data matrix at time t + 1;

12. The method of claim 11, wherein the blind source separation algorithm is Independent Component Analysis (ICA).

13. A computer program product for controlling an apparatus based on streaming data analysis using blind source separation, the computer program product comprising:

computer-readable instructions stored on a non-transitory computer-readable medium, the computer-readable instructions executable by a computer having one or more processors to cause the processors to:

using the tensor factor weight calculation for sensor data reconstruction;

extracting hidden sensor data based on the sensor data reconstruction; and

14. The computer program product of claim 13, wherein in performing the loop of unmixing, transfer matching, and tensor factor weight calculations, the one or more processors perform the following:

processing the tensor slice into a unmixed output;

determining a tensor factor weight value using the assigned mode factor;

15. The computer program product of claim 14, wherein the tensor factor weight value is determined by building a linear system of equations using sensor data and solving the tensor weight factor.

16. The computer program product of claim 13, wherein a rate of unmixing is increased by multiplying newly sampled streaming tensor data by a previous unmixing matrix.

17. The computer program product of claim 13, wherein, in updating the set of parallel processing pipelines, the one or more processors further perform the following:

measuring a tensor data matrix at time t over a sliding time window;

measuring a new tensor data matrix at time t + 1;

18. The computer program product of claim 17, wherein the blind source separation algorithm is Independent Component Analysis (ICA).