CN112740234B - Neuromorphic system for authorized user detection - Google Patents
Neuromorphic system for authorized user detection Download PDFInfo
- Publication number
- CN112740234B CN112740234B CN201980031921.5A CN201980031921A CN112740234B CN 112740234 B CN112740234 B CN 112740234B CN 201980031921 A CN201980031921 A CN 201980031921A CN 112740234 B CN112740234 B CN 112740234B
- Authority
- CN
- China
- Prior art keywords
- neuromorphic
- sensor data
- output
- input
- spike
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 63
- 238000000034 method Methods 0.000 claims description 31
- 230000007704 transition Effects 0.000 claims description 16
- 238000001914 filtration Methods 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 230000009471 action Effects 0.000 claims description 6
- 238000005315 distribution function Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 6
- 230000004048 modification Effects 0.000 claims description 5
- 238000012986 modification Methods 0.000 claims description 5
- 230000003595 spectral effect Effects 0.000 claims description 5
- 238000009499 grossing Methods 0.000 claims description 4
- 230000001131 transforming effect Effects 0.000 claims description 4
- 238000012544 monitoring process Methods 0.000 claims 1
- 230000008569 process Effects 0.000 description 14
- 230000006870 function Effects 0.000 description 10
- 238000004590 computer program Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 210000002569 neuron Anatomy 0.000 description 6
- 230000006399 behavior Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 238000013500 data storage Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 239000007788 liquid Substances 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000000875 corresponding effect Effects 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000002547 anomalous effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000002964 excitative effect Effects 0.000 description 2
- 238000010304 firing Methods 0.000 description 2
- 230000002401 inhibitory effect Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 230000005021 gait Effects 0.000 description 1
- 230000007274 generation of a signal involved in cell-cell signaling Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/70—Protecting specific internal or peripheral components, in which the protection of a component leads to protection of the entire computer
- G06F21/88—Detecting or preventing theft or loss
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/31—User authentication
- G06F21/316—User authentication by observing the pattern of computer usage, e.g. typical user behaviour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Computer Security & Cryptography (AREA)
- Social Psychology (AREA)
- Neurology (AREA)
- Debugging And Monitoring (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
A neuromorphic system for authorized user detection is described. The system includes a client device including multiple types of sensors providing streaming sensor data and one or more processors. The one or more processors include an input processing component and an output processing component. Neuromorphic electronic components are embedded in or on the client device to continuously monitor streaming sensor data and generate output spikes based on the streaming sensor data. In addition, the output processing component classifies the streaming sensor data based on output spikes to detect and classify the anomaly signals.
Description
Cross Reference to Related Applications
This application is a continuation-in-part application of U.S. application Ser. No.15/338,228, filed 10/28 in 2016 (which is a non-provisional patent application of U.S. provisional application Ser. No.62/247,557 filed 10/28 in 2015), the entire contents of which are incorporated herein by reference.
The present application is also a non-provisional patent application filed in the united states at 2018, 6, 13, U.S. provisional application No.62/684,515, the entire contents of which are incorporated herein by reference.
Technical Field
The present invention relates to anomaly detection systems, and more particularly to low power neuromorphic detection systems that detect classification changes by autonomously discovering significant differences between sensor data.
Background
Anomaly detection systems are typically used to classify sensor data or otherwise identify system dynamics that may be designated as anomalous behavior. Conventional automated inspection systems operate on top of conventional electronics, thus using a large amount of energy to solve complex classification problems. Such systems typically run on large-scale processors or other relatively large computing systems. However, implementing anomaly detection in a variety of mobile and field applications requires a small system that can effectively handle complex problems. Notably, such detection or classification systems have not been incorporated into small-scale chips that can handle complex problems with little computational overhead.
Therefore, there is a continuing need for low power, small neuromorphic anomaly detection systems.
Disclosure of Invention
The present disclosure relates to a neuromorphic system for authorized user detection. The system includes neuromorphic electronic components embedded in or attached to a client device. The neuromorphic electronic assembly has a neuromorphic chip that is operable to continuously monitor streaming sensor data from a client device and generate an output spike (out-spike) based on the streaming sensor data.
In another aspect, the neuromorphic system further comprises a client device that includes an input processing component, an output processing component, and a plurality of types of sensors for providing streaming sensor data.
In another aspect, the output processing component classifies the streaming sensor data based on output spikes to detect user transitions.
In yet another aspect, the input processing component is configured to further perform the following:
normalizing the streaming sensor data from the plurality of types of sensors into a normalized time series;
grouping the normalized time series from the plurality of types of sensors into a single scalar;
collecting packets of samples of the single scalar into a queue;
transforming the queue into discrete one-dimensional frequency domain data;
modifying the one-dimensional frequency domain data with a window function to reduce spectral leakage and generate frequency domain modified data;
filtering the frequency domain modification data to obtain a scaled windowed frequency interval;
scaling both the normalized time series and the scaled windowed frequency to generate an input rate;
mapping the input rate to a distribution function; and
An input spike is generated based on the mapped input rate.
In another aspect, the neuromorphic electronic component generates an output spike based on the input spike.
In another aspect, the neuromorphic electronic component generates an output spike based on an input spike using a randomly connected excitatory-inhibitory pulse network.
In yet another aspect, the output processing component further performs the following:
smoothing the output spike to a rate;
applying the rate to a linear classifier to generate a reading (readout);
filtering the readings; and
the streaming sensor data is classified as an anomaly signal based on the readings.
In addition, after classifying the abnormal signal, the output processing component further performs at least one of the following operations:
locking or unlocking access to the client device;
starting a new processing task;
executing a new logical branch of executable code;
transmitting information associated with the anomaly signal;
information associated with the exception signal is saved to a memory device.
Finally, the invention also includes a computer program product and a computer implemented method. The computer program product includes computer readable instructions stored on a non-transitory computer readable medium, the computer readable instructions being executable by a computer having one or more processors such that when the instructions are executed, the one or more processors perform the operations listed herein. Alternatively, the computer-implemented method includes acts of causing a computer to execute such instructions and perform the resulting operations.
Drawings
The objects, features and advantages of the present invention will become apparent from the following detailed description of various aspects of the invention, with reference to the following drawings, in which:
FIG. 1 is an illustration of a neuromorphic system for authorized user detection, according to various embodiments of the present disclosure;
FIG. 2 is a block diagram depicting components of an input processing component and an output processing component according to some embodiments of the present disclosure;
FIG. 3 is an illustration of a computer program product according to some embodiments of the present disclosure;
FIG. 4 provides a flow chart depicting processing flow within an input processing component according to some embodiments of the present disclosure;
FIG. 5 is a graphical representation of a path that an electrical signal may travel through neuromorphic electronic components, according to some embodiments of the present disclosure;
FIG. 6 illustrates a process flow of an output processing component according to some embodiments of the present disclosure;
FIG. 7 is a flow chart illustrating an example of a hierarchical threat detection system in which the neuromorphic system of the present disclosure is implemented;
FIG. 8 illustrates a schematic diagram of an example mobile device and neuromorphic electronic assembly in which the neuromorphic-detection system of the present disclosure is implemented;
FIG. 9 is an illustration of an example mobile device in which the neuromorphic-detection system of the present disclosure is implemented; and
fig. 10 is a graph illustrating exponentially smoothed read signals from 5 users of the neuromorphic platform, the graph depicting detected user transitions.
Detailed Description
The present invention relates to anomaly detection systems, and more particularly to low power neuromorphic detection systems that detect classification changes by autonomously discovering significant differences between sensor data. The following description is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of a particular application. Many modifications and various uses will be apparent to those skilled in the art, and the general principles defined herein may be applied in a broad sense. Thus, the present invention is not intended to be limited to the aspects shown, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. It will be apparent, however, to one skilled in the art that the invention may be practiced without limitation to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.
The reader should be careful to all documents and files filed concurrently with this specification, which are disclosed with this specification for public inspection, the contents of all such documents and files are incorporated herein by reference. All the features disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.
Furthermore, any element in the claims that does not explicitly state "means" for performing a particular function or "step" for performing a particular function is not to be construed as a "means" or "step" clause as specified in 35U.S. c. section 112, clause 6. In particular, the use of "… … step" or "… … action" in the claims herein is not intended to introduce 35U.S. C.112 section 6 specification.
Before describing the present invention in detail, a description of several principal aspects of the invention is provided first, followed by an introduction. Next, specific details of the invention are provided to give an understanding of the specific aspects. Finally, a number of example implementations are provided.
(1) Principal aspects
Various embodiments of the present invention include three "primary" aspects. As shown in fig. 1, the first major aspect is a low power neuromorphic system for authorized user detection. The system has three general components, namely an input processing component 102, a neuromorphic electronics component 104, and an output processing component 106. Both the input process 102 and the output process 106 components are implemented in software and/or hardware as a computer system that is owned or otherwise within a "client" system (such as memory and processing components within a mobile device, vehicle, etc.). Neuromorphic electronic component 104 is a component that is implemented in neuromorphic hardware and performs most of the computing processing. The second main aspect is a method typically in the form of software or other program that is run using a data processing system (computer) and neuromorphic hardware as described herein. The third main aspect is a computer program product. The computer program product generally represents computer readable instructions stored on a non-transitory computer readable medium, such as an optical storage device, such as a Compact Disc (CD) or Digital Versatile Disc (DVD), a Field Programmable Gate Array (FPGA), or a magnetic storage device, such as a floppy disk or magnetic tape. Other non-limiting examples of computer readable media include hard disk, read Only Memory (ROM), and flash memory. These aspects will be described in more detail below.
Fig. 2 provides a block diagram depicting a non-limiting example of a computer system 200, which computer system 200 may be implemented to function as an input processing component and/or an output processing component (i.e., elements 102 and/or 106 of fig. 1). Such computer system 200 is configured to perform computations, processes, operations, and/or functions associated with programs or algorithms. In one aspect, certain processes and steps discussed herein are implemented as a series of instructions (e.g., software programs) residing in computer readable storage units and executed by one or more processors of computer system 200. When executed, the instructions cause the computer system 200 to perform certain actions and exhibit certain behavior as may be required to perform the processes described herein.
Computer system 200 may include an address/data bus 202 configured to transfer information. In addition, one or more data processing units, such as processor 204(s), are coupled to address/data bus 202. The processor 204 is configured to process information and instructions. In one aspect, the processor 204 is a microprocessor. Alternatively, the processor 204 may be a different type of processor, such as a parallel processor, an Application Specific Integrated Circuit (ASIC), a Programmable Logic Array (PLA), a Complex Programmable Logic Device (CPLD), or a Field Programmable Gate Array (FPGA).
Computer system 200 is configured to utilize one or more data storage units. Computer system 200 may include a volatile memory unit 206 (e.g., random access memory ("RAM"), static RAM, dynamic RAM, etc.) coupled to address/data bus 202, wherein volatile memory unit 206 is configured to store information and instructions for processor 204. The computer system 200 may also include a nonvolatile memory unit 208 (e.g., read only memory ("ROM"), programmable ROM ("PROM"), erasable programmable ROM ("EPROM"), electrically erasable programmable ROM "EEPROM", flash memory, etc.) coupled to the address/data bus 202, wherein the nonvolatile memory unit 208 is configured to store static information and instructions for the processor 204. Alternatively, computer system 200 may execute instructions retrieved from an online data storage unit (such as in a "cloud" computing). In an aspect, computer system 200 may also include one or more interfaces coupled with address/data bus 202 (such as interface 210) or other interfaces as described in further detail below (e.g., a digital interface to neuromorphic electronic components). The one or more interfaces are configured to enable the computer system 200 to interface with other electronic devices and computer systems. The communication interface implemented by the one or more interfaces may include wired communication technology (e.g., serial cable, modem, network adapter, etc.) and/or wireless communication technology (e.g., wireless modem, wireless network adapter, etc.).
In one aspect, the computer system 200 may include an input device 212 coupled to the address/data bus 202, wherein the input device 212 is configured to communicate information and command selections to the processor 200. According to one aspect, the input device 212 is an alphanumeric input device (such as a keyboard) that may include alphanumeric and/or function keys. Alternatively, the input device 212 may be an input device other than an alphanumeric input device. In one aspect, the computer system 200 may include a cursor control device 214 coupled to the address/data bus 202, wherein the cursor control device 214 is configured to communicate user input information and/or command selections to the processor 200. In one aspect, the cursor control device 214 is implemented using a device such as a mouse, trackball, track pad, optical tracking device, or touch screen. Despite the foregoing, in one aspect, cursor control device 214 is directed and/or activated via input from input device 212 (such as in response to the use of special keys and key sequence commands associated with input device 212). In an alternative aspect, cursor control device 214 is configured to be guided or directed by voice commands.
In an aspect, computer system 200 may also include one or more optional computer usable data storage devices (such as storage device 216) coupled to address/data bus 202. Storage 216 is configured to store information and/or computer-executable instructions. In one aspect, the display device 218 is coupled to the address/data bus 202, wherein the display device 218 is configured to display video and/or graphics.
The computer system 200 presented herein is an example computing environment in accordance with an aspect. However, the non-limiting example of computer system 200 is not strictly limited to computer systems. For example, one aspect provides that computer system 200 represents one type of data processing analysis that may be used in accordance with aspects described herein. In addition, other computing systems may also be implemented. Indeed, the spirit and scope of the present technology is not limited to any single data processing environment. Thus, in one aspect, computer-executable instructions (such as program modules) executed by a computer are used to control or implement one or more operations of various aspects of the present technology. In one implementation, such program modules include routines, programs, objects, components, and/or data structures that are configured to perform particular tasks or implement particular abstract data types. Additionally, an aspect provides for implementing one or more aspects of the technology by utilizing one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where individual program modules are located in both local and remote computer storage media including memory-storage devices.
An exemplary diagram of a computer program product (i.e., a storage device) embodying an aspect of the invention is shown in fig. 3. The computer program product is shown as a floppy disk 300 or an optical disk 302 such as a CD or DVD. However, as previously mentioned, a computer program product generally represents computer readable instructions stored on any compatible non-transitory computer readable medium. The term "instruction" as used in relation to the present invention generally means a set of operations to be performed on a computer and may represent a fragment of an entire program or a single separable software module. Non-limiting examples of "instructions" include computer program code (source code or object code) and "hard-coded" electronics (i.e., computer operations encoded into a computer chip or otherwise in a neuromorphic hardware chip). The "instructions" are stored on any non-transitory computer readable medium, such as in the memory of a computer or in floppy disks, CD-ROMs, FPGAs, and flash drives.
(2) Introduction to the invention
A neuromorphic system for authorized user detection is described. More specifically and referring again to fig. 1, the system is a low power neuromorphic anomaly detection system 100, the low power neuromorphic anomaly detection system 100 identifying anomalies by detecting classification changes by way of significant differences between autonomous discovery sensor data 108. Once such a change is detected, the system generates an output signal 110 as appropriate for the particular application. The output signal 110 may trigger any number of subsequent actions, such as computer processing tasks, transmitting information related to anomalies or environmental changes, and storing data.
The neuromorphic electronic assembly 104 described herein may be implemented using any suitable neuromorphic hardware. In one aspect, neuromorphic electronic assembly 104 is implemented using neuromorphic hardware as described in U.S. patent No.8,977,578, the entire contents of which are incorporated herein by reference. The low power neuromorphic system of the present disclosure uses neuromorphic hardware (such as that described in the' 578 patent) along with additional data flow steps to generate the single output signal 110. In addition, the (tune)/configuration neuromorphic hardware is tuned in a specific manner to produce the desired result.
The system is unique in that it uses neuromorphic hardware (e.g., electronically implemented nonlinear liquid state machines) in conjunction with a software linear classifier to perform classification of sensor data (not just images). This combination of linear and nonlinear steps is unique and provides a significant improvement over the prior art. Neuromorphic techniques are scalable in scale and perform fast enough for complex problems. Furthermore, the system uses machine learning techniques to find differences in normal (nominal) and abnormal (off-nominal) conditions that can be practically applied to modern "client" systems that include electronic devices. Importantly, the system applies the linear classifier to a liquid state machine with a coupling of electronics and software as arranged in the figures filed herewith. In so doing, the system provides low power consumption handling of complex problems that occur on "client" systems, and enables the system to use fewer resources than in the prior art. This implementation allows very low power (less than 100 milliwatts) solutions to complex classification problems with fast response times (from a few milliseconds to a few seconds depending on the application) and small size and weight occupation.
Conventional automated detection systems (without neuromorphic hardware) utilize more energy to solve complex classification problems because their software runs on top of conventional electronics, which are not power efficient and do not parallelize the problem (and therefore run slower). Because the system of the present disclosure may perform activities when a trigger occurs, the client system does not have to perform those activities all the time. Thus, resources are substantially saved in terms of processor utilization, power consumption, data storage, and/or transmission bandwidth.
(3) Specific details of various embodiments
As shown in fig. 1, neuromorphic system 100 includes three serial components: an input processing component 102 (which receives sensor inputs 108), a neuromorphic electronics component 104, and an output processing component 106 (which generates output signals 110). The sensor input 108 includes various types of sensor data. Non-limiting examples of sources of sensor input 106 data include gyroscopes, accelerometers, altimeters, fuel levels, computer network services, etc. feeding the input processing component 102.
As described above, both the input process 102 and the output process 106 components are implemented in software and run on a conventional computer processor, and may be incorporated into any platform or client device that may receive streaming data. Examples of such client devices include mobile devices (e.g., telephones, ipads, etc.), autonomous vehicles, computers, or any other platform/client device that uses a processor and receives streaming data. Thus, while a particular client device is described below and illustrated as a mobile device such as a telephone, it should be understood that the invention is not intended to be limited thereto as all of the features described and illustrated may be incorporated into a variety of different applications.
Neuromorphic electronic component 104 is a component that is implemented in neuromorphic hardware and performs most of the computing processing. The neuromorphic electronic component 104 communicates with the other processing components 102 and 106 through a digital interface, such as a Serial Peripheral Interface (SPI). The output process 106 generates a binary output signal 110, the binary output signal 110 representing a normal state (e.g., authorized user) or an abnormal detection state (e.g., unauthorized user).
For further understanding, FIG. 4 provides a flow chart depicting the processing flow within the input processing 102 component. The input processing 102 component cleans up (i.e., normalizes 400) the sensor input 108 (i.e., streams the sensor data) and then maps the data to either the frequency domain (fd) or the time domain (td). All data (both td and fd) is then converted to a rate, and then to an input spike (in-spikes) 418, as described in further detail below.
The sensor input 108 data is normalized 400 by mapping the sensor specific range onto a zero to one scale to generate normalized time series data. In the frequency domain, the data is combined 402 by grouping the data types into a single scalar by time instance. Thereafter, packets of samples are collected 404 into a queue of sample size suitable for the application. The queue is converted 406 into discrete one-dimensional (1D) frequency domain data. The 1d frequency domain data is then modified 408 by multiplying the data by a window function, such as a Hamming window (Hamming window) or any other suitable window function, to reduce spectral leakage. Thereafter, the data is filtered 410 by rejecting frequency bins (frequency bins) outside the particular application frequency range, resulting in scaled windowed frequency bins. Non-limiting examples of such applications and frequency ranges include selecting a frequency between 0.5Hz and 10Hz in gait applications.
The input processing 102 component then scales 412 all values (both the normalized time series and the scaled windowed frequency interval) such that the values are limited to a maximum execution time to generate an input rate (in-rate). For example, the values are mapped linearly onto a range of spike rates (e.g., 0Hz to 200Hz, etc.). The input rate is mapped 414 to a distribution function (e.g., poisson probability distribution function (P)). Finally, input spikes 418 are generated 416 for the respective input pads of the neuromorphic electronic component 104 and the neuromorphic chip 821. The input spike 418 is a binary value for each input pad that is generated based on a comparison of a randomly generated number and a poisson probability distribution (P) value associated with the input rate value. The input peaks 418 are transmitted in a structure equivalent to a one-dimensional binary array. The maximum number of input peaks 418 is equal to the number of input pads present on a neuromorphic chip (i.e., a neuromorphic electronic component having an electronically-implemented liquid state machine with leakage integration firing (leaky integrate and fire) neurons, such as disclosed in U.S. patent No.8,977,578).
Fig. 5 provides a graphical representation of neuromorphic electronic component 104, and the path that an electronic signal may travel from input pad 500 to output pad 502 for a particular configuration. The bright square 504 represents an excitatory neuron and the dark square 506 represents an inhibitory neuron. The neuromorphic electronic component 104 receives the input spike 418 as an input and generates the output spike 508 as an output pad signal. The output spikes 508 are also sent in a structure equivalent to a one-dimensional binary array. The maximum number of output spikes 508 is equal to the number of output pads 502 present on the neuromorphic chip of the neuromorphic electronic assembly 104.
Fig. 6 illustrates the processing flow of the output processing 106 component. As shown in fig. 6, the output spikes 508 are smoothed 600 into a firing rate or output rate 602, which is a measure of the number of spikes per second. The velocity 602 is then applied to the linear classifier 604 to calculate 606 readings 608. The linear classifier 604 acts as a mapping from the rate 602 to the readings 608. Linear classifiers are generally understood by those skilled in the art (see, e.g., byA "linear classifier" is defined, the entire contents of which are incorporated herein by reference). The linear classifier 604 is referred to because each reading 608 is a linear combination of the rates 602. Thus, this is a simple matrix multiplication operation in which the learning process is to determine a matrix by which the rate 602 is multiplied to obtain the reading 608. Desirably, discriminant training 603 must be performed on the linear classifier 604 before reliable anomaly signal generation can occur. Training 603 is based on supervised machine learning techniques using truth values 614. Each training stage is specific to one category at a time (there may be two or more categories). The differences between the categories (time and/or frequency domain) are the cause of anomalies or anomaly signals that will result. The frequency domain data is calculated in real time and used in a sliding window manner. The amount of frequency domain output data size is equal to the queue sample size.
The readings 608 are a plurality of floating point values corresponding to each category (2 or more). The readings are then filtered 610 to remove noise and an anomaly detection process 612 is performed, which produces a final output signal 110 (e.g., an anomaly signal specifying the presence of an authorized or unauthorized user). The anomaly detection process 612 identifies unauthorized or authorized users by signaling that a user change has occurred. When there is a user transition, the reading 608 signal becomes anomalous and the anomaly (in the reading 608 signal) is detected by the system.
(4) Example implementation
As will be appreciated by those skilled in the art, the neuromorphic anomaly detection system of the present disclosure has many applications. For example, the system has been applied to detect user changes of a mobile device using biometric sensor data. In this implementation, the system is physically attached or embedded into the mobile device, where power usage by the mobile device is a focus of resource conservation. When an unauthorized user is detected, the mobile device locks the user from further use until an authorized user is detected (at which point the features/functions of the mobile device are unlocked and accessible). In addition, the system may be used in other applications where power consumption, feature size, and/or device accessibility are very limited. The system is also beneficial if it would be most beneficial if complex anomaly detection (for an off-board server/cloud solution) were performed directly on the client system.
As a specific example, the low power neuromorphic system of the present disclosure is applied in a hierarchical anomaly classification solution, such as in stage 1 block 700 of a hierarchical threat detection algorithm 702 (shown in fig. 7) further described in U.S. application No.15/338,228. In other words, the process described in U.S. application Ser. No.15/338,228 is modified to include the low power neuromorphic system of the present disclosure as stage 1 block 700. In this modified system, sensor data is generated from gyroscopes and accelerometers at a frequency of 50 Hz. The queue size is 200 samples. The application frequency range of interest is 0.3Hz to 20.0Hz. The upper limit of the rate is 200Hz. The execution time is in the range from 1.5ms to 5 ms. Effectively (active) about 25 input pads are used and effectively about 50 output pads are used. A Liquid State Machine (LSM) is configured such that 300 excitatory neurons and 25 inhibitory neurons are activated. The network map of neurons is connected at 1%. The system successfully distinguishes between multiple users operating a phone and issuing a phone alert when an unverified user walks with the phone.
Fig. 8 shows a schematic diagram of an example mobile device 800 (e.g., a phone) and neuromorphic-electronic component 104 (i.e., neuromorphic-chip 821 in communication with FPGA 823 or any other hardware or component required to allow neuromorphic chip 821 to operate). FPGA 823 is responsible for the configuration of neuromorphic chip 821 (e.g., a non-limiting example of which includes loading a particular network, such as the network described above with respect to fig. 5). Sub-components that have been developed and exist on mobile device 800 include a sensor data collection application 802 and a data resampler 804, the data resampler 804 converting all sensor streams from non-uniform sensor data 803 to uniform sampling rate (i.e., uniform data (binary) 805). The output of the resampler 804 goes directly to the local EWS application 806 and to the rate-based spike encoder 808 (which acts as the input processing component described above). It should be noted that a shielded cable 830 for transmission is illustrated; however, the present invention is not limited to such cables, as the particular cable 830 shown is provided as a non-limiting example of such transmission media.
The sensor signals encoded as input peaks 418 are sent to neuromorphic electronic component 104 via Serial Peripheral Interface (SPI) connection 807 and corresponding line driver and receiver 827 on mobile device 800. The alert generated by neuromorphic electronics component 104 is encoded into output spike 518 and sent to mobile device 800 over SPI connection 807 (and corresponding line driver and receiver 827), where output spike 518 is decoded 809, then read by EWS application 806 (i.e., which acts as an output processing component as described above) to determine intent or classification, and then broadcast 813 to optional policy engine 815, which optional policy engine 815 maintains a policy regarding acceptable intent.
As a non-limiting example, if policy engine 815 specifies that the class of "unauthorized users" cannot be allowed to continue to use device 800, various protocols or actions 817 may be sent for mobile device 800 implementation, such as the following operations: all device accesses 819 are locked until an authorized user is detected (e.g., an authorized user enters an appropriate access code into the system). Other examples based on exception detection (i.e., unauthorized user access, etc.) include starting a new processing task or executing a new logical branch of executable code, sending information associated with the exception, and saving the information associated with the exception to a memory device. In some aspects, the activity is terminated upon or shortly after the signal transitions from the anomaly back to the normal state. Or, as yet another example, if the signal is classified as an authorized user, other features of the device 800 in the device 800 may be unlocked. As an optional step, the activity between devices may be correlated through EWS 825.
Fig. 9 illustrates an example mobile device 800 implementing the neuromorphic-detection system of the present disclosure and included neuromorphic-electronic components 104 on an interface backplane attached to the mobile device 800. The signal output from device 800 via the SPI connection was tested. As shown in fig. 10, the exponentially smoothed read signals from 5 users of the neuromorphic platform were input to the neuromorphic classifier and performed with an average of 84.16% accuracy. In fig. 10, the "filtered reading" axis 1002 is dimensionless. They are the result of multiplying the output rate by a linear classifier weight matrix and then applying a low pass filter. The "sample ID" axis 1004 is referred to as a data point index and is therefore dimensionless as well. It should be appreciated that since the index and time are isomorphic (isomorphic) in this case, the x-axis may also be generated based on time.
The classification accuracy of each user is 66.53% at the minimum and 99.21% at the maximum. The average true positive and negative rates were 89.67% and 60.32%, respectively. Since the chip was trained on data for 5 different subjects, the chance classification rate was 20%. These results indicate a reasonably good user classification for low power systems. The spike transition across the time domain represents a user transition 1000, which is measured by the read signal. As described above, when there is a user transition 1000, the reading signal becomes abnormal, and the abnormality (in the reading signal) is detected by the system.
Perhaps more important than user classification in some embodiments is the ability of the neuromorphic system to detect changes in user identity. To create meaningful performance estimates, a continuous alarm aggregation strategy is formulated on the user transition output and the time series is divided into equal length blocks from which a true value (ground-trunk) is determined. First, the user transition strategy sets the minimum interval (margin) between detection of false alarms to 8.2 seconds. Doing so prevents unwanted successive alarms from occurring for the same transition event. Next, the total number of experiments for each alarm test was set to: (total time in sample set)/(2 x interval). If an alarm is not detected within +/-interval seconds of the real time associated with the alarm, the alarm is classified as false negative. If an alarm is detected, the alarm is considered to be true positive. Using this experimental setup, the resulting metric produced 98.74% accuracy in detecting user transitions, with 99.57% true positive and 75% true negative. These results demonstrate the great advantage of using neuromorphic systems as user transition detection systems because of their high accuracy, minimizing the amount of time that local EWS will run, thereby further conserving power.
Continuous behavior-based authentication of devices (e.g., mobile devices) is an important core technology area. In the field of national defense and business where the problem of stealing personal data from lost or stolen mobile devices (e.g., telephones) is growing, the development of improved low power security and authentication techniques for mobile devices has received great attention. Mobile devices are increasingly embedded in vehicles and airplanes, and authentication of these devices is becoming increasingly important in view of the intentional adoption of these systems by adversaries. The security necessary to continuously and reliably authenticate a user with minimal burden requires power-efficient behavior-based inference. The invention described herein provides transformation capabilities for the development of next generation behavior-based authentication and enhanced security protocols for various devices, including mobile devices. Upon detection of an unauthorized user, the device may be caused to perform various automated operations including stopping the operation, pulling the autonomous vehicle safely to the curb and shutting down, locking the user in place, etc.
Finally, while the invention has been described in terms of various embodiments, those skilled in the art will readily recognize that the invention may have other applications in other environments. It should be noted that many embodiments and implementations are possible. In addition, the following claims are in no way intended to limit the scope of the invention to the specific embodiments described above. In addition, any expression of "means for … …" is intended to cause a reading of the elements and the means of the claims without specifically using the term "means for … …" is not intended to be interpreted as an element of the means plus the function even if the claim additionally includes the term "means". Furthermore, although certain method steps have been recited in a particular order, these method steps may occur in any desired order and are within the scope of the invention.
Claims (13)
1. A neuromorphic system for authorizing user detection, the neuromorphic system comprising:
a neuromorphic electronic component for embedding in or attaching to a client device, the neuromorphic electronic component having a neuromorphic chip operable to continuously monitor streaming sensor data from the client device and generate output spikes based on the streaming sensor data;
a client device comprising a plurality of types of sensors for providing the streaming sensor data, an input processing component, and an output processing component, and wherein the output processing component classifies the streaming sensor data based on the output spike to detect user transitions; and is also provided with
Wherein the input processing component is configured to further perform the following:
normalizing the streaming sensor data from the plurality of types of sensors into a normalized time series;
grouping the normalized time series from the plurality of types of sensors into a single scalar;
collecting packets of samples of the single scalar into a queue;
Transforming the queue into discrete one-dimensional frequency domain data;
modifying the one-dimensional frequency domain data with a window function to reduce spectral leakage and generate frequency domain modified data;
filtering the frequency domain modification data to obtain a scaled windowed frequency interval;
scaling both the normalized time series and the scaled windowed frequency to generate an input rate;
mapping the input rate to a distribution function; and
an input spike is generated based on the mapped input rate.
2. The neuromorphic system of claim 1 wherein the neuromorphic electronic component generates an output spike based on the input spike.
3. The neuromorphic system of claim 2 wherein the neuromorphic electronic component generates the output spike based on the input spike using a randomly-connected excitatory-inhibitory spike network.
4. A neuromorphic system for authorizing user detection, the neuromorphic system comprising:
a neuromorphic electronic component for embedding in or attaching to a client device, the neuromorphic electronic component having a neuromorphic chip operable to continuously monitor streaming sensor data from the client device and generate output spikes based on the streaming sensor data;
A client device comprising a plurality of types of sensors for providing the streaming sensor data, an input processing component, and an output processing component, and wherein the output processing component classifies the streaming sensor data based on the output spike to detect user transitions; and is also provided with
Wherein the output processing component further performs the following:
smoothing the output spike to a rate;
applying the rate to a linear classifier to generate readings;
filtering the readings; and
classifying the streaming sensor data as an anomaly signal based on the reading.
5. The neuromorphic system of claim 4 wherein, after classifying the abnormal signal, the output processing component further performs at least one of:
locking or unlocking access to the client device;
starting a new processing task;
executing a new logical branch of executable code;
transmitting information associated with the anomaly signal;
information associated with the exception signal is saved to a memory device.
6. A neuromorphic method for authorizing user detection, the neuromorphic method comprising the acts of:
Continuously monitoring streaming sensor data from a client device using a neuromorphic electronic component having a neuromorphic chip;
generating, with the neuromorphic-electronic-component having the neuromorphic-chip, an output spike based on the streaming sensor data; and
classifying, using the client device, the streaming sensor data based on the output spike to detect user transitions, the client device having multiple types of sensors for providing the streaming sensor data, an input processing component, and an output processing component, wherein the output processing component classifies the streaming sensor data; and is also provided with
The neuromorphic method further performs the following actions by the input processing component:
normalizing the streaming sensor data from the plurality of types of sensors into a normalized time series;
grouping the normalized time series from the plurality of types of sensors into a single scalar;
collecting packets of samples of the single scalar into a queue;
transforming the queue into discrete one-dimensional frequency domain data;
modifying the one-dimensional frequency domain data with a window function to reduce spectral leakage and generate frequency domain modified data;
Filtering the frequency domain modification data to obtain a scaled windowed frequency interval;
scaling both the normalized time series and the scaled windowed frequency to generate an input rate;
mapping the input rate to a distribution function; and
an input spike is generated based on the mapped input rate.
7. The neuromorphic method of claim 6, further comprising the acts of: the output spike is generated by the neuromorphic electronic component based on the input spike.
8. The neuromorphic method of claim 7 wherein the neuromorphic electronic component generates the output spike based on the input spike using a randomly-connected excitatory-inhibitory spike network.
9. The neuromorphic method of claim 6, further comprising, by the output processing component:
smoothing the output spike to a rate;
applying the rate to a linear classifier to generate readings;
filtering the readings; and
classifying the streaming sensor data as an anomaly signal based on the reading.
10. The neuromorphic method of claim 9 wherein, after classifying the abnormal signal, the neuromorphic method further comprises performing at least one of the following actions:
Locking access to the client device;
starting a new processing task;
executing a new logical branch of executable code;
transmitting information associated with the anomaly signal; and
information associated with the exception signal is saved to a memory device.
11. A non-transitory computer-readable medium encoded with executable instructions such that, when the instructions are executed by one or more processors, the one or more processors perform the operations of:
receiving output spikes from neuromorphic electronic components having neuromorphic chips that continuously monitor streaming sensor data from a client device and generate output spikes based on the streaming sensor data;
classifying, using the client device, the streaming sensor data based on the output spike to detect user transitions, the client device having multiple types of sensors for providing the streaming sensor data, an input processing component, and an output processing component, wherein the output processing component classifies the streaming sensor data;
The non-transitory computer-readable medium further stores instructions for causing the input processing component to:
normalizing the streaming sensor data from the plurality of types of sensors into a normalized time series;
grouping the normalized time series from the plurality of types of sensors into a single scalar;
collecting packets of samples of the single scalar into a queue;
transforming the queue into discrete one-dimensional frequency domain data;
modifying the one-dimensional frequency domain data with a window function to reduce spectral leakage and generate frequency domain modified data;
filtering the frequency domain modification data to obtain a scaled windowed frequency interval;
scaling both the normalized time series and the scaled windowed frequency to generate an input rate;
mapping the input rate to a distribution function; and
an input spike is generated based on the mapped input rate.
12. The non-transitory computer-readable medium of claim 11, further storing instructions for causing the output processing component to:
smoothing the output spike to a rate;
Applying the rate to a linear classifier to generate readings;
filtering the readings; and
the streaming sensor data is classified as an anomaly signal based on the readings.
13. The non-transitory computer-readable medium of claim 12, wherein, after classifying the anomaly signal, the instructions further cause the one or more processors to at least one of:
locking access to the client device;
starting a new processing task;
executing a new logical branch of executable code;
transmitting information associated with the anomaly signal; and
information associated with the exception signal is saved to a memory device.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US201862684515P | 2018-06-13 | 2018-06-13 | |
| US62/684,515 | 2018-06-13 | ||
| PCT/US2019/026830 WO2019240868A1 (en) | 2018-06-13 | 2019-04-10 | Neuromorphic system for authorized user detection |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112740234A CN112740234A (en) | 2021-04-30 |
| CN112740234B true CN112740234B (en) | 2024-01-02 |
Family
ID=68842032
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201980031921.5A Active CN112740234B (en) | 2018-06-13 | 2019-04-10 | Neuromorphic system for authorized user detection |
Country Status (3)
| Country | Link |
|---|---|
| EP (1) | EP3807822A4 (en) |
| CN (1) | CN112740234B (en) |
| WO (1) | WO2019240868A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111669514B (en) * | 2020-06-08 | 2021-02-26 | 北京大学 | High dynamic range imaging method and apparatus |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104685516A (en) * | 2012-08-17 | 2015-06-03 | 高通技术公司 | Apparatus and methods for spiking neuron network learning |
| CN107016437A (en) * | 2016-01-13 | 2017-08-04 | 国际商业机器公司 | The method efficiently generated, system and computer-readable recording medium for the random peaks pattern in core base neuromorphic system |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8515885B2 (en) * | 2010-10-29 | 2013-08-20 | International Business Machines Corporation | Neuromorphic and synaptronic spiking neural network with synaptic weights learned using simulation |
| US8977578B1 (en) * | 2012-06-27 | 2015-03-10 | Hrl Laboratories, Llc | Synaptic time multiplexing neuromorphic network that forms subsets of connections during different time slots |
| BR112016002229A2 (en) * | 2013-08-09 | 2017-08-01 | Behavioral Recognition Sys Inc | cognitive neurolinguistic behavior recognition system for multisensor data fusion |
| US9798751B2 (en) * | 2013-10-16 | 2017-10-24 | University Of Tennessee Research Foundation | Method and apparatus for constructing a neuroscience-inspired artificial neural network |
| US10157629B2 (en) * | 2016-02-05 | 2018-12-18 | Brainchip Inc. | Low power neuromorphic voice activation system and method |
| US20170227995A1 (en) * | 2016-02-09 | 2017-08-10 | The Trustees Of Princeton University | Method and system for implicit authentication |
| US10628568B2 (en) * | 2016-03-31 | 2020-04-21 | Fotonation Limited | Biometric recognition system |
| US20170337469A1 (en) * | 2016-05-17 | 2017-11-23 | Agt International Gmbh | Anomaly detection using spiking neural networks |
| US11003984B2 (en) * | 2016-05-31 | 2021-05-11 | Samsung Electronics Co., Ltd. | Timing sequence for digital STDP synapse and LIF neuron-based neuromorphic system |
| US10671912B2 (en) * | 2016-09-13 | 2020-06-02 | Sap Se | Spatio-temporal spiking neural networks in neuromorphic hardware systems |
-
2019
- 2019-04-10 EP EP19820220.2A patent/EP3807822A4/en active Pending
- 2019-04-10 WO PCT/US2019/026830 patent/WO2019240868A1/en active Search and Examination
- 2019-04-10 CN CN201980031921.5A patent/CN112740234B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104685516A (en) * | 2012-08-17 | 2015-06-03 | 高通技术公司 | Apparatus and methods for spiking neuron network learning |
| CN107016437A (en) * | 2016-01-13 | 2017-08-04 | 国际商业机器公司 | The method efficiently generated, system and computer-readable recording medium for the random peaks pattern in core base neuromorphic system |
Non-Patent Citations (1)
| Title |
|---|
| Neuromorphic and early warning behavior-based authentication for mobile devices;PHILLIPS MATTHEW E;IEEE;1-4 * |
Also Published As
| Publication number | Publication date |
|---|---|
| EP3807822A4 (en) | 2022-03-23 |
| WO2019240868A1 (en) | 2019-12-19 |
| EP3807822A1 (en) | 2021-04-21 |
| WO2019240868A8 (en) | 2020-11-26 |
| CN112740234A (en) | 2021-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Aldweesh et al. | Deep learning approaches for anomaly-based intrusion detection systems: A survey, taxonomy, and open issues | |
| Koroniotis et al. | A new network forensic framework based on deep learning for Internet of Things networks: A particle deep framework | |
| Azmoodeh et al. | Detecting crypto-ransomware in IoT networks based on energy consumption footprint | |
| Selim et al. | Anomaly events classification and detection system in critical industrial internet of things infrastructure using machine learning algorithms | |
| Yang et al. | Intrusion detection based on bidirectional long short-term memory with attention mechanism | |
| Yang et al. | Application of meta-learning in cyberspace security: A survey | |
| Du et al. | Digital Forensics as Advanced Ransomware Pre‐Attack Detection Algorithm for Endpoint Data Protection | |
| Shahin et al. | A novel fully convolutional neural network approach for detection and classification of attacks on industrial IoT devices in smart manufacturing systems | |
| CN113470695A (en) | Sound abnormality detection method, sound abnormality detection device, computer device, and storage medium | |
| Wang et al. | Computational intelligence for information security: A survey | |
| WO2022115419A1 (en) | Method of detecting an anomaly in a system | |
| CN113746780B (en) | Abnormal host detection method, device, medium and equipment based on host image | |
| Zhang et al. | A binarized neural network approach to accelerate in-vehicle network intrusion detection | |
| Ndife et al. | Cyber-Security Audit for Smart Grid Networks: An Optimized Detection Technique Based on Bayesian Deep Learning. | |
| Khan et al. | DivaCAN: Detecting in-vehicle intrusion attacks on a controller area network using ensemble learning | |
| Shahriar et al. | CANShield: Deep Learning-Based Intrusion Detection Framework for Controller Area Networks at the Signal-Level | |
| CN112740234B (en) | Neuromorphic system for authorized user detection | |
| Solani et al. | A novel approach to reduce false-negative alarm rate in network-based intrusion detection system using linear discriminant analysis | |
| Joshi | Classification, clustering and intrusion detection system | |
| US10902115B2 (en) | Neuromorphic system for authorized user detection | |
| Lu et al. | One intrusion detection method based on uniformed conditional dynamic mutual information | |
| Shahin | Evaluating the Fidelity and Efficiency of Network Intrusion Detection Systems Via Deep Learning, Machine Learning, and Deep Hybrid Learning in Industrial IoT Devices | |
| Shahin et al. | An AI-Powered Network Intrusion Detection System in Industrial IoT Devices via Deep Learning | |
| Himthani et al. | Application of machine learning techniques in intrusion detection systems: A systematic review | |
| Wu et al. | Intrusion Detection System Using a Distributed Ensemble Design Based Convolutional Neural Network in Fog Computing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |