WO2016108962A1 - Fast efficient evaluation of messages on automotive networks using look-up tables - Google Patents
Fast efficient evaluation of messages on automotive networks using look-up tables Download PDFInfo
- Publication number
- WO2016108962A1 WO2016108962A1 PCT/US2015/050615 US2015050615W WO2016108962A1 WO 2016108962 A1 WO2016108962 A1 WO 2016108962A1 US 2015050615 W US2015050615 W US 2015050615W WO 2016108962 A1 WO2016108962 A1 WO 2016108962A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- kernel
- svm
- look
- classifier
- svm classifier
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N7/00—Computing arrangements based on specific mathematical models
Definitions
- the message ID is 1 1 bits (2048 possible message IDs) or 29 bits (-500 million possible message IDs).
- Some CAN bus architectures can support both 1 1 -bit and 29-bit message IDs. Collisions are arbitrated by prioritizing the message ID with the most leading zeros; the lower priority message is then re-sent after a specified delay.
- OBD-II OBD-II
- USB USB
- Bluetooth WiFi enabled on-vehicle hotspots
- cellular connections The consequences of an unintentional glitch or intentional hack can range from very minor (e.g. the car stereo volume decreasing) to very serious (e.g. loss of vehicle braking, uncontrolled engine acceleration, or aircraft flight control interference).
- CAN Controller Area Network
- the SVM classifier comprises a microprocessor or microcontroller programmed to perform classification using the SVM classifier with at least some terms F(x i d , y d ) of the kernel f ⁇ xt, y) determined using a look-up table.
- the kernel f ⁇ xt, y) may, for example, be a Radial Basis Function (RBF) kernel.
- RBF Radial Basis Function
- the dimension y d can only assume values defined by a finite set of values, or in more limited embodiments by a finite set of integers.
- the microprocessor or microcontroller programmed to perform classification using the SVM classifier comprises the microprocessor or microcontroller of an embedded system.
- an embedded system comprises the microprocessor or microcontroller of the immediately preceding paragraph.
- the microprocessor or microcontroller may, for example, be further programmed to perform an embedded system function such as engine control, brake control, entertainment system operation, or so forth, or an embedded system function for a vehicle, ocean buoy, or satellite.
- an SVM classifier training system comprises a computer (for example, a cloud-based computing resource, a parallel processor system, or a cluster computer) programmed to train an SVM classifier employing a kernel f ⁇ Xi, y) ⁇ x F(x d , y d ) where D is the dimensionality of Support Vectors x t and of data vectors y, and to generate a look-up table containing the possible values for at least one term F ⁇ x i d , y d ) of the kernel
- the applying is suitably performed by a microprocessor or microcontroller.
- FIGURE 1 diagrammatically shows a classifier training system in accord with illustrative embodiments disclosed herein.
- FIGURE 2 diagrammatically illustrates a classifier in accord with illustrative embodiments disclosed herein.
- An approach to CAN bus security in the face of this "zero knowledge" context is zero knowledge anomaly detection.
- an anomaly is either a glitch or an intrusion (e.g. hacking) - more generally, CAN bus traffic that should not be present.
- a priori knowledge of what types of messages may be transmitted by a given ECU may be limited (e.g., due to traffic introduced by aftermarket components connected with the CAN bus).
- an anomaly is defined as follows: an anomaly is a statistically significant deviation from normal CAN bus behavior as gleamed from CAN bus message traffic. "Statistically significant" means that the deviation from normal CAN bus behavior is sufficient, measureable and greater than the allowable limit.
- An anomaly detection system can be constructed based on statistical analysis of a training data set consisting of CAN bus traffic acquired from a vehicle known to be operating normally. For example, training data can be acquired at the automobile manufacturer facility, using a vehicle straight off the assembly line, or a pre-production model, or some other vehicle. This approach is unusual in that the training data are all "negative” samples, that is, all of the training data are "normal” and none of the training data are anomalies. Thus, ideally, after training on such data the anomaly detection system should not generate any anomaly alerts.
- the anomaly detection algorithm comprises a Support Vector Machine (SVM) trained to operate as a one-class classifier outputting either a positive value (indicating an anomaly) or a negative value (indicating a normal message).
- SVM Support Vector Machine
- the one-class classifier can also be thought of as a binary classifier in which one "class” is “anomaly” and the other "class” is "normal message” or "not anomaly”).
- Each message is treated as a vector residing in a high-dimensional vector space.
- a message includes (in time-order): an 1 1 -bit message ID (with an optional 18-bit extension message ID); network ID denoting the network bus the message arrived on; and eight data bytes.
- the training set consists of a set of training vectors, and a sub-set of the training vectors forms the set of Support Vectors.
- a hyperplane is defined respective to the Support Vectors that is optimized by the SVM training to separate positive samples from negative samples while maximizing margins to the nearest training samples in the vector space.
- the illustrative one-class SVM classifier employs a Radial Basis Function (RBF) as the kernel for defining the hyperplane.
- RBF Radial Basis Function
- the Support Vectors are chosen from the training sub-set by the SVM training algorithm so as to provide positive/negative sample segregation with maximized margins, typically subject to additional constraints defined by tuning parameters.
- the tuning parameters include the already-mentioned y parameter, which controls the shape or curvature of the separating hyperplane. A larger value of y increases hyperplane curvature and also tends to increase the number of Support Vectors chosen during the SVM training.
- Another tuning parameter, v defines a lower bound on the number of Support Vectors, and also presents an upper bound on the number of training samples that may be erroneously classified (this error count can be reduced by forced adding of Support Vectors).
- a tuning parameter ⁇ defines a penalty for data points inside the margin (i.e. too close to the hyperplane), and may be viewed as controlling granularity of the hyperplane.
- SVM training is performed as follows.
- the tuning parameters y, v, ⁇ (or additional or other tuning parameters depending upon the particular SVM model being employed) are chosen, and SVM training is performed to optimize selection of the Support Vectors so as to optimally segregate positive from negative samples while satisfying constraints defined by the tuning parameters y, , ⁇ .
- the user may then choose to adjust the tuning parameters y, v, ⁇ and repeat the training, so as to tune the SVM classifier by trial-and-error.
- the trained SVM classifier is applied to classify a new message represented by the message vector y according to:
- the tuning parameters y, , ⁇ from the training are used, N is the number of support vectors chosen by the SVM training, and the vectors x lt ... , x N are the Support Vectors chosen by the training.
- the trained SVM classifier is implemented on an ECU (either as an additional function of an existing ECU, or as a separate ECU dedicated to providing CAN network security).
- the computing power of the ECU is limited, as it can usually be, at most, a standalone multi-core CPU system.
- the SVM training can be performed offline with massive computing power brought to bear, for example using a cloud computing system or other parallel computing system that may include dozens or more CPUs.
- Equation (2) by inserting Equation (1 ) yields: e -Y ⁇ (xsvi,i-yi) 2 +--+(xsvi,D-yD) 2 ) _
- the dimensions 3 ... 10 correspond to the eight data bytes.
- Each of these dimensions 3, ... ,10 can thus assume only integer values in the range 0, ... ,255.
- the values ...,3 ⁇ 4 il0 are fixed values for the Support Vector x t .
- T(i, 3, y 3 ) is the look-up table entry for Support Vector x third dimension, for value y 3 .
- T(i, 3, y 3 ) is the look-up table entry for Support Vector x third dimension, for value y 3 .
- all eight message byte values e ⁇ 3 ⁇ y3 ⁇ , e(* o-y i o) can be stored in a table with dimensions (1 ... N ; 3 ... 10 ; 0 ... 255) .
- the message and network ID dimensions that is, the terms e ⁇ i l_3/1 ) and e ⁇ xi, 2 -y 2 ) j are more complex, and whether a look-up table approach is viable depends upon the particular message ID format, especially the number of bits. Assuming the first dimension is a message ID of fixed 1 1 bits, corresponding to 2048 possible message IDs, a look-up table for the message ID having dimensions (1 ... N ; 0 ... 2047) is feasible. If a second ID field (network ID) is provided, a tabular approach for this would employ a table of dimensions (1 ... N ; 0 ... 262,143), which may be feasible depending upon storage capacity.
- the CAN network architecture employs a single message ID that is 29 bits, this corresponds to 536,870,912 possible message IDs, so that a table approach may not be feasible.
- a look-up table such as one of dimensions (1 ... N ; 1 ... S ; 1 ... S) where S denotes the actually used message ID values.
- FIGURES 1 and 2 an illustrative embodiment is shown, including the training system (FIGURE 1 ) and the embedded system (FIGURE 2).
- the one-class SVM classifier training is suitably performed by a high-capacity computing system, such as an illustrative server computer 10, which may be cloud-based (i.e. a cloud computing resource), a parallel processor system, cluster computer, or so forth.
- the training is performed by a one-class SVM classifier trainer 12 using a training set of CAN bus traffic 14 represented as vectors as described herein (e.g. one or two dimensions for message/network ID information, and eight dimensions for the data bytes) and SVM tuning parameters 16.
- the SVM trainer 12 outputs a set of Support Vectors 20 chosen from the training data 14.
- the training may also entail optimizing the SVM tuning parameters 16, either by trial-and-error (e.g. a human analyst re-runs the SVM trainer 12 using different values for the SVM tuning parameters 16, or an automated approach such as a grid search is used to optimize the SVM tuning parameters 16).
- the illustrative SVM that is trained by the SVM trainer 12 employs a Radial Basis Function (RBF) as the kernel.
- RBF Radial Basis Function
- the disclosed approaches are applicable to any SVM that employs a kernel: where Expression (7) is written for Support Vector x D denotes the number of dimensions of the vector, the symbol "oc" is conventionally used to denote a proportional relationship, and the right-most equality is the illustrative RBF earlier set forth using alternative notation as Expression (1 ).
- the disclosed approaches are applicable to any SVM that employs a kernel:
- a look-up table generator 22 is applied to exhaustively compute all possible values for at least those terms F(x i ,y d ) corresponding to data bytes. More generally, the look-up table generator 22 can be applied to exhaustively compute all possible values any term F(x i , y d ) for which x i and y d can only assume values defined by a finite (i.e. closed) set of values, such as the set of integers ⁇ 0, ... ,255 ⁇ for an 8-bit data byte, or the set of integers ⁇ 0, ... ,2047 ⁇ for an 1 1 -bit message ID dimension.
- a finite (i.e. closed) set of values such as the set of integers ⁇ 0, ... ,255 ⁇ for an 8-bit data byte, or the set of integers ⁇ 0, ... ,2047 ⁇ for an 1 1 -bit message ID dimension.
- the output of the look-up table generator 22 is a set of one or more look-up tables 30. It should be noted that while the look-up table generator 22 may require substantial computing resources, those computing resources are available in the form of the illustrative server computer 10 (which may be cloud-based, a parallel processor system, cluster computer, or so forth). Thus, the look-up table generator 22 is not limited by the limited computing power or resources of any automotive-class ECU.
- a vehicle 40 such as an illustrative automobile, or a truck, watercraft (boat, ship, et cetera), aircraft (airplane, blimp, helicopter, et cetera) includes at least one (and typically many) embedded system 42 comprising an electronic control unit (ECU) 44 (shown diagrammatically in the inset).
- the vehicle 40 also includes at least one (and possibly two or more) Controller Area Network bus (CAN bus) 46 via which the embedded systems can intercommunicate.
- the illustrative ECU 44 includes a transceiver 48 that transmits and receives messages via the CAN bus 46.
- the ECU 44 comprises a microprocessor or microcontroller and ancillary electronics (e.g. memory, the illustrative transceiver 48, or so forth) programmed to provide the desired functionality of the embedded system 42.
- This functionality is represented in the inset diagram representing the ECU 44 as an ECU functional module 50.
- the ECU functional module 50 may provide engine control, brake control, entertainment system operation, or so forth.
- the ECU 44 is further programmed to implement a one-class SVM message classifier 60 that classifies each message passing through the transceiver 48 as either "normal” or "anomaly".
- a one-class SVM message classifier 60 that classifies each message passing through the transceiver 48 as either "normal” or "anomaly".
- both sent and received messages are classified, although it is contemplated to only classify received messages under the assumption that the embedded system 42 is secure, and/or under the assumption that any anomalous transmission events generated by the embedded system 42 will be detected by other ECUs on the CAN bus 46 also running instances of the one-class SVM message classifier 60.
- every ECU on the CAN network receives every message, not just those messages relevant to the ECU.
- the SVM message classifier 60 can classify every message on the CAN bus 46 as either "normal” or “anomalous".
- the one-class SVM message classifier 60 employs the trained SVM produced by the system of FIGURE 1 , including using the N Support Vectors 20, the (optionally optimized) SVM tuning parameters 16, and the look-up table(s) 30.
- Expression (2) is evaluated, using the look-up table(s) 30 to evaluate at least some terms F(x i , y d ) of the kernel function f(x it y) for each Support Vector x t .
- the one-class SVM message classifier 60 is thus computationally efficient as compared with evaluating all terms of the F(x i d , y d ) of the kernel function f(x it y) directly, i.e. without reference to the look-up table(s) 30.
- the output of the one-class SVM message classifier 60 for each input message y is either "anomaly" or "normal". These classifications are input to an anomaly alert module 62.
- the anomaly alert module 62 issues an anomalous event alert if the detected anomalous messages meet some alert criterion, such as observing a density of anomalous messages (i.e. a count of anomalous messages in a defined time window) that exceeds a threshold.
- the issued anomalous event alert may be used in various ways. For example, if the anomalous events are traced to a particular ECU, then a warning that the ECU is compromised (either intentionally or due to some flawed firmware upgrade, malfunction, or the like that is causing the ECU to generate glitches). Such a warning may appear on the on-board computer display of the vehicle 40, or may be wirelessly transmitted to a centralized location (e.g. a monitoring center of the automobile manufacturer, who uses the information to monitor firmware upgrades) or so forth.
- a centralized location e.g
- the disclosed look-up table(s)-based SVM classifiers may be used in other applications where computational power for training is plentiful but the implemented SVM classifier is intended to execute on a more computationally limited electronic data processing device.
- the disclosed SVM classifiers can be used for anomaly detection in the context of ocean buoys, orbiting satellites, and other systems with limited computational capacity. While anomaly detection is the illustrative application, in other cases the classifier may be used to classify other received or transmitted information, such as classifying data acquired by an ocean buoy, classifying satellite telemetry information, and so forth.
- the look-up table(s) may be used to compute some, most, or all of the terms F(x i , y d ) of the SVM kernel f ⁇ xt, y) of Equation (7) or Equation (8).
- the disclosed techniques may be embodied as a non-transitory storage medium storing instructions readable and executable by a computer, (microprocessor or microcontroller of an) embedded system, or various combinations thereof.
- the non-transitory storage medium may, for example, comprise a hard disk drive, RAID or the like of a computer; an electronic, magnetic, optical, or other memory of an embedded system, or so forth.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Small-Scale Networks (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A Support Vector Machine (SVM) classifier employs a kernel f(xi, y) ∞∏Dd≃1f(xi,d,yd) or f(xi,y) ΣD≃1f(xi,d,yd) where D is the dimensionality of Support Vectors x?i#191 and vector y is a vector being classified, e.g. representing a message communicated on a Controller Area Network (CAN) bus of a vehicle. The SVM classifier comprises a microprocessor or microcontroller programmed to perform classification using the SVM classifier with at least some terms F(x
i,d,yd) of the kernel f(x
I,y) determined using a look up table. For those terms F(x
i,y) of the kernel f(x
i,y) determined using a look up table, the dimension y
d can only assume values defined by a finite set of values. In an embedded system, the microprocessor or microcontroller performs an embedded system function such as engine control, brake control, entertainment system operation, or so forth, or an embedded system function for a vehicle, ocean buoy, or satellite.
Description
FAST EFFICIENT EVALUATION OF MESSAGES ON AUTOMOTIVE
NETWORKS USING LOOK-UP TABLES
[0001] This application claims the benefit of U.S. Provisional Application No. 62/097,895 filed December 30, 2014 and titled "FAST EFFICIENT EVALUATION OF MESSAGES ON AUTOMOTIVE NETWORKS USING LOOK-UP TABLES". U.S. Provisional Application No. 62/097,895 filed December 30, 2014 is incorporated herein by reference in its entirety.
BACKGROUND
[0002] The following relates to the machine classification arts, embedded system arts, automotive network analysis arts, and related arts.
[0003] Vehicles (automobiles, trucks, watercraft, aircraft, et cetera) are increasingly utilizing embedded electronic systems. In the automotive context, embedded system units are usually referred to as Electronic Control Units (ECUs), and are commonly networked on a Controller Area Network bus (CAN bus). In a typical communication protocol, a message includes a header, typically including a message identifier followed by a set of data bytes (e.g. 8 bytes = 64 bits). In standard automotive CAN bus architectures, the message ID is 1 1 bits (2048 possible message IDs) or 29 bits (-500 million possible message IDs). Some CAN bus architectures can support both 1 1 -bit and 29-bit message IDs. Collisions are arbitrated by prioritizing the message ID with the most leading zeros; the lower priority message is then re-sent after a specified delay.
[0004] There is an expectation in the automotive industry (supported by governmental regulations in some instances) that the CAN bus architecture should be "open" in the sense that third-party manufacturers should be able to provide ECU- equipped components that can connect with an existing CAN bus. This creates a potential entry point for exploitation and challenges in providing CAN bus security against inadvertent glitches and intentional intrusion (e.g. hacking of an ECU or of the entire CAN bus network). Likewise, there are several new entry points for exploitation and pose challenges. A few examples of vulnerable entry points are the tire pressure monitoring system (TPMS), RADAR, LIDAR, camera, on-board diagnostics (e.g. OBD-II), USB, Bluetooth, WiFi enabled on-vehicle hotspots, and cellular connections.
The consequences of an unintentional glitch or intentional hack can range from very minor (e.g. the car stereo volume decreasing) to very serious (e.g. loss of vehicle braking, uncontrolled engine acceleration, or aircraft flight control interference).
BRIEF SUMMARY
[0005] In one disclosed aspect, a Support Vector Machine (SVM) classifier is disclosed which employs a kernel f{xt, y) <x
F{xi>d, yd) or a kernel f{xt, y) <x ∑d=1 F(xi , yd) where D is the dimensionality of Support Vectors xt and vector y is a vector being classified, for example y may represent a message communicated on a Controller Area Network (CAN) bus of a vehicle. The SVM classifier comprises a microprocessor or microcontroller programmed to perform classification using the SVM classifier with at least some terms F(xi d, yd) of the kernel f{xt, y) determined using a look-up table. The kernel f{xt, y) may, for example, be a Radial Basis Function (RBF) kernel. In some embodiments, for those terms F(xi d, yd) of the kernel f{xt, y) determined using a look-up table the dimension yd can only assume values defined by a finite set of values, or in more limited embodiments by a finite set of integers. In some embodiments the microprocessor or microcontroller programmed to perform classification using the SVM classifier comprises the microprocessor or microcontroller of an embedded system.
[0006] In another disclosed aspect, an embedded system comprises the microprocessor or microcontroller of the immediately preceding paragraph. The microprocessor or microcontroller may, for example, be further programmed to perform an embedded system function such as engine control, brake control, entertainment system operation, or so forth, or an embedded system function for a vehicle, ocean buoy, or satellite.
[0007] In another disclosed aspect, an SVM classifier training system comprises a computer (for example, a cloud-based computing resource, a parallel processor system, or a cluster computer) programmed to train an SVM classifier employing a kernel f{Xi, y) <x
F(x d, yd) where D is the dimensionality of Support Vectors xt and of data vectors y, and to generate a look-up
table containing the possible values for at least one term F{xi d, yd) of the kernel
[0008] In another disclosed aspect, a classification method comprises applying an SVM classifier employing a kernel f{xt,y) <x
F (xi,d>yd) or a kernel f{xt, y) <x ∑d=1 F(xi ,yd) to classify a vector y, where D is the dimensionality of Support Vectors xt, where the applying includes determining at least some terms F(xi d,yd) of the kernel f{Xi, y) using a look-up table. The applying is suitably performed by a microprocessor or microcontroller.
BRIEF DESCRIPTION OF THE DRAWINGS
[0009] FIGURE 1 diagrammatically shows a classifier training system in accord with illustrative embodiments disclosed herein.
[0010] FIGURE 2 diagrammatically illustrates a classifier in accord with illustrative embodiments disclosed herein.
DETAILED DESCRIPTION
[0011] An approach to CAN bus security in the face of this "zero knowledge" context is zero knowledge anomaly detection. In a rough conceptual sense, an anomaly is either a glitch or an intrusion (e.g. hacking) - more generally, CAN bus traffic that should not be present. In the CAN bus context, a priori knowledge of what types of messages may be transmitted by a given ECU may be limited (e.g., due to traffic introduced by aftermarket components connected with the CAN bus).
[0012] As used herein, an anomaly is defined as follows: an anomaly is a statistically significant deviation from normal CAN bus behavior as gleamed from CAN bus message traffic. "Statistically significant" means that the deviation from normal CAN bus behavior is sufficient, measureable and greater than the allowable limit. An anomaly detection system can be constructed based on statistical analysis of a training data set consisting of CAN bus traffic acquired from a vehicle known to be operating normally. For example, training data can be acquired at the automobile manufacturer facility, using a vehicle straight off the assembly line, or a pre-production model, or some other vehicle. This approach is unusual in that the training data are all "negative" samples,
that is, all of the training data are "normal" and none of the training data are anomalies. Thus, ideally, after training on such data the anomaly detection system should not generate any anomaly alerts.
[0013] In approaches disclosed herein, the anomaly detection algorithm comprises a Support Vector Machine (SVM) trained to operate as a one-class classifier outputting either a positive value (indicating an anomaly) or a negative value (indicating a normal message). (The one-class classifier can also be thought of as a binary classifier in which one "class" is "anomaly" and the other "class" is "normal message" or "not anomaly"). Each message is treated as a vector residing in a high-dimensional vector space. For example, in one CAN bus architecture a message includes (in time-order): an 1 1 -bit message ID (with an optional 18-bit extension message ID); network ID denoting the network bus the message arrived on; and eight data bytes. This can be represented by a vector having ten dimensions: one dimension for the message ID; one dimension for the network (or BUS) ID; and eight additional dimensions for the eight respective data bytes. In the SVM paradigm, the training set consists of a set of training vectors, and a sub-set of the training vectors forms the set of Support Vectors. A hyperplane is defined respective to the Support Vectors that is optimized by the SVM training to separate positive samples from negative samples while maximizing margins to the nearest training samples in the vector space. The illustrative one-class SVM classifier employs a Radial Basis Function (RBF) as the kernel for defining the hyperplane. Representing a Support Vector by x and a (message) vector to be classified by y, the RBF f(x, y) is defined as: f(x, y) = e~H(*i-yi)2 +(*2 -y2)2 +---KxD-yD)2) (1 ) where D is the number of dimensions of the vectors (e.g. D = 10 in the illustrative CAN bus message with a message and a network ID element and eight data bytes) and γ characterizes the curvature of the RBF kernel f(pc, y) . During SVM training, the Support Vectors are chosen from the training sub-set by the SVM training algorithm so as to provide positive/negative sample segregation with maximized margins, typically subject to additional constraints defined by tuning parameters.
[0014] In a typical SVM approach, the tuning parameters include the already-mentioned y parameter, which controls the shape or curvature of the separating hyperplane. A larger value of y increases hyperplane curvature and also tends to increase the number of Support Vectors chosen during the SVM training. Another tuning parameter, v, defines a lower bound on the number of Support Vectors, and also presents an upper bound on the number of training samples that may be erroneously classified (this error count can be reduced by forced adding of Support Vectors). A tuning parameter ε defines a penalty for data points inside the margin (i.e. too close to the hyperplane), and may be viewed as controlling granularity of the hyperplane.
[0015] Conventionally, SVM training is performed as follows. The tuning parameters y, v, ε (or additional or other tuning parameters depending upon the particular SVM model being employed) are chosen, and SVM training is performed to optimize selection of the Support Vectors so as to optimally segregate positive from negative samples while satisfying constraints defined by the tuning parameters y, , ε. In the conventional approach, the user may then choose to adjust the tuning parameters y, v, ε and repeat the training, so as to tune the SVM classifier by trial-and-error. After the training phase is complete, the trained SVM classifier is applied to classify a new message represented by the message vector y according to:
where / is given in Expression (1 ), the tuning parameters y, , ε from the training are used, N is the number of support vectors chosen by the SVM training, and the vectors xlt ... , xN are the Support Vectors chosen by the training. The message y is classified as anomalous if the sum in Expression (2) is greater than the threshold p because it is too far away from the trained hyperplane; otherwise, the message is classified as normal. Since the parameter ε relates to the margin around the trained hyperplane, in some embodiments the threshold p = Νε is used in Expression (2).
[0016] For CAN network security, the trained SVM classifier is implemented on an ECU (either as an additional function of an existing ECU, or as a separate ECU
dedicated to providing CAN network security). The computing power of the ECU is limited, as it can usually be, at most, a standalone multi-core CPU system. On the other hand, the SVM training can be performed offline with massive computing power brought to bear, for example using a cloud computing system or other parallel computing system that may include dozens or more CPUs.
[0017] A problem can arise because the ECU may have insufficient computing capacity to execute the trained one-class SVM classifier. Expanding Expression (2) by inserting Equation (1 ) yields: e -Y{(xsvi,i-yi)2 +--+(xsvi,D-yD)2) _|_ |_ e -Y{(xsvN,i-yi†+--+(xsvN,D-yD)2) > p ^
where in Expression (3) the subscripts SV1, ... , SVN denote the N Support Vectors, so that, for example, xSVi,2 denotes the second dimension of Support Vector xl Existing commercial production automobiles have CAN bus traffic density on the order of 3,000-5,000 messages per second, each of which needs to be classified by the trained SVM classifier in accordance with Expression (3). This level of computation requires roughly 200M FLOPS, which is not achievable with presently existing automotive class hardware specifications.
[0018] To reduce the number of computations, the computational complexity of Expression (1 ), the exponential for a single Support Vector (denoted without loss of generality as Support Vector xt) can be greatly reduced as follows. First, the e ~Y term can be pulled out in accord with the rule eab = (e b)a so as to yield:
where xi denotes the value of the first dimension of Support Vector xu and xi>2 denotes the value of the second dimension of Support Vector x and so forth. Then the rule e a+b = ea ■ e b can be applied to yield:
f(x y) = . . e {xi,D-yo) (5)
For an illustrative ten-dimensional message (D = 10) having a message ID, an optional extension message ID (or network ID), and eight data bytes, the dimensions 3 ... 10 correspond to the eight data bytes. Each of these dimensions 3, ... ,10 can thus assume only integer values in the range 0, ... ,255. Moreover, the values ...,¾il0 are fixed values for the Support Vector xt. Thus, for example, e^Xi^~y^ can only assume 256 possible values corresponding to the respective possible data byte values y3 = 0, ... ,255. Thus, Expression (5) reduces to: f{x y) = (e (¾-yi)2 - e (¾-y2)2 . T i> 3j y3) . r(i, 4, y4)■ T(i, S, y5)■ T(i, 6, y6)
v (6)
■ T(i, 7, y7)■ T(i, 8, y8)■ T(i, 9, yg)■ T(i, 10, y10)J where, for example, T(i, 3, y3) is the look-up table entry for Support Vector x third dimension, for value y3. Thus, all eight message byte values e^3~y3^ , e(* o-yio) can be stored in a table with dimensions (1 ... N ; 3 ... 10 ; 0 ... 255) . The inventors estimate that N = 520 should be a sufficient number of Support Vectors for most automotive CAN bus security monitoring applications, so this table is of reasonable size.
[0019] The message and network ID dimensions, that is, the terms e^i l_3/1) and e {xi,2 -y2) j are more complex, and whether a look-up table approach is viable depends upon the particular message ID format, especially the number of bits. Assuming the first dimension is a message ID of fixed 1 1 bits, corresponding to 2048 possible message IDs, a look-up table for the message ID having dimensions (1 ... N ; 0 ... 2047) is feasible. If a second ID field (network ID) is provided, a tabular approach for this would employ a table of dimensions (1 ... N ; 0 ... 262,143), which may be feasible depending upon storage capacity. On the other hand, if the CAN network architecture employs a single message ID that is 29 bits, this corresponds to 536,870,912 possible message IDs, so that a table approach may not be feasible. However, if a small sub-set of the
536,870,912 possible message IDs are actually used (as determined, for example, based on training data), then it may be possible to use a look-up table such as one of dimensions (1 ... N ; 1 ... S ; 1 ... S) where S denotes the actually used message ID values.
[0020] In sum, for a standard CAN bus architecture, all but (at most) one dimension of Expression (6) can be handled using a look-up table, or two or three look-up tables if additional table(s) are needed for the message ID dimension(s). Evaluation of Expression (6) is reduced in complexity by about an order of magnitude as compared with evaluation of Expression (3), down to about 20M FLOPS compared with 200M FLOPS for Expression (3). The cost of this approach is higher storage space for the trained one-class SVM classifier due to the need to store the look-up tables. In one design, the look-up tables are estimated to require about 10 megabytes of storage space, which is within the storage capacity of existing automotive class embedded systems.
[0021] With reference to FIGURES 1 and 2, an illustrative embodiment is shown, including the training system (FIGURE 1 ) and the embedded system (FIGURE 2).
[0022] With particular reference to FIGURE 1 , the one-class SVM classifier training is suitably performed by a high-capacity computing system, such as an illustrative server computer 10, which may be cloud-based (i.e. a cloud computing resource), a parallel processor system, cluster computer, or so forth. The training is performed by a one-class SVM classifier trainer 12 using a training set of CAN bus traffic 14 represented as vectors as described herein (e.g. one or two dimensions for message/network ID information, and eight dimensions for the data bytes) and SVM tuning parameters 16. The SVM trainer 12 outputs a set of Support Vectors 20 chosen from the training data 14. The training may also entail optimizing the SVM tuning parameters 16, either by trial-and-error (e.g. a human analyst re-runs the SVM trainer 12 using different values for the SVM tuning parameters 16, or an automated approach such as a grid search is used to optimize the SVM tuning parameters 16). The illustrative SVM that is trained by the SVM trainer 12 employs a Radial Basis Function (RBF) as the kernel. More generally, the disclosed approaches are applicable to any SVM that employs a kernel:
where Expression (7) is written for Support Vector x D denotes the number of dimensions of the vector, the symbol "oc" is conventionally used to denote a proportional relationship, and the right-most equality is the illustrative RBF earlier set forth using alternative notation as Expression (1 ). The disclosed approaches are applicable to any SVM that employs a kernel:
With continuing reference to FIGURE 1 , after the N support vectors 20 are determined by the training system 12, a look-up table generator 22 is applied to exhaustively compute all possible values for at least those terms F(xi ,yd) corresponding to data bytes. More generally, the look-up table generator 22 can be applied to exhaustively compute all possible values any term F(xi , yd) for which xi and yd can only assume values defined by a finite (i.e. closed) set of values, such as the set of integers {0, ... ,255} for an 8-bit data byte, or the set of integers {0, ... ,2047} for an 1 1 -bit message ID dimension. The output of the look-up table generator 22 is a set of one or more look-up tables 30. It should be noted that while the look-up table generator 22 may require substantial computing resources, those computing resources are available in the form of the illustrative server computer 10 (which may be cloud-based, a parallel processor system, cluster computer, or so forth). Thus, the look-up table generator 22 is not limited by the limited computing power or resources of any automotive-class ECU.
[0023] With particular reference to FIGURE 2, a vehicle 40, such as an illustrative automobile, or a truck, watercraft (boat, ship, et cetera), aircraft (airplane, blimp, helicopter, et cetera) includes at least one (and typically many) embedded system 42 comprising an electronic control unit (ECU) 44 (shown diagrammatically in the inset). The vehicle 40 also includes at least one (and possibly two or more) Controller Area Network bus (CAN bus) 46 via which the embedded systems can intercommunicate. To
this end, the illustrative ECU 44 includes a transceiver 48 that transmits and receives messages via the CAN bus 46. The ECU 44 comprises a microprocessor or microcontroller and ancillary electronics (e.g. memory, the illustrative transceiver 48, or so forth) programmed to provide the desired functionality of the embedded system 42. This functionality is represented in the inset diagram representing the ECU 44 as an ECU functional module 50. For example, the ECU functional module 50 may provide engine control, brake control, entertainment system operation, or so forth.
[0024] To provide CAN network security functionality, the ECU 44 is further programmed to implement a one-class SVM message classifier 60 that classifies each message passing through the transceiver 48 as either "normal" or "anomaly". Preferably, both sent and received messages are classified, although it is contemplated to only classify received messages under the assumption that the embedded system 42 is secure, and/or under the assumption that any anomalous transmission events generated by the embedded system 42 will be detected by other ECUs on the CAN bus 46 also running instances of the one-class SVM message classifier 60. In this regard, it should be noted that under typical CAN network architectures, every ECU on the CAN network receives every message, not just those messages relevant to the ECU. Thus, the SVM message classifier 60 can classify every message on the CAN bus 46 as either "normal" or "anomalous". The one-class SVM message classifier 60 employs the trained SVM produced by the system of FIGURE 1 , including using the N Support Vectors 20, the (optionally optimized) SVM tuning parameters 16, and the look-up table(s) 30. To classify a message represented by a vector y, Expression (2) is evaluated, using the look-up table(s) 30 to evaluate at least some terms F(xi , yd) of the kernel function f(xit y) for each Support Vector xt. The one-class SVM message classifier 60 is thus computationally efficient as compared with evaluating all terms of the F(xi d, yd) of the kernel function f(xit y) directly, i.e. without reference to the look-up table(s) 30.
[0025] The output of the one-class SVM message classifier 60 for each input message y is either "anomaly" or "normal". These classifications are input to an anomaly alert module 62. The anomaly alert module 62 issues an anomalous event alert if the detected anomalous messages meet some alert criterion, such as observing
a density of anomalous messages (i.e. a count of anomalous messages in a defined time window) that exceeds a threshold. The issued anomalous event alert may be used in various ways. For example, if the anomalous events are traced to a particular ECU, then a warning that the ECU is compromised (either intentionally or due to some flawed firmware upgrade, malfunction, or the like that is causing the ECU to generate glitches). Such a warning may appear on the on-board computer display of the vehicle 40, or may be wirelessly transmitted to a centralized location (e.g. a monitoring center of the automobile manufacturer, who uses the information to monitor firmware upgrades) or so forth.
[0026] While described in the context of vehicle embedded system security monitoring, it will be appreciated that the disclosed look-up table(s)-based SVM classifiers may be used in other applications where computational power for training is plentiful but the implemented SVM classifier is intended to execute on a more computationally limited electronic data processing device. For example, the disclosed SVM classifiers can be used for anomaly detection in the context of ocean buoys, orbiting satellites, and other systems with limited computational capacity. While anomaly detection is the illustrative application, in other cases the classifier may be used to classify other received or transmitted information, such as classifying data acquired by an ocean buoy, classifying satellite telemetry information, and so forth. As previously noted, the look-up table(s) may be used to compute some, most, or all of the terms F(xi , yd) of the SVM kernel f{xt, y) of Equation (7) or Equation (8).
[0027] It will further be appreciated that the disclosed techniques may be embodied as a non-transitory storage medium storing instructions readable and executable by a computer, (microprocessor or microcontroller of an) embedded system, or various combinations thereof. The non-transitory storage medium may, for example, comprise a hard disk drive, RAID or the like of a computer; an electronic, magnetic, optical, or other memory of an embedded system, or so forth.
[0028] The preferred embodiments have been illustrated and described. Obviously, modifications and alterations will occur to others upon reading and understanding the preceding detailed description. It is intended that the invention be construed as
including all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims
1 . A Support Vector Machine (SVM) classifier employing a kernel f(Xi, y) <x Y[d=1 F(x d,yd) or a kernel f{x y) <x∑d=1 F(xi ,yd) where D is the dimensionality of Support Vectors xt and vector y is a vector being classified, the SVM classifier comprising a microprocessor or microcontroller programmed to perform classification using the SVM classifier with at least some terms F(xi d, yd) of the kernel f{xt, y) determined using a look-up table.
2. The SVM classifier of claim 1 wherein y represents a message communicated on a Controller Area Network (CAN) bus of a vehicle.
3. The SVM classifier of any one of claims 1 -2 wherein the kernel f{xt,y) is a Radial Basis Function (RBF) kernel.
4. The SVM classifier of any one of claims 1 -3 wherein for those terms p{xi,d> yd) of the kernel f{xt,y) determined using a look-up table, the dimension yd can only assume values defined by a finite set of values.
5. The SVM classifier of any one of claims 1 -3 wherein for those terms p{xi,d> yd) of the kernel f{xt,y) determined using a look-up table, the dimension yd can only assume values defined by a finite set of integers.
6. The SVM classifier of any one of claims 1 -5 wherein the microprocessor or microcontroller programmed to perform classification using the SVM classifier comprises the microprocessor or microcontroller of an embedded system.
7. An embedded system comprising the microprocessor or microcontroller of any one of claims 1 -5.
8. The embedded system of claim 7 wherein the microprocessor or microcontroller is further programmed to perform an embedded system function such as engine control, brake control, entertainment system operation, or so forth.
9. A The embedded system of claim 7 wherein the microprocessor or microcontroller is further programmed to perform an embedded system function for a vehicle, ocean buoy, or satellite.
10. A Support Vector Machine (SVM) classifier training system comprising a computer programmed to:
train an SVM classifier employing a kernel f {xt, y) <x
F(xi , yd) or a kernel f Xi, y <x∑d=1 F(xi , yd) where D is the dimensionality of Support Vectors xt and of data vectors y; and
generate a look-up table containing the possible values for at least one term F(xi,d, yd) of the kernel f {x y) .
1 1 . The SVM classifier training system of claim 10 wherein the computer comprises a cloud-based computing resource, a parallel processor system, or a cluster computer.
12. The SVM classifier training system of any one of claims 10-1 1 wherein the kernel f {xt, y) is a Radial Basis Function (RBF) kernel.
13. The SVM classifier training system of any one of claims 10-12 wherein, for the at least one term F(xi d,yd) of the kernel f {xt, y) for which a look-up table is generated, the dimension yd can only assume values defined by a finite set of values.
14. The SVM classifier training system of any one of claims 10-12 wherein, for the at least one term F(xi d,yd) of the kernel f {xt, y) for which a look-up table is
generated, the dimension yd can only assume values defined by a finite set of integers.
15. A system comprising:
a Support Vector Machine (SVM) classifier or embedded system as set forth in any one of claims 1 -9; and
an SVM classifier training system as set forth in any one of claims 10-14.
16. A classification method comprising:
applying a Support Vector Machine (SVM) classifier employing a kernel f(xi, y)
F(xiid,yd) to classify a vector y, where D is the dimensionality of Support Vectors ΧΪ,
wherein the applying includes determining at least some terms F(xi d, yd) of the kernel f Xi, y using a look-up table; and
wherein the applying is performed by a microprocessor or microcontroller.
17. A method of generating a Support Vector Machine (SVM) classifier comprising:
training an SVM classifier employing a kernel f{xt, y) X-
F{xi>d, yd) or a kernel f Xi, y
F{xi>d, yd) where D is the dimensionality of Support Vectors xt and of data vectors y; and
generating a look-up table containing the possible values for at least one term F(xiid, yd) of the kernel f{xi, y) which term F(xi d,yd) has a finite number of possible values.
18. A method comprising:
(1 ) training a Support Vector Machine (SVM) classifier employing a kernel f{Xi, y) <x
F{xiid,yd) where D is the dimensionality of Support Vectors xt and of data vectors y;
(2) generating a look-up table containing the possible values for at least one term F(xi d, yd) of the kernel f{xt,y) which term F(xi d, yd) has a finite number of possible values; and
(3) applying the trained SVM classifier to classify a vector y, wherein the applying includes determining at least some terms F(xi d, yd) of the kernel f{xt,y) using the generated look-up table.
19. The method of claim 18 wherein operations (1 ) and (2) are performed by a computer and operation (3) is performed by a microprocessor or microcontroller of an embedded system other than the computer.
20. The method of claim 18 wherein operations (1 ) and (2) are performed by a computer and operation (3) is performed by an embedded system of a vehicle.
21 . A non-transitory storage medium storing instructions readable and executable by at least one computer and/or embedded system to perform a method as set forth in any one of claims 16-21 .
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US201462097895P | 2014-12-30 | 2014-12-30 | |
| US62/097,895 | 2014-12-30 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2016108962A1 true WO2016108962A1 (en) | 2016-07-07 |
Family
ID=56164605
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/US2015/050615 Ceased WO2016108962A1 (en) | 2014-12-30 | 2015-09-17 | Fast efficient evaluation of messages on automotive networks using look-up tables |
Country Status (2)
| Country | Link |
|---|---|
| US (1) | US10832159B2 (en) |
| WO (1) | WO2016108962A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108875244A (en) * | 2018-06-29 | 2018-11-23 | 北京航空航天大学 | A kind of orbit prediction accuracy improvements method based on random forest |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6846991B2 (en) * | 2016-07-05 | 2021-03-24 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America | Anomaly detection electronic control unit, in-vehicle network system and anomaly detection method |
| US10484425B2 (en) | 2017-09-28 | 2019-11-19 | The Mitre Corporation | Controller area network frame override |
| US12380746B2 (en) | 2018-09-30 | 2025-08-05 | Strong Force Tp Portfolio 2022, Llc | Digital twin systems and methods for transportation systems |
| WO2020069517A2 (en) | 2018-09-30 | 2020-04-02 | Strong Force Intellectual Capital, Llc | Intelligent transportation systems |
| US12169987B2 (en) * | 2018-09-30 | 2024-12-17 | Strong Force Tp Portfolio 2022, Llc | Intelligent transportation systems including digital twin interface for a passenger vehicle |
| US11631151B2 (en) | 2018-09-30 | 2023-04-18 | Strong Force Tp Portfolio 2022, Llc | Intelligent transportation systems |
| US11700270B2 (en) * | 2019-02-19 | 2023-07-11 | The Aerospace Corporation | Systems and methods for detecting a communication anomaly |
| CN110427960B (en) * | 2019-06-18 | 2023-04-25 | 南京大学 | A Reconfigurable Multi-Classification Support Vector Machine System |
| WO2021016307A1 (en) | 2019-07-25 | 2021-01-28 | Battelle Memorial Institute | Can bus protection systems and methods |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8280828B2 (en) * | 2009-06-12 | 2012-10-02 | Xerox Corporation | Fast and efficient nonlinear classifier generated from a trained linear classifier |
| US20140297707A1 (en) * | 2013-04-01 | 2014-10-02 | Canon Kabushiki Kaisha | Information processing apparatus, information processing method and storage medium |
-
2015
- 2015-09-17 US US14/857,073 patent/US10832159B2/en active Active
- 2015-09-17 WO PCT/US2015/050615 patent/WO2016108962A1/en not_active Ceased
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8280828B2 (en) * | 2009-06-12 | 2012-10-02 | Xerox Corporation | Fast and efficient nonlinear classifier generated from a trained linear classifier |
| US20140297707A1 (en) * | 2013-04-01 | 2014-10-02 | Canon Kabushiki Kaisha | Information processing apparatus, information processing method and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| THEISSIER.: "Detecting anomalies in multivariate time series from automotive systems.", vol. 104, no. 109, 9 December 2013 (2013-12-09), pages 37 - 38, 56, 72, Retrieved from the Internet <URL:bura.brunel.ac.uk/bitstream/2438/7902/1/FulltextThesis.pdf> [retrieved on 20151110] * |
| THEISSLER: "An Anomaly Detection Approach to Detect Unexpected Faults in Recordings from Test Drives.", WORLD ACADEMY OF SCIENCE , ENGINEERING AND TECHNOLOGY INTERNATIONAL JOURNAL OF COMPUTER, ELECTRICAL, AUTOMATION, CONTROL AND INFORMATION ENGINEERING, vol. 7, no. 7, 2013, pages 400, Retrieved from the Internet <URL:http://www.it-designers-gruppe.de/uploads/media/Theissler_Dear-Anomaly_detection_approach-ICVES-reprint_02-20130912.pdf> [retrieved on 20151110] * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108875244A (en) * | 2018-06-29 | 2018-11-23 | 北京航空航天大学 | A kind of orbit prediction accuracy improvements method based on random forest |
| CN108875244B (en) * | 2018-06-29 | 2020-05-12 | 北京航空航天大学 | A method for improving orbit prediction accuracy based on random forest |
Also Published As
| Publication number | Publication date |
|---|---|
| US10832159B2 (en) | 2020-11-10 |
| US20160189056A1 (en) | 2016-06-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12118080B2 (en) | Anomaly detection for vehicular networks for intrusion and malfunction detection | |
| US10832159B2 (en) | Fast efficient evaluation of messages on automotive networks using look-up tables | |
| US10846166B2 (en) | Temporal anomaly detection on automotive networks | |
| US11323876B2 (en) | Method for adaptively adjusting security level of V2X communication message and apparatus therefor | |
| EP3553712B1 (en) | Information processing device, information processing method, and program | |
| US11595431B2 (en) | Information processing apparatus, moving apparatus, and method | |
| US20160308887A1 (en) | In-vehicle network intrusion detection system and method for controlling the same | |
| US11222213B2 (en) | Vehicle behavior detection system | |
| US11984033B2 (en) | Artificial intelligence-based persistence of vehicle black box data | |
| Martín-Lammerding et al. | An ontology-based system to avoid UAS flight conflicts and collisions in dense traffic scenarios | |
| WO2022023861A1 (en) | Fair anomaly detection and localization | |
| CN116894225A (en) | Driving behavior abnormality analysis method and its device, equipment and media | |
| EP3972217B1 (en) | Ml-based voltage fingerprinting for ground truth and controlled message error for message and ecu mapping for can bus | |
| CN112712608B (en) | Systems and methods for collecting performance data from vehicles | |
| EP4414185B1 (en) | Systems and methods for detecting and removing erroneous tire pressure values | |
| US12291209B2 (en) | State driven micro-value architecture | |
| CN118214644A (en) | Method and system for incremental centroid clustering | |
| US20200193737A1 (en) | System and method to automatically determine a driving mode of a vehicle | |
| US12052641B2 (en) | System and method for real-time extraction and processing of video data from vehicles | |
| US20240157934A1 (en) | Systems and methods for generating vehicle safety scores and predicting vehicle collision probabilities | |
| US20250131776A1 (en) | Data retention using nearby resources | |
| CN119763067A (en) | Traffic light recognition model training method, traffic light recognition method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 15875845 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 15875845 Country of ref document: EP Kind code of ref document: A1 |

