CN112363859A - Method and device for determining abnormality determination threshold - Google Patents

Method and device for determining abnormality determination threshold Download PDF

Info

Publication number
CN112363859A
CN112363859A CN202011245175.0A CN202011245175A CN112363859A CN 112363859 A CN112363859 A CN 112363859A CN 202011245175 A CN202011245175 A CN 202011245175A CN 112363859 A CN112363859 A CN 112363859A
Authority
CN
China
Prior art keywords
anomaly
abnormal
target object
value
normal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011245175.0A
Other languages
Chinese (zh)
Inventor
刘波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Life Insurance Company of China Ltd
Original Assignee
Ping An Life Insurance Company of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Life Insurance Company of China Ltd filed Critical Ping An Life Insurance Company of China Ltd
Priority to CN202011245175.0A priority Critical patent/CN112363859A/en
Publication of CN112363859A publication Critical patent/CN112363859A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/079Root cause analysis, i.e. error or fault diagnosis
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00Computing arrangements based on specific mathematical models
    • G06N7/01Probabilistic graphical models, e.g. probabilistic networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Optimization (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Pure & Applied Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Algebra (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Quality & Reliability (AREA)
  • Testing And Monitoring For Control Systems (AREA)

Abstract

The embodiment of the application provides a method and a device for determining an abnormality determination threshold, wherein the method comprises the following steps: performing anomaly measurement operation on the target object to obtain an anomaly measurement matrix; calculating the data distribution characteristics of the anomaly metric matrix; and determining an abnormality judgment threshold value of the target object based on the data distribution characteristics of the abnormality metric matrix. According to the method and the device, the abnormal measurement matrix of the target object is calculated, and the abnormal judgment threshold is calculated according to the data distribution characteristics of the abnormal measurement matrix, so that the problem of fuzzy judgment of the threshold limit in the abnormal object judgment process is solved, the abnormal threshold limit of the target object is determined, and the accuracy of abnormal judgment is improved.

Description

Method and device for determining abnormality determination threshold
Technical Field
The present disclosure relates to the field of anomaly analysis and detection, and in particular, to a method and an apparatus for determining an anomaly determination threshold.
Background
With the development of information technology, the demand for intelligent detection is also rapidly increasing, and anomaly analysis becomes a main task of intelligent detection. When anomaly analysis is performed, anomaly measurement of a target object can be realized by methods such as a distance-based algorithm and a pattern-based algorithm, but threshold division for determining whether a detected object is abnormal in the existing method does not meet the requirement of anomaly determination, for example, when the anomaly measurement is performed by using an isolated forest algorithm based on the pattern-based anomaly measurement algorithm, an anomaly measurement value in a value range of [0,1] can be obtained, and the anomaly judgment standard is as follows: judging the abnormal measurement value to be abnormal when the result of the abnormal measurement value is close to 1, and judging the abnormal measurement value to be normal when the result of the abnormal measurement value is close to 0; as another example, in the distance-based abnormal metric algorithm, when mahalanobis distance is used for abnormal analysis, the obtained abnormal metric value has a large difference from the actual abnormal metric value, and an accurate partition threshold cannot be directly obtained. At present, the phenomenon that an abnormality judgment threshold value is unclear often exists in an abnormality analysis method, so that an abnormality judgment error occurs.
Disclosure of Invention
The embodiment of the application provides a method and a device for determining an abnormality threshold, which can be used for determining the boundary of the abnormality threshold of a target object and improving the accuracy of abnormality determination.
In a first aspect, an embodiment of the present application provides a method for determining an anomaly determination threshold, where the method includes:
performing anomaly measurement operation on the target object to obtain an anomaly measurement matrix;
calculating the data distribution characteristics of the anomaly metric matrix;
and determining an abnormality judgment threshold value of the target object based on the data distribution characteristics of the abnormality metric matrix.
Optionally, the method further includes: acquiring an input factor of the target object, wherein the input factor is at least one parameter required for judging the target object to be abnormal;
the performing an anomaly metric operation on the target object to obtain an anomaly metric matrix includes:
calculating an observation value of the input factor, and dividing the input factor into a normal input factor and an abnormal input factor based on the observation value; and respectively carrying out anomaly measurement operation on the normal input factors and the anomaly input factors to obtain a normal anomaly measurement matrix and an anomaly measurement matrix.
Optionally, the calculating the data distribution characteristic of the anomaly metric matrix includes:
and calculating the mean value and the standard deviation of the normal abnormal measurement matrix and the mean value and the standard deviation of the abnormal measurement matrix based on the observed value of the input factor to obtain a normal mean value, a normal standard deviation, an abnormal mean value and an abnormal standard deviation.
Optionally, the determining an anomaly determination threshold of the target object based on the data distribution characteristics of the anomaly metric matrix includes:
substituting the normal class mean, the normal class standard deviation, the abnormal class mean and the abnormal class standard deviation into a Bayesian minimum error formula p (Tw |)u)p(wu)=p(T|wc)p(wc) Calculating the abnormality determination threshold;
wherein, the p (w)u) Is the prior probability of the normal class input factor, p (w)c) The prior probability of the input factor of the anomaly class, p (Tw |)u) The p (Tw) is a conditional density function of the normal class anomaly metric matrixc) Measure the conditional density function of the matrix for exception-like exceptions, the p (Tw |)u) And said p (T | w)c) The abnormal judgment threshold value is T.
Optionally, the method further includes: determining an anomaly analysis result of the target object based on the anomaly determination threshold, the anomaly analysis result including normality and anomaly.
Optionally, when the measured value of the input factor of the target object is greater than or equal to the abnormality determination threshold, the abnormality analysis result of the target object is normal; and when the measured value of the input factor of the target object is smaller than the abnormity judgment threshold value, judging that the abnormity analysis result of the target object is abnormal.
Optionally, the method further includes: comparing the actual results of the target objects with the corresponding abnormal analysis results respectively to obtain the false recognition rate; and adjusting the abnormity judgment threshold value based on the error recognition rate.
In a second aspect, an embodiment of the present application provides an apparatus for determining an abnormality determination threshold, where the apparatus includes:
the operation unit is used for performing anomaly measurement operation on the target object to obtain an anomaly measurement matrix;
the operation unit is further used for calculating a mean value and a standard deviation of the abnormal measurement matrix;
a determination unit configured to determine an abnormality determination threshold of the target object based on the mean and the standard deviation.
Optionally, the apparatus further comprises: an acquisition unit, configured to acquire an input factor of the target object, where the input factor is at least one parameter required for determining that the target object is abnormal;
in terms of performing an anomaly metric operation on a target object to obtain an anomaly metric matrix, the operation unit is specifically configured to:
calculating an observation value of the input factor, and dividing the input factor into a normal input factor and an abnormal input factor based on the observation value; and respectively carrying out anomaly measurement operation on the normal input factors and the anomaly input factors to obtain a normal anomaly measurement matrix and an anomaly measurement matrix.
Optionally, in terms of calculating the data distribution characteristic of the abnormal metric matrix, the operation unit is specifically configured to:
and calculating the mean value and the standard deviation of the normal abnormal measurement matrix and the mean value and the standard deviation of the abnormal measurement matrix based on the observed value of the input factor to obtain a normal mean value, a normal standard deviation, an abnormal mean value and an abnormal standard deviation.
Optionally, the determining unit is specifically configured to: substituting the normal class mean, the normal class standard deviation, the abnormal class mean and the abnormal class standard deviation into a Bayesian minimum error formula p (Tw |)u)p(wu)=p(T|wc)p(wc) Calculating the abnormality determination threshold;
wherein, the p (w)u) Is the prior probability of the normal class input factor, p (w)c) The prior probability of the input factor of the anomaly class, p (Tw |)u) The p (Tw) is a conditional density function of the normal class anomaly metric matrixc) Conditional density function for measuring matrix for exception-like exceptionsNumber, said p (T | w)u) And said p (T | w)c) The abnormal judgment threshold value is T.
Optionally, the determining unit is further configured to: determining an anomaly analysis result of the target object based on the anomaly determination threshold, the anomaly analysis result including normality and anomaly.
Optionally, when the measured value of the input factor of the target object is greater than or equal to the abnormality determination threshold, the abnormality analysis result of the target object is normal; and when the measured value of the input factor of the target object is smaller than the abnormity judgment threshold value, judging that the abnormity analysis result of the target object is abnormal.
Optionally, the device further includes a comparing unit and an adjusting unit, where the comparing unit is configured to compare actual results of the plurality of target objects with the corresponding abnormal analysis results, respectively, to obtain the false identification rate;
the adjusting unit is used for adjusting the abnormity judgment threshold value based on the error recognition rate.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps described in any of the methods in the first aspect of the embodiment of the present application.
In a fourth aspect, the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program for electronic data exchange, where the computer program makes a computer perform part or all of the steps described in any one of the methods in the first aspect of the present application.
In a fifth aspect, the present application provides a computer program product, where the computer program product includes a non-transitory computer-readable storage medium storing a computer program, where the computer program is operable to cause a computer to perform some or all of the steps described in any of the methods of the first aspect of the embodiments of the present application. The computer program product may be a software installation package.
The embodiment of the application provides a method and a device for determining an anomaly determination threshold, wherein an anomaly measurement matrix is obtained by performing anomaly measurement operation on a target object; calculating the data distribution characteristics of the anomaly metric matrix; and determining an abnormality judgment threshold value of the target object based on the data distribution characteristics of the abnormality metric matrix. According to the method and the device, the abnormal measurement matrix of the target object is calculated, and the abnormal judgment threshold is calculated according to the data distribution characteristics of the abnormal measurement matrix, so that the problem of fuzzy judgment of the threshold limit in the abnormal object judgment process is solved, the abnormal threshold limit of the target object is determined, and the accuracy of abnormal judgment is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic flowchart of a method for determining an anomaly determination threshold according to an embodiment of the present application;
fig. 2 is a schematic flowchart of another method for determining an abnormality determination threshold according to an embodiment of the present application;
fig. 3a is a block diagram of functional units of an abnormality determination threshold determination apparatus according to an embodiment of the present application;
fig. 3b is a block diagram of functional units of another apparatus for determining an abnormality determination threshold according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and successfully with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
In this embodiment, the electronic device related to this embodiment may include various handheld devices, desktop computers, vehicle-mounted devices, wearable devices (smart watches, smart bracelets, wireless headsets, augmented reality/virtual reality devices, smart glasses), computing devices or other processing devices connected to wireless modems, and various forms of User Equipment (UE), Mobile Stations (MS), terminal devices (terminal device), and the like, and the electronic device may also be a server.
In order to better understand the scheme of the embodiments of the present application, the following first introduces the related terms and concepts that may be involved in the embodiments of the present application.
The Bayes decision is that under incomplete information, subjective probability estimation is carried out on partially unknown states, then the Bayes formula is used for correcting the occurrence probability, and finally the expectation value and the correction probability are used for making the optimal decision. The Bayesian decision theory method is a basic method in statistical model decision, and the basic idea is as follows: for any vector observed in the feature space, x ═ x (x)1,x2,…,xd)TThe class-conditional probability density p (x | w) is knowni) i-1, 2 … … c and the prior probability p (w) of the class statei) Obtaining the posterior probability of the state by using a Bayesian formula
Figure BDA0002769757760000051
According to basic decision rules
Figure BDA0002769757760000052
And attributing x to the class decision classification with the maximum posterior probability. And a decision method for selecting the optimal class mark by combining the false judgment loss under the condition that all relevant probabilities are known.
When anomaly analysis is performed, anomaly measurement of a target object can be realized by methods such as a distance-based algorithm and a pattern-based algorithm, but threshold division for determining whether the target object is abnormal in the existing method does not meet the requirement of anomaly determination, for example, when anomaly measurement is performed by applying an isolated forest algorithm based on a pattern-based anomaly measurement algorithm, an anomaly measurement value in a value range of [0,1] can be obtained, and the anomaly judgment standard is as follows: judging the abnormal measurement value to be abnormal when the result of the abnormal measurement value is close to 1, and judging the abnormal measurement value to be normal when the result of the abnormal measurement value is close to 0; as another example, in the distance-based abnormal metric algorithm, when mahalanobis distance is used for abnormal analysis, the obtained abnormal metric value has a large difference from the actual abnormal metric value, and an accurate partition threshold cannot be directly obtained. At present, the phenomenon that an abnormality judgment threshold value is unclear often exists in an abnormality analysis method, so that an abnormality judgment error occurs.
In order to solve the above problems, the present application provides a method for determining an anomaly determination threshold, the method includes performing anomaly metric operation on a target object to obtain an anomaly metric matrix; calculating the data distribution characteristics of the anomaly metric matrix; and determining an abnormality judgment threshold value of the target object based on the data distribution characteristics of the abnormality metric matrix. Specifically, the target object abnormal measurement matrix can be calculated, and the abnormal judgment threshold value is calculated according to the data distribution characteristics of the abnormal measurement matrix, so that the problem of fuzzy threshold value limit judgment in the abnormal object judgment process is solved, the abnormal threshold value limit of the target object is clarified, and the accuracy of abnormal judgment is improved.
The present application will be described in detail with reference to specific examples.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a method for determining an anomaly determination threshold according to an embodiment of the present application. As shown in fig. 1, the method comprises the steps of:
and S110, carrying out anomaly measurement operation on the target object to obtain an anomaly measurement matrix.
The target object may be a target of anomaly analysis, the anomaly analysis may be applied to network data, platform data, device security, device usage state, object behavior, object characteristics, and the like, application scenarios of the anomaly analysis are different, the target object is also different, and the embodiment of the present application does not limit the application scenarios. Specifically, when analyzing whether the bank account of the user is abnormal, the target object may be the bank account of the user; when the quality of the bulb is detected to be qualified, the target object is the bulb; when monitoring whether the network data is abnormal, the target object may be an electronic device, an Application platform, an Application program, such as a certain mobile phone, a certain Application software (APP), a certain website, and the like.
In one possible example, the method further comprises: and acquiring an input factor of the target object, wherein the input factor is at least one parameter required for judging the target object to be abnormal.
When performing anomaly analysis, relevant parameters of a target object need to be collected to judge whether the target object is abnormal or not. The input factors of different target objects are different, for example, when performing anomaly analysis on network data, the target object is an electronic device, and the input factor of the target object may be log data in the electronic device, where the log data is from security and hardware in the electronic device, and includes a flow log, an operation log, a security audit log, an alarm log, and so on. For example, in face recognition, the target object is a face image, and the input factors may be facial features including face contour points, eyebrow contour points, lip contour points, eye pupils, nose tip nostrils, and the like.
Optionally, in the step 110, performing an anomaly metric operation on the target object to obtain an anomaly metric matrix, which may include the following steps:
11. and calculating the observed value of the input factor.
The observed value of the input factor is a probability value of the input factor, which is between [0 and 1] and is used for representing the probability that the value of the input factor is abnormal. Specifically, a model can be trained in advance through a known input factor and an observed value of the input factor, and the model gives a corresponding probability value through the input factor; or comparing the value of the input factor with a preset normal value, and determining the observed value of the input factor according to the difference value of the input factor and the preset normal value.
12. Based on the observations, the input factors are divided into normal class input factors and abnormal class input factors.
When the observed value is greater than or equal to the preset value, the input factor corresponding to the observed value greater than or equal to the preset value may be classified as an abnormal class factor, and the input factor corresponding to the observed value smaller than the preset value may be classified as a normal class factor, where the preset value may be set according to a business rule or may be obtained through model calculation, which is not limited in the embodiment of the present application.
13. And respectively carrying out anomaly measurement operation on the normal input factors and the anomaly input factors to obtain a normal anomaly measurement matrix and an anomaly measurement matrix.
The abnormal measurement operation is to detect an abnormal value of an input factor, and can be realized by a distance-based algorithm, a mode-based algorithm and other methodsThe anomaly metric matrix of the input factor is obtained, and the anomaly metric matrix can be expressed as D ═ D1,d2,…,dmAnd m is the number of input factors. And respectively carrying out anomaly measurement on the normal input factors and the anomaly input factors to obtain an input factor normal anomaly measurement matrix or an input factor anomaly measurement matrix.
And S120, calculating the data distribution characteristics of the abnormal measurement matrix.
Because the way of realizing the anomaly measurement of the target object by methods such as a distance-based algorithm or a pattern-based algorithm has the problem of unclear division of the anomaly threshold, in the embodiment of the application, after the anomaly measurement is calculated, the anomaly judgment threshold of the target object is determined by adopting Bayesian decision, and the accurate threshold is automatically obtained.
Optionally, the calculating the data distribution characteristic of the anomaly metric matrix includes:
and calculating the mean value and the standard deviation of the normal abnormal measurement matrix and the mean value and the standard deviation of the abnormal measurement matrix based on the observed value of the input factor to obtain a normal mean value, a normal standard deviation, an abnormal mean value and an abnormal standard deviation.
Wherein the conditional density function p (x | w) of the anomaly measure matrix of the normal classu) Conditional density function p (x | w) of anomaly metric matrix for sum anomaly classc) All obey Gaussian density function distribution
Figure BDA0002769757760000071
Said wuDenoted as normal class, wcExpressed as an anomaly class, the total density function p (x) of the target object anomaly measure matrix may be expressed as p (x) p (x | w)u)p(wu)+p(x|wc)p(wc) Wherein p (w)u) And p (w)c) Inputting a priori probability value of the factor for the target object, for example, knowing that the sample number of the input factor of the target object is A, the input factor is an abnormal class wcB, the input factor is a normal class wuOfA number C, A ═ B + C, then p (w)u)=C/A,p(wc) B/a. Thus, after the anomaly metrics are implemented, the process of solving for the change threshold can be transformed into a process of estimating the mean and standard deviation of the anomaly class and the normal class.
In particular, the estimation of the mean and standard deviation can be implemented using the Expectation Maximization (EM) algorithm. The EM algorithm is realized by an iteration loop, and each iteration consists of two steps of expectation and expectation maximization. The expected value can be obtained by directly estimating the expected value of the probability density from the observation data according to the current value of the parameter to be estimated; the estimate of the parameter is then updated by maximizing this expectation. The two steps are alternately performed in sequence in the whole iteration process until the iteration process converges, namely when the prior probability, the mean value and the standard deviation of the two adjacent iteration calculations are smaller than a given threshold value, the iteration is terminated.
S130, determining an abnormity judgment threshold value of the target object based on the data distribution characteristics of the abnormity metric matrix.
Optionally, the determining an anomaly determination threshold of the target object based on the data distribution characteristics of the anomaly metric matrix includes: substituting the normal class mean, the normal class standard deviation, the abnormal class mean and the abnormal class standard deviation into a Bayesian minimum error formula p (Tw |)u)p(wu)=p(T|wc)p(wc) Calculating the abnormality determination threshold; wherein, the p (w)u) Is the prior probability of the normal class input factor, p (w)c) The prior probability of the input factor of the anomaly class, p (Tw |)u) The p (Tw) is a conditional density function of the normal class anomaly metric matrixc) Measure the conditional density function of the matrix for exception-like exceptions, the p (Tw |)u) And said p (T | w)c) The abnormal judgment threshold value is T.
After the mean value and the standard deviation of the normal class and the abnormal class are obtained according to the EM algorithm, the abnormal judgment threshold value T can be calculated according to the Bayes minimum error rate theory. The bayesian minimum error can be expressed as: p (T | w)u)p(wu)=p(T|wc)p(wc) Separately, p (w)u)、p(wc)、p(x|wu) And p (x | w)c) Substituting into the formula, the solving formula of the obtained abnormality determination threshold T is:
Figure BDA0002769757760000072
thereby calculating the value of the abnormality determination threshold T.
In one possible example, the method further comprises: determining an anomaly analysis result of the target object based on the anomaly determination threshold, the anomaly analysis result including normality and anomaly.
Optionally, when the measured value of the input factor of the target object is greater than or equal to the abnormality determination threshold, the abnormality analysis result of the target object is normal; and when the measured value of the input factor of the target object is smaller than the abnormity judgment threshold value, judging that the abnormity analysis result of the target object is abnormal.
According to the T of the abnormity judgment threshold value, the abnormity analysis result of the target object by performing binary segmentation on the abnormity measurement matrix can be obtained. Specifically, when the observed value of the target object input factor is greater than or equal to T of the abnormality determination threshold, the abnormality analysis result of the target object is normal; and when the observed value of the target object input factor is smaller than T of the abnormity judgment threshold value, judging that the abnormity analysis result of the target object is abnormal. For example, when the value of the input parameter of the bulb is greater than or equal to T of the abnormality determination threshold, it indicates that the quality of the bulb is acceptable; when the value of the input parameter of the bulb is smaller than T of the abnormality determination threshold value, it indicates that the quality of the bulb is not good.
In one possible example, the method further comprises: comparing the actual results of the target objects with corresponding abnormal analysis results respectively to obtain the false recognition rate; and adjusting the abnormity judgment threshold value based on the mapping relation between the error recognition rate and the adjustment coefficient.
In the embodiment of the present application, the abnormality determination threshold determined by the embodiment may still have a deviation in practical application, so that when performing abnormality analysis or abnormality detection, a normal target object is determined to be abnormal, or an abnormal target object is determined to be normal. Therefore, in order to improve the accuracy of abnormality determination, after the abnormality determination threshold is determined, the abnormality determination threshold is adjusted after passing through the actual result of the target object.
Specifically, the plurality of target objects are subjected to anomaly analysis by a determined anomaly determination threshold value to obtain anomaly analysis results of the plurality of target objects, the anomaly analysis results of the plurality of target objects are compared with actual results of the plurality of target objects, and a first error identification rate and a second error identification rate of the plurality of target objects are calculated, wherein the first error identification rate is the probability of determining a normal target object as an abnormal target object, the second error identification rate is the probability of determining an abnormal target object as a normal target object, the first error identification rate is the number of target objects determining the normal target object as the abnormal target object/the total number of target objects, and the second error identification rate is the number of target objects determining the abnormal target object as the normal target object/the total number of target objects. And then adjusting the abnormity judgment threshold value through the difference value between the first error recognition rate and the second error recognition rate and the mapping relation between the difference value and the adjustment coefficient.
Further, under the condition that the difference value between the first error recognition rate and the second error recognition rate is larger than zero, the abnormal judgment threshold value is reduced based on the mapping relation between the difference value and the adjustment coefficient; and under the condition that the difference value between the first error identification rate and the second error identification rate is less than or equal to zero, increasing the abnormity judgment threshold value based on the mapping relation between the difference value and the adjustment coefficient.
In one possible example, the adjusted anomaly determination threshold value is an anomaly determination threshold value-data sensitivity coefficient is an adjustment coefficient.
Wherein the mapping relation between the difference value and the adjustment coefficient is different for different target objects. Specifically, different target objects have different data sensitivities, for example, target objects related to security class and property class have a larger influence of occurrence of an abnormality, the error recognition rate of abnormality determination is smaller corresponding to the error recognition rate of other types of target objects, and the interval between the difference values is also smaller. Therefore, in order to reduce the excessive adjustment amplitude of the abnormality determination threshold, the adjustment coefficient may be multiplied by the data sensitivity coefficient, and the data sensitivity coefficient may be smaller as the data sensitivity of the target object is larger.
In specific implementation, the electronic device may perform multi-scale feature decomposition on the relevant parameters of the target object by using a multi-scale decomposition algorithm to obtain a low-frequency feature component and a high-frequency feature component, where the multi-scale decomposition algorithm may be at least one of the following: pyramid transform algorithms, wavelet transforms, contourlet transforms, shear wave transforms, etc., and are not limited herein. Further, the low frequency feature component may be divided into a plurality of dimensions, each of which may include the same or different input factors. The low frequency feature component reflects the subject feature of the target object, and the high frequency feature component reflects the detail information of the target object. For example, when performing face recognition, the electronic device may perform multi-scale feature decomposition on the target vein image by using a multi-scale decomposition algorithm to obtain a low-frequency feature component and a high-frequency feature component, and divide the low-frequency feature component into a plurality of regions, where the area size of each region is the same or different. The low-frequency feature component reflects the main features of the image, and the high-frequency feature component reflects the detail information of the image.
Further, the electronic device may determine an information entropy corresponding to each of the dimensions or each of the regions to obtain a plurality of information entropies, determine an average information entropy and a target mean square error according to the plurality of information entropies, where the information entropy reflects the amount of information of the target object to a certain extent, and the mean square error may reflect the stability of the target object. The electronic device may pre-store a mapping relationship between a preset mean square error and an adjustment coefficient, and further determine a target adjustment coefficient corresponding to the target mean square error according to the mapping relationship, in this embodiment, a value range of the adjustment coefficient may be-0.15 to 0.15.
Further, the electronic device may adjust the average information entropy according to a target adjustment coefficient to obtain a target information entropy, where the target information entropy is (1+ target adjustment coefficient) × the average information entropy. The electronic device may pre-store a mapping relationship between a preset information entropy and an evaluation value, and further, may determine a first evaluation value corresponding to the target information entropy according to the mapping relationship between the preset information entropy and the evaluation value.
Further, the electronic device may determine an input factor distribution density or a target feature point distribution density according to the high-frequency feature component, where the input factor distribution density is the total number of input factors/dimension of the high-frequency feature component, and the target feature point distribution density is the total number of feature points/region area of the high-frequency feature component. The electronic device may further pre-store a preset mapping relationship between the input factor distribution density or the feature point distribution density and the evaluation value, further determine a second evaluation value corresponding to the input factor distribution density or the target feature point distribution density according to the preset mapping relationship between the feature point distribution density and the evaluation value, and finally perform a weighting operation according to the first evaluation value, the second evaluation value, the target low-frequency weight, and the target high-frequency weight to obtain an input factor quality evaluation value of the target object, which is specifically as follows:
the input factor quality evaluation value is first evaluation value target low-frequency weight + second evaluation value target high-frequency weight
According to the method for determining the anomaly determination threshold value, the anomaly measurement matrix is obtained by performing anomaly measurement operation on the target object; calculating the data distribution characteristics of the anomaly metric matrix; and determining an abnormality judgment threshold value of the target object based on the data distribution characteristics of the abnormality metric matrix. According to the method and the device, the abnormal measurement matrix of the target object is calculated, and the abnormal judgment threshold is calculated according to the data distribution characteristics of the abnormal measurement matrix, so that the problem of fuzzy judgment of the threshold limit in the abnormal object judgment process is solved, the abnormal threshold limit of the target object is determined, and the accuracy of abnormal judgment is improved.
The above description has introduced the solution of the embodiment of the present application mainly from the perspective of the method-side implementation process. It is understood that the electronic device comprises corresponding hardware structures and/or software modules for performing the respective functions in order to realize the above-mentioned functions. Those of skill in the art will readily appreciate that the present application is capable of hardware or a combination of hardware and computer software implementing the various illustrative elements and algorithm steps described in connection with the embodiments provided herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiment of the present application, the electronic device may be divided into the functional units according to the method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. It should be noted that the division of the unit in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation.
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a method for determining an abnormal determination threshold according to an embodiment of the present application, and as shown in fig. 2, the method for determining an abnormal determination threshold includes:
s210, obtaining an input factor of the target object, wherein the input factor is at least one parameter required for judging the target object to be abnormal.
S220, calculating the observation value of the input factor, and dividing the input factor into a normal input factor and an abnormal input factor based on the observation value.
And S230, respectively carrying out anomaly measurement operation on the normal input factors and the anomaly input factors to obtain a normal anomaly measurement matrix and an anomaly measurement matrix.
S240, calculating the data distribution characteristics of the normal anomaly measurement matrix and the anomaly measurement matrix.
S250, determining an abnormity judgment threshold value of the target object based on the data distribution characteristics of the normal class abnormity metric matrix and the data distribution characteristics of the abnormity metric matrix.
The detailed description of the steps 210 to 250 may refer to the corresponding steps described in the above fig. 1, and will not be described herein again.
It can be seen that, in the method for determining an anomaly determination threshold described in this embodiment of the application, an input factor of the target object is obtained, where the input factor is at least one parameter required for determining an anomaly of the target object, an observation value of the input factor is calculated, the input factor is divided into a normal class input factor and an anomaly class input factor based on the observation value, anomaly measurement operations are performed on the normal class input factor and the anomaly class input factor respectively to obtain a normal class anomaly measurement matrix and an anomaly measurement matrix, data distribution characteristics of the normal class anomaly measurement matrix and the anomaly measurement matrix are calculated, and the anomaly determination threshold of the target object is determined based on the data distribution characteristics of the normal class anomaly measurement matrix and the data distribution characteristics of the anomaly measurement matrix. By calculating the normal class anomaly measurement matrix and the anomaly class anomaly measurement matrix of the target object and calculating the anomaly judgment threshold according to the data distribution characteristics of the normal class anomaly measurement matrix and the data distribution characteristics of the anomaly class anomaly measurement matrix, the problem of fuzzy threshold limit judgment in the anomaly object judgment process is solved, the anomaly threshold limit of the target object is determined, and the accuracy of anomaly judgment is improved.
Referring to fig. 3a, fig. 3a is a block diagram of functional units of an apparatus 300 for determining an abnormality determination threshold according to an embodiment of the present application, as shown in fig. 3a, the apparatus 300 includes an arithmetic unit 310 and a determining unit 320, wherein,
an operation unit 310, configured to perform an anomaly metric operation on a target object to obtain an anomaly metric matrix;
the operation unit 310 is further configured to calculate a mean and a standard deviation of the anomaly metric matrix;
a determining unit 320, configured to determine an abnormality determination threshold of the target object based on the mean and the standard deviation.
Optionally, as shown in fig. 3b, the functional units of another apparatus 300 for determining an abnormality determination threshold provided in the embodiment of the present application form a block diagram, where the apparatus 300 further includes: an obtaining unit 330, configured to obtain an input factor of the target object, where the input factor is at least one parameter required for determining that the target object is abnormal;
in terms of performing an anomaly metric operation on a target object to obtain an anomaly metric matrix, the operation unit 310 is specifically configured to:
calculating an observation value of the input factor, and dividing the input factor into a normal input factor and an abnormal input factor based on the observation value; and respectively carrying out anomaly measurement operation on the normal input factors and the anomaly input factors to obtain a normal anomaly measurement matrix and an anomaly measurement matrix.
Optionally, in terms of calculating the data distribution characteristic of the abnormal metric matrix, the operation unit 310 is specifically configured to:
and calculating the mean value and the standard deviation of the normal abnormal measurement matrix and the mean value and the standard deviation of the abnormal measurement matrix based on the observed value of the input factor to obtain a normal mean value, a normal standard deviation, an abnormal mean value and an abnormal standard deviation.
Optionally, the determining unit 320 is specifically configured to: substituting the normal class mean, the normal class standard deviation, the abnormal class mean and the abnormal class standard deviation into a Bayesian minimum error formula p (Tw |)u)p(wu)=p(T|wc)p(wc) Calculating the abnormality determination threshold;
wherein, the p (w)u) Is the prior probability of the normal class input factor, p (w)c) The prior probability of the input factor of the anomaly class, p (Tw |)u) The p (Tw) is a conditional density function of the normal class anomaly metric matrixc) Measure the conditional density function of the matrix for exception-like exceptions, the p (Tw |)u) And said p (T | w)c) The abnormal judgment threshold value is T.
Optionally, the determining unit 320 is further configured to: determining an anomaly analysis result of the target object based on the anomaly determination threshold, the anomaly analysis result including normality and anomaly.
Optionally, when the measured value of the input factor of the target object is greater than or equal to the abnormality determination threshold, the abnormality analysis result of the target object is normal; and when the measured value of the input factor of the target object is smaller than the abnormity judgment threshold value, judging that the abnormity analysis result of the target object is abnormal.
Optionally, the apparatus 300 further includes a comparing unit 340 and an adjusting unit 350, where the comparing unit 340 is configured to compare actual results of a plurality of target objects with the corresponding abnormal analysis results, respectively, to obtain the false identification rate;
the adjusting unit 350 is configured to adjust the abnormality determination threshold based on the false recognition rate.
It can be understood that the functions of each program module of the apparatus for determining an abnormality determination threshold according to the embodiment of the present application may be specifically implemented according to the method in the foregoing method embodiment, and the specific implementation process may refer to the relevant description of the foregoing method embodiment, which is not described herein again.
Referring to fig. 4, fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure, as shown in fig. 4, the electronic device includes a memory, a communication interface, and a processor, wherein the electronic device may further include a communication bus, and the processor, the communication interface, and the memory may be connected to each other through the bus.
The processor is configured to implement the following steps when executing the program stored in the memory:
performing anomaly measurement operation on the target object to obtain an anomaly measurement matrix; calculating the data distribution characteristics of the anomaly metric matrix; and determining an abnormality judgment threshold value of the target object based on the data distribution characteristics of the abnormality metric matrix.
Further, the processor may be a general-purpose Central Processing Unit (CPU) or multiple CPUs, a single or multiple block Graphics Processing Unit (GPU), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits for controlling the execution of programs according to the present invention.
The Memory may be a Read-Only Memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that can store information and instructions, an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical Disc storage, optical Disc storage (including Compact Disc, laser Disc, optical Disc, digital versatile Disc, blu-ray Disc, etc.), a magnetic Disc storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to these.
In some examples, the memory may be self-contained, with the communication interface and communication bus connected to the processor. The memory may also be integral to the processor. A communication bus transfers information between the above components.
Embodiments of the present application also provide a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any one of the methods as described in the above method embodiments.
Embodiments of the present application also provide a computer program product comprising a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods as described in the above method embodiments. The computer program product may be a software installation package.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the above-described division of the units is only one type of division of logical functions, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of some interfaces, devices or units, and may be an electric or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit may be stored in a computer readable memory if it is implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in a software product, which is stored in a memory and includes several instructions for causing an electronic device (which may be a personal computer, a terminal device, or a network device) to execute all or part of the steps of the above-mentioned method of the embodiments of the present application. And the aforementioned memory comprises: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable memory, which may include: flash disk, ROM, RAM, magnetic or optical disk, and the like.
The foregoing detailed description of the embodiments of the present application has been presented to illustrate the principles and implementations of the present application, and the above description of the embodiments is only provided to help understand the method and the core concept of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims (10)

1. A method for determining an abnormality determination threshold value, applied to an electronic device, includes:
performing anomaly measurement operation on the target object to obtain an anomaly measurement matrix;
calculating the data distribution characteristics of the anomaly metric matrix;
and determining an abnormality judgment threshold value of the target object based on the data distribution characteristics of the abnormality metric matrix.
2. The method of claim 1, further comprising:
acquiring an input factor of the target object, wherein the input factor is at least one parameter required for judging the target object to be abnormal;
the performing an anomaly metric operation on the target object to obtain an anomaly metric matrix includes:
calculating an observation value of the input factor, and dividing the input factor into a normal input factor and an abnormal input factor based on the observation value;
and respectively carrying out anomaly measurement operation on the normal input factors and the anomaly input factors to obtain a normal anomaly measurement matrix and an anomaly measurement matrix.
3. The method of claim 2, wherein the computing the data distribution characteristic of the anomaly metric matrix comprises:
and calculating the mean value and the standard deviation of the normal abnormal measurement matrix and the mean value and the standard deviation of the abnormal measurement matrix based on the observed value of the input factor to obtain a normal mean value, a normal standard deviation, an abnormal mean value and an abnormal standard deviation.
4. The method of claim 3, wherein determining the anomaly determination threshold for the target object based on the data distribution characteristics of the anomaly metric matrix comprises:
substituting the normal class mean, the normal class standard deviation, the abnormal class mean and the abnormal class standard deviation into a Bayesian minimum error formula p (Tw |)u)p(wu)=p(T|wc)p(wc) Calculating the abnormality determination threshold;
wherein, the p (w)u) Is the prior probability of the normal class input factor, p (w)c) The prior probability of the input factor of the anomaly class, p (Tw |)u) A conditional density function for said normal class anomaly metric matrix, saidp(T|wc) Measure the conditional density function of the matrix for exception-like exceptions, the p (Tw |)u) And said p (T | w)c) The abnormal judgment threshold value is T.
5. The method according to any one of claims 1-4, further comprising:
determining an anomaly analysis result of the target object based on the anomaly determination threshold, the anomaly analysis result including normality and anomaly.
6. The method according to claim 4, characterized in that when the measured value of the input factor of the target object is greater than or equal to the abnormality determination threshold value, the result of the abnormality analysis of the target object is normal; and when the measured value of the input factor of the target object is smaller than the abnormity judgment threshold value, judging that the abnormity analysis result of the target object is abnormal.
7. The method of claim 5 or 6, further comprising:
comparing the actual results of the target objects with the corresponding abnormal analysis results respectively to obtain the false recognition rate;
and adjusting the abnormity judgment threshold value based on the mapping relation between the error recognition rate and the adjustment coefficient.
8. An abnormality determination threshold determination device, characterized by comprising:
the operation unit is used for performing anomaly measurement operation on the target object to obtain an anomaly measurement matrix;
the arithmetic unit is further used for calculating the data distribution characteristics of the abnormal measurement matrix;
and the determining unit is used for determining an abnormity judgment threshold value of the target object based on the data distribution characteristics of the abnormity metric matrix.
9. An electronic device, comprising a processor, memory, a communication interface, and one or more programs stored in the memory and configured to be executed by the processor, the programs comprising instructions for performing the steps in the method of any of claims 1-7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a computer program stored for data exchange, which computer program, when being executed by a processor, carries out the method according to any one of claims 1-7.
CN202011245175.0A 2020-11-10 2020-11-10 Method and device for determining abnormality determination threshold Pending CN112363859A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011245175.0A CN112363859A (en) 2020-11-10 2020-11-10 Method and device for determining abnormality determination threshold

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011245175.0A CN112363859A (en) 2020-11-10 2020-11-10 Method and device for determining abnormality determination threshold

Publications (1)

Publication Number Publication Date
CN112363859A true CN112363859A (en) 2021-02-12

Family

ID=74509138

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011245175.0A Pending CN112363859A (en) 2020-11-10 2020-11-10 Method and device for determining abnormality determination threshold

Country Status (1)

Country Link
CN (1) CN112363859A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116629706A (en) * 2023-07-19 2023-08-22 山东新中鲁建设有限公司 BIM-based green energy-saving building construction quality assessment method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6076083A (en) * 1995-08-20 2000-06-13 Baker; Michelle Diagnostic system utilizing a Bayesian network model having link weights updated experimentally
CN108985553A (en) * 2018-06-05 2018-12-11 中国平安人寿保险股份有限公司 A kind of recognition methods and equipment of abnormal user
CN111092757A (en) * 2019-12-06 2020-05-01 网宿科技股份有限公司 Abnormal data detection method, system and equipment
CN111680727A (en) * 2020-05-29 2020-09-18 深圳市英维克信息技术有限公司 Naive Bayes-based fault diagnosis method, diagnosis device, and storage medium

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6076083A (en) * 1995-08-20 2000-06-13 Baker; Michelle Diagnostic system utilizing a Bayesian network model having link weights updated experimentally
CN108985553A (en) * 2018-06-05 2018-12-11 中国平安人寿保险股份有限公司 A kind of recognition methods and equipment of abnormal user
CN111092757A (en) * 2019-12-06 2020-05-01 网宿科技股份有限公司 Abnormal data detection method, system and equipment
CN111680727A (en) * 2020-05-29 2020-09-18 深圳市英维克信息技术有限公司 Naive Bayes-based fault diagnosis method, diagnosis device, and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116629706A (en) * 2023-07-19 2023-08-22 山东新中鲁建设有限公司 BIM-based green energy-saving building construction quality assessment method
CN116629706B (en) * 2023-07-19 2023-09-22 山东新中鲁建设有限公司 BIM-based green energy-saving building construction quality assessment method

Similar Documents

Publication Publication Date Title
US10635890B2 (en) Facial recognition method and apparatus, electronic device, and storage medium
CN108364195B (en) User retention probability prediction method and device, prediction server and storage medium
CN109376786A (en) A kind of image classification method, device, terminal device and readable storage medium storing program for executing
US20190087737A1 (en) Anomaly detection and automated analysis in systems based on fully masked weighted directed
US20140032450A1 (en) Classifying unclassified samples
CN111445304B (en) Information recommendation method, device, computer equipment and storage medium
CN109257383B (en) BGP anomaly detection method and system
US20190087248A1 (en) Anomaly detection and automated analysis using weighted directed graphs
CN112363859A (en) Method and device for determining abnormality determination threshold
CN109002758B (en) Face feature point positioning method, device, equipment and storage medium
CN110889817A (en) Image fusion quality evaluation method and device
CN110083777B (en) Social network user grouping method and device and server
CN113448955A (en) Data set quality evaluation method and device, computer equipment and storage medium
CN110580483A (en) indoor and outdoor user distinguishing method and device
CN113034387A (en) Image denoising method, device, equipment and medium
CN115881304B (en) Risk assessment method, device, equipment and medium based on intelligent detection
CN111476668A (en) Identification method and device of credible relationship, storage medium and computer equipment
CN108985350B (en) Method and device for recognizing blurred image based on gradient amplitude sparse characteristic information, computing equipment and storage medium
CN110781410A (en) Community detection method and device
JP6797854B2 (en) Information processing device and information processing method
JP2020095583A (en) Bankruptcy probability calculation system utilizing artificial intelligence
CN114549062A (en) Consumer preference prediction method, system, electronic equipment and storage product
CN114355234A (en) Intelligent quality detection method and system for power module
CN114398964A (en) Fault diagnosis method, fault diagnosis device, electronic equipment and storage medium
CN114022284A (en) Abnormal transaction detection method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination