CN109782342B - Method for selecting seismic event correlation detection algorithm with better performance - Google Patents
Method for selecting seismic event correlation detection algorithm with better performance Download PDFInfo
- Publication number
- CN109782342B CN109782342B CN201811505562.6A CN201811505562A CN109782342B CN 109782342 B CN109782342 B CN 109782342B CN 201811505562 A CN201811505562 A CN 201811505562A CN 109782342 B CN109782342 B CN 109782342B
- Authority
- CN
- China
- Prior art keywords
- algorithm
- ratio
- precision
- performance
- recall
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Geophysics And Detection Of Objects (AREA)
Abstract
The invention provides a method for selecting an automatic earthquake event correlation detection algorithm with better performance, which belongs to the field of detection algorithm performance evaluation.
Description
Technical Field
The invention belongs to the field of performance evaluation of detection algorithms, and can be used for performance evaluation of automatic detection algorithms of seismic events.
Background
The detection of the seismic event is a process of inversion forming an event according to the signals recorded by the monitoring station and the characteristics thereof, and generally comprises the processes of detection of the station signals, identification of seismic phases, association and positioning of multiple seismic phases and the like. Seismic event detection can be viewed as a two-class problem, with any detection system containing two classes of errors: one is missing inspection; a false detection. Reducing false positives can make the system more sensitive, but can increase false positives. Conversely, reducing false positives reduces system sensitivity and increases the risk of false negatives. Aiming at the earthquake event detection problem, the academic community provides a plurality of detection algorithms, including a GA algorithm based on a global lattice point, a NET-VISA method based on a Bayesian probability model and the like. It is desirable to provide a seismic event correlation detection method with superior performance.
How to judge which algorithm is better and is more suitable for the requirement of earthquake event detection needs to compare and evaluate the performance of the detection algorithm.
In many studies of detection algorithm optimization, ROC curves are used for algorithm performance evaluation. The method plots true rate (TPR) and False Positive Rate (FPR) curves as algorithm sensitivities. Since the calculation of false positive rate requires the knowledge of the value of true negative, and for seismic event detection algorithms, the calculation of true negative requires the statistics of the number of false events formed by the possible combination of detections for all stations per day, which is difficult to calculate.
For the field of data retrieval and the like, Precision (Precision) and recall (recall) are generally adopted to evaluate the performance of the algorithm. Precision is defined as the proportion of data returned that is relevant to a query, and recall is defined as how much information of interest to the user is retrieved. And evaluating the performance of the detection algorithm by drawing a P-R curve, searching a balance point, calculating a harmonic mean value and the like. The method can be used as a performance evaluation index of a seismic event detection algorithm because the calculation of the recall ratio and the precision ratio does not need to know the value of a true counter example. However, the detection process of the seismic event is complex, the number of parameters is large, and the common methods such as P-R curve, balance point, harmonic mean and the like of the algorithm are difficult to draw, and are not suitable for performance measurement of the algorithm. Therefore, the performance evaluation method and indexes are designed, the performance of the algorithm is reflected more reasonably, intuitively and reliably, and the method is an urgent requirement for optimization research of the earthquake event detection algorithm.
Disclosure of Invention
The invention aims to provide a method for selecting an automatic earthquake event correlation detection algorithm with better performance, which detects earthquake events through a plurality of automatic earthquake event correlation detection algorithms, judges and evaluates results of earthquake time detection of various methods, determines recall ratio and precision ratio of the earthquake event correlation detection method, and quantitatively compares detection results of various methods through performance measurement curves in two-dimensional space of the recall ratio and the precision ratio, thereby obtaining the earthquake event correlation detection method with better performance, which is used for better detection of earthquake events.
The technical scheme of the invention is as follows: a method for selecting a seismic event correlation detection algorithm with better performance is characterized by comprising the following steps: which comprises the following steps:
1) acquiring signals acquired by a plurality of seismograph sensors in various regions of the world;
2) outputting signals acquired by a seismograph sensor to a seismic event detection system, wherein the seismic event detection system adopts an automatic seismic event detection algorithm to detect the signals, identify seismic phases, associate a plurality of seismic phases and position seismic events to obtain seismic event information;
3) the earthquake event detection system adopts other automatic detection algorithms of various earthquake events to respectively obtain earthquake event information corresponding to the automatic detection algorithms of various earthquake events;
4) for the event bulletin produced by various earthquake event automatic detection algorithms, the daily recall ratio and the daily precision ratio of the detection algorithm are calculated based on the defined event matching rule by taking the reference event bulletin as a standard;
5) establishing a precision ratio-recall ratio two-dimensional coordinate space diagram, referred to as a P-R space for short, by taking the precision ratio as a vertical axis and the recall ratio as a horizontal axis, and drawing the daily recall ratio and the precision ratio obtained by calculation in the P-R space;
6) drawing a performance measurement curve in a P-R space to realize the visual comparison of various algorithm results;
the concrete contents are as follows:
6.1 by FβAs function value, the precision ratio P and the recall ratio R are variables, and F is establishedβThe function between P, R is as follows:
i.e. a performance metric curve, FβReferred to as a performance metric value; determining a weight value beta of a weight parameter of the formula;
6.2 plotting the performance metric function curve in P-R space: drawing a function curve between a plurality of variable precision ratios P and recall ratios R corresponding to the performance metric values when the performance metric values take a plurality of determined values;
6.3 comparing the results of the daily recall ratio and the daily precision ratio of each algorithm in the P-R space to be positioned at the position of the performance measurement curve; if the performance metric value F of the performance metric curve of a certain algorithm isβThe larger the detection performance of the algorithm.
Preferably, if a certain calculation is madePerformance metric value F of performance metric curve where method is locatedβThe larger the weight value is, and the more 1 the weight value is selected, the better the recall ratio of the algorithm is; if the performance metric value F of the performance metric curve of a certain algorithm isβThe larger the weight value is, and the more optimal the precision ratio of the algorithm is when the weight value is selected to be less than 1.
Preferably, for established FβAnd P, R, selecting different weight values for different emphasis of recall ratio and precision ratio: and when the accuracy rate is checked, the weight value is less than 1.
The invention has the beneficial effects that:
the method solves the performance evaluation problem of the seismic data processing algorithm, and has the advantages of simplicity, intuition, comprehensiveness, reliability and the like; the method adopts a performance measurement curve based on weighted harmonic mean to measure the performance of the algorithm, and can reflect different biases of the algorithm on the recall ratio and the precision ratio; and drawing a performance curve (namely a performance measurement curve) of the weighted harmonic mean in the P-R two-dimensional space diagram, and visually reflecting the performance of the earthquake detection algorithm by using the performance curve values of the daily recall ratio and the daily precision ratio of the detection result.
Drawings
FIG. 1 is a schematic diagram of a performance measurement curve in a P-R two-dimensional space;
FIG. 2 is a graph illustrating a comparison of performance metrics according to an embodiment of the present invention;
Detailed Description
The invention is further illustrated by the following figures and examples.
A method for selecting a seismic event correlation detection algorithm with better performance is characterized by comprising the following steps: which comprises the following steps:
1) acquiring signals acquired by a plurality of seismograph sensors in various regions of the world;
2) outputting signals acquired by a seismograph sensor to a seismic event detection system, wherein the seismic event detection system adopts an automatic seismic event detection algorithm to detect the signals, identify seismic phases, associate a plurality of seismic phases and position seismic events to obtain seismic event information;
3) the earthquake event detection system adopts other automatic detection algorithms of various earthquake events to respectively obtain earthquake event information corresponding to the automatic detection algorithms of various earthquake events;
4) for the event bulletin produced by various earthquake event automatic detection algorithms, the daily recall ratio and the daily precision ratio of the detection algorithm are calculated based on the defined event matching rule by taking the reference event bulletin as a standard.
5) And establishing a precision ratio-recall ratio two-dimensional coordinate space diagram, namely a P-R space for short, by taking the precision ratio as a vertical axis and the recall ratio as a horizontal axis, and drawing the daily recall ratio and the precision ratio obtained by calculation in the P-R space.
6) And drawing a performance measurement curve in a P-R space to realize the visual comparison of various algorithm results.
The specific implementation mode is as follows:
6.1 the weighted harmonic mean formula of recall and precision is:
determining a weight value beta of a weight parameter of the formula; (in the formula, different weights are selected for recall ratio and precision ratio, when the recall ratio is checked on the side, the weight is greater than 1, when the precision ratio is checked on the side, the weight is less than 1, that is, the weight parameter of the formula of the weighted harmonic mean of recall ratio and precision ratio is determined;)
With FβAs function value, the precision ratio P and the recall ratio R are variables, and F is establishedβAnd P, R, where FβReferred to as a performance metric value;
6.2 plotting the Performance metric function curves in P-R recall (in contour mode): plotting a function curve between a plurality of lines P, R for a plurality of determined values of the performance metric;
6.3 comparing the results of the daily recall ratio and the precision ratio of each algorithm in the P-R space to be positioned at the position of the performance measurement curve; if the measurement F of the performance metric curve of an algorithm isβThe larger, and the weight in step 6.1When the reselection rate is more than 1, the recall ratio of the algorithm is more optimal; if the measurement F of the performance metric curve of an algorithm isβThe larger the algorithm is, and when the weight selection in the step 6.1 is less than 1, the precision ratio of the algorithm is better, and the intuitive comparison of the algorithm is realized.
The five-pointed star in fig. 1 represents the hypothetical procedure one result, the dots represent the hypothetical procedure two result, the black curve represents the performance curve when β is 2, and the gray curve represents the performance curve when β is 0.5. The method in the figure has a higher performance curve value than the method two.
The specific limitations are as follows:
4) for the event bulletin generated by the earthquake event detection algorithm, the daily recall ratio and the daily precision ratio of the detection algorithm are calculated and recorded as the unit time of each day based on the reference event bulletin as a standard and defined event matching standards (such as arrival time difference <30s and position difference <2 degrees). Precision (Precision) is defined as the probability that the prediction is correct in the sample predicted as a positive example; recall (Recall) is defined as the probability that a positive sample is predicted to be correct. The calculation formulas of the daily precision and the daily recall ratio are as follows:
wherein TP (true positive): true and true every day; FP (false positive): false positive case every day; fn (false negative): false counterexample was taken every day.
5) And constructing a Precision-Recall (Precision-Recall) two-dimensional space diagram, namely a P-R space, by taking the Precision as a vertical axis and the Recall as a horizontal axis, and drawing the daily Recall and the Precision obtained by calculation in the P-R space.
6) Taking weighted harmonic mean formula as performance metric function, drawing metric function curves in P-R graph, each curve representing a determined performance metric value FβBeta is a weight value representing a sum of recall ratiosDifferent precision ratios are emphasized. Beta is a>The 1 hour recall is more important, beta<1 precision is more important.
The principle is as follows: the larger the performance metric value is, the better the performance metric value is, the comparison result of the daily recall ratio and the daily recall ratio of each algorithm in the P-R space diagram is positioned in the area of a performance metric curve, and the corresponding F isβThe larger the performance metric, the better the detection performance
Intuitively, as in the figure, it can be said that the closer to the upper right corner of the P-R diagram, the corresponding FβThe larger the performance metric, the better the detection performance.
The basic idea of the invention is to calculate the recall ratio and precision ratio index of the algorithm, draw a performance measurement curve in a P-R two-dimensional space diagram in a contour line mode, and quantitatively and intuitively realize the performance evaluation of the algorithm by comparing the position of the performance measurement curve where the result of the algorithm is located.
For example: two seismic event correlation algorithms are evaluated by adopting the method, one is a NET-VISA method based on a Bayesian model, and the other is a GA method based on global lattice point correlation. The two methods are processed off-line by using Data of a year which completely forbids an International Monitoring System (IMS), and the two algorithms are compared and evaluated by taking an International Data Center (IDC) manual review bulletin (REB) as a reference event.
The precision and recall per day of each algorithm for the year are first calculated with defined event matching criteria (arrival time difference <30 s; position difference <2 degrees). The precision ratio is used as the ordinate, the recall ratio is used as the abscissa, and the above calculation results are drawn in a P-R two-dimensional space diagram, as shown in FIG. 2.
In fig. 2, the five-pointed star represents the daily recall ratio and precision ratio of the NET-VISA algorithm, the dots represent the daily recall ratio and precision ratio of the GA algorithm, and the dot size represents the number of daily reference events; by quantitative comparison, the NET-VISA algorithm generally has higher performance curve value than the GA algorithm.
Beta-2 is taken as the weight of the weighted harmonic average performance function, the evaluation of the algorithm is focused on high recall ratio, a performance measurement curve is drawn in a P-R two-dimensional space diagram in a contour line mode, the recall ratio of the algorithm and the performance measurement curve value of the recall ratio result are compared, and the NET-VISA method has higher performance value (generally located at the upper right of the GA result, and the results are distributed on the performance curve with higher value) which is shown in the figure 2. By utilizing the method, the evaluation result of the earthquake event correlation algorithm is comprehensively, intuitively and reliably given.
Claims (4)
1. A method for selecting a seismic event correlation detection algorithm with better performance is characterized by comprising the following steps: which comprises the following steps:
1) acquiring signals acquired by a plurality of seismograph sensors in various regions of the world;
2) outputting signals acquired by a seismograph sensor to a seismic event detection system, wherein the seismic event detection system adopts an automatic seismic event detection algorithm to detect the signals, identify seismic phases, associate a plurality of seismic phases and position seismic events to obtain seismic event information;
3) the earthquake event detection system adopts other automatic detection algorithms of various earthquake events to respectively obtain earthquake event information corresponding to the automatic detection algorithms of various earthquake events;
4) for the event bulletin produced by various earthquake event automatic detection algorithms, the daily recall ratio and the daily precision ratio of the detection algorithm are calculated based on the defined event matching rule by taking the reference event bulletin as a standard;
5) establishing a precision-recall ratio two-dimensional coordinate space diagram, referred to as a P-R space for short, by taking the precision ratio as a vertical axis and the recall ratio as a horizontal axis, and drawing the daily recall ratio and the daily recall ratio obtained by calculation in the P-R space;
6) drawing a performance measurement curve in a P-R space to realize the visual comparison of various algorithm results;
the concrete contents are as follows:
6.1 by FβAs function value, the precision ratio P and the recall ratio R are variables, and F is establishedβThe function between P, R is as follows:
i.e. a performance metric curve, FβReferred to as a performance metric value; determining a weight value beta of a weight parameter of the function;
6.2 plotting the performance metric function curve in P-R space: drawing a function curve between a plurality of variable precision ratios P and recall ratios R corresponding to the performance metric values when the performance metric values take a plurality of determined values;
6.3 comparing the results of the daily recall ratio and the daily precision ratio of each algorithm in the P-R space to be positioned at the position of the performance measurement curve; performance metric value F if its positionβThe larger the algorithm is, the better the detection performance of the algorithm is;
where precision is defined as the proportion of data returned that is relevant to the query, and recall is defined as how much information the user is interested in is retrieved.
2. The method of claim 1, wherein the method comprises the steps of: if the performance metric value F of the performance metric curve of a certain algorithm isβThe larger the weight value is, and the more 1 the weight value is selected, the better the recall ratio of the algorithm is; if the performance metric value F of the performance metric curve of a certain algorithm isβThe larger the weight value is, and the more optimal the precision ratio of the algorithm is when the weight value is selected to be less than 1.
3. The method of claim 1, wherein the method comprises the steps of: for established FβAnd P, R, selecting different weight values for different emphasis of recall ratio and precision ratio: and when the accuracy rate is checked, the weight value is less than 1.
4. The method of claim 1, wherein the method comprises the steps of: the event matching criteria based on the definition are: the arrival difference is <30s and the position difference is <2 degrees.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811505562.6A CN109782342B (en) | 2018-12-10 | 2018-12-10 | Method for selecting seismic event correlation detection algorithm with better performance |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811505562.6A CN109782342B (en) | 2018-12-10 | 2018-12-10 | Method for selecting seismic event correlation detection algorithm with better performance |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109782342A CN109782342A (en) | 2019-05-21 |
| CN109782342B true CN109782342B (en) | 2021-01-15 |
Family
ID=66495799
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811505562.6A Active CN109782342B (en) | 2018-12-10 | 2018-12-10 | Method for selecting seismic event correlation detection algorithm with better performance |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109782342B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4721945B2 (en) * | 2006-04-14 | 2011-07-13 | 電気化学工業株式会社 | Crustal deformation detection device and crustal deformation detection method |
| JP2008249485A (en) * | 2007-03-30 | 2008-10-16 | Railway Technical Res Inst | Method of estimating earthquake damage in evaluation objective point |
| RU2008127931A (en) * | 2008-07-08 | 2010-01-20 | Игорь Николаевич Сушкин (RU) | EARTHQUAKE PREDICTION METHOD |
| CN107045140A (en) * | 2017-02-22 | 2017-08-15 | 中国科学院寒区旱区环境与工程研究所 | The Di-qi map of earthquake prediction |
| CN108932311B (en) * | 2018-06-20 | 2022-06-03 | 天津大学 | Method for detecting and predicting emergency |
-
2018
- 2018-12-10 CN CN201811505562.6A patent/CN109782342B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109782342A (en) | 2019-05-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Bifet et al. | Pitfalls in benchmarking data stream classification and how to avoid them | |
| CN112506990B (en) | Hydrological data anomaly detection method based on spatiotemporal information | |
| CN1750021A (en) | Methods and apparatus for managing and predicting performance of automatic classifiers | |
| CN104931960A (en) | Trend message and radar target state information whole-track data correlation method | |
| CN107679734A (en) | It is a kind of to be used for the method and system without label data classification prediction | |
| CN112633412A (en) | Abnormal electricity consumption detection method, equipment and storage medium | |
| US7373332B2 (en) | Methods and apparatus for detecting temporal process variation and for managing and predicting performance of automatic classifiers | |
| CN113537337A (en) | Training method, abnormality detection method, apparatus, device, and storage medium | |
| US20050144537A1 (en) | Method to use a receiver operator characteristics curve for model comparison in machine condition monitoring | |
| US8072324B2 (en) | Methods for remote device status determination | |
| CN116415931A (en) | Big data-based power equipment operation state monitoring method and system | |
| CN112766301A (en) | Similarity judgment method for indicator diagram of oil extraction machine | |
| CN116416884A (en) | Testing device and testing method for display module | |
| CN104198998B (en) | Clustering treatment based CFAR (Constant False Alarm Rate) detection method under non-uniform background | |
| CN113553319A (en) | LOF outlier detection cleaning method, device and equipment based on information entropy weighting and storage medium | |
| CN109782342B (en) | Method for selecting seismic event correlation detection algorithm with better performance | |
| CN1749988A (en) | Methods and apparatus for managing and predicting performance of automatic classifiers | |
| CN107277777A (en) | A kind of indoor orientation method and device | |
| CN109709604B (en) | Method for selecting seismic event correlation detection algorithm with low error cost | |
| CN1750020A (en) | Methods and apparatus for managing and predicting performance of automatic classifiers | |
| CN111882135B (en) | Internet of things equipment intrusion detection method and related device | |
| CN113255820B (en) | Training method for falling-stone detection model, falling-stone detection method and related device | |
| CN111798237B (en) | Abnormal transaction diagnosis method and system based on application log | |
| CN103903258A (en) | Method for detecting changes of remote sensing image based on order statistic spectral clustering | |
| CN107506824B (en) | Method and device for detecting bad observation data of power distribution network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |