CN113064890A - Quality evaluation method, device, server and medium for operator data - Google Patents
Quality evaluation method, device, server and medium for operator data Download PDFInfo
- Publication number
- CN113064890A CN113064890A CN202110367597.3A CN202110367597A CN113064890A CN 113064890 A CN113064890 A CN 113064890A CN 202110367597 A CN202110367597 A CN 202110367597A CN 113064890 A CN113064890 A CN 113064890A
- Authority
- CN
- China
- Prior art keywords
- data
- cluster
- files
- base stations
- storage size
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000013441 quality evaluation Methods 0.000 title claims abstract description 31
- 230000002159 abnormal effect Effects 0.000 claims abstract description 27
- 238000004458 analytical method Methods 0.000 claims description 18
- 101150116940 AGPS gene Proteins 0.000 claims description 13
- 230000005856 abnormality Effects 0.000 claims description 12
- 230000005540 biological transmission Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000001303 quality assessment method Methods 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 abstract description 27
- 238000011897 real-time detection Methods 0.000 abstract description 2
- 238000001514 detection method Methods 0.000 description 18
- 238000012545 processing Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 11
- 238000007726 management method Methods 0.000 description 7
- 238000010606 normalization Methods 0.000 description 6
- 238000010276 construction Methods 0.000 description 5
- 238000003745 diagnosis Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000000275 quality assurance Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Abstract
The invention discloses a quality evaluation method, a device, a server and a medium of operator data, comprising the following steps: collecting MR data of a data source, an interface machine, a cluster and a shared server; judging whether the number of the data source, the interface machine and the cluster files and the file storage size are equal, and reporting whether the number of the base stations is equal to the number of the working parameter base stations; if the deviation of the number of the files is larger than a first preset threshold value, sending alarm information, if the deviation of the storage size of the files is larger than a second preset threshold value, sending the alarm information, and if the difference value of the number of the reported base stations which is smaller than the number of the working parameter base stations exceeds a third preset threshold value, sending the alarm information; and analyzing whether the data of the operator has abnormity, abnormal links and reasons of various abnormal judgments according to whether the alarm information is sent and the type frequency of the alarm information, and outputting a data quality evaluation result. The data quality monitoring of the invention realizes the whole monitoring process, the rule configuration and the real-time detection.
Description
Technical Field
The embodiment of the invention relates to the field of data processing, in particular to a method, a device, a server and a medium for evaluating the quality of operator data.
Background
Information resources, which are strategic resources and production elements, are becoming the basis for the normal operation of enterprises, and abnormal data such as missing, error and the like necessarily produce wrong or inaccurate processing results, so that wrong or inaccurate decisions are made, and serious or even fatal consequences can be caused to the enterprises. Thus, data quality appears to be critical to the enterprise.
The data quality assurance work currently has the following problems:
(1) the data source is complex, the integrity is difficult to guarantee and identify
A. The data volume is large: the operator data is in the amount of ten thousand G per day. B. The data is of a wide variety: the operator data comprises O-domain wireless data, perception DPI data, fixed network DPI data, mobile internet logs, B-domain data, M-domain data and the like. C. The data sources are many: the operator data source comprises OMC equipment, SCA equipment, core network equipment, provincial terminal unified gateway equipment and the like. Based on the situations of large data quantity, multiple types and multiple data sources, the integrity of the data is difficult to guarantee and identify.
(2) Long data processing chain and low problem positioning efficiency
The data processing chain is long, and comprises a data source link, a transmission link, an acquisition link, a processing link, a warehousing link, a shared interface link and the like, the whole data processing process is closed, data is difficult to locate when a problem occurs, the links with the problem cannot be rapidly identified, and an automatic early warning mechanism is not provided. Most of the methods rely on manual positioning and screening, the operation and maintenance cost is high, the automation degree is low, and the current situation of high cost and low efficiency of maintenance is caused.
(3) Lack of standardized and platformized data quality detection system
The whole operator data lacks a standardized data quality monitoring guarantee system, data quality detection points are configured, the whole-process monitoring of the data quality is supported, a comprehensive and timely data quality report is provided, and configured alarm setting is provided.
Disclosure of Invention
The embodiment of the invention provides a quality evaluation method, a quality evaluation device, a quality evaluation server and a quality evaluation medium for operator data, so as to realize whole monitoring, rule configuration and real-time detection.
In a first aspect, an embodiment of the present invention provides a method for evaluating quality of operator data, including:
the method comprises the following steps of firstly, acquiring MR data of a data source link, an interface machine link, a cluster link and a shared server link, wherein the MR data comprises: A. the number of files of the data source, the number of files of the interface machine and the number of files of the cluster; B. the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster; C. reporting the number of base stations by the cluster and reporting the number of base stations by the sharing server;
judging whether the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster are equal, whether the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster are equal, and whether the number of the cluster reporting base stations and the number of the sharing server reporting base stations are equal to the number of the working parameter base stations;
step three, if the deviation of the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster is larger than a first preset threshold value, alarm information is sent out, if the deviation of the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster is larger than a second preset threshold value, the alarm information is sent out, and if the difference value of the number of the cluster reporting base stations and the number of the sharing server reporting base stations which are respectively smaller than the number of the industrial parameter base stations exceeds a third preset threshold value, the alarm information is sent out;
and step four, analyzing whether the data of the operator is abnormal or not, links in which the data are abnormal and reasons of all judgment abnormalities exist according to whether the alarm information is sent or not and the type frequency of the alarm information, and outputting a data quality evaluation result.
Optionally, the MR data further includes: D. call drop rate, imsi effective rate, agps reporting rate.
Optionally, in the second step, the method further includes:
and judging whether the call drop rate is less than 0.3 percent, the imsi effective rate is more than 90 percent, and the agps reporting rate is between 2 and 5 percent.
Optionally, in step three, the method further includes:
and if the call drop rate exceeds 0.3 percent, or the imsi effective rate is less than 90 percent, or the agps reporting rate is not between 2 and 5 percent, sending out alarm information.
Optionally, in step three, if an abnormal judgment occurs: a. the number of cluster files is less than that of the interface machine files; b. the storage size of the cluster file is smaller than that of the interface machine file; c. the number of the cluster reporting base stations is less than the number of the engineering parameter base stations; and (4) judging the abnormality, namely a, b and c belong to the abnormality judgment of the cluster link, and determining that the problem occurs in the transmission process from the interface machine file to the cluster file and the cluster data is lost.
Optionally, in step three, if an abnormal judgment occurs: d. and if the number of the shared service reporting base stations is less than the data volume of the engineering parameter base stations, performing source tracing analysis on the judgment d, and determining that the judgment d is abnormal due to the loss of the cluster link data.
Optionally, in step four, if the data is determined to be lost, the evaluation data is substantially available when the data loss rate is less than 1%.
In a second aspect, an embodiment of the present invention further provides an apparatus for evaluating quality of operator data, including:
the acquisition unit is used for acquiring MR data of a data source link, an interface machine link, a cluster link and a shared server link, wherein the MR data comprises: A. the number of files of the data source, the number of files of the interface machine and the number of files of the cluster; B. the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster; C. reporting the number of base stations by the cluster and reporting the number of base stations by the sharing server;
the judging unit is used for judging whether the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster are equal, whether the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster are equal, and whether the number of the cluster reporting base stations and the number of the sharing server reporting base stations are equal to the number of the industrial parameter base stations;
the alarm unit is used for sending alarm information if the deviation of the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster is larger than a first preset threshold value, sending alarm information if the deviation of the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster is larger than a second preset threshold value, and sending alarm information if the difference value of the number of the cluster reporting base stations and the number of the sharing server reporting base stations, which is respectively smaller than the number of the industrial parameter base stations, exceeds a third preset threshold value;
and the analysis unit is used for analyzing whether the data of the operator has abnormity, abnormal links and various abnormal reasons according to the alarm information and the type frequency of the alarm information, and outputting a data quality evaluation result.
In a third aspect, an embodiment of the present invention further provides a server, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor, when executing the computer program, implements the method for evaluating the quality of the operator data in any of the foregoing embodiments.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the quality assessment method for operator data in any of the foregoing embodiments.
The technical scheme of the embodiment of the invention provides powerful support for data quality management work through a data quality monitoring and guaranteeing system, and realizes the following system construction targets: (1) the monitoring is completed. And corresponding data quality monitoring means are provided for a data source link, a transmission link, an acquisition link, a processing link, a warehousing link, an interface sharing link and the like, so that the data quality monitoring of the whole process is realized. (2) And (5) rule configuration. And formulating a data quality evaluation general rule and a user-defined quality detection rule to realize the configuration of the detection rule and the alarm rule. (3) The detection is real-time. The online real-time diagnosis is realized, and the normalizations, the online and the real-time performances of fault discovery, fault positioning, influence analysis, problem tracing and the like of the data quality problem are realized.
Drawings
Fig. 1 is a schematic flowchart of a method for evaluating quality of operator data according to a first embodiment of the present invention;
fig. 2 is a schematic structural diagram of an apparatus for evaluating quality of operator data according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a server in the third embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Furthermore, the terms "first," "second," and the like may be used herein to describe various orientations, actions, steps, elements, or the like, but the orientations, actions, steps, or elements are not limited by these terms. These terms are only used to distinguish one direction, action, step or element from another direction, action, step or element. For example, the first preset threshold may be referred to as a second preset threshold, and similarly, the second preset threshold may be referred to as a first preset threshold, without departing from the scope of the present application. Both the first preset threshold and the second preset threshold are preset thresholds, but they are not the same preset threshold. The terms "first", "second", etc. are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Example one
Fig. 1 is a schematic flow chart of a method for evaluating quality of operator data according to an embodiment of the present invention, which is applicable to data quality monitoring. The method of the embodiment of the invention can be executed by a quality evaluation device of operator data, which can be realized by software and/or hardware, and can be generally integrated in a server or a terminal device. Referring to fig. 1, the method for evaluating the quality of operator data according to the embodiment of the present invention specifically includes the following steps:
step S110, collecting MR data of a data source link, an interface machine link, a cluster link and a shared server link, wherein the MR data comprises: A. the number of files of the data source, the number of files of the interface machine and the number of files of the cluster; B. the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster; C. the cluster reports the number of the base stations, and the sharing server reports the number of the base stations.
Specifically, data quality management is performed on MR data, acquisition information of the MR data is configured in the acquisition process of a data quality management center, the data quality management center mainly performs management and issuing of various rule configurations, a data quality information acquisition module is mainly responsible for acquiring quality original information of the data, and is equivalent to a data quality embedding point, the acquisition rule is set according to the data quality management center for acquiring the data information, for example, information fields of which links are acquired, and a user can complete configuration through an interface of the data quality management center. In this embodiment, the data monitoring step includes: a data source link, an interface machine link, a cluster link and a shared server link. The acquired MR data includes: A. the number of files of the data source, the number of files of the interface machine and the number of files of the cluster; B. the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster; C. the cluster reports the number of the base stations, and the sharing server reports the number of the base stations. As an alternative embodiment, the MR data further comprises: D. call drop rate, imsi effective rate, agps reporting rate.
For example, the collected data information is: the number of files of the data source is 94282, the file storage size of the data source is 864954M, the number of files of the interface machine is 94282, the file storage size of the interface machine is 864954M, the number of cluster reporting base stations is 135687, the call drop rate is 0.09%, the effective rate of imsi is 99.5%, and the report rate of agps is 2.98%.
Step S120, judging whether the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster are equal, whether the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster are equal, and whether the number of the cluster reporting base stations and the number of the sharing server reporting base stations are equal to the number of the working parameter base stations.
Specifically, after data quality information of each link is collected, index judgment is carried out on the information, the judgment standard of the index judgment is also from a data quality management center, and data quality evaluation judgment is carried out according to the rule set by the data quality management center for judging each index. In this embodiment, the data quality evaluation requirement determines whether the number of files of the data source, the number of files of the interface machine, and the number of files of the cluster are equal; whether the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster are equal or not is judged; and whether the number of the cluster reporting base stations and the number of the sharing server reporting base stations are equal to the number of the working parameter base stations or not is judged. As an alternative embodiment, in step S120, the method further includes: and judging whether the call drop rate is less than 0.3 percent, the imsi effective rate is more than 90 percent, and the agps reporting rate is between 2 and 5 percent.
Step S130, if the deviation of the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster is larger than a first preset threshold value, alarm information is sent out, if the deviation of the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster is larger than a second preset threshold value, the alarm information is sent out, and if the difference value of the number of the cluster reporting base stations and the number of the sharing server reporting base stations, which is respectively smaller than the number of the industrial parameter base stations, exceeds a third preset threshold value, the alarm information is sent out.
For example, the deviation of the number of files of the data source, the number of files of the interface machine and the number of files of the cluster is 0.01%, the deviation of the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster is 0.1%, the number of the cluster reporting base stations and the number of the sharing server reporting base stations are less than 5% of the number of the industrial parameter base stations, and the alarm is generated. As an alternative embodiment, in step S130, the method further includes: and if the call drop rate exceeds 0.3 percent, or the imsi effective rate is less than 90 percent, or the agps reporting rate is not between 2 and 5 percent, sending out alarm information.
Step S140, analyzing whether the data of the operator has abnormity, abnormal links and reasons of various abnormal judgments according to whether the alarm information is sent out and the type frequency of the alarm information, and outputting a data quality evaluation result.
For example, according to the discrimination discovery of step S130: a. the number of cluster files is less than that of the interface machine files; b. the storage size of the cluster file is smaller than that of the interface machine file; c. the number of the cluster reporting base stations is less than the number of the engineering parameter base stations; d. the number of the base stations reported by the shared service is less than the data volume of the working parameter base stations. Further analysis was as follows: the abnormity discrimination a, b and c belong to abnormity discrimination occurring in a cluster link, normalization analysis is carried out on the abnormity discrimination a, b and c, the three abnormity discrimination results from the same problem, namely the problem occurs in the transmission process from the interface machine file to the cluster file, and the cluster data is lost. Judging d belongs to a link of a shared server, an upstream link of the shared server is a cluster link, tracing analysis is carried out on the judging d, and the abnormal judging d is found to be caused by data loss of the cluster link, and the problem root is the cluster link.
And finally obtaining a problem analysis conclusion after the reason of discriminant analysis is passed, wherein the four abnormal discriminants are caused by data loss in a cluster link, and the data is not completely put in storage due to file packet damage and data processing task error reporting according to the historical processing result suggestion. And processing the proposal to negotiate a data source manufacturer for data retransmission and performing additional recording on the retransmitted data. And (4) evaluating the data quality, obtaining a conclusion according to the indexes, losing 1% of the data quantity, not influencing the overall use effect, and displaying other normal detection indexes through a visual report chart.
The technical scheme of the embodiment of the invention provides powerful support for data quality management work through a data quality monitoring and guaranteeing system, and realizes the following system construction targets: (1) the monitoring is completed. And corresponding data quality monitoring means are provided for a data source link, a transmission link, an acquisition link, a processing link, a warehousing link, an interface sharing link and the like, so that the data quality monitoring of the whole process is realized. (2) And (5) rule configuration. And formulating a data quality evaluation general rule and a user-defined quality detection rule to realize the configuration of the detection rule and the alarm rule. (3) The detection is real-time. The online real-time diagnosis is realized, and the normalizations, the online and the real-time performances of fault discovery, fault positioning, influence analysis, problem tracing and the like of the data quality problem are realized.
Example two
The operator data quality evaluation device provided by the embodiment of the invention can execute the operator data quality evaluation method provided by any embodiment of the invention, has corresponding functional modules and beneficial effects of the execution method, can be realized in a software and/or hardware (integrated circuit) mode, and can be generally integrated in a server or terminal equipment. Fig. 2 is a schematic structural diagram of an operator data quality evaluation apparatus according to a second embodiment of the present invention. Referring to fig. 2, the quality evaluation apparatus 200 for operator data according to the embodiment of the present invention may specifically include:
an acquisition unit 210, configured to acquire MR data of a data source link, an interface machine link, a cluster link, and a shared server link, where the MR data includes: A. the number of files of the data source, the number of files of the interface machine and the number of files of the cluster; B. the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster; C. reporting the number of base stations by the cluster and reporting the number of base stations by the sharing server;
the judging unit 220 is configured to judge whether the number of the files of the data source, the number of the files of the interface machine, and the number of the files of the cluster are equal, whether the file storage size of the data source, the file storage size of the interface machine, and the file storage size of the cluster are equal, and whether the number of the cluster reporting base stations and the number of the sharing server reporting base stations are equal to the number of the working parameter base stations;
the alarm unit 230 is configured to send alarm information if a deviation between the number of files in the data source, the number of files in the interface machine, and the number of files in the cluster is greater than a first preset threshold, send alarm information if a deviation between the file storage size of the data source, the file storage size of the interface machine, and the file storage size of the cluster is greater than a second preset threshold, and send alarm information if a difference between the number of reporting base stations in the cluster and the number of reporting base stations in the shared server, which is respectively less than the number of working parameter base stations, exceeds a third preset threshold;
and the analysis unit 240 is configured to analyze whether the data of the operator has an abnormality, a link in which the abnormality occurs, and reasons for each abnormal judgment according to whether the alarm information is sent and the type frequency of the alarm information, and output a data quality evaluation result.
Optionally, the MR data further includes: D. call drop rate, imsi effective rate, agps reporting rate.
Optionally, the determining unit 220 is further configured to:
and judging whether the call drop rate is less than 0.3 percent, the imsi effective rate is more than 90 percent, and the agps reporting rate is between 2 and 5 percent.
Optionally, the alarm unit 230 is further configured to:
and if the call drop rate exceeds 0.3 percent, or the imsi effective rate is less than 90 percent, or the agps reporting rate is not between 2 and 5 percent, sending out alarm information.
Optionally, the alarm unit 230 is further configured to: if the abnormity occurs, judging: a. the number of cluster files is less than that of the interface machine files; b. the storage size of the cluster file is smaller than that of the interface machine file; c. the number of the cluster reporting base stations is less than the number of the engineering parameter base stations; and (4) judging the abnormality, namely a, b and c belong to the abnormality judgment of the cluster link, and determining that the problem occurs in the transmission process from the interface machine file to the cluster file and the cluster data is lost.
Optionally, the alarm unit 230 is further configured to: if the abnormity occurs, judging: d. and if the number of the shared service reporting base stations is less than the data volume of the engineering parameter base stations, performing source tracing analysis on the judgment d, and determining that the judgment d is abnormal due to the loss of the cluster link data.
Optionally, the analysis unit 240 is further configured to: if the data is determined to be lost, the evaluation data is basically available when the data loss rate is less than 1%.
The technical scheme of the embodiment of the invention provides powerful support for data quality management work through a data quality monitoring and guaranteeing system, and realizes the following system construction targets: (1) the monitoring is completed. And corresponding data quality monitoring means are provided for a data source link, a transmission link, an acquisition link, a processing link, a warehousing link, an interface sharing link and the like, so that the data quality monitoring of the whole process is realized. (2) And (5) rule configuration. And formulating a data quality evaluation general rule and a user-defined quality detection rule to realize the configuration of the detection rule and the alarm rule. (3) The detection is real-time. The online real-time diagnosis is realized, and the normalizations, the online and the real-time performances of fault discovery, fault positioning, influence analysis, problem tracing and the like of the data quality problem are realized.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a server according to a third embodiment of the present invention, as shown in fig. 3, the server includes a processor 310, a memory 320, an input device 330, and an output device 340; the number of the processors 310 in the server may be one or more, and one processor 310 is taken as an example in fig. 3; the processor 310, the memory 320, the input device 330 and the output device 340 in the server may be connected by a bus or other means, and the bus connection is taken as an example in fig. 3.
The memory 320 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the quality evaluation method of the operator data in the embodiment of the present invention (for example, the acquisition unit 210, the judgment unit 220, the alarm unit 230, and the analysis unit 240 in the quality evaluation device of the operator data). The processor 310 executes various functional applications of the server and data processing by executing software programs, instructions, and modules stored in the memory 320, that is, implements the above-described quality evaluation method of the operator data.
Namely:
the method comprises the following steps of firstly, acquiring MR data of a data source link, an interface machine link, a cluster link and a shared server link, wherein the MR data comprises: A. the number of files of the data source, the number of files of the interface machine and the number of files of the cluster; B. the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster; C. reporting the number of base stations by the cluster and reporting the number of base stations by the sharing server;
judging whether the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster are equal, whether the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster are equal, and whether the number of the cluster reporting base stations and the number of the sharing server reporting base stations are equal to the number of the working parameter base stations;
step three, if the deviation of the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster is larger than a first preset threshold value, alarm information is sent out, if the deviation of the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster is larger than a second preset threshold value, the alarm information is sent out, and if the difference value of the number of the cluster reporting base stations and the number of the sharing server reporting base stations which are respectively smaller than the number of the industrial parameter base stations exceeds a third preset threshold value, the alarm information is sent out;
and step four, analyzing whether the data of the operator is abnormal or not, links in which the data are abnormal and reasons of all judgment abnormalities exist according to whether the alarm information is sent or not and the type frequency of the alarm information, and outputting a data quality evaluation result.
Of course, the processor of the server provided in the embodiment of the present invention is not limited to execute the method operations described above, and may also execute the relevant operations in the method for evaluating the quality of the operator data provided in any embodiment of the present invention.
The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 320 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 320 may further include memory located remotely from processor 310, which may be connected to a server over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 330 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the server. The output device 340 may include a display device such as a display screen.
The technical scheme of the embodiment of the invention provides powerful support for data quality management work through a data quality monitoring and guaranteeing system, and realizes the following system construction targets: (1) the monitoring is completed. And corresponding data quality monitoring means are provided for a data source link, a transmission link, an acquisition link, a processing link, a warehousing link, an interface sharing link and the like, so that the data quality monitoring of the whole process is realized. (2) And (5) rule configuration. And formulating a data quality evaluation general rule and a user-defined quality detection rule to realize the configuration of the detection rule and the alarm rule. (3) The detection is real-time. The online real-time diagnosis is realized, and the normalizations, the online and the real-time performances of fault discovery, fault positioning, influence analysis, problem tracing and the like of the data quality problem are realized.
Example four
A fourth embodiment of the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, perform a method for quality assessment of operator data, the method including:
the method comprises the following steps of firstly, acquiring MR data of a data source link, an interface machine link, a cluster link and a shared server link, wherein the MR data comprises: A. the number of files of the data source, the number of files of the interface machine and the number of files of the cluster; B. the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster; C. reporting the number of base stations by the cluster and reporting the number of base stations by the sharing server;
judging whether the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster are equal, whether the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster are equal, and whether the number of the cluster reporting base stations and the number of the sharing server reporting base stations are equal to the number of the working parameter base stations;
step three, if the deviation of the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster is larger than a first preset threshold value, alarm information is sent out, if the deviation of the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster is larger than a second preset threshold value, the alarm information is sent out, and if the difference value of the number of the cluster reporting base stations and the number of the sharing server reporting base stations which are respectively smaller than the number of the industrial parameter base stations exceeds a third preset threshold value, the alarm information is sent out;
and step four, analyzing whether the data of the operator is abnormal or not, links in which the data are abnormal and reasons of all judgment abnormalities exist according to whether the alarm information is sent or not and the type frequency of the alarm information, and outputting a data quality evaluation result.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the method described above, and may also perform related operations in the method for evaluating the quality of the operator data provided by any embodiment of the present invention.
The computer-readable storage media of embodiments of the invention may take any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or terminal. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The technical scheme of the embodiment of the invention provides powerful support for data quality management work through a data quality monitoring and guaranteeing system, and realizes the following system construction targets: (1) the monitoring is completed. And corresponding data quality monitoring means are provided for a data source link, a transmission link, an acquisition link, a processing link, a warehousing link, an interface sharing link and the like, so that the data quality monitoring of the whole process is realized. (2) And (5) rule configuration. And formulating a data quality evaluation general rule and a user-defined quality detection rule to realize the configuration of the detection rule and the alarm rule. (3) The detection is real-time. The online real-time diagnosis is realized, and the normalizations, the online and the real-time performances of fault discovery, fault positioning, influence analysis, problem tracing and the like of the data quality problem are realized.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method for evaluating quality of operator data, comprising:
the method comprises the following steps of firstly, acquiring MR data of a data source link, an interface machine link, a cluster link and a shared server link, wherein the MR data comprises: A. the number of files of the data source, the number of files of the interface machine and the number of files of the cluster; B. the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster; C. reporting the number of base stations by the cluster and reporting the number of base stations by the sharing server;
judging whether the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster are equal, whether the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster are equal, and whether the number of the cluster reporting base stations and the number of the sharing server reporting base stations are equal to the number of the working parameter base stations;
step three, if the deviation of the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster is larger than a first preset threshold value, alarm information is sent out, if the deviation of the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster is larger than a second preset threshold value, the alarm information is sent out, and if the difference value of the number of the cluster reporting base stations and the number of the sharing server reporting base stations which are respectively smaller than the number of the industrial parameter base stations exceeds a third preset threshold value, the alarm information is sent out;
and step four, analyzing whether the data of the operator is abnormal or not, links in which the data are abnormal and reasons of all judgment abnormalities exist according to whether the alarm information is sent or not and the type frequency of the alarm information, and outputting a data quality evaluation result.
2. Method for quality assessment of operator data according to claim 1, characterized in that the MR data further comprises: D. call drop rate, imsi effective rate, agps reporting rate.
3. The method for evaluating the quality of the operator data according to claim 2, further comprising, in step two:
and judging whether the call drop rate is less than 0.3 percent, the imsi effective rate is more than 90 percent, and the agps reporting rate is between 2 and 5 percent.
4. The method for quality assessment of operator data according to claim 3, characterized in that in step three, further comprising:
and if the call drop rate exceeds 0.3 percent, or the imsi effective rate is less than 90 percent, or the agps reporting rate is not between 2 and 5 percent, sending out alarm information.
5. The method for evaluating the quality of operator data according to claim 1, wherein in step three, if an abnormal judgment occurs: a. the number of cluster files is less than that of the interface machine files; b. the storage size of the cluster file is smaller than that of the interface machine file; c. the number of the cluster reporting base stations is less than the number of the engineering parameter base stations; and (4) judging the abnormality, namely a, b and c belong to the abnormality judgment of the cluster link, and determining that the problem occurs in the transmission process from the interface machine file to the cluster file and the cluster data is lost.
6. The method of claim 5, wherein in step three, if the operator data is determined to be abnormal: d. and if the number of the shared service reporting base stations is less than the data volume of the engineering parameter base stations, performing source tracing analysis on the judgment d, and determining that the judgment d is abnormal due to the loss of the cluster link data.
7. The method of claim 6, wherein in step four, if data loss is determined, the assessment data is substantially available when the data loss rate is less than 1%.
8. An apparatus for quality assessment of operator data, comprising:
the acquisition unit is used for acquiring MR data of a data source link, an interface machine link, a cluster link and a shared server link, wherein the MR data comprises: A. the number of files of the data source, the number of files of the interface machine and the number of files of the cluster; B. the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster; C. reporting the number of base stations by the cluster and reporting the number of base stations by the sharing server;
the judging unit is used for judging whether the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster are equal, whether the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster are equal, and whether the number of the cluster reporting base stations and the number of the sharing server reporting base stations are equal to the number of the industrial parameter base stations;
the alarm unit is used for sending alarm information if the deviation of the number of the files of the data source, the number of the files of the interface machine and the number of the files of the cluster is larger than a first preset threshold value, sending alarm information if the deviation of the file storage size of the data source, the file storage size of the interface machine and the file storage size of the cluster is larger than a second preset threshold value, and sending alarm information if the difference value of the number of the cluster reporting base stations and the number of the sharing server reporting base stations, which is respectively smaller than the number of the industrial parameter base stations, exceeds a third preset threshold value;
and the analysis unit is used for analyzing whether the data of the operator has abnormity, abnormal links and various abnormal reasons according to the alarm information and the type frequency of the alarm information, and outputting a data quality evaluation result.
9. A server comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method for quality assessment of operator data according to any of claims 1-7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a method for quality assessment of operator data according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110367597.3A CN113064890A (en) | 2021-04-06 | 2021-04-06 | Quality evaluation method, device, server and medium for operator data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110367597.3A CN113064890A (en) | 2021-04-06 | 2021-04-06 | Quality evaluation method, device, server and medium for operator data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113064890A true CN113064890A (en) | 2021-07-02 |
Family
ID=76566025
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110367597.3A Pending CN113064890A (en) | 2021-04-06 | 2021-04-06 | Quality evaluation method, device, server and medium for operator data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113064890A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116609801A (en) * | 2023-04-04 | 2023-08-18 | 北京讯腾智慧科技股份有限公司 | Main and standby service system and method for base station observation data |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104883298A (en) * | 2015-06-12 | 2015-09-02 | 中国通信建设集团设计院有限公司 | Business quality detection method and router |
| CN109062754A (en) * | 2018-06-26 | 2018-12-21 | 平安科技（深圳）有限公司 | Data monitoring and alarm method, device, storage medium and server |
| CN109714196A (en) * | 2018-12-11 | 2019-05-03 | 中国联合网络通信集团有限公司 | Data monitoring method and platform |
| CN110913426A (en) * | 2019-12-27 | 2020-03-24 | 树蛙信息科技(南京)有限公司 | Passenger flow big data signaling acquisition point information automatic detection synchronization method |
-
2021
- 2021-04-06 CN CN202110367597.3A patent/CN113064890A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104883298A (en) * | 2015-06-12 | 2015-09-02 | 中国通信建设集团设计院有限公司 | Business quality detection method and router |
| CN109062754A (en) * | 2018-06-26 | 2018-12-21 | 平安科技（深圳）有限公司 | Data monitoring and alarm method, device, storage medium and server |
| CN109714196A (en) * | 2018-12-11 | 2019-05-03 | 中国联合网络通信集团有限公司 | Data monitoring method and platform |
| CN110913426A (en) * | 2019-12-27 | 2020-03-24 | 树蛙信息科技(南京)有限公司 | Passenger flow big data signaling acquisition point information automatic detection synchronization method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116609801A (en) * | 2023-04-04 | 2023-08-18 | 北京讯腾智慧科技股份有限公司 | Main and standby service system and method for base station observation data |
| CN116609801B (en) * | 2023-04-04 | 2023-12-22 | 北京讯腾智慧科技股份有限公司 | Main and standby service system and method for base station observation data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108763957B (en) | Database security audit system, method and server | |
| CN107294808B (en) | Interface test method, device and system | |
| EP1834444B1 (en) | Binary class based analysis and monitoring | |
| US20140245451A1 (en) | System and method for managing industrial processes | |
| CN112422344A (en) | Log abnormity warning method and device, storage medium and electronic device | |
| WO2022105318A1 (en) | Machine bench operating-state monitoring method and apparatus, storage medium, and electronic device | |
| CN105721184A (en) | Network link quality monitoring method and apparatus | |
| CN105471932A (en) | Front-end application monitoring method, front-end application and front-end application monitoring system | |
| CN113671909A (en) | Safety monitoring system and method for steel industrial control equipment | |
| CN110929896A (en) | Security analysis method and device for system equipment | |
| CN114715139A (en) | Automatic parking abnormal data acquisition method and device, storage medium and product | |
| CN116880412A (en) | Visual production management platform based on cloud | |
| CN115529595A (en) | Method, device, equipment and medium for detecting abnormity of log data | |
| CN110808856A (en) | Big data operation and maintenance method and system based on data center | |
| CN113495820A (en) | Method and device for collecting and processing abnormal information and abnormal monitoring system | |
| CN113064890A (en) | Quality evaluation method, device, server and medium for operator data | |
| CN110647417A (en) | Energy internet abnormal data processing method, device and system | |
| CN113537590A (en) | Data anomaly prediction method and system | |
| CN113760634A (en) | Data processing method and device | |
| CN110609761B (en) | Method and device for determining fault source, storage medium and electronic equipment | |
| CN116760732A (en) | Power distribution terminal data communication monitoring method and device and related equipment | |
| CN116594840A (en) | Log fault acquisition and analysis method, system, equipment and medium based on ELK | |
| CN103067101B (en) | Communication terminal testing and monitoring method and device | |
| CN115276234A (en) | Power network safety monitoring system | |
| CN113254313A (en) | Monitoring index abnormality detection method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |