CN117557007A - Quality evaluation method and system for data processing - Google Patents
Quality evaluation method and system for data processing Download PDFInfo
- Publication number
- CN117557007A CN117557007A CN202410040204.1A CN202410040204A CN117557007A CN 117557007 A CN117557007 A CN 117557007A CN 202410040204 A CN202410040204 A CN 202410040204A CN 117557007 A CN117557007 A CN 117557007A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- quality
- quality evaluation
- acquiring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 602
- 238000013441 quality evaluation Methods 0.000 title claims abstract description 282
- 238000000034 method Methods 0.000 title claims abstract description 227
- 238000004458 analytical method Methods 0.000 claims abstract description 137
- 230000008859 change Effects 0.000 claims abstract description 27
- 230000008569 process Effects 0.000 claims description 168
- 238000001303 quality assessment method Methods 0.000 claims description 70
- 238000007781 pre-processing Methods 0.000 claims description 69
- 238000011282 treatment Methods 0.000 claims description 46
- 238000012958 reprocessing Methods 0.000 claims description 19
- 239000013598 vector Substances 0.000 claims description 5
- 238000011221 initial treatment Methods 0.000 claims 1
- 238000004519 manufacturing process Methods 0.000 claims 1
- 238000011156 evaluation Methods 0.000 abstract description 39
- 238000005457 optimization Methods 0.000 abstract description 29
- 230000000694 effects Effects 0.000 description 86
- 238000012512 characterization method Methods 0.000 description 15
- 230000014509 gene expression Effects 0.000 description 12
- 230000009286 beneficial effect Effects 0.000 description 8
- 230000000007 visual effect Effects 0.000 description 6
- 238000009825 accumulation Methods 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000008092 positive effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 238000013480 data collection Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06395—Quality analysis or management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention provides a quality evaluation method and a system for data processing, and relates to the technical field of data processing. Acquiring data processing information of different processing types in a quality evaluation period, and performing quality parameter analysis based on the data types to form quality evaluation parameter data of different processing types; according to the quality evaluation parameter data of different processing types, performing processing type quality evaluation analysis to form processing type quality evaluation result data; and carrying out data processing comprehensive quality evaluation analysis according to the processing type quality evaluation analysis result data to form data processing comprehensive quality evaluation result data. According to the method, reasonable and continuous processing achievement evaluation analysis is performed based on real-time processing information of data processing, continuous basic reference information is provided for optimization of engineering data processing, and therefore timeliness and accuracy of a processing result can be further improved while the achievement of the data processing is kept stable along with the change of the engineering data.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a quality evaluation method and a system for data processing.
Background
Processing of data is an important contributing means that each data use object cannot span. Regarding the efficiency and effect of data processing, the difference in processing results is presented due to the different data processing modes, processing forms and types of processed data. Currently, enterprises in particular pay more and more attention to data processing, and the success of data processing can directly affect the interests of the enterprises. Therefore, most enterprises also gradually establish analysis and evaluation means for the data processing effect so as to implement further optimization on the data processing and improve the data processing effect.
Engineering data is important data which has high accuracy and high timeliness and needs to be reasonably analyzed and processed based on purposes. At present, most of engineering data processing is only stopped on a model for once establishing data processing, and a reasonable data processing effect evaluation mechanism is lacking in a subsequent data processing model, so that monitoring and analysis can not be continuously and effectively carried out on the data processing effect, and further basic reference data for reasonably carrying out data processing optimization is provided, and the effect of engineering data processing is greatly limited.
Therefore, the quality evaluation method and the system for data processing are designed, and reasonable and continuous processing achievement evaluation analysis is carried out based on real-time processing information of the data processing, so that continuous basic reference information is provided for optimization of engineering data processing, and timeliness and accuracy of a processing result can be further improved while the achievement of the data processing is kept stable along with the change of the engineering data, so that the method and the system are the problems to be solved urgently at present.
Disclosure of Invention
The invention aims to provide a quality evaluation method for data processing, which is characterized in that multiple types of quality evaluation parameters capable of representing the processing effect of engineering data are obtained in an evaluation period, so that the processing quality of the effect of the engineering data processing is analyzed and evaluated from different aspects of the engineering data processing, and real-time and reasonable processing effect evaluation reference information is formed in each quality evaluation aspect, and real-time and accurate basic reference data is provided for optimizing the processing effect based on different aspects of the engineering data processing. Meanwhile, the engineering data processing quality evaluation data in different aspects are synthesized, the overall evaluation analysis is carried out on the engineering data processing effect on the whole, and important and accurate data reference is provided for the optimization adjustment of the follow-up data using object from the overall effect of engineering data processing and the comprehensive situation. The continuous quality evaluation analysis of the engineering data processing effect can ensure that the data use object is optimally adjusted by combining the quality evaluation result in real time under the condition that the engineering data changes along with time, so that the matching property of the engineering data processing effect and the required processing result is improved, and the timeliness and the accuracy of the processing result are further improved.
The invention also aims to provide a data processing quality evaluation system, which fully realizes the collection of the processing effect characterization information in different processing stages of engineering data processing through a data preprocessing information collection unit, a data process processing information collection unit and a data result processing information collection unit, so as to provide important, accurate and effective basic data for the follow-up targeted optimization adjustment based on the processing effect in three stages of preprocessing, process processing and result processing of engineering data processing. The quality evaluation unit is used for timely and effectively reasonably analyzing the acquired parameter data capable of representing the engineering data processing effect to form quality evaluation data capable of accurately and reasonably representing the processing effect, and visual, accurate and effective quality analysis data information is provided for the data processing optimization of the data using object.
In a first aspect, the present invention provides a quality evaluation method for data processing, including acquiring data processing information of different processing types in a quality evaluation period, and performing quality parameter analysis based on the data types to form quality evaluation parameter data of different processing types; according to the quality evaluation parameter data of different processing types, performing processing type quality evaluation analysis to form processing type quality evaluation result data; and carrying out data processing comprehensive quality evaluation analysis according to the processing type quality evaluation analysis result data to form data processing comprehensive quality evaluation result data.
According to the method, multiple types of quality evaluation parameters capable of representing the processing effect of the engineering data are obtained in an evaluation period, so that the processing quality of the processing effect of the engineering data is analyzed and evaluated from different aspects of the processing of the engineering data, and then real-time and reasonable processing effect evaluation reference information is formed in each quality evaluation aspect, and real-time and accurate basic reference data is provided for optimizing the processing effect-based on different aspects of the processing of the engineering data. Meanwhile, the engineering data processing quality evaluation data in different aspects are synthesized, the overall evaluation analysis is carried out on the engineering data processing effect on the whole, and important and accurate data reference is provided for the optimization adjustment of the follow-up data using object from the overall effect of engineering data processing and the comprehensive situation. The continuous quality evaluation analysis of the engineering data processing effect can ensure that the data use object is optimally adjusted by combining the quality evaluation result in real time under the condition that the engineering data changes along with time, so that the matching property of the engineering data processing effect and the required processing result is improved, and the timeliness and the accuracy of the processing result are further improved.
As one possible implementation manner, acquiring data processing information of different processing types in a quality evaluation period, and performing quality parameter analysis based on the data types to form quality evaluation parameter data of different processing types, including: acquiring data preprocessing information in a quality evaluation period, and analyzing preprocessing quality parameters based on data types to form data preprocessing quality evaluation parameter data; acquiring data processing process information in a quality evaluation period, and analyzing process processing quality parameters based on data types to form data processing quality evaluation parameter data; and acquiring data result processing information in the quality evaluation period, and analyzing result processing quality parameters based on the data types to form data result processing quality evaluation parameter data.
In the present invention, it is understood that, in general, for engineering data, the acquisition of engineering data is subjected to an acquisition phase of raw data, an analysis processing phase of raw processing, and a utilization phase of processing result data. The three changing stages of the engineering data just correspond to and match with the three stages of the data processing, namely, the preprocessing stage of the data corresponds to the original data acquisition stage of the engineering data, and the original data is mainly acquired by performing primary screening, acquisition frequency, acquisition conditions and the like on the acquisition of the original data to acquire more reasonable and accurate original engineering data so as to ensure the accuracy and rationality of the subsequent data processing result data. The quality evaluation analysis of the pretreatment stage is beneficial to optimizing and adjusting the acquisition of the original treatment, and the acquired original engineering data is ensured to meet the requirements of engineering data processing purposes. The process processing stage of the data, namely the process processing stage corresponding to the engineering data, is mainly formed based on the purpose of using the engineering data, namely the quality evaluation analysis of the process processing stage of the engineering data can be used for improving the accuracy and rationality of the process processing result in the follow-up process in consideration of the aspects of meeting the purpose precision, the data quantity, the integrity of the data information and the like of the engineering data, which are required to be achieved by the engineering data processing. The result processing stage of the data corresponds to the result processing stage of the engineering processing, the stage mainly obtains data information meeting the target requirement from the result data, and the quality evaluation analysis of the data processing stage is beneficial to improving the accuracy of the data result and ensuring that the processing result fully meets the requirement on the engineering data. It should be noted that, for the data processing of engineering data, due to the characteristics of high timeliness and accuracy of engineering data, when quality evaluation analysis is performed, consideration of evaluation span needs to be performed from a time level, so that the quality evaluation period can be reasonably determined according to actual analysis needs. In addition, for engineering data, the result data formed by data processing may be formed by comprehensive processing action of data of different types in the early stage, but the processing process and effect of the data of different types are greatly different, so that when quality evaluation analysis is performed, quality analysis and evaluation of the processing effect are required to be performed separately according to different data types, so that the accuracy and rationality of the quality analysis and evaluation result are ensured.
As one possible implementation manner, acquiring data preprocessing information in a quality evaluation period, and performing preprocessing quality parameter analysis based on a data type to form data preprocessing quality evaluation parameter data, including: acquisition quality assessment periodData preprocessing amount information of different types of data is obtained, and the following quality parameter analysis based on preprocessing efficiency is carried out: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodRaw data volume collected inAnd determining the original data volume acquisition efficiency of the corresponding type of dataWherein n represents the number of different types of data; raw data volume acquisition efficiency for acquiring different types of dataForming preprocessing efficiency quality evaluation parameters of different types of data; acquisition quality assessment periodAnd effectively preprocessing data information of different types of data, and analyzing the following quality parameters based on the preprocessing effective throughput: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodRaw data volume collected inThe amount of effective utilization data to be effectively utilized by the effective processingAnd determining the effective data utilization rate of the corresponding type of data The method comprises the steps of carrying out a first treatment on the surface of the Obtaining effective data utilization of different types of dataForming quality evaluation parameters of effective pretreatment throughput of different types of data; acquisition quality assessment periodProcess item change information of different types of data in the data processing system, and the following quality parameter analysis based on the pretreatment process item change is performed: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodPretreatment item with internal increase and decreaseAnd quality assessment periodInitial processing item total amount of (a)And determining the throughput change rate of the corresponding type of data according to the following formulaWherein, the method comprises the steps of, wherein,during quality assessment period for pre-processing itemThe number of change vectors in which an increase and decrease occur; acquiring throughput rate of change of different types of dataThe preprocessing effective processing items that form the different types of data vary the quality assessment parameters.
In the invention, three quality evaluation parameters are mainly considered for the purpose and effect of combining the quality evaluation analysis of the engineering data preprocessing stage with the engineering data processing. The first is the collection efficiency of the original engineering data, and because the data processing adopts a high-efficiency informatization processing means, the factor limiting the processing efficiency of the original engineering data is the collection amount of the original engineering data, so that the analysis of the collection efficiency of the original engineering data is an important aspect for representing the quality of the pretreatment effect. And secondly, the ratio of the processed and utilized original engineering data reflects the capability of acquiring effective original engineering data in a data preprocessing stage, and the acquisition of the effective original engineering data provides enough original data accumulation for the subsequent data analysis meeting the processing requirement, so that the information acquisition of the acquisition quantity of the effective original engineering data is an important quality representation of the engineering processing in the convenience of data validity. The third aspect is a change in processing items of the pre-process, and it is understood that when real-time consideration is made as to whether the processing result meets the requirement of the use of the engineering data based on the engineering data processing performed based on the purpose of the use of the engineering data, flow items for the engineering data processing are increased or decreased due to comparison of the processing result with the requirement, and these flow items are not considered or are redundant in the early stage of the engineering data processing, and most of these processing items are effective in the pre-process, and flow breaking of data information occurs due to the relevance of the data processing in the subsequent process and the result processing, so that analysis of the change in processing items performed in the pre-process is an important characterization of quality analysis in terms of redundancy and resource utilization of the engineering data processing sufficiently performed. Of course, for the process item to increase and decrease during a reasonable quality assessment period of engineering data processing, the rate of change of the process item is a combination of the overall change effect in the period formed after the increase and decrease based on the number change, and the combination is characterized in the form of positive and negative vectors. Such a representation is also based on the fact that the possible reduction of processing items is due to the occurrence of redundant and invalid processing items, and the increasing is due to the occurrence of insufficient data, but in general, the use requirement of the engineering data processing result is unchanged, so that the increase and decrease are merely the adjustment of the processing manner and condition inside the engineering data processing, and the increase and decrease are reasonably represented after synthesis to reflect the quality effect of the engineering data preprocessing as a whole.
As one possible implementation manner, performing a process type quality evaluation analysis according to quality evaluation parameter data of different process types to form process type quality evaluation result data, including: acquiring original data volume acquisition efficiency of all types of dataEffective data utilization rateThroughput rate of changeAnd determining a data preprocessing quality evaluation parameter according to the following formula：Wherein, the method comprises the steps of, wherein,、andAnd (5) an influence factor of the corresponding quality evaluation parameter.
In the invention, it can be understood that the quality characterization of the pretreatment of the engineering data is carried out through the acquisition efficiency of the original engineering data, the effective data acquisition efficiency of the original engineering data and the change of the treatment items, and the quality evaluation of the pretreatment stage of the engineering data is objectively and directly carried out from different aspects. Therefore, three quality analysis parameters are utilized to carry out quality evaluation of the pretreatment of engineering data, and the influence of the three quality parameters on the quality evaluation of the pretreatment stage can be determined according to actual conditions by utilizing the correlation influence factors. After all, the different data use objects are different in the aspect of focusing on the process optimization of engineering data based on the processing result by taking the quality evaluation data into consideration, and the influence factors can selectively highlight the quality aspect focused by the different data objects, so that the optimization adjustment is more adaptive, and the processing result requirement of the data use objects is met.
As one possible implementation manner, acquiring data processing process information in a quality evaluation period, and performing process processing quality parameter analysis based on a data type to form data processing quality evaluation parameter data, including: acquisition quality assessment periodData process throughput information of different types of data in the process, and carrying out the following quality parameter analysis based on process processing time length: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodTotal duration of single processing for each processing item in the listAcquiring the sum of single effective processing time lengths of all nodes corresponding to each processing item under the corresponding type data to form an effective processing total time lengthAnd determining the effective processing time efficiency of the corresponding type dataWherein m represents the number of the processing item under the type data with the number n; efficient process duration for obtaining different types of dataProcess processing effectiveness for forming different types of dataQuality evaluation parameters of the processing time length; acquisition quality assessment periodData process throughput information of different types of data in the node processing efficiency based on the data processing efficiency is analyzed, wherein the following quality parameters are analyzed: for the same type of data, acquiring the corresponding type of data in a quality evaluation period Data volume processed by each node in the networkAnd nodes in quality evaluation periodIs not limited to the total effective processing time of (1)And determining node processing efficiency of the corresponding type of dataThe method comprises the steps of carrying out a first treatment on the surface of the Node processing efficiency for acquiring different types of dataForming process processing node processing efficiency quality evaluation parameters of different types of data; acquisition quality assessment periodData process throughput information of different types of data, and the following quality parameter analysis based on the process intermediate data generation amount is performed: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodAverage amount of intermediate data generated by internal processingAnd determining the intermediate data volume generation rate of the corresponding type of dataThe method comprises the steps of carrying out a first treatment on the surface of the Obtaining intermediate data volume generation rate of different types of dataThe process of forming the different types of data processes intermediate data generation amount quality evaluation parameters.
In the present invention, three aspects of quality assessment analysis are mainly considered for the engineering data process processing stage. The first aspect is the duration validity of the process phase of the engineering process for the engineering data process. After the original engineering data is processed, the engineering data is processed in different levels and stages to extract important engineering data information and characteristics in the original engineering data, wherein for the engineering data processing, the step-by-step processing of the original engineering data information can be orderly and efficiently performed is an ideal condition for the engineering data process processing, but the conditions of data accumulation, long processing time and the like caused by low processing efficiency of certain processing nodes in the processing process are considered to influence the quality of the true data processing process, so that the evaluation analysis of the processing time length is an important quality evaluation aspect for representing the processing efficiency of the engineering data process. The second aspect is node processing efficiency of engineering data process processing, it can be understood that for processing duration efficiency, the relevance characterization between the processing stage and the processing level set in engineering data process processing is reflected on the whole, whether the configuration of the processing steps is reasonable or not is reflected, after all, the processing of engineering data has various processing expressions, for example, the acquisition of power data can be obtained on the basis of step-by-step acquisition of current, voltage and time, or on the basis of direct acquisition of efficacy, and the two modes are different in step setting on the process, so that consumed durations are different. The processing efficiency of the node is the quality representation of the processing speed of the node, the deep influencing factor can be the intensity of information resources, the reasonable programming of engineering processing and other aspects, and the analysis of the processing efficiency of the node is beneficial to optimizing the setting of the node. The third aspect is the throughput of intermediate data, which for informationized engineering data processing affects the operation of the system and the use of resources, particularly space resources, which is an important quality characterization of engineering data process processing in terms of operating efficiency and resource usage. Here, the intermediate data may be expressed in consideration of space occupation units due to various types of data that may be generated, so that the occupation size may be more intuitively represented, and of course, the intermediate data output may fluctuate during the evaluation period, so that the representation in terms of average amount may be more reasonable.
As one possible implementation manner, performing a process type quality evaluation analysis according to quality evaluation parameter data of different process types to form process type quality evaluation result data, including: obtaining efficient processing duration efficiency for all types of dataNode processing efficiencyIntermediate data volume generation rateAnd determining a data process processing quality assessment parameter according to the following formula：Wherein, the method comprises the steps of, wherein,、andAnd (5) an influence factor of the corresponding quality evaluation parameter.
In the invention, similarly, for the comprehensive quality characterization of engineering data process processing by taking into consideration the effective duration evaluation parameter, the node processing efficiency evaluation parameter and the intermediate data output evaluation parameter, in order to more adaptively characterize different data applicable objects, the quality reaction can be targeted by utilizing the influence factors to carry out weight distribution according to actual conditions. It should be noted that, when combining different aspects of the quality assessment parameters, it is considered that the expression has uniform directionality, that is, the expression in the formula has a positive effect on uniform increase, and uniform decrease has a negative effect, which is why the intermediate data output adopts negative expression.
As one possible implementation manner, performing a process type quality evaluation analysis according to quality evaluation parameter data of different process types to form process type quality evaluation result data, including: acquisition quality assessment periodAnd (3) data result processing amount information of different types of data, and carrying out the following quality parameter analysis based on the result processing utilization rate: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodThe effective utilized amount of the result data formed thereinAnd determining the data utilization efficiency of the corresponding type of dataThe method comprises the steps of carrying out a first treatment on the surface of the Data utilization efficiency for acquiring different types of dataForming result processing utilization rate quality evaluation parameters of different types of data; acquisition quality assessment periodAnd (3) data result processing amount information of different types of data, and carrying out the following quality parameter analysis based on the result processing data reprocessing rate: for the same type of data, obtain the correspondingType data during quality assessment periodTotal amount of internal result dataAmount of reprocessed result dataAnd determining the data reprocessing rate of the corresponding type of dataThe method comprises the steps of carrying out a first treatment on the surface of the Data reprocessing rate for acquiring different types of dataAnd forming quality evaluation parameters of the reprocessing rate of the result processing data of different types of data.
In the invention, two main aspects are considered for quality evaluation analysis of engineering data result processing. The first aspect is the utilization rate at which the result data formed by the processing can be effectively utilized. It will be appreciated that most of engineering data is continuous data formed by analyzing collected discrete data, so that all result data formed after engineering data processing may have selective utilization by data applicable objects, and thus the rest data is subjected to invalid processing, and resources are wasted, so that the utilization rate of the result data is an important representation of the engineering data result processing in terms of quality of success, and after all, the most ideal state is that each result formed by processing is effectively used. The second aspect is the case of secondary processing of the result data. Considering that the data processing modes aiming at engineering data are various, the formed result data have a certain gap in matching the requirements of data users, so that certain reprocessing is needed, and the reprocessing is the quality representation of the processing modes.
As a possible implementation manner, according to the quality evaluation parameter data of different processing types, processing type quality evaluation analysis is performed to form processing type quality evaluation Result data, comprising: obtaining data utilized efficiency of all types of dataAnd data reprocessing rateAnd determining a data result processing quality assessment parameter according to the following formula：Wherein, the method comprises the steps of, wherein,andand (5) an influence factor of the corresponding quality evaluation parameter.
In the invention, the influence factors are adopted to establish reasonable correlation expression of the quality analysis parameters, and the uniformity of the quality analysis parameters to the expression direction of the quality evaluation result is ensured to be more visual and accurate for effective quality evaluation of engineering data result processing.
As one possible implementation manner, performing data processing comprehensive quality assessment analysis according to the processing type quality assessment analysis result data to form data processing comprehensive quality assessment result data, including: respectively acquiring data preprocessing quality evaluation parametersData process quality assessment parametersData result processing quality assessment parametersAnd determining a data processing integrated quality assessment parameter according to the following formula：Wherein, the method comprises the steps of, wherein,、andEfficient weighting of parameters for quality assessment of corresponding data processing procedures, and。
in the present invention, after establishing reasonable quality assessment for different aspects of engineering data processing, the engineering data processing is considered to be a compact whole, and usually, when the data user performs optimization adjustment on the whole, the data user needs to consider not only different processing stages but also self conditions. There is also a contrast adjustment analysis of the same type of processing system. It is therefore necessary to conduct quality assessment analysis as a whole. In comprehensive quality evaluation analysis, the conditions of different data using objects are considered, and the weight is adopted to carry out the adjustment according to the prominent aspect of the actual condition, so that the analysis optimization adjustment is more adaptive.
In a second aspect, the present invention provides a quality evaluation system for data processing, which is applied to the quality evaluation method for data processing in the first aspect, and includes a data preprocessing information acquisition unit, configured to acquire quality parameter information of a preprocessing part in a data processing process; the data process processing information acquisition unit is used for acquiring quality parameter information of a process processing part in the data processing process; the data result processing information acquisition unit is used for acquiring quality parameter information of a result processing part in the data processing process; the quality evaluation unit is used for acquiring the quality parameter information of the pretreatment part acquired by the data pretreatment information acquisition unit, carrying out pretreatment quality parameter analysis to form pretreatment quality evaluation parameter data, acquiring the quality parameter information of the process treatment part acquired by the data process treatment information acquisition unit, carrying out process treatment quality parameter analysis to form process treatment quality evaluation parameter data, acquiring the quality parameter information of the result treatment part acquired by the data result treatment information acquisition unit, carrying out result treatment quality parameter analysis to form result treatment quality evaluation parameter data, respectively acquiring the pretreatment quality evaluation parameter data, the process treatment quality evaluation parameter data and the result treatment quality evaluation parameter data, and carrying out comprehensive quality evaluation analysis to form comprehensive quality evaluation result data.
According to the invention, the system fully realizes the collection of the processing effect characterization information in different processing stages of engineering data processing through the data preprocessing information collection unit, the data process processing information collection unit and the data result processing information collection unit, so that important, accurate and effective basic data is provided for the follow-up targeted optimization adjustment based on the processing effect in three stages of preprocessing, process processing and result processing of engineering data processing. The quality evaluation unit is used for timely and effectively reasonably analyzing the acquired parameter data capable of representing the engineering data processing effect to form quality evaluation data capable of accurately and reasonably representing the processing effect, and visual, accurate and effective quality analysis data information is provided for the data processing optimization of the data using object.
The quality evaluation method and the system for data processing provided by the invention have the beneficial effects that:
according to the method, multiple types of quality evaluation parameters capable of representing the processing effect of the engineering data are obtained in an evaluation period, so that the processing quality of the processing effect of the engineering data is analyzed and evaluated from different aspects of the processing of the engineering data, and then real-time and reasonable processing effect evaluation reference information is formed in each quality evaluation aspect, and real-time and accurate basic reference data is provided for optimizing the processing effect-based on different aspects of the processing of the engineering data. Meanwhile, the engineering data processing quality evaluation data in different aspects are synthesized, the overall evaluation analysis is carried out on the engineering data processing effect on the whole, and important and accurate data reference is provided for the optimization adjustment of the follow-up data using object from the overall effect of engineering data processing and the comprehensive situation. The continuous quality evaluation analysis of the engineering data processing effect can ensure that the data use object is optimally adjusted by combining the quality evaluation result in real time under the condition that the engineering data changes along with time, so that the matching property of the engineering data processing effect and the required processing result is improved, and the timeliness and the accuracy of the processing result are further improved.
The system fully realizes the collection of the processing effect characterization information in different processing stages of engineering data processing through the data preprocessing information collection unit, the data process processing information collection unit and the data result processing information collection unit, so as to provide important, accurate and effective basic data for the targeted optimization adjustment based on the processing effect in the three stages of the follow-up preprocessing, the process processing and the result processing of the engineering data processing. The quality evaluation unit is used for timely and effectively reasonably analyzing the acquired parameter data capable of representing the engineering data processing effect to form quality evaluation data capable of accurately and reasonably representing the processing effect, and visual, accurate and effective quality analysis data information is provided for the data processing optimization of the data using object.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments of the present invention will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and should not be considered as limiting the scope, and other related drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a step diagram of a quality evaluation method for data processing according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention.
Processing of data is an important contributing means that each data use object cannot span. Regarding the efficiency and effect of data processing, the difference in processing results is presented due to the different data processing modes, processing forms and types of processed data. Currently, enterprises in particular pay more and more attention to data processing, and the success of data processing can directly affect the interests of the enterprises. Therefore, most enterprises also gradually establish analysis and evaluation means for the data processing effect so as to implement further optimization on the data processing and improve the data processing effect.
Engineering data is important data which has high accuracy and high timeliness and needs to be reasonably analyzed and processed based on purposes. At present, most of engineering data processing is only stopped on a model for once establishing data processing, and a reasonable data processing effect evaluation mechanism is lacking in a subsequent data processing model, so that monitoring and analysis can not be continuously and effectively carried out on the data processing effect, and further basic reference data for reasonably carrying out data processing optimization is provided, and the effect of engineering data processing is greatly limited.
Referring to fig. 1, an embodiment of the present invention provides a quality evaluation method for data processing, where multiple types of quality evaluation parameters capable of characterizing an effect of processing engineering data are obtained in an evaluation period, so that analysis and evaluation of processing quality are performed on the effect of processing engineering data from different aspects of processing engineering data, and real-time and reasonable processing effect evaluation reference information is formed in each quality evaluation aspect, so that real-time and accurate basic reference data is provided for optimizing the different aspects of processing engineering data based on the processing effect. Meanwhile, the engineering data processing quality evaluation data in different aspects are synthesized, the overall evaluation analysis is carried out on the engineering data processing effect on the whole, and important and accurate data reference is provided for the optimization adjustment of the follow-up data using object from the overall effect of engineering data processing and the comprehensive situation. The continuous quality evaluation analysis of the engineering data processing effect can ensure that the data use object is optimally adjusted by combining the quality evaluation result in real time under the condition that the engineering data changes along with time, so that the matching property of the engineering data processing effect and the required processing result is improved, and the timeliness and the accuracy of the processing result are further improved.
The quality evaluation method for data processing specifically comprises the following steps:
s1: and acquiring data processing information of different processing types in the quality evaluation period, and analyzing quality parameters based on the data types to form quality evaluation parameter data of different processing types.
Acquiring data processing information of different processing types in a quality evaluation period, and analyzing quality parameters based on the data types to form quality evaluation parameter data of different processing types, wherein the data processing information comprises: acquiring data preprocessing information in a quality evaluation period, and analyzing preprocessing quality parameters based on data types to form data preprocessing quality evaluation parameter data; acquiring data processing process information in a quality evaluation period, and analyzing process processing quality parameters based on data types to form data processing quality evaluation parameter data; and acquiring data result processing information in the quality evaluation period, and analyzing result processing quality parameters based on the data types to form data result processing quality evaluation parameter data.
It will be appreciated that for engineering data, typically the acquisition of engineering data will go through the acquisition phase of raw data, the analysis processing phase of raw processing, and the utilization phase of processing result data. The three changing stages of the engineering data just correspond to and match with the three stages of the data processing, namely, the preprocessing stage of the data corresponds to the original data acquisition stage of the engineering data, and the original data is mainly acquired by performing primary screening, acquisition frequency, acquisition conditions and the like on the acquisition of the original data to acquire more reasonable and accurate original engineering data so as to ensure the accuracy and rationality of the subsequent data processing result data. The quality evaluation analysis of the pretreatment stage is beneficial to optimizing and adjusting the acquisition of the original treatment, and the acquired original engineering data is ensured to meet the requirements of engineering data processing purposes. The process processing stage of the data, namely the process processing stage corresponding to the engineering data, is mainly formed based on the purpose of using the engineering data, namely the quality evaluation analysis of the process processing stage of the engineering data can be used for improving the accuracy and rationality of the process processing result in the follow-up process in consideration of the aspects of meeting the purpose precision, the data quantity, the integrity of the data information and the like of the engineering data, which are required to be achieved by the engineering data processing. The result processing stage of the data corresponds to the result processing stage of the engineering processing, the stage mainly obtains data information meeting the target requirement from the result data, and the quality evaluation analysis of the data processing stage is beneficial to improving the accuracy of the data result and ensuring that the processing result fully meets the requirement on the engineering data. It should be noted that, for the data processing of engineering data, due to the characteristics of high timeliness and accuracy of engineering data, when quality evaluation analysis is performed, consideration of evaluation span needs to be performed from a time level, so that the quality evaluation period can be reasonably determined according to actual analysis needs. In addition, for engineering data, the result data formed by data processing may be formed by comprehensive processing action of data of different types in the early stage, but the processing process and effect of the data of different types are greatly different, so that when quality evaluation analysis is performed, quality analysis and evaluation of the processing effect are required to be performed separately according to different data types, so that the accuracy and rationality of the quality analysis and evaluation result are ensured.
The method comprises the steps of acquiring data preprocessing information in a quality evaluation period, and analyzing preprocessing quality parameters based on data types to form data preprocessing quality evaluation parameter data, wherein the method comprises the following steps: acquisition quality assessment periodData preprocessing amount information of different types of data is obtained, and the following quality parameter analysis based on preprocessing efficiency is carried out: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodRaw data volume collected inAnd determining the original data volume acquisition efficiency of the corresponding type of dataWherein n represents the number of different types of data; raw data volume acquisition efficiency for acquiring different types of dataForming preprocessing efficiency quality evaluation parameters of different types of data; acquisition quality assessment periodAnd effectively preprocessing data information of different types of data, and analyzing the following quality parameters based on the preprocessing effective throughput: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodRaw data volume collected inThe amount of effective utilization data to be effectively utilized by the effective processingAnd determining the effective data utilization rate of the corresponding type of dataThe method comprises the steps of carrying out a first treatment on the surface of the Obtaining effective data utilization of different types of data Forming quality evaluation parameters of effective pretreatment throughput of different types of data; acquisition quality assessment periodChange information of processing items of different types of data in the processing items, and thenThe following quality parameter analysis based on pretreatment process item variation: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodPretreatment item with internal increase and decreaseAnd quality assessment periodInitial processing item total amount of (a)And determining the throughput change rate of the corresponding type of data according to the following formulaWherein, the method comprises the steps of, wherein,during quality assessment period for pre-processing itemThe number of change vectors in which an increase and decrease occur; acquiring throughput rate of change of different types of dataThe preprocessing effective processing items that form the different types of data vary the quality assessment parameters.
For the purpose and effect of combining the quality evaluation analysis of the engineering data preprocessing stage with the engineering data processing, three quality evaluation parameters are mainly considered. The first is the collection efficiency of the original engineering data, and because the data processing adopts a high-efficiency informatization processing means, the factor limiting the processing efficiency of the original engineering data is the collection amount of the original engineering data, so that the analysis of the collection efficiency of the original engineering data is an important aspect for representing the quality of the pretreatment effect. And secondly, the ratio of the processed and utilized original engineering data reflects the capability of acquiring effective original engineering data in a data preprocessing stage, and the acquisition of the effective original engineering data provides enough original data accumulation for the subsequent data analysis meeting the processing requirement, so that the information acquisition of the acquisition quantity of the effective original engineering data is an important quality representation of the engineering processing in the convenience of data validity. The third aspect is a change in processing items of the pre-process, and it is understood that when real-time consideration is made as to whether the processing result meets the requirement of the use of the engineering data based on the engineering data processing performed based on the purpose of the use of the engineering data, flow items for the engineering data processing are increased or decreased due to comparison of the processing result with the requirement, and these flow items are not considered or are redundant in the early stage of the engineering data processing, and most of these processing items are effective in the pre-process, and flow breaking of data information occurs due to the relevance of the data processing in the subsequent process and the result processing, so that analysis of the change in processing items performed in the pre-process is an important characterization of quality analysis in terms of redundancy and resource utilization of the engineering data processing sufficiently performed. Of course, for the process item to increase and decrease during a reasonable quality assessment period of engineering data processing, the rate of change of the process item is a combination of the overall change effect in the period formed after the increase and decrease based on the number change, and the combination is characterized in the form of positive and negative vectors. Such a representation is also based on the fact that the possible reduction of processing items is due to the occurrence of redundant and invalid processing items, and the increasing is due to the occurrence of insufficient data, but in general, the use requirement of the engineering data processing result is unchanged, so that the increase and decrease are merely the adjustment of the processing manner and condition inside the engineering data processing, and the increase and decrease are reasonably represented after synthesis to reflect the quality effect of the engineering data preprocessing as a whole.
Acquiring data processing process information in a quality evaluation period, and analyzing process processing quality parameters based on data types to form data processing quality evaluation parameter data, wherein the data processing quality evaluation parameter data comprises: acquisition quality assessment periodData process throughput information of different types of data in the process, and carrying out the following quality parameter analysis based on process processing time length: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodTotal duration of single processing for each processing item in the listAcquiring the sum of single effective processing time lengths of all nodes corresponding to each processing item under the corresponding type data to form an effective processing total time lengthAnd determining the effective processing time efficiency of the corresponding type dataWherein m represents the number of the processing item under the type data with the number n; efficient process duration for obtaining different types of dataForming process processing effective processing duration quality evaluation parameters of different types of data; acquisition quality assessment periodData process throughput information of different types of data in the node processing efficiency based on the data processing efficiency is analyzed, wherein the following quality parameters are analyzed: for the same type of data, acquiring the corresponding type of data in a quality evaluation period Data volume processed by each node in the networkAnd nodes in quality evaluation periodIs not limited to the total effective processing time of (1)And determining node processing efficiency of the corresponding type of dataThe method comprises the steps of carrying out a first treatment on the surface of the Node processing efficiency for acquiring different types of dataForming process processing node processing efficiency quality evaluation parameters of different types of data; acquisition quality assessment periodData process throughput information of different types of data, and the following quality parameter analysis based on the process intermediate data generation amount is performed: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodAverage amount of intermediate data generated by internal processingAnd determining the intermediate data volume generation rate of the corresponding type of dataThe method comprises the steps of carrying out a first treatment on the surface of the Obtaining intermediate data volume generation rate of different types of dataThe process of forming the different types of data processes intermediate data generation amount quality evaluation parameters.
Three aspects of quality assessment analysis are mainly considered for the engineering data process processing stage. The first aspect is the duration validity of the process phase of the engineering process for the engineering data process. After the original engineering data is processed, the engineering data is processed in different levels and stages to extract important engineering data information and characteristics in the original engineering data, wherein for the engineering data processing, the step-by-step processing of the original engineering data information can be orderly and efficiently performed is an ideal condition for the engineering data process processing, but the conditions of data accumulation, long processing time and the like caused by low processing efficiency of certain processing nodes in the processing process are considered to influence the quality of the true data processing process, so that the evaluation analysis of the processing time length is an important quality evaluation aspect for representing the processing efficiency of the engineering data process. The second aspect is node processing efficiency of engineering data process processing, it can be understood that for processing duration efficiency, the relevance characterization between the processing stage and the processing level set in engineering data process processing is reflected on the whole, whether the configuration of the processing steps is reasonable or not is reflected, after all, the processing of engineering data has various processing expressions, for example, the acquisition of power data can be obtained on the basis of step-by-step acquisition of current, voltage and time, or on the basis of direct acquisition of efficacy, and the two modes are different in step setting on the process, so that consumed durations are different. The processing efficiency of the node is the quality representation of the processing speed of the node, the deep influencing factor can be the intensity of information resources, the reasonable programming of engineering processing and other aspects, and the analysis of the processing efficiency of the node is beneficial to optimizing the setting of the node. The third aspect is the throughput of intermediate data, which for informationized engineering data processing affects the operation of the system and the use of resources, particularly space resources, which is an important quality characterization of engineering data process processing in terms of operating efficiency and resource usage. Here, the intermediate data may be expressed in consideration of space occupation units due to various types of data that may be generated, so that the occupation size may be more intuitively represented, and of course, the intermediate data output may fluctuate during the evaluation period, so that the representation in terms of average amount may be more reasonable.
According to the quality evaluation parameter data of different processing types, performing processing type quality evaluation analysis to form processing type quality evaluation result data, including: acquisition quality assessment periodData result of different types of data in a memoryThe quality information and the following quality parameter analysis based on the result processing utilization rate are carried out: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodThe effective utilized amount of the result data formed thereinAnd determining the data utilization efficiency of the corresponding type of dataThe method comprises the steps of carrying out a first treatment on the surface of the Data utilization efficiency for acquiring different types of dataForming result processing utilization rate quality evaluation parameters of different types of data; acquisition quality assessment periodAnd (3) data result processing amount information of different types of data, and carrying out the following quality parameter analysis based on the result processing data reprocessing rate: for the same type of data, acquiring the corresponding type of data in a quality evaluation periodTotal amount of internal result dataAmount of reprocessed result dataAnd determining the data reprocessing rate of the corresponding type of dataThe method comprises the steps of carrying out a first treatment on the surface of the Data reprocessing rate for acquiring different types of dataAnd forming quality evaluation parameters of the reprocessing rate of the result processing data of different types of data.
Two main aspects are considered for quality assessment analysis of engineering data result processing. The first aspect is the utilization rate at which the result data formed by the processing can be effectively utilized. It will be appreciated that most of engineering data is continuous data formed by analyzing collected discrete data, so that all result data formed after engineering data processing may have selective utilization by data applicable objects, and thus the rest data is subjected to invalid processing, and resources are wasted, so that the utilization rate of the result data is an important representation of the engineering data result processing in terms of quality of success, and after all, the most ideal state is that each result formed by processing is effectively used. The second aspect is the case of secondary processing of the result data. Considering that the data processing modes aiming at engineering data are various, the formed result data have a certain gap in matching the requirements of data users, so that certain reprocessing is needed, and the reprocessing is the quality representation of the processing modes.
S2: and carrying out quality evaluation analysis on the processing types according to the quality evaluation parameter data of different processing types to form processing type quality evaluation result data.
And for the engineering data preprocessing stage, carrying out processing type quality evaluation analysis according to quality evaluation parameter data of different processing types to form processing type quality evaluation result data, wherein the processing type quality evaluation result data comprises: acquiring original data volume acquisition efficiency of all types of dataEffective data utilization rateThroughput rate of changeAnd determining a data preprocessing quality evaluation parameter according to the following formula：Wherein, the method comprises the steps of, wherein,、andAnd (5) an influence factor of the corresponding quality evaluation parameter.
It can be understood that the quality characterization of the pretreatment of the engineering data is performed by the collection efficiency of the original engineering data, the effective data collection efficiency of the original engineering data and the change of the treatment items, and all three parameters are all the quality assessment of the pretreatment stage of the engineering data in an objective and direct way from different aspects. Therefore, three quality analysis parameters are utilized to carry out quality evaluation of the pretreatment of engineering data, and the influence of the three quality parameters on the quality evaluation of the pretreatment stage can be determined according to actual conditions by utilizing the correlation influence factors. After all, the different data use objects are different in the aspect of focusing on the process optimization of engineering data based on the processing result by taking the quality evaluation data into consideration, and the influence factors can selectively highlight the quality aspect focused by the different data objects, so that the optimization adjustment is more adaptive, and the processing result requirement of the data use objects is met.
For the engineering data process processing stage, according to the quality evaluation parameter data of different processing types, performing processing type quality evaluation analysis to form processing type quality evaluation result data, including: obtaining efficient processing duration efficiency for all types of dataNode processing efficiencyIntermediate data volume generation rateAnd determining a data process processing quality assessment parameter according to the following formula：Wherein, the method comprises the steps of, wherein,、andAnd (5) an influence factor of the corresponding quality evaluation parameter.
Likewise, for comprehensive quality characterization of engineering data process processing by considering the effective duration evaluation parameter, the node processing efficiency evaluation parameter and the intermediate data output evaluation parameter, in order to more adaptively characterize different data applicable objects, quality reaction can be performed in a targeted manner by using influence factors to perform weight distribution according to actual conditions. It should be noted that, when combining different aspects of the quality assessment parameters, it is considered that the expression has uniform directionality, that is, the expression in the formula has a positive effect on uniform increase, and uniform decrease has a negative effect, which is why the intermediate data output adopts negative expression.
And for the engineering data result processing stage, carrying out processing type quality evaluation analysis according to quality evaluation parameter data of different processing types to form processing type quality evaluation result data, wherein the processing type quality evaluation result data comprises: obtaining data utilized efficiency of all types of dataAnd data reprocessing rateAnd determining a data result processing quality assessment parameter according to the following formula：Wherein, the method comprises the steps of, wherein,andand (5) an influence factor of the corresponding quality evaluation parameter.
And similarly, a reasonable correlation expression of the quality analysis parameters is established by adopting the influence factors, and the uniformity of the quality analysis parameters to the expression direction of the quality evaluation result is ensured to be capable of more intuitively and accurately performing effective quality evaluation of engineering data result processing.
S3: and carrying out data processing comprehensive quality evaluation analysis according to the processing type quality evaluation analysis result data to form data processing comprehensive quality evaluation result data.
According to the processing type quality evaluation analysis result data, performing data processing comprehensive quality evaluation analysis to form data processing comprehensive quality evaluation result data, including: respectively acquiring data preprocessing quality evaluation parametersData process quality assessment parametersData result processing quality assessment parameters And determining a data processing integrated quality assessment parameter according to the following formula：Wherein, the method comprises the steps of, wherein,、andEfficient weighting of parameters for quality assessment of corresponding data processing procedures, and。
of course, after establishing reasonable quality assessment for different aspects of engineering data processing, the engineering data processing is considered to be a compact whole, and usually, when the data user performs optimization adjustment on the whole, the data user needs to consider not only different processing stages but also self conditions. There is also a contrast adjustment analysis of the same type of processing system. It is therefore necessary to conduct quality assessment analysis as a whole. In comprehensive quality evaluation analysis, the conditions of different data using objects are considered, and the weight is adopted to carry out the adjustment according to the prominent aspect of the actual condition, so that the analysis optimization adjustment is more adaptive.
The invention also provides a data processing quality evaluation system, which adopts the data processing quality evaluation method provided by the invention, and comprises a data preprocessing information acquisition unit for acquiring quality parameter information of a preprocessing part in the data processing process; the data process processing information acquisition unit is used for acquiring quality parameter information of a process processing part in the data processing process; the data result processing information acquisition unit is used for acquiring quality parameter information of a result processing part in the data processing process; the quality evaluation unit is used for acquiring the quality parameter information of the pretreatment part acquired by the data pretreatment information acquisition unit, carrying out pretreatment quality parameter analysis to form pretreatment quality evaluation parameter data, acquiring the quality parameter information of the process treatment part acquired by the data process treatment information acquisition unit, carrying out process treatment quality parameter analysis to form process treatment quality evaluation parameter data, acquiring the quality parameter information of the result treatment part acquired by the data result treatment information acquisition unit, carrying out result treatment quality parameter analysis to form result treatment quality evaluation parameter data, respectively acquiring the pretreatment quality evaluation parameter data, the process treatment quality evaluation parameter data and the result treatment quality evaluation parameter data, and carrying out comprehensive quality evaluation analysis to form comprehensive quality evaluation result data.
The system fully realizes the collection of the processing effect characterization information in different processing stages of engineering data processing through the data preprocessing information collection unit, the data process processing information collection unit and the data result processing information collection unit, so as to provide important, accurate and effective basic data for the targeted optimization adjustment based on the processing effect in the three stages of the follow-up preprocessing, the process processing and the result processing of the engineering data processing. The quality evaluation unit is used for timely and effectively reasonably analyzing the acquired parameter data capable of representing the engineering data processing effect to form quality evaluation data capable of accurately and reasonably representing the processing effect, and visual, accurate and effective quality analysis data information is provided for the data processing optimization of the data using object.
In summary, the method and the device for evaluating the quality of data processing provided by the embodiment of the invention have the beneficial effects that:
according to the method, multiple types of quality evaluation parameters capable of representing the processing effect of the engineering data are obtained in an evaluation period, so that the processing quality of the processing effect of the engineering data is analyzed and evaluated from different aspects of the processing of the engineering data, and then real-time and reasonable processing effect evaluation reference information is formed in each quality evaluation aspect, and real-time and accurate basic reference data is provided for optimizing the processing effect-based on different aspects of the processing of the engineering data. Meanwhile, the engineering data processing quality evaluation data in different aspects are synthesized, the overall evaluation analysis is carried out on the engineering data processing effect on the whole, and important and accurate data reference is provided for the optimization adjustment of the follow-up data using object from the overall effect of engineering data processing and the comprehensive situation. The continuous quality evaluation analysis of the engineering data processing effect can ensure that the data use object is optimally adjusted by combining the quality evaluation result in real time under the condition that the engineering data changes along with time, so that the matching property of the engineering data processing effect and the required processing result is improved, and the timeliness and the accuracy of the processing result are further improved.
The system fully realizes the collection of the processing effect characterization information in different processing stages of engineering data processing through the data preprocessing information collection unit, the data process processing information collection unit and the data result processing information collection unit, so as to provide important, accurate and effective basic data for the targeted optimization adjustment based on the processing effect in the three stages of the follow-up preprocessing, the process processing and the result processing of the engineering data processing. The quality evaluation unit is used for timely and effectively reasonably analyzing the acquired parameter data capable of representing the engineering data processing effect to form quality evaluation data capable of accurately and reasonably representing the processing effect, and visual, accurate and effective quality analysis data information is provided for the data processing optimization of the data using object.
In the present invention, "at least one" means one or more, and "a plurality" means two or more. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b, or c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or plural.
It should be understood that, in various embodiments of the present invention, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
Those of ordinary skill in the art will appreciate that the elements and method steps of the examples described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or as a combination of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.
In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the invention, which are described in detail and are not to be construed as limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.
Claims (10)
1. A method of quality assessment of data processing, comprising:
acquiring data processing information of different processing types in a quality evaluation period, and analyzing quality parameters based on the data types to form quality evaluation parameter data of different processing types;
performing processing type quality evaluation analysis according to the quality evaluation parameter data of different processing types to form processing type quality evaluation result data;
And carrying out data processing comprehensive quality evaluation analysis according to the processing type quality evaluation analysis result data to form data processing comprehensive quality evaluation result data.
2. The method for evaluating the quality of data processing according to claim 1, wherein the steps of acquiring the data processing information of different processing types in the quality evaluation period, and performing quality parameter analysis based on the data types to form quality evaluation parameter data of different processing types, comprise:
acquiring data preprocessing information in the quality evaluation period, and analyzing preprocessing quality parameters based on data types to form data preprocessing quality evaluation parameter data;
acquiring data processing process information in the quality evaluation period, and analyzing process processing quality parameters based on data types to form data processing quality evaluation parameter data;
and acquiring data result processing information in the quality evaluation period, and analyzing result processing quality parameters based on the data types to form data result processing quality evaluation parameter data.
3. The method for evaluating the quality of data processing according to claim 2, wherein the acquiring the data preprocessing information in the quality evaluation period and performing preprocessing quality parameter analysis based on the data type to form data preprocessing quality evaluation parameter data comprises:
Acquiring the quality assessment periodData preprocessing amount information of different types of data is obtained, and the following quality parameter analysis based on preprocessing efficiency is carried out:
for the same type of data, acquiring the corresponding type of data in the quality evaluation periodRaw data volume collected inAnd determining the original data amount acquisition efficiency of the corresponding type of data +.>Wherein n represents the number of different types of data;
acquiring the original data volume acquisition efficiency of different types of dataForming preprocessing efficiency quality evaluation parameters of different types of data;
acquiring the quality assessment periodAnd effectively preprocessing data information of different types of data, and analyzing the following quality parameters based on the preprocessing effective throughput:
for the same type of data, acquiring the corresponding type of data in the quality evaluation periodRaw data volume collected inThe effective utilization data amount of which is utilized by the effective processing +.>And determining the effective data utilization rate of the corresponding type of data；
Acquiring the effective data utilization of different types of dataForming quality evaluation parameters of effective pretreatment throughput of different types of data;
acquiring the quality assessment periodChange information of processing items of different types of data in the processing item and processing item based on the change information The following quality parameters of the pretreatment item changes were analyzed:
for the same type of data, acquiring the corresponding type of data in the quality evaluation periodPretreatment item of internal increase and decrease->And the quality assessment period->Total amount of initial treatment items->And determining the throughput change rate of the corresponding type of data according to the following formula>Wherein->For the pretreatment item +.>The number of change vectors in which an increase and decrease occur;
acquiring the throughput rate of change of different types of dataThe preprocessing effective processing items that form the different types of data vary the quality assessment parameters.
4. A quality assessment method for data processing according to claim 3, wherein said performing a process type quality assessment analysis based on said quality assessment parameter data of different process types to form process type quality assessment result data comprises:
acquiring the raw data volume of all types of dataEfficiency of collectionThe effective data utilization rate +.>Rate of change of throughput->And determining a data preprocessing quality evaluation parameter +.>：
Wherein->、/>And +.>And (5) an influence factor of the corresponding quality evaluation parameter.
5. The method of claim 4, wherein the step of obtaining the data processing process information in the quality evaluation period and performing a process quality parameter analysis based on the data type to form data process quality evaluation parameter data comprises:
Acquiring the quality assessment periodData process throughput information of different types of data in the process, and carrying out the following quality parameter analysis based on process processing time length:
for the same type of data, obtain the correspondingType data during the quality assessment periodThe total duration of a single treatment per treatment item +.>Acquiring the sum of single effective processing time lengths of all nodes corresponding to each processing item under corresponding type data to form effective processing total time length +.>And determines the effective processing time length efficiency of the corresponding type data>Wherein m represents the number of the processing item under the type data with the number n;
acquiring the effective processing time length efficiency of different types of dataForming process processing effective processing duration quality evaluation parameters of different types of data;
acquiring the quality assessment periodData process throughput information of different types of data in the node processing efficiency based on the data processing efficiency is analyzed, wherein the following quality parameters are analyzed:
for the same type of data, acquiring the corresponding type of data in the quality evaluation periodData volume processed by each node in the networkAnd nodes are +.>Is +.>And determining the node processing efficiency of the corresponding type data >；
Processing efficiency of the node for obtaining different types of dataForming process processing node processing efficiency quality evaluation parameters of different types of data;
acquiring the quality assessment periodData process throughput information of different types of data, and the following quality parameter analysis based on the process intermediate data generation amount is performed:
for the same type of data, acquiring the corresponding type of data in the quality evaluation periodAverage amount of intermediate data generated by internal processing +.>And determining intermediate data volume generation rate of corresponding type data +.>；
Acquiring the intermediate data volume generation rate of different types of dataThe process of forming the different types of data processes intermediate data generation amount quality evaluation parameters.
6. The method according to claim 5, wherein the performing a process type quality evaluation analysis based on the quality evaluation parameter data of different process types to form process type quality evaluation result data includes:
acquiring the effective processing time length efficiency of all types of dataSaid node processing efficiency->Intermediate data volume production rate +.>And determines the data process processing quality evaluation parameter +. >：
Wherein->、/>And +.>And (5) an influence factor of the corresponding quality evaluation parameter.
7. The method according to claim 6, wherein the performing a process type quality evaluation analysis based on the quality evaluation parameter data of different process types to form process type quality evaluation result data includes:
acquiring the quality assessment periodAnd (3) data result processing amount information of different types of data, and carrying out the following quality parameter analysis based on the result processing utilization rate:
for the same type of data, acquiring the corresponding type of data in the quality evaluation periodThe effective utilized amount of the result data formed therein +.>And determines data utilization efficiency of the corresponding type of data +.>；
Acquiring different types of data is utilized efficientlyForming result processing utilization rate quality evaluation parameters of different types of data;
acquiring the quality assessment periodAnd (3) data result processing amount information of different types of data, and carrying out the following quality parameter analysis based on the result processing data reprocessing rate:
for the same type of data, acquiring the corresponding type of data in the quality evaluation periodTotal amount of internal result data- >And the amount of result data reprocessed +.>And determines the data reprocessing rate of the corresponding type of data>；
Acquiring the data reprocessing rate of different types of dataAnd forming quality evaluation parameters of the reprocessing rate of the result processing data of different types of data.
8. The method according to claim 7, wherein the performing a process type quality evaluation analysis based on the quality evaluation parameter data of different process types to form process type quality evaluation result data includes:
acquiring all types of data is utilized efficientlyAnd the data reprocessing rate->And determines the data result processing quality evaluation parameter +.>：
Wherein->And->And (5) an influence factor of the corresponding quality evaluation parameter.
9. The method for evaluating the quality of a data process according to claim 8, wherein the performing a data processing integrated quality evaluation analysis based on the process type quality evaluation analysis result data to form data processing integrated quality evaluation result data comprises:
respectively acquiring the data preprocessing quality evaluation parametersThe data process quality evaluation parameter +. >Said data result processing quality assessment parameter +.>And determining a data processing integrated quality assessment parameter according to the following formula：
Wherein->、/>And +.>Effective weights of parameters are evaluated for the quality of the corresponding data processing process, and +.>。
10. A quality assessment system of data processing employing the quality assessment method of data processing according to any one of claims 1 to 9, characterized by comprising:
the data preprocessing information acquisition unit is used for acquiring quality parameter information of a preprocessing part in the data processing process;
the data process processing information acquisition unit is used for acquiring quality parameter information of a process processing part in the data processing process;
the data result processing information acquisition unit is used for acquiring quality parameter information of a result processing part in the data processing process;
the quality evaluation unit is used for acquiring the quality parameter information of the pretreatment part acquired by the data pretreatment information acquisition unit, carrying out pretreatment quality parameter analysis to form pretreatment quality evaluation parameter data, acquiring the quality parameter information of the process treatment part acquired by the data process treatment information acquisition unit, carrying out process treatment quality parameter analysis to form process treatment quality evaluation parameter data, acquiring the quality parameter information of the result treatment part acquired by the data result treatment information acquisition unit, carrying out result treatment quality parameter analysis to form result treatment quality evaluation parameter data, respectively acquiring the pretreatment quality evaluation parameter data, the process treatment quality evaluation parameter data and the result treatment quality evaluation parameter data, carrying out comprehensive quality evaluation analysis to form comprehensive quality evaluation result data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410040204.1A CN117557007B (en) | 2024-01-11 | 2024-01-11 | Quality evaluation method and system for data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410040204.1A CN117557007B (en) | 2024-01-11 | 2024-01-11 | Quality evaluation method and system for data processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117557007A true CN117557007A (en) | 2024-02-13 |
| CN117557007B CN117557007B (en) | 2024-04-26 |
Family
ID=89813251
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410040204.1A Active CN117557007B (en) | 2024-01-11 | 2024-01-11 | Quality evaluation method and system for data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117557007B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108345985A (en) * | 2018-01-09 | 2018-07-31 | 国网瑞盈电力科技（北京）有限公司 | A kind of power distribution network Data Quality Assessment Methodology and system |
| CN109583722A (en) * | 2018-11-16 | 2019-04-05 | 北京奇虎科技有限公司 | A kind of channel data analysis system and method |
| CN113159468A (en) * | 2020-01-22 | 2021-07-23 | 普天信息技术有限公司 | Energy consumption data quality comprehensive evaluation method and system |
| CN113408927A (en) * | 2021-06-30 | 2021-09-17 | 四川交达预应力工程检测科技有限公司 | Big data-based prestressed construction quality evaluation method and system |
| US20230036483A1 (en) * | 2021-07-28 | 2023-02-02 | S&P Global Inc. | Data Anomaly Forecasting From Data Record Meta-Statistics |
| CN116611712A (en) * | 2023-07-17 | 2023-08-18 | 国网信息通信产业集团有限公司 | Semantic inference-based power grid work ticket evaluation system |
-
2024
- 2024-01-11 CN CN202410040204.1A patent/CN117557007B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108345985A (en) * | 2018-01-09 | 2018-07-31 | 国网瑞盈电力科技（北京）有限公司 | A kind of power distribution network Data Quality Assessment Methodology and system |
| CN109583722A (en) * | 2018-11-16 | 2019-04-05 | 北京奇虎科技有限公司 | A kind of channel data analysis system and method |
| CN113159468A (en) * | 2020-01-22 | 2021-07-23 | 普天信息技术有限公司 | Energy consumption data quality comprehensive evaluation method and system |
| CN113408927A (en) * | 2021-06-30 | 2021-09-17 | 四川交达预应力工程检测科技有限公司 | Big data-based prestressed construction quality evaluation method and system |
| US20230036483A1 (en) * | 2021-07-28 | 2023-02-02 | S&P Global Inc. | Data Anomaly Forecasting From Data Record Meta-Statistics |
| CN116611712A (en) * | 2023-07-17 | 2023-08-18 | 国网信息通信产业集团有限公司 | Semantic inference-based power grid work ticket evaluation system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117557007B (en) | 2024-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Fullmer et al. | Analysis of difficulty control in bitcoin and proof-of-work blockchains | |
| Chatterjee et al. | Markov decision processes with multiple objectives | |
| CN109471847B (en) | I/O congestion control method and control system | |
| Panneerselvam et al. | Workload analysis for the scope of user demand prediction model evaluations in cloud environments | |
| CN112398700B (en) | Service degradation method and device, storage medium and computer equipment | |
| Gupta et al. | Big data energy management, analytics and visualization for residential areas | |
| CN110390563A (en) | Quantization method, device, computer equipment and the storage medium of user's value | |
| CN117557007B (en) | Quality evaluation method and system for data processing | |
| Zhou et al. | The role of communication time in the convergence of federated edge learning | |
| TW200809633A (en) | Method for autonomic system management using adaptive allocation of resources | |
| CN113822492A (en) | Short-term power load prediction method and device and readable storage medium | |
| CN116566696B (en) | Security assessment system and method based on cloud computing | |
| CN112416590A (en) | Server system resource adjusting method and device, computer equipment and storage medium | |
| Tang et al. | Tackling system induced bias in federated learning: Stratification and convergence analysis | |
| Abdallah et al. | Online energy-efficient resource allocation in cloud computing data centers | |
| Dai et al. | Robust design using preference aggregation methods | |
| CN113449008B (en) | Modeling method and device | |
| CN114511760A (en) | Sample equalization method, device, equipment and storage medium | |
| Ding et al. | Comat: an effective composite matching framework for content-based pub/sub systems | |
| Huh et al. | Adaptive data-driven inventory control policies based on Kaplan-Meier estimator | |
| Zhang et al. | Online model racing based on extreme performance | |
| Lin et al. | Improving robustness of learning-based adaptive video streaming in wildly fluctuating networks | |
| CN115033390B (en) | Load balancing method and device | |
| Lakzaei et al. | An optimized method for Radio and Computational Resources Allocation To IoT Users In Fog Computing | |
| Liu et al. | Crowd Learning: Improving Online Decision Making Using Crowdsourced Data. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |