CN116842240A - Data management and control system based on full-link management and control - Google Patents

Data management and control system based on full-link management and control Download PDF

Info

Publication number
CN116842240A
CN116842240A CN202311100015.0A CN202311100015A CN116842240A CN 116842240 A CN116842240 A CN 116842240A CN 202311100015 A CN202311100015 A CN 202311100015A CN 116842240 A CN116842240 A CN 116842240A
Authority
CN
China
Prior art keywords
data
scheme
processing
subset
analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202311100015.0A
Other languages
Chinese (zh)
Other versions
CN116842240B (en
Inventor
万力
韩东明
王庆焕
邢军鹏
李晓阳
刘其敏
邵龙
李冬冬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Haibo Technology Information System Co ltd
Original Assignee
Shandong Haibo Technology Information System Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong Haibo Technology Information System Co ltd filed Critical Shandong Haibo Technology Information System Co ltd
Priority to CN202311100015.0A priority Critical patent/CN116842240B/en
Publication of CN116842240A publication Critical patent/CN116842240A/en
Application granted granted Critical
Publication of CN116842240B publication Critical patent/CN116842240B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/906Clustering; Classification
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/258Data format conversion from or to a database
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/901Indexing; Data structures therefor; Storage structures
    • G06F16/9024Graphs; Linked lists

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a data management system based on full-link management and control. Relates to the field of data management, comprising: and a data classification module: the method comprises the steps of performing standardization processing on target data, and classifying the target data based on links to which the target data belong to obtain a first data set; and an interaction determining module: the method comprises the steps of obtaining a data interaction mode and interaction degree between each subset in a first data set to obtain a comprehensive interaction result; management analysis module: the data analysis and verification method comprises the steps of performing data analysis and verification on first data of each sub-set, and obtaining a first analysis sub-set based on analysis and verification results; and the data management module is used for: the method is used for matching the data treatment scheme to the first analysis subset based on the scheme database, and adjusting the data treatment scheme based on the comprehensive interaction result to obtain the comprehensive treatment scheme for data treatment. By classifying the target data of different links and carrying out data analysis and inspection, the method is matched with a precise data management scheme, and precise management and control of all-link data are realized.

Description

Data management and control system based on full-link management and control
Technical Field
The invention relates to the field of data management, in particular to a data management system based on full-link management and control.
Background
At present, with the continuous development of big data, the management and application of various data resources are increasingly strengthened by utilizing the big data. The data management by using big data is mainly controlled strictly by data quality.
However, the existing data management and control often cannot meet the requirement of accurate management of different links, so that different data quality problems occur on the data of the different links, and the final data result is affected.
Therefore, the invention provides a data management system based on full-link management and control.
Disclosure of Invention
The invention provides a data management system based on full-link management control, which is used for classifying target data of different links, carrying out data analysis and data quality inspection according to different classifications, and matching a precise data management scheme according to analysis and inspection results so as to realize precise management of full-link data.
The invention provides a data management system based on full link management and control, which comprises:
and a data classification module: the method comprises the steps of performing standardization processing on target data, and classifying the target data based on different links to which the data belong to obtain a first data set;
And an interaction determining module: the method comprises the steps of obtaining a data interaction mode and a data interaction degree between each subset in a first data set, and obtaining a comprehensive interaction result of each subset;
management analysis module: the data analysis and data quality inspection method comprises the steps of performing data analysis and data quality inspection on first data of each sub-set in a first data set, and obtaining a first analysis sub-set based on analysis inspection results;
and the data management module is used for: the method is used for matching the data treatment scheme to the first analysis subset based on the scheme database, and adjusting the data treatment scheme based on the comprehensive interaction result of the corresponding subset, so that the comprehensive treatment scheme of the target data is obtained, and the data treatment is realized.
In one possible implementation, the data classification module includes:
a data acquisition unit: all target data of the full link are acquired and imported into a data processing platform;
a data processing unit: judging whether a data sample containing a data missing value exists in the target data;
if the data exists, when the proportion of the data samples containing the data missing values to the total data samples is smaller than the preset missing proportion, eliminating the fields containing the data missing values;
when the proportion of the data samples containing the data missing values to the total data samples is smaller than the second missing proportion, the data samples containing the data missing values are removed;
And otherwise, predicting the field of the data missing value part based on the data sample, so as to complement the data missing value.
In one possible implementation, the data classification module includes:
data classification unit: the method comprises the steps of obtaining links to which each piece of data in target data belongs, and classifying the target data based on link types of the links to obtain a plurality of first initial subsets;
a second processing unit: and the data processing module is used for carrying out data scaling on the first data in the first initial subset to obtain first processing data in a preset data range, and constructing a first processing data subset based on the first processing data to obtain a first data set.
In one possible implementation, the interaction determining module includes:
a first interaction unit: the method comprises the steps of acquiring a link type corresponding to each first processing data subset in a first data set, and judging a first interaction mode and a first interaction degree between the link type corresponding to each first processing data subset and the rest link types;
a second interaction unit: the method comprises the steps of acquiring a second interaction mode and a second interaction degree between a feature type corresponding to first processing data of each first processing data subset and a feature type corresponding to the first processing data in the rest first processing data subsets one by one;
Interaction mode determining unit: the comprehensive interaction mode is used for determining each first processing data subset based on the corresponding first interaction mode and second interaction mode;
interaction degree determining unit: the method comprises the steps of determining a comprehensive interaction degree of each first processing data subset based on a corresponding first interaction degree and second interaction degree;
interaction result determining unit: and the method is used for sorting the corresponding comprehensive interaction modes and the comprehensive interaction degrees to obtain the comprehensive interaction result of each first processing subset.
In one possible implementation, the management analysis module includes:
a feature acquisition unit: the method comprises the steps of acquiring a link type of each first processing sub-set in a first data set and a data characteristic of first processing data in the first processing sub-set;
rule construction unit: the method comprises the steps of screening test rules matched with a corresponding first processing subset based on the link type and corresponding data characteristics, and constructing an initial analysis test rule library;
rule processing unit: the method comprises the steps of performing verification adjustment on an initial analysis verification rule base based on a running log corresponding to first data in a first processing subset to obtain a first analysis verification rule base;
A first rule classification unit: the method comprises the steps of performing first classification on rules in a first analysis and inspection rule base according to data integrity inspection standards to obtain a first classification rule base;
a second rule classification unit: the method comprises the steps of performing second classification on rules in a first analysis and inspection rule base according to data availability and traceability standards to obtain a second classification rule base;
minimum test judging unit: the minimum test proportion is used for determining data test based on the control precision of the current full-link control;
a first inspection unit: the first classification rule base is used for carrying out first verification on the first processing data in the first processing subset, and dividing the first processing subset into a first verified subset and a first unverified subset according to whether the first verification is carried out or not;
a second inspection unit: the first processing data processing method comprises the steps of performing a first check on first processing data in a first processing subset based on a first classification rule base, and dividing the first processing subset into a first checking subset and a first non-checking subset according to whether the first check is performed or not;
unverified data processing unit: for constructing an unverified data set based on the first unverified sub-set and the second unverified sub-set and reprocessing the unverified data;
And a checking and comparing unit: for comparing the first ratio of the first subset of tests to the first subset of treatments to a minimum ratio of tests, and simultaneously comparing the second ratio of the second subset of tests to the second subset of treatments to the minimum ratio of tests;
a test result determination unit: if the first proportion and the second proportion are both larger than the lowest checking proportion, checking results in the first checking sub-set and the second checking sub-set are obtained;
classifying the test results in the first test subset and the second test subset according to the difference of the corresponding first processing data;
archiving the inspection results based on a plurality of first processing data corresponding to each same inspection result, and establishing a corresponding inspection index;
obtaining a first analysis subset based on each inspection index and the corresponding classified archiving result;
otherwise, judging that the current first classification rule base or the second classification rule base is in error, and reconstructing the initial analysis and inspection rule base.
In one possible implementation, an unverified data processing unit comprises:
a data comparison subunit: the method comprises the steps of forming an unverified data set based on a first unverified sub-set and a second unverified sub-set, and comparing each unverified data in the verified data set with a preset data unverified reason;
A data adjustment subunit: the data adjustment method comprises the steps of carrying out data adjustment based on a data adjustment scheme corresponding to the unverified reason to obtain first adjustment data;
judging whether the first adjustment data can be subjected to a first test or a second test;
if the first adjustment data can be subjected to the first check or the second check, the first adjustment data is extracted and filled into the corresponding first check sub-set or the second check sub-set.
In one possible implementation, the data governance module includes:
scheme matching unit: the method comprises the steps of obtaining a scheme database consistent with the current link type, screening a first data treatment scheme with highest matching degree with the first analysis subset based on the scheme database, and screening a second data treatment scheme with next highest matching degree;
scheme extraction unit: the method comprises the steps of extracting a part of a first data management scheme, which is overlapped with a scheme existing in a second data management scheme, and performing scheme processing to obtain a data management scheme matched with the first analysis subset;
scheme adjusting unit: the method comprises the steps of obtaining a comprehensive interaction result corresponding to a current first processing subset, and adjusting a data treatment scheme based on the comprehensive interaction result and corresponding adjustment weights, so as to obtain a first adjustment scheme of the first processing subset;
Comprehensive scheme determination unit: the method comprises the steps of obtaining a comprehensive treatment scheme based on all first treatment subsets, and carrying out data treatment on target data based on the comprehensive treatment scheme;
and the comprehensive treatment schemes corresponding to all the first treatment subsets are the comprehensive treatment schemes of the target data.
In one possible implementation, the scheme extraction unit includes:
scheme extraction subunit: the method comprises the steps of extracting a part of a first data management scheme, which is overlapped with a scheme existing in a second data management scheme, to obtain a third data management scheme;
scheme judging subunit: the method comprises the steps of judging whether a sub-scheme in a third data management scheme can be completely executed;
if the sub-schemes which cannot be completely executed exist in the third data management scheme, extracting the sub-schemes in the first data management scheme and the second data management scheme to complement the part which cannot be completely executed, judging whether scheme conflicts exist in different complementing schemes, and if not, obtaining the current complementing scheme as a fourth data management scheme;
otherwise, the completion scheme is replaced to obtain a fourth data treatment scheme;
scheme determination subunit: and the fourth processing scheme is used for sorting the fourth processing scheme and the third processing scheme to obtain a data management scheme matched with the first analysis subset.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
The technical scheme of the invention is further described in detail through the drawings and the embodiments.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the invention, serve to explain the invention. In the drawings:
FIG. 1 is a block diagram of a data management system based on full link management and control in an embodiment of the invention;
FIG. 2 is a block diagram of an interaction determination module in an embodiment of the invention;
fig. 3 is a block diagram of a scheme extraction unit in an embodiment of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described below with reference to the accompanying drawings, it being understood that the preferred embodiments described herein are for illustration and explanation of the present invention only, and are not intended to limit the present invention.
Example 1:
The embodiment of the invention provides a data management system based on full link management and control, as shown in fig. 1, comprising:
and a data classification module: the method comprises the steps of performing standardization processing on target data, and classifying the target data based on different links to which the data belong to obtain a first data set;
and an interaction determining module: the method comprises the steps of obtaining a data interaction mode and a data interaction degree between each subset in a first data set, and obtaining a comprehensive interaction result of each subset;
management analysis module: the data analysis and data quality inspection method comprises the steps of performing data analysis and data quality inspection on first data of each sub-set in a first data set, and obtaining a first analysis sub-set based on analysis inspection results;
and the data management module is used for: the method is used for matching the data treatment scheme to the first analysis subset based on the scheme database, and adjusting the data treatment scheme based on the comprehensive interaction result of the corresponding subset, so that the comprehensive treatment scheme of the target data is obtained, and the data treatment is realized.
In this embodiment, the target data refers to data that needs to be processed in all links, for example, data in the directions of intelligent data modeling, global data integration, efficient data development, active data processing, comprehensive data security, rapid analysis service, and the like can be processed.
In this embodiment, normalization refers to removing or predicting missing values in target data.
In this embodiment, the link refers to a data transmission channel of the target data, and the links to which the data of different data transmission channels belong are different.
In this embodiment, the first data set refers to classifying the target data according to different links to which the target data belongs to obtain first data subsets, and performing data scaling on the first data in each first data subset to obtain the first data set.
In this embodiment, the data interaction manner includes: database interactions, file interactions, information interactions based on different protocols.
In this embodiment, the data exchange degree refers to the exchange degree of data exchange between different links for target data.
In this embodiment, the comprehensive interaction result refers to an interaction result of determining first data between different links according to a data interaction mode and a data interaction degree.
In this embodiment, data analysis and data quality inspection refer to constructing an analysis and inspection rule base according to data characteristics and link types, and performing analysis and inspection on corresponding data according to the analysis and inspection rule base.
In this embodiment, the first analysis subset refers to a corresponding analysis set obtained according to the analysis and inspection result and the corresponding first data.
In this embodiment, the scheme database refers to a database of corresponding abatement schemes matching different first data according to link types.
In this embodiment, the data governance scheme refers to matching the corresponding data governance scheme of each first analysis subset according to the scheme database.
In this embodiment, the comprehensive treatment scheme refers to all data treatment schemes of the target data, and the comprehensive treatment scheme is obtained after the scheme treatment.
The beneficial effects of the technical scheme are as follows: the target data of different links are classified, data analysis and data quality inspection are carried out according to the different classifications, and the accurate data management scheme is matched according to analysis and inspection results, so that the accurate management of the data of all links is realized.
Example 2:
based on embodiment 1, the data classification module includes:
a data acquisition unit: all target data of the full link are acquired and imported into a data processing platform;
a data processing unit: judging whether a data sample containing a data missing value exists in the target data;
If the data exists, when the proportion of the data samples containing the data missing values to the total data samples is smaller than the preset missing proportion, eliminating the fields containing the data missing values;
when the proportion of the data samples containing the data missing values to the total data samples is smaller than the second missing proportion, the data samples containing the data missing values are removed;
and otherwise, predicting the field of the data missing value part based on the data sample, so as to complement the data missing value.
In this embodiment, the link refers to a data transmission channel of the target data, and links to which data of different data transmission channels belong are different.
In this embodiment, the target data refers to data that needs to be managed in the full link.
In this embodiment, the data processing platform refers to a platform capable of processing target data, where a data processing manner included in the data processing platform includes: data normalization, data summarization, data addition, data classification and the like.
In this embodiment, the data missing value refers to an unknown or missing data value existing in the target data, for example, the data sample 1 in the target data is a, b, the data sample 2 is c, d, the data sample 3 is e, the data sample 4 is blank, the missing data missing value exists in the data sample 3 according to the size of each data sample, and the unknown data missing value exists in the data sample 4.
In this embodiment, the data samples refer to sub-data obtained by splitting target data according to a data size, where one data sample is one sub-data belonging to the target data, and each data sample includes a plurality of data units.
In this embodiment, the total data samples refer to all data samples contained in the target data.
In this embodiment, the preset miss ratio refers to a feature value miss ratio determined according to the data governance fineness of the target data, where the range of the preset miss ratio is (0, 1).
In this embodiment, the field containing the data missing value refers to a data unit containing the data missing value in each data sample, where one data unit is a field.
In this embodiment, the second missing proportion refers to a characteristic value missing proportion determined according to the data governance finesse of the target data, wherein the second true proportion ranges from (0, 1).
In this embodiment, each data sample contains several fields.
In this embodiment, predicting the data field refers to predicting the data characteristic of the missing value portion according to the data characteristic and the data value of the data sample, where the influence weights of the data characteristic of the missing value portion are different according to the difference of the distance between the data field and the missing value portion in the data sample, and predicting the missing value according to the prediction characteristic and the data value of the adjacent portion of the missing value portion, specifically as follows:
The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Indicate->A data value of the data field; />Representing a total number of data fields in the corresponding data sample; />Indicate->Weights of distances corresponding to the data fields; />A data value representing a data field corresponding to the second largest weight; />A data value representing a data field corresponding to the first large weight;representing the total weight of the data fields in the corresponding data samples and being less than 1; />Representing the corresponding missing value.
The beneficial effects of the technical scheme are as follows: by classifying the target data of different links, data analysis and data quality inspection can be more accurately performed according to different classifications, and accurate data management schemes are matched according to analysis and inspection results, so that accurate management of all-link data is realized.
Example 3:
based on embodiment 1, the data classification module includes:
data classification unit: the method comprises the steps of obtaining links to which each piece of data in target data belongs, and classifying the target data based on link types of the links to obtain a plurality of first initial subsets;
a second processing unit: and the data processing module is used for carrying out data scaling on the first data in the first initial subset to obtain first processing data in a preset data range, and constructing a first processing data subset based on the first processing data to obtain a first data set.
In this embodiment, the link types include: ATM, POS, FDDI, HIPPI, HDMI, etc.
In this embodiment, the first initial subset refers to a data set obtained by classifying the target data according to different link types of links to which each target data belongs in the target data, where each link type corresponds to a first initial subset, for example, a set formed by data obtained from links with a link type of HDMI is the first initial subset.
In this embodiment, the first data refers to data contained in the first initial subset.
In this embodiment, the data scaling refers to a process of performing data scaling on the first data in the first initial subset so that the first data is within a preset data range in order to eliminate differences in dimension and magnitude between feature values of different data and ensure reliability of the data.
In this embodiment, the first processing data refers to data obtained by performing data scaling on the first data.
In this embodiment, the first subset of processing data refers to a set containing corresponding first processing data.
In this embodiment, the first data set refers to a data set obtained by performing data classification and scaling on target data of a full link.
The beneficial effects of the technical scheme are as follows: by carrying out data scaling on the target data of different links, data analysis and data quality inspection can be more accurately carried out according to corresponding classifications, and according to analysis inspection results, an accurate data management scheme is matched, so that accurate management on all-link data is realized.
Example 4:
based on the embodiment 3, the interaction determining module, as shown in fig. 2, includes:
a first interaction unit: the method comprises the steps of acquiring a link type corresponding to each first processing data subset in a first data set, and judging a first interaction mode and a first interaction degree between the link type corresponding to each first processing data subset and the rest link types;
a second interaction unit: the method comprises the steps of acquiring a second interaction mode and a second interaction degree between a feature type corresponding to first processing data of each first processing data subset and a feature type corresponding to the first processing data in the rest first processing data subsets one by one;
interaction mode determining unit: the comprehensive interaction mode is used for determining each first processing data subset based on the corresponding first interaction mode and second interaction mode;
Interaction degree determining unit: the method comprises the steps of determining a comprehensive interaction degree of each first processing data subset based on a corresponding first interaction degree and second interaction degree;
interaction result determining unit: and the method is used for sorting the corresponding comprehensive interaction modes and the comprehensive interaction degrees to obtain the comprehensive interaction result of each first processing subset.
In this embodiment, the first interaction means a link interaction between the link type corresponding to each first processing subset and the remaining link types, for example, parallel communication, serial communication, and the like. The first interaction degree refers to the link interaction degree between the link type corresponding to each first processing subset and the rest link types, wherein the value range of the first interaction degree is (0, 1).
In this embodiment, the data interaction manner includes: conditional interactions, text interactions, menu interactions, object interactions, etc. The data interaction degree refers to the degree of data interaction between different first processing sub-data sets, wherein the value range of the data interaction degree between two first processing sub-data sets is (0, 1).
In this embodiment, the feature types generally include category, ID-type features, numeric-type features, and the like.
In this embodiment, the second interaction manner refers to a feature interaction manner between a feature type of each first processing data in each first processing subset and a feature type of the remaining first processing data in the same first processing subset, for example, the second interaction manner includes: information interaction, database interaction, file interaction, etc. The second interaction degree refers to the feature interaction degree between the feature type of each first processing data in each first processing sub-set and the feature types of the rest first processing data in the same first processing sub-set, wherein the value range of the second interaction degree is (0, 1).
In this embodiment, the integrated interaction manner refers to an integrated interaction manner obtained by integrating the first interaction manner and the second interaction manner corresponding to each first processing data subset, and the integrated interaction degree refers to an integrated interaction degree obtained by integrating the first interaction degree and the second interaction degree corresponding to each first processing subset according to the interaction weight.
In this embodiment, the integrated interaction result is determined according to the integrated interaction mode and the integrated interaction degree of the same first processing subset.
The beneficial effects of the technical scheme are as follows: by determining the interaction results among the target data of different links, the data analysis and inspection results are adjusted according to the interaction results, and a more accurate data management scheme can be obtained, so that accurate management of all-link data is realized.
Example 5:
based on embodiment 3, the management analysis module includes:
a feature acquisition unit: the method comprises the steps of acquiring a link type of each first processing sub-set in a first data set and a data characteristic of first processing data in the first processing sub-set;
rule construction unit: the method comprises the steps of screening test rules matched with a corresponding first processing subset based on the link type and corresponding data characteristics, and constructing an initial analysis test rule library;
rule processing unit: the method comprises the steps of performing verification adjustment on an initial analysis verification rule base based on a running log corresponding to first data in a first processing subset to obtain a first analysis verification rule base;
a first rule classification unit: the method comprises the steps of performing first classification on rules in a first analysis and inspection rule base according to data integrity inspection standards to obtain a first classification rule base;
a second rule classification unit: the method comprises the steps of performing second classification on rules in a first analysis and inspection rule base according to data availability and traceability standards to obtain a second classification rule base;
minimum test judging unit: the minimum test proportion is used for determining data test based on the control precision of the current full-link control;
A first inspection unit: the first classification rule base is used for carrying out first verification on the first processing data in the first processing subset, and dividing the first processing subset into a first verified subset and a first unverified subset according to whether the first verification is carried out or not;
a second inspection unit: the first processing data processing method comprises the steps of performing a first check on first processing data in a first processing subset based on a first classification rule base, and dividing the first processing subset into a first checking subset and a first non-checking subset according to whether the first check is performed or not;
unverified data processing unit: for constructing an unverified data set based on the first unverified sub-set and the second unverified sub-set and reprocessing the unverified data;
and a checking and comparing unit: for comparing the first ratio of the first subset of tests to the first subset of treatments to a minimum ratio of tests, and simultaneously comparing the second ratio of the second subset of tests to the second subset of treatments to the minimum ratio of tests;
a test result determination unit: if the first proportion and the second proportion are both larger than the lowest checking proportion, checking results in the first checking sub-set and the second checking sub-set are obtained;
Classifying the test results in the first test subset and the second test subset according to the difference of the corresponding first processing data;
archiving the inspection results based on a plurality of first processing data corresponding to each same inspection result, and establishing a corresponding inspection index;
obtaining a first analysis subset based on each inspection index and the corresponding classified archiving result;
otherwise, judging that the current first classification rule base or the second classification rule base is in error, and reconstructing the initial analysis and inspection rule base.
In this embodiment, the verification rules include data integrity verification rules, data availability and traceability verification rules.
In this embodiment, the initial analysis and inspection rule base refers to a data inspection rule that is configured differently according to the link type and the data characteristic corresponding to each first processing subset and matches with the link type and the data characteristic of the current first processing subset.
In this embodiment, the content of the operation log covers the device status, operation mode, time zone, exception handling, device health, notes, and the like.
In this embodiment, the first analysis and inspection rule base is an inspection rule base obtained by adjusting the initial analysis and inspection rule base according to the first data corresponding operation log included in the first processing subset.
In this embodiment, the data integrity check criteria refers to the lowest integrity criteria that includes the necessary information and details in the various analysis and check rules.
In this embodiment, the first classification rule base refers to a rule base obtained by classifying rules in the first analysis and inspection rule base according to different data integrity inspection standards corresponding to different analysis and inspection rules.
In this embodiment, the data availability and traceability criteria refer to criteria for determining whether the sources and rules corresponding to different analysis and inspection rules are reliable or not, and whether analysis and inspection can be performed.
In this embodiment, the second classification rule base refers to a rule base obtained by classifying rules in the first analysis and inspection rule base according to different availability and traceability standards of data corresponding to different analysis and inspection rules.
In this embodiment, the lowest test proportion refers to a data test proportion determined according to the control accuracy of the current full-link control, where the value range of the first test proportion is (0, 1).
In this embodiment, the first processing data is processing data obtained by data scaling the first data.
In this embodiment, the first checking means performing a data quality check on the first processed data in the first subset of processes according to the first classification rule base.
In this embodiment, the first processing data corresponds to a plurality of analysis and inspection rules, and determining whether the first processing data needs to be inspected based on each analysis and inspection rule specifically includes:
the method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Based on the +.>Analysis of the test results under the test rules, +.>To correspond to the first processing data and the second processing dataNumber of data samples matched by each analysis and test rule, +.>To correspond to the first processed data can not be associated with the +.>Number of data samples matched by each analysis and test rule, +.>For the data characteristic value and the first processing data corresponding to the data sample in the first processing dataConversion coefficients between data characteristic values matched by individual analysis and test rules, < >>For corresponding to the first processing data and +.>Standard conversion factor of data characteristic value of ith data sample matched by individual analysis and test rule,/for data characteristic value of ith data sample matched by individual analysis and test rule>Standard conversion coefficients of characteristic values of corresponding consistent data in the j1 st analysis and test rule matched for the n1 st data sample,/for the data sample>Standard conversion coefficients of feature values corresponding to inconsistent data in the j1 st analysis and test rule matched with the n1 st data sample; max represents the maximum value symbol; min represents a minimum symbol; />The value range of (1, 0);
If T is larger than T0, the corresponding first processing data is judged to be capable of being checked, otherwise, the corresponding first processing data is judged to be incapable of being checked, wherein T0 is a checking threshold value of the corresponding first processing data based on the j1 st analysis checking rule.
In this embodiment, j1 has values of 1 and 2, and when 1, indicates a first test on the first processed data, and when 2, indicates a second test on the first processed data.
In this embodiment, the determining whether the first processed data in the first processing subset is capable of performing the first test is by determining whether the data samples in the first processed data have a matching analysis test rule, if no corresponding analysis test rule exists, determining that the first processed data corresponding to the current data sample is not capable of performing the first test, and otherwise, determining that the first processed data corresponding to the current data sample is capable of performing the first test.
In this embodiment, the first processing subset may be divided into a first verified subset and a first unverified subset according to whether the first processed data is capable of being subjected to the first verification, wherein the first verified subset and the first unverified subset include all the first processed data in the first processing subset.
In this embodiment, the second checking means performing a data quality check on the first processed data in the first subset of processes according to the second classification rule base.
In this embodiment, the first processing subset may be divided into a second subset of the first processing subset and a second subset of the second processing subset, where the second subset of the first processing subset includes all of the first processing data, depending on whether the first processing data is capable of performing the second verification.
In this embodiment, the non-verified data set refers to a set including all non-verified data in the first non-verified subset and the second non-verified subset.
In this embodiment, the test results in the first test sub-set and the second test sub-set refer to the test result corresponding to the data subjected to the first test in the first processing data and the test result corresponding to the data subjected to the second test in the first processing data, where the test result in the first test sub-set is the data integrity test result of the data capable of performing the first test in the first processing data, and the test result in the second test sub-set is the data availability and traceability test result of the data capable of performing the second test in the first processing data, for example, the data integrity of the data a is determined to be 90% after comparing the data a with the data integrity standard, and the data availability and traceability of the data B is determined to be 62% after comparing the data B with the data availability and traceability test standard.
In this embodiment, reprocessing the unverified data refers to a process of performing data processing by matching corresponding data adjustment schemes according to the unverified cause of the unverified data.
In this embodiment, the first ratio refers to a ratio of an amount of data included in the first syndrome to an amount of data included in the corresponding first processing subset, and the second ratio refers to a ratio of an amount of data included in the second syndrome to an amount of data included in the corresponding first processing subset.
In this embodiment, the archiving of the test results refers to sorting the test results of the first test set and the test results of the second test set corresponding to the same first processing data to obtain one test set.
In this embodiment, the checking index refers to constructing an index by using the data characteristics of the first data corresponding to the archiving result of the current checking result and the corresponding link characteristics.
In this embodiment, the first analysis sub-sets refer to analysis test result sets corresponding to the first processing sub-sets, which are obtained according to test indexes and classification filing results of the first processing data corresponding to each first processing sub-set.
The beneficial effects of the technical scheme are as follows: by matching the corresponding analysis and inspection rules for the first data set and analyzing different data by adopting different analysis and inspection rules, the analysis and inspection of the target data can be more accurate, so that the data management of all links is more accurate.
Example 6:
based on embodiment 5, the unverified data processing unit comprises:
a data comparison subunit: the method comprises the steps of forming an unverified data set based on a first unverified sub-set and a second unverified sub-set, and comparing each unverified data in the verified data set with a preset data unverified reason;
a data adjustment subunit: the data adjustment method comprises the steps of carrying out data adjustment based on a data adjustment scheme corresponding to the unverified reason to obtain first adjustment data;
judging whether the first adjustment data can be subjected to a first test or a second test;
if the first adjustment data can be subjected to the first check or the second check, the first adjustment data is extracted and filled into the corresponding first check sub-set or the second check sub-set.
In this embodiment, the predetermined data non-verification reason refers to a reason that the target data predetermined according to the feature type of the target data and the link type to which the target data belongs cannot be subjected to data verification.
In this embodiment, the unverified data set is comprised of a first unverified subset and a second unverified subset.
In this embodiment, the data adjustment scheme refers to a data adjustment scheme of unverified data determined according to a unverified reason of preset data, and after the unverified data is adjusted according to the data adjustment scheme, it is further required to perform a first inspection or a second inspection on the adjusted data again, so as to determine whether the adjusted data can perform the first inspection or the second inspection.
In this embodiment, the first adjustment data is adjustment data obtained after adjusting the current unverified data according to a comparison result of the unverified reason of the preset data matched with the current unverified data.
The beneficial effects of the technical scheme are as follows: the data in the unverified data set is subjected to data adjustment, so that the data is subjected to data inspection again, the data inspection result can be more comprehensive and accurate, and the accuracy of the full-link data management is improved.
Example 7:
based on embodiment 5, the data governance module includes:
scheme matching unit: the method comprises the steps of obtaining a scheme database consistent with the current link type, screening a first data treatment scheme with highest matching degree with the first analysis subset based on the scheme database, and screening a second data treatment scheme with next highest matching degree;
scheme extraction unit: the method comprises the steps of extracting a part of a first data management scheme, which is overlapped with a scheme existing in a second data management scheme, and performing scheme processing to obtain a data management scheme matched with the first analysis subset;
scheme adjusting unit: the method comprises the steps of obtaining a comprehensive interaction result corresponding to a current first processing subset, and adjusting a data treatment scheme based on the comprehensive interaction result and corresponding adjustment weights, so as to obtain a first adjustment scheme of the first processing subset;
Comprehensive scheme determination unit: the method comprises the steps of obtaining a comprehensive treatment scheme based on all first treatment subsets, and carrying out data treatment on target data based on the comprehensive treatment scheme;
and the comprehensive treatment schemes corresponding to all the first treatment subsets are the comprehensive treatment schemes of the target data.
In this embodiment, the scheme database refers to a database of corresponding abatement schemes matching different first data according to link types.
In this embodiment, the first data governance scheme is a scheme of screening a data governance scheme with the highest matching degree with the first analysis subset in the scheme database, and the second data governance scheme is a scheme of screening a data governance scheme with the second highest matching degree with the first analysis subset in the scheme database, wherein the screening in the scheme database is a process of matching according to an analysis and inspection result in the first analysis subset and a scheme corresponding to each analysis and inspection result in the scheme database.
In this embodiment, the scheme overlap refers to a portion where the first data management scheme overlaps with a scheme existing in the second data management scheme, for example, the first data management scheme includes 1,2,3, a, b, and the second data management scheme includes 1,2,3,4, a, and then the scheme overlap portion of the first data management scheme and the second data management scheme is 1,2,3, a.
In this embodiment, the data governance scheme refers to matching the corresponding data governance scheme of each first analysis subset according to the scheme database.
In this embodiment, the first adjustment scheme is an adjustment scheme obtained after adjusting the data management scheme according to the comprehensive interaction result of the first processing subset corresponding to the current first analysis subset.
In this embodiment, the comprehensive treatment scheme refers to all data treatment schemes of the target data, and the comprehensive treatment scheme is obtained after the scheme treatment.
In this embodiment, the comprehensive treatment schemes corresponding to all the first processing subsets in the first data set are the comprehensive treatment schemes of the target data.
The beneficial effects of the technical scheme are as follows: the classification analysis and inspection results are matched with the corresponding data management schemes and are integrated, so that the management of all-link data can be more accurate.
Example 8:
based on the embodiment 7, the scheme extraction unit, as shown in fig. 3, includes:
scheme extraction subunit: the method comprises the steps of extracting a part of a first data management scheme, which is overlapped with a scheme existing in a second data management scheme, to obtain a third data management scheme;
scheme judging subunit: the method comprises the steps of judging whether a sub-scheme in a third data management scheme can be completely executed;
If the sub-schemes which cannot be completely executed exist in the third data management scheme, extracting the sub-schemes in the first data management scheme and the second data management scheme to complement the part which cannot be completely executed, judging whether scheme conflicts exist in different complementing schemes, and if not, obtaining the current complementing scheme as a fourth data management scheme;
otherwise, the completion scheme is replaced to obtain a fourth data treatment scheme;
scheme determination subunit: and the fourth processing scheme is used for sorting the fourth processing scheme and the third processing scheme to obtain a data management scheme matched with the first analysis subset.
In this embodiment, the third data governance scheme refers to a data governance scheme formed by a portion where the first data governance scheme corresponding to the same analysis subset overlaps with the extraction scheme in the second data governance scheme.
In this embodiment, the complete execution refers to whether or not the execution is consistent and complete between different sub-schemes in the third data management scheme.
In this embodiment, the completion scheme refers to that if there is a sub-scheme that cannot be completely executed in the third data management scheme, the sub-schemes in the first data management scheme and the second data management scheme are extracted to complete the portion that cannot be completely executed, where the sub-schemes in the first data management scheme and the stele data management scheme can be simultaneously extracted when the scheme is completed.
In this embodiment, the fourth data governance scheme is combined with the third data governance scheme and adjusted according to the scheme execution order to obtain a data governance scheme matching the first analysis subset.
The beneficial effects of the technical scheme are as follows: by adjusting the data management scheme, the comprehensive treatment is performed, so that the management of all-link data can be more accurate.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims (8)

1. A data governance system based on full link governance control, comprising:
and a data classification module: the method comprises the steps of performing standardization processing on target data, and classifying the target data based on different links to which the data belong to obtain a first data set;
and an interaction determining module: the method comprises the steps of obtaining a data interaction mode and a data interaction degree between each subset in a first data set, and obtaining a comprehensive interaction result of each subset;
Management analysis module: the data analysis and data quality inspection method comprises the steps of performing data analysis and data quality inspection on first data of each sub-set in a first data set, and obtaining a first analysis sub-set based on analysis inspection results;
and the data management module is used for: the method is used for matching the data treatment scheme to the first analysis subset based on the scheme database, and adjusting the data treatment scheme based on the comprehensive interaction result of the corresponding subset, so that the comprehensive treatment scheme of the target data is obtained, and the data treatment is realized.
2. The data governance system based on full link governance of claim 1, wherein the data classification module comprises:
a data acquisition unit: all target data of the full link are acquired and imported into a data processing platform;
a data processing unit: judging whether a data sample containing a data missing value exists in the target data;
if the data exists, when the proportion of the data samples containing the data missing values to the total data samples is smaller than the preset missing proportion, eliminating the fields containing the data missing values;
when the proportion of the data samples containing the data missing values to the total data samples is smaller than the second missing proportion, the data samples containing the data missing values are removed;
And otherwise, predicting the field of the data missing value part based on the data sample, so as to complement the data missing value.
3. The data governance system based on full link governance of claim 1, wherein the data classification module comprises:
data classification unit: the method comprises the steps of obtaining links to which each piece of data in target data belongs, and classifying the target data based on link types of the links to obtain a plurality of first initial subsets;
a second processing unit: and the data processing module is used for carrying out data scaling on the first data in the first initial subset to obtain first processing data in a preset data range, and constructing a first processing data subset based on the first processing data to obtain a first data set.
4. A data governance system according to claim 3 and wherein said interaction determination module comprises:
a first interaction unit: the method comprises the steps of acquiring a link type corresponding to each first processing data subset in a first data set, and judging a first interaction mode and a first interaction degree between the link type corresponding to each first processing data subset and the rest link types;
A second interaction unit: the method comprises the steps of acquiring a second interaction mode and a second interaction degree between a feature type corresponding to first processing data of each first processing data subset and a feature type corresponding to the first processing data in the rest first processing data subsets one by one;
interaction mode determining unit: the comprehensive interaction mode is used for determining each first processing data subset based on the corresponding first interaction mode and second interaction mode;
interaction degree determining unit: the method comprises the steps of determining a comprehensive interaction degree of each first processing data subset based on a corresponding first interaction degree and second interaction degree;
interaction result determining unit: and the method is used for sorting the corresponding comprehensive interaction modes and the comprehensive interaction degrees to obtain the comprehensive interaction result of each first processing subset.
5. A data governance system based on full link governance according to claim 3 and wherein the management analysis module comprises:
a feature acquisition unit: the method comprises the steps of acquiring a link type of each first processing sub-set in a first data set and a data characteristic of first processing data in the first processing sub-set;
rule construction unit: the method comprises the steps of screening test rules matched with a corresponding first processing subset based on the link type and corresponding data characteristics, and constructing an initial analysis test rule library;
Rule processing unit: the method comprises the steps of performing verification adjustment on an initial analysis verification rule base based on a running log corresponding to first data in a first processing subset to obtain a first analysis verification rule base;
a first rule classification unit: the method comprises the steps of performing first classification on rules in a first analysis and inspection rule base according to data integrity inspection standards to obtain a first classification rule base;
a second rule classification unit: the method comprises the steps of performing second classification on rules in a first analysis and inspection rule base according to data availability and traceability standards to obtain a second classification rule base;
minimum test judging unit: the minimum test proportion is used for determining data test based on the control precision of the current full-link control;
a first inspection unit: the first classification rule base is used for carrying out first verification on the first processing data in the first processing subset, and dividing the first processing subset into a first verified subset and a first unverified subset according to whether the first verification is carried out or not;
a second inspection unit: the first processing data processing method comprises the steps of performing a first check on first processing data in a first processing subset based on a first classification rule base, and dividing the first processing subset into a first checking subset and a first non-checking subset according to whether the first check is performed or not;
Unverified data processing unit: for constructing an unverified data set based on the first unverified sub-set and the second unverified sub-set and reprocessing the unverified data;
and a checking and comparing unit: for comparing the first ratio of the first subset of tests to the first subset of treatments to a minimum ratio of tests, and simultaneously comparing the second ratio of the second subset of tests to the second subset of treatments to the minimum ratio of tests;
a test result determination unit: if the first proportion and the second proportion are both larger than the lowest checking proportion, checking results in the first checking sub-set and the second checking sub-set are obtained;
classifying the test results in the first test subset and the second test subset according to the difference of the corresponding first processing data;
archiving the inspection results based on a plurality of first processing data corresponding to each same inspection result, and establishing a corresponding inspection index;
obtaining a first analysis subset based on each inspection index and the corresponding classified archiving result;
otherwise, judging that the current first classification rule base or the second classification rule base is in error, and reconstructing the initial analysis and inspection rule base.
6. A data governance system based on full link governance according to claim 5 and wherein the unverified data processing unit comprises:
a data comparison subunit: the method comprises the steps of forming an unverified data set based on a first unverified sub-set and a second unverified sub-set, and comparing each unverified data in the verified data set with a preset data unverified reason;
a data adjustment subunit: the data adjustment method comprises the steps of carrying out data adjustment based on a data adjustment scheme corresponding to the unverified reason to obtain first adjustment data;
judging whether the first adjustment data can be subjected to a first test or a second test;
if the first adjustment data can be subjected to the first check or the second check, the first adjustment data is extracted and filled into the corresponding first check sub-set or the second check sub-set.
7. The data governance system based on full link governance of claim 5, wherein the data governance module comprises:
scheme matching unit: the method comprises the steps of obtaining a scheme database consistent with the current link type, screening a first data treatment scheme with highest matching degree with the first analysis subset based on the scheme database, and screening a second data treatment scheme with next highest matching degree;
Scheme extraction unit: the method comprises the steps of extracting a part of a first data management scheme, which is overlapped with a scheme existing in a second data management scheme, and performing scheme processing to obtain a data management scheme matched with the first analysis subset;
scheme adjusting unit: the method comprises the steps of obtaining a comprehensive interaction result corresponding to a current first processing subset, and adjusting a data treatment scheme based on the comprehensive interaction result and corresponding adjustment weights, so as to obtain a first adjustment scheme of the first processing subset;
comprehensive scheme determination unit: the method comprises the steps of obtaining a comprehensive treatment scheme based on all first treatment subsets, and carrying out data treatment on target data based on the comprehensive treatment scheme;
and the comprehensive treatment schemes corresponding to all the first treatment subsets are the comprehensive treatment schemes of the target data.
8. The data governance system based on full link governance of claim 7, wherein the scheme extraction unit comprises:
scheme extraction subunit: the method comprises the steps of extracting a part of a first data management scheme, which is overlapped with a scheme existing in a second data management scheme, to obtain a third data management scheme;
scheme judging subunit: the method comprises the steps of judging whether a sub-scheme in a third data management scheme can be completely executed;
If the sub-schemes which cannot be completely executed exist in the third data management scheme, extracting the sub-schemes in the first data management scheme and the second data management scheme to complement the part which cannot be completely executed, judging whether scheme conflicts exist in different complementing schemes, and if not, obtaining the current complementing scheme as a fourth data management scheme;
otherwise, the completion scheme is replaced to obtain a fourth data treatment scheme;
scheme determination subunit: and the fourth processing scheme is used for sorting the fourth processing scheme and the third processing scheme to obtain a data management scheme matched with the first analysis subset.
CN202311100015.0A 2023-08-30 2023-08-30 Data management and control system based on full-link management and control Active CN116842240B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311100015.0A CN116842240B (en) 2023-08-30 2023-08-30 Data management and control system based on full-link management and control

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311100015.0A CN116842240B (en) 2023-08-30 2023-08-30 Data management and control system based on full-link management and control

Publications (2)

Publication Number Publication Date
CN116842240A true CN116842240A (en) 2023-10-03
CN116842240B CN116842240B (en) 2023-12-01

Family

ID=88165563

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311100015.0A Active CN116842240B (en) 2023-08-30 2023-08-30 Data management and control system based on full-link management and control

Country Status (1)

Country Link
CN (1) CN116842240B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117149746A (en) * 2023-10-30 2023-12-01 山东海博科技信息系统股份有限公司 Data warehouse management system based on cloud primordial and memory calculation separation
CN118394970A (en) * 2024-07-01 2024-07-26 中数通信息有限公司 Data supervision method for data management process management

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106408028A (en) * 2016-09-26 2017-02-15 珠海市测绘院 Urban-rural planning inspection and plotting data processing method
US20180068181A1 (en) * 2015-03-23 2018-03-08 Brite:Bill Limited A document verification system
CN110837545A (en) * 2019-11-13 2020-02-25 贵州医渡云技术有限公司 Interactive data analysis method, device, medium and electronic equipment
US20220050828A1 (en) * 2020-08-17 2022-02-17 Lucidum, Inc. Asset discovery data classification and risk evaluation
CN114116690A (en) * 2021-10-29 2022-03-01 北京银盾泰安网络科技有限公司 Data cleaning platform
US20220067828A1 (en) * 2020-08-31 2022-03-03 SoftWorks Al, LLC Validation method and system to improve data accuracy
CN115576999A (en) * 2022-12-09 2023-01-06 矩阵起源(深圳)信息科技有限公司 Task data processing method, device and equipment based on cloud platform and storage medium
CN115994187A (en) * 2023-02-08 2023-04-21 国网河南省电力公司郑州供电公司 Hidden danger link type full-period safety management method and system based on distributed collaboration

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180068181A1 (en) * 2015-03-23 2018-03-08 Brite:Bill Limited A document verification system
CN106408028A (en) * 2016-09-26 2017-02-15 珠海市测绘院 Urban-rural planning inspection and plotting data processing method
CN110837545A (en) * 2019-11-13 2020-02-25 贵州医渡云技术有限公司 Interactive data analysis method, device, medium and electronic equipment
US20220050828A1 (en) * 2020-08-17 2022-02-17 Lucidum, Inc. Asset discovery data classification and risk evaluation
US20220067828A1 (en) * 2020-08-31 2022-03-03 SoftWorks Al, LLC Validation method and system to improve data accuracy
CN114116690A (en) * 2021-10-29 2022-03-01 北京银盾泰安网络科技有限公司 Data cleaning platform
CN115576999A (en) * 2022-12-09 2023-01-06 矩阵起源(深圳)信息科技有限公司 Task data processing method, device and equipment based on cloud platform and storage medium
CN115994187A (en) * 2023-02-08 2023-04-21 国网河南省电力公司郑州供电公司 Hidden danger link type full-period safety management method and system based on distributed collaboration

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
RENE ABRAHAM 等: "Data governance:A conceptual framework, structured review, and research agenda", ELSEVIER, pages 424 - 438 *
洪之旭 等: "基于大数据的社会治理数据集成及决策分析方法", 清华大学学报(自然科学版), pages 43 - 48 *
龚俊 等: "顾及多细节层次的三维R树索引扩展方法", 测绘学报, pages 119 - 125 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117149746A (en) * 2023-10-30 2023-12-01 山东海博科技信息系统股份有限公司 Data warehouse management system based on cloud primordial and memory calculation separation
CN117149746B (en) * 2023-10-30 2024-02-02 山东海博科技信息系统股份有限公司 Data warehouse management system based on cloud primordial and memory calculation separation
CN118394970A (en) * 2024-07-01 2024-07-26 中数通信息有限公司 Data supervision method for data management process management

Also Published As

Publication number Publication date
CN116842240B (en) 2023-12-01

Similar Documents

Publication Publication Date Title
CN116842240B (en) Data management and control system based on full-link management and control
Pang et al. Predicting vulnerable software components through deep neural network
CN111858242B (en) System log abnormality detection method and device, electronic equipment and storage medium
Gegick et al. Prioritizing software security fortification throughcode-level metrics
EP3716165A1 (en) Esg criteria-based enterprise evaluation device and operation method thereof
US20200082083A1 (en) Apparatus and method for verifying malicious code machine learning classification model
CN106376002B (en) Management method and device and spam monitoring system
US20090222390A1 (en) Method, program and apparatus for generating two-class classification/prediction model
CN107168995B (en) Data processing method and server
CN112733146B (en) Penetration testing method, device and equipment based on machine learning and storage medium
CN116414815A (en) Data quality detection method, device, computer equipment and storage medium
CN113567605A (en) Method and device for constructing automatic interpretation model of mass chromatogram and electronic equipment
CN111582647A (en) User data processing method and device and electronic equipment
Gegick et al. Predicting attack-prone components
Conrado et al. Adjusting the Scott-Knott cluster analyses for unbalanced designs
CN115658620B (en) Data authorization sharing method and system based on big data
CN114519187B (en) Android malicious application detection method and system based on multi-dimensional hybrid features
CN115757034A (en) Log analysis processing method and device, computer equipment and storage medium
CN111539576B (en) Risk identification model optimization method and device
KR20230141147A (en) Method and non-transitory computer-readable recording medium for classifying patent by technical field
CN115423600A (en) Data screening method, device, medium and electronic equipment
CN113986970A (en) Method for detecting measurement fee calculation result based on baseline database data
CN113792141A (en) Feature selection method based on covariance measure factor
CN118069898B (en) Log generalization method and device for multiple log sources
CN117495319A (en) Resume analysis method, resume analysis device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant