WO2017092447A1 - Method and apparatus for data quality management and control - Google Patents

Method and apparatus for data quality management and control Download PDF

Info

Publication number
WO2017092447A1
WO2017092447A1 PCT/CN2016/097717 CN2016097717W WO2017092447A1 WO 2017092447 A1 WO2017092447 A1 WO 2017092447A1 CN 2016097717 W CN2016097717 W CN 2016097717W WO 2017092447 A1 WO2017092447 A1 WO 2017092447A1
Authority
WO
WIPO (PCT)
Prior art keywords
datasheet
target
information
service sub
datasheets
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/097717
Other languages
French (fr)
Inventor
Pingping Fu
Yong Ma
Hanyong HAO
Yanqing Chen
Xuelian CHEN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Information and Telecommunication Branch of State Grid Jiangxi Electric Power Co Ltd
State Grid Corp of China SGCC
Original Assignee
Information and Telecommunication Branch of State Grid Jiangxi Electric Power Co Ltd
State Grid Corp of China SGCC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Information and Telecommunication Branch of State Grid Jiangxi Electric Power Co Ltd, State Grid Corp of China SGCC filed Critical Information and Telecommunication Branch of State Grid Jiangxi Electric Power Co Ltd
Publication of WO2017092447A1 publication Critical patent/WO2017092447A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/219—Managing data history or versioning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments

Definitions

  • Embodiments of the present invention relate to computer technologies and, in particular, to a method and an apparatus for data quality management and control.
  • the service application system includes a plurality of sub-systems. There is a data transmission interface between each sub-system and the data center, through which a sub-system stores data generated by itself into a tablespace allocated by the data center for the sub-system.
  • the datasheets in the data center have the following four problems: 1) one tablespace may store datasheets generated by a plurality of sub-systems, and the datasheets are stored in the tablespace disorderly, resulting in difficulty in searching a datasheet; 2) same datasheets may be stored in a plurality of tablespaces, resulting in greater redundancy for the datasheets; 3) after a datasheet is updated, both an old version and a new version exist, resulting in inconsistency of datasheets; 4) datasheet flow is not clear.
  • Embodiments of the present invention provide a method and an apparatus for data quality management and control, which aim at lowering difficulty in searching a datasheet, reducing redundancy of the datasheet, improving consistency of datasheets and guaranteeing a clear flow of the datasheet to achieve management and control of data quality.
  • An aspect of an embodiment of the present invention provides a method for data quality management and control, including:
  • the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information;
  • the datasheet operation information is adding a datasheet, then querying pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier; if a name of the target datasheet is different from the plurality of datasheet names, then instructing the service sub-system to store the target datasheet into a data center;
  • the datasheet operation information is updating a datasheet, then querying pieces of history information according to the name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet, and instructing the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet;
  • the pieces of history information at least comprise the service sub-system identifier and the datasheet name.
  • Another aspect of an embodiment of the present invention provides an apparatus for data quality management and control, including:
  • a receiving module configured to receive application information transmitted by a service sub-system, where the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information; receive user information transmitted by the service sub-system and a target datasheet transmitted by the service sub-system;
  • a resolving module configured to resolve the datasheet operation trigger information to obtain datasheet flow information
  • a querying module configured to: if the datasheet operation information is adding a datasheet, then query pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier; if the datasheet operation information is updating a datasheet, then query pieces of history information according to a name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet; and
  • a transmitting module configured to: if the name of the target datasheet is different from the plurality of datasheet names, then instruct the service sub-system to store the target datasheet into a data center; and instruct the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet;
  • the pieces of history information at least comprise the service sub-system identifier and the datasheet name.
  • a target datasheet may be found through a datasheet name in history information stored by a system for data quality management and control; redundancy of datasheets is reduced by querying pieces of history information to obtain a plurality of datasheet names corresponding to a service sub-system identifier, and determining whether a name of the target datasheet is the same as the plurality of datasheet names, instructing a service sub-system to store the target datasheet into a data center only when the name of the target datasheet is different from the plurality of datasheet names; consistency of the datasheets is improved by querying pieces of history information to obtain a datasheet name that is the same as the name of the target datasheet, and replacing datasheet contents corresponding to the datasheet name with contents of the target datasheet; and flow of the datasheet is guaranteed to be clear by resolving datasheet operation trigger information to obtain datasheet flow information; thus, management and control of data quality are achieved.
  • FIG. 1 is a flow chart of a method for data quality management and control according to an embodiment of the present invention
  • FIG. 2 is a network structural diagram applicable to a method for data quality management and control according to an embodiment of the present invention
  • FIG. 3 is a structural diagram of an apparatus for data quality management and control according to an embodiment of the present invention.
  • FIG. 4 is a structural diagram of an apparatus for data quality management and control according to another embodiment of the present invention.
  • FIG. 1 is a flow chart of a method for data quality management and control according to an embodiment of the present invention
  • FIG. 2 is a network structural diagram applicable to a method for data quality management and control according to an embodiment of the present invention.
  • An embodiment of the present invention provides a method for data quality management and control, targeting at several problems occurring in data of a data center due to faster growth rates of databases and datasheets and larger amount of data stored by the data center, where the method has the following specific steps.
  • Step S101 receiving application information transmitted by a service sub-system, where the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information;
  • An executive agent of embodiments of the present invention is a system for data quality management and control, as shown in FIG. 2, the system for data quality management and control may perform information interactions with a plurality of service sub-systems in a service system, and may also perform information interactions with a data center.
  • the data center is used to store a database and/or a datasheet generated by the service sub-system.
  • the service system includes a plurality of service sub-systems, such as a production management system, a marketing business system, a human resource system, an electricity trading system, a financial system, etc.
  • each service sub-system has a plurality of users correspondingly, and each user may generate a database or a datasheet in the service sub-system; when a user generates a database or a datasheet in the service sub-system, the service sub-system transmits application information to the system for data quality management and control, where the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information; one database includes a plurality of datasheets, and the service sub-system identifier uniquely identifies one service sub-system; the datasheet operation information indicates an operation performed by a user for a database or a datasheet generated by itself, such as introducing a datasheet, updating a datasheet, deleting a datasheet, etc. ; and the datasheet operation trigger information indicates a reason why the user performs the above-described operation for the database or the datasheet.
  • Step S102 resolving the datasheet operation trigger information to obtain datasheet flow information
  • the system for data quality management and control resolves the reason why the user performs the above-described operation for the database or the datasheet so as to obtain datasheet flow information, where the datasheet flow information specifically indicates a datasheet generated by the user is viewed, called or stored by which service sub-system, that is, the datasheet flow information may view, call or store the service sub-system identifier of the datasheet.
  • Step S103 receiving user information transmitted by the service sub-system and a target datasheet transmitted by the service sub-system;
  • the system for data quality management and control reviews the application information transmitted by the service sub-system, after the review passes, the service sub-system transmits user information and a target datasheet to the system for data quality management and control; the target datasheet is a datasheet generated by a user corresponding to the user information in the service sub-system.
  • Step S104 if the datasheet operation information is adding a datasheet, then querying pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier; if a name of the target datasheet is different from the plurality of datasheet names, then instructing the service sub-system to store the target datasheet into a data center;
  • the system for data quality management and control queries pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier; the system for data quality management and control is pre-stored with pieces of history information, each piece of history information at least includes the service sub-system identifier and the datasheet name; the system for data quality management and control determines whether a name of the target datasheet is the same as each datasheet name in the plurality of datasheet names respectively, if the name of the target datasheet is different from the plurality of datasheet names, then the system for data quality management and control instructs the service sub-system to store the target datasheet into the data center; if the name of the target datasheet is the same as any one of the plurality of datasheet names, then transmit warning information to the service sub-system, and notify the user of a repetition of the name of the target datasheet to avoid
  • Step S105 if the datasheet operation information is updating a datasheet, then querying pieces of history information according to the name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet, and instructing the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet;
  • the pieces of history information at least comprise the service sub-system identifier and the datasheet name.
  • the data center is stored with a datasheet having a name that is the same as that of the target datasheet, and history information pre-stored by the system for data quality management and control also includes the datasheet name, then the system for data quality management and control queries pieces of history information according to the name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet, and instructs the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet, thereby avoiding a phenomenon where there is inconsistency of datasheets stored in the data center.
  • a target datasheet may be found through a datasheet name in history information stored by a system for data quality management and control; redundancy of datasheets is reduced by querying pieces of history information to obtain a plurality of datasheet names corresponding to a service sub-system identifier, and determining whether a name of the target datasheet is the same as the plurality of datasheet names, instructing a service sub-system to store the target datasheet into a data center only when the name of the target datasheet is different from the plurality of datasheet names; consistency of the datasheets is improved by querying pieces of history information to obtain a datasheet name that is the same as the name of the target datasheet, and replacing datasheet contents corresponding to the datasheet name with contents of the target datasheet; and flow of the datasheet is guaranteed to be clear by resolving datasheet operation trigger information to obtain datasheet flow information; thus, management and control of data quality are achieved.
  • the instructing the service sub-system to store the target datasheet into the data center further including: receiving a name of a tablespace storing the target datasheet as transmitted by the data center and an interface mode between the service sub-system and the data center transmitted by the data center; and generating current record information, where the current record information includes the service sub-system identifier, the name of the tablespace, the name of the target datasheet, the user information, the datasheet flow information, operation authorization information of the target datasheet and the interface mode.
  • the service sub-subsystem After the system for data quality management and control instructs the service sub-system to store the target datasheet into the data center, the service sub-subsystem transmits the target datasheet to the data center through an interface mode between the service sub-system and the data center, where the interface mode specifically includes any one of data warehousing technologies (Extract Transform Load, ETL for short) , OGG, DBlink and Webservice.
  • ETL Extract Transform Load
  • the data center may open up a tablespace for several service sub-systems, preferably, the data center opens up a tablespace for each service sub-system, the data center stores into the tablespace a target datasheet transmitted by the service sub-system through the interface mode, and transmits a name of the tablespace and the interface mode between the service sub-system and the data center to the system for data quality management and control.
  • the system for data quality management and control generates current record information according to the application information, the user information and the target datasheet which are transmitted by the service sub-system, the name of the tablespace and the interface mode which are transmitted by the data center as described in the above embodiment, where the current record information includes the service sub-system identifier, the name of the tablespace, the name of the target datasheet, the user information, the datasheet flow information, operation authorization information of the target datasheet and the interface mode, and the operation authorization information of the target datasheet is an authority set by the system for data quality management and control for the user for operating the target datasheet.
  • the system for data quality management and control receives application information transmitted by another service sub-system, the current record information then becomes history information.
  • the receiving the user information transmitted by the service sub-system and the target datasheet transmitted by the service sub-system further including: determining whether data in the target datasheet is in conformity with a preset data rule; if the data in the target datasheet is not in conformity with the preset data rule, then transmitting warning information to the service sub-system to enable a user to modify a data format in the target datasheet.
  • the system for data quality management and control is pre-stored with a data rule, data in target datasheets transmitted by each service sub-system should be in conformity with the data rule, that is, unified standardization is performed through the data rule to the data in the target datasheets transmitted by each service sub-system, so that the target datasheets transmitted by each service sub-system may be identified by other service sub-systems, specifically, after receiving the user information transmitted by the service sub-system and the target datasheet transmitted by the service sub-system, the system for data quality management and control determines whether data in the target datasheet is in conformity with a preset data rule; if the data in the target datasheet is not in conformity with the preset data rule, then transmit warning information to the service sub-system to enable a user to modify a data format in the target datasheet.
  • unified standardization is performed to data in target datasheets transmitted by each service sub-system through a data rule pre-stored in the system for data quality management and control, which ensures that the target datasheets transmitted by each service sub-system can be identified by other service sub-systems.
  • an importance degree of each datasheet may be obtained by analyzing all pieces of record information in the record information library, and the importance degree of each datasheet may be measured through quantity of datasheet flow information corresponding to counted datasheet names, for instance, if a name of a target datasheet occurs in pieces of record information, it is obtained from analysis of the pieces of record information that the target datasheet is viewed, called or stored by a plurality of service sub-systems, then it indicates that the target datasheet has higher importance degree, and the quantity of the datasheet flow information corresponding to the datasheet names is proportional to the importance degree of the datasheet.
  • the method for data quality management and control also includes: setting a datasheet collecting rule, and collecting a plurality of datasheets from the data center according to the datasheet collecting rule; determining whether names of any two datasheets in the plurality of the datasheets are the same, if the names of the two datasheets are the same, then determining whether contents of the two datasheets are the same; if the contents of the two datasheets are the same, then transmitting a first deleting command to the data center to enable the data center to delete any one of the two datasheets; if the contents of the two datasheets are different, then acquiring timestamps of the two datasheets from the data center, and transmitting a second deleting command to the data center to enable the data center to delete a datasheet having a smaller timestamp in the two datasheets.
  • the system for data quality management and control may also set a datasheet collecting rule, where the datasheet collecting rule includes periodic collecting, event triggered collecting, datasheet feature-based collecting, preset time point collecting, etc.
  • the system for data quality management and control collects a plurality of datasheets from the data center according to the datasheet collecting rule, and determines whether names of any two datasheets in the plurality of datasheets are the same, if the names of the two datasheets are the same, then determine whether contents of the two datasheets are the same; if the contents of the two datasheets are the same, then transmit a first deleting command to the data center to enable the data center to delete any one of the two datasheets, thereby further reducing the redundancy of the datasheets in the data center; if the contents of the two datasheets are different, then acquire timestamps of the two datasheets from the data center, and transmit a second deleting command to the data center to enable the data center to delete a datasheet having a smaller timestamp in the
  • a plurality of datasheets in the data center is collected, and names and contents of any two datasheets in the plurality of datasheets are determined; the redundancy of the datasheets in the data center is further reduced by deleting any one of two datasheets having same names and same contents; and meanwhile the consistency of the datasheets in the data center is further improved by deleting a datasheet having a smaller timestamp in the two datasheets having the same names but different contents.
  • FIG. 3 is a structural diagram of an apparatus for data quality management and control according to an embodiment of the present invention.
  • the apparatus for data quality management and control as shown in FIG. 3 is specifically the system for data quality management and control as shown in FIG. 2.
  • the apparatus for data quality management and control according to the embodiment of the present invention may perform processing flows according to embodiments of the method for data quality management and control, as shown in FIG. 3, the apparatus 30 for data quality management and control includes: a receiving module 31, a resolving module 32, a querying module 33 and a transmitting module 34.
  • the receiving module 31 is configured to receive application information transmitted by a service sub-system, where the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information; and receive user information transmitted by the service sub-system and a target datasheet transmitted by the service sub-system.
  • the resolving module 32 is configured to resolve the datasheet operation trigger information to obtain datasheet flow information.
  • the querying module 33 is configured to: if the datasheet operation information is adding a datasheet, then query pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier; if the datasheet operation information is updating a datasheet, then query pieces of history information according to a name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet.
  • the transmitting module 34 is configured to: if the name of the target datasheet is different from the plurality of datasheet names, then instruct the service sub-system to store the target datasheet into a data center; and instruct the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet, where the pieces of history information at least include the service sub-system identifier and the datasheet name.
  • a target datasheet may be found through a datasheet name in history information stored by a system for data quality management and control; redundancy of datasheets is reduced by querying pieces of history information to obtain a plurality of datasheet names corresponding to a service sub-system identifier, and determining whether a name of the target datasheet is the same as the plurality of datasheet names, instructing a service sub-system to store the target datasheet into a data center only when the name of the target datasheet is different from the plurality of datasheet names; consistency of the datasheets is improved by querying pieces of history information to obtain a datasheet name that is the same as the name of the target datasheet, and replacing datasheet contents corresponding to the datasheet name with contents of the target datasheet; and flow of the datasheet is guaranteed to be clear by resolving datasheet operation trigger information to obtain datasheet flow information; thus, management and control of data quality are achieved.
  • FIG. 4 is a structural diagram of an apparatus for data quality management and control according to another embodiment of the present invention.
  • the receiving module 31 is further configured to, after the transmitting module 34 instructs the service sub-system to store the target datasheet into the data center, receive a name of a tablespace storing the target datasheet as transmitted by the data center and an interface mode between the service sub-system and the data center transmitted by the data center;
  • the apparatus 30 for data quality management and control also includes a recording module 35.
  • the recording module 35 is configured to generate current record information, where the current record information includes the service sub-system identifier, the name of the tablespace, the name of the target datasheet, the user information, the datasheet flow information, operation authorization information of the target datasheet and the interface mode.
  • the apparatus 30 for data quality management and control also includes a determining module 36.
  • the determining module 36 is configured to: after the receiving module 31 receives the user information transmitted by the service sub-system and the target datasheet transmitted by the service sub-system, determine whether data in the target datasheet is in conformity with a preset data rule; the transmitting module 34 is further configured to: if the data in the target datasheet is not in conformity with the preset data rule, then transmit warning information to the service sub-system to enable a user to modify a data format in the target datasheet.
  • the determining module 36 is further configured to analyze an importance degree of each datasheet according to the current record information and the pieces of history information, the more the datasheet flow information corresponding to the datasheet name, the higher the importance degree of the datasheet.
  • the apparatus 30 for data quality management and control also includes a collecting module 37.
  • the collecting module 37 is configured to set a datasheet collecting rule, and collect a plurality of datasheets from the data center according to the datasheet collecting rule;
  • the determining module 36 is further configured to determine whether names of any two datasheets in the plurality of the datasheets are the same, if the names of the two datasheets are the same, then determine whether contents of the two datasheets are the same;
  • the transmitting module 34 is further configured to: if the contents of the two datasheets are the same, then transmit a first deleting command to the data center to enable the data center to delete any one of the two datasheets; if the contents of the two datasheets are different, then acquire timestamps of the two datasheets from the data center, and transmit a second deleting command to the data center to enable the data center to delete a datasheet having a smaller timestamp in the two datasheets.
  • the apparatus for data quality management and control according to the embodiment of the present invention may be specifically used to perform the method embodiments provided in FIG. 1 above, and specific functions will not be repeated herein.
  • unified standardization is performed to data in target datasheets transmitted by each service sub-system through a data rule pre-stored in the system for data quality management and control, which ensures that the target datasheets transmitted by each service sub-system can be identified by other service sub-systems; through the system for data quality management and control, a plurality of datasheets in the data center is collected, and names and contents of any two datasheets in the plurality of datasheets are determined; the redundancy of the datasheets in the data center is further reduced by deleting any one of two datasheets having same names and same contents; and meanwhile the consistency of the datasheets in the data center is further improved by deleting a datasheet having a smaller timestamp in the two datasheets having the same names but different contents.
  • a target datasheet may be found through a datasheet name in history information stored by a system for data quality management and control; redundancy of datasheets is reduced by querying pieces of history information to obtain a plurality of datasheet names corresponding to a service sub-system identifier, and determining whether a name of the target datasheet is the same as the plurality of datasheet names, instructing a service sub-system to store the target datasheet into a data center only when the name of the target datasheet is different from the plurality of datasheet names; consistency of the datasheets is improved by querying pieces of history information to obtain a datasheet name that is the same as the name of the target datasheet, and replacing datasheet contents corresponding to the datasheet name with contents of the target datasheet; and flow of the datasheet is guaranteed to be clear by resolving datasheet operation trigger information to obtain datasheet flow information; thus, management and control of data quality are achieved; unified standardization is performed to data in target datasheets transmitted by each service sub
  • the disclosed apparatus and method may be implemented in other manners.
  • the above-described apparatus embodiments are merely exemplary.
  • the unit division is merely logical function division and there may be other divisions during actual implementation.
  • a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed.
  • the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces.
  • the indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.
  • the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, that is, may be located in one position, or may be distributed on a plurality of network units. A part or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • functional units in the embodiments of the present invention may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.
  • the integrated unit may be implemented in a form of hardware, or may be implemented in a form of a hardware plus software functional unit.
  • the integrated unit implemented in the form of the software functional unit as described above may be stored in a computer-readable storage medium.
  • the software functional unit described above is stored in a storage medium, and includes several instructions capable of enabling a computer device (which may be a personal computer, a server, a network device, or the like) or a processor to perform a part of the steps of the methods described in the embodiments of the present invention.
  • the foregoing storage medium includes: any medium that can store program codes, such as a USB flash disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM) , a random access memory (Random Access Memory, RAM) , a magnetic disk, or an optical disc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

A method and an apparatus for data quality management and control are provided. The method includes: receiving application information transmitted by a service sub-system, where the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information (S101); resolving datasheet operation trigger information to obtain datasheet flow information (S102); receiving user information transmitted by the service sub-system and a target datasheet transmitted by the service sub-system (S103); if a name of the target datasheet is different from a plurality of datasheet names corresponding to the service sub-system identifier, then instructing the service sub-system to store the target datasheet into a data center (S104); if the datasheet operation information is updating a datasheet, instructing the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet (S105).

Description

METHOD AND APPARATUS FOR DATA QUALITY MANAGEMENT AND CONTROL
CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims priority to Chinese Patent Application No. 201510889904.9, filed on December 04, 2015, the content of which is hereby incorporated by reference in its entirety.
TECHNICAL FIELD
Embodiments of the present invention relate to computer technologies and, in particular, to a method and an apparatus for data quality management and control.
BACKGROUND
With rapid development of enterprise informatization, a great number of data of a service application system accesses a data center, which results in rapid growth for databases and datasheets in the data center and a huge number of data resources.
In the prior art, the service application system includes a plurality of sub-systems. There is a data transmission interface between each sub-system and the data center, through which a sub-system stores data generated by itself into a tablespace allocated by the data center for the sub-system.
However, as growth rates of the databases and the datasheets become faster, and the amount of data stored by the data center becomes larger, the datasheets in the data center have the following four problems: 1) one tablespace may store datasheets generated by a plurality of sub-systems, and the datasheets are stored in the tablespace disorderly, resulting in difficulty in searching a datasheet； 2) same datasheets may be stored in a plurality of tablespaces, resulting in greater redundancy for the datasheets； 3) after a datasheet is updated,  both an old version and a new version exist, resulting in inconsistency of datasheets； 4) datasheet flow is not clear.
SUMMARY
Embodiments of the present invention provide a method and an apparatus for data quality management and control, which aim at lowering difficulty in searching a datasheet, reducing redundancy of the datasheet, improving consistency of datasheets and guaranteeing a clear flow of the datasheet to achieve management and control of data quality.
An aspect of an embodiment of the present invention provides a method for data quality management and control, including:
receiving application information transmitted by a service sub-system, where the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information；
resolving the datasheet operation trigger information to obtain datasheet flow information；
receiving user information transmitted by the service sub-system and a target datasheet transmitted by the service sub-system；
if the datasheet operation information is adding a datasheet, then querying pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier； if a name of the target datasheet is different from the plurality of datasheet names, then instructing the service sub-system to store the target datasheet into a data center；
if the datasheet operation information is updating a datasheet, then querying pieces of history information according to the name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet, and instructing the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet；
where the pieces of history information at least comprise the service sub-system identifier and the datasheet name.
Another aspect of an embodiment of the present invention provides an apparatus for data quality management and control, including:
a receiving module, configured to receive application information transmitted by a service sub-system, where the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information； receive user information transmitted by the service sub-system and a target datasheet transmitted by the service sub-system；
a resolving module, configured to resolve the datasheet operation trigger information to obtain datasheet flow information；
a querying module, configured to: if the datasheet operation information is adding a datasheet, then query pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier； if the datasheet operation information is updating a datasheet, then query pieces of history information according to a name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet； and
a transmitting module, configured to: if the name of the target datasheet is different from the plurality of datasheet names, then instruct the service sub-system to store the target datasheet into a data center； and instruct the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet；
where the pieces of history information at least comprise the service sub-system identifier and the datasheet name.
According to a method and an apparatus for data quality management and control provided in embodiments of the present invention, a target datasheet may be found through a datasheet name in history information stored by a system for data quality management and control； redundancy of datasheets is reduced by querying pieces of history information to obtain a plurality of datasheet names corresponding to a service sub-system identifier, and determining whether a name of the target datasheet is the same as the plurality of datasheet names, instructing a service sub-system to store the target datasheet into a data center only when the name of the target datasheet is different from the plurality of datasheet names； consistency of the datasheets is improved by querying pieces of history information to obtain  a datasheet name that is the same as the name of the target datasheet, and replacing datasheet contents corresponding to the datasheet name with contents of the target datasheet； and flow of the datasheet is guaranteed to be clear by resolving datasheet operation trigger information to obtain datasheet flow information； thus, management and control of data quality are achieved.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 is a flow chart of a method for data quality management and control according to an embodiment of the present invention；
FIG. 2 is a network structural diagram applicable to a method for data quality management and control according to an embodiment of the present invention；
FIG. 3 is a structural diagram of an apparatus for data quality management and control according to an embodiment of the present invention；
FIG. 4 is a structural diagram of an apparatus for data quality management and control according to another embodiment of the present invention.
DESCRIPTION OF EMBODIMENTS
FIG. 1 is a flow chart of a method for data quality management and control according to an embodiment of the present invention； FIG. 2 is a network structural diagram applicable to a method for data quality management and control according to an embodiment of the present invention. An embodiment of the present invention provides a method for data quality management and control, targeting at several problems occurring in data of a data center due to faster growth rates of databases and datasheets and larger amount of data stored by the data center, where the method has the following specific steps.
Step S101, receiving application information transmitted by a service sub-system, where the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information；
An executive agent of embodiments of the present invention is a system for data quality management and control, as shown in FIG. 2, the system for data quality management  and control may perform information interactions with a plurality of service sub-systems in a service system, and may also perform information interactions with a data center. The data center is used to store a database and/or a datasheet generated by the service sub-system. The service system includes a plurality of service sub-systems, such as a production management system, a marketing business system, a human resource system, an electricity trading system, a financial system, etc. ； each service sub-system has a plurality of users correspondingly, and each user may generate a database or a datasheet in the service sub-system； when a user generates a database or a datasheet in the service sub-system, the service sub-system transmits application information to the system for data quality management and control, where the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information； one database includes a plurality of datasheets, and the service sub-system identifier uniquely identifies one service sub-system； the datasheet operation information indicates an operation performed by a user for a database or a datasheet generated by itself, such as introducing a datasheet, updating a datasheet, deleting a datasheet, etc. ； and the datasheet operation trigger information indicates a reason why the user performs the above-described operation for the database or the datasheet.
Step S102, resolving the datasheet operation trigger information to obtain datasheet flow information；
The system for data quality management and control resolves the reason why the user performs the above-described operation for the database or the datasheet so as to obtain datasheet flow information, where the datasheet flow information specifically indicates a datasheet generated by the user is viewed, called or stored by which service sub-system, that is, the datasheet flow information may view, call or store the service sub-system identifier of the datasheet.
Step S103, receiving user information transmitted by the service sub-system and a target datasheet transmitted by the service sub-system；
The system for data quality management and control reviews the application information transmitted by the service sub-system, after the review passes, the service sub-system transmits user information and a target datasheet to the system for data quality  management and control； the target datasheet is a datasheet generated by a user corresponding to the user information in the service sub-system.
Step S104, if the datasheet operation information is adding a datasheet, then querying pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier； if a name of the target datasheet is different from the plurality of datasheet names, then instructing the service sub-system to store the target datasheet into a data center；
If the target datasheet is a newly-added datasheet, that is, the target datasheet has not been stored in the data center, then the system for data quality management and control queries pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier； the system for data quality management and control is pre-stored with pieces of history information, each piece of history information at least includes the service sub-system identifier and the datasheet name； the system for data quality management and control determines whether a name of the target datasheet is the same as each datasheet name in the plurality of datasheet names respectively, if the name of the target datasheet is different from the plurality of datasheet names, then the system for data quality management and control instructs the service sub-system to store the target datasheet into the data center； if the name of the target datasheet is the same as any one of the plurality of datasheet names, then transmit warning information to the service sub-system, and notify the user of a repetition of the name of the target datasheet to avoid a repetition of datasheets in the data center.
Step S105, if the datasheet operation information is updating a datasheet, then querying pieces of history information according to the name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet, and instructing the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet；
where the pieces of history information at least comprise the service sub-system identifier and the datasheet name.
If the target datasheet is an updated datasheet, that is, the target datasheet has been stored in the data center, the data center is stored with a datasheet having a name that is the  same as that of the target datasheet, and history information pre-stored by the system for data quality management and control also includes the datasheet name, then the system for data quality management and control queries pieces of history information according to the name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet, and instructs the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet, thereby avoiding a phenomenon where there is inconsistency of datasheets stored in the data center.
According to the embodiment of the present invention, a target datasheet may be found through a datasheet name in history information stored by a system for data quality management and control； redundancy of datasheets is reduced by querying pieces of history information to obtain a plurality of datasheet names corresponding to a service sub-system identifier, and determining whether a name of the target datasheet is the same as the plurality of datasheet names, instructing a service sub-system to store the target datasheet into a data center only when the name of the target datasheet is different from the plurality of datasheet names； consistency of the datasheets is improved by querying pieces of history information to obtain a datasheet name that is the same as the name of the target datasheet, and replacing datasheet contents corresponding to the datasheet name with contents of the target datasheet； and flow of the datasheet is guaranteed to be clear by resolving datasheet operation trigger information to obtain datasheet flow information； thus, management and control of data quality are achieved.
Based on the above embodiment, after the instructing the service sub-system to store the target datasheet into the data center, further including: receiving a name of a tablespace storing the target datasheet as transmitted by the data center and an interface mode between the service sub-system and the data center transmitted by the data center； and generating current record information, where the current record information includes the service sub-system identifier, the name of the tablespace, the name of the target datasheet, the user information, the datasheet flow information, operation authorization information of the target datasheet and the interface mode.
After the system for data quality management and control instructs the service sub-system to store the target datasheet into the data center, the service sub-subsystem  transmits the target datasheet to the data center through an interface mode between the service sub-system and the data center, where the interface mode specifically includes any one of data warehousing technologies (Extract Transform Load, ETL for short) , OGG, DBlink and Webservice. The data center may open up a tablespace for several service sub-systems, preferably, the data center opens up a tablespace for each service sub-system, the data center stores into the tablespace a target datasheet transmitted by the service sub-system through the interface mode, and transmits a name of the tablespace and the interface mode between the service sub-system and the data center to the system for data quality management and control.
The system for data quality management and control generates current record information according to the application information, the user information and the target datasheet which are transmitted by the service sub-system, the name of the tablespace and the interface mode which are transmitted by the data center as described in the above embodiment, where the current record information includes the service sub-system identifier, the name of the tablespace, the name of the target datasheet, the user information, the datasheet flow information, operation authorization information of the target datasheet and the interface mode, and the operation authorization information of the target datasheet is an authority set by the system for data quality management and control for the user for operating the target datasheet. When the system for data quality management and control receives application information transmitted by another service sub-system, the current record information then becomes history information.
After the receiving the user information transmitted by the service sub-system and the target datasheet transmitted by the service sub-system, further including: determining whether data in the target datasheet is in conformity with a preset data rule； if the data in the target datasheet is not in conformity with the preset data rule, then transmitting warning information to the service sub-system to enable a user to modify a data format in the target datasheet.
The system for data quality management and control is pre-stored with a data rule, data in target datasheets transmitted by each service sub-system should be in conformity with the data rule, that is, unified standardization is performed through the data rule to the data in the target datasheets transmitted by each service sub-system, so that the target datasheets  transmitted by each service sub-system may be identified by other service sub-systems, specifically, after receiving the user information transmitted by the service sub-system and the target datasheet transmitted by the service sub-system, the system for data quality management and control determines whether data in the target datasheet is in conformity with a preset data rule； if the data in the target datasheet is not in conformity with the preset data rule, then transmit warning information to the service sub-system to enable a user to modify a data format in the target datasheet.
According to the embodiment of the present invention, unified standardization is performed to data in target datasheets transmitted by each service sub-system through a data rule pre-stored in the system for data quality management and control, which ensures that the target datasheets transmitted by each service sub-system can be identified by other service sub-systems.
Based on the above-described embodiment, after the generating the current record information, further including: analyzing an importance degree of each datasheet according to the current record information and the pieces of history information, the more the datasheet flow information corresponding to the datasheet name, the higher the importance degree of the datasheet.
Each time a piece of current record information is generated, the system for data quality management and control puts the current record information into a record information library, an importance degree of each datasheet may be obtained by analyzing all pieces of record information in the record information library, and the importance degree of each datasheet may be measured through quantity of datasheet flow information corresponding to counted datasheet names, for instance, if a name of a target datasheet occurs in pieces of record information, it is obtained from analysis of the pieces of record information that the target datasheet is viewed, called or stored by a plurality of service sub-systems, then it indicates that the target datasheet has higher importance degree, and the quantity of the datasheet flow information corresponding to the datasheet names is proportional to the importance degree of the datasheet.
The method for data quality management and control also includes: setting a datasheet collecting rule, and collecting a plurality of datasheets from the data center  according to the datasheet collecting rule； determining whether names of any two datasheets in the plurality of the datasheets are the same, if the names of the two datasheets are the same, then determining whether contents of the two datasheets are the same； if the contents of the two datasheets are the same, then transmitting a first deleting command to the data center to enable the data center to delete any one of the two datasheets； if the contents of the two datasheets are different, then acquiring timestamps of the two datasheets from the data center, and transmitting a second deleting command to the data center to enable the data center to delete a datasheet having a smaller timestamp in the two datasheets.
The system for data quality management and control may also set a datasheet collecting rule, where the datasheet collecting rule includes periodic collecting, event triggered collecting, datasheet feature-based collecting, preset time point collecting, etc. The system for data quality management and control collects a plurality of datasheets from the data center according to the datasheet collecting rule, and determines whether names of any two datasheets in the plurality of datasheets are the same, if the names of the two datasheets are the same, then determine whether contents of the two datasheets are the same； if the contents of the two datasheets are the same, then transmit a first deleting command to the data center to enable the data center to delete any one of the two datasheets, thereby further reducing the redundancy of the datasheets in the data center； if the contents of the two datasheets are different, then acquire timestamps of the two datasheets from the data center, and transmit a second deleting command to the data center to enable the data center to delete a datasheet having a smaller timestamp in the two datasheets, thereby further increasing the consistency of the datasheets in the data center.
According to the embodiment of the present invention, through the system for data quality management and control, a plurality of datasheets in the data center is collected, and names and contents of any two datasheets in the plurality of datasheets are determined； the redundancy of the datasheets in the data center is further reduced by deleting any one of two datasheets having same names and same contents； and meanwhile the consistency of the datasheets in the data center is further improved by deleting a datasheet having a smaller timestamp in the two datasheets having the same names but different contents.
FIG. 3 is a structural diagram of an apparatus for data quality management and control according to an embodiment of the present invention. The apparatus for data quality management and control as shown in FIG. 3 is specifically the system for data quality management and control as shown in FIG. 2. The apparatus for data quality management and control according to the embodiment of the present invention may perform processing flows according to embodiments of the method for data quality management and control, as shown in FIG. 3, the apparatus 30 for data quality management and control includes: a receiving module 31, a resolving module 32, a querying module 33 and a transmitting module 34. The receiving module 31 is configured to receive application information transmitted by a service sub-system, where the application information includes a service sub-system identifier, datasheet operation information and datasheet operation trigger information； and receive user information transmitted by the service sub-system and a target datasheet transmitted by the service sub-system. The resolving module 32 is configured to resolve the datasheet operation trigger information to obtain datasheet flow information. The querying module 33 is configured to: if the datasheet operation information is adding a datasheet, then query pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier； if the datasheet operation information is updating a datasheet, then query pieces of history information according to a name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet. The transmitting module 34 is configured to: if the name of the target datasheet is different from the plurality of datasheet names, then instruct the service sub-system to store the target datasheet into a data center； and instruct the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet, where the pieces of history information at least include the service sub-system identifier and the datasheet name.
According to the embodiment of the present invention, a target datasheet may be found through a datasheet name in history information stored by a system for data quality management and control； redundancy of datasheets is reduced by querying pieces of history information to obtain a plurality of datasheet names corresponding to a service sub-system identifier, and determining whether a name of the target datasheet is the same as the plurality  of datasheet names, instructing a service sub-system to store the target datasheet into a data center only when the name of the target datasheet is different from the plurality of datasheet names； consistency of the datasheets is improved by querying pieces of history information to obtain a datasheet name that is the same as the name of the target datasheet, and replacing datasheet contents corresponding to the datasheet name with contents of the target datasheet； and flow of the datasheet is guaranteed to be clear by resolving datasheet operation trigger information to obtain datasheet flow information； thus, management and control of data quality are achieved.
FIG. 4 is a structural diagram of an apparatus for data quality management and control according to another embodiment of the present invention. Based on the above-described embodiment, the receiving module 31 is further configured to, after the transmitting module 34 instructs the service sub-system to store the target datasheet into the data center, receive a name of a tablespace storing the target datasheet as transmitted by the data center and an interface mode between the service sub-system and the data center transmitted by the data center； the apparatus 30 for data quality management and control also includes a recording module 35. The recording module 35 is configured to generate current record information, where the current record information includes the service sub-system identifier, the name of the tablespace, the name of the target datasheet, the user information, the datasheet flow information, operation authorization information of the target datasheet and the interface mode.
The apparatus 30 for data quality management and control also includes a determining module 36. The determining module 36 is configured to: after the receiving module 31 receives the user information transmitted by the service sub-system and the target datasheet transmitted by the service sub-system, determine whether data in the target datasheet is in conformity with a preset data rule； the transmitting module 34 is further configured to: if the data in the target datasheet is not in conformity with the preset data rule, then transmit warning information to the service sub-system to enable a user to modify a data format in the target datasheet.
The determining module 36 is further configured to analyze an importance degree of each datasheet according to the current record information and the pieces of history  information, the more the datasheet flow information corresponding to the datasheet name, the higher the importance degree of the datasheet.
The apparatus 30 for data quality management and control also includes a collecting module 37. The collecting module 37 is configured to set a datasheet collecting rule, and collect a plurality of datasheets from the data center according to the datasheet collecting rule； the determining module 36 is further configured to determine whether names of any two datasheets in the plurality of the datasheets are the same, if the names of the two datasheets are the same, then determine whether contents of the two datasheets are the same； the transmitting module 34 is further configured to: if the contents of the two datasheets are the same, then transmit a first deleting command to the data center to enable the data center to delete any one of the two datasheets； if the contents of the two datasheets are different, then acquire timestamps of the two datasheets from the data center, and transmit a second deleting command to the data center to enable the data center to delete a datasheet having a smaller timestamp in the two datasheets.
The apparatus for data quality management and control according to the embodiment of the present invention may be specifically used to perform the method embodiments provided in FIG. 1 above, and specific functions will not be repeated herein.
According to the embodiment of the present invention, unified standardization is performed to data in target datasheets transmitted by each service sub-system through a data rule pre-stored in the system for data quality management and control, which ensures that the target datasheets transmitted by each service sub-system can be identified by other service sub-systems； through the system for data quality management and control, a plurality of datasheets in the data center is collected, and names and contents of any two datasheets in the plurality of datasheets are determined； the redundancy of the datasheets in the data center is further reduced by deleting any one of two datasheets having same names and same contents； and meanwhile the consistency of the datasheets in the data center is further improved by deleting a datasheet having a smaller timestamp in the two datasheets having the same names but different contents.
In conclusion, according to embodiments of the present invention, a target datasheet may be found through a datasheet name in history information stored by a system  for data quality management and control； redundancy of datasheets is reduced by querying pieces of history information to obtain a plurality of datasheet names corresponding to a service sub-system identifier, and determining whether a name of the target datasheet is the same as the plurality of datasheet names, instructing a service sub-system to store the target datasheet into a data center only when the name of the target datasheet is different from the plurality of datasheet names； consistency of the datasheets is improved by querying pieces of history information to obtain a datasheet name that is the same as the name of the target datasheet, and replacing datasheet contents corresponding to the datasheet name with contents of the target datasheet； and flow of the datasheet is guaranteed to be clear by resolving datasheet operation trigger information to obtain datasheet flow information； thus, management and control of data quality are achieved； unified standardization is performed to data in target datasheets transmitted by each service sub-system through a data rule pre-stored in the system for data quality management and control, which ensures that the target datasheets transmitted by each service sub-system can be identified by other service sub-systems； through the system for data quality management and control, a plurality of datasheets in the data center is collected, and names and contents of any two datasheets in the plurality of datasheets are determined； the redundancy of the datasheets in the data center is further reduced by deleting any one of two datasheets having same names and same contents； and meanwhile the consistency of the datasheets in the data center is further improved by deleting a datasheet having a smaller timestamp in the two datasheets having the same names but different contents.
In embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely exemplary. For example, the unit division is merely logical function division and there may be other divisions during actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. The indirect couplings or  communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, that is, may be located in one position, or may be distributed on a plurality of network units. A part or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in a form of hardware, or may be implemented in a form of a hardware plus software functional unit.
The integrated unit implemented in the form of the software functional unit as described above may be stored in a computer-readable storage medium. The software functional unit described above is stored in a storage medium, and includes several instructions capable of enabling a computer device (which may be a personal computer, a server, a network device, or the like) or a processor to perform a part of the steps of the methods described in the embodiments of the present invention. The foregoing storage medium includes: any medium that can store program codes, such as a USB flash disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM) , a random access memory (Random Access Memory, RAM) , a magnetic disk, or an optical disc.
Persons of ordinary skill in the art may clearly understand that, for the purpose of convenient and brief description, the division of the above functional modules is merely described for exemplary purposes, during practical use, the above functions may be assigned to different functional modules as required, that is, an internal structure of the apparatus is divided into different functional modules to perform all or a part of functions as described above. For a specific operating process of the above apparatus, reference may be made to corresponding procedures in the foregoing method embodiments, which will not be repeated herein.
Finally, it should be noted that the foregoing embodiments are merely intended for describing technical solutions of the present invention rather than limiting the present  invention. Although the present invention is described in detail with reference to the foregoing embodiments, persons of ordinary skill in the art should understand that they may still make modifications to the technical solutions described in the foregoing embodiments, or make equivalent replacements to some or all technical features therein； however, these modifications or replacements do not make the essence of corresponding technical solutions depart from the scope of the technical solutions in the embodiments of the present invention.

Claims (10)

  1. A method for data quality management and control, comprising:
    receiving application information transmitted by a service sub-system, wherein the application information comprises a service sub-system identifier, datasheet operation information and datasheet operation trigger information；
    resolving the datasheet operation trigger information to obtain datasheet flow information；
    receiving user information transmitted by the service sub-system and a target datasheet transmitted by the service sub-system；
    if the datasheet operation information is adding a datasheet, then querying pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier； if a name of the target datasheet is different from the plurality of datasheet names, then instructing the service sub-system to store the target datasheet into a data center；
    if the datasheet operation information is updating a datasheet, then querying pieces of history information according to the name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet, and instructing the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet；
    wherein the pieces of history information at least comprise the service sub-system identifier and the datasheet name.
  2. The method according to claim 1, after the instructing the service sub-system to store the target datasheet into the data center, further comprising:
    receiving a name of a tablespace storing the target datasheet as transmitted by the data center and an interface mode between the service sub-system and the data center transmitted by the data center； and
    generating current record information, wherein the current record information comprises the service sub-system identifier, the name of the tablespace, the name of the target datasheet,  the user information, the datasheet flow information, operation authorization information of the target datasheet and the interface mode.
  3. The method according to claim 2, after the receiving the user information transmitted by the service sub-system and the target datasheet transmitted by the service sub-system, further comprising:
    determining whether data in the target datasheet is in conformity with a preset data rule；
    if the data in the target datasheet is not in conformity with the preset data rule, then transmitting warning information to the service sub-system to enable a user to modify a data format in the target datasheet.
  4. The method according to claim 3, after the generating the current record information, further comprising:
    analyzing an importance degree of each datasheet according to the current record information and the pieces of history information, the more the datasheet flow information corresponding to the datasheet name, the higher the importance degree of the datasheet.
  5. The method according to claim 4, further comprising:
    setting a datasheet collecting rule, and collecting a plurality of datasheets from the data center according to the datasheet collecting rule；
    determining whether names of any two datasheets in the plurality of the datasheets are the same, if the names of the two datasheets are the same, then determining whether contents of the two datasheets are the same；
    if the contents of the two datasheets are the same, then transmitting a first deleting command to the data center to enable the data center to delete any one of the two datasheets；
    if the contents of the two datasheets are different, then acquiring timestamps of the two datasheets from the data center, and transmitting a second deleting command to the data center to enable the data center to delete a datasheet having a smaller timestamp in the two datasheets.
  6. An apparatus for data quality management and control, comprising:
    a receiving module, configured to receive application information transmitted by a service sub-system, wherein the application information comprises a service sub-system identifier, datasheet operation information and datasheet operation trigger information；  receive user information transmitted by the service sub-system and a target datasheet transmitted by the service sub-system；
    a resolving module, configured to resolve the datasheet operation trigger information to obtain datasheet flow information；
    a querying module, configured to: if the datasheet operation information is adding a datasheet, then query pieces of history information according to the service sub-system identifier to obtain a plurality of datasheet names corresponding to the service sub-system identifier； if the datasheet operation information is updating a datasheet, then query pieces of history information according to a name of the target datasheet to obtain a datasheet name that is the same as the name of the target datasheet； and
    a transmitting module, configured to: if the name of the target datasheet is different from the plurality of datasheet names, then instruct the service sub-system to store the target datasheet into a data center； and instruct the data center to replace datasheet contents corresponding to the datasheet name with contents of the target datasheet；
    wherein the pieces of history information at least comprise the service sub-system identifier and the datasheet name.
  7. The apparatus for data quality management and control according to claim 6, wherein the receiving module is further configured to, after the transmitting module instructs the service sub-system to store the target datasheet into the data center, receive a name of a tablespace storing the target datasheet as transmitted by the data center and an interface mode between the service sub-system and the data center transmitted by the data center；
    the apparatus for data quality management and control further comprises a recording module configured to generate current record information, wherein the current record information comprises the service sub-system identifier, the name of the tablespace, the name of the target datasheet, the user information, the datasheet flow information, operation authorization information of the target datasheet and the interface mode.
  8. The apparatus for data quality management and control according to claim 7, further comprising:
    a determining module, configured to: after the receiving module receives the user information transmitted by the service sub-system and the target datasheet transmitted by the  service sub-system, determine whether data in the target datasheet is in conformity with a preset data rule；
    the transmitting module is further configured to: if the data in the target datasheet is not in conformity with the preset data rule, then transmit warning information to the service sub-system to enable a user to modify a data format in the target datasheet.
  9. The apparatus for data quality management and control according to claim 8, wherein, the determining module is further configured to analyze an importance degree of each datasheet according to the current record information and the pieces of history information, the more the datasheet flow information corresponding to the datasheet name, the higher the importance degree of the datasheet.
  10. The apparatus for data quality management and control according to claim 9, further comprising:
    a collecting module, configured to set a datasheet collecting rule, and collect a plurality of datasheets from the data center according to the datasheet collecting rule；
    the determining module is further configured to determine whether names of any two datasheets in the plurality of the datasheets are the same, if the names of the two datasheets are the same, then determine whether contents of the two datasheets are the same；
    the transmitting module is further configured to: if the contents of the two datasheets are the same, then transmit a first deleting command to the data center to enable the data center to delete any one of the two datasheets； if the contents of the two datasheets are different, then acquire timestamps of the two datasheets from the data center, and transmit a second deleting command to the data center to enable the data center to delete a datasheet having a smaller timestamp in the two datasheets.
PCT/CN2016/097717 2015-12-04 2016-08-31 Method and apparatus for data quality management and control Ceased WO2017092447A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510889904.9 2015-12-04
CN201510889904.9A CN105512283B (en) 2015-12-04 2015-12-04 Data quality management control method and device

Publications (1)

Publication Number Publication Date
WO2017092447A1 true WO2017092447A1 (en) 2017-06-08

Family

ID=55720265

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/097717 Ceased WO2017092447A1 (en) 2015-12-04 2016-08-31 Method and apparatus for data quality management and control

Country Status (3)

Country Link
US (1) US10248674B2 (en)
CN (1) CN105512283B (en)
WO (1) WO2017092447A1 (en)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105512283B (en) * 2015-12-04 2019-05-03 国网江西省电力公司信息通信分公司 Data quality management control method and device
CN107222351A (en) * 2017-06-29 2017-09-29 济南浪潮高新科技投资发展有限公司 A kind of analysis method of the network equipment quality of data
CN108089872B (en) * 2017-12-19 2021-02-19 联想(北京)有限公司 Method for acquiring BIOS (basic input output System) update data, data update method and electronic equipment
CN108304464B (en) * 2017-12-26 2021-01-29 北京明略软件系统有限公司 Data cleaning method and device
CN108595563A (en) * 2018-04-13 2018-09-28 林秀丽 A kind of data quality management method and device
CN109034623B (en) * 2018-07-27 2021-11-30 国家电网有限公司 Power grid planning data processing method and terminal equipment
CN109933580B (en) * 2019-02-14 2020-12-25 北京奇艺世纪科技有限公司 Training data generation method and device and server
CN112243244A (en) * 2019-07-16 2021-01-19 中兴通讯股份有限公司 Homologous data processing device and method and base station
WO2021189283A1 (en) * 2020-03-25 2021-09-30 深圳市欢太科技有限公司 Method and device for data processing, electronic device, and storage medium
CN112052138A (en) * 2020-08-31 2020-12-08 平安科技（深圳）有限公司 Service data quality detection method and device, computer equipment and storage medium
CN114911781A (en) * 2022-03-31 2022-08-16 华能信息技术有限公司 Data quality management method and system
CN115455010B (en) * 2022-11-09 2023-02-28 以萨技术股份有限公司 Data processing method based on milvus database, electronic equipment and storage medium
CN117762949B (en) * 2024-02-19 2024-05-24 平凯星辰(北京)科技有限公司 Data extraction method, device, electronic equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102202073A (en) * 2010-03-24 2011-09-28 杭州华三通信技术有限公司 Distributed system and data synchronization method thereof
CN103617255A (en) * 2013-11-29 2014-03-05 国网电网公司 Service data exchange synchronizing system and method used for electric information system
US20150339329A1 (en) * 2013-03-14 2015-11-26 International Business Machines Corporation Row-based data filtering at a database level
CN105512283A (en) * 2015-12-04 2016-04-20 国网江西省电力公司信息通信分公司 Data quality management and control method and device

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050097122A1 (en) * 2003-08-29 2005-05-05 Thierry Schafflutzel Redundancy-free provision of multi-purpose data
US8082301B2 (en) * 2006-11-10 2011-12-20 Virtual Agility, Inc. System for supporting collaborative activity
US20110099095A1 (en) * 2009-10-28 2011-04-28 Microsoft Corporation Processing internal use of data-center resources
US8434128B2 (en) * 2010-02-22 2013-04-30 Avaya Inc. Flexible security requirements in an enterprise network
EP2766803B1 (en) * 2011-10-12 2022-12-07 Tata Consultancy Services Limited A method and system for consolidating a plurality of heterogeneous storage systems in a data center
US9445529B2 (en) * 2012-05-23 2016-09-13 International Business Machines Corporation Liquid cooled data center design selection
US9569476B2 (en) * 2013-04-02 2017-02-14 International Business Machines Corporation Intelligent data routing and storage provisioning
JP6820741B2 (en) * 2013-05-31 2021-01-27 コーニンクレッカ フィリップス エヌ ヴェＫｏｎｉｎｋｌｉｊｋｅ Ｐｈｉｌｉｐｓ Ｎ．Ｖ． Systems and methods for automatically downloading data such as sleep test data
WO2015119753A2 (en) * 2014-02-05 2015-08-13 Utc Fire & Security Americas Corporation, Inc. Uploading data from mobile devices
US9274710B1 (en) * 2014-03-31 2016-03-01 Amazon Technologies, Inc. Offset-based congestion control in storage systems
US9495478B2 (en) * 2014-03-31 2016-11-15 Amazon Technologies, Inc. Namespace management in distributed storage systems
US9824093B1 (en) * 2014-06-30 2017-11-21 EMC IP Holding Company LLC Datacenter maintenance
EP3224744A4 (en) * 2014-11-28 2018-08-01 Nasuni Corporation Versioned file system with global lock
CN104899257B (en) * 2015-05-18 2018-04-13 北京京东尚科信息技术有限公司 Data-updating method and device in Distributed Data Warehouse
US10168762B2 (en) * 2015-09-17 2019-01-01 Advanced Micro Devices, Inc. Power management for heterogeneous computing systems
US20160247243A1 (en) * 2016-04-29 2016-08-25 EdgeConneX, Inc. System for measuring, analyzing, allocating and provisioning available electrical capacity in a data center visa vie a sales order request to use electrical capacity
US10152490B2 (en) * 2015-12-29 2018-12-11 Successfactors, Inc. Sequential replication with limited number of objects

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102202073A (en) * 2010-03-24 2011-09-28 杭州华三通信技术有限公司 Distributed system and data synchronization method thereof
US20150339329A1 (en) * 2013-03-14 2015-11-26 International Business Machines Corporation Row-based data filtering at a database level
CN103617255A (en) * 2013-11-29 2014-03-05 国网电网公司 Service data exchange synchronizing system and method used for electric information system
CN105512283A (en) * 2015-12-04 2016-04-20 国网江西省电力公司信息通信分公司 Data quality management and control method and device

Also Published As

Publication number Publication date
CN105512283B (en) 2019-05-03
CN105512283A (en) 2016-04-20
US10248674B2 (en) 2019-04-02
US20170161306A1 (en) 2017-06-08

Similar Documents

Publication Publication Date Title
US10248674B2 (en) Method and apparatus for data quality management and control
CN113094385B (en) Data sharing fusion platform and method based on software defined open tool set
US12278809B2 (en) Computer-implemented methods, systems comprising computer-readable media, and electronic devices for narrative-structured representation of and intervention into a network computing environment
US9361338B2 (en) Offloaded, incremental database statistics collection and optimization
CN106815125A (en) A kind of log audit method and platform
US20100088197A1 (en) Systems and methods for generating remote system inventory capable of differential update reports
US20240364750A1 (en) Computer-implemented methods, systems comprising computer-readable media, and electronic devices for autonomous cybersecurity within a network computing environment
CN111767297B (en) Big data processing method, device, equipment and medium
CN118468320B (en) Data authority control method and system
CN111666205A (en) Data auditing method, system, computer equipment and storage medium
CN107491463B (en) Optimization method and system for data query
CN113810475A (en) Wifi probe equipment management and control system based on big data architecture
CN113010494A (en) Database auditing method and device and database proxy server
US20190005252A1 (en) Device for self-defense security based on system environment and user behavior analysis, and operating method therefor
US10003492B2 (en) Systems and methods for managing data related to network elements from multiple sources
CN118838944B (en) Correlation query method and device based on established model
CN110543509B (en) Monitoring system, method and device for user access data and electronic equipment
CN116628042B (en) Data processing method, device, equipment and medium
CN114780993A (en) Data calling method and device
CN113553344A (en) Business data management method, system, computer equipment and storage medium
CN107622124B (en) Data query method and system based on block data
WO2019120629A1 (en) On-demand snapshots from distributed data storage systems
CN119128941A (en) Access rights management method and device, storage medium and program product
CN121924054A (en) Task processing method, device, equipment and medium based on data probe
CN118193552A (en) Database-based data processing method, device and computer program product

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16869758

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16869758

Country of ref document: EP

Kind code of ref document: A1