CN111708779A - Data management method, system, management equipment and storage medium - Google Patents

Data management method, system, management equipment and storage medium Download PDF

Info

Publication number
CN111708779A
CN111708779A CN202010529626.7A CN202010529626A CN111708779A CN 111708779 A CN111708779 A CN 111708779A CN 202010529626 A CN202010529626 A CN 202010529626A CN 111708779 A CN111708779 A CN 111708779A
Authority
CN
China
Prior art keywords
data
field
name
mapping relation
warehouse
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010529626.7A
Other languages
Chinese (zh)
Inventor
孙清越
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
CCB Finetech Co Ltd
Original Assignee
China Construction Bank Corp
CCB Finetech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp, CCB Finetech Co Ltd filed Critical China Construction Bank Corp
Priority to CN202010529626.7A priority Critical patent/CN111708779A/en
Publication of CN111708779A publication Critical patent/CN111708779A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/283Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data management method, a data management system, a management device and a storage medium. The method comprises the following steps: importing a table structure of a data table according to a basic table of a data warehouse, wherein the data table comprises a table name and a field name; creating a field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in a data warehouse, wherein the field mapping relation table comprises: table name, field name, source table, and source field. According to the technical scheme, the bloody border relationship is established for each field of the data table in the data warehouse by creating the field mapping relationship table, so that the data can be conveniently tracked, and the reliability of data management is improved.

Description

Data management method, system, management equipment and storage medium
Technical Field
The embodiment of the invention relates to the technical field of databases, in particular to a data management method, a data management system, a data management device and a storage medium.
Background
An enterprise-level Data Warehouse (Data Warehouse) constructed based on a big Data technology can provide a strategic set of all types of Data support for decision making processes of all levels of an enterprise, and provides powerful Data support for enterprise development. The data warehouse can perform layered and parallel batch processing on mass data according to certain business rules based on the requirements of business scenes. The source service data is processed for many times in the data warehouse, and under the scene, the correctness of the data stored in the data warehouse and the consistency of the source service data and the processed data are the key for efficiently managing the data. In a big data era, the frequency of data updating is higher and higher, data needs to be tested or inquired frequently, and the like.
Disclosure of Invention
The invention provides a data management method, a data management system, a management device and a storage medium, which are used for comprehensively tracking data and improving the reliability of data management.
In a first aspect, an embodiment of the present invention provides a data management method, including:
importing a table structure of a data table according to a basic table of a data warehouse, wherein the data table comprises a table name and a field name;
creating a field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in a data warehouse, wherein the field mapping relation table comprises: table name, field name, source table, and source field.
Further, the importing a table structure of a data table according to a base table of a data warehouse includes:
filling a table structure of the data table according to the basic table template;
if a history table structure which is the same as the table structure is created, covering the history table structure with the table structure;
and if the history table structure identical to the table structure is not created, creating the table structure and adding the table name of the corresponding data table.
Further, the creating a field mapping relationship table according to the table structure and the mapping relationship of the data table in different data layers in the data warehouse includes:
if the data table associated with the field mapping relation table does not exist, loading the data table associated with the field mapping relation table;
and if the data table associated with the field mapping relation table exists, creating the field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in a data warehouse.
Further, the creating a field mapping relationship table further includes:
and if the field name of the field mapping relation table does not exist, generating field supplementary recording prompt information.
Further, the method also comprises the following steps:
and searching and modifying or deleting the corresponding field mapping relation in the field mapping relation table according to the changed data mapping document.
Further, the searching, modifying or deleting the corresponding field mapping relationship in the field mapping relationship table according to the changed data mapping document includes:
determining a field to be changed according to the changed data mapping document, and searching all field mapping relations associated with the field to be changed;
selecting fields to be changed in batch and modifying or deleting the fields to be changed according to all field mapping relations associated with the fields to be changed;
and if the modified field does not exist, importing the data table corresponding to the modified field.
Further, the method also comprises the following steps:
and generating a visualized field mesh structure relation graph according to the field mapping relation.
In a second aspect, an embodiment of the present invention provides a data management system, including: a management device and a data server;
the data server is used for providing a data warehouse, and processing and storing a data table of source service data based on different data layers through the data warehouse;
the management apparatus includes:
the import module is used for importing a table structure of a data table according to a basic table of the data warehouse, wherein the data table comprises a table name and a field name;
an acquisition module, configured to create a field mapping relationship table according to the table structure and mapping relationships of the data table in different data layers in a data warehouse, where the field mapping relationship table includes: table name, field name, source table, and source field.
In a third aspect, an embodiment of the present invention provides a management device, including:
one or more processors;
storage means for storing one or more programs;
when the one or more programs are executed by the one or more processors, cause the one or more processors to implement the data management method of the first aspect.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements the data management method according to the first aspect.
The embodiment of the invention provides a data management method, a system, a management device and a storage medium, wherein the method comprises the following steps: importing a table structure of a data table according to a basic table of a data warehouse, wherein the data table comprises a table name and a field name; creating a field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in a data warehouse, wherein the field mapping relation table comprises: table name, field name, source table, and source field. According to the technical scheme, the bloody border relationship is established for each field of the data table in the data warehouse by creating the field mapping relationship table, so that the data can be conveniently tracked, and the reliability of data management is improved.
Drawings
Fig. 1 is a flowchart of a data management method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a data warehouse according to an embodiment of the present invention;
fig. 3 is a flowchart of a data management method according to a second embodiment of the present invention;
fig. 4 is a flowchart illustrating an implementation of data presentation according to a second embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data management system according to a third embodiment of the present invention;
fig. 6 is a schematic diagram illustrating an implementation of a data management system according to a third embodiment of the present invention;
fig. 7 is a schematic diagram of a hardware structure of a management device according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. In addition, the embodiments and features of the embodiments in the present invention may be combined with each other without conflict. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a data management method according to an embodiment of the present invention, which is applicable to a situation where a blood relationship is established for data in a data warehouse for overall management. Specifically, the data management method may be executed by a management device in the data management system, where the management device includes, but is not limited to: desktop computers, notebook computers, integrated servers, cloud servers, and the like. The data management system also comprises a data server used for providing a data warehouse, and processing and storing the data table of the source service data based on different data layers through the data warehouse.
As shown in fig. 1, the method specifically includes the following steps:
and S110, importing a table structure of a data table according to a basic table of the data warehouse, wherein the data table comprises a table name and a field name.
Specifically, the data warehouse is a theme-Oriented (Subject organized), integrated (integrated), relatively stable (Non-Volatile), and historical change-reflecting (Time variance) data set, and is used for supporting management decisions, and performing operations such as extraction, screening, cleaning, and integration on source service data distributed everywhere according to requirements of decision analysis, and finally integrating the source service data into the data warehouse. The data warehouse at least comprises a data falling layer, a data subject layer and a data mart layer, and the source business data is processed in the data warehouse in a layering way. The base Table (Driving Table) is the underlying Table that stores information about the database, is the Table that is accessed first of all (typically accessed in a full Table scan) and is the object that is created first in any Oracle database. A table structure is a table used to describe data derived from one or more base tables, and may also be understood as a private table of several subset composition services extracted from existing base tables, the table structure being built on top of the base tables, with both its structure (i.e. the defined columns) and its content (i.e. all data rows) coming from the base tables. One table structure may correspond to one or more base tables. The information of each data table includes a table name and a field name.
In this embodiment, the table structure is determined according to the base table of the data warehouse, and the table structure is imported into the data management system for storage. The information for each data table in the data warehouse includes a table name and a field name, which are stored with the table structure.
S120, creating a field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in a data warehouse, wherein the field mapping relation table comprises: table name, field name, source table, and source field.
Specifically, each field in each data table is processed by different data layers (for example, at a data falling layer, a data subject layer and a data mart layer) in the data warehouse, and the data tables or fields have a certain mapping relationship among the different data layers and have traceability. When a certain data table or a certain field needs to be queried or changed, the position of the data table or the field in all data layers needs to be queried or changed. According to the embodiment, a data tracking platform can be provided for the data tables by constructing a data management system, and the field mapping relation table of each field can be obtained by the management device according to the imported table structure and the mapping relation of each data table between different data layers in the data warehouse, wherein the field mapping relation table is used for reflecting which basic table in the table structure should be accessed from first for one field, and on the basis of the basic table, other basic tables with the field need to be accessed, so that a consanguinity relation is established for the fields in each data table, and a reliable basis is provided for comprehensive access.
In this embodiment, the field mapping relationship table includes the following fields: table names, field names, source table names and source fields, so that the source and the association relation of each field among data packages at different layers of the data warehouse are reflected.
In this embodiment, the content that the data management system needs to store mainly includes: a base data table and a relational data table. The basic data table is mainly used for storing field information of each data table in the data warehouse, and the field information comprises a field ID, a field name, a data table name ID, whether the field is an initial node, whether the field is a leaf node, a level number and the like. In the data warehouse, the field of the data buffer layer is set as an initial node, and the field of the data table in the offsite mart is set as a leaf node. Due to the layered design of the data warehouse, data of each layer is processed on upper layer data, and fields correspond to different layer numbers at different levels. The field mapping relationship table created in this embodiment is substantially a relationship data table, and is mainly used for storing the relationship between fields, for example, including relationship ID, field ID, table ID, mapping field ID, mapping table ID, and the like. In addition, the data management system can also store a basic table name table which is mainly used for storing the names of all data tables in the enterprise-level data warehouse, including data table IDs, data table names, layer numbers and the like.
The management device of the embodiment is used for establishing a field mapping relation table by importing the table structure, constructing a data tracking platform by combining a data warehouse, and definitely establishing the blood relationship of data by using the mapping relation of each table field of the data warehouse, so that the whole data processing process in the data warehouse is conveniently managed in a unified manner.
Fig. 2 is a schematic diagram of a data warehouse according to an embodiment of the present invention. In this embodiment, the data warehouse is created based on an Oracle database. The data warehouse may be divided into five layers: the system comprises a data buffer layer, a data source attaching model layer, a theme model layer, a common processing layer and an in-bin market region, wherein the main functions of the layers are as follows:
data buffer Layer (LDS): and carrying source service data from all places without moving. The source service database has diversity, such as Oracle, Mysql, DB2 database, etc., and before extracting the source service data, according to the service rules and the constraint rules of the tables of the source service database, a first mapping relationship between the source data and the data buffer layer can be made, and accordingly, the source service data is extracted to the data buffer layer.
Data paste model layer (ODS): the data of the data layer is derived from the LDS layer, and the ODS layer needs to retain data of longer period. Meanwhile, the ODS layer provides data support for the common process layer and the in-bin bazaar. The data with high timeliness are also polymerized in the ODS layer. Data has a mapping relationship between the ODS layer and the LDS layer.
Theme model layer (Physical Data Management, PDM): and establishing an enterprise-level model according to the business theme of the enterprise, establishing a single data view of the enterprise level, and storing data used for analysis in the enterprise. The data of the data layer is from the LDS layer, and the data has a mapping relation with the PDM layer in the LDS layer.
Common processing region (General detail management, GDM): and analyzing the application commonality requirement to extract common data access and statistical requirement with commonality, thereby constructing an application-oriented common data providing shared data access service. The data of the data layer is derived from the PDM layer, and the data has a data mapping relation between the GDM layer and the PDM layer.
In-warehouse Data Model (ADM): and the system is oriented to a specific business system and is used for processing partial data which needs to be processed and processed in an application market. The data originates from the ODS layer, the LDS layer, and the GDM layer, and thus, there is a mapping relationship between the layers.
Based on the data warehouse, the source data of the front-end service system is extracted to the data buffer layer according to the mapping relation and the rule of the data; and then, the Oracle storage process processes the source data according to the data aggregation rule, the data mapping relation and the time sequence relation of the storage process, stores the data in the extrawarehouse mart, and provides a data access interface uniformly through the data service bus. According to the embodiment, a data warehouse and a data management system based on an Oracle database are built, the management equipment builds a data blood relationship by collecting the mapping relationship of data among all layers of the data warehouse, and creates a field mapping relationship table, so that a data processing result can be quickly and efficiently inquired, abnormal data can be positioned, or data can be changed.
According to the data management method provided by the embodiment of the invention, the table structure of the data table is imported according to the basic table of the data warehouse, the field mapping relation table is created according to the table structure and the mapping relation of the data table in different data layers of the data warehouse, the blood relationship is established for each field of the data table in the data warehouse, the data is convenient to track, and the reliability of data management is improved.
Example two
Fig. 3 is a flowchart of a data management method according to a second embodiment of the present invention, where the present embodiment is optimized based on the second embodiment, and specifically describes a display process of importing a table structure, creating a mapping table, and mapping a field. It should be noted that technical details that are not described in detail in the present embodiment may be referred to any of the above embodiments.
Specifically, as shown in fig. 3, the method specifically includes the following steps:
s201, filling a table structure of the data table according to the basic table template.
Specifically, the basic table template mainly includes two fields: table names and field names of the data tables. In this embodiment, the table name and the field name of each data table are filled in the basic table template on the basis of the basic table in the data warehouse, so as to obtain the table structure.
S202, has created a history table structure identical to the table structure? If yes, go to S203; if not, go to S204.
Specifically, if the same table structure has already been created, the created table structure is overwritten, and if the same table structure has not already been created, a new table structure is directly created and the table name of the corresponding data table is added.
And S203, covering the history table structure.
And S204, creating the table structure and adding the table name of the corresponding data table.
S205, there is a data table associated with the field mapping relationship table? If yes, executing S207; if not, go to S206.
Specifically, in the process of creating the field mapping relationship table, the management device needs to establish the field mapping relationship based on the created data table, and first determines whether the data table related to the field mapping relationship is imported into the system, if the relevant data table is not imported, the relevant data table is loaded first, and the field mapping relationship table is created after the loading is completed; if the relevant data table has already been imported, it is further determined whether the corresponding field name already exists.
S206, loading the data table associated with the field mapping relation table.
S207, field name of field mapping table exists? If yes, go to S209; if not, go to step S208.
Specifically, if the field name of the field mapping relationship table already exists, the field mapping relationship table may be created; if the field name of the field mapping relation table does not exist, field additional entry prompt information needs to be generated to prompt a user to perform additional entry on the corresponding field, and the field mapping relation table is created after the additional entry is completed.
And S208, generating field supplementary recording prompt information.
S209, creating the field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in the data warehouse.
S210, searching and modifying or deleting the corresponding field mapping relation in the field mapping relation table according to the changed data mapping document.
Specifically, due to the continuous change of the business requirements, the data tables and fields in each layer in the data warehouse also change, and a changed data mapping document is generated. The management equipment can realize the functions of inquiring, modifying or deleting the mapping relation among the fields in the data warehouse according to the changed data mapping document, thereby ensuring the dynamic update and the accuracy of the field mapping relation.
And S211, generating a visualized field mesh structure relation graph according to the field mapping relation.
Specifically, based on the data table and the field mapping relationship, a field mesh structure relationship diagram can be displayed, for example, a blood relationship diagram of fields from top to bottom is displayed in a tree form, so that developers and testers can more intuitively and conveniently and comprehensively master the field mapping relationship between different data layers of the data warehouse.
Fig. 4 is a flowchart illustrating an implementation of data presentation according to a second embodiment of the present invention. As shown in fig. 4, the data management system may log in by the user and select a data display mode in a platform interface form, where the display mode includes a surface display and a field display; if the user selects the surface layer indication, a basic table in the system is screened, and if the user selects the field display, the field to be displayed needs to be input; the user may then select the data layer to be presented, which may be all or part of the data layers in the data warehouse; according to the display mode and the data layer selected by the user, the blooding relationships of the fields or the corresponding data tables among different data layers can be displayed in a tree form. For example, the base table corresponding to the field input by the user is used as the root of the tree structure, the data table related to the base table is used as the branch, the field of the data buffer layer in the data warehouse can be the initial node, and the field in the offsite mart can be the leaf node. When the field needs to be inquired or changed, the fields can be intuitively and conveniently positioned to all related data tables from the root of the tree structure along different branches according to the field mesh structure relation graph, so that the fields are selected in batches, and the fields are modified or deleted in batches.
In an embodiment, S210 specifically includes: determining a field to be changed according to the changed data mapping document, and searching all field mapping relations associated with the field to be changed; selecting fields to be changed in batch and modifying or deleting the fields to be changed according to all field mapping relations associated with the fields to be changed; and if the modified field does not exist, importing the data table corresponding to the modified field.
In the embodiment, the field to be changed is determined according to the changed data mapping document generated by the data warehouse, the change mainly refers to the modification or deletion of data, and only one field to be changed can be used, so that the field can be directly modified or deleted; there may be a plurality of fields to be changed, which appear at different positions of a plurality of layers, have undergone different processing, etc., and in this case, it is necessary to select and modify or delete fields in batches. In addition, if the field to be changed is modified, it is also determined whether the modified new field and the corresponding data table are imported into the data management system, and if not, the data table corresponding to the modified field needs to be imported, so as to update the field mapping relationship.
In the data management method provided by the second embodiment of the present invention, optimization is performed on the basis of the above embodiments, a blood relationship of data is established by using a mapping relationship of fields of each data table of a data warehouse, and in the process of creating a field mapping relationship, a missing data table can be introduced in time or a nonexistent field can be subjected to additional entry, so as to ensure the correctness of the field mapping relationship; on the basis, the data processing process can be comprehensively inquired, and the fields are modified or deleted in batches, so that the data management and change efficiency is improved; by using the data table and the fields as the basis and displaying the blood relationship diagram of the tree-shaped fields from top to bottom, developers and testers can conveniently and visually master the field mapping relationship among different data layers of the data warehouse conveniently and comprehensively.
EXAMPLE III
Fig. 5 is a schematic structural diagram of a data management system according to a third embodiment of the present invention. As shown in fig. 5, the data management system provided in this embodiment includes: a management device 310 and a data server 320;
the data server 320 is used for providing a data warehouse through which data tables of source service data are processed and stored based on different data layers;
the management device 310 includes:
the import module 311 is configured to import a table structure of a data table according to a base table of a data warehouse, where the data table includes a table name and a field name;
an acquisition module 312, configured to create a field mapping relationship table according to the table structure and mapping relationships of the data table in different data layers in the data warehouse, where the field mapping relationship table includes: table name, field name, source table, and source field.
On the basis of the foregoing embodiment, the importing module 311 is specifically configured to:
filling a table structure of the data table according to the basic table template;
if a history table structure which is the same as the table structure is created, covering the history table structure with the table structure;
and if the history table structure identical to the table structure is not created, creating the table structure and adding the table name of the corresponding data table.
On the basis of the foregoing embodiment, the acquisition module 320 is specifically configured to:
if the data table associated with the field mapping relation table does not exist, loading the data table associated with the field mapping relation table;
and if the data table associated with the field mapping relation table exists, creating the field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in a data warehouse.
On the basis of the above embodiment, the method further includes:
and the field additional recording module is used for generating field additional recording prompt information if the field name of the field mapping relation table does not exist.
On the basis of the above embodiment, the method further includes:
and the data change module is used for searching and modifying or deleting the corresponding field mapping relation in the field mapping relation table according to the changed data mapping document.
On the basis of the foregoing embodiment, the data change module is specifically configured to:
determining a field to be changed according to the changed data mapping document, and searching all field mapping relations associated with the field to be changed;
selecting fields to be changed in batch and modifying or deleting the fields to be changed according to all field mapping relations associated with the fields to be changed;
and if the modified field does not exist, importing the data table corresponding to the modified field.
On the basis of the above embodiment, the method further includes:
and the display module is used for generating a visualized field mesh structure relation graph according to the field mapping relation.
Fig. 6 is a schematic diagram illustrating an implementation of a data management system according to a third embodiment of the present invention. As shown in fig. 6, the data management system provides a data tracking platform, where the source service data and the data in the data warehouse may be data of Oracle, Mysql, DB2 databases, and according to the data mapping relationship document of each layer, (the table structure of) the data table of the data warehouse may be imported through a database adapter (i.e. an import module, as a database base table of the data tracking platform, and the main information stored in the table structure is a table name and a field name;
the upstream and downstream relation collectors (i.e. collection modules) can read corresponding mapping documents according to the mapping relation between the data tables of each data layer of the data warehouse, form the many-to-many relation between fields, and create and store a field mapping relation table, wherein the stored main information is table names, field names, source tables and source fields.
Further, the data management system further comprises: kindred relationship module (i.e., data alteration module): the module mainly searches, modifies or deletes the corresponding field mapping relation according to the changed data mapping document.
Further, the system also comprises a real-time data display module (namely a display module): the module forms a visualized mesh structure relationship diagram based on the field mapping relationship table. And the display module can self-define the display interface according to the depth of the field mapping relation.
The data management system of this embodiment realizes that data management provides reliable data support for the enterprise to visual show mode does benefit to the blood relationship that developer and tester clearly mastered the data, can both the quick positioning data problem in development stage and test stage, carries out inquiry or change high-efficiently, ensures the uniformity of data, improves the data quality in the data warehouse.
The data management system provided by the third embodiment of the present invention can be used to execute the data management method provided by any of the above embodiments, and has corresponding functions and beneficial effects.
Example four
Fig. 7 is a schematic diagram of a hardware structure of a management device according to a fourth embodiment of the present invention. Management devices include, but are not limited to: desktop computer, notebook computer, integration server and high in the clouds server. As shown in fig. 7, the management device provided in this embodiment includes: a processor 410 and a storage 420. The number of the processors in the management device may be one or more, fig. 7 illustrates one processor 410, the processor 410 and the storage device 420 in the management device may be connected by a bus or in other manners, and fig. 7 illustrates the connection by a bus.
The one or more programs are executed by the one or more processors 410, causing the one or more processors to implement the data management method of any of the embodiments described above.
The storage device 420 in the management apparatus is used as a computer-readable storage medium, and can be used to store one or more programs, which may be software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the data management method in the embodiment of the present invention (for example, the modules in the data management system shown in the figure, including the importing module 311 and the collecting module 312). The processor 410 executes various functional applications of the management apparatus and data processing by executing software programs, instructions, and modules stored in the storage device 420, that is, implements the data management method in the above-described method embodiment.
The storage device 420 mainly includes a storage program area and a storage data area, wherein the storage program area can store an operating system and an application program required by at least one function; the storage data area may store data or the like created according to the use of the management apparatus (such as the field mapping table, the table structure, and the like in the above-described embodiments). Further, the storage 420 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the storage 420 may further include memory located remotely from the processor 410, which may be connected to the management device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
And, when one or more programs included in the above-described management apparatus are executed by the one or more processors 410, the following operations are performed: importing a table structure of a data table according to a basic table of a data warehouse, wherein the data table comprises a table name and a field name; creating a field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in a data warehouse, wherein the field mapping relation table comprises: table name, field name, source table, and source field.
The management device proposed by the present embodiment is the same inventive concept as the data management method proposed by the above embodiments, and technical details that are not described in detail in the present embodiment can be referred to any of the above embodiments, and the present embodiment has the same advantageous effects as the data management method.
On the basis of the above-described embodiments, the present embodiment also provides a computer-readable storage medium on which a computer program is stored, the program implementing the data management method in any of the above-described embodiments of the present invention when executed by a management apparatus, the method including: importing a table structure of a data table according to a basic table of a data warehouse, wherein the data table comprises a table name and a field name; creating a field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in a data warehouse, wherein the field mapping relation table comprises: table name, field name, source table, and source field.
Of course, the storage medium provided by the embodiment of the present invention includes computer-executable instructions, and the computer-executable instructions are not limited to the operations of the data management method described above, and may also perform related operations in the data management method provided by any embodiment of the present invention, and have corresponding functions and advantages.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the data management method according to the embodiments of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (10)

1. A method for managing data, comprising:
importing a table structure of a data table according to a basic table of a data warehouse, wherein the data table comprises a table name and a field name;
creating a field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in a data warehouse, wherein the field mapping relation table comprises: table name, field name, source table, and source field.
2. The method of claim 1, wherein importing the table structure of the data table from the base table of the data warehouse comprises:
filling a table structure of the data table according to the basic table template;
if a history table structure which is the same as the table structure is created, covering the history table structure with the table structure;
and if the history table structure identical to the table structure is not created, creating the table structure and adding the table name of the corresponding data table.
3. The method of claim 1, wherein creating a field mapping table according to the table structure and the mapping of the data table to different data layers in a data warehouse comprises:
if the data table associated with the field mapping relation table does not exist, loading the data table associated with the field mapping relation table;
and if the data table associated with the field mapping relation table exists, creating the field mapping relation table according to the table structure and the mapping relation of the data table in different data layers in a data warehouse.
4. The method of claim 3, further comprising:
and if the field name of the field mapping relation table does not exist, generating field supplementary recording prompt information.
5. The method of claim 1, further comprising:
and searching and modifying or deleting the corresponding field mapping relation in the field mapping relation table according to the changed data mapping document.
6. The method according to claim 5, wherein the finding, modifying or deleting the corresponding field mapping relationship in the field mapping relationship table according to the changed data mapping document comprises:
determining a field to be changed according to the changed data mapping document, and searching all field mapping relations associated with the field to be changed;
selecting fields to be changed in batch and modifying or deleting the fields to be changed according to all field mapping relations associated with the fields to be changed;
and if the modified field does not exist, importing the data table corresponding to the modified field.
7. The method of any one of claims 1-6, further comprising:
and generating a visualized field mesh structure relation graph according to the field mapping relation.
8. A data management system, comprising: a management device and a data server;
the data server is used for providing a data warehouse, and processing and storing a data table of source service data based on different data layers through the data warehouse;
the management apparatus includes:
the import module is used for importing a table structure of a data table according to a basic table of the data warehouse, wherein the data table comprises a table name and a field name;
an acquisition module, configured to create a field mapping relationship table according to the table structure and mapping relationships of the data table in different data layers in a data warehouse, where the field mapping relationship table includes: table name, field name, source table, and source field.
9. A management device, comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data management method as recited in any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data management method according to any one of claims 1 to 7.
CN202010529626.7A 2020-06-11 2020-06-11 Data management method, system, management equipment and storage medium Pending CN111708779A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010529626.7A CN111708779A (en) 2020-06-11 2020-06-11 Data management method, system, management equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010529626.7A CN111708779A (en) 2020-06-11 2020-06-11 Data management method, system, management equipment and storage medium

Publications (1)

Publication Number Publication Date
CN111708779A true CN111708779A (en) 2020-09-25

Family

ID=72540145

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010529626.7A Pending CN111708779A (en) 2020-06-11 2020-06-11 Data management method, system, management equipment and storage medium

Country Status (1)

Country Link
CN (1) CN111708779A (en)

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112260874A (en) * 2020-10-23 2021-01-22 南京鹏云网络科技有限公司 Management system and method based on distributed storage unit
CN112395850A (en) * 2020-11-05 2021-02-23 中国人寿保险股份有限公司 Method and device for checking data definition consistency and electronic equipment
CN112685415A (en) * 2020-12-30 2021-04-20 平安普惠企业管理有限公司 Data import method and device, computer equipment and storage medium
CN112817984A (en) * 2021-02-22 2021-05-18 杭州数梦工场科技有限公司 Data processing method and device, and data source obtaining method and device
CN113111084A (en) * 2021-03-31 2021-07-13 北京沃东天骏信息技术有限公司 Method and device for processing data
CN113486008A (en) * 2021-06-30 2021-10-08 平安信托有限责任公司 Data blood margin analysis method, device, equipment and storage medium
CN113688131A (en) * 2021-08-09 2021-11-23 招商银行股份有限公司 Method, system, device and computer program product for field level impact analysis
CN113722341A (en) * 2021-08-31 2021-11-30 北京达佳互联信息技术有限公司 Operation data processing method and related device
CN114064752A (en) * 2021-11-09 2022-02-18 珠海市新德汇信息技术有限公司 Data influence analysis method based on record-level blood relationship, storage medium and equipment
CN114064640A (en) * 2021-11-09 2022-02-18 珠海市新德汇信息技术有限公司 Blood relationship construction method, storage medium and equipment applied to data tracing
CN114328759A (en) * 2021-12-28 2022-04-12 厦门市美亚柏科信息股份有限公司 Data construction and management method and terminal of data warehouse
CN114490627A (en) * 2020-10-27 2022-05-13 杭州数梦工场科技有限公司 Data processing method and device, electronic equipment and storage medium
CN114936199A (en) * 2022-07-21 2022-08-23 平安银行股份有限公司 Data processing method for system reconstruction, computer equipment and storage medium
CN115794827A (en) * 2022-11-29 2023-03-14 广发银行股份有限公司 Data table structure management system and method

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2011213842A1 (en) * 2010-09-03 2012-03-22 Tata Consultancy Services Limited A system and method of managing mapping information
CN108520008A (en) * 2018-03-15 2018-09-11 链家网(北京)科技有限公司 The construction method and construction device of data warehouse model
CN108984620A (en) * 2018-06-14 2018-12-11 中国科学院重庆绿色智能技术研究院 A kind of relevant database and its design method of dynamic scalable
CN109800273A (en) * 2018-12-03 2019-05-24 北京明略软件系统有限公司 A kind of method, apparatus of data processing, computer storage medium and terminal
CN110780860A (en) * 2019-09-10 2020-02-11 中国平安财产保险股份有限公司 Generation method and device of table building script, computer equipment and storage medium
CN110807033A (en) * 2019-10-12 2020-02-18 中思博安科技(北京)有限公司 Data management method, device and system
CN111159184A (en) * 2019-12-25 2020-05-15 上海中信信息发展股份有限公司 Metadata tracing method and device and server
CN111241185A (en) * 2020-04-26 2020-06-05 浙江网商银行股份有限公司 Data processing method and device

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2011213842A1 (en) * 2010-09-03 2012-03-22 Tata Consultancy Services Limited A system and method of managing mapping information
CN108520008A (en) * 2018-03-15 2018-09-11 链家网(北京)科技有限公司 The construction method and construction device of data warehouse model
CN108984620A (en) * 2018-06-14 2018-12-11 中国科学院重庆绿色智能技术研究院 A kind of relevant database and its design method of dynamic scalable
CN109800273A (en) * 2018-12-03 2019-05-24 北京明略软件系统有限公司 A kind of method, apparatus of data processing, computer storage medium and terminal
CN110780860A (en) * 2019-09-10 2020-02-11 中国平安财产保险股份有限公司 Generation method and device of table building script, computer equipment and storage medium
CN110807033A (en) * 2019-10-12 2020-02-18 中思博安科技(北京)有限公司 Data management method, device and system
CN111159184A (en) * 2019-12-25 2020-05-15 上海中信信息发展股份有限公司 Metadata tracing method and device and server
CN111241185A (en) * 2020-04-26 2020-06-05 浙江网商银行股份有限公司 Data processing method and device

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112260874A (en) * 2020-10-23 2021-01-22 南京鹏云网络科技有限公司 Management system and method based on distributed storage unit
CN114490627A (en) * 2020-10-27 2022-05-13 杭州数梦工场科技有限公司 Data processing method and device, electronic equipment and storage medium
CN112395850A (en) * 2020-11-05 2021-02-23 中国人寿保险股份有限公司 Method and device for checking data definition consistency and electronic equipment
CN112685415A (en) * 2020-12-30 2021-04-20 平安普惠企业管理有限公司 Data import method and device, computer equipment and storage medium
CN112817984B (en) * 2021-02-22 2023-10-20 杭州数梦工场科技有限公司 Data processing method and device, and data source acquisition method and device
CN112817984A (en) * 2021-02-22 2021-05-18 杭州数梦工场科技有限公司 Data processing method and device, and data source obtaining method and device
CN113111084A (en) * 2021-03-31 2021-07-13 北京沃东天骏信息技术有限公司 Method and device for processing data
CN113486008A (en) * 2021-06-30 2021-10-08 平安信托有限责任公司 Data blood margin analysis method, device, equipment and storage medium
CN113688131A (en) * 2021-08-09 2021-11-23 招商银行股份有限公司 Method, system, device and computer program product for field level impact analysis
CN113688131B (en) * 2021-08-09 2024-03-26 招商银行股份有限公司 Method, system, device and computer program product for analyzing field level influence
CN113722341A (en) * 2021-08-31 2021-11-30 北京达佳互联信息技术有限公司 Operation data processing method and related device
CN113722341B (en) * 2021-08-31 2024-03-01 北京达佳互联信息技术有限公司 Operation data processing method and related device
CN114064752A (en) * 2021-11-09 2022-02-18 珠海市新德汇信息技术有限公司 Data influence analysis method based on record-level blood relationship, storage medium and equipment
CN114064640A (en) * 2021-11-09 2022-02-18 珠海市新德汇信息技术有限公司 Blood relationship construction method, storage medium and equipment applied to data tracing
CN114328759A (en) * 2021-12-28 2022-04-12 厦门市美亚柏科信息股份有限公司 Data construction and management method and terminal of data warehouse
CN114936199A (en) * 2022-07-21 2022-08-23 平安银行股份有限公司 Data processing method for system reconstruction, computer equipment and storage medium
CN115794827A (en) * 2022-11-29 2023-03-14 广发银行股份有限公司 Data table structure management system and method
CN115794827B (en) * 2022-11-29 2023-07-21 广发银行股份有限公司 Data table structure management system and method

Similar Documents

Publication Publication Date Title
CN111708779A (en) Data management method, system, management equipment and storage medium
US9519701B2 (en) Generating information models in an in-memory database system
US9652497B2 (en) Processing queries using hybrid access paths
CN112966004B (en) Data query method, device, electronic equipment and computer readable medium
US8959475B2 (en) Generating a semantic graph that allows a user to intelligently and efficiently connect with people and assets involved in projects
US10515069B2 (en) Utilization of a concept to obtain data of specific interest to a user from one or more data storage locations
US9798833B2 (en) Accessing information content in a database platform using metadata
EP1989644A2 (en) Systems and methods for finding log files generated by a distributed computer
CN113312392A (en) Lightweight rule engine processing method and device
US10536381B2 (en) Determining connections of a network between source and target nodes in a database
US20200342029A1 (en) Systems and methods for querying databases using interactive search paths
US20180365312A1 (en) Systems, Methods, and Apparatuses for Creating, Linking and Discovering Business Navigation Maps for Analyzing Data
US10169725B2 (en) Change-request analysis
CN117829657A (en) Spacecraft system evaluation system and evaluation method
CN117453980A (en) Metadata management, configuration page generation method, server and storage medium
US20120159376A1 (en) Editing data records associated with static images
CN116737113A (en) Metadata catalog management system and method for mass scientific data
CN110928895A (en) Data query method, data table establishing method, device and equipment
US9886520B2 (en) Exposing relationships between universe objects
CN114218590A (en) Authority configuration method, device, equipment and medium
US20070022081A1 (en) Record of data repository access
WO2015021907A1 (en) Utilization of a concept to obtain data of specific interest to a user from one or more data storage locations
CN117527785B (en) Method and system for supporting space engineering file data uploading and full link management
CN117421297B (en) Statistical information collection method for dynamic partition table of distributed database
US11995584B2 (en) Training assignment tool

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20220919

Address after: 12 / F, 15 / F, 99 Yincheng Road, China (Shanghai) pilot Free Trade Zone, Pudong New Area, Shanghai, 200120

Applicant after: Jianxin Financial Science and Technology Co.,Ltd.

Address before: 25 Financial Street, Xicheng District, Beijing 100033

Applicant before: CHINA CONSTRUCTION BANK Corp.

Applicant before: Jianxin Financial Science and Technology Co.,Ltd.