CN113535722A

CN113535722A - DAG (demand directed Access control) source tracing sampling method, system, equipment and storage medium based on mapping

Info

Publication number: CN113535722A
Application number: CN202110784915.6A
Authority: CN
Inventors: 高晗露
Original assignee: Beijing Minglue Zhaohui Technology Co Ltd
Current assignee: Beijing Minglue Zhaohui Technology Co Ltd
Priority date: 2021-07-12
Filing date: 2021-07-12
Publication date: 2021-10-22

Abstract

The invention discloses a DAG (demand-directed Access control) source tracing sampling method, a system, equipment and a storage medium based on mapping, wherein the method comprises the following steps: acquiring an input field and an output field of an operator node, and generating a mapping map; acquiring all father nodes of the operator nodes and storing the father nodes as DAG; sequentially inquiring whether a father node is an original table or not by adopting a DAG (directed access) recursive algorithm, if so, searching a corresponding original table field through a mapping map, and acquiring corresponding sample data through a database according to the original table field; and returning the obtained sample data and the mapping map. When field mapping recommendation is carried out, through DAG tracing, a source can be found more correctly, and original table sample data can be taken quickly and accurately for recommendation; the data table is free of manual filling, data disorder caused by multiple data tables is avoided, accuracy is high, and user experience is improved.

Description

DAG (demand directed Access control) source tracing sampling method, system, equipment and storage medium based on mapping

Technical Field

The invention relates to the technical field of data management, in particular to a DAG (demand-directed Access) traceability sampling method, system, equipment and storage medium based on mapping.

Background

In the data processing, a plurality of intermediate tables, dependence and mapping relations are generated from the original table to the corresponding target table, a DAG graph is formed, when the DAG is constructed, field mapping recommendation is carried out when an operator needs sample data of the original table, and since the intermediate table is not processed and has no corresponding data, recursive tracing is carried out to find relevant data of the original table to carry out recommendation mapping. In the existing method, the related information of an original table is manually carried in data for treatment in the process of constructing a DAG (direct current access), but the method can cause data disorder when a plurality of data tables exist, and the accuracy rate of the manual information is low.

Disclosure of Invention

Aiming at the technical problems that the method for carrying out data management by manually carrying information is low in accuracy and can cause data disorder, the invention provides a DAG (direct current) traceability sampling method, a DAG traceability sampling system, DAG traceability sampling equipment and a storage medium based on mapping.

In a first aspect, an embodiment of the present application provides a DAG source tracing sampling method based on mapping, including:

a mapping map generation step: acquiring an input field and an output field of an operator node, and generating a mapping map;

a DAG obtaining step: acquiring all father nodes of the operator nodes and storing the father nodes as DAG;

parent node query step: sequentially inquiring whether the father node is an original table or not by the DAG through a recursive algorithm, if so, searching a corresponding original table field through the mapping map, and acquiring corresponding sample data through a database according to the original table field;

and a data return step: and returning the acquired sample data and the mapping map.

In the above mapping-based DAG source tracing sampling method, the parent node querying step further includes:

and (3) mapping operator query step: if not, inquiring whether the type of the father node is a mapping operator, and if the type is the mapping operator and the output field of the field mapping stored in the database is the same as the key value in the mapping map, storing the input field of the field mapping stored in the database into the mapping map.

The mapping-based DAG source tracing sampling method includes the steps of:

a field acquisition step: connecting the database to obtain the input field and the output field of the operator node and storing the input field and the output field in Map < String, List >;

field arrangement: and sorting the input field and the output field of the operator node, and respectively storing the input field and the output field into an inputMap and an OutputMap to generate the mapping map.

In the DAG source tracing sampling method based on mapping, in the mapping map, key is field id, and value is "table type # field id".

The mapping-based DAG source tracing sampling method further includes:

field mapping recommendation step: and recommending field mapping according to the acquired sample data.

In a second aspect, an embodiment of the present application provides a mapping-based DAG traceability sampling system, including:

a map generation unit: acquiring an input field and an output field of an operator node, and generating a mapping map;

a DAG acquisition unit: acquiring all father nodes of the operator nodes and storing the father nodes as DAG;

a parent node query unit: sequentially inquiring whether the father node is an original table or not by the DAG through a recursive algorithm, if so, searching a corresponding original table field through the mapping map, and acquiring corresponding sample data through a database according to the original table field;

a data return unit: returning the obtained sample data and the mapping map;

a field mapping recommendation unit: and recommending field mapping according to the acquired sample data.

The DAG source tracing sampling system based on mapping described above, wherein the parent node query unit further includes:

the mapping operator query module: and if the father node is not an original table, inquiring whether the type of the father node is a mapping operator, and if the type is the mapping operator and the output field of the field mapping stored in the database is the same as the key value in the mapping map, storing the input field of the field mapping stored in the database into the mapping map.

The mapping-based DAG traceability sampling system, wherein the mapping map generating unit comprises:

a field acquisition module: connecting the database to obtain the input field and the output field of the operator node and storing the input field and the output field in Map < String, List >;

a field arrangement module: and sorting the input field and the output field of the operator node, and respectively storing the input field and the output field into an inputMap and an OutputMap to generate the mapping map.

In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor, when executing the computer program, implements the mapping-based DAG traceability sampling method according to the first aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, implements the mapping-based DAG source-tracing sampling method according to the first aspect.

Compared with the prior art, the invention has the advantages and positive effects that:

1. when field mapping recommendation is carried out, through DAG tracing, a source can be found more correctly, and original table sample data can be taken quickly and accurately for recommendation;

2. the data management method has the advantages that manual filling is not needed, data disorder caused by multiple data tables is avoided, accuracy is high, user experience is improved, and data management capability is improved.

Drawings

FIG. 1 is a schematic diagram illustrating steps of a DAG source tracing sampling method based on mapping according to the present invention;

FIG. 2 is a schematic flow chart based on step S1 in FIG. 1 according to the present invention;

FIG. 3 is a schematic flow chart of a DAG source tracing sampling method based on mapping according to the present invention;

FIG. 4 is a flowchart illustrating a DAG source tracing sampling method according to an embodiment of the present invention;

FIG. 5 is a block diagram of a DAG traceability sampling system based on mapping according to the present invention;

fig. 6 is a block diagram of a computer device according to an embodiment of the present application.

Wherein the reference numerals are:

1. a map generation unit; 11. a field acquisition module; 12. a field arrangement module; 2. a DAG acquisition unit; 3. a parent node query unit; 31. a mapping operator query module; 4. a data return unit; 5. a field mapping recommendation unit; 81. a processor; 82. a memory; 83. a communication interface; 80. a bus.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.

It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.

The present invention is described in detail with reference to the embodiments shown in the drawings, but it should be understood that these embodiments are not intended to limit the present invention, and those skilled in the art should understand that functional, methodological, or structural equivalents or substitutions made by these embodiments are within the scope of the present invention.

Before describing in detail the various embodiments of the present invention, the core inventive concepts of the present invention are summarized and described in detail by the following several embodiments.

The invention provides a method for carrying out DAG recursion to find sample data for field recommendation based on field mapping between tables, which can quickly and accurately obtain the sample data of an original table for recommendation.

The first embodiment is as follows:

fig. 1 is a schematic step diagram of a DAG source tracing sampling method based on mapping according to the present invention. As shown in fig. 1, this embodiment discloses a specific implementation of a DAG source-tracing sampling method (hereinafter referred to as "method") based on mapping.

Specifically, the method disclosed in this embodiment mainly includes the following steps:

step S1: acquiring an input field and an output field of an operator node, and generating a mapping map;

as shown in fig. 2, step S1 specifically includes the following steps:

step S11: acquiring the input field and the output field of the operator node by connecting a database and storing the input field and the output field in Map < String, List >;

step S12: and sorting the input field and the output field of the operator node, and respectively storing the input field and the output field into an inputMap and an OutputMap to generate the mapping map. In the mapping map, key is field id, and value is 'table type # field id'.

Step S2: acquiring all father nodes of the operator nodes and storing the father nodes as DAG;

step S3: sequentially inquiring whether the father node is an original table or not by the DAG through a recursive algorithm, if so, searching a corresponding original table field through the mapping map, and acquiring corresponding sample data through a database according to the original table field;

if not, inquiring whether the type of the father node is a mapping operator, and if the type is the mapping operator and the output field of the field mapping stored in the database is the same as the key value in the mapping map, storing the input field of the field mapping stored in the database into the mapping map.

Step S4: and returning the acquired sample data and the mapping map.

Step S5: and recommending field mapping according to the acquired sample data.

The invention aims to quickly and accurately take original table sample data for recommendation when field mapping recommendation is carried out. As shown in fig. 3, the present invention provides a method for performing field recommendation by finding sample data through DAG recursion based on field mapping between tables, which depends on a database of storage mapping and DAG, and the flow is as follows:

1. supposing that an operator node exists in the database, the input and the output of the operator node respectively have n and m fields, and matching is not performed;

2. connecting a database, acquiring the input and output fields of the node and storing the input and output fields in Map < String, List >, acquiring all father nodes of the node and storing the father nodes as DAG;

3. sorting input and output fields of operators, respectively storing the input and output fields into an inputMap and an OutputMap, and generating a mapping map, wherein key is field id, and value is 'table type # field id';

4. acquiring a superior father node list from a DAG, and taking one node x;

5. if x is an original table, searching a corresponding original table field through a mapping map, acquiring sample data in the original table field, executing the next step, otherwise, checking whether the x type is a mapping operator, and if so, storing a field mapping input field stored in a database into the mapping map by comparing an output field with the same key value in the mapping map;

6. if the father node list is not inquired, executing the steps of 4-5, otherwise returning to the mapping map and the sample data list;

7. and recommending the inter-operator field according to the sample data.

Referring to fig. 4, the application flow of the method is described as follows with reference to a specific embodiment:

as shown in FIG. 4, taking the example of a DAG, assume that there is one mapping operator node x, the input field: "name", "sex"; an output field: "person _ name", "person _ sex". The database now needs to be queried and the input field "name" matched to the output field "person _ name" according to the sample data content.

The specific implementation method comprises the following steps in sequence:

1. and connecting the database, acquiring the input and output fields of the node and storing the input and output fields in Map < String, List >, acquiring all the father nodes of the node and storing the father nodes as DAG.

2. And (3) sorting input and output fields of operators, respectively storing the input and output fields into an inputMap and an OutputMap, and generating a mapping map, wherein key is field id, and value is 'table type # field id'.

3. And acquiring a parent node list of the previous level from the DAG, acquiring a person of the previous level, wherein the person is neither an original table nor a mapping operator, continuously acquiring the node list of the previous level, acquiring a node y, and repeating the step.

4. y is a mapping operator, and the field mapping stored in the database is as follows: "teacher _ name" - > "name", "teacher _ six" - > "six", the input field is stored in the mapping map by comparing the same key value in the output field and the mapping map, at this time, "teacher _ name" - > "name", "teacher _ six" - > "six" is in the map, the previous level node list is continuously taken, one node, teacher _1, is taken, and the step is repeated.

5. After the y and the front node execute the above steps, returning to the father node list of the person, and finding that z nodes are not executed, repeating the 4 steps until the original table student is the original table, searching the corresponding original table field through mapping the map, obtaining the sample data therein, and searching the corresponding mapping in the map: "student _ name" - > "name", "student _ sex" - > "sex", and sample data is acquired.

6. And carrying out field mapping recommendation according to the acquired sample data.

Example two:

in combination with the mapping-based DAG source-tracing sampling method disclosed in the first embodiment, this embodiment discloses a specific implementation example of a mapping-based DAG source-tracing sampling system (hereinafter referred to as "system").

Referring to fig. 5, the system includes:

map generation unit 1: acquiring an input field and an output field of an operator node, and generating a mapping map;

DAG acquisition unit 2: acquiring all father nodes of the operator nodes and storing the father nodes as DAG;

parent node querying unit 3: sequentially inquiring whether the father node is an original table or not by the DAG through a recursive algorithm, if so, searching a corresponding original table field through the mapping map, and acquiring corresponding sample data through a database according to the original table field;

the data return unit 4: returning the obtained sample data and the mapping map;

the field mapping recommendation unit 5: and recommending field mapping according to the acquired sample data.

Specifically, the map generation unit 1 includes:

the field acquisition module 11: connecting the database to obtain the input field and the output field of the operator node and storing the input field and the output field in Map < String, List >;

the field sorting module 12: and sorting the input field and the output field of the operator node, and respectively storing the input field and the output field into an inputMap and an OutputMap to generate the mapping map.

Specifically, the parent node querying unit 3 further includes:

mapping operator query module 31: and if the father node is not an original table, inquiring whether the type of the father node is a mapping operator, and if the type is the mapping operator and the output field of the field mapping stored in the database is the same as the key value in the mapping map, storing the input field of the field mapping stored in the database into the mapping map.

For reference, please refer to the first embodiment, and details thereof are not repeated herein.

Example three:

referring to fig. 6, the present embodiment discloses an embodiment of a computer device. The computer device may comprise a processor 81 and a memory 82 in which computer program instructions are stored.

Specifically, the processor 81 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 82 may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory 82 may include a Hard Disk Drive (Hard Disk Drive, abbreviated to HDD), a floppy Disk Drive, a Solid State Drive (SSD), flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 82 may include removable or non-removable (or fixed) media, where appropriate. The memory 82 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 82 is a Non-Volatile (Non-Volatile) memory. In particular embodiments, Memory 82 includes Read-Only Memory (ROM) and Random Access Memory (RAM). The ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), Electrically rewritable ROM (EAROM), or FLASH Memory (FLASH), or a combination of two or more of these, where appropriate. The RAM may be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), where the DRAM may be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended data output Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), and the like.

The memory 82 may be used to store or cache various data files for processing and/or communication use, as well as possible computer program instructions executed by the processor 81.

The processor 81 implements any of the mapping-based DAG traceback sampling methods described in the embodiments above by reading and executing computer program instructions stored in the memory 82.

In some of these embodiments, the computer device may also include a communication interface 83 and a bus 80. As shown in fig. 6, the processor 81, the memory 82, and the communication interface 83 are connected via the bus 80 to complete communication therebetween.

The communication interface 83 is used for implementing communication between modules, devices, units and/or equipment in the embodiment of the present application. The communication port 83 may also be implemented with other components such as: the data communication is carried out among external equipment, image/data acquisition equipment, a database, external storage, an image/data processing workstation and the like.

Bus 80 includes hardware, software, or both to couple the components of the computer device to each other. Bus 80 includes, but is not limited to, at least one of the following: data Bus (Data Bus), Address Bus (Address Bus), Control Bus (Control Bus), Expansion Bus (Expansion Bus), and Local Bus (Local Bus). By way of example, and not limitation, Bus 80 may include an Accelerated Graphics Port (AGP) or other Graphics Bus, an Enhanced Industry Standard Architecture (EISA) Bus, a Front-Side Bus (FSB), a Hyper Transport (HT) Interconnect, an ISA (ISA) Bus, an InfiniBand (InfiniBand) Interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a microchannel Architecture (MCA) Bus, a PCI (Peripheral Component Interconnect) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a Video Electronics Bus (audio Electronics Association), abbreviated VLB) bus or other suitable bus or a combination of two or more of these. Bus 80 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

In addition, in combination with the mapping-based DAG source tracing sampling method in the foregoing embodiments, embodiments of the present application may provide a computer-readable storage medium to implement the mapping-based DAG source tracing sampling method. The computer readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the mapping-based DAG traceback sampling methods of the embodiments described above.

The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

In summary, the method has the advantages that when field mapping recommendation is performed, through DAG tracing, a source can be found more correctly, and original table sample data can be taken quickly and accurately for recommendation; the data management method has the advantages that manual filling is not needed, data disorder caused by multiple data tables is avoided, accuracy is high, user experience is improved, and data management capability is improved.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A DAG source tracing sampling method based on mapping is characterized by comprising the following steps:

2. The mapping-based DAG traceback sampling method of claim 1, wherein the parent node querying step further comprises:

3. The map-based DAG traceback sampling method of claim 2, wherein the map generating step comprises:

4. The DAG traceable sampling method based on mapping of claim 3, wherein in the mapping map, key is field id and value is "table type # field id".

5. The mapping-based DAG traceback sampling method of claim 1, further comprising:

6. A mapping-based DAG traceback sampling system, comprising:

a data return unit: returning the obtained sample data and the mapping map;

7. The mapping-based DAG traceback sampling system of claim 6, wherein the parent node query unit further comprises:

8. The mapping-based DAG traceback sampling system of claim 7, wherein the mapping map generation unit comprises:

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the mapping-based DAG traceback sampling method of any of claims 1 to 5.

10. A computer readable storage medium having stored thereon a computer program, characterized in that the program, when executed by a processor, implements the mapping-based DAG traceback sampling method of any of claims 1 to 5.