CN117828149A - Management method, system, equipment and storage medium of data catalogue - Google Patents

Management method, system, equipment and storage medium of data catalogue Download PDF

Info

Publication number
CN117828149A
CN117828149A CN202311871306.XA CN202311871306A CN117828149A CN 117828149 A CN117828149 A CN 117828149A CN 202311871306 A CN202311871306 A CN 202311871306A CN 117828149 A CN117828149 A CN 117828149A
Authority
CN
China
Prior art keywords
attribute information
model
meta
data
metadata
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311871306.XA
Other languages
Chinese (zh)
Inventor
缪海旭
李潇
庞少勇
杜宇航
朱敬茹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Travelsky Technology Co Ltd
Original Assignee
China Travelsky Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Travelsky Technology Co Ltd filed Critical China Travelsky Technology Co Ltd
Priority to CN202311871306.XA priority Critical patent/CN117828149A/en
Publication of CN117828149A publication Critical patent/CN117828149A/en
Pending legal-status Critical Current

Links

Abstract

The embodiment of the invention provides a method, a system, equipment and a storage medium for managing a data directory, wherein the method comprises the following steps: responding to a data catalog generation instruction, obtaining an attribute information group of at least one metadata of at least one meta-model, wherein the attribute information group comprises a plurality of attribute information of the metadata and hierarchical labels of all the attribute information, the attribute information type comprises an input type and an acquisition type, the attribute information of the acquisition type is database storage information and/or non-database storage information acquired by a preset acquisition adapter based on the attribute information of the input type, and the attribute information of the acquisition type is used for all the meta-models: and merging the attribute information of the metadata of which the meta-model has the same hierarchical labels to obtain a plurality of attribute information sets, and generating a data catalog based on each meta-model, each attribute information set of each meta-model and the hierarchical labels of each attribute information set. The invention improves the management precision of the data catalogue.

Description

Management method, system, equipment and storage medium of data catalogue
Technical Field
The present invention relates to the field of data management technologies, and in particular, to a method, a system, an apparatus, and a storage medium for managing a data directory.
Background
A data directory is a centralized database that stores and manages enterprise data assets. The existing way to generate the data directory is generally to use the database list directly as the data directory.
However, the data assets of the enterprise include, in addition to data stored in the database, data that is not stored in the database, such as interface data transfer records, file transfer records, report transfer records, and the like. This results in a reduction in the accuracy of managing the data directory when a need to manage the data directory (e.g., update or create the data directory) arises because the database list does not have the data that is not stored in the database.
Disclosure of Invention
The embodiment of the invention aims to provide a method, a system, equipment and a storage medium for managing a data directory, so as to improve the management precision of the data directory. The specific technical scheme is as follows:
a method of managing a data directory, the method comprising:
responding to a data catalog generation instruction, obtaining an attribute information group of at least one metadata of at least one meta-model, wherein the attribute information group comprises a plurality of attribute information of the metadata and hierarchical labels of the attribute information, the attribute information type comprises an input type and an acquisition type, and the attribute information of the acquisition type is database storage information and/or non-database storage information acquired by a preset acquisition adapter based on the attribute information of the input type;
For each of the metamodels: combining the attribute information of the metadata of the meta model with the same hierarchical label to obtain a plurality of attribute information sets;
generating a data catalog based on each of the metamodels, each of the sets of attribute information for each of the metamodels, and the hierarchical labels for each of the sets of attribute information.
Optionally, the method further comprises:
obtaining the attribute information set of the updated metadata of a target meta-model in response to a data catalog update instruction, wherein the target meta-model is any one or more of the meta-models;
based on each updating hierarchical label in the updating metadata, correspondingly updating the hierarchical label of each history metadata of the target meta-model;
merging the attribute information of the metadata of the target meta-model with the same hierarchical label to obtain a plurality of attribute information sets;
generating a data catalog based on each of the metamodels, each of the sets of attribute information for each of the metamodels, and the hierarchical labels for each of the sets of attribute information.
Optionally, the obtaining, in response to a data directory update instruction, the attribute information set of the update metadata of the target meta-model includes:
Responding to the data catalog updating instruction, and calling a target meta-model attribute information group template corresponding to a target meta-model identifier according to the target meta-model identifier in the data catalog updating instruction, wherein the target meta-model attribute information group template comprises all attribute information types of meta-data of the target meta-model and preset level labels of all the attribute information types;
the target meta-model attribute information group template is sent to a preset front end, and the target meta-model attribute information group template fed back by the preset front end and filled with data is obtained;
and carrying out data extraction on the target meta-model attribute information group template filled with the data to obtain the attribute information group of the updated meta-data of the target meta-model.
Optionally, the obtaining, in response to a data directory update instruction, the attribute information set of the update metadata of the target meta-model includes:
responding to the data catalog updating instruction, generating an updating metadata acquisition instruction according to acquisition instruction configuration information and a storage address identifier in the data catalog updating instruction, and sending the updating metadata acquisition instruction to the preset acquisition adapter corresponding to the storage address identifier so that the preset acquisition adapter acquires initial information of the updating metadata from an acquisition log according to the updating metadata acquisition instruction;
According to the target meta-model identification in the data catalog updating instruction, a target meta-model attribute information group template corresponding to the target meta-model identification is called, wherein the target meta-model attribute information group template comprises attribute information types of meta-data of the target meta-model and preset level labels of all the attribute information types;
based on the attribute information type, carrying out data extraction on the initial information, and importing an extraction result into the target meta-model attribute information group template;
and carrying out data extraction on the imported attribute information group template of the target meta-model to obtain the attribute information group of the updated meta-data of the target meta-model.
Optionally, the preset collection adaptor stores information and/or non-stored information based on a database collected by the attribute information of the input type, including:
searching a storage address identifier corresponding to the input type attribute information from a preset storage address list, wherein the preset storage address list stores the storage address identifier of the metadata and at least one input type attribute information of the metadata;
generating a target information acquisition instruction based on the input type attribute information and the storage address identifier corresponding to the input type attribute information, and sending the target information acquisition instruction to the preset acquisition adapter corresponding to the storage address identifier, so that the preset acquisition adapter acquires at least one acquisition type attribute information in the same attribute information group as the input type attribute information.
A system for managing a data directory, the system comprising:
the data acquisition unit is used for responding to a data catalog generation instruction, acquiring an attribute information group of at least one metadata of at least one meta-model, wherein the attribute information group comprises a plurality of attribute information of the metadata and hierarchical labels of the attribute information, the attribute information type comprises an input type and an acquisition type, and the attribute information of the acquisition type is database storage information and/or non-database storage information acquired by a preset acquisition adapter based on the attribute information of the input type;
a data merging unit, configured to, for each of the metamodels: combining the attribute information of the metadata of the meta model with the same hierarchical label to obtain a plurality of attribute information sets;
and a catalog generating unit configured to generate a data catalog based on each of the metamodels, each of the attribute information sets of each of the metamodels, and the hierarchical labels of each of the attribute information sets.
Optionally, the system further comprises:
a catalog updating unit configured to obtain, in response to a data catalog updating instruction, the attribute information set of updated metadata of a target meta model, the target meta model being any one or more of the meta models; based on each updating hierarchical label in the updating metadata, correspondingly updating the hierarchical label of each history metadata of the target meta-model; merging the attribute information of the metadata of the target meta-model with the same hierarchical label to obtain a plurality of attribute information sets; generating a data catalog based on each of the metamodels, each of the sets of attribute information for each of the metamodels, and the hierarchical labels for each of the sets of attribute information.
Optionally, the catalog update unit is configured to, in response to a data catalog update instruction, obtain the attribute information set of update metadata of a target meta model:
responding to the data catalog updating instruction, and calling a target meta-model attribute information group template corresponding to a target meta-model identifier according to the target meta-model identifier in the data catalog updating instruction, wherein the target meta-model attribute information group template comprises all attribute information types of meta-data of the target meta-model and preset level labels of all the attribute information types;
the target meta-model attribute information group template is sent to a preset front end, and the target meta-model attribute information group template fed back by the preset front end and filled with data is obtained;
and carrying out data extraction on the target meta-model attribute information group template filled with the data to obtain the attribute information group of the updated meta-data of the target meta-model.
Optionally, the directory updating unit obtains the attribute information set of the update metadata of the target meta-model in response to a data directory update instruction, including:
responding to the data catalog updating instruction, generating an updating metadata acquisition instruction according to acquisition instruction configuration information and a storage address identifier in the data catalog updating instruction, and sending the updating metadata acquisition instruction to the preset acquisition adapter corresponding to the storage address identifier so that the preset acquisition adapter acquires initial information of the updating metadata from an acquisition log according to the updating metadata acquisition instruction;
According to the target meta-model identification in the data catalog updating instruction, a target meta-model attribute information group template corresponding to the target meta-model identification is called, wherein the target meta-model attribute information group template comprises attribute information types of meta-data of the target meta-model and preset level labels of all the attribute information types;
based on the attribute information type, carrying out data extraction on the initial information, and importing an extraction result into the target meta-model attribute information group template;
and carrying out data extraction on the imported attribute information group template of the target meta-model to obtain the attribute information group of the updated meta-data of the target meta-model.
Optionally, the preset collection adaptor is configured to, when the database stores information and/or the non-database stores information based on the attribute information collection of the input type:
searching a storage address identifier corresponding to the input type attribute information from a preset storage address list, wherein the preset storage address list stores the storage address identifier of the metadata and at least one input type attribute information of the metadata;
generating a target information acquisition instruction based on the input type attribute information and the storage address identifier corresponding to the input type attribute information, and sending the target information acquisition instruction to the preset acquisition adapter corresponding to the storage address identifier, so that the preset acquisition adapter acquires at least one acquisition type attribute information in the same attribute information group as the input type attribute information.
A management apparatus for a data directory, the apparatus comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement a method of managing a data directory as described in any one of the above.
A computer readable storage medium, which when executed by a processor of a management device of a data directory, causes the management device of the data directory to perform a method of managing a data directory as claimed in any one of the above.
According to the data catalog management method, system, device and storage medium provided by the embodiment of the invention, the data storage information and the non-database storage information can be acquired through the preset acquisition adapter, so that compared with the mode of carrying out data catalog management only based on a database list in the prior art, the data catalog management method, system, device and storage medium provided by the embodiment of the invention avoid omission of the non-database storage information and improve the management precision of the data catalog. And the attribute information of the metadata with the same level label is combined through configuration, and the data catalog is generated based on each meta-model, each attribute information set of each meta-model and the level label of each attribute information set. It can be seen that the present invention provides management accuracy for data directories.
Of course, it is not necessary for any one product or method of practicing the invention to achieve all of the advantages set forth above at the same time.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
FIG. 1 is a flowchart of a method for managing a data directory according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a target system data directory according to an alternative embodiment of the present invention;
FIG. 3 is a block diagram of a management system for a data directory according to an embodiment of the present invention;
fig. 4 is a block diagram of a management device for a data directory according to an embodiment of the present invention.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.
It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units. [ ordinal words ]
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise. [ Single plural ]
The embodiment of the invention provides a method for managing a data directory, as shown in fig. 1, the method for managing the data directory comprises the following steps:
s101, responding to a data catalog generation instruction, obtaining an attribute information group of at least one metadata of at least one meta model, wherein the attribute information group comprises a plurality of attribute information of the metadata and hierarchical labels of all attribute information, the attribute information type comprises an input type and an acquisition type, and the attribute information of the acquisition type is database storage information and/or non-database storage information acquired by a preset acquisition adapter based on the attribute information of the input type.
Alternatively, in an alternative embodiment of the present invention, the data directory generation instruction may be an instruction generated by the front end for generating a current system data directory.
In the practical application scenario, the Meta-model (Meta-model) is a model for defining attributes and relationships of a certain type of metadata. Since the meta model defines the attribute of the meta data, attribute information of all the meta data defined by the meta model can be obtained and managed based on the meta model.
Optionally, in an alternative embodiment of the present invention, the types of metamodels described above include, but are not limited to: file class, interface class, message class, etc.
It should be noted that, in a practical application scenario, the Metadata (Metadata) is data for describing data, and the Metadata may represent a structural relationship between all data associated with a certain type of service. May be used to identify, describe and track data.
In the actual application scenario, the attribute information is an attribute description related to defining the metadata according to the actual service scenario. For example, the file class metadata may include attribute information such as file name, file size, storage address, and the like.
In the actual application scenario, the input type of attribute information may be metadata attribute information input in a front page through a metadata operation manner when an attribute information acquisition requirement for metadata is generated. The specific implementation process can be as follows: and displaying the meta model corresponding to the meta model identification in the data catalog generation instruction on a front page so that an operator fills in the attribute information of the meta data according to the attribute information defined by the meta model.
It should be noted that, in an actual application scenario, the preset collection adapter may be a data collection device configured in each data source in a development and production environment of the system. According to the invention, the data storage information and the non-database storage information are collected by configuring the preset collection adapter, so that compared with the mode of carrying out data catalog management only based on a database list in the prior art, the method and the device have the advantages that omission of the non-database storage information is avoided, and the management precision of the data catalog is improved.
It should be noted that, in an actual application scenario, the types of the preset collection adapter may be various, for example, an Oracle collection adapter, a greenplus collection adapter, an EDB collection adapter, and the like. The configuration process can be as follows: and acquiring data such as access addresses, names and the like of data sources to be deployed by the preset acquisition adapter, and establishing a corresponding relation between the preset acquisition adapter and the deployed data sources based on the data. When the data acquisition requirement is generated, the preset acquisition adapter can acquire the related data in the data source according to the acquisition frequency in the acquisition instruction.
Optionally, in an optional embodiment of the present invention, the types of the non-database storage information include, but are not limited to: data stored in a business system cache, such as interface data transmission records, file transmission records, report transmission records and the like.
S102, for each meta-model: and merging the attribute information of the metadata of the meta-model with the same level label to obtain a plurality of attribute information sets.
S103, generating a data catalog based on each meta-model, each attribute information set of each meta-model and the hierarchical labels of each attribute information set.
It should be noted that, in the actual application scenario, the above-mentioned hierarchical label is a label for characterizing a hierarchy to which attribute information belongs in the data directory. According to the method, the attribute information of the metadata with the same level label is combined through configuration, and the data catalogue is generated based on each meta-model, each attribute information set of each meta-model and the level label of each attribute information set.
In the practical application scenario, there are various embodiments of step S103 shown in fig. 1, and an exemplary method is provided herein:
A first directory hierarchy of data directories is set as a target system data directory. The presence file class meta-model and the interface class meta-model are set, and the level labels of the two types of meta-models correspond to the second directory level. For a file type meta-model, attribute information of metadata thereof includes: business attributes, technical attributes, and operational attributes. For the interface class meta-model, attribute information of metadata thereof includes: business attributes, security attributes, and management attributes. For convenience of description, it is set that the hierarchical labels of the metadata attribute information corresponding to the above two types of metamodels each correspond to the third directory hierarchy.
Then, when generating the data directory based on each meta-model, each set of attribute information of each meta-model, and the hierarchical labels of each set of attribute information, the target system data directory as shown in fig. 2 may be generated from the affiliations of the hierarchical labels by configuring a recursive algorithm.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
According to the invention, the data storage information and the non-database storage information are acquired through the preset acquisition adapter, so that compared with the mode of carrying out data catalog management only based on a database list in the prior art, the method and the device have the advantages that omission of the non-database storage information is avoided, and the management precision of the data catalog is improved. And the attribute information of the metadata with the same level label is combined through configuration, and the data catalog is generated based on each meta-model, each attribute information set of each meta-model and the level label of each attribute information set. It can be seen that the present invention provides management accuracy for data directories.
Optionally, the method for managing a data directory as shown in fig. 1 further includes:
responding to a data catalog updating instruction, obtaining an attribute information set of updating metadata of a target meta-model, wherein the target meta-model is any one or more of the meta-models;
based on each update level label in the update metadata, the level label of each history metadata of the update target meta-model is correspondingly updated;
Combining attribute information of metadata of the target meta-model with the same level label to obtain a plurality of attribute information sets;
a data catalog is generated based on each meta-model, each set of attribute information for each meta-model, and the hierarchical labels for each set of attribute information.
Optionally, in response to the data directory update instruction, obtaining the attribute information set of the update metadata of the target meta-model includes:
responding to the data catalog updating instruction, and calling a target meta-model attribute information group template corresponding to the target meta-model identifier according to the target meta-model identifier in the data catalog updating instruction, wherein the target meta-model attribute information group template comprises all attribute information types of metadata of the target meta-model and preset level labels of all attribute information types;
the method comprises the steps of sending a target meta-model attribute information group template to a preset front end, and obtaining a target meta-model attribute information group template fed back by the preset front end and filled with data;
and carrying out data extraction on the target meta-model attribute information group template filled with the data to obtain an attribute information group of the updated meta-data of the target meta-model.
Alternatively, in an alternative embodiment of the present invention, the data directory update instruction may be an instruction generated based on a metadata registration instruction received by the front end when unregistered metadata occurs.
Optionally, in response to the data directory update instruction, obtaining the attribute information set of the update metadata of the target meta-model includes:
responding to the data catalog update instruction, generating an update metadata acquisition instruction according to acquisition instruction configuration information and a storage address identifier in the data catalog update instruction, and sending the update metadata acquisition instruction to a preset acquisition adapter corresponding to the storage address identifier so that the preset acquisition adapter acquires initial information of update metadata from an acquisition log according to the update metadata acquisition instruction;
according to the target meta-model identification in the data catalog updating instruction, a target meta-model attribute information group template corresponding to the target meta-model identification is called, wherein the target meta-model attribute information group template comprises attribute information types of meta-data of the target meta-model and preset level labels of all attribute information types;
based on the attribute information type, extracting data from the initial information, and importing the extraction result into a target meta-model attribute information group template;
and extracting data from the imported target meta-model attribute information group template to obtain an attribute information group of the updated meta-data of the target meta-model.
In the actual application scene, the initial information of the updated metadata is acquired from the acquisition log according to the updated metadata acquisition instruction by configuring the preset acquisition adapter, so that the problem of reduced updated metadata acquisition efficiency caused by a manual addition mode is avoided, and the generation efficiency of the data catalogue is improved.
In the actual application scenario, the information in the data directory may be displayed in a display manner such as a line graph, a double-circle graph, a bar graph, a pie graph, etc. in order to improve the display efficiency of the data directory due to the huge amount of the meta-model, the meta-data and the attribute information. Meanwhile, the configuration display page only displays the catalog information with the same data label in the data catalog, so that the display effect is improved.
Optionally, the presetting collection adapter collects database storage information and/or non-database storage information based on the attribute information of the input type, including:
searching a storage address identifier corresponding to the input type attribute information from a preset storage address list, wherein the storage address identifier of metadata and at least one input type attribute information of the metadata are stored in the preset storage address list;
Generating a target information acquisition instruction based on the input type attribute information and the storage address identifier corresponding to the input type attribute information, and sending the target information acquisition instruction to a preset acquisition adapter corresponding to the storage address identifier, so that the preset acquisition adapter acquires the attribute information of at least one acquisition type in the same attribute information group as the input type attribute information.
The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information. [ Nap is not limiting ]
Although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Corresponding to the embodiment of the method, the invention also provides a management system of the data directory, as shown in fig. 3, the management system of the data directory comprises:
a data collection unit 301, configured to obtain, in response to a data catalog generation instruction, an attribute information set of at least one metadata of at least one meta model, where the attribute information set includes a plurality of attribute information of the metadata and a hierarchical label of each attribute information, the attribute information type includes an input type and an acquisition type, and the attribute information of the acquisition type is database storage information and/or non-database storage information acquired by a preset acquisition adapter based on the attribute information of the input type;
A data merging unit 302, configured to, for each metamodel: combining attribute information of metadata of the meta model with the same level label to obtain a plurality of attribute information sets;
the catalog generation unit 303 is configured to generate a data catalog based on each meta-model, each attribute information set of each meta-model, and a hierarchical label of each attribute information set.
Optionally, the management system of the data directory shown in fig. 3 further includes:
a catalog updating unit, configured to obtain, in response to a data catalog updating instruction, an attribute information set of updated metadata of a target meta-model, where the target meta-model is any one or more of the meta-models; based on each update level label in the update metadata, the level label of each history metadata of the update target meta-model is correspondingly updated; combining attribute information of metadata of the target meta-model with the same level label to obtain a plurality of attribute information sets; a data catalog is generated based on each meta-model, each set of attribute information for each meta-model, and the hierarchical labels for each set of attribute information.
Optionally, the above-mentioned catalog updating unit is configured to, in response to the data catalog updating instruction, when obtaining the attribute information group of the update metadata of the target meta model:
Responding to the data catalog updating instruction, and calling a target meta-model attribute information group template corresponding to the target meta-model identifier according to the target meta-model identifier in the data catalog updating instruction, wherein the target meta-model attribute information group template comprises all attribute information types of metadata of the target meta-model and preset level labels of all attribute information types;
the method comprises the steps of sending a target meta-model attribute information group template to a preset front end, and obtaining a target meta-model attribute information group template fed back by the preset front end and filled with data;
and carrying out data extraction on the target meta-model attribute information group template filled with the data to obtain an attribute information group of the updated meta-data of the target meta-model.
Optionally, the catalog updating unit obtains the attribute information set of the update metadata of the target meta-model in response to the data catalog updating instruction, including:
responding to the data catalog update instruction, generating an update metadata acquisition instruction according to acquisition instruction configuration information and a storage address identifier in the data catalog update instruction, and sending the update metadata acquisition instruction to a preset acquisition adapter corresponding to the storage address identifier so that the preset acquisition adapter acquires initial information of update metadata from an acquisition log according to the update metadata acquisition instruction;
According to the target meta-model identification in the data catalog updating instruction, a target meta-model attribute information group template corresponding to the target meta-model identification is called, wherein the target meta-model attribute information group template comprises attribute information types of meta-data of the target meta-model and preset level labels of all attribute information types;
based on the attribute information type, extracting data from the initial information, and importing the extraction result into a target meta-model attribute information group template;
and extracting data from the imported target meta-model attribute information group template to obtain an attribute information group of the updated meta-data of the target meta-model.
Optionally, the preset collection adaptor is configured to, when the database stores information and/or the non-database stores information, collect attribute information based on the input type:
searching a storage address identifier corresponding to the input type attribute information from a preset storage address list, wherein the storage address identifier of metadata and at least one input type attribute information of the metadata are stored in the preset storage address list;
generating a target information acquisition instruction based on the input type attribute information and the storage address identifier corresponding to the input type attribute information, and sending the target information acquisition instruction to a preset acquisition adapter corresponding to the storage address identifier, so that the preset acquisition adapter acquires the attribute information of at least one acquisition type in the same attribute information group as the input type attribute information.
The embodiment of the invention also provides a management device of the data catalogue, as shown in fig. 4, the management device of the data catalogue comprises:
a processor 401;
a memory 402 for storing instructions executable by the processor 401;
wherein the processor 401 is configured to execute instructions to implement a method of managing a data directory as any one of the above.
The embodiment of the present invention also provides a computer-readable storage medium, which when executed by a processor of a management apparatus of a data directory, enables the management apparatus of the data directory to perform a management method of the data directory as any one of the above.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims.
While several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).

Claims (10)

1. A method for managing a data directory, the method comprising:
responding to a data catalog generation instruction, obtaining an attribute information group of at least one metadata of at least one meta-model, wherein the attribute information group comprises a plurality of attribute information of the metadata and hierarchical labels of the attribute information, the attribute information type comprises an input type and an acquisition type, and the attribute information of the acquisition type is database storage information and/or non-database storage information acquired by a preset acquisition adapter based on the attribute information of the input type;
For each of the metamodels: combining the attribute information of the metadata of the meta model with the same hierarchical label to obtain a plurality of attribute information sets;
generating a data catalog based on each of the metamodels, each of the sets of attribute information for each of the metamodels, and the hierarchical labels for each of the sets of attribute information.
2. The method according to claim 1, wherein the method further comprises:
obtaining the attribute information set of the updated metadata of a target meta-model in response to a data catalog update instruction, wherein the target meta-model is any one or more of the meta-models;
based on each updating hierarchical label in the updating metadata, correspondingly updating the hierarchical label of each history metadata of the target meta-model;
merging the attribute information of the metadata of the target meta-model with the same hierarchical label to obtain a plurality of attribute information sets;
generating a data catalog based on each of the metamodels, each of the sets of attribute information for each of the metamodels, and the hierarchical labels for each of the sets of attribute information.
3. The method of claim 2, wherein said obtaining said set of attribute information for update metadata for a target meta-model in response to a data directory update instruction comprises:
Responding to the data catalog updating instruction, and calling a target meta-model attribute information group template corresponding to a target meta-model identifier according to the target meta-model identifier in the data catalog updating instruction, wherein the target meta-model attribute information group template comprises all attribute information types of meta-data of the target meta-model and preset level labels of all the attribute information types;
the target meta-model attribute information group template is sent to a preset front end, and the target meta-model attribute information group template fed back by the preset front end and filled with data is obtained;
and carrying out data extraction on the target meta-model attribute information group template filled with the data to obtain the attribute information group of the updated meta-data of the target meta-model.
4. The method of claim 2, wherein said obtaining said set of attribute information for update metadata for a target meta-model in response to a data directory update instruction comprises:
responding to the data catalog updating instruction, generating an updating metadata acquisition instruction according to acquisition instruction configuration information and a storage address identifier in the data catalog updating instruction, and sending the updating metadata acquisition instruction to the preset acquisition adapter corresponding to the storage address identifier so that the preset acquisition adapter acquires initial information of the updating metadata from an acquisition log according to the updating metadata acquisition instruction;
According to the target meta-model identification in the data catalog updating instruction, a target meta-model attribute information group template corresponding to the target meta-model identification is called, wherein the target meta-model attribute information group template comprises attribute information types of meta-data of the target meta-model and preset level labels of all the attribute information types;
based on the attribute information type, carrying out data extraction on the initial information, and importing an extraction result into the target meta-model attribute information group template;
and carrying out data extraction on the imported attribute information group template of the target meta-model to obtain the attribute information group of the updated meta-data of the target meta-model.
5. The method according to claim 1, wherein the preset collection adaptor collects database storage information and/or non-database storage information based on the attribute information of the input type, comprising:
searching a storage address identifier corresponding to the input type attribute information from a preset storage address list, wherein the preset storage address list stores the storage address identifier of the metadata and at least one input type attribute information of the metadata;
Generating a target information acquisition instruction based on the input type attribute information and the storage address identifier corresponding to the input type attribute information, and sending the target information acquisition instruction to the preset acquisition adapter corresponding to the storage address identifier, so that the preset acquisition adapter acquires at least one acquisition type attribute information in the same attribute information group as the input type attribute information.
6. A system for managing a data directory, the system comprising:
the data acquisition unit is used for responding to a data catalog generation instruction, acquiring an attribute information group of at least one metadata of at least one meta-model, wherein the attribute information group comprises a plurality of attribute information of the metadata and hierarchical labels of the attribute information, the attribute information type comprises an input type and an acquisition type, and the attribute information of the acquisition type is database storage information and/or non-database storage information acquired by a preset acquisition adapter based on the attribute information of the input type;
a data merging unit, configured to, for each of the metamodels: combining the attribute information of the metadata of the meta model with the same hierarchical label to obtain a plurality of attribute information sets;
And a catalog generating unit configured to generate a data catalog based on each of the metamodels, each of the attribute information sets of each of the metamodels, and the hierarchical labels of each of the attribute information sets.
7. The system of claim 6, wherein the system further comprises:
a catalog updating unit configured to obtain, in response to a data catalog updating instruction, the attribute information set of updated metadata of a target meta model, the target meta model being any one or more of the meta models; based on each updating hierarchical label in the updating metadata, correspondingly updating the hierarchical label of each history metadata of the target meta-model; merging the attribute information of the metadata of the target meta-model with the same hierarchical label to obtain a plurality of attribute information sets; generating a data catalog based on each of the metamodels, each of the sets of attribute information for each of the metamodels, and the hierarchical labels for each of the sets of attribute information.
8. The system according to claim 7, wherein the catalog update unit, in response to a data catalog update instruction, is configured to, when obtaining the attribute information set of update metadata of a target metamodel:
Responding to the data catalog updating instruction, and calling a target meta-model attribute information group template corresponding to a target meta-model identifier according to the target meta-model identifier in the data catalog updating instruction, wherein the target meta-model attribute information group template comprises all attribute information types of meta-data of the target meta-model and preset level labels of all the attribute information types;
the target meta-model attribute information group template is sent to a preset front end, and the target meta-model attribute information group template fed back by the preset front end and filled with data is obtained;
and carrying out data extraction on the target meta-model attribute information group template filled with the data to obtain the attribute information group of the updated meta-data of the target meta-model.
9. A management device for a data directory, the device comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the method of managing a data directory as claimed in any one of claims 1 to 5.
10. A computer-readable storage medium, characterized in that instructions in the computer-readable storage medium, when executed by a processor of a management device of a data directory, enable the management device of a data directory to perform the management method of a data directory as claimed in any one of claims 1 to 5.
CN202311871306.XA 2023-12-29 2023-12-29 Management method, system, equipment and storage medium of data catalogue Pending CN117828149A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311871306.XA CN117828149A (en) 2023-12-29 2023-12-29 Management method, system, equipment and storage medium of data catalogue

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311871306.XA CN117828149A (en) 2023-12-29 2023-12-29 Management method, system, equipment and storage medium of data catalogue

Publications (1)

Publication Number Publication Date
CN117828149A true CN117828149A (en) 2024-04-05

Family

ID=90518747

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311871306.XA Pending CN117828149A (en) 2023-12-29 2023-12-29 Management method, system, equipment and storage medium of data catalogue

Country Status (1)

Country Link
CN (1) CN117828149A (en)

Similar Documents

Publication Publication Date Title
CN109660397B (en) System, method and device for collecting logs
CN109413127B (en) Data synchronization method and device
CN109446274B (en) Method and device for managing BI metadata of big data platform
CN112685433B (en) Metadata updating method and device, electronic equipment and computer-readable storage medium
US10614087B2 (en) Data analytics on distributed databases
US11153071B2 (en) Citation and attribution management methods and systems
US20200286014A1 (en) Information updating method and device
CN112416923A (en) Metadata management method and device, equipment and storage medium
CN113886485A (en) Data processing method, device, electronic equipment, system and storage medium
CN111699484A (en) System and method for data management
CN115858488A (en) Parallel migration method and device based on data governance and readable medium
US20170193409A1 (en) Business resource modeling
CN111090803A (en) Data processing method and device, electronic equipment and storage medium
CN112559495A (en) System for supporting multi-system label mapping to realize unified label management
CN117828149A (en) Management method, system, equipment and storage medium of data catalogue
CN111104450A (en) Target data importing method, medium, device and computing equipment
US10664501B2 (en) Deriving and interpreting users collective data asset use across analytic software systems
CN113868253A (en) Data relationship capturing and big data relationship tree construction method
CN112860292A (en) Configuration management method and device based on application
CN111198805B (en) Abnormality monitoring method and device
US11726979B2 (en) Determining a chronological order of transactions executed in relation to an object stored in a storage system
CN113779017A (en) Method and apparatus for data asset management
CN111126961A (en) Complex product full life cycle digital mainline service system
CN106408199B (en) Information processing method and device
CN109710673B (en) Work processing method, device, equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination