CN112395340B - Data asset management method and device - Google Patents
Data asset management method and device Download PDFInfo
- Publication number
- CN112395340B CN112395340B CN202011279767.4A CN202011279767A CN112395340B CN 112395340 B CN112395340 B CN 112395340B CN 202011279767 A CN202011279767 A CN 202011279767A CN 112395340 B CN112395340 B CN 112395340B
- Authority
- CN
- China
- Prior art keywords
- metadata
- data
- database
- asset management
- management system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data asset management method and a data asset management device, which are suitable for a data asset management system for managing at least one database, wherein the data asset management system comprises a unified interface and at least one plug-in, the database corresponds to the plug-in one by one, and the at least one plug-in is respectively connected with the corresponding database through the unified interface; the data asset management system receives the retrieval information; the data asset management system determines first metadata and first connection information from the data resource pool according to the retrieval information; the data asset management system calls a first plug-in corresponding to the first database according to the first connection information; the data asset management system calls a unified interface through a first plug-in unit and establishes connection with a first database according to first connection information; the data asset management system obtains first data in a first database. The method realizes multi-source data access and management and simplifies the multi-source data management flow.
Description
Technical Field
The present disclosure relates to the field of network technologies, and in particular, to a method and an apparatus for managing data assets.
Background
With the rapid growth of the internet, the volume of data has been increasing, and more people have come to appreciate the importance of data, so that a data center for providing a data global service has been developed. The core of the data center is the flow mechanism of the data application and sustainable data operation.
The flow mechanism of the data application refers to a pipeline for constructing enterprise data assets, the value of the data assets is mined, and a whole set of data use mechanism from convergence, development, management and asset service is formed, and the core content of the flow mechanism is a data asset system. Continuous data operation refers to strengthening enterprise data culture construction by continuously operating data assets. That is, the nature of the data center is service sharing, the goal is to support rapid innovation or trial and error of the application, and the means of implementation is data operation through micro-service architecture, agile infrastructure and public basic services. For example, in data development, the change of the core data model of an enterprise is relatively slow, and the workload of maintaining data is also very large; but the speed of business application innovation of enterprises and the change of requirements on data are very rapid. Therefore, through the data operation of the data center, required core data can be provided for application development, and the problem that response force cannot keep pace between data development and application development due to mismatching of development speed is solved. The data center station needs to be in butt joint with different databases, so that in order to obtain data assets in the different databases, various acquisition modes of the databases need to be formulated, the data acquisition work is complex, and data corresponding to different data sources cannot be well managed.
Therefore, there is a need for a method and apparatus for managing data assets, which can implement multi-source data access and management, and simplify multi-source data management process.
Disclosure of Invention
The embodiment of the invention provides a data asset management method and device, which realize multi-source data access and management and simplify multi-source data management flow.
In a first aspect, an embodiment of the present invention provides a data asset management method, which is applicable to a data asset management system for managing at least one database, where the data asset management system includes a unified interface and at least one plugin, the database corresponds to the plugin one by one, and the at least one plugin is connected to the corresponding database through the unified interface respectively; the method comprises the following steps: the data asset management system receiving retrieval information; the data asset management system determines first metadata and first connection information from a data resource pool according to the retrieval information; metadata of each database and connection information of each database are stored in the data resource pool; the first connection information is connection information of a first database, and the first metadata is description information of first data in the first database; the data asset management system calls a first plug-in corresponding to the first database according to the first connection information; the data asset management system calls the unified interface through the first plug-in and establishes connection with the first database according to the first connection information; the data asset management system obtains the first data in the first database.
In the above method, the data asset management system manages and/or connects at least one database through the unified interface and at least one plug-in. When the data asset management system receives the search information, corresponding first metadata ID is determined according to the search information, and corresponding first metadata and first connection information are obtained in the data resource pool according to the first metadata ID. And then, determining a first plug-in corresponding to the first database according to the first connection information, and establishing connection with the first database through the first connection information and the unified interface to acquire specific data of the first metadata. The method has the advantages that the user side can acquire metadata information and specific data of a plurality of databases, multi-source access and management of the databases are realized, the multi-source data management flow is simplified, and network resources are saved.
Optionally, the method further comprises: the data asset management system receives an asset view creation instruction; the data asset management system determines an asset directory corresponding to the asset view creation instruction, and loads metadata information acquired from the data resource pool to the corresponding asset directory, so that creation of the asset view is completed, and the metadata information comprises unique identifiers of metadata.
In the method, the data asset management system may generate asset directories of the databases (may generate corresponding asset directories for the databases respectively, or may generate total asset directories for the databases) according to metadata information of the databases or a program preset in the data asset management system, and after receiving an asset view creation instruction, the data asset management system loads metadata information acquired from the data asset pool into the corresponding asset directories. Thus, the user can intuitively acquire the information such as the data type in each database.
Optionally, the method further comprises: the data asset management system detects metadata in databases; and after the data asset management system determines that the first metadata of the first database is changed, updating the first metadata recorded in the data resource pool.
In the method, after detecting that the first metadata of the first database is changed, the data asset management system acquires related information of the changed first metadata and updates the first metadata recorded in the data resource pool. And ensuring the accuracy of the first metadata.
Optionally, after the data asset management system updates the first metadata recorded in the data resource pool, the method further includes: the data asset management system updates a metadata state of the first metadata in the data asset pool, the metadata state comprising: newly added metadata, valid metadata and invalid metadata; and the data asset management system updates the asset view where the first metadata is located according to the metadata state of the first metadata.
In the above method, after the data asset management system updates the first metadata, the metadata state of the first metadata in the data asset pool and the asset view is also updated. The user can acquire the current state of the first metadata, and the availability of the first metadata is improved.
Optionally, the method further comprises: after the data asset management system determines that the first metadata of the first database are changed, the first metadata recorded in the association system is updated through the message middleware, and the association system contains the first metadata.
In the method, the data asset management system sends the change result of the first metadata to the association system through the message middleware. So that the associated system synchronously updates the first metadata, and the accuracy of the first metadata is ensured.
Optionally, the method further comprises: the data asset management system receives a second plug-in of the newly added database and performs validity check on the second plug-in; after the verification is passed, the data asset management system determines second connection information and is connected with the newly-added database through the second plug-in and the second connection information; the data asset management system acquires second metadata of the newly-added database, and stores the second metadata and the second connection information into the data resource pool, wherein the second metadata are description information of the second data in the newly-added database.
In the above method, the data asset management system may receive the second plugin and run the second plugin after verification passes. Thus, the newly added database corresponding to the second plug-in is accessed. Increased flexibility in data sources (databases) in a data asset management system is achieved. And storing the second metadata and the second connection information into a data resource pool. The user can intuitively acquire the information such as the data type in the newly-added database after the metadata information of the newly-added database is loaded to the corresponding asset directory in the asset view.
Optionally, the method further comprises: and the data asset management system performs all statistical tasks according to the set metadata statistical tasks.
In the method, the data asset management system executes each statistical task according to the set metadata statistical task so as to improve the accuracy of the data volume statistical information acquired by the user side.
In a second aspect, an embodiment of the present invention provides a data asset management device, which is applicable to a data asset management system for managing at least one database, where the data asset management system includes a unified interface and at least one plugin, the database corresponds to the plugin one by one, and the at least one plugin is connected to the corresponding database through the unified interface respectively; the device comprises:
the receiving and transmitting module is used for receiving the retrieval information;
the determining module is used for determining first metadata and first connection information from the data resource pool according to the retrieval information; metadata of each database and connection information of each database are stored in the data resource pool; the first connection information is connection information of a first database, and the first metadata is description information of first data in the first database;
the processing module is used for calling a first plug-in corresponding to the first database according to the first connection information;
the processing module is further used for calling the unified interface through the first plug-in and establishing connection with the first database according to the first connection information;
the processing module is further configured to obtain the first data in the first database.
In a third aspect, embodiments of the present application further provide a computing device, including: a memory for storing a program; a processor for calling a program stored in said memory, and executing the method as described in the various possible designs of the first aspect according to the obtained program.
In a fourth aspect, embodiments of the present application also provide a computer-readable non-volatile storage medium, including a computer-readable program, which when read and executed by a computer, causes the computer to perform the method as described in the various possible designs of the first aspect.
These and other implementations of the present application will be more readily understood in the following description of the embodiments.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it will be apparent that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data asset management architecture according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for managing data assets according to an embodiment of the present invention;
FIG. 3 is a flow chart of a method for managing data assets according to an embodiment of the present invention;
FIG. 4 is a flow chart of a method for managing data assets according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a data asset management device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail below with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
FIG. 1 is a system architecture for data asset management according to an embodiment of the present invention; the at least one database 101 may include a plurality of databases, such as a relational database (e.g., oracle, mySQL, postgreSQL, etc.), a big data related database (e.g., hbase, hive, kafka, etc.), a graph database (e.g., neo 4J), and an index database (e.g., elastic search), etc. At least one plug-in 105, such as a relational database plug-in, big data relational database plug-in, graph database plug-in, index database plug-in, may be included in the corresponding data asset management system 103; and each plug-in is respectively packaged with a data source configuration method of a corresponding database, and the like. For example, a relational database plug-in encapsulates a data source configuration method of a relational database, that is, the database and the plug-in are in a one-to-one correspondence. Assuming that the big data related database 102 is the first database 102, the big data related database plug-in 106 is the first plug-in 106; when the data asset management system 103 receives the search information, the data asset management system 103 obtains the first metadata and the first connection information in the data resource pool 107 according to the search information, the data asset management system 103 determines the first database 102 according to the first connection information, and the data asset management system 103 obtains the first data in the first database 102 by calling the unified interface 104 through the first plugin 106 to establish connection with the first database 102. The detection module 108 periodically acquires connection information of each database in the at least one database 101 from the data resource pool 107, acquires each metadata in each database according to the connection information of each database, acquires metadata change information, and updates corresponding metadata in the data resource pool 107 according to the metadata change information; the change information of the metadata is also sent to the association system 110, so that the association system 110 updates the local metadata according to the change information of the metadata. The statistics module 109 performs each statistics task according to the set metadata statistics task, that is, periodically obtains connection information of each database in the at least one database 101 from the data resource pool 107, obtains data amount information corresponding to each statistics task in each database according to the connection information of each database, and records the data amount information.
Based on this, the embodiment of the application provides a flow of a data asset management method, as shown in fig. 2, including:
step 201, the data asset management system receives retrieval information;
here, the retrieval information may include a first metadata ID of the retrieved first metadata, a related parameter and a sentence for acquiring the first data, and the like.
Step 202, the data asset management system determines first metadata and first connection information from a data resource pool according to the retrieval information; metadata of each database and connection information of each database are stored in the data resource pool; the first connection information is connection information of a first database, and the first metadata is description information of first data in the first database;
here, the first connection information may include information such as a port number, IP, etc. of the first database.
Step 203, the data asset management system calls a first plugin corresponding to the first database according to the first connection information;
here, the first plugin encapsulates the configuration method of the first database, for example, if the first database is an HIVE database, by encapsulating a unified SQL editing box in the plugin, the user may manually write an SQL script to implement data query.
Step 204, the data asset management system calls the unified interface through the first plug-in unit, and establishes connection with the first database according to the first connection information;
step 205, the data asset management system obtains the first data in the first database.
In the above method, the data asset management system manages and/or connects at least one database through the unified interface and at least one plug-in. When the data asset management system receives the search information, corresponding first metadata ID is determined according to the search information, and corresponding first metadata and first connection information are obtained in the data resource pool according to the first metadata ID. And then, determining a first plug-in corresponding to the first database according to the first connection information, and establishing connection with the first database through the first connection information and the unified interface to acquire specific data of the first metadata. The method has the advantages that the user side can acquire metadata information and specific data of a plurality of databases, multi-source access and management of the databases are realized, the multi-source data management flow is simplified, and network resources are saved.
The embodiment of the application also provides an asset view creation method, which comprises the following steps: the data asset management system receives an asset view creation instruction; the data asset management system determines an asset directory corresponding to the asset view creation instruction, and loads metadata information acquired from the data resource pool to the corresponding asset directory, so that creation of the asset view is completed, and the metadata information comprises unique identifiers of metadata. That is, an asset directory is set in advance in the data asset management system; or a program preset by the data asset management system, and generating an asset catalog according to the information such as the data type, the data property, the business corresponding to the data and the like in the metadata. After receiving the asset view creation instruction, the data asset management system loads metadata information such as the state, unique identification and the like of each metadata in the data resource pool to the corresponding asset directory, and completes creation of the asset view. Thus, the user can acquire information such as data types, data properties and the like contained in each database through the asset view, and acquire data usable in the databases. Wherein the asset directory may be presented in the form of a directory tree, and the metadata in the data resource pool may include: metadata of a database that can be connected to the data asset management system through the unified interface, and metadata of a database that cannot be connected to the data asset management system through the unified interface. The metadata of the database which cannot establish connection can be obtained through uploading, inputting and other modes. Accordingly, the asset view may include two views, one being the asset view of the database that can establish a connection and the other being the asset view of the database that cannot establish a connection. The two parts of the asset view can be characterized by view colors and the like, so that a user can know that metadata can acquire real specific data from a database directly through a data asset management system, and that the metadata cannot acquire real specific data from the database directly through the data asset management system.
Based on the above-mentioned flow, the embodiment of the present application provides a flow of a data asset management method, as shown in fig. 3, including:
step 301, the data asset management system receives an asset view creation instruction.
Step 302, the data asset management system determines an asset directory corresponding to the asset view creation instruction, and loads metadata information obtained from the data resource pool to the corresponding asset directory to complete creation of the asset view.
Step 303, the data asset management system receives the instruction for displaying the asset view and displays the asset view.
Step 304, the data asset management system receives the first metadata checking instruction, and obtains the first metadata from the data resource pool according to the unique identifier and other information of the first metadata in the asset view.
Step 305, the data asset management system receives the retrieval information.
Step 306, the data asset management system determines the first metadata and the first connection information from the data asset pool based on the retrieval information.
Step 307, the data asset management system determines a first database according to the first connection information, and further invokes a first plug-in corresponding to the first database.
Step 308, the data asset management system invokes the unified interface through the first plug-in, and establishes a connection with the first database according to the first connection information.
Step 309, the data asset management system obtains the first data in the first database.
Here, the above-described flow steps are not exclusive, and the data asset management system may generate the search information through steps 303 and 304, or may directly acquire the search information without executing steps 303 and 304. For example, if the data asset management system provides two search interfaces: a metadata retrieval interface, an asset view retrieval interface; when the user opens the metadata retrieval interface, the retrieval information can be generated by selecting or inputting corresponding sentences, parameters, unique identification of metadata and other information, i.e. step 303 and step 304 are not executed; when the user opens the asset view retrieval interface, the retrieval information may be generated by manipulating the asset view, i.e. steps 303, 304 are performed. The steps of the process of generating the search information by operating the asset view in step 303 and step 304 are merely an example, and are not limited in particular. In addition, steps 301, 302 may be performed once at the time of asset view creation, and may not be performed subsequently.
The embodiment of the application also provides a metadata updating method, which comprises the following steps: the data asset management system detects metadata in databases; and after the data asset management system determines that the first metadata of the first database is changed, updating the first metadata recorded in the data resource pool. That is, a detection module (such as detection module 108 shown in the architecture of fig. 1) may be present in the data management system, and detection parameters such as a detection period, detected metadata, a detected database, and the like may be set in the detection module. The detection module in the data asset management system periodically acquires the change information of the metadata to update the metadata in the data resource pool, and the accuracy of the metadata in the data resource pool is ensured. Here, the change information of the first metadata may include deleting a field, modifying a field, deleting the first metadata, etc., and the data asset management system may detect the full amount metadata or part of metadata of each database, and correspondingly, update the metadata in the data resource pool may also update the full amount metadata or part of metadata, and specifically may set detection parameters according to the importance, variability, etc. of the real specific data corresponding to the metadata. For example, if the average change speed of some data is once a year and the average change speed of some data is once a day, the corresponding detection periods may be set respectively. The above detection parameters are merely examples, and may be set according to specific requirements, and are not limited in particular.
The embodiment of the application provides an asset view updating method, wherein after the data asset management system updates the first metadata recorded in the data resource pool, the method further comprises the following steps: the data asset management system updates a metadata state of the first metadata in the data asset pool, the metadata state comprising: newly added metadata, valid metadata and invalid metadata; and the data asset management system updates the asset view where the first metadata is located according to the metadata state of the first metadata. That is, updating the metadata state of the first metadata in the data asset pool is further included after updating the first metadata recorded in the data asset pool. For example, the first metadata in the data resource pool has timeliness, if the first metadata exceeds a preset time and is not detected and/or updated by the detection module, the data asset management system modifies the valid metadata of the first metadata state into invalid metadata, and after the detection module detects and/or updates the first metadata, the data asset management system modifies the invalid metadata state of the first metadata into the valid metadata state. If the first metadata is newly added metadata, determining that the first metadata is in a newly added metadata state from the newly added moment of the first metadata until the preset time is exceeded, and deleting the newly added metadata state of the first metadata. In this process, if metadata information in the asset view includes metadata states, the metadata information of the first metadata in the asset view is updated accordingly each time the state of the first metadata is changed. The asset view contains metadata states, so that a user can intuitively acquire the state of the current metadata, and the user can conveniently judge the availability of the metadata.
The embodiment of the application also provides a method for updating the metadata of the association system, which comprises the following steps: after the data asset management system determines that the first metadata of the first database are changed, the first metadata recorded in the association system is updated through the message middleware, and the association system contains the first metadata. That is, the first metadata is also included in the association system, and after the data asset management system determines that the first metadata of the first database is changed, the first metadata recorded in the association system is updated through the message middleware. For example, when the association system is a sharing system, the first metadata is provided for the relevant development department of the enterprise, and the development department user can acquire the first data in the first database through the first metadata in the sharing system.
Based on the above method, the embodiment of the present application provides a flow of a data asset management method, as shown in fig. 4, including:
step 401, the data asset management system acquires first metadata in a first database according to first connection information in the data resource pool.
Step 402, the data asset management system determines modification information of the first metadata according to the acquired first metadata in the first database and the first metadata in the data resource pool.
Step 403, the data asset management system updates the first metadata in the data resource pool according to the change information of the first metadata.
Step 404, the data asset management system updates the status of the first metadata in the data asset pool to valid metadata.
Step 405, the data asset management system updates the asset view according to the status of the first metadata.
Step 406, the data asset management system sends the change information of the first metadata to the association system through the message middleware so as to enable the association system to update synchronously.
The embodiment of the application also provides a data statistics method, which comprises the following steps: and the data asset management system performs all statistical tasks according to the set metadata statistical tasks. That is, the data asset management system may further include a statistics module (such as statistics module 109 shown in fig. 1), in which the statistics module includes a set metadata statistics task, for example, a data amount, a statistics period, etc. of a database corresponding to the set metadata are set, and a statistics record is generated according to a result of executing each statistics task and stored in the statistics module. The data volume change information and the like in each database can be acquired by calling the statistical records, so that the data volume change of the database can be conveniently observed and analyzed, the statistical records corresponding to the metadata can be used as metadata information to be loaded into the asset view, and the subsequent data asset management system updates the asset view according to the generated statistical records, so that the intuitiveness of the data volume of the database is improved.
The embodiment of the application also provides a database access method, which comprises the following steps: the data asset management system receives a second plug-in of the newly added database and performs validity check on the second plug-in; after the verification is passed, the data asset management system determines second connection information and is connected with the newly-added database through the second plug-in and the second connection information; the data asset management system acquires second metadata of the newly-added database, and stores the second metadata and the second connection information into the data resource pool, wherein the second metadata are description information of the second data in the newly-added database. That is, when the data asset management system accesses the newly-added data, the second plug-in is loaded in the data asset management system, the validity of the second plug-in is checked, after the second plug-in passes the check, the second connection information of the newly-added database is obtained and is connected with the newly-added database, the second metadata of the newly-added database is obtained, and the second connection information and the second metadata are stored in the data resource pool so as to facilitate the subsequent management of the data asset of the newly-added database.
A generic plug-in framework design approach based on the OSGI specification for dynamic extension of microservices may be employed in the data asset management system herein. The method can be used for packaging the data source configuration of different databases into different plugins, the plugins are not mutually affected, and a certain database plugin can be dynamically controlled to be started or stopped according to specific service requirements. The plug-in frame may include: plug-in configurator and plug-in framework core. The plug-in configurator is used for configuring the plug-in running environment and the plug-in path. And the plug-in framework core is used for determining and storing functional components for realizing various service functions in the plug-in according to the plug-in running environment and the plug-in path, and establishing an association relation with the micro service system so that the micro service system can call the functional components for realizing the service functions needing to be expanded in the stored functional components. The plug-in frame core mainly comprises: application context, plug-in factory, plug-in function component manager, plug-in controller, and plug-in listener. The application context is used for storing functional components for realizing various service functions in the plug-in unit and functional components of the self service of the micro service system, and is mainly used for establishing association relation between the plug-in unit and the micro service system. Therefore, when the micro service business system determines that the function expansion is needed, the plug-in interface is called, and the function component for realizing the business function needing to be expanded is called from the application context of the plug-in framework core. The plug-in factory is used for loading and analyzing the plug-in according to the plug-in running environment and the plug-in path configured by the plug-in configurator.
Based on the same conception, the embodiment of the invention provides a data asset management device which is suitable for a data asset management system for managing at least one database, wherein the data asset management system comprises a unified interface and at least one plug-in, the database corresponds to the plug-in one by one, and the at least one plug-in is respectively connected with the corresponding database through the unified interface; fig. 5 is a schematic diagram of a data asset management device according to an embodiment of the present application, as shown in fig. 5, including:
a transceiver module 501 for receiving the search information;
a determining module 502, configured to determine first metadata and first connection information from the data resource pool according to the retrieval information; metadata of each database and connection information of each database are stored in the data resource pool; the first connection information is connection information of a first database, and the first metadata is description information of first data in the first database;
a processing module 503, configured to invoke a first plug-in corresponding to the first database according to the first connection information;
the processing module 503 is further configured to invoke the unified interface through the first plug-in, and establish a connection with the first database according to the first connection information;
the processing module 503 is further configured to obtain the first data in the first database.
Optionally, the transceiver module 501 is further configured to receive an asset view creation instruction;
the determining module 502 is further configured to determine an asset directory corresponding to the asset view creation instruction, and load metadata information obtained from the data resource pool to the corresponding asset directory, so as to complete creation of an asset view, where the metadata information includes unique identifiers of metadata.
Optionally, the processing module 503 is further configured to detect each metadata in each database; the determining module 502 is further configured to update the first metadata recorded in the data resource pool after determining that the first metadata of the first database is changed.
Optionally, the processing module 503 is further configured to update a metadata state of the first metadata in the data resource pool, where the metadata state includes: newly added metadata, valid metadata and invalid metadata; the processing module 503 is further configured to update an asset view in which the first metadata is located according to a metadata state of the first metadata.
Optionally, the determining module 502 is further configured to update, through a message middleware, the first metadata recorded in an association system after determining that the first metadata of the first database is changed, where the association system includes the first metadata.
Optionally, the transceiver module 501 is further configured to receive a second plug-in of the newly added database, and perform validity check on the second plug-in; the determining module 502 is further configured to determine second connection information after the verification is passed, and connect with the new database through the second plug-in and the second connection information; the processing module 503 is further configured to obtain second metadata of the new database, and store the second metadata and the second connection information to the data resource pool, where the second metadata is description information of the second data in the new database.
Optionally, the processing module 503 is further configured to perform each statistical task according to the set metadata statistical task.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various modifications and variations can be made in the present application without departing from the spirit or scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to cover such modifications and variations.
Claims (9)
1. The data asset management method is characterized by being suitable for a data asset management system for managing a plurality of databases, wherein the data asset management system comprises a unified interface and a plurality of plugins, the databases are in one-to-one correspondence with the plugins, and the plugins are respectively connected with the corresponding databases through the unified interface; the method comprises the following steps:
the data asset management system receiving retrieval information;
the data asset management system determines first metadata and first connection information from a data resource pool according to the retrieval information; metadata of each database and connection information of each database are stored in the data resource pool; the first connection information is connection information of a first database, and the first metadata is description information of first data in the first database;
the data asset management system calls a first plug-in corresponding to the first database according to the first connection information;
the data asset management system calls the unified interface through the first plug-in and establishes connection with the first database according to the first connection information;
the data asset management system obtains the first data in the first database;
the data asset management system receives a second plug-in of the newly added database and performs validity check on the second plug-in;
after the verification is passed, the data asset management system determines second connection information and is connected with the newly-added database through the second plug-in and the second connection information;
the data asset management system acquires second metadata of the newly-added database, and stores the second metadata and the second connection information into the data resource pool, wherein the second metadata are description information of the second data in the newly-added database.
2. The method as recited in claim 1, further comprising:
the data asset management system receives an asset view creation instruction;
the data asset management system determines an asset directory corresponding to the asset view creation instruction, and loads metadata information acquired from the data resource pool to the corresponding asset directory, so that creation of the asset view is completed, and the metadata information comprises unique identifiers of metadata.
3. The method as recited in claim 1, further comprising:
the data asset management system detects metadata in databases;
and after the data asset management system determines that the first metadata of the first database is changed, updating the first metadata recorded in the data resource pool.
4. The method of claim 3, wherein after the data asset management system updates the first metadata recorded in the data asset pool, further comprising:
the data asset management system updates a metadata state of the first metadata in the data asset pool, the metadata state comprising: newly added metadata, valid metadata and invalid metadata;
and the data asset management system updates the asset view where the first metadata is located according to the metadata state of the first metadata.
5. A method as recited in claim 3, further comprising:
after the data asset management system determines that the first metadata of the first database are changed, the first metadata recorded in the association system is updated through the message middleware, and the association system contains the first metadata.
6. The method as claimed in any one of claims 1 to 5, further comprising:
and the data asset management system performs all statistical tasks according to the set metadata statistical tasks.
7. The data asset management device is characterized by being suitable for a data asset management system for managing a plurality of databases, wherein the data asset management system comprises a unified interface and a plurality of plug-ins, the databases are in one-to-one correspondence with the plug-ins, and the plug-ins are respectively connected with the corresponding databases through the unified interface; the device comprises:
the receiving and transmitting module is used for receiving the retrieval information;
the determining module is used for determining first metadata and first connection information from the data resource pool according to the retrieval information; metadata of each database and connection information of each database are stored in the data resource pool; the first connection information is connection information of a first database, and the first metadata is description information of first data in the first database;
the processing module is used for calling a first plug-in corresponding to the first database according to the first connection information;
the processing module is further used for calling the unified interface through the first plug-in and establishing connection with the first database according to the first connection information;
the processing module is further used for acquiring the first data in the first database;
the receiving and transmitting module is also used for receiving a second plug-in of the newly added database and carrying out validity check on the second plug-in through the processing module;
the processing module is further used for determining second connection information after verification is passed, and connecting the second connection information with the newly-added database through the second plug-in and the second connection information;
the processing module is further configured to obtain second metadata of the new database, and store the second metadata and the second connection information to the data resource pool, where the second metadata is description information of the second data in the new database.
8. A computer readable storage medium, characterized in that the computer readable storage medium stores a program which, when run on a computer, causes the computer to implement the method of any one of claims 1 to 6.
9. A computer device, comprising:
a memory for storing a computer program;
a processor for invoking a computer program stored in said memory, performing the method according to any of claims 1 to 6 in accordance with the obtained program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011279767.4A CN112395340B (en) | 2020-11-16 | 2020-11-16 | Data asset management method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011279767.4A CN112395340B (en) | 2020-11-16 | 2020-11-16 | Data asset management method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112395340A CN112395340A (en) | 2021-02-23 |
| CN112395340B true CN112395340B (en) | 2023-07-28 |
Family
ID=74599521
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011279767.4A Active CN112395340B (en) | 2020-11-16 | 2020-11-16 | Data asset management method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112395340B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672983A (en) * | 2021-08-25 | 2021-11-19 | 山大地纬软件股份有限公司 | Service handling data privacy protection system and method based on block chain |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202452A (en) * | 2016-07-15 | 2016-12-07 | 复旦大学 | The uniform data resource management system of big data platform and method |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9767172B2 (en) * | 2014-10-03 | 2017-09-19 | Palantir Technologies Inc. | Data aggregation and analysis system |
| CN106203828A (en) * | 2016-07-11 | 2016-12-07 | 浪潮软件集团有限公司 | Data management platform based on data full life cycle management |
| CN107330080B (en) * | 2017-07-03 | 2020-05-22 | 北京希嘉创智教育科技有限公司 | Data processing method and device and computer equipment applying same |
| CN107590208A (en) * | 2017-08-25 | 2018-01-16 | 国网辽宁省电力有限公司 | A kind of electric network data heterogeneous database data cut-in method of plug-in type |
| CN109582722A (en) * | 2018-11-30 | 2019-04-05 | 珠海市新德汇信息技术有限公司 | Public security resource data service system |
| CN110990726A (en) * | 2019-10-30 | 2020-04-10 | 武汉大学 | Space-time big data intelligent service system |
-
2020
- 2020-11-16 CN CN202011279767.4A patent/CN112395340B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202452A (en) * | 2016-07-15 | 2016-12-07 | 复旦大学 | The uniform data resource management system of big data platform and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112395340A (en) | 2021-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210405629A1 (en) | System and method for interoperable communication of an automation system component with multiple information sources | |
| US7814459B2 (en) | System and method for automated on demand replication setup | |
| US10936659B2 (en) | Parallel graph events processing | |
| US9280568B2 (en) | Zero downtime schema evolution | |
| CN110471949B (en) | Data blood margin analysis method, device, system, server and storage medium | |
| CN107077388A (en) | System and method for providing end-to-end life cycle in multi-tenant application server environment | |
| CN109492053B (en) | Method and device for accessing data | |
| CN107077389A (en) | For using system and method during global operation in multi-tenant application server environment | |
| JP2017107555A (en) | Methods, systems and programs for determining identities of software in software containers | |
| US8966442B2 (en) | Custom code innovation management | |
| US20140123114A1 (en) | Framework for integration and execution standardization (fiesta) | |
| CN109408493A (en) | A kind of moving method and system of data source | |
| US20220398186A1 (en) | Lineage-driven source code generation for building, testing, deploying, and maintaining data marts and data pipelines | |
| US20130339488A1 (en) | Enterprise services framework for mobile devices | |
| US20160274874A1 (en) | Method and apparatus for processing request | |
| WO2015154435A1 (en) | Software parameter management method and device | |
| Munonye et al. | Evaluation of data storage patterns in microservices archicture | |
| CN111552728A (en) | Data processing method, system, terminal and storage medium of block chain | |
| US20140006000A1 (en) | Built-in response time analytics for business applications | |
| US9141517B2 (en) | Public solution model test automation framework | |
| CN112395340B (en) | Data asset management method and device | |
| US10826965B2 (en) | Network monitoring to identify network issues | |
| CN115796758A (en) | Factory rule management platform | |
| CN112860948A (en) | Metadata management method and system based on multi-tenant SaaS architecture and electronic equipment | |
| Strauch et al. | Migrating eScience applications to the cloud: methodology and evaluation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |