CN117435558A - Metadata management method, computing device and storage medium - Google Patents
Metadata management method, computing device and storage medium Download PDFInfo
- Publication number
- CN117435558A CN117435558A CN202311754773.4A CN202311754773A CN117435558A CN 117435558 A CN117435558 A CN 117435558A CN 202311754773 A CN202311754773 A CN 202311754773A CN 117435558 A CN117435558 A CN 117435558A
- Authority
- CN
- China
- Prior art keywords
- metadata
- scene
- database
- scene model
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007726 management method Methods 0.000 title claims abstract description 66
- 238000000034 method Methods 0.000 claims abstract description 40
- 230000008859 change Effects 0.000 claims description 16
- 238000012986 modification Methods 0.000 claims description 9
- 230000004048 modification Effects 0.000 claims description 9
- 238000012544 monitoring process Methods 0.000 claims description 4
- 230000004044 response Effects 0.000 claims description 4
- 238000011161 development Methods 0.000 abstract description 6
- 238000012423 maintenance Methods 0.000 abstract description 5
- 238000004590 computer program Methods 0.000 description 19
- 230000008569 process Effects 0.000 description 14
- 238000012545 processing Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 4
- 238000012217 deletion Methods 0.000 description 4
- 230000037430 deletion Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000003068 static effect Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000003780 insertion Methods 0.000 description 2
- 230000037431 insertion Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the specification provides a metadata management method, which constructs a scene model with a tree structure based on scene objects and metadata hung under the scene objects, wherein one node comprises one scene object in the scene model, the hierarchical relationship among the nodes corresponds to the hierarchical relationship among the scene objects included by the nodes, and the aim of classifying and managing actual service scenes can be achieved based on adding, deleting and modifying the tree structure. Meanwhile, the metadata management method is used for carrying out metadata mounting under scene objects based on metadata driving, so that the method only needs to carry out data mounting based on standardized metadata when facing different actual service scenes, and even when facing more complex service scenes, the requirements of the complex service scenes can be met by simply inheriting and expanding the metadata, and the development and maintenance cost is effectively reduced.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a metadata management technology in the field of data processing technologies, and more particularly, to a metadata management method, a computing device, and a storage medium.
Background
With the continuous acceleration of the informatization process, digital transformation has become a necessary trend for many enterprises. With the continuous enhancement of industry competition, the enterprise digital transformation can bring more than optimization in cost and improvement in operation and more than strategic improvement to the enterprise. The digitized transformation of an enterprise relates to management of data assets of the enterprise, wherein the data assets of the enterprise refer to data resources owned or controlled by the enterprise, and the data resources of the enterprise can be understood as data which can bring value to the enterprise.
Metadata is used as data for describing data, metadata management runs through a data asset management flow, and is an important basis for realizing data asset management, so that the metadata management method has important significance for managing metadata of enterprises. In this context, how to provide a technical solution to optimize metadata management becomes a technical problem that needs to be solved by those skilled in the art.
Disclosure of Invention
The embodiment of the specification provides a metadata management method, a computing device and a storage medium, and the aim of optimizing metadata management is fulfilled by classifying and managing scene objects.
In order to achieve the technical purpose, the embodiment of the specification provides the following technical scheme:
In a first aspect, an embodiment of the present specification provides a metadata management method, where metadata includes a logical entity, the metadata management method including:
establishing scene objects, wherein one scene object corresponds to one actual service scene;
acquiring metadata and loading the metadata under the scene object;
and constructing a scene model, wherein the scene model comprises a plurality of nodes, the nodes form a tree structure, one node comprises one scene object, and the hierarchical relationship between the nodes corresponds to the hierarchical relationship between the scene objects included by the nodes.
In a second aspect, one embodiment of the present specification also provides a computing device including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the metadata management method as described above when executing the computer program.
In a third aspect, an embodiment of the present specification further provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the metadata management method as described above.
In a fourth aspect, the present description embodiments provide a computer program product or a computer program, the computer program product comprising a computer program stored in a computer readable storage medium; the processor of the computer device reads the computer program from the computer readable storage medium, and the processor implements the steps of the metadata management method described above when executing the computer program.
As can be seen from the above technical solutions, in the metadata management method provided in the embodiments of the present disclosure, a scene model with a tree structure is constructed based on scene objects and metadata that is mounted under the scene objects, in the scene model, one of the nodes includes one of the scene objects, and a hierarchical relationship between the nodes corresponds to a hierarchical relationship between the scene objects included in the nodes, so that the purpose of classification management on an actual service scene can be achieved based on addition, deletion and modification of the tree structure. Meanwhile, the metadata management method is used for carrying out metadata mounting under scene objects based on metadata driving, so that the method only needs to carry out data mounting based on standardized metadata when facing different actual service scenes, and even when facing more complex service scenes, the requirements of the complex service scenes can be met by simply inheriting and expanding the metadata, and the development and maintenance cost is effectively reduced. That is, the metadata management method provided in the embodiment of the present disclosure implements, by means of metadata driving, metadata mounting under a scene object and classification management of an actual service scene, thereby achieving the purpose of optimizing metadata management.
Drawings
In order to more clearly illustrate the embodiments of the present description or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present description, and that other drawings may be obtained according to the drawings provided without inventive effort to a person skilled in the art.
Fig. 1 is a schematic flow chart of a metadata management method according to an embodiment of the present disclosure.
Fig. 2 is a schematic diagram of a scene model according to an embodiment of the present disclosure.
Fig. 3 is a schematic flow chart of data synchronization between a first database and a second database according to an embodiment of the present disclosure.
Fig. 4 is a schematic flow chart of a scene model publishing according to an embodiment of the present disclosure.
Fig. 5 is a schematic flow chart of a scene model comparison according to an embodiment of the present disclosure.
Fig. 6 is a schematic structural diagram of a metadata management apparatus according to an embodiment of the present disclosure.
Fig. 7 is a schematic structural diagram of a computing device according to an embodiment of the present disclosure.
Detailed Description
Unless defined otherwise, technical or scientific terms used in the embodiments of the present specification should be given the ordinary meaning as understood by one of ordinary skill in the art to which the present specification belongs. The terms "first," "second," and the like, as used in the embodiments of the present disclosure, do not denote any order, quantity, or importance, but rather are used to avoid intermixing of the components.
Throughout the specification, unless the context requires otherwise, the word "plurality" means "at least two", and the word "comprising" is to be construed as open, inclusive meaning, i.e. as "comprising, but not limited to. In the description of the present specification, the terms "one embodiment," "some embodiments," "example embodiments," "examples," "particular examples," or "some examples," etc., are intended to indicate that a particular feature, structure, material, or characteristic associated with the embodiment or example is included in at least one embodiment or example of the present specification. The schematic representations of the above terms do not necessarily refer to the same embodiment or example.
The technical solutions of the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is apparent that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.
In the related technology, when a scene knowledge platform is constructed, the realization is generally realized based on industry related data management business, namely, in order to adapt to enterprise business, the data characteristics of business data are abstracted, metadata are integrated and managed, and the realization mode has strong business. When metadata is constructed and managed for different business scenes, different adaptations are needed, when complex scenes are faced, codes are needed to be repeatedly written, and development cost is high.
In order to solve the problem, the inventor of the present specification finds that a scene model can be constructed based on a tree structure when the scene model is constructed, and realizes classification and management of actual business scenes corresponding to different scene objects in the scene model; meanwhile, based on the characteristic of metadata driving, the metadata is built and is mounted under the scene object, so that when different actual service scenes are faced, only the standardized metadata is required to be used for data mounting, even when more complex service scenes are faced, the requirements of the complex service scenes can be met by simply inheriting and expanding the metadata, and the development and maintenance cost is effectively reduced.
Based on this, the embodiment of the present disclosure provides an improved metadata management scheme, where a scene model with a tree structure may be constructed based on scene objects and metadata that is mounted under the scene objects, where one of the nodes includes one of the scene objects, and a hierarchical relationship between the nodes corresponds to a hierarchical relationship between the scene objects included in the nodes, so as to achieve the purpose of classification management of an actual service scene. Meanwhile, the metadata management method is used for carrying out metadata mounting under scene objects based on metadata driving, so that the method only needs to carry out data mounting based on standardized metadata when facing different actual service scenes, and even when facing more complex service scenes, the requirements of the complex service scenes can be met by simply inheriting and expanding the metadata, and the development and maintenance cost is effectively reduced. That is, the metadata management method provided in the embodiment of the present disclosure implements, by means of metadata driving, metadata mounting under a scene object and classification management of an actual service scene, thereby achieving the purpose of optimizing metadata management.
Based on the above-described concept, the metadata management method provided in the embodiment of the present specification will be exemplarily described below with reference to the accompanying drawings.
As an alternative embodiment, fig. 1 illustrates an alternative flowchart of a metadata management method provided by an embodiment of the present specification, where the method flow may be implemented by a scenario knowledge platform, for example, by a server forming the scenario knowledge platform.
Referring to fig. 1, the metadata management method may include the following steps.
S101: scene objects are established, one corresponding to each actual business scene.
The actual business scenario may include a business scenario that an enterprise or industry needs to pay attention to, for example, for an e-commerce enterprise, the actual business scenario may include a commodity management scenario, an order processing scenario, a customer service scenario, and the like; in these scenes, the commodity management scene may further include sub-scenes such as a commodity shelf scene, and an information modification scene, where the relationship between the commodity management scene and the sub-scenes is a parent-child level relationship. Similarly, the order processing scene can comprise sub-scenes such as an order placing scene, a payment scene, a warehouse-out scene, a shipping scene and the like, and the relationship between the order processing scene and the sub-scenes is also a parent-child level relationship. The customer service scene can comprise sub-scenes such as a customer consultation scene, a refund processing scene, a complaint processing scene and the like, and the relation between the customer service scene and the sub-scenes is also a parent-child level relation. Each of the above mentioned scenes may be referred to as one actual service scene, and the scene objects may refer to virtual objects corresponding to the actual service scenes. It should be noted that the actual business scenario is not limited to the business scenario under the electronic market scenario, but may be selected according to actual requirements, and the above description is only an example.
S102: and acquiring metadata and loading the metadata under the scene object.
Metadata is data describing the data, and in an alternative embodiment, the metadata may include logical model information, logical entity attributes, ER (Entity Relationship ) relationships, and the like, and the logical model may also be referred to as a logical entity. Among the data models obtained by data modeling, the logical model and the physical model are mainly involved. Data modeling is the process of finding, analyzing, and determining data requirements.
It can be found that in some actual service scenarios, metadata has versatility and can be applied to each actual service scenario. For example, in an electronic market scenario, the logical model information included in the metadata may include commodity information, customer information, order information, and the like, the logical entity attribute may include commodity ID in commodity information, commodity name, customer ID in customer information, customer address, order ID in order information, order time, and the like, and the ER relationship may include a relationship between order information and commodity information, and the like, and these metadata are applicable in various actual business scenarios (commodity management scenario, order processing scenario, customer service scenario, and sub-scenario thereof). Therefore, in this embodiment, the metadata constructed in step S102 may be referred to as standard metadata, where the metadata is standard metadata, and the standard metadata is applicable to a plurality of actual service scenarios, and by loading the standard metadata under a scenario object, a metadata management method driven by metadata is implemented, so that when the method faces different actual service scenarios, only the standardized metadata is required to be used for data loading, and even when the method faces more complex service scenarios, only simple inheritance and expansion of metadata are required to meet the requirements of the complex service scenarios, thereby effectively reducing the cost of development and maintenance.
S103: and constructing a scene model, wherein the scene model comprises a plurality of nodes, the nodes form a tree structure, one node comprises one scene object, and the hierarchical relationship between the nodes corresponds to the hierarchical relationship between the scene objects included by the nodes.
The scene model may be a tree structure including a plurality of nodes, and the tree structure may include a plurality of edges (edges) connecting two nodes in addition to the plurality of nodes. In the tree structure, nodes at different levels and connected by edges have parent-child relationships. Referring to fig. 2, fig. 2 shows a schematic diagram of a scene model formed by each actual service scene in the aforementioned electronic market scene. By means of fig. 2, it is easy to find out that by constructing a scene model, one node includes one scene object, and the hierarchical relationship between the nodes corresponds to the hierarchical relationship between the scene objects included by the nodes, so that the purpose of classifying and managing the actual service scene is achieved.
In one embodiment of the present disclosure, classification and management of scene objects in a scene model may be implemented by a method of adding, deleting, and modifying nodes based on a tree structure. The method can adopt a recursion idea to add, delete and modify the nodes of the tree structure on the premise of ensuring the stability of the tree structure.
Specifically, the adding (or creating) operation to the nodes of the tree structure may include: the child nodes needing to be added are added into the child node list of the parent node through recursion operation on the child node list of the parent node.
Performing a Delete operation on a node of the tree structure may include: starting from the node to be deleted, each child node is recursively accessed and deleted. It is important to note that the remaining tree structure is guaranteed to be valid during the deletion process, either by deleting all child nodes or by mounting child nodes under the parent node of the deleted node.
The look-up (or Read) operation for a node of the tree structure may include: and recursively accessing each child node from the root node, returning a result when the target node is found, and otherwise, continuing to recursively search.
Performing a change (or Update) operation on a node of the tree structure may include: updating the value of the node and updating the location of the node. In both operations we need to find the target node first, which is in fact a read (look-up) procedure, but after finding the target node an update operation is performed, such as changing its value or moving its location. Recursion can be used in this process to ensure that the process is performed completely throughout the tree, while ensuring that the legitimacy of the operation, such as the moved position, does not introduce loops.
As previously described, in some embodiments, the metadata includes logical model information, logical entity attributes, and ER relationships; while ER relationships are often complex, to facilitate managing complex ER relationship data, in one embodiment of the present description, the metadata management method further includes:
storing a first portion of the metadata in a first database; a first portion of the metadata including at least the logical model information and the ER relationship in the metadata, the first database including a graph database;
and storing the second part of the metadata in a second database, wherein the second part of the metadata at least comprises logical entity attributes in the metadata, and the second database comprises a relational database.
A graph database is a database used to store, query and process graph structure data. Such a data structure of the graph is made up of nodes (which may also be referred to as vertices) and connecting lines called edges. This enables the graph database to represent relationships between objects directly and essentially, and also makes the graph database a database well suited for storing and querying complex relationships. Thus, in this embodiment, the first portion of metadata including ER relationships is stored in the first database, facilitating lookup operations for complex relationships.
However, the graph database may not have much advantage over the relational database when storing some detail data, so in this embodiment, the second portion including the logical entity attribute in the metadata is stored in the second database, which achieves the purpose of fully exploiting the respective advantages of the relational database and the graph database for metadata storage.
Optionally, in order to take advantage of the graph database on the basis of metadata storage by using the relational database and the graph database, in one embodiment of the present specification, the metadata management method further includes:
and responding to the carrying relation query request, querying ER relation corresponding to the relation query request in the first database, and returning the ER relation as a response result to a requester of the relation query request.
In this embodiment, when a relationship query request for ER relationship query is responded, ER relationship query is performed in the first database, so that the advantage that the graph database is convenient for querying complex relationships is fully exerted, and query efficiency is improved.
Optionally, the first database comprises a Neo4j database and the second database comprises a PostgreSQL database. The Neo4j database can represent information in a natural and visual way, and can realize efficient graph data processing, so that data query and processing become simpler and faster. The PostgreSQL database is a powerful open source object relational database system. It uses and extends the SQL (Structured Query Language ) language, enabling it to store and query various types of data.
In order to achieve data synchronization of the first database and the second database, in one embodiment of the present specification, the metadata management method further includes:
monitoring data change in the second database, and generating a synchronization instruction carrying metadata change information when metadata stored in the second database changes, wherein the synchronization instruction comprises an instruction based on a graph database script language;
the synchronization instruction is executed to synchronize metadata in the first database with metadata in the second database.
Taking the example that the first database includes Neo4j database and the second database includes PostgreSQL database, the synchronization instruction may be an instruction based on a CQL (Cypher Query Language, cytoer query language) statement, and the above synchronization process may refer to fig. 3, and the process may include:
s1, monitoring data change of a PostgreSQL database: this can be done by setting a trigger on PostgreSQL or using a listening tool. Changes in PostgreSQL can be captured in real time as they occur, such as insertions, updates, or deletions.
S2, converting the CQL sentences into CQL sentences corresponding to the graph database: after capturing the data change information in the PostgreSQL database, the goal of this step is to translate these changes into corresponding CQL statements; and adding the CQL statement to a message queue to wait for execution. CQL is a graph query language for Neo4j, just like SQL is for a relational database. This step acts as a translation process to translate specific data changes into CQL for execution on Neo4 j.
S3, executing CQL statement: after converting the data changes to CQL statements, the last step is to execute these CQL statements on Neo4 j. In this way, data synchronization from the PostgreSQL database to the Neo4j database can be accomplished.
After the scene model is constructed, the process of model release is also involved, and the metadata management method further comprises the following steps:
responding to a model release request carrying a release position of the scene model, and comparing the historical scene model with the scene model when the historical scene model exists at the release position to obtain metadata difference information;
and updating the metadata in the historical scene model according to the metadata difference information after the modification approval of the metadata difference information passes.
In this embodiment, when a historical scene model exists at the release position, it indicates that the current scene model is an update to the historical scene model, metadata difference information may be obtained by comparing the historical scene model with the scene model, and metadata in the historical scene model may be updated after the modification approval of the metadata difference information passes.
The modification approval application of the metadata difference information can be submitted to a developer through an open-source flow engine Camunda tool, so that the developer can approve the modification information of the model. Specifically, an approval flow chart can be constructed on a flow engine interface, task labels are defined, a background needs to follow an interface protocol of a Canunda, corresponding task labels are bound, and an approval flow interface is created.
Wherein referring to fig. 4, the responding to the model publishing request of the publishing location carrying the scene model, when the historical scene model exists in the publishing location, comparing the historical scene model with the scene model, and obtaining the metadata difference information includes:
responding to a model release request carrying a release position of the scene model, and calling a change release model interface for designating a preset naming space, wherein the preset naming space comprises an object set of a database, and the preset naming space corresponds to the model release request of the release position of the scene model;
creating the preset namespace when the preset namespace does not exist;
inquiring the historical scene model in the preset naming space, and if the historical scene model exists, comparing the historical scene model with the scene model to obtain metadata difference information; if the historical scene model does not exist, the scene model is created in the preset naming space.
When the historical scene model is queried, the historical scene model can be queried through querying logic model information and logic entity attributes in the scene model.
The metadata difference information obtained by comparing the old and new models (i.e., the historical scene model and the scene model) may include: the method comprises the steps of adding information of a logic entity attribute, renaming information of the logic entity attribute, non-null constraint of the logic entity attribute, inconsistent information of uniqueness constraint, inconsistent information of a default value of the logic entity attribute, change information of a type change or precision change of the logic entity attribute, change information of an index and the like.
Creating the scene model in the preset namespace without the historical scene model may refer to creating a scene model in the preset namespace from the constructed scene model.
To simplify the process of comparing the historical scene model with the scene model, in one embodiment of the present description, the comparing the historical scene model with the scene model includes:
creating a first cursor in the historical scene model and a second cursor in the scene model; the first cursor is used for positioning key value pairs in the historical scene model, and the second cursor is used for positioning key value pairs in the scene model; the key value pair comprises a scene object and metadata hung under the scene object;
And sequentially comparing the historical scene model with key value pairs in the scene model by moving the first cursor and the second cursor so as to determine the metadata difference information.
Referring to fig. 5, the comparison process may specifically include:
acquiring a cursor A and a cursor B: the cursor a and the cursor B may refer to a first cursor and a second cursor created in the historical scene model and the scene model, respectively;
cursor a moves down one line: it may mean that the cursor a moves downward by one line at the position pointed by the cursor a in the database corresponding to the historical scene model;
cursor B moves down one line: it may mean that the cursor B moves one line down at the position pointed to in the database corresponding to the scene model;
judging the cursor AB to the bottom: the method can be used for judging whether the positions of the cursor A and the cursor B in the respective databases point to the last row, and if so, judging that the comparison is finished; if not, judging whether the cursor A is at the bottom;
in the case of cursor a to the bottom, determining that the remaining data in a is not in B (i.e., the remaining data in the database pointed to by cursor a is not in the database pointed to by cursor B);
judging whether the cursor B is bottomed or not under the condition that the cursor A is bottomed, and determining that the rest data in the B is not in A under the condition that the cursor B is bottomed (namely, the rest data in a database pointed by the cursor B is not in a database pointed by the cursor A);
In the case that cursor B is not at bottom, the primary keys keyA and keyB at the cursor are fetched: i.e. the field or combination of fields that fetches a row of data of the database to which cursor a and cursor B respectively point, the primary key can be any value that ensures the uniqueness of each row of data.
If the key a=key b, the key a represents that the primary key values at the two cursors are equal, judging whether other fields are equal, outputting a conclusion that the data pointed by the two cursors are equal under the condition that the other fields are equal, returning to the step that the cursor a moves down one line, and executing the comparison of the data of the next line; outputting a conclusion that the data pointed by the two cursors are different under the condition that other fields are not equal, returning to the step of moving the cursor A down one row, and executing the comparison of the data of the next row;
if the keyA is not equal to the keyB and the keyA is less than the keyB, the method indicates that the line data pointed by the cursor A is not in B, namely the line data pointed by the cursor A is not in a database pointed by the cursor B, and the operation of moving the cursor A down one line is executed, after the cursor A moves down one line, whether the cursor A is at the bottom is judged, and under the condition that the cursor A is at the bottom, a conclusion that the rest data in the A is not in B is output; returning to the step of taking out the key A and key B of the main key at the cursor when the cursor A is not at the bottom;
If the keyA is not equal to the keyB and the keyA is more than the keyB, the data of the row pointed by the cursor B is not in A, namely the data of the row pointed by the cursor B is not in the database pointed by the cursor A, the operation of moving the cursor B down one row is executed, after the cursor B moves down one row, whether the cursor B is at the bottom is judged, and under the condition that the cursor B is at the bottom, the conclusion that the rest data in the cursor B is not in A is output; in the case that cursor B is not at the bottom, return to the step of fetching the primary keys keyA and keyB at the cursor.
The objective of comparing historical scene models with scene models can be achieved efficiently and reliably in the manner described above, because only the portions that have changed are compared and processed between the two databases by the cursor comparison method, rather than traversing the entire database. The incremental comparison and synchronization method can remarkably improve the efficiency of processing the large database and reduce unnecessary calculation and transmission cost; in addition, by comparing the primary key values, the cursor comparison method can ensure that the synchronized data cannot be repeatedly processed in the synchronization process, and repeated insertion or deletion of the data is avoided. Meanwhile, due to the adoption of the incremental comparison and synchronization modes, even if errors or interruptions occur in the synchronization process, the synchronization can be easily restored and the rest data can be continuously processed, so that the consistency and the integrity of the data are ensured.
The embodiment of the present specification also provides a metadata management apparatus, and the contents described below may be referred to in correspondence with the above description. Fig. 6 exemplarily shows an alternative block diagram of a metadata management apparatus, which may include, with reference to fig. 6:
an object establishing module 601, configured to establish a scene object, where one scene object corresponds to one actual service scene;
the data acquisition module 602 is configured to acquire metadata and mount the metadata under the scene object;
the model building module 603 is configured to build a scene model, where the scene model includes a plurality of nodes, the plurality of nodes form a tree structure, one node includes one scene object, and a hierarchical relationship between the nodes corresponds to a hierarchical relationship between the scene objects included by the nodes.
Optionally, the metadata includes logical model information, logical entity attributes, and ER relationships;
the metadata management apparatus further includes:
a data storage module for storing a first portion of the metadata in a first database; a first portion of the metadata including at least the logical model information and the ER relationship in the metadata, the first database including a graph database;
And storing the second part of the metadata in a second database, wherein the second part of the metadata at least comprises logical entity attributes in the metadata, and the second database comprises a relational database.
Optionally, the metadata management apparatus further includes:
the data synchronization module is used for monitoring the data change in the second database, and generating a synchronization instruction carrying metadata change information when the metadata stored in the second database changes, wherein the synchronization instruction comprises an instruction based on a graph database script language;
the synchronization instruction is executed to synchronize metadata in the first database with metadata in the second database.
Optionally, the metadata management apparatus further includes: and the relation query module is used for responding to the carrying relation query request, querying ER relation corresponding to the relation query request in the first database, and returning the ER relation as a response result to a requester of the relation query request.
Optionally, the first database comprises a Neo4j database and the second database comprises a PostgreSQL database.
Optionally, the metadata management apparatus further includes:
The data updating module is used for responding to a model release request of a release position carrying the scene model, and comparing the historical scene model with the scene model when the historical scene model exists in the release position, so as to obtain metadata difference information;
and updating the metadata in the historical scene model according to the metadata difference information after the modification approval of the metadata difference information passes.
Optionally, the data updating module compares the historical scene model with the scene model specifically for:
creating a first cursor in the historical scene model and a second cursor in the scene model; the first cursor is used for positioning key value pairs in the historical scene model, and the second cursor is used for positioning key value pairs in the scene model; the key value pair comprises a scene object and metadata hung under the scene object;
and sequentially comparing the historical scene model with key value pairs in the scene model by moving the first cursor and the second cursor so as to determine the metadata difference information.
Optionally, the data updating module responds to a model publishing request carrying a publishing position of the scene model, when a historical scene model exists in the publishing position, the historical scene model is compared with the scene model, and metadata difference information is obtained specifically for:
Responding to a model release request carrying a release position of the scene model, and calling a change release model interface for designating a preset naming space, wherein the preset naming space comprises an object set of a database, and the preset naming space corresponds to the model release request of the release position of the scene model;
creating the preset namespace when the preset namespace does not exist;
inquiring the historical scene model in the preset naming space, and if the historical scene model exists, comparing the historical scene model with the scene model to obtain metadata difference information; if the historical scene model does not exist, the scene model is created in the preset naming space.
The specific limitation regarding the metadata management apparatus may be referred to the limitation regarding the metadata management method hereinabove, and will not be described herein. The respective modules in the above-described sorting apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
Another embodiment of the present application further provides a computing device, referring to fig. 7, and an exemplary embodiment of the present specification further provides a computing device, including: a memory storing a computer program, and a processor that when executing the computer program performs the steps in the metadata management method according to various embodiments of the present specification described in the above embodiments of the present specification.
The internal structure of the computing device may be as shown in fig. 7, including a processor, memory, network interface, and input devices connected by a system bus. Wherein the processor of the computing device is configured to provide computing and control capabilities. The memory of the computing device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computing device is for communicating with an external terminal through a network connection. The computer program, when executed by a processor, performs the steps in the metadata management method according to various embodiments of the present specification described in the above embodiments of the present specification.
The processor may include a host processor, and may also include a baseband chip, modem, and the like.
The memory stores programs for executing the technical scheme of the invention, and can also store an operating system and other key services. In particular, the program may include program code including computer-operating instructions. More specifically, the memory may include read-only memory (ROM), other types of static storage devices that may store static information and instructions, random access memory (random access memory, RAM), other types of dynamic storage devices that may store information and instructions, disk storage, flash, and the like.
The processor may be a general-purpose processor, such as a general-purpose Central Processing Unit (CPU), microprocessor, etc., or may be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs in accordance with aspects of the present invention. But may also be a Digital Signal Processor (DSP), application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.
The input device may include means for receiving data and information entered by a user, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer or gravity sensor, etc.
The output device may include means, such as a display screen, printer, speakers, etc., that allow information to be output to the user.
The communication interface may include means, such as any transceiver, for communicating with other devices or communication networks, such as ethernet, radio Access Network (RAN), wireless Local Area Network (WLAN), etc.
The processor executes the program stored in the memory and invokes other devices, which may be used to implement the steps of any of the metadata management methods provided in the embodiments of the present application.
The computing device can also comprise a display component and a voice component, wherein the display component can be a liquid crystal display screen or an electronic ink display screen, and an input device of the computing device can be a touch layer covered on the display component, can also be a key, a track ball or a touch pad arranged on a shell of the computing device, and can also be an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the architecture associated with the present description and is not limiting of the computing devices to which the present description may be applied, and that a particular computing device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In addition to the methods and apparatus described above, the metadata management methods provided by the embodiments of the present description may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps in the metadata management methods according to the various embodiments of the present description described in the "exemplary methods" section of the present description.
The computer program product may write program code for performing the operations of embodiments of the present description in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, the present specification embodiment also provides a computer-readable storage medium having stored thereon a computer program that is executed by a processor to perform the steps in the metadata management method according to the various embodiments of the present specification described in the above-described "exemplary method" section of the present specification.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples merely represent a few implementations of the present description, which are described in more detail and are not to be construed as limiting the scope of the solutions provided by the examples of the present description. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the present description, which is within the scope of the present description. Accordingly, the protection scope of the patent should be determined by the appended claims.
Claims (9)
1. A metadata management method, wherein metadata comprises a logical entity, the metadata management method comprising:
establishing scene objects, wherein one scene object corresponds to one actual service scene;
acquiring metadata, and loading the metadata under the scene object, wherein the metadata is standard metadata which is applicable to a plurality of actual service scenes;
Constructing a scene model, wherein the scene model comprises a plurality of nodes, the nodes form a tree structure, one node comprises one scene object, and the hierarchical relationship between the nodes corresponds to the hierarchical relationship between the scene objects included by the nodes;
responding to a model release request carrying a release position of the scene model, and comparing the historical scene model with the scene model when the historical scene model exists at the release position to obtain metadata difference information;
and updating the metadata in the historical scene model according to the metadata difference information after the modification approval of the metadata difference information passes.
2. The method of claim 1, wherein the metadata comprises logical model information, logical entity attributes, and ER relationships;
the metadata management method further includes:
storing a first portion of the metadata in a first database; a first portion of the metadata including at least the logical model information and the ER relationship in the metadata, the first database including a graph database;
and storing the second part of the metadata in a second database, wherein the second part of the metadata at least comprises logical entity attributes in the metadata, and the second database comprises a relational database.
3. The method as recited in claim 2, further comprising:
monitoring data change in the second database, and generating a synchronization instruction carrying metadata change information when metadata stored in the second database changes, wherein the synchronization instruction comprises an instruction based on a graph database script language;
the synchronization instruction is executed to synchronize metadata in the first database with metadata in the second database.
4. The method as recited in claim 2, further comprising:
and responding to the carrying relation query request, querying ER relation corresponding to the relation query request in the first database, and returning the ER relation as a response result to a requester of the relation query request.
5. The method of any one of claims 2-4, wherein the first database comprises a Neo4j database and the second database comprises a PostgreSQL database.
6. The method of claim 1, wherein the comparing the historical scene model with the scene model comprises:
creating a first cursor in the historical scene model and a second cursor in the scene model; the first cursor is used for positioning key value pairs in the historical scene model, and the second cursor is used for positioning key value pairs in the scene model; the key value pair comprises a scene object and metadata hung under the scene object;
And sequentially comparing the historical scene model with key value pairs in the scene model by moving the first cursor and the second cursor so as to determine the metadata difference information.
7. The method of claim 1, wherein the obtaining metadata difference information in response to a model publication request of a publication location carrying the scene model, when a historical scene model exists at the publication location, comparing the historical scene model with the scene model comprises:
responding to a model release request carrying a release position of the scene model, and calling a change release model interface for designating a preset naming space, wherein the preset naming space comprises an object set of a database, and the preset naming space corresponds to the model release request of the release position of the scene model;
creating the preset namespace when the preset namespace does not exist;
inquiring the historical scene model in the preset naming space, and if the historical scene model exists, comparing the historical scene model with the scene model to obtain metadata difference information; if the historical scene model does not exist, the scene model is created in the preset naming space.
8. A computing device, comprising: at least one memory storing one or more computer-executable instructions and at least one processor invoking the one or more computer-executable instructions to perform the metadata management method of any of claims 1-7.
9. A storage medium storing one or more computer-executable instructions which, when executed, implement the metadata management method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311754773.4A CN117435558B (en) | 2023-12-20 | 2023-12-20 | Metadata management method, computing device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311754773.4A CN117435558B (en) | 2023-12-20 | 2023-12-20 | Metadata management method, computing device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117435558A true CN117435558A (en) | 2024-01-23 |
| CN117435558B CN117435558B (en) | 2024-03-29 |
Family
ID=89553855
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311754773.4A Active CN117435558B (en) | 2023-12-20 | 2023-12-20 | Metadata management method, computing device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117435558B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101944209A (en) * | 2010-09-21 | 2011-01-12 | 用友软件股份有限公司 | Method and device for customizing metadata |
| US20140368522A1 (en) * | 2013-06-13 | 2014-12-18 | Autodesk, Inc. | File format and system for distributed scene graphs |
| US20170139987A1 (en) * | 2015-11-18 | 2017-05-18 | Sap Se | Calculation scenarios with extended semantic nodes |
| CN108733713A (en) * | 2017-04-21 | 2018-11-02 | 阿里巴巴集团控股有限公司 | Data query method and device in data warehouse |
| CN109344173A (en) * | 2018-09-10 | 2019-02-15 | 阿里巴巴集团控股有限公司 | Data managing method and device, data structure |
| CN109859319A (en) * | 2018-12-14 | 2019-06-07 | 浙江永拓信息科技有限公司 | Environment Visualization editing system |
| CN113392227A (en) * | 2021-05-31 | 2021-09-14 | 交控科技股份有限公司 | Metadata knowledge map engine system facing rail transit field |
| CN114970318A (en) * | 2022-04-22 | 2022-08-30 | 同济大学 | Automatic driving system safety evaluation method based on behavior result double-agent model |
| CN114996930A (en) * | 2022-05-26 | 2022-09-02 | 广东电网有限责任公司 | Modeling method and device, electronic equipment and storage medium |
| CN115185891A (en) * | 2022-09-14 | 2022-10-14 | 联想凌拓科技有限公司 | Data management method and device of file system, electronic equipment and storage medium |
| CN115712623A (en) * | 2022-11-22 | 2023-02-24 | 中国司法大数据研究院有限公司 | Batch data fault-tolerant acquisition method based on capture metadata change |
| CN115858498A (en) * | 2022-12-20 | 2023-03-28 | 上海方融科技有限责任公司 | Five-dimensional space-time distributed database construction method and device |
| CN116360766A (en) * | 2023-04-10 | 2023-06-30 | 深圳市惟客数据科技有限公司 | Low-code system and device based on DDD visual designer |
| CN116400900A (en) * | 2023-04-04 | 2023-07-07 | 上海太美数字科技有限公司 | Method and device for generating index |
| CN116992887A (en) * | 2023-08-31 | 2023-11-03 | 广东粤电南水发电有限责任公司 | Metadata data catalog processing method, device and processing equipment |
-
2023
- 2023-12-20 CN CN202311754773.4A patent/CN117435558B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101944209A (en) * | 2010-09-21 | 2011-01-12 | 用友软件股份有限公司 | Method and device for customizing metadata |
| US20140368522A1 (en) * | 2013-06-13 | 2014-12-18 | Autodesk, Inc. | File format and system for distributed scene graphs |
| US20170139987A1 (en) * | 2015-11-18 | 2017-05-18 | Sap Se | Calculation scenarios with extended semantic nodes |
| CN108733713A (en) * | 2017-04-21 | 2018-11-02 | 阿里巴巴集团控股有限公司 | Data query method and device in data warehouse |
| CN109344173A (en) * | 2018-09-10 | 2019-02-15 | 阿里巴巴集团控股有限公司 | Data managing method and device, data structure |
| CN109859319A (en) * | 2018-12-14 | 2019-06-07 | 浙江永拓信息科技有限公司 | Environment Visualization editing system |
| CN113392227A (en) * | 2021-05-31 | 2021-09-14 | 交控科技股份有限公司 | Metadata knowledge map engine system facing rail transit field |
| CN114970318A (en) * | 2022-04-22 | 2022-08-30 | 同济大学 | Automatic driving system safety evaluation method based on behavior result double-agent model |
| CN114996930A (en) * | 2022-05-26 | 2022-09-02 | 广东电网有限责任公司 | Modeling method and device, electronic equipment and storage medium |
| CN115185891A (en) * | 2022-09-14 | 2022-10-14 | 联想凌拓科技有限公司 | Data management method and device of file system, electronic equipment and storage medium |
| CN115712623A (en) * | 2022-11-22 | 2023-02-24 | 中国司法大数据研究院有限公司 | Batch data fault-tolerant acquisition method based on capture metadata change |
| CN115858498A (en) * | 2022-12-20 | 2023-03-28 | 上海方融科技有限责任公司 | Five-dimensional space-time distributed database construction method and device |
| CN116400900A (en) * | 2023-04-04 | 2023-07-07 | 上海太美数字科技有限公司 | Method and device for generating index |
| CN116360766A (en) * | 2023-04-10 | 2023-06-30 | 深圳市惟客数据科技有限公司 | Low-code system and device based on DDD visual designer |
| CN116992887A (en) * | 2023-08-31 | 2023-11-03 | 广东粤电南水发电有限责任公司 | Metadata data catalog processing method, device and processing equipment |
Non-Patent Citations (4)
| Title |
|---|
| XIUQI HUANG; YUANNING GAO; XINYI ZHOU; XIAOFENG GAO; GUIHAI CHEN: "An Adaptive Metadata Management Scheme Based on Deep Reinforcement Learning for Large-Scale Distributed File Systems", IEEE/ACM TRANSACTIONS ON NETWORKING, 21 April 2023 (2023-04-21), pages 2840 * |
| 乔嘉林: "物联网时序数据库管理系统文件结构设计与优化研究", 中国博士学位论文全文数据库, 15 January 2023 (2023-01-15), pages 138 - 27 * |
| 王威;刘卫东;宋佳兴;: "基于Globe的分布式LDAP元数据目录服务框架", 计算机工程, no. 11, 5 June 2006 (2006-06-05), pages 72 - 74 * |
| 衡星辰;陈丰;张诗军;甘杉;: "元数据管理系统在电力企业的研究与实践", 自动化与仪器仪表, no. 04, 25 April 2017 (2017-04-25), pages 107 - 109 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117435558B (en) | 2024-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10558642B2 (en) | Mechanism for deprecating object oriented data | |
| US8751437B2 (en) | Single persistence implementation of business objects | |
| US7599948B2 (en) | Object relational mapping layer | |
| US20100228764A1 (en) | Offline Validation of Data in a Database System for Foreign Key Constraints | |
| CN109492053B (en) | Method and device for accessing data | |
| CN109240946A (en) | The multi-level buffer method and terminal device of data | |
| WO2020192141A1 (en) | Method and device for simulating interface, computer apparatus, and storage medium | |
| US20210256012A1 (en) | Methods and apparatuses for reading and updating data structures, and electronic devices | |
| US20100185928A1 (en) | Extensibility for web based diagram visualization | |
| WO2023130933A1 (en) | Model processing method and apparatus, code aquisition method and apparatus, and electronic device | |
| CN112860953A (en) | Data importing method, device, equipment and storage medium of graph database | |
| CN117435558B (en) | Metadata management method, computing device and storage medium | |
| US9411555B2 (en) | Systems and methods for creating context sensitive graph topologies based on multidimensional context information | |
| US11016830B2 (en) | Entity-based service operation for object-based persistence | |
| US10970316B1 (en) | Maintaining a class hierarchy of an object-oriented data model using shared columns and/or overflow tables | |
| CN115543428A (en) | Simulated data generation method and device based on strategy template | |
| CN112347794B (en) | Data translation method, device, equipment and computer storage medium | |
| CN112699148A (en) | Method, device and equipment for refreshing cache and storage medium | |
| US8250090B2 (en) | Method for performing message-based distributed computing, involves executing expression of matched message response on members identified by set instance definition using dimension values derived from message | |
| TWI620134B (en) | Integration device and integration method thereof | |
| WO2024001569A1 (en) | Network configuration method and apparatus, storage medium and electronic apparatus | |
| WO2024016789A1 (en) | Log data query method and apparatus, and device and medium | |
| TWI699658B (en) | Data sharing system and data sharing method | |
| CN114997111B (en) | Service processing method, device, computer equipment and storage medium | |
| CN111708806B (en) | Data access method, device, server, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |