CN112035455A - Metadata processing method and device, computer equipment and storage medium - Google Patents

Metadata processing method and device, computer equipment and storage medium Download PDF

Info

Publication number
CN112035455A
CN112035455A CN202010890063.4A CN202010890063A CN112035455A CN 112035455 A CN112035455 A CN 112035455A CN 202010890063 A CN202010890063 A CN 202010890063A CN 112035455 A CN112035455 A CN 112035455A
Authority
CN
China
Prior art keywords
metadata information
metadata
information
processed
determining
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010890063.4A
Other languages
Chinese (zh)
Inventor
邱荣明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Property and Casualty Insurance Company of China Ltd
Original Assignee
Ping An Property and Casualty Insurance Company of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Property and Casualty Insurance Company of China Ltd filed Critical Ping An Property and Casualty Insurance Company of China Ltd
Priority to CN202010890063.4A priority Critical patent/CN112035455A/en
Publication of CN112035455A publication Critical patent/CN112035455A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application relates to the technical field of data acquisition, and particularly discloses a metadata processing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring metadata information of a source end system, and determining the metadata information as metadata information to be processed; determining invalid metadata information in the metadata information to be processed; determining accessed metadata information in the metadata information to be processed; determining metadata information to be accessed according to target metadata information in the metadata information to be processed, wherein the target metadata information comprises the metadata information except invalid metadata information and accessed metadata information in the metadata information to be processed; and accessing the metadata information to be accessed into a data warehouse. The embodiment of the application can improve the data integrity of the data warehouse, and improve the data quality and the data management capability. In addition, the present application also relates to blockchain techniques, and the metadata information can be stored in blockchain nodes.

Description

Metadata processing method and device, computer equipment and storage medium
Technical Field
The present application relates to the field of data acquisition technologies, and in particular, to a metadata processing method and apparatus, a computer device, and a storage medium.
Background
The data warehouse is used as a data source of enterprise intelligent operation analysis, and is a strategic set for providing data support of all types for decision making processes of all levels of an enterprise, so that whether the data warehouse needs to be accessed to all application systems of the enterprise, particularly to effective data of all application systems of the enterprise, has been accessed to all effective data, namely the data integrity of the current data warehouse is an important ring for judging the data quality management of the data warehouse.
In the traditional data warehouse construction, a user of downstream data is usually required to determine a demand, then the demand is provided for a data access person, and the data access person accesses corresponding data according to the demand, but the construction mode has many disadvantages, including the problems of high management difficulty, slow data response time, scattered data demands and management demands, so that the data integrity of the data warehouse is difficult to guarantee.
Disclosure of Invention
The application provides a metadata processing method, a metadata processing device, computer equipment and a storage medium, which can improve the data integrity of a data warehouse and improve the data quality and data management capability.
In a first aspect, the present application provides a metadata processing method, including:
acquiring metadata information of a source end system, and determining the metadata information as metadata information to be processed;
determining invalid metadata information in the metadata information to be processed;
determining accessed metadata information in the metadata information to be processed;
determining metadata information to be accessed according to target metadata information in the metadata information to be processed, wherein the target metadata information comprises the metadata information except invalid metadata information and accessed metadata information in the metadata information to be processed;
and accessing the metadata information to be accessed into a data warehouse.
In a second aspect, the present application provides a metadata processing apparatus, the apparatus comprising:
the data acquisition module is used for acquiring metadata information of a source end system and determining the metadata information as metadata information to be processed;
the invalidation determining module is used for determining invalid metadata information in the metadata information to be processed;
the accessed determining module is used for determining the accessed metadata information in the metadata information to be processed;
the to-be-accessed determining module is used for determining to-be-accessed metadata information according to target metadata information in the to-be-processed metadata information, wherein the target metadata information comprises the metadata information except invalid metadata information and accessed metadata information in the to-be-processed metadata information;
and the data access module is used for accessing the metadata information to be accessed into a data warehouse.
In a third aspect, the present application further provides a computer device comprising a memory and a processor; the memory is used for storing a computer program; the processor is configured to execute the computer program and to implement the metadata processing method as described above when executing the computer program.
In a fourth aspect, the present application also provides a computer-readable storage medium storing a computer program which, when executed by a processor, causes the processor to implement the metadata processing method as described above.
The application discloses a metadata processing method, a metadata processing device, computer equipment and a storage medium, wherein the infringement comprises the following steps: acquiring metadata information of a source end system, and determining the metadata information as metadata information to be processed; determining invalid metadata information in the metadata information to be processed; determining accessed metadata information in the metadata information to be processed; determining metadata information to be accessed according to target metadata information in the metadata information to be processed, wherein the target metadata information comprises the metadata information except invalid metadata information and accessed metadata information in the metadata information to be processed; and accessing the metadata information to be accessed into a data warehouse. According to the embodiment of the application, the metadata information can be acquired from the source data, invalid metadata information and accessed metadata information in the metadata information are removed, the removed metadata information can be accessed into the data warehouse, the data integrity of the data warehouse can be improved, and the data quality of the data warehouse is improved. The data warehouse is driven by the demand to change into the data driving type, so that the data service capability of the data warehouse can be improved, complicated data access demand management work can be omitted, the labor cost is reduced, and the communication timeliness is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flow chart of a metadata processing method provided in an embodiment of the present application;
FIG. 2 is a schematic flow chart diagram of another metadata processing method provided by an embodiment of the present application;
fig. 3 is a schematic block diagram of a metadata processing apparatus provided in an embodiment of the present application;
fig. 4 is a schematic block diagram of a structure of a computer device provided in an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The flow diagrams depicted in the figures are merely illustrative and do not necessarily include all of the elements and operations/steps, nor do they necessarily have to be performed in the order depicted. For example, some operations/steps may be decomposed, combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
It is to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
The embodiment of the application provides a metadata processing method and device, computer equipment and a storage medium. Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
Referring to fig. 1, fig. 1 is a schematic flowchart of a metadata processing method according to an embodiment of the present application, and as shown in fig. 1, the metadata processing method specifically includes steps S101 to S105.
S101, obtaining metadata information of a source end system, and determining the metadata information as metadata information to be processed.
The Metadata information (Metadata) is data for describing data, and the Metadata information mainly includes information for describing a data property (property) and is used for supporting functions such as indicating a storage location, history data, resource searching, file recording and the like. The metadata information is an electronic catalog, and is used for describing and collecting the content or characteristics of data so as to achieve the aim of compiling the catalog and further achieve the aim of assisting data retrieval. Alternatively, the metadata information includes table information, field information, and the like.
The source end system can be an application system of an enterprise, the source end system can generate metadata information along with the application system of the enterprise, and the metadata information of the source end system can be used as a source of the metadata information needing to be accessed in the data warehouse, so that the metadata information can be obtained from the source end system.
In the using process of the data warehouse and the source end system, the metadata information in the source end system may change, and meanwhile, the use requirement of the data warehouse may change, so that the effective data required by the data warehouse may change. Therefore, when managing the metadata in the data warehouse, all the metadata information can be acquired from the source end system, so that new valid data can be determined from all the metadata information to be accessed into the data warehouse.
Optionally, newly added metadata information may also be obtained from the source end system, for example, if the metadata information is obtained from the source end system for the first time, all metadata information in the source end system may be obtained, and determined as the metadata information to be processed. If the metadata information is not acquired from the source end system for the first time, the newly-added metadata information in the source end system can be acquired, and the metadata information is determined to be the metadata information to be processed.
Optionally, the obtained metadata information may be stored into the chunk link points to increase security.
And S102, determining invalid metadata information in the metadata information to be processed.
Different metadata information corresponds to different description functions, according to different requirements of the data warehouse, part of the metadata information is useful for the data warehouse and needs to be accessed into the data warehouse, and part of the metadata information is invalid for the data warehouse and does not need to be accessed into the data warehouse. Therefore, invalid metadata information in the metadata information to be processed needs to be determined, and the invalid metadata information does not need to be accessed into the data warehouse.
In one embodiment, the operation of determining invalid metadata information in the metadata information to be processed may be implemented as follows:
acquiring metadata analysis information, and determining invalid metadata information in the to-be-processed metadata information according to the metadata analysis information, wherein the metadata analysis information comprises pre-stored metadata information and analysis results thereof, and the analysis results comprise whether each piece of metadata information is invalid or not.
The metadata analysis information comprises pre-stored metadata information and an analysis result thereof, and which of the pre-stored metadata information is invalid can be determined according to the metadata analysis information. The metadata analysis information may correspond to a currently applied data warehouse, that is, an analysis result in the metadata analysis information is obtained according to a requirement of the data warehouse, and invalid metadata information included in the analysis result is invalid according to the current data warehouse.
The metadata analysis information may be stored in a preset database, and the metadata analysis information may be obtained from the preset database. The metadata information to be processed includes metadata information acquired from the source end system, and may be one or more pieces of metadata information, and the metadata information to be processed is searched from preset metadata information of the metadata analysis information, so that invalid metadata information in the metadata information to be processed can be determined.
The analysis result may include a determination result of whether the metadata information is valid for the data warehouse, and for example, the metadata information whose analysis result is "no access" is the metadata information invalid for the data warehouse, and the metadata information whose analysis result is "access required" is the metadata information useful for the data warehouse. Alternatively, the metadata analysis information may be manually analyzing each metadata information to determine a corresponding analysis result; the metadata analysis information may also be a result of analyzing the metadata information through a preset analysis model to output a corresponding analysis result.
In one embodiment, as shown in fig. 2, the method of the embodiment of the present application further includes the following operations:
s201, obtaining metadata information of a source end system, and inputting the metadata information of the source end system into a preset analysis model, so that the preset analysis model outputs an analysis result of the metadata information.
S202, determining the metadata information of the source end system and the corresponding analysis result as metadata analysis information.
In operation S201, obtaining the metadata information of the source end system may be obtaining the metadata information from the source end system for the first time, or may be subsequently collecting the metadata information of the source end system. The first acquisition may be to acquire all metadata information of the source end system, and subsequently, if the metadata information is newly added to the source end system, the newly added metadata information may be acquired.
The acquiring of the metadata information of the source end system in operation S201 may be performed simultaneously with the acquiring of the metadata information of the source end system in operation S101, and the acquiring of the metadata information of the source end system in operation S201 may be performed before the acquiring of the metadata information of the source end system in operation S101. Illustratively, if the metadata information of the get source system in operation S101 is the first get, the metadata information of the get source system in operation S201 may be performed simultaneously with the metadata information of the get source system in operation S101; after metadata information of a source end system is obtained, the metadata information is determined to be metadata information to be processed, and the metadata information of the source end system is input into a preset analysis model, so that the preset analysis model outputs an analysis result of the metadata information. Whereas, if the acquisition of the metadata information of the source end system in operation S101 is not the first acquisition, operation S201 may be performed before operation S101.
The preset analysis model may be a preset model and corresponds to a currently applied data warehouse, and the preset analysis model may output an analysis result corresponding to the metadata information according to the input metadata information, where the analysis result includes whether the metadata information is invalid for the current data warehouse.
After the analysis result of the metadata information is obtained according to the preset analysis model, the metadata information and the corresponding analysis result thereof can be determined as metadata analysis information. Optionally, the metadata analysis information may also be stored in a preset database.
S103, determining accessed metadata information in the metadata information to be processed.
The accessed metadata information is the metadata information already accessed into the data warehouse, and the metadata information already accessed into the data warehouse does not need to be accessed into the data warehouse again, so that the accessed metadata information in the to-be-processed metadata information can be determined.
In one embodiment, the operation of determining the accessed metadata information in the metadata information to be processed may be implemented as follows:
obtaining metadata information in an operation data storage layer in a data warehouse, and determining to-be-processed metadata information corresponding to the metadata information in the operation data storage layer as accessed metadata information.
Wherein, an Operation Data Store (ODS) layer is a database belonging to one layer in the Data warehouse as a temporary area of the Data warehouse. The operation data storage layer integrates metadata information from different data sources into a theme-oriented, integrated, enterprise-global and consistent data set after extraction, cleaning, transmission and other operations, is used for meeting enterprise quasi-real-time Online analysis Processing (OLAP) operation and enterprise-global Online analysis Processing operation, and provides integrated data for a data warehouse.
The operation data storage layer in the data warehouse stores integrated global-oriented metadata information, so that the metadata information in the operation data storage layer in the data warehouse can be acquired, that is, the metadata information which is accessed into the operation data storage layer can be determined, and further, the to-be-processed metadata information corresponding to the metadata information in the operation data storage layer can be determined as the accessed metadata information.
S104, determining metadata information to be accessed according to target metadata information in the metadata information to be processed, wherein the target metadata information comprises the metadata information except invalid metadata information and accessed metadata information in the metadata information to be processed.
Invalid metadata information and accessed metadata information in the metadata information to be processed can be removed to obtain target metadata information. The accessed metadata information and the invalid metadata information are removed from the target metadata information, and then the metadata information to be accessed can be determined according to the target metadata information.
In one embodiment, the operation of determining to-be-accessed metadata information according to target metadata information in the to-be-processed metadata information may be implemented as follows:
and determining invalid metadata information in target metadata information in the metadata information to be processed, and determining the metadata information except the invalid metadata information in the target metadata information in the metadata information to be processed as the metadata information to be accessed.
In operation S102, invalid metadata information in the to-be-processed metadata information is determined, and there may be some invalid metadata information in the to-be-processed metadata information that is not determined. For example, if invalid metadata information in the to-be-processed metadata information is determined according to the metadata analysis information, and the metadata analysis information may only include metadata information obtained from the source end system for the first time and an analysis result corresponding to the metadata information, if the metadata information obtained from the source end system in operation S101 includes newly added metadata information in the source end system, invalid metadata information existing in the newly added metadata information cannot be determined according to the metadata analysis information. There is a need to further determine invalid metadata information in the target metadata information.
After invalid metadata information in the target metadata information is removed, the obtained to-be-accessed metadata information does not include the invalid metadata information and the accessed metadata information, and then the to-be-accessed metadata information can be accessed to a data warehouse. Alternatively, the target metadata information may be analyzed manually to determine invalid metadata information in the target metadata information.
In one embodiment, the operation of determining invalid metadata information in target metadata information in the to-be-processed metadata information may be implemented as follows:
inputting target metadata information of the metadata information to be processed into a preset analysis model so that the preset analysis model outputs an analysis result of the target metadata information; and determining invalid metadata information in the target metadata information according to the analysis result of the target metadata information.
The preset analysis model may be a preset model and corresponds to a currently applied data warehouse, and the preset analysis model may output an analysis result corresponding to the metadata information according to the input metadata information, where the analysis result includes whether the metadata information is invalid for the current data warehouse. Therefore, the target metadata information is input to the preset analysis model, and invalid metadata information in the target metadata information can be determined.
In this embodiment, the metadata information to be processed needs to be analyzed to obtain invalid metadata information in the metadata information to be processed; and determining the invalid metadata information in the metadata information to be processed in the already executed operation S102, that is, the target metadata information has already removed part of the invalid metadata information in the metadata information to be processed, and the removed invalid metadata information does not need to be analyzed again, so that it is possible to avoid that secondary analysis reduces the efficiency of data processing.
In one embodiment, after determining invalid metadata information in target metadata information in the metadata information to be processed, the method further includes the following operations: and updating the metadata analysis information according to the invalid metadata information of the target metadata information.
The metadata analysis information may be updated according to invalid metadata information of the target metadata information, and the invalid metadata information in the target metadata and the analysis result thereof may be determined to be stored in the metadata analysis information. Therefore, the metadata analysis information can be expanded, so that more invalid metadata information in the metadata information to be processed can be eliminated when the next time the invalid metadata information in the metadata information to be processed is determined by using the metadata analysis information.
And S105, accessing the metadata information to be accessed into a data warehouse.
The finally obtained metadata information to be accessed does not include accessed metadata information or invalid metadata information, namely the metadata information to be accessed includes the metadata information which is useful for the current data warehouse and needs to be accessed, and then the metadata information to be accessed can be accessed into the data warehouse so as to improve the data coverage rate in the data warehouse.
The metadata information to be accessed can be accessed to an operation data storage layer in the data warehouse, for example, the metadata information to be accessed can be sent to a data access person, and the data access person can access the metadata information to be accessed to the operation data storage layer of the data warehouse through an access tool.
In one embodiment, the method provided by the embodiment of the present application further includes the following operations:
and determining the sum of the metadata information in the operation data storage layer and the metadata information to be accessed as first metadata information, and determining the ratio of the metadata information in the operation data storage layer and the first metadata information as data coverage rate.
Wherein, the data coverage rate is the metadata information in the operation data storage layer/(the metadata information in the operation data storage layer + the metadata information to be accessed); the metadata information in the operation data storage layer is the metadata information which has been accessed to the data warehouse, and the metadata information to be accessed is the metadata information which comes from the source end system and is valid for the current data warehouse, but has not been accessed to the data warehouse, so that the sum of the metadata information in the operation data storage layer and the metadata information to be accessed can be determined as first metadata information, and the first metadata information includes all the metadata information valid for the current data warehouse. The ratio between the metadata information in the operational data store layer and said first metadata information is then determined as the data coverage, i.e. the ratio of all metadata information valid for the current data warehouse that has been accessed to the data warehouse.
The data service capability of the current data warehouse can be determined according to the data coverage rate, and managers can optimize the data warehouse according to the data service capability.
According to the metadata processing method, the metadata information is obtained from the source data, invalid metadata information and accessed metadata information are removed, the removed metadata information can be accessed into the data warehouse, the data integrity of the data warehouse can be improved, and the data quality of the data warehouse can be improved. The data warehouse is driven by the demand to change into the data driving type, so that the data service capability of the data warehouse can be improved, complicated data access demand management work can be omitted, the labor cost is reduced, and the communication timeliness is improved.
Referring to fig. 3, fig. 3 is a schematic block diagram of a metadata processing apparatus according to an embodiment of the present application, where the metadata processing apparatus may be configured in a server, and is configured to perform any one of the metadata processing methods described above. As shown in fig. 3, the metadata processing apparatus 10 includes:
the data acquisition module 11 is configured to acquire metadata information of a source end system and determine the metadata information as to-be-processed metadata information;
an invalidity determining module 12, configured to determine invalid metadata information in the to-be-processed metadata information;
an accessed determining module 13, configured to determine accessed metadata information in the to-be-processed metadata information;
a to-be-accessed determining module 14, configured to determine to-be-accessed metadata information according to target metadata information in the to-be-processed metadata information, where the target metadata information includes metadata information in the to-be-processed metadata information, except for invalid metadata information and accessed metadata information;
and the data access module 15 is configured to access the metadata information to be accessed into a data warehouse.
In one embodiment, the invalidation determining module is specifically configured to: acquiring metadata analysis information; and determining invalid metadata information in the metadata information to be processed according to the metadata analysis information, wherein the metadata analysis information comprises prestored metadata information and an analysis result thereof, and the analysis result comprises whether each metadata information is invalid or not.
In one embodiment, the metadata processing apparatus further comprises:
the source end system analysis system comprises a first analysis module, a second analysis module and a third analysis module, wherein the first analysis module is used for acquiring metadata information of a source end system and inputting the metadata information of the source end system into a preset analysis model so that the preset analysis model outputs an analysis result of the metadata information;
and the second analysis module is used for determining the metadata information of the source end system and the corresponding analysis result as metadata analysis information.
In one embodiment, the accessed determining module is specifically configured to: acquiring metadata information in an operation data storage layer in a data warehouse; and determining the metadata information to be processed corresponding to the metadata information in the operation data storage layer as the accessed metadata information.
In one embodiment, the metadata processing apparatus further comprises:
the calculation module is used for determining the sum of the metadata information in the operation data storage layer and the metadata information to be accessed as first metadata information;
a coverage determination module for determining a ratio of the metadata information in the operational data storage layer and the first metadata information as a data coverage.
In one embodiment, the module for determining to be accessed is specifically configured to: determining invalid metadata information in target metadata information in the metadata information to be processed; and determining the metadata information except the invalid metadata information in the target metadata information in the metadata information to be processed as the metadata information to be accessed.
In one embodiment, the metadata processing apparatus further comprises:
and the updating module is used for updating the metadata analysis information according to the invalid metadata information of the target metadata information after the invalid metadata information in the target metadata information in the metadata information to be processed is determined.
Referring to fig. 4, fig. 4 is a schematic block diagram of a computer device according to an embodiment of the present application.
Referring to fig. 4, the computer device 100 includes a processor 110 and a memory 120 connected by a system bus, wherein the memory 120 may include a nonvolatile storage medium and an internal memory.
The non-volatile storage medium may store an operating system and a computer program. The computer program includes program instructions that, when executed, cause a processor to perform any one of the metadata processing methods.
The processor is used for providing calculation and control capability and supporting the operation of the whole computer equipment.
The internal memory provides an environment for the execution of a computer program on a non-volatile storage medium, which when executed by a processor, causes the processor to perform any of the metadata processing methods.
Those skilled in the art will appreciate that the architecture shown in fig. 4 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
It should be understood that the Processor may be a Central Processing Unit (CPU), and the Processor may be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Wherein, in one embodiment, the processor is configured to execute a computer program stored in the memory to implement the steps of:
acquiring metadata information of a source end system, and determining the metadata information as metadata information to be processed;
determining invalid metadata information in the metadata information to be processed;
determining accessed metadata information in the metadata information to be processed;
determining metadata information to be accessed according to target metadata information in the metadata information to be processed, wherein the target metadata information comprises the metadata information except invalid metadata information and accessed metadata information in the metadata information to be processed;
and accessing the metadata information to be accessed into a data warehouse.
In one embodiment, the processor, when implementing the determining invalid metadata information in the to-be-processed metadata information, is configured to implement:
acquiring metadata analysis information;
and determining invalid metadata information in the metadata information to be processed according to the metadata analysis information, wherein the metadata analysis information comprises prestored metadata information and an analysis result thereof, and the analysis result comprises whether each metadata information is invalid or not.
In one embodiment, the processor is further configured to implement:
acquiring metadata information of a source end system, and inputting the metadata information of the source end system into a preset analysis model so that the preset analysis model outputs an analysis result of the metadata information;
and determining the metadata information of the source end system and the corresponding analysis result as metadata analysis information.
In one embodiment, the processor, in implementing the determining the accessed metadata information in the metadata information to be processed, is configured to implement:
acquiring metadata information in an operation data storage layer in a data warehouse;
and determining the metadata information to be processed corresponding to the metadata information in the operation data storage layer as the accessed metadata information.
In one embodiment, the processor is further configured to implement:
determining the sum of the metadata information in the operation data storage layer and the metadata information to be accessed as first metadata information;
determining a ratio of the metadata information in the operational data storage layer and the first metadata information as a data coverage.
In one embodiment, when the determining of metadata information to be accessed according to target metadata information in the metadata information to be processed is implemented, the processor is configured to implement:
determining invalid metadata information in target metadata information in the metadata information to be processed;
and determining the metadata information except the invalid metadata information in the target metadata information in the metadata information to be processed as the metadata information to be accessed.
In one embodiment, after implementing the determining of invalid metadata information in target metadata information of the to-be-processed metadata information, the processor is further configured to implement:
and updating the metadata analysis information according to the invalid metadata information of the target metadata information.
The embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, where the computer program includes program instructions, and the processor executes the program instructions to implement any metadata processing method provided in the embodiment of the present application.
The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, for example, a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
While the invention has been described with reference to specific embodiments, the scope of the invention is not limited thereto, and those skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. A metadata processing method, comprising:
acquiring metadata information of a source end system, and determining the metadata information as metadata information to be processed;
determining invalid metadata information in the metadata information to be processed;
determining accessed metadata information in the metadata information to be processed;
determining metadata information to be accessed according to target metadata information in the metadata information to be processed, wherein the target metadata information comprises the metadata information except invalid metadata information and accessed metadata information in the metadata information to be processed;
and accessing the metadata information to be accessed into a data warehouse.
2. The method according to claim 1, wherein the determining invalid metadata information in the metadata information to be processed comprises:
acquiring metadata analysis information;
and determining invalid metadata information in the metadata information to be processed according to the metadata analysis information, wherein the metadata analysis information comprises prestored metadata information and an analysis result thereof, and the analysis result comprises whether each metadata information is invalid or not.
3. The metadata processing method according to claim 2, wherein the method further comprises:
acquiring metadata information of a source end system, and inputting the metadata information of the source end system into a preset analysis model so that the preset analysis model outputs an analysis result of the metadata information;
and determining the metadata information of the source end system and the corresponding analysis result as metadata analysis information.
4. The method according to claim 1, wherein the determining the accessed metadata information in the metadata information to be processed comprises:
acquiring metadata information in an operation data storage layer in a data warehouse;
and determining the metadata information to be processed corresponding to the metadata information in the operation data storage layer as the accessed metadata information.
5. The metadata processing method according to claim 4, wherein the method further comprises:
determining the sum of the metadata information in the operation data storage layer and the metadata information to be accessed as first metadata information;
determining a ratio of the metadata information in the operational data storage layer and the first metadata information as a data coverage.
6. The method according to claim 1, wherein the determining metadata information to be accessed according to target metadata information in the metadata information to be processed comprises:
determining invalid metadata information in target metadata information in the metadata information to be processed;
and determining the metadata information except the invalid metadata information in the target metadata information in the metadata information to be processed as the metadata information to be accessed.
7. The method according to claim 6, wherein after determining invalid metadata information in target metadata information in the metadata information to be processed, the method further comprises:
and updating the metadata analysis information according to the invalid metadata information of the target metadata information.
8. A metadata processing apparatus, comprising:
the data acquisition module is used for acquiring metadata information of a source end system and determining the metadata information as metadata information to be processed;
the invalidation determining module is used for determining invalid metadata information in the metadata information to be processed;
the accessed determining module is used for determining the accessed metadata information in the metadata information to be processed;
the to-be-accessed determining module is used for determining to-be-accessed metadata information according to target metadata information in the to-be-processed metadata information, wherein the target metadata information comprises the metadata information except invalid metadata information and accessed metadata information in the to-be-processed metadata information;
and the data access module is used for accessing the metadata information to be accessed into a data warehouse.
9. A computer device, wherein the computer device comprises a memory and a processor;
the memory is used for storing a computer program;
the processor for executing the computer program and implementing the metadata processing method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to implement the metadata processing method according to any one of claims 1 to 7.
CN202010890063.4A 2020-08-28 2020-08-28 Metadata processing method and device, computer equipment and storage medium Pending CN112035455A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010890063.4A CN112035455A (en) 2020-08-28 2020-08-28 Metadata processing method and device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010890063.4A CN112035455A (en) 2020-08-28 2020-08-28 Metadata processing method and device, computer equipment and storage medium

Publications (1)

Publication Number Publication Date
CN112035455A true CN112035455A (en) 2020-12-04

Family

ID=73585929

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010890063.4A Pending CN112035455A (en) 2020-08-28 2020-08-28 Metadata processing method and device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112035455A (en)

Similar Documents

Publication Publication Date Title
CN112445854B (en) Multi-source service data real-time processing method, device, terminal and storage medium
CN110795455A (en) Dependency relationship analysis method, electronic device, computer device and readable storage medium
US20110016451A1 (en) Method and system for generating test cases for a software application
US11593336B2 (en) Data pipeline branching
CN114528008A (en) Code control method, device and medium based on distributed version control system
CN111737227A (en) Data modification method and system
CN114416703A (en) Method, device, equipment and medium for automatically monitoring data integrity
US20080189150A1 (en) Supply chain multi-dimensional serial containment process
US20100005203A1 (en) Method of Merging and Incremantal Construction of Minimal Finite State Machines
Srivastava et al. NetSeekR: a network analysis pipeline for RNA-Seq time series data
CN111523921B (en) Funnel analysis method, analysis device, electronic device, and readable storage medium
CN104217032A (en) Method and device for processing database dimensions
CN110866007B (en) Information management method, system and computer equipment for big data application and table
CN111752958A (en) Intelligent associated label method, device, computer equipment and storage medium
CN115086047B (en) Interface authentication method and device, electronic equipment and storage medium
CN116339716A (en) Flow chart analysis method
CN115878592A (en) Government affair data management method and device, storage medium and electronic equipment
CN112035455A (en) Metadata processing method and device, computer equipment and storage medium
CN115543428A (en) Simulated data generation method and device based on strategy template
US20210064694A1 (en) Templated form generation from meta model based rules with concurrent editing
US7269610B2 (en) System and method to observe user behavior and perform actions introspectable objects
CN113312410B (en) Data map construction method, data query method and terminal equipment
CN111831874B (en) Webpage data information acquisition method and device, computer equipment and storage medium
CN116383777B (en) Data management platform and data right determining method facing data management
AU2021287457B2 (en) "Log Data Compliance"

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination