CN115934864A - Data asset management method and related device - Google Patents
Data asset management method and related device Download PDFInfo
- Publication number
- CN115934864A CN115934864A CN202211732906.3A CN202211732906A CN115934864A CN 115934864 A CN115934864 A CN 115934864A CN 202211732906 A CN202211732906 A CN 202211732906A CN 115934864 A CN115934864 A CN 115934864A
- Authority
- CN
- China
- Prior art keywords
- target
- data
- asset
- information
- bin
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000007726 management method Methods 0.000 title claims abstract description 96
- 238000007781 pre-processing Methods 0.000 claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 17
- 238000000034 method Methods 0.000 claims description 34
- 238000004140 cleaning Methods 0.000 claims description 13
- 238000003860 storage Methods 0.000 claims description 11
- 238000005516 engineering process Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 abstract description 6
- 239000008280 blood Substances 0.000 abstract description 5
- 210000004369 blood Anatomy 0.000 abstract description 5
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 5
- 238000001914 filtration Methods 0.000 description 4
- 238000012937 correction Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
- 230000033764 rhythmic process Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 238000005406 washing Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application discloses a data asset management method and a related device, comprising the following steps: acquiring a target asset, and preprocessing the target asset; judging whether preset asset management conditions are met or not according to the preprocessed target assets; if yes, matching the processing mode according to the target asset and obtaining a matching result; when the matching result is a type of management mode, acquiring a target data bin corresponding to the type of management mode; acquiring associated bin information according to the target data bin and establishing associated information of the target assets and the associated bin information; generating a target metadata unit in the target data bin according to the association information and the target assets; the timeliness of asset management is improved, and the technical effect of recording the blood relationship into the counting bin is achieved.
Description
Technical Field
The present application relates to the field of metadata processing, and in particular, to a data asset management method and related apparatus.
Background
With the gradual development of enterprises, the business scale of the enterprises is increased, the data management of the enterprises is more and more important, and the construction of enterprise-level data warehouses is indispensable. In the process of constructing the digital warehouse, how to keep consistent with the rhythm of developing and modeling is an important problem to be solved urgently in the development of the digital warehouse.
The existing data asset management systems put metadata into a warehouse in a manual input or semi-automatic mode, and the data warehousing mode is not only troublesome, but also poor in timeliness and even incapable of inputting the blood relationship of a meter into a warehouse.
Therefore, how to realize effectiveness and rationality in the process of managing data assets becomes a technical problem to be solved urgently.
Disclosure of Invention
In order to realize effectiveness and rationality in the process of managing the data assets, the application provides a data asset management method and a related device.
In a first aspect, the data asset management method provided by the present application adopts the following technical scheme:
a data asset management method, comprising:
acquiring target assets, and preprocessing the target assets;
judging whether preset asset management conditions are met or not according to the preprocessed target assets;
if yes, matching the processing mode according to the target asset and obtaining a matching result;
when the matching result is a type of management mode, acquiring a target data bin corresponding to the type of management mode;
acquiring associated bin information according to the target data bin and establishing associated information of the target assets and the associated bin information;
and generating a target metadata unit in the target data bin according to the association information and the target asset.
Optionally, the step of acquiring the target asset and preprocessing the target asset includes:
acquiring target assets, and performing data cleaning on the target assets;
classifying the target assets subjected to data cleaning according to a label classification rule;
and adjusting the weight information of the first label in the target asset according to the classification result.
Optionally, the step of adjusting the weight information of the first tag in the target asset according to the classification result includes:
matching in a preset strategy set according to the classification result to obtain a matching result;
generating a weight adjustment strategy according to the matching result;
and adjusting the weight information of the first label in the target asset according to the weight adjusting strategy.
Optionally, the step of determining whether the preset asset management condition is met according to the preprocessed target asset includes:
generating basic information of the target assets according to the preprocessed target assets;
acquiring preset asset management conditions;
determining an asset base requirement in the preset asset management condition;
and judging whether the target asset meets the preset asset management condition or not according to the asset basic requirement and the target asset basic information.
Optionally, after the step of determining whether the preset asset management condition is met according to the preprocessed target asset, the method further includes:
if not, determining non-compliant items according to the preset asset management conditions;
generating asset replenishment suggestions according to the non-compliant projects;
generating a feedback report in conjunction with the target asset based on the asset replenishment suggestion.
Optionally, the step of obtaining associated bin information according to the target data bin and establishing associated information between the target asset and the associated bin information includes:
acquiring associated bin information according to the position information of the target data bin;
determining a main data field according to stored data asset tag information in the associated bin information;
and establishing the association information of the target assets and the association bin information according to the main field.
Optionally, the step of generating a target metadata unit in the target data bin according to the association information and the target asset includes:
forming an index for the metadata by utilizing a solr technology in the target data bin according to the associated information and the target assets so as to provide metadata full-text retrieval capability, and retrieving associated data in the whole data bin according to keywords in a data map;
and generating a target metadata unit according to the index.
In a second aspect, the present application provides a geographic information system-based path planning apparatus, including:
the system comprises a preprocessing module, a storage module and a processing module, wherein the preprocessing module is used for acquiring target assets and preprocessing the target assets;
the condition judgment module is used for judging whether preset asset management conditions are met or not according to the preprocessed target assets;
the result acquisition module is used for matching the processing mode and acquiring a matching result according to the target asset if the target asset is matched with the processing mode;
the data bin acquisition module is used for acquiring a target data bin corresponding to a first type of management mode when the matching result is the first type of management mode;
the associated information acquisition module is used for acquiring associated bin information according to the target data bin and establishing associated information of the target assets and the associated bin information;
and the metadata generation module is used for generating a target metadata unit in the target data bin according to the association information and the target assets.
In a third aspect, the present application provides a computer apparatus, the apparatus comprising: a memory, a processor that, when executing computer instructions stored by the memory, performs a method as in any one of the above.
In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method as described above.
In summary, the present application includes the following beneficial technical effects:
the method comprises the steps of obtaining target assets, and preprocessing the target assets; judging whether preset asset management conditions are met or not according to the preprocessed target assets; if yes, matching the processing mode according to the target asset and obtaining a matching result; when the matching result is a type of management mode, acquiring a target data bin corresponding to the type of management mode; acquiring associated bin information according to the target data bin and establishing associated information of the target assets and the associated bin information; generating a target metadata unit in the target data bin according to the association information and the target assets; the timeliness of asset management is improved, and the technical effect of recording the blood relationship into the counting bin is achieved.
Drawings
FIG. 1 is a schematic diagram of a computer device architecture of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart diagram of a first embodiment of a data asset management method of the present invention;
FIG. 3 is an architecture diagram of metadata collection and synchronization for a first embodiment of the data asset management method of the present invention;
FIG. 4 is a schematic flow chart diagram of a second embodiment of a data asset management method of the present invention;
fig. 5 is a block diagram showing the construction of a first embodiment of the data asset management device of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a computer device in a hardware operating environment according to an embodiment of the present invention.
As shown in fig. 1, the computer device may include: a processor 1001, such as a Central Processing Unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement connection communication among these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a Wireless interface (e.g., a Wireless-Fidelity (Wi-Fi) interface). The Memory 1005 may be a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as a disk Memory. The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the configuration shown in FIG. 1 does not constitute a limitation of a computer device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a storage medium, may include therein an operating system, a network communication module, a user interface module, and a data asset management program.
In the computer device shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 of the computer device of the present invention may be provided in a computer device that calls the data asset management program stored in the memory 1005 through the processor 1001 and executes the data asset management method provided by the embodiment of the present invention.
An embodiment of the present invention provides a data asset management method, and referring to fig. 2, fig. 2 is a schematic flow chart of a first embodiment of the data asset management method of the present invention.
In this embodiment, the data asset management method includes the following steps:
step S10: and acquiring target assets, and preprocessing the target assets.
It should be noted that Metadata (also called intermediary data and relay data) is data describing data (data about data), and is mainly information describing data property (property), and is used to support functions such as indicating storage location, history data, resource search, file record, and the like. Metadata is an electronic catalog, and in order to achieve the goal of creating a catalog, the contents or features of data must be described and collected, so as to achieve the goal of assisting data retrieval. Dublin Core Metadata Initiative (DCMI) is an application of Metadata, and is a workshop sponsored by International library computer Center (OCLC) and National Center for Supercomputing Applications (NCSA) in 2 months 1995, and 52 requests from librarians and computer experts are created to jointly specify a set of features describing electronic files on the network. Metadata is information about the organization of data, data fields, and their relationships, and in short, metadata is data about data. The metadata is defined as: data describing the data, descriptive information for the data and information resources.
Metadata (Metadata) is data (data about other data) describing other data, or structural data (structured data) for providing information about a certain resource. Metadata is data that describes an object such as an information resource or data, and is used for the purpose of: identifying a resource; evaluating the resources; tracking changes of the resource in the using process; the method realizes simple and efficient management of a large amount of networked data; the information resource can be effectively found, searched, integrally organized and effectively managed. The basic characteristics of metadata are as follows:
a) Once the metadata is established, it can be shared. The structure and integrity of the metadata depend on the value and use environment of the information resources; the development and utilization environment of metadata is often a variable distributed environment; either format cannot fully meet the different needs of different groups;
b) Metadata is first a coding scheme. Metadata is a coding system used to describe digital information resources, especially network information resources, which leads to fundamental differences between metadata and traditional data coding systems; the most important feature and function of metadata is to build a machine understandable framework for digitized information resources.
The metadata system constructs a logical framework and a basic model of the e-government affairs, so that the functional characteristics, the operation mode and the overall performance of the system operation of the e-government affairs are determined. The operation of the e-government is implemented based on the metadata. It has the main functions as follows: description functions, integration functions, control functions, and agent functions.
Since metadata is also data, it can be stored and retrieved in a database in a data-like manner. The use of data elements will be made accurate and efficient if the organization of the data elements is provided while the metadata describing the data elements is provided. The user may first view his metadata when using the data in order to be able to obtain the information he wants.
It is understood that in the field of data warehouses, metadata is divided by use into technical metadata and business metadata. First, the metadata can provide user-based information, and the metadata, such as business description information of the recorded data items, can help the user to use the data. Second, metadata can support the management and maintenance of data by the system, e.g., metadata about the data item storage method can support the system in accessing data in the most efficient manner. Specifically, in a data warehouse system, a metadata mechanism mainly supports the following five types of system management functions:
(1) Describing which data is in the data warehouse;
(2) Defining data to be entered into and data to be generated from the data warehouse;
(3) Recording the data extraction working time arrangement carried out along with the occurrence of the business event;
(4) Recording and detecting the requirement and the execution condition of the system data consistency;
(5) And measuring the data quality.
In a specific implementation, the metadata collection and synchronization architecture diagram of the present embodiment is shown in fig. 3.
Firstly, building dependent technical components such as Atlas kafka hbase mysql solr, anger and the like, and getting through the connectivity among the components; configuring metadata acquisition plug-in atlas plug at Hive and HBase ends, and recording the water level acquired each time; the collected technical metadata is combined into a unique component by an example, a library and a table of the metadata, and a complete data warehouse metadata unit is formed by combining the service metadata; and forming the metadata into an index by utilizing a solr technology, providing metadata full-text retrieval capability, and retrieving the associated data in the whole data bin according to keywords in the data map.
It should be noted that, after a user generates a real table according to a service logic in a development, test, and production environment, the system can automatically synchronize table metadata information to the data warehouse platform, and then a data administrator can query the service metadata and technical metadata information of the table in the data warehouse platform in time, such as: the incidence relation between tables, business or technical responsible person, field and type, belonging subject domain and number storehouse level information.
In a specific implementation, one of the methods of this embodiment includes:
1. and (3) installing atlas plugin (the plugin can acquire component metadata change) in the service of Hive, hbase and the like, synchronizing the metadata acquired by the plugin to kafka by the atlas server, and synchronously storing the kafka data in real time.
2. The metadata is put in a database as technical metadata (including the relationship between tables), and the metadata needs to be classified, labeled and the like in a plurality of bins, and a set of service metadata needs to be maintained.
3. And combining the technical metadata and the service metadata through the metadata primary key name to form complete asset metadata information.
Step S20: and judging whether preset asset management conditions are met or not according to the preprocessed target assets.
Further, in order to accurately determine whether the preset asset management condition is met, the step of determining whether the preset asset management condition is met according to the preprocessed target asset includes: generating basic information of the target assets according to the preprocessed 7hyup target assets; acquiring preset asset management conditions; determining an asset base requirement in the preset asset management condition; and judging whether the target asset meets the preset asset management condition or not according to the asset basic requirement and the target asset basic information.
Further, in order to provide a feedback report when the preset asset management condition is not met, after the step of determining whether the preset asset management condition is met according to the preprocessed target asset, the method further includes: if not, determining non-compliant items according to the preset asset management conditions; generating an asset replenishment suggestion according to the non-compliance project; generating a feedback report in conjunction with the target asset based on the asset replenishment suggestion.
Step S30: and if so, matching the processing mode according to the target asset and obtaining a matching result.
Step S40: and when the matching result is in the first class of management mode, acquiring the target data bin corresponding to the first class of management mode.
It should be noted that, one type of management method in this embodiment refers to that the target asset can be stored in an existing data warehouse, and the corresponding two types of management methods need to newly create a data warehouse according to the target asset.
Step S50: and acquiring the associated bin information according to the target data bin and establishing the associated information of the target assets and the associated bin information.
Further, in order to improve the integrity of the establishment of the associated information, the step of obtaining the associated bin information according to the target data bin and establishing the associated information between the target asset and the associated bin information includes: acquiring associated bin information according to the position information of the target data bin; determining a main data field according to stored data asset tag information in the associated bin information; and establishing the association information of the target assets and the association bin information according to the main field.
It should be noted that, firstly, the label needs to have data thereon, so that the label can be called a data label. The data label is a management of the data, can perform deep analysis according to the data, refine the data and better improve the working efficiency and the method. A tag alone is a form of data that is used to characterize a business entity. The two are combined to use, so that the effect is better.
Step S60: and generating a target metadata unit in the target data bin according to the association information and the target asset.
Further, in order to improve the rationality of target data unit generation, the step of generating a target metadata unit in the target data bin according to the association information and the target asset includes: forming an index for the metadata by utilizing a solr technology in the target data bin according to the associated information and the target assets so as to provide metadata full-text retrieval capability, and retrieving associated data in the whole data bin according to keywords in a data map; and generating a target metadata unit according to the index.
The embodiment acquires a target asset and preprocesses the target asset; judging whether preset asset management conditions are met or not according to the preprocessed target assets; if yes, matching the processing mode according to the target asset and obtaining a matching result; when the matching result is a type of management mode, acquiring a target data bin corresponding to the type of management mode; acquiring associated bin information according to the target data bin and establishing associated information of the target assets and the associated bin information; generating a target metadata unit in the target data bin according to the association information and the target assets; the timeliness of asset management is improved, and the technical effect of recording the blood relationship into a counting bin is achieved.
Referring to fig. 4, fig. 4 is a flowchart illustrating a data asset management method according to a second embodiment of the present invention.
Based on the first embodiment, the step S10 of the data asset management method of this embodiment further includes:
step S101: and acquiring the target asset, and performing data cleaning on the target asset.
It should be noted that, data cleansing (Data cleansing), a process of reviewing and verifying Data, is intended to delete duplicate information, correct existing errors, and provide Data consistency. Data cleansing also looks by name to "wash out" dirty, meaning the last procedure to find and correct recognizable errors in a data file, including checking data consistency, handling invalid and missing values, etc. Because the data in the data warehouse is a collection of data oriented to a certain subject, the data is extracted from a plurality of business systems and contains historical data, so that the problems that some data are wrong and some data conflict with each other are avoided, and the wrong or conflicting data are obviously unwanted and are called as 'dirty data' are avoided. We need to "wash" dirty data according to certain rules, which is data washing. The task of data cleaning is to filter the data which do not meet the requirements, and the filtered result is sent to a business administration department to confirm whether the data are filtered or corrected by a business unit and then extracted. The data which is not qualified is mainly three categories of incomplete data, error data and repeated data. Data cleaning is different from questionnaire examination, and data cleaning after recording is generally completed by a computer instead of a human
It can be understood that the preprocessing targets aimed at by the present embodiment mainly include: incomplete data: this kind of data is mainly the information missing that should be, such as the name of the supplier, the name of the branch company, the regional information missing of the customer, the unmatched main and detail tables in the business system, etc. And filtering the data, respectively writing different Excel files according to the missing content, submitting the Excel files to the client, and requiring completion within the specified time. And writing the data into a data warehouse after completion. Error data: the reason for this kind of error is that the service system is not sound enough, and it is not judged after receiving the input and directly written into the background database, for example, the numerical data is input into full-angle numerical characters, there is a carriage return operation behind the character string data, the date format is incorrect, the date is out of bounds, etc. The data is also classified, and for the problem that characters similar to full-angle characters and invisible characters exist before and after the data, the data can be found only by writing SQL sentences, and then a client is required to extract the data after the business system is corrected. Errors such as incorrect date format or date out-of-bounds errors can cause ETL operation failure, and the errors need to be picked out by a business system database in an SQL mode, are submitted to a business administration department to require correction in a limited period, and are extracted after correction. Repeating data: for this type of data, particularly those that occur in dimension tables, all fields of the duplicate data records are exported for validation and collation by the client. Data cleaning is a repeated process which cannot be completed within a few days, and only problems are continuously found and solved. Whether filtering is performed or not, whether correction is performed or not generally requires customer confirmation, filtered data is written into an Excel file or a data table, and mails for filtering data can be sent to business units every day in the initial stage of ETL development to prompt the business units to correct errors as soon as possible, and meanwhile, the filtered data can also be used as a basis for verifying data in the future. Data cleansing requires care not to filter out useful data, to verify carefully for each filtering rule, and to confirm by the user.
Step S102; and classifying the target assets subjected to data cleaning according to the label classification rule.
It is understood that the label classification rule is preset by a system administrator according to actual use requirements, for example: according to the asset industry type in the asset management requirement, the following can be set: financial assets, internet entertainment assets, and industrial production assets, among others.
Step S103: and adjusting the weight information of the first label in the target asset according to the classification result.
Further, in order to increase the actual weight of the tag in the target asset, the step of adjusting the weight information of the first tag in the target asset according to the classification result includes: matching in a preset strategy set according to the classification result to obtain a matching result; generating a weight adjustment strategy according to the matching result; and adjusting the weight information of the first label in the target asset according to the weight adjusting strategy.
The embodiment acquires target assets and performs data cleaning on the target assets; classifying the target assets subjected to data cleaning according to a label classification rule; adjusting the weight information of a first label in the target asset according to the classification result; the weight information of the first label in the target asset is readjusted through preprocessing, and the accuracy of data asset management is improved.
Furthermore, an embodiment of the present invention further provides a computer readable storage medium, where a program for data asset management is stored, and the program for data asset management when executed by a processor implements the steps of the method for data asset management as described above.
Referring to fig. 5, fig. 5 is a block diagram showing a first embodiment of the data asset management device according to the present invention.
As shown in fig. 5, the data asset management apparatus according to the embodiment of the present invention includes:
the system comprises a preprocessing module 10, a storage module and a processing module, wherein the preprocessing module is used for acquiring target assets and preprocessing the target assets;
a condition judgment module 20, configured to judge whether a preset asset management condition is met according to the preprocessed target asset;
a result obtaining module 30, configured to, if yes, obtain a matching result according to the target asset matching processing manner;
the data bin obtaining module 40 is configured to obtain a target data bin corresponding to a first type of management mode when the matching result is the first type of management mode;
the associated information acquisition module 50 is configured to acquire associated bin information according to the target data bin and establish associated information between the target asset and the associated bin information;
and a metadata generating module 60, configured to generate a target metadata unit in the target data bin according to the association information and the target asset.
It should be understood that the above is only an example, and the technical solution of the present invention is not limited in any way, and in a specific application, a person skilled in the art may set the technical solution as needed, and the present invention is not limited thereto.
The embodiment acquires a target asset and preprocesses the target asset; judging whether preset asset management conditions are met or not according to the preprocessed target assets; if yes, matching the processing mode according to the target asset and obtaining a matching result; when the matching result is a type of management mode, acquiring a target data bin corresponding to the type of management mode; acquiring associated bin information according to the target data bin and establishing associated information of the target assets and the associated bin information; generating a target metadata unit in the target data bin according to the association information and the target assets; the timeliness of asset management is improved, and the technical effect of recording the blood relationship into the counting bin is achieved.
In an embodiment, the preprocessing module 10 is further configured to acquire a target asset, and perform data cleansing on the target asset; classifying the target assets subjected to data cleaning according to a label classification rule; and adjusting the weight information of the first label in the target asset according to the classification result.
In an embodiment, the preprocessing module 10 is further configured to perform matching in a preset policy set according to the classification result to obtain a matching result; generating a weight adjustment strategy according to the matching result; and adjusting the weight information of the first label in the target asset according to the weight adjusting strategy.
In an embodiment, the condition determining module 20 is further configured to generate basic information of the target asset according to the preprocessed target asset; acquiring preset asset management conditions; determining asset base requirements in the preset asset management conditions; and judging whether the target asset meets the preset asset management condition or not according to the asset basic requirement and the target asset basic information.
In an embodiment, the condition determining module 20 is further configured to determine, if the condition is not met, an out-of-compliance item according to the preset asset management condition; generating an asset replenishment suggestion according to the non-compliance project; generating a feedback report in conjunction with the target asset based on the asset replenishment suggestion.
In an embodiment, the associated information obtaining module 50 is further configured to obtain associated bin information according to the position information of the target data bin; determining a main data field according to stored data asset tag information in the associated bin information; and establishing the association information of the target assets and the association bin information according to the main field.
In an embodiment, the metadata generating module 60 is further configured to index metadata in the target data bin by using a solr technique according to the association information and the target asset to provide a full-text metadata retrieval capability, and retrieve the association data in the whole data bin according to a keyword in a data map; and generating a target metadata unit according to the index.
It should be noted that the above-described work flows are only exemplary, and do not limit the scope of the present invention, and in practical applications, a person skilled in the art may select some or all of them to achieve the purpose of the solution of the embodiment according to actual needs, and the present invention is not limited herein.
In addition, the technical details that are not described in detail in this embodiment may refer to the method for managing data assets provided by any embodiment of the present invention, and are not described herein again.
Further, it is to be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention or portions thereof that contribute to the prior art may be embodied in the form of a software product, where the computer software product is stored in a storage medium (e.g. Read Only Memory (ROM)/RAM, magnetic disk, optical disk), and includes several instructions for enabling a terminal device (e.g. a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all equivalent structures or equivalent processes performed by the present invention or directly or indirectly applied to other related technical fields are also included in the scope of the present invention.
Claims (10)
1. A method for data asset management, comprising:
acquiring target assets, and preprocessing the target assets;
judging whether preset asset management conditions are met or not according to the preprocessed target assets;
if yes, matching the processing mode according to the target asset and obtaining a matching result;
when the matching result is a type of management mode, acquiring a target data bin corresponding to the type of management mode;
acquiring associated bin information according to the target data bin and establishing associated information of the target assets and the associated bin information;
and generating a target metadata unit in the target data bin according to the association information and the target asset.
2. The data asset management method according to claim 1, wherein said step of obtaining a target asset and preprocessing said target asset comprises:
acquiring target assets, and performing data cleaning on the target assets;
classifying the target assets subjected to data cleaning according to a label classification rule;
and adjusting the weight information of the first label in the target asset according to the classification result.
3. The data asset management method according to claim 2, wherein said step of adjusting the weight information of the first tag in the target asset according to the classification result comprises:
matching in a preset strategy set according to the classification result to obtain a matching result;
generating a weight adjustment strategy according to the matching result;
and adjusting the weight information of the first label in the target asset according to the weight adjusting strategy.
4. The data asset management method according to claim 1, wherein said step of determining whether a preset asset management condition is met according to the preprocessed target asset comprises:
generating basic information of the target assets according to the preprocessed target assets;
acquiring preset asset management conditions;
determining asset base requirements in the preset asset management conditions;
and judging whether the target asset meets the preset asset management condition or not according to the asset basic requirement and the target asset basic information.
5. The data asset management method according to claim 1, wherein after the step of determining whether the pre-processed target asset meets the preset asset management condition, the method further comprises:
if not, determining non-compliance projects according to the preset asset management conditions;
generating asset replenishment suggestions according to the non-compliant projects;
generating a feedback report in conjunction with the target asset based on the asset replenishment suggestion.
6. The method of claim 1, wherein the step of obtaining associated bin information from the target data bin and establishing association information between the target asset and the associated bin information comprises:
acquiring associated bin information according to the position information of the target data bin;
determining a main data field according to stored data asset tag information in the associated bin information;
and establishing the association information of the target assets and the association bin information according to the main field.
7. The method of claim 1, wherein the step of generating a target metadata unit in the target data warehouse from the association information and the target asset comprises:
forming an index for the metadata by utilizing a solr technology in the target data bin according to the associated information and the target assets so as to provide metadata full-text retrieval capability, and retrieving associated data in the whole data bin according to keywords in a data map;
and generating a target metadata unit according to the index.
8. A data asset management device, characterized in that the data asset management device comprises:
the system comprises a preprocessing module, a storage module and a processing module, wherein the preprocessing module is used for acquiring target assets and preprocessing the target assets;
the condition judgment module is used for judging whether preset asset management conditions are met or not according to the preprocessed target assets;
the result acquisition module is used for matching the processing mode and acquiring a matching result according to the target asset if the target asset is matched with the processing mode;
the data bin acquisition module is used for acquiring a target data bin corresponding to a first type of management mode when the matching result is the first type of management mode;
the associated information acquisition module is used for acquiring associated bin information according to the target data bin and establishing associated information of the target asset and the associated bin information;
and the metadata generation module is used for generating a target metadata unit in the target data bin according to the association information and the target assets.
9. A computer device, the device comprising: a memory, a processor that, when executing computer instructions stored by the memory, performs the method of any of claims 1-7.
10. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211732906.3A CN115934864A (en) | 2022-12-30 | 2022-12-30 | Data asset management method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211732906.3A CN115934864A (en) | 2022-12-30 | 2022-12-30 | Data asset management method and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115934864A true CN115934864A (en) | 2023-04-07 |
Family
ID=86700799
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211732906.3A Withdrawn CN115934864A (en) | 2022-12-30 | 2022-12-30 | Data asset management method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115934864A (en) |
-
2022
- 2022-12-30 CN CN202211732906.3A patent/CN115934864A/en not_active Withdrawn
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11281626B2 (en) | Systems and methods for management of data platforms | |
| US11360950B2 (en) | System for analysing data relationships to support data query execution | |
| US20210011891A1 (en) | System for importing data into a data repository | |
| US10678810B2 (en) | System for data management in a large scale data repository | |
| US10296658B2 (en) | Use of context-dependent statistics to suggest next steps while exploring a dataset | |
| CA2701046C (en) | Analysis of a system for matching data records | |
| JP4580518B2 (en) | Database design system | |
| US20150356123A1 (en) | Systems and methods for management of data platforms | |
| US20070162482A1 (en) | Method and system of using artifacts to identify elements of a component business model | |
| US20090172024A1 (en) | Systems and methods for collecting and analyzing business intelligence data | |
| CN116097241A (en) | Data preparation using semantic roles | |
| Nikiforova | Open Data Quality Evaluation: A comparative analysis of open data in Latvia | |
| JP6623754B2 (en) | Tabular data processing program, method and apparatus | |
| Gui et al. | IFC-based partial data model retrieval for distributed collaborative design | |
| JP6375029B2 (en) | A metadata-based online analytical processing system that analyzes the importance of reports | |
| Bicevska et al. | Models of data quality | |
| CN112631889A (en) | Portrayal method, device and equipment for application system and readable storage medium | |
| CN115982429B (en) | Knowledge management method and system based on flow control | |
| CN109636303B (en) | Storage method and system for semi-automatically extracting and structuring document information | |
| EP3152678B1 (en) | Systems and methods for management of data platforms | |
| US11580479B2 (en) | Master network techniques for a digital duplicate | |
| CN115934864A (en) | Data asset management method and related device | |
| US20140149186A1 (en) | Method and system of using artifacts to identify elements of a component business model | |
| CN114331679A (en) | Data problem analysis method and related device | |
| EP3652669A1 (en) | Systems and methods for compiling a database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 518000 2201, block D, building 1, Chuangzhi Yuncheng bid section 1, Liuxian Avenue, Xili community, Xili street, Nanshan District, Shenzhen City, Guangdong Province Applicant after: Qizhi Technology Co.,Ltd. Address before: 518000 2201, block D, building 1, Chuangzhi Yuncheng bid section 1, Liuxian Avenue, Xili community, Xili street, Nanshan District, Shenzhen City, Guangdong Province Applicant before: Qizhi Network Technology Co.,Ltd. |
|
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20230407 |