CN111813842B - Data processing method, device, system, equipment and storage medium - Google Patents
Data processing method, device, system, equipment and storage medium Download PDFInfo
- Publication number
- CN111813842B CN111813842B CN202010944093.9A CN202010944093A CN111813842B CN 111813842 B CN111813842 B CN 111813842B CN 202010944093 A CN202010944093 A CN 202010944093A CN 111813842 B CN111813842 B CN 111813842B
- Authority
- CN
- China
- Prior art keywords
- data
- target
- database
- target data
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention belongs to the technical field of data management, and in particular, to a data processing method, apparatus, system, device, and storage medium. The data processing method comprises the steps of obtaining target data of a target database; calling a preset intelligent analysis algorithm engine according to a preset scheduling task to process the target data and generate a data image; and performing data catalog arrangement on the target data according to the data image to generate a target data catalog. The data processing method provided by the embodiment of the invention automatically and intelligently analyzes the data of the database to obtain the data portrait, and arranges the data to form the data catalog. The whole process has less manual intervention components, high efficiency and high quality of the generated directory, is convenient for government departments to know the core services of all parties more accurately, carries out data sharing in a targeted manner and develops more extensive cooperation and collaboration.
Description
Technical Field
The present invention belongs to the technical field of data management, and in particular, to a data processing method, apparatus, system, device, and storage medium.
Background
In the process of digital transformation of the government, government affair data resources are integrated, the 'data base' is found out, the data resources are reasonably developed and utilized, the data sharing capacity among government departments is improved, the measures can powerfully support the cooperation of different business lines among the government departments, support the work of government data asset formation and listing, and meet the requirements of comprehensive sharing, development and use of the data resources.
However, information barriers exist among government departments at present, the investment of manpower is insufficient, and the number of government affair data resource servings and cataloguing is low, so that the data resource sharing work among the government departments is difficult to widely develop and cooperate. At present, in order to solve the above problems, the following three methods are generally adopted: 1. manually reporting data resources mastered by government departments in a directory template mode, and cataloging and warehousing the data resources by a summarizing department to form a data resource directory; 2. filling data resources mastered by government departments in a system form mode, and finally cataloguing and warehousing to form a data resource catalog; 3. and acquiring metadata information of the database as cataloging initialization data resources, supplementing and filling other attribute items meeting the catalog standard, and finally warehousing to form a data resource catalog.
Although the three methods carry out inventory and cataloging management on government affair data resources, the investment is large, the manual intervention mode is easy to make mistakes, and the quantity and the quality of the formed data resource catalog are not high.
Disclosure of Invention
Embodiments of the present invention provide a data processing method, apparatus, system, device, and storage medium, which are used to solve the problems in the prior art that the government affair data resource arrangement investment is large, the manual intervention mode is prone to errors, and the number and quality of the formed data resource catalog are not high.
The embodiment of the invention is realized in such a way that a data processing method comprises the following steps:
acquiring target data of a target database;
calling a preset intelligent analysis algorithm engine according to a preset scheduling task to process the target data and generate a data image;
and performing data catalog arrangement on the target data according to the data image to generate a target data catalog.
Preferably, before the obtaining the target data of the target database, the method further includes:
and performing connection configuration with the target database, and testing whether the database connection is communicated.
Preferably, after the connecting configuration with the target database and testing whether the database connection is connected, the method further includes:
and configuring the data acquisition authority of the target database.
Preferably, the step of processing the target data by calling a preset intelligent analysis algorithm engine according to a preset scheduling task to generate a data image specifically includes:
analyzing the database type of the target data, and executing a corresponding scheduling task according to the database type of the target data to call a corresponding preset intelligent analysis algorithm engine;
analyzing the target data according to a preset data evaluation index through the intelligent analysis algorithm engine, and determining an index attribute of the target data, wherein the index attribute comprises but is not limited to one or a combination of more of data effectiveness, data integrity and data activity;
and screening the target data according to the index attribute of the target data to determine core target data.
Preferably, the step of analyzing the target data according to a preset data evaluation index by the intelligent analysis algorithm engine to determine an index attribute of the target data further includes:
and carrying out identification classification on the data in the target data according to the index attribute of the target data.
Preferably, the step of performing data catalog layout on the target data according to the data image to generate a target data catalog further includes:
and sending the target data directory to a user side, receiving directory modification information returned by the user side, and adjusting the target data directory according to the directory modification information.
Another object of an embodiment of the present invention is to provide a data processing apparatus, including:
the data acquisition module is used for acquiring target data of a target database, wherein the target data comprises but is not limited to database metadata information and database log information;
the data processing module is used for calling a preset intelligent analysis algorithm engine according to a preset scheduling task to process the target data and generate a data image;
and the catalog generation module is used for arranging the data catalog of the target data according to the data image to generate a target data catalog.
Another object of an embodiment of the present invention is to provide a data processing system, including:
the government affair data platform is used for storing government affair data information;
and the data processing center comprises the data processing device and is used for acquiring target data from the government affair data platform, executing the steps of the data processing method to process the target data and generating a target data directory.
It is a further object of embodiments of the present invention to provide a computer device, comprising a memory and a processor, the memory having stored therein a computer program, which, when executed by the processor, causes the processor to perform the steps of the data processing method.
It is another object of an embodiment of the present invention to also provide a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, causes the processor to execute the steps of the data processing method.
The data processing method provided by the embodiment of the invention automatically and intelligently analyzes the data of the database to obtain the data portrait, then judges the data related to the core business of the department, and arranges the data to form the data catalog. The whole process has less manual intervention components, high efficiency and high quality of the generated catalog. Meanwhile, based on a high-quality data directory, government departments can more accurately know core services of all parties, and data sharing is performed in a targeted manner by combining the service characteristics of the government departments, so that a service line is opened, and more extensive cooperation and collaboration are developed.
Drawings
FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of generating a data representation according to an embodiment of the present invention;
FIG. 3 is a flow chart of another data processing method according to an embodiment of the present invention;
FIG. 4 is a flow chart of another data processing method according to an embodiment of the present invention;
FIG. 5 is a block diagram of a data processing apparatus according to an embodiment of the present invention;
FIG. 6 is a block diagram of a data processing system, according to an embodiment;
FIG. 7 is a block diagram showing an internal configuration of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
It will be understood that, as used herein, the terms "first," "second," and the like may be used herein to describe various elements, but these elements are not limited by these terms unless otherwise specified. These terms are only used to distinguish one element from another. For example, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script, without departing from the scope of the present application.
The data processing method in the embodiment of the invention can be applied to the environment comprising the terminal and the computer equipment. The computer device may be an independent physical server or terminal, may also be a server cluster formed by a plurality of physical servers, and may be a cloud server providing basic cloud computing services such as a cloud server, a cloud database, a cloud storage, a CDN, and the like; the terminal may be, but is not limited to, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and the like.
Example one
As shown in fig. 1, in the embodiment of the present invention, a data processing method is provided, and the embodiment is mainly illustrated by applying the method to a terminal. A data processing method, which may specifically include the steps of:
step S110, acquiring target data of a target database, wherein the target data comprises but is not limited to database metadata information and database log information;
step S120, a preset intelligent analysis algorithm engine is called according to a preset scheduling task to process target data, and a data image is generated;
step S130, data catalog arrangement is carried out on the target data according to the data portrait, and a target data catalog is generated.
In the embodiment of the invention, the target database can support various commonly used database types such as mysql, oracle, sqlserver, rds and the like, and various heterogeneous data corresponding to various database types in the embodiment of the invention can also execute corresponding scheduling tasks according to the corresponding database types to call corresponding intelligent analysis algorithms for analysis, so that the requirement of uniformly processing various heterogeneous data can be met, and the flexibility is high.
In the embodiment of the present invention, the database Metadata information refers to Metadata (also called intermediary data or relay data), which is data describing data (data about data), and is mainly information describing data attribute (property), and is used to support functions such as indicating storage location, history data, resource search, file record, and the like. Metadata is an electronic catalog, and in order to achieve the purpose of creating a catalog, the contents or features of data must be described and collected, so as to achieve the purpose of assisting data retrieval. The database log information is log information for recording data processing changes of the database, and those skilled in the art can understand the data processing changes according to the above description, and will not be described and exemplified in more detail here.
In the embodiment of the present invention, the preset scheduling task refers to presetting an intelligent analysis algorithm engine for analyzing data of different data structure types, after acquiring the target data, analyzing and determining the structure type of the target data, and then executing a corresponding scheduling task according to the structure type of the target data to call the corresponding intelligent analysis algorithm engine to analyze and process the target data.
In the embodiment of the invention, the preset intelligent analysis algorithm engine is mainly used for accurately reflecting the core degree of data and reflecting which tables are temporary tables, zombie tables and production tables by analyzing evaluation indexes of 10-dimension data of medium database metadata information integrity, data heat, data importance and data activity of target data. The whole process has less manual intervention, the intelligent analysis algorithm engine is initialized in the system and is executed regularly, and the execution result is automatically pushed to the system.
In the embodiment of the present invention, as shown in fig. 2, which is a flowchart of generating a data image in the embodiment of the present invention, step S120 calls a preset intelligent analysis algorithm engine according to a preset scheduling task to process target data, so as to generate the data image, and specifically includes:
step S210, analyzing the database type of the target data, executing a corresponding scheduling task according to the database type of the target data, and calling a corresponding preset intelligent analysis algorithm engine;
step S220, analyzing the target data according to a preset data evaluation index through an intelligent analysis algorithm engine, and determining the index attribute of the target data, wherein the index attribute comprises but is not limited to one or a combination of more of data effectiveness, data integrity and data activity;
step S230, screening the target data according to the index attribute of the target data, and determining core target data.
In the embodiment of the invention, the result of the intelligent analysis on the target data carries index attributes such as effective scores and integrity of the data, namely the data image, and meanwhile, the pushed back data excludes data resources such as temporary tables, zombie tables, dimension tables and the like, namely the pushed back data resources are the data resources after screening, and meanwhile, the data resources are scored and identified in the data image, and the higher the effective score carried by the data resources is, the more important the core degree of the data is. Therefore, the target data after analysis can be catalogued, sorted and filled, and the catalog arrangement of data resources is completed.
In the embodiment of the present invention, step S220 is to analyze the target data according to the preset data evaluation index through the intelligent analysis algorithm engine, and determine the index attribute of the target data, further including:
and identifying and classifying the data in the target data according to the index attribute of the target data.
Specifically, for example, in the process of processing the government affair data, according to the data after intelligent analysis processing, the service type or specific department to which the data belongs is identified, or the confidentiality degree and the importance degree of the data are identified and classified, which is also convenient for the arrangement and management of the government affair data.
In the embodiment of the present invention, as shown in fig. 3, another data processing method provided in the embodiment of the present invention is different from that in fig. 1, before acquiring target data of a target database, the method further includes:
step S310, connecting and configuring with a target database, and testing whether the database connection is connected.
In the embodiment of the invention, different database connection modes can be selected according to different database types with different structures for connection configuration with the database, the connection configuration of the database belongs to a common technical means in the field, the specific operation and process are not further described, and the test after connection can also be performed by selecting different test methods according to the corresponding database types.
In the embodiment of the present invention, as shown in fig. 4, another data processing method provided in the embodiment of the present invention is different from that in fig. 3 in that after performing step S310 to perform connection configuration with a target database and testing whether database connections are connected, the method further includes:
step S410, configuring the data acquisition authority of the target database.
In the embodiment of the present invention, a target data portion of a database may have a confidential status, for example, data of some departments in government affairs data is not public, and at this time, a setting of a data acquisition authority of the database is required, which is mainly to determine whether the database can be authorized to acquire data by configuring the data acquisition authority. Correspondingly, in the execution process, the configured corresponding verification information may be sent to the target database, and the target database performs verification processing on the configured corresponding verification information after receiving verification, and then allows to acquire data. The implementation of more specific data access rights is not further illustrated and described here, and can be simply designed by a person skilled in the art according to the actual mood.
In the embodiment of the present invention, step S130 performs data catalog layout on the target data according to the data representation to generate a target data catalog, further including:
and sending the target data directory to the user side, receiving directory modification information returned by the user side, and adjusting the target data directory according to the directory modification information.
In the embodiment of the invention, the target data directory is sent to the user side for verification or audit, and then further improvement is carried out on the basis, so that a user can conveniently adjust the data directory according to actual conditions or requirements, and the flexibility is improved.
The data processing method provided by the embodiment of the invention automatically and intelligently analyzes the data of the database to obtain the data portrait, then judges the data related to the core business of the department, and arranges the data to form the data catalog. The whole process has less manual intervention components, high efficiency and high quality of the generated catalog. Meanwhile, based on a high-quality data directory, government departments can more accurately know core services of all parties, and data sharing is performed in a targeted manner by combining the service characteristics of the government departments, so that a service line is opened, and more extensive cooperation and collaboration are developed.
Example two
As shown in fig. 5, in the embodiment of the present invention, a data processing apparatus is provided, which may be integrated in the terminal or the computer device, and specifically includes:
a data obtaining module 510, configured to obtain target data of a target database, where the target data includes, but is not limited to, database metadata information and database log information;
the data processing module 520 is configured to invoke a preset intelligent analysis algorithm engine to process the target data according to a preset scheduling task, and generate a data image;
and a catalog generation module 530 for arranging the data catalog of the target data according to the data image to generate a target data catalog.
In the embodiment of the invention, the target database can support various commonly used database types such as mysql, oracle, sqlserver, rds and the like, and various heterogeneous data corresponding to various database types in the embodiment of the invention can also execute corresponding scheduling tasks according to the corresponding database types to call corresponding intelligent analysis algorithms for analysis, so that the requirement of uniformly processing various heterogeneous data can be met, and the flexibility is high.
In the embodiment of the present invention, the database Metadata information refers to Metadata (also called intermediary data or relay data), which is data describing data (data about data), and is mainly information describing data attribute (property), and is used to support functions such as indicating storage location, history data, resource search, file record, and the like. Metadata is an electronic catalog, and in order to achieve the purpose of creating a catalog, the contents or features of data must be described and collected, so as to achieve the purpose of assisting data retrieval. The database log information is log information for recording data processing changes of the database, and those skilled in the art can understand the data processing changes according to the above description, and will not be described and exemplified in more detail here.
In the embodiment of the present invention, the preset scheduling task refers to presetting an intelligent analysis algorithm engine for analyzing data of different data structure types, after acquiring the target data, analyzing and determining the structure type of the target data, and then executing a corresponding scheduling task according to the structure type of the target data to call the corresponding intelligent analysis algorithm engine to analyze and process the target data.
In the embodiment of the invention, the preset intelligent analysis algorithm engine is mainly used for accurately reflecting the core degree of data and reflecting which tables are temporary tables, zombie tables and production tables by analyzing evaluation indexes of 10-dimension data of medium database metadata information integrity, data heat, data importance and data activity of target data. The whole process has less manual intervention, the intelligent analysis algorithm engine is initialized in the system and is executed regularly, and the execution result is automatically pushed to the system.
In this embodiment of the present invention, the data processing module 520 invokes a preset intelligent analysis algorithm engine to process the target data according to a preset scheduling task, and generates a data image, which specifically includes:
analyzing the database type of the target data, executing a corresponding scheduling task according to the database type of the target data, and calling a corresponding preset intelligent analysis algorithm engine;
analyzing target data according to preset data evaluation indexes through an intelligent analysis algorithm engine, and determining index attributes of the target data, wherein the index attributes include but are not limited to one or a combination of more of data effectiveness, data integrity and data activity;
and screening the target data according to the index attribute of the target data to determine core target data.
In the embodiment of the invention, the result of the intelligent analysis on the target data carries index attributes such as effective scores and integrity of the data, namely the data image, and meanwhile, the pushed back data excludes data resources such as temporary tables, zombie tables, dimension tables and the like, namely the pushed back data resources are the data resources after screening, and meanwhile, the data resources are scored and identified in the data image, and the higher the effective score carried by the data resources is, the more important the core degree of the data is. Therefore, the target data after analysis can be catalogued, sorted and filled, and the catalog arrangement of data resources is completed.
In this embodiment of the present invention, the data processing module 520 analyzes the target data according to the preset data evaluation index through the intelligent analysis algorithm engine, and determines the index attribute of the target data, further including:
and identifying and classifying the data in the target data according to the index attribute of the target data.
Specifically, for example, in the process of processing the government affair data, according to the data after intelligent analysis processing, the service type or specific department to which the data belongs is identified, or the confidentiality degree and the importance degree of the data are identified and classified, which is also convenient for the arrangement and management of the government affair data.
In this embodiment of the present invention, before the data obtaining module 510 obtains the target data of the target database, the method further includes:
and performing connection configuration with the target database, and testing whether the database connection is communicated.
In the embodiment of the invention, different database connection modes can be selected according to different database types with different structures for connection configuration with the database, the connection configuration of the database belongs to a common technical means in the field, the specific operation and process are not further described, and the test after connection can also be performed by selecting different test methods according to the corresponding database types.
In this embodiment of the present invention, after the data obtaining module 510 performs connection configuration with the target database, and tests whether the database connection is connected, the method further includes:
and configuring the data acquisition authority of the target database.
In the embodiment of the present invention, a target data portion of a database may have a confidential status, for example, data of some departments in government affairs data is not public, and at this time, a setting of a data acquisition authority of the database is required, which is mainly to determine whether the database can be authorized to acquire data by configuring the data acquisition authority. Correspondingly, in the execution process, the configured corresponding verification information may be sent to the target database, and the target database performs verification processing on the configured corresponding verification information after receiving verification, and then allows to acquire data. The implementation of more specific data access rights is not further illustrated and described here, and can be simply designed by a person skilled in the art according to the actual mood.
In this embodiment of the present invention, the catalog generating module 530 performs data catalog layout on the target data according to the data representation to generate a target data catalog, further comprising:
and sending the target data directory to the user side, receiving directory modification information returned by the user side, and adjusting the target data directory according to the directory modification information.
In the embodiment of the invention, the target data directory is sent to the user side for verification or audit, and then further improvement is carried out on the basis, so that a user can conveniently adjust the data directory according to actual conditions or requirements, and the flexibility is improved.
The data processing device provided by the embodiment of the invention automatically and intelligently analyzes the data of the database to obtain the data portrait, then judges the data related to the core business of the department, and arranges the data to form the data catalog. The whole process has less manual intervention components, high efficiency and high quality of the generated catalog. Meanwhile, based on a high-quality data directory, government departments can more accurately know core services of all parties, and data sharing is performed in a targeted manner by combining the service characteristics of the government departments, so that a service line is opened, and more extensive cooperation and collaboration are developed.
EXAMPLE III
As shown in fig. 6, in an embodiment of the present invention, there is provided a data processing system including:
the government affair data platform 610 stores government affair data information;
and the data processing center 620 comprises a data processing device in the embodiment, and is used for acquiring the target data from the government affair data platform 610 and executing the steps of the data processing method in the embodiment to process the target data to generate the target data directory.
In the embodiment of the present invention, the government affair data platform 610 may be a data platform set for each different department of the government affair department, for example, the tax administration includes an office department, a policy and regulation department, a goods and labor tax department, a income tax department, and the like, so that each different department may independently set a corresponding data platform to record each business data. The foregoing is by way of example only.
The data processing system provided by the embodiment of the invention automatically and intelligently analyzes the data of the database to obtain the data portrait, then judges the data related to the core services of the departments, arranges the data to form the data directory, can more accurately know the core services of each party among government departments, and specifically shares the data by combining the service characteristics of the government departments, thereby getting through the service lines and developing more extensive cooperation and collaboration.
Example four
In an embodiment of the present invention, a computer device is provided, where the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:
acquiring target data of a target database, wherein the target data comprises but is not limited to database metadata information and database log information;
calling a preset intelligent analysis algorithm engine according to a preset scheduling task to process target data and generate a data image;
and arranging the target data according to the data catalog of the data portrait to generate a target data catalog.
FIG. 7 is a diagram illustrating an internal structure of a computer device in one embodiment. The computer device may be specifically an independent physical server or a terminal, may also be a server cluster formed by a plurality of physical servers, and may be a cloud server providing basic cloud computing services such as a cloud server, a cloud database, a cloud storage, and a CDN. But not limited thereto, the smart phone, the tablet computer, the notebook computer, the desktop computer, the smart speaker, the smart watch, and the like may also be used. As shown in fig. 7, the computer apparatus includes a processor, a memory, a network interface, an input device, and a display screen linked by a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program which, when executed by the processor, causes the processor to implement the data processing method. The internal memory may also have stored therein a computer program that, when executed by the processor, causes the processor to perform a data processing method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
EXAMPLE five
In an embodiment of the present invention, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, causes the processor to perform the steps of:
acquiring target data of a target database, wherein the target data comprises but is not limited to database metadata information and database log information;
calling a preset intelligent analysis algorithm engine according to a preset scheduling task to process target data and generate a data image;
and arranging the target data according to the data catalog of the data portrait to generate a target data catalog.
It should be understood that, although the steps in the flowcharts of the embodiments of the present invention are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in various embodiments may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only show some embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (7)
1. A data processing method, comprising:
performing connection configuration with a target database, and testing whether the database connection is communicated;
configuring data acquisition permission of the target database;
acquiring target data of a target database, wherein the target data comprises but is not limited to database metadata information and database log information, the database metadata information is metadata which is data describing data and mainly describes data attributes, and the database log information is log information recording database data processing changes;
analyzing the database type of the target data, and executing a corresponding scheduling task according to the database type of the target data to call a corresponding preset intelligent analysis algorithm engine;
analyzing the target data according to a preset data evaluation index through the intelligent analysis algorithm engine, and determining an index attribute of the target data, wherein the index attribute comprises but is not limited to one or a combination of more of data effectiveness, data integrity and data activity;
screening the target data according to the index attribute of the target data, determining core target data, and generating a data image according to the core target data;
and performing data catalog arrangement on the target data according to the data image to generate a target data catalog.
2. The data processing method according to claim 1, wherein the step of analyzing the target data according to a preset data evaluation index by the intelligent analysis algorithm engine to determine an index attribute of the target data further comprises:
and carrying out identification classification on the data in the target data according to the index attribute of the target data.
3. The data processing method according to claim 1, wherein the step of generating a target data list by performing data list layout on the target data based on the data image further comprises:
and sending the target data directory to a user side, receiving directory modification information returned by the user side, and adjusting the target data directory according to the directory modification information.
4. A data processing apparatus, comprising:
the data acquisition module is used for carrying out connection configuration with a target database and testing whether the database connection is communicated; configuring data acquisition permission of the target database; acquiring target data of a target database, wherein the target data comprises but is not limited to database metadata information and database log information, the database metadata information is metadata which is data describing data and mainly describes data attributes, and the database log information is log information recording database data processing changes;
the data processing module is used for analyzing the database type of the target data and executing a corresponding scheduling task according to the database type of the target data to call a corresponding preset intelligent analysis algorithm engine; analyzing the target data according to a preset data evaluation index through the intelligent analysis algorithm engine, and determining an index attribute of the target data, wherein the index attribute comprises but is not limited to one or a combination of more of data effectiveness, data integrity and data activity; screening the target data according to the index attribute of the target data, determining core target data, and generating a data image according to the core target data;
and the catalog generation module is used for arranging the data catalog of the target data according to the data image to generate a target data catalog.
5. A data processing system, comprising:
the government affair data platform is used for storing government affair data information;
a data processing center, comprising the data processing apparatus of claim 4, configured to obtain target data from the government data platform, and perform the steps of the data processing method of any one of claims 1 to 3 to process the target data to generate a target data directory.
6. A computer arrangement, characterized by a memory and a processor, in which a computer program is stored which, when executed by the processor, causes the processor to carry out the steps of the data processing method of any one of claims 1 to 3.
7. A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, causes the processor to carry out the steps of the data processing method of any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010944093.9A CN111813842B (en) | 2020-09-10 | 2020-09-10 | Data processing method, device, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010944093.9A CN111813842B (en) | 2020-09-10 | 2020-09-10 | Data processing method, device, system, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111813842A CN111813842A (en) | 2020-10-23 |
| CN111813842B true CN111813842B (en) | 2021-03-05 |
Family
ID=72860172
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010944093.9A Active CN111813842B (en) | 2020-09-10 | 2020-09-10 | Data processing method, device, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111813842B (en) |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108280194B (en) * | 2018-01-26 | 2019-05-21 | 重庆市地理信息中心 | A kind of search and methods of exhibiting towards complex space data |
| CN109145047B (en) * | 2018-06-29 | 2023-02-03 | 土巴兔集团股份有限公司 | User tag portrait configuration method, data processing device and storage medium |
| CN109241062A (en) * | 2018-09-27 | 2019-01-18 | 国信优易数据有限公司 | A kind of generation method and device of government data catalogue |
| CN109992632A (en) * | 2019-01-14 | 2019-07-09 | 江苏智途科技股份有限公司 | A kind of spatial data intelligence distribution method of servicing based on big data |
| CN109858971A (en) * | 2019-02-03 | 2019-06-07 | 北京字节跳动网络技术有限公司 | Processing method, device, storage medium and the electronic equipment of user's portrait |
| CN109977308B (en) * | 2019-03-20 | 2021-07-13 | 北京字节跳动网络技术有限公司 | User group portrait construction method and device, storage medium and electronic equipment |
| CN110008228A (en) * | 2019-03-26 | 2019-07-12 | 北京字节跳动网络技术有限公司 | Acquisition methods and device, the storage medium and electronic equipment of user group's data |
| CN110399476A (en) * | 2019-06-18 | 2019-11-01 | 平安科技（深圳）有限公司 | Generation method, device, equipment and the storage medium of talent's portrait |
| CN110399432A (en) * | 2019-07-23 | 2019-11-01 | 杭州城市大数据运营有限公司 | A kind of classification method of table, device, computer equipment and storage medium |
| CN111475673A (en) * | 2020-03-02 | 2020-07-31 | 视联动力信息技术股份有限公司 | Display method and device of monitoring directory and storage medium |
-
2020
- 2020-09-10 CN CN202010944093.9A patent/CN111813842B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111813842A (en) | 2020-10-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11036771B2 (en) | Data processing systems for generating and populating a data inventory | |
| US11240273B2 (en) | Data processing and scanning systems for generating and populating a data inventory | |
| US11138336B2 (en) | Data processing systems for generating and populating a data inventory | |
| US11921894B2 (en) | Data processing systems for generating and populating a data inventory for processing data access requests | |
| US10437860B2 (en) | Data processing systems for generating and populating a data inventory | |
| US10438016B2 (en) | Data processing systems for generating and populating a data inventory | |
| US10831724B2 (en) | Method of reconciling resources in the metadata hierarchy | |
| CN111191221A (en) | Method and device for configuring authority resources and computer readable storage medium | |
| CN114756627B (en) | Alliance-chain-based digital asset processing method, system and storage medium | |
| CN112288400A (en) | Multi-system data interaction method and device, computer equipment and storage medium | |
| US10970675B2 (en) | Data processing systems for generating and populating a data inventory | |
| US11222309B2 (en) | Data processing systems for generating and populating a data inventory | |
| CN111813842B (en) | Data processing method, device, system, equipment and storage medium | |
| CN114756293A (en) | Service processing method, device, computer equipment and storage medium | |
| CN114268638A (en) | Block chain platform channel management method and device, computer equipment and storage medium | |
| CN116108089A (en) | Big data testing method, computer equipment and storage medium | |
| CN115470180A (en) | File processing method and device, electronic equipment and computer readable storage medium | |
| CN117455386A (en) | Resource auditing method and device, computer equipment and storage medium thereof | |
| CN116562777A (en) | Approval process method, device, computer equipment and storage medium | |
| WO2019023511A1 (en) | Data processing systems for generating and populating a data inventory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |