CN116795412A - Buried data processing method, buried data processing device, buried data processing equipment and storage medium - Google Patents
Buried data processing method, buried data processing device, buried data processing equipment and storage medium Download PDFInfo
- Publication number
- CN116795412A CN116795412A CN202310273271.3A CN202310273271A CN116795412A CN 116795412 A CN116795412 A CN 116795412A CN 202310273271 A CN202310273271 A CN 202310273271A CN 116795412 A CN116795412 A CN 116795412A
- Authority
- CN
- China
- Prior art keywords
- buried
- data
- buried point
- writing
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 71
- 238000003672 processing method Methods 0.000 title claims abstract description 33
- 238000010276 construction Methods 0.000 claims abstract description 17
- 238000005192 partition Methods 0.000 claims description 45
- 238000000034 method Methods 0.000 claims description 35
- 238000004458 analytical method Methods 0.000 abstract description 15
- 238000007429 general method Methods 0.000 abstract description 11
- 230000006870 function Effects 0.000 description 23
- 238000010586 diagram Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 239000000284 extract Substances 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000009933 burial Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application provides a buried point data processing method, a device, equipment and a storage medium, which can store parameter information of buried points into MySQL in advance, can automatically acquire the parameter information of the buried points from MySQL even facing the buried points with different data structures, construct the corresponding relation between the parameter information and table information, and automatically realize the construction and writing of a table corresponding to buried point data without repeatedly developing corresponding codes to realize the table construction and table writing of the buried point data, thereby remarkably improving the processing efficiency of the buried point data. The embodiment realizes a general method and function for analyzing massive buried point data, avoids the need of developing codes for each new service or service logic adjustment, obviously improves the processing speed of the buried point data and the efficiency of writing the data into a database, and simultaneously ensures the availability and timeliness of the buried point analysis function.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing buried data.
Background
With the rapid development of computers and the internet, such clients that install and use APP (application) are becoming more popular, such as mobile phone APP, portable devices, software and services installed on traditional computers. These clients typically need to interact with a remote server to provide the user with functionality and use of data, and thus there are increasing statistical requirements for user behavior, active operational effects, data access frequency, conversion rate, etc. At present, one of the main modes of client data is to perform certain business, data and interactive buried points on a client, and transmit the buried point data to a server for statistical analysis, and evaluate the effect of operation activities, access quantity and the like.
In the related art, the number of buried points corresponding to the APP is increasing, and how to efficiently process a large number of buried point data is a problem generally considered in the industry.
It should be noted that the information disclosed in the foregoing background section is only for enhancement of understanding of the background of the application and thus may include information that does not form the prior art that is already known to those of ordinary skill in the art.
Disclosure of Invention
Aiming at the problems in the prior art, the application aims to provide a buried data processing method, a device, equipment and a storage medium, which overcome the difficulties in the prior art and can improve the buried data processing efficiency.
The embodiment of the disclosure provides a buried data processing method, which comprises the following steps:
extracting parameter information of the buried point from MySQL;
writing corresponding table information in MySQL according to parameter information of the buried point;
and constructing a table of the buried points in the database according to the table information and the parameter information of the buried points, wherein the table structure of the constructed data table is determined according to the parameter information of the buried points.
Optionally, the embedded point data processing method is implemented by calling a Spark program; the buried data processing method further comprises the following steps:
and constructing Spark partitions for the buried points, wherein the Spark partitions are used for transmitting data of the buried points.
Optionally, the method for processing the buried data further includes:
generating a data writing instruction according to the parameter information of the buried point and the corresponding table information, and storing the data writing instruction into MySQL, wherein the data writing instruction is used for indicating to write the buried point data into the corresponding data table in the database.
Optionally, writing the corresponding table information in MySQL according to the parameter information of the buried point, including:
and under the condition that the table information of the buried point is not stored in MySQL, writing the corresponding table information in MySQL according to the parameter information of the buried point.
Optionally, the method for processing the buried data further includes:
under the condition that the table information of the buried point is stored in MySQL, the data table is positioned in the database according to the stored table information, and the data table is updated according to the parameter information of the buried point.
Alternatively, the database is implemented based on clickHouse.
The embodiment of the disclosure also provides a buried data processing method, which comprises the following steps:
collecting buried point data, and analyzing the buried point data to obtain parameter information of the buried point;
performing parameter matching in MySQL according to the parameter information of the buried points to obtain corresponding target table information;
and positioning the target table in the database according to the target table information, and writing the buried point data into the target table.
Optionally, the embedded point data processing method is implemented by calling a Spark program; writing buried point data to a target table, comprising:
and writing the buried point data into the target table through a Spark partition corresponding to the buried point in the Spark program.
Optionally, writing the embedded point data into the target table through the Spark partition corresponding to the embedded point in the Spark program includes:
and under the condition that the buried point data of a plurality of different buried points are acquired and matched with a plurality of corresponding target tables, the corresponding target tables are written in through Spark partitions corresponding to the different buried points.
Optionally, the database is implemented based on a clickHouse, and the plurality of target tables are respectively stored in a plurality of corresponding table partitions in the clickHouse; the method for writing the embedded point data of different embedded points into the corresponding target table through the Spark partition corresponding to each of the different embedded points comprises the following steps:
and writing the buried point data of the different buried points into the target table of the corresponding table partition through the Spark partition corresponding to each of the different buried points.
Optionally, locating the target table in the database according to the target table information, writing the buried point data into the target table, including:
under the condition that a data writing instruction is obtained from MySQL, in response to the data writing instruction, positioning a target table in a database according to target table information, and writing buried point data into the target table.
The embodiment of the disclosure also provides a buried data processing device, which includes:
the extracting module is used for extracting parameter information of the buried point from MySQL;
the first writing module writes corresponding table information in MySQL according to parameter information of the buried point;
and the table construction module is used for constructing a table of the buried points in the database according to the table information and the parameter information of the buried points, and the table structure of the constructed data table is determined according to the parameter information of the buried points.
The embodiment of the disclosure also provides a buried data processing device, which includes:
the acquisition module acquires buried point data and analyzes the buried point data to obtain parameter information of the buried point;
the matching module is used for carrying out parameter matching in MySQL according to the parameter information of the buried point to obtain corresponding target table information;
and the second writing module is used for positioning the target table in the database according to the target table information and writing the buried point data into the target table.
The embodiment of the application also provides electronic equipment, which comprises:
a processor;
a memory having stored therein executable instructions of a processor;
wherein the processor is configured to perform the steps of the above-described buried data processing method via execution of executable instructions.
The embodiment of the present application also provides a computer-readable storage medium storing a program which, when executed, implements the steps of the above-described buried data processing method.
According to the embedded point data processing method, device, equipment and storage medium, the parameter information of the embedded point can be stored in MySQL in advance, even if the embedded point is faced with the embedded point of different data structures, the corresponding relation between the parameter information and the table information can be built by automatically acquiring the parameter information of the embedded point from MySQL, the building and writing of the table corresponding to the embedded point data can be automatically realized, the table building and the table writing of the embedded point data can be realized without repeatedly developing corresponding codes, and the processing efficiency of the embedded point data can be remarkably improved. The embodiment realizes a general method and function for analyzing massive buried point data, avoids the need of developing codes for each new service or service logic adjustment, obviously improves the processing speed of the buried point data and the efficiency of writing the data into a database, and simultaneously ensures the availability and timeliness of the buried point analysis function.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the following drawings.
FIG. 1 shows one of the flowcharts of the buried data processing method of the embodiment of the present disclosure.
FIG. 2 shows a second flowchart of a buried data processing method according to an embodiment of the present disclosure.
FIG. 3 is a third flowchart of a method for processing buried data according to an embodiment of the present disclosure.
Fig. 4 is a schematic diagram of a buried data processing method according to an embodiment of the present disclosure.
Fig. 5 is one of schematic structural diagrams of a buried data processing apparatus according to an embodiment of the present disclosure.
FIG. 6 is a second schematic diagram of a buried data processing apparatus according to an embodiment of the present disclosure.
Fig. 7 is a schematic structural view of the electronic device of the present application. And
fig. 8 is a schematic structural view of a computer-readable storage medium according to an embodiment of the present application.
Detailed Description
Other advantages and effects of the present application will be readily apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present application by way of specific examples. The application may be practiced or carried out in other embodiments and with various details, and various modifications and alterations may be made to the details of the application from various points of view and applications without departing from the spirit of the application. It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other.
The embodiments of the present application will be described in detail below with reference to the attached drawings so that those skilled in the art to which the present application pertains can easily implement the present application. This application may be embodied in many different forms and is not limited to the embodiments described herein.
In the related art, mass buried data is analyzed using a clickHouse, which has the following problems. Typically, different buried points have different private parameters, and the data structure of each buried point data is different. If analysis of these buried points is required, a custom data table needs to be developed for each buried point, the structure of the table needs to contain the parameters contained by the buried points. The huge number of data tables affects the query speed of the buried data, and the daily data amount of the buried data is in the order of billions, which obviously further affects the data query speed. Furthermore, due to low efficiency of processing the buried data, timeliness of the output of the buried data is affected.
Therefore, the embodiment of the disclosure proposes an efficient buried point data processing scheme, by storing parameter information of a buried point into MySQL, the parameter information includes a data structure thereof, writing corresponding table information into MySQL according to the parameter information of the buried point stored in MySQL, and constructing a table in a database according to the table information and the parameter information of the buried point. In this way, in the application stage, when fresh buried point data is received, the buried point data is analyzed to obtain parameter information of the buried point, and information matching is performed in MySQL according to the parameter information to obtain target table information, so that a target table can be positioned in a database according to the target table information, and the buried point data is written into the target table.
The embodiment provides an automatic buried point data writing program, even if the buried points of different data structures are faced, the corresponding relation between the parameter information and the table information can be built by automatically acquiring the parameter information of the buried points from MySQL, and the construction and writing of the table corresponding to the buried point data can be automatically realized, and the table writing of the table building and the buried point data can be realized without repeatedly developing corresponding codes, so that the buried point data processing efficiency can be remarkably improved. The embodiment realizes a general method and function for analyzing massive buried point data, avoids the need of developing codes for each new service or service logic adjustment, obviously improves the processing speed of the buried point data and the efficiency of writing the data into a database, and simultaneously ensures the availability and timeliness of the buried point analysis function.
Fig. 1 shows a flowchart of a method for processing buried data according to an embodiment of the present disclosure, as shown in fig. 1, the method includes, but is not limited to, the following steps:
step 110: extracting parameter information of the buried point from MySQL;
step 120: writing corresponding table information in MySQL according to parameter information of the buried point;
step 130: and constructing a table of the buried points in the database according to the table information and the parameter information of the buried points, wherein the table structure of the constructed data table is determined according to the parameter information of the buried points.
By storing the parameter information of the buried point in MySQL in advance, the embodiment can automatically acquire the parameter information of the buried point from MySQL, construct the corresponding relation between the parameter information and the table information, and automatically realize the construction and writing of the table corresponding to the buried point data, without repeatedly developing corresponding codes to realize the table construction and the table writing of the buried point data, thus being capable of remarkably improving the processing efficiency of the buried point data. The embodiment realizes a general method and function for analyzing massive buried point data, avoids the need of developing codes for each new service or service logic adjustment, obviously improves the processing speed of the buried point data and the efficiency of writing the data into a database, and simultaneously ensures the availability and timeliness of the buried point analysis function.
In the embodiment of the disclosure, the above-mentioned buried data processing method is implemented by calling a Spark program. Specifically, by writing unified codes in the Spark program, parameter information of the buried points can be automatically read from MySQL, a table is further correspondingly built in a database, and the corresponding relation between the parameter information of the buried points and the table information is stored in MySQL.
In this case, a Spark partition is further constructed for the buried point in the Spark program, and the Spark partition is used to transmit data of the buried point. Specifically, in the application stage, when the buried point data of the buried point is collected, the buried point data is written into a corresponding table in the database through the Spark partition.
By using the Spark program, the embedded point data can be written concurrently, and the embedded point data writing efficiency is improved. Spark is a fast and general-purpose computing engine designed for large-scale data processing, and can read data in a distributed manner, and the processed data is written into a target medium in a distributed manner through various conversion, processing and the like on the data.
In the embodiment of the present disclosure, when step 120 is performed, the parameter information of the buried point is determined to be the latest information, and may be determined according to the date of the parameter information. Further, when the corresponding table information is not matched in MySQL according to the parameter information of the buried point, the buried point is indicated to be the latest reported buried point, at this time, the corresponding table information can be written in MySQL according to the parameter information of the buried point, that is, the corresponding relation between the parameter information of the buried point and the table information is constructed, and finally, the table is built in the database.
In a further embodiment of the present disclosure, in the case that the parameter information according to the buried point is matched to the stored table information, the data table may be directly located in the database according to the stored table information, and updated according to the parameter information of the buried point.
In this embodiment, the parameter information of the newly added buried point is updated parameter information of the existing buried point, for example, a newly added field or a modified field, and the corresponding data table in the database is consistently modified based on the updated parameter information, so as to ensure the accuracy of writing the data of the later buried point.
In the disclosed embodiment, the database is implemented based on clickHouse. The ClickHouse is a column type database management system for data analysis, and can process massive data in parallel through vectorization execution and use of a CPU bottom instruction set, so that the data processing speed is increased, and an analysis result is obtained rapidly.
The embodiment of the disclosure provides a mass buried point data analysis method based on Spark computing engine and ClickHouse database. The Spark program is written for analyzing and processing the custom parameter information of the buried point, then the data is parallelly imported into the ClickHouse through the Spark program, and various processing is carried out on massive big data in a database mode through the ClickHouse, so that the method is simpler and easier to use, convenient to maintain and high in query speed, and various analysis functions are supported.
In the embodiment of the disclosure, the table is built for the buried point in the database according to the table information and the parameter information of the buried point, specifically, the table name of the data table to be built is determined according to the table information, for example, the table name may be the buried point name, then the data structure of the data table to be built, such as the column field name, is determined according to the parameter information of the buried point, and then the table is built according to the table name and the data structure of the data table to be built.
Fig. 2 shows a flowchart of a method for processing buried point data according to an embodiment of the present disclosure, where the method is used for writing collected buried point data into a database, specifically, as shown in fig. 2, the method for processing buried point data includes, but is not limited to, the following steps:
step 210: collecting buried point data, and analyzing the buried point data to obtain parameter information of the buried point;
step 220: performing parameter matching in MySQL according to the parameter information of the buried points to obtain corresponding target table information;
step 230: and positioning the target table in the database according to the target table information, and writing the buried point data into the target table.
By using the embodiment, for any buried point, table information corresponding to the parameter information of the buried point can be automatically extracted from MySQL, so that a target table is automatically positioned in a database according to the table information and table writing is performed. According to the embodiment, the table construction and table writing of the embedded data are realized without repeatedly developing corresponding codes, so that the embedded data processing efficiency can be remarkably improved. The embodiment realizes a general method and function for analyzing massive buried point data, avoids the need of developing codes for each new service or service logic adjustment, obviously improves the processing speed of the buried point data and the efficiency of writing the data into a database, and simultaneously ensures the availability and timeliness of the buried point analysis function.
In the embodiment of the present disclosure, the correspondence between the parameter information of the buried point and the table information stored in MySQL may be configured based on the method shown in fig. 1, or may be obtained by using other methods, which is not limited herein.
In the embodiment of the disclosure, the buried data processing method may be implemented by calling a Spark program. In this case, the Spark partition corresponding to the buried point in the Spark program writes the buried point data in the target table. Spark is a fast and general-purpose computing engine designed for large-scale data processing, and can read data in a distributed manner, and write the processed data into a target medium in a distributed manner through various conversion, processing and the like on the data.
In the embodiment of the disclosure, when buried point data of a plurality of different buried points are collected and matched with a plurality of corresponding target tables, spark partitions corresponding to the different buried points are written into the corresponding target tables concurrently.
In this case, a Spark partition is created for the buried point in the Spark program, and the Spark partition is used to transmit data of the buried point. By using the Spark program, the embedded point data can be written concurrently, and the embedded point data writing efficiency is improved.
In the embodiment of the disclosure, the database is realized based on the ClickHouse, and a plurality of target tables are respectively stored in a plurality of corresponding table partitions in the ClickHouse; the method for writing the embedded point data of different embedded points into the corresponding target table through the Spark partition corresponding to each of the different embedded points comprises the following steps:
and writing the buried point data of the different buried points into the target table of the corresponding table partition through the Spark partition corresponding to each of the different buried points.
In the disclosed embodiment, a data writing instruction may be generated by a Spark program according to parameter information and corresponding table information of the buried point, where the data writing instruction is used to instruct writing of buried point data into a corresponding data table in the database. In this case, in the case where the data writing instruction is acquired from MySQL, the target table is located in the database according to the target table information in response to the data writing instruction, and the buried point data is written into the target table.
The embodiment of the disclosure provides a mass buried point data analysis method based on Spark computing engine and ClickHouse database. The Spark program is written for analyzing and processing the custom parameter information of the buried point, then the data is parallelly imported into the ClickHouse through the Spark program, and various processing is carried out on massive big data in a database mode through the ClickHouse, so that the method is simpler and easier to use, convenient to maintain and high in query speed, and various analysis functions are supported.
In the embodiment of the disclosure, the table is built for the buried point in the database according to the table information and the parameter information of the buried point, specifically, the table name of the data table to be built is determined according to the table information, for example, the table name may be the buried point name, then the data structure of the data table to be built, such as the column field name, is determined according to the parameter information of the buried point, and then the table is built according to the table name and the data structure of the data table to be built.
Fig. 3 shows a buried data processing method of a specific application scenario of the present disclosure, and as shown in fig. 3, the method specifically includes the following steps.
Step 310: when the reported buried point is obtained, saving a parameter structure of the buried point in MySQL, extracting the parameter structure of the buried point from the MySQL by the Sprak program, and analyzing the custom parameter of the buried point to obtain the parameter structure of the buried point;
step 320: generating a statement of writing the ClickHouse according to the parameter structure of the buried point, and storing the statement in MySQL.
Step 330: by analyzing the custom parameters of the buried points, a corresponding table is built in the ClickHouse for each buried point, and the column names correspond to all the custom parameters of the buried points. Specifically, it is determined whether a table corresponding to the buried point exists.
If yes, step 340 is executed, and according to the parameter change information, a column corresponding to the parameter is newly added in the stored table and the data in MySQL is updated;
if not, go to step 350 and newly build a correspondence table in the clickHouse.
Specifically, the ClickHouse builds a table statement:
CREATE TABLE table_name if not exists table_name(
Column A String,
Column B String,
`d`Date DEFAULT toDate(receive_time,'Asia/Hong_Kong'),
)
ENGINE＝ReplacingMergeTree(current_time)
PARTITION BY(d)
TTL d+toIntervalDay(31)
SETTINGS index_granularity＝8192
a displacingmergtree ENGINE table is created by an engine=displacingmergtree (current_time) statement, and the uniqueness of the data is ensured according to the current_time field.
The PARTION BY (d) TTL d+toIntervalDay (31) carries out PARTITION management on data, PARTITIONs the data according to date, and keeps data of a period of the latest history.
The SETTINGS index_granularity=8192 divides each column of data according to index granularity (index granularity, default 8192 lines), and can directly locate the corresponding index granularity by performing binary search on the primary key index, so that full-table scanning is avoided and query is accelerated.
Step 360: the embedded point data are stored in the Hive table in a Jason format, so that the embedded point data are acquired from the Hive table and each piece of embedded point data are analyzed through a Spark program written in advance to obtain the custom parameters of the embedded point.
Step 370: the method comprises the steps of connecting MySQL through a Spark program, obtaining data in the MySQL, recording a statement of 'writing in a ClickHouse' of each buried point in the MySQL, specifically a statement of writing in a same-name ClickHouse table, and writing the buried point data into the ClickHouse by calling the statements.
In the embodiment of the disclosure, referring to fig. 4, by running a Spark program, first performing Spark partition on buried point data, ordering data inside the Spark partition according to buried point names by a sortwisinin partition method, then traversing the buried point data in the Spark partition by using a foreachPartition method, and writing the buried point data in a click house in batches by a concurrency mode:
the batchUpdate may be used to write data in the Spark partition to the clickHouse in batch by setting a threshold, e.g., 10000, when the number of data stripes is greater than or equal to 10000. Batch insertion can significantly improve efficiency. If the number of records is less than 10000, the data in the Spark partition will continue to be traversed.
The technical scheme provided by the embodiment of the disclosure can achieve the following effects: by using Spark computing engine and ClickHouse database system, a general method and function for analyzing mass buried data are realized. Only the function is required to be called in the project, the problem that codes need to be developed for new service or service logic adjustment each time is avoided, the processing speed of buried point data and the efficiency of writing the data into the ClickHouse cluster are obviously improved, and meanwhile, the availability and timeliness of the buried point analysis function are ensured.
Fig. 5 is a schematic diagram of the structure of the buried data processing apparatus according to the present application. As shown in fig. 5, a buried data processing apparatus 500 of the present application includes:
the extracting module 510 extracts parameter information of the buried point from MySQL;
the first writing module 520 writes corresponding table information in MySQL according to parameter information of the buried point;
the table construction module 530 constructs a table of the buried points in the database according to the table information and the parameter information of the buried points, and the table structure of the constructed data table is determined according to the parameter information of the buried points.
In an alternative embodiment, the method for processing the embedded point data is implemented by calling a Spark program; the table building module 530 is specifically further configured to:
and constructing Spark partitions for the buried points, wherein the Spark partitions are used for transmitting data of the buried points.
In an alternative embodiment, the table creation module 530 is specifically further configured to:
generating a data writing instruction according to the parameter information of the buried point and the corresponding table information, and storing the data writing instruction into MySQL, wherein the data writing instruction is used for indicating to write the buried point data into the corresponding data table in the database.
In an alternative embodiment, the first writing module 520 is specifically configured to:
and under the condition that the table information of the buried point is not stored in MySQL, writing the corresponding table information in MySQL according to the parameter information of the buried point.
In an alternative embodiment, the table creation module 530 is specifically further configured to:
under the condition that the table information of the buried point is stored in MySQL, the data table is positioned in the database according to the stored table information, and the data table is updated according to the parameter information of the buried point.
In an alternative embodiment, the database is implemented based on a clickHouse.
According to the buried point data processing device, by storing the parameter information of the buried point in the MySQL in advance, even if the buried point is faced with buried points of different data structures, the corresponding relation between the parameter information and the table information can be built by automatically acquiring the parameter information of the buried point from the MySQL, and the construction and writing of the table corresponding to the buried point data can be automatically realized, and the table writing of the table building and the buried point data can be realized without repeatedly developing corresponding codes, so that the buried point data processing efficiency can be remarkably improved. The embodiment realizes a general method and function for analyzing massive buried point data, avoids the need of developing codes for each new service or service logic adjustment, obviously improves the processing speed of the buried point data and the efficiency of writing the data into a database, and simultaneously ensures the availability and timeliness of the buried point analysis function.
Fig. 6 is a schematic diagram of the structure of the buried data processing apparatus according to the present application. As shown in fig. 6, a buried data processing apparatus 600 of the present application includes:
the acquisition module 610 acquires buried point data and analyzes the buried point data to obtain parameter information of the buried point;
the matching module 620 performs parameter matching in MySQL according to the parameter information of the buried point to obtain corresponding target table information;
the second writing module 630 locates the target table in the database according to the target table information and writes the buried point data into the target table.
In an alternative embodiment, the method for processing the embedded point data is implemented by calling a Spark program; the second writing module 630 is specifically configured to:
and writing the buried point data into the target table through a Spark partition corresponding to the buried point in the Spark program.
In an alternative embodiment, the second writing module 630 is specifically configured to:
and under the condition that the buried point data of a plurality of different buried points are acquired and matched with a plurality of corresponding target tables, the corresponding target tables are written in through Spark partitions corresponding to the different buried points.
In an alternative embodiment, the database is implemented based on a clickHouse, and the plurality of target tables are stored in a corresponding plurality of table partitions in the clickHouse, respectively; the second writing module 630 is specifically configured to:
and writing the buried point data of the different buried points into the target table of the corresponding table partition through the Spark partition corresponding to each of the different buried points.
In an alternative embodiment, the second writing module 630 is specifically configured to:
under the condition that a data writing instruction is obtained from MySQL, in response to the data writing instruction, positioning a target table in a database according to target table information, and writing buried point data into the target table.
The buried point data processing device disclosed by the embodiment of the disclosure can automatically extract the table information corresponding to the parameter information of any buried point from MySQL, so that a target table is automatically positioned in a database according to the table information and the table writing is performed. According to the embodiment, the table construction and table writing of the embedded data are realized without repeatedly developing corresponding codes, so that the embedded data processing efficiency can be remarkably improved. The embodiment realizes a general method and function for analyzing massive buried point data, avoids the need of developing codes for each new service or service logic adjustment, obviously improves the processing speed of the buried point data and the efficiency of writing the data into a database, and simultaneously ensures the availability and timeliness of the buried point analysis function.
The embodiment of the application also provides electronic equipment which comprises a processor. A memory having stored therein executable instructions of a processor. Wherein the processor is configured to execute the steps of the buried data processing method via execution of the executable instructions.
As described above, by storing the parameter information of the buried point in MySQL in advance, the electronic device of the present application can automatically acquire the parameter information of the buried point from MySQL, construct the correspondence between the parameter information and the table information, and automatically implement the construction and writing of the table corresponding to the buried point data, without repeatedly developing corresponding codes to implement the table construction and the table writing of the buried point data, which can significantly improve the processing efficiency of the buried point data. The embodiment realizes a general method and function for analyzing massive buried point data, avoids the need of developing codes for each new service or service logic adjustment, obviously improves the processing speed of the buried point data and the efficiency of writing the data into a database, and simultaneously ensures the availability and timeliness of the buried point analysis function.
Those skilled in the art will appreciate that the various aspects of the application may be implemented as a system, method, or program product. Accordingly, aspects of the application may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" platform.
Fig. 7 is a schematic structural view of the electronic device of the present application. An electronic device 700 according to this embodiment of the application is described below with reference to fig. 7. The electronic device 700 shown in fig. 7 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present application.
As shown in fig. 7, the electronic device 700 is embodied in the form of a general purpose computing device. Components of electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one memory unit 720, a bus 730 connecting the different platform components (including memory unit 720 and processing unit 710), a display unit 740, and the like.
Wherein the storage unit stores program code that can be executed by the processing unit 710 such that the processing unit 710 performs the steps according to various exemplary embodiments of the present application described in the burial point data processing method section of the present specification. For example, the processing unit 710 may perform the steps as shown in fig. 1 or 3.
The memory unit 720 may include readable media in the form of volatile memory units, such as Random Access Memory (RAM) 721 and/or cache memory 722, and may further include Read Only Memory (ROM) 723.
The storage unit 720 may also include a program/utility 724 having a set (at least one) of program modules 725, such program modules 725 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 730 may be a bus representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 700 may also communicate with one or more external devices 70 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 700, and/or any device (e.g., router, modem, etc.) that enables the electronic device 700 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 750. Also, electronic device 700 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through network adapter 760. Network adapter 760 may communicate with other modules of electronic device 700 via bus 730. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 700, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage platforms, and the like.
The embodiment of the application also provides a computer readable storage medium for storing a program, and the steps of the buried data processing method implemented when the program is executed. In some possible embodiments, the aspects of the present application may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the application as described in the above description of the method of processing buried data, when the program product is run on the terminal device.
As described above, when the program of the computer readable storage medium of this embodiment is executed, it is possible to automatically obtain the parameter information of the buried point from MySQL, construct the correspondence between the parameter information and the table information, and automatically implement the construction and writing of the table corresponding to the buried point data by storing the parameter information of the buried point in MySQL in advance, even for buried points of different data structures, without repeatedly developing corresponding codes to implement the table construction and table writing of the buried point data, which can significantly improve the processing efficiency of the buried point data. The embodiment realizes a general method and function for analyzing massive buried point data, avoids the need of developing codes for each new service or service logic adjustment, obviously improves the processing speed of the buried point data and the efficiency of writing the data into a database, and simultaneously ensures the availability and timeliness of the buried point analysis function.
Fig. 8 is a schematic structural view of a computer-readable storage medium of the present application. Referring to fig. 8, a program product 800 for implementing the above-described method according to an embodiment of the present application is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
In summary, the method, the device, the equipment and the storage medium for processing the embedded point data can store the parameter information of the embedded point into the MySQL in advance, even if the embedded point of different data structures is faced, the corresponding relation between the parameter information and the table information can be constructed by automatically acquiring the parameter information of the embedded point from the MySQL, and the construction and the writing of the table corresponding to the embedded point data can be automatically realized without repeatedly developing corresponding codes to realize the table construction and the table writing of the embedded point data, so that the processing efficiency of the embedded point data can be remarkably improved. The embodiment realizes a general method and function for analyzing massive buried point data, avoids the need of developing codes for each new service or service logic adjustment, obviously improves the processing speed of the buried point data and the efficiency of writing the data into a database, and simultaneously ensures the availability and timeliness of the buried point analysis function.
The foregoing is a further detailed description of the application in connection with the preferred embodiments, and it is not intended that the application be limited to the specific embodiments described. It will be apparent to those skilled in the art that several simple deductions or substitutions may be made without departing from the spirit of the application, and these should be considered to be within the scope of the application.
Claims (15)
1. A method for processing buried data, comprising:
extracting parameter information of the buried point from MySQL;
writing corresponding table information in the MySQL according to the parameter information of the buried point;
and constructing a table of the buried points in a database according to the table information and the parameter information of the buried points, wherein the table structure of the constructed data table is determined according to the parameter information of the buried points.
2. The buried data processing method according to claim 1, wherein said buried data processing method is implemented by calling Spark program; the buried data processing method further comprises the following steps:
and constructing Spark partitions for the buried points, wherein the Spark partitions are used for transmitting the data of the buried points.
3. The buried data processing method according to claim 1, characterized in that said buried data processing method further comprises:
generating a data writing instruction according to the parameter information of the buried point and the corresponding table information, and storing the data writing instruction into the MySQL, wherein the data writing instruction is used for indicating to write buried point data into the corresponding data table in the database.
4. The method for processing buried point data according to claim 1, wherein writing the correspondence table information in MySQL according to the parameter information of the buried point includes:
and under the condition that the table information of the buried point is not stored in the MySQL, writing corresponding table information in the MySQL according to the parameter information of the buried point.
5. The buried data processing method according to claim 4, characterized in that said buried data processing method further comprises:
and under the condition that the table information of the buried point is stored in the MySQL, positioning a data table in the database according to the stored table information, and updating the data table according to the parameter information of the buried point.
6. The method of claim 1, wherein the database is implemented based on clickHouse.
7. A method for processing buried data, comprising:
collecting buried point data, and analyzing the buried point data to obtain parameter information of buried points;
performing parameter matching in MySQL according to the parameter information of the buried point to obtain corresponding target table information;
and positioning a target table in a database according to the target table information, and writing the buried point data into the target table.
8. The method for processing buried data according to claim 7, characterized in that,
the buried point data processing method is realized by calling a Spark program; writing the buried point data into the target table, including:
and writing the buried point data into the target table through a Spark partition corresponding to the buried point in the Spark program.
9. The buried point data processing method according to claim 8, wherein writing the buried point data into the target table through a Spark partition corresponding to the buried point in the Spark program includes:
and under the condition that buried point data of a plurality of different buried points are acquired and matched with a plurality of corresponding target tables, the corresponding target tables are written in through Spark partitions corresponding to the different buried points.
10. The method of claim 9, wherein the database is implemented based on a clickHouse, and wherein the plurality of target tables are stored in a corresponding plurality of table partitions in the clickHouse, respectively; the step of writing the buried point data of the different buried points into the corresponding target table through the Spark partition corresponding to each of the different buried points, includes:
and writing the buried point data of the different buried points into the target table of the corresponding table partition through the Spark partition corresponding to each of the different buried points.
11. The buried point data processing method according to claim 7, wherein locating a target table in a database according to the target table information, writing the buried point data into the target table, comprises:
and under the condition that a data writing instruction is obtained from the MySQL, responding to the data writing instruction, positioning a target table in a database according to the target table information, and writing the buried point data into the target table.
12. A buried data processing apparatus, comprising:
the extracting module is used for extracting parameter information of the buried point from MySQL;
the first writing module writes corresponding table information in the MySQL according to the parameter information of the buried point;
and the table construction module is used for constructing a table of the buried points in a database according to the table information and the parameter information of the buried points, and the table structure of the constructed data table is determined according to the parameter information of the buried points.
13. A buried data processing apparatus, comprising:
the acquisition module is used for acquiring buried point data and analyzing the buried point data to obtain parameter information of the buried point;
the matching module is used for carrying out parameter matching in MySQL according to the parameter information of the buried point to obtain corresponding target table information;
and the second writing module is used for positioning a target table in a database according to the target table information and writing the buried point data into the target table.
14. An electronic device, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the buried data processing method of any one of claims 1 to 11 via execution of the executable instructions.
15. A computer-readable storage medium storing a program, characterized in that the program when executed implements the steps of the buried data processing method according to any one of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310273271.3A CN116795412A (en) | 2023-03-17 | 2023-03-17 | Buried data processing method, buried data processing device, buried data processing equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310273271.3A CN116795412A (en) | 2023-03-17 | 2023-03-17 | Buried data processing method, buried data processing device, buried data processing equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116795412A true CN116795412A (en) | 2023-09-22 |
Family
ID=88045250
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310273271.3A Pending CN116795412A (en) | 2023-03-17 | 2023-03-17 | Buried data processing method, buried data processing device, buried data processing equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116795412A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118035060A (en) * | 2024-04-11 | 2024-05-14 | 云筑信息科技(成都)有限公司 | Method for dynamically generating buried point standard basic data model |
-
2023
- 2023-03-17 CN CN202310273271.3A patent/CN116795412A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118035060A (en) * | 2024-04-11 | 2024-05-14 | 云筑信息科技(成都)有限公司 | Method for dynamically generating buried point standard basic data model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7146007B2 (en) | Target data acquisition method, target data acquisition device, electronic device, non-transitory computer-readable storage medium, and computer program | |
| CN110825767A (en) | Data query method, platform, equipment and storage medium | |
| CN111709527A (en) | Operation and maintenance knowledge map library establishing method, device, equipment and storage medium | |
| US11200231B2 (en) | Remote query optimization in multi data sources | |
| US11934287B2 (en) | Method, electronic device and computer program product for processing data | |
| CN113312377A (en) | Automatic-association SQL query statement processing method and device and electronic equipment | |
| CN112069261B (en) | Data synchronization method, system, equipment and storage medium based on distributed system | |
| CN116795412A (en) | Buried data processing method, buried data processing device, buried data processing equipment and storage medium | |
| CN116244387A (en) | Entity relationship construction method, device, electronic equipment and storage medium | |
| CN115168398A (en) | Data query method and device, electronic equipment and storage medium | |
| CN114356964A (en) | Data blood margin construction method and device, storage medium and electronic equipment | |
| CN112988583A (en) | Method and device for testing syntax compatibility of database | |
| CN111723134A (en) | Information processing method, information processing device, electronic equipment and storage medium | |
| CN110515807B (en) | Database table monitoring method, device, equipment and storage medium | |
| CN110795424B (en) | Characteristic engineering variable data request processing method and device and electronic equipment | |
| CN111782834A (en) | Image retrieval method, device, equipment and computer readable storage medium | |
| CN110928898A (en) | Data acquisition method, data acquisition device, storage medium and electronic equipment | |
| CN113127496A (en) | Method, apparatus, medium, and device for determining change data in database | |
| CN111178014A (en) | Method and device for processing business process | |
| CN112527880B (en) | Method, device, equipment and medium for collecting metadata information of big data cluster | |
| CN110908993A (en) | Method and device for analyzing reasonability of database index | |
| CN116301869B (en) | Front-end page configuration management and control system, method, equipment and medium | |
| CN112965747B (en) | Method, apparatus, device and computer readable medium for mining code loopholes | |
| JP7241201B2 (en) | Sampling target identification method, device, electronic device and computer readable storage medium | |
| US10558680B2 (en) | Efficient data retrieval in big-data processing systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |