CN117093597B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN117093597B CN117093597B CN202311331566.8A CN202311331566A CN117093597B CN 117093597 B CN117093597 B CN 117093597B CN 202311331566 A CN202311331566 A CN 202311331566A CN 117093597 B CN117093597 B CN 117093597B
- Authority
- CN
- China
- Prior art keywords
- data
- target
- database
- updating
- storage table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 39
- 230000008859 change Effects 0.000 claims abstract description 142
- 238000013500 data storage Methods 0.000 claims abstract description 135
- 238000004458 analytical method Methods 0.000 claims abstract description 47
- 238000007726 management method Methods 0.000 claims abstract description 46
- 238000012545 processing Methods 0.000 claims description 35
- 238000012217 deletion Methods 0.000 claims description 32
- 230000037430 deletion Effects 0.000 claims description 32
- 238000000034 method Methods 0.000 claims description 16
- 238000010586 diagram Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 6
- 239000003054 catalyst Substances 0.000 description 5
- 238000003780 insertion Methods 0.000 description 5
- 230000037431 insertion Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/235—Update request formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/243—Natural language query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application provides a data processing method and a device, wherein the data processing method is applied to a data operation terminal and comprises the following steps: receiving a data change statement submitted for a target database; analyzing the data change statement according to a user-defined analysis rule, and acquiring metadata information at a metadata management end according to an analysis result; loading a data storage table associated with the target database according to the metadata information, and extracting a data field to be changed from the data change statement; and selecting a target updating strategy based on the table type of the data storage table, updating the data storage table by utilizing the data field to be changed according to the target updating strategy, and writing the updating result back to the target database.
Description
Technical Field
The present disclosure relates to the field of database technologies, and in particular, to a data processing method and apparatus.
Background
With the development of internet technology, more and more services start to be online, and with the popularization of online services, data needs to be attached to a database to be stored so as to support service development. Compared with the data in the database, the operations of adding, deleting, modifying, checking and the like exist according to the service requirements. In practice, the data computation engine Spark SQL may support the operation of a variety of relational databases, such as supporting the reading and writing of data from MySQL, oracle, postgreSQL and the like. However, the current native Spark SQL only supports the query and insertion of data when operating database data, and does not support the update and deletion operations in the database. In the prior art, the technical scheme disclosed by the publication No. CN 110399386A, although the technical scheme for realizing data updating operation by Spark SQL is disclosed; however, the flexibility is poor, and the implementation of the update and delete operations is completed based on the most basic parsing rules, and the problem of scene complexity in the actual application scene is not considered, so an effective solution is needed to solve the above problem.
Disclosure of Invention
In view of the foregoing, embodiments of the present application provide a data processing method to solve the technical drawbacks of the prior art. Embodiments of the present application also provide a data processing apparatus, a computing device, and a computer-readable storage medium.
According to a first aspect of an embodiment of the present application, there is provided a data processing method, applied to a data operation terminal, including:
receiving a data change statement submitted for a target database;
analyzing the data change statement according to a user-defined analysis rule, and acquiring metadata information at a metadata management end according to an analysis result;
loading a data storage table associated with the target database according to the metadata information, and extracting a data field to be changed from the data change statement;
and selecting a target updating strategy based on the table type of the data storage table, updating the data storage table by utilizing the data field to be changed according to the target updating strategy, and writing the updating result back to the target database.
According to a second aspect of embodiments of the present application, there is provided a data processing apparatus, applied to a data operation terminal, including:
A receiving module configured to receive a data change statement submitted for a target database;
the analysis module is configured to analyze the data change statement according to a custom analysis rule and acquire metadata information at a metadata management end according to an analysis result;
the loading module is configured to load a data storage table associated with the target database according to the metadata information and extract a data field to be changed in the data change statement;
and the updating module is configured to select a target updating strategy based on the table type of the data storage table, update the data storage table by utilizing the data field to be changed according to the target updating strategy, and write the updating result back to the target database.
According to a third aspect of embodiments of the present application, there is provided a computing device comprising:
a memory and a processor;
the memory is used for storing computer executable instructions, and the processor implements the steps of the data processing method when executing the computer executable instructions.
According to a fourth aspect of embodiments of the present application, there is provided a computer readable storage medium storing computer executable instructions which, when executed by a processor, implement the steps of the data processing method.
In order to improve the flexibility of data operation in the relational database and complete data operation according to actual scene requirements, the data processing method provided by the embodiment can analyze the data change statement by utilizing a custom analysis rule under the condition that the data change statement submitted for the target database is received, so that metadata information is obtained at a metadata management end according to an analysis result, and the metadata information is used as a basis of the data operation; then, a data storage table corresponding to the data operation requirement in the current scene can be loaded in the target database according to the metadata information, and the data field to be changed is extracted from the data change statement, so that the data operation can be flexibly completed for any data operation scene; after the data storage table and the data field to be changed are obtained, in order to ensure that any type of operation can be completed for the database, a corresponding target updating strategy can be selected according to the table type of the data storage table, the data storage table is updated by the data field to be changed according to the target updating strategy, and the updating result is written back to the target database. When the data in the database is operated, the processing of any requirement can be completed on the database, so that the data can be used more flexibly to meet the actual requirement.
Drawings
FIG. 1 is a schematic diagram of a data processing method according to an embodiment of the present application;
FIG. 2 is a flow chart of a data processing method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a data processing method according to an embodiment of the present application for executing a data change statement;
FIG. 4 is a process flow diagram of a data processing method according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;
FIG. 6 is a block diagram of a computing device according to one embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is, however, susceptible of embodiment in many other ways than those herein described and similar generalizations can be made by those skilled in the art without departing from the spirit of the application and the application is therefore not limited to the specific embodiments disclosed below.
The terminology used in one or more embodiments of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the application. As used in this application in one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present application refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that, although the terms first, second, etc. may be used in one or more embodiments of the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, a first may also be referred to as a second, and similarly, a second may also be referred to as a first, without departing from the scope of one or more embodiments of the present application.
First, terms related to one or more embodiments of the present invention will be explained.
Spark SQL is a module of Spark to process structured data that provides two programming abstractions, dataFrame and DataSet, respectively, and that works as a distributed SQL query engine.
Relational database: by relational model is meant a database that employs relational models to organize data, which store data in rows and columns for the user to understand, a series of rows and columns of the relational database are referred to as tables, a group of tables making up the database. The user retrieves the data in the database by querying, which is an executable code that defines certain areas in the database. A relational model can be understood simply as a two-dimensional tabular model, and a relational database is a data organization consisting of two-dimensional tables and relationships between them.
MySQL: is a relational database management system that keeps data in different tables rather than placing all the data in one large warehouse, which increases speed and increases flexibility.
In the present application, a data processing method is provided. The present application relates to a data processing apparatus, a computing device, and a computer-readable storage medium, and is described in detail in the following embodiments.
In practical application, when the conventional Spark SQL operates the relational database, only the Select and Insert methods are supported, and the Update and Delete methods are not supported, so that in order to realize the Update and Delete operations, a user is required to place data on the database first, and then the data is realized by a JDBC (Java Database Connectivity, java database connection) mode. But this method has a problem that Update and Delete statements cannot be used directly in Spark SQL clients.
Referring to the schematic diagram shown in fig. 1, in order to improve flexibility of data operation in a relational database and complete data operation according to actual scene requirements, the data processing method according to the present embodiment may analyze a data change statement by using a custom analysis rule under the condition that the data change statement submitted for a target database is received, so as to obtain metadata information at a metadata management end according to an analysis result, and use the metadata information as a basis of the data operation; then, a data storage table corresponding to the data operation requirement in the current scene can be loaded in the target database according to the metadata information, and the data field to be changed is extracted from the data change statement, so that the data operation can be flexibly completed for any data operation scene; after the data storage table and the data field to be changed are obtained, in order to ensure that any type of operation can be completed for the database, a corresponding target updating strategy can be selected according to the table type of the data storage table, the data storage table is updated by the data field to be changed according to the target updating strategy, and the updating result is written back to the target database. When the data in the database is operated, the processing of any requirement can be completed on the database, so that the data can be used more flexibly to meet the actual requirement.
Fig. 2 shows a flowchart of a data processing method according to an embodiment of the present application, where the method is applied to a data operation end, and specifically includes the following steps:
step S202, receiving a data change statement submitted for a target database.
The data processing method provided by the embodiment is applied to a data operation end, wherein the data operation end is a client end with an access relation with a target database, for example, a Spark SQL client end can be used; the target database is a relational database, such as MySQL, oracle, postgreSQL. In order to realize the update or deletion operation of the data in the target database by the data operation end, the capacity expansion can be performed on the grammar analysis rule of the data operation end based on the Antlr4, the custom grammar analysis rule is realized by utilizing the Antlr4, meanwhile, the processing rule of the grammar expansion container interface writing statement execution plan of the data operation end is inherited, and when the update or deletion operation of the data in the target database is realized, the processing rule is based, so that the support of the more flexible operation of the data operation end in the database, such as the processing of adding, deleting, and the like, is ensured.
Specifically, the data change statement specifically refers to a statement submitted when the operation needs to be performed on the data in the target database, and the statement is used for deleting or updating the data; it may be an SQL statement or a custom statement such as regular expression, HQL, JPQL, CQL, etc. corresponding to other custom languages. The present embodiment describes a data processing method using an example in which a data change statement is an SQL statement.
In this case, when the data change statement submitted to the target database is received, it is described that the data in the target database needs to be changed at this time, but since the database in which the operation is performed is a relational database and the terminal that triggers the operation is a data operation terminal, operations such as deletion and correction can be completed to support, for example, the data change statement is received, and then, subsequent analysis processing can be performed, and thus, a corresponding operation processing can be performed according to the analysis result.
In addition, on the premise that the data operation end can carry out data change processing on the target database, the insertion and selection on the target database still need to be supported, so that the execution of tasks can be carried out according to the change task set by inserting and selecting corresponding sentences, and the update of the data in the database is realized; in this embodiment, the specific implementation manner is as follows:
analyzing the data change sentence by using a reference analysis rule under the condition that the sentence type of the data change sentence is the target sentence type; generating a change task set according to the analysis result, and executing the change task contained in the change task set according to a logic execution strategy; and updating the target data contained in the target database according to the execution result of the change task.
Specifically, the target sentence type is specifically that the index change sentence is an insert sentence type or a select sentence type; correspondingly, the reference parsing rule is specifically a parsing rule existing in the data operation terminal. Correspondingly, the change task set specifically refers to the combination of task components created based on the target statement type, and the update of the data contained in the target database, namely the process of data insertion or data selection, can be realized by executing the task. Correspondingly, the logic execution strategy is specifically a task execution strategy which is native to the data operation end and is used for executing the change task to realize data operation processing.
Based on the above, when the statement type of the data change statement is the target statement type, the operation on the target data in the target database can be completed by utilizing the processing rule of the data operation terminal native at the moment; therefore, the data change statement can be analyzed by utilizing the standard analysis rule; generating a change task set according to the analysis result, and executing the change task contained in the change task set by utilizing a logic execution strategy preset by the data operation end; and updating the target data contained in the target database according to the execution result of the change task.
That is, after receiving the data change statement of the target statement type, the statement may be parsed by using a parser native to the data operation end, so as to generate a task set according to the parsing result, and then the tasks in the task set may be executed according to the logic execution plan, so as to implement insertion and query processing on the data in the target database.
For example, when receiving the SQL statement associated with the set or Insert, the SQL statement may be parsed by using a grammar parsing rule carried by the Spark SQL client, and the set or Insert operation needs to be performed on the relational database a according to the parsing result.
In conclusion, through supporting the execution of any statement by the data operation end aiming at the target database, the data in the target database can be operated according to the needs, so that the needs in the actual scene are met, and the data can be operated more flexibly.
And S204, analyzing the data change statement according to a user-defined analysis rule, and acquiring metadata information from a metadata management terminal according to an analysis result.
Specifically, after the data change statement is received, considering that the data change statement may be a statement for deleting and updating data in the target database, and the terminal for executing the operation is a data operation terminal, for supporting the data operation terminal to complete any operation on the data in the target database, the data change statement may be analyzed by using a custom analysis rule, so that metadata information is obtained at the metadata management terminal according to the analysis result; the metadata information can be used for loading the data storage table from the database, and the influence of the self architecture of the limited data operation end is not needed, so that the data operation flexibility is improved.
The custom parsing rule specifically refers to parsing rules added according to actual service requirements, and can be added by using Antlr 4; correspondingly, the metadata management end specifically refers to a terminal for managing metadata corresponding to the data storage tables in the target database, and is used for storing metadata information of all the tables, so that when data in the target database is operated, the metadata information can be queried to acquire the corresponding tables to finish the operation. Corresponding to the above. The metadata information specifically refers to data that indexes all tables stored in the target database, so as to query the corresponding tables of the corresponding data, and perform operations such as adding, deleting, modifying and checking the corresponding data fields.
Further, when data operation is performed in the target database, it is required to ensure that the metadata management end stores table information corresponding to a table corresponding to data to be operated, and when the operation is performed, the operation can be completed by reading the table information; in this embodiment, the specific implementation manner is as follows:
extracting a registered form statement from the data change statement for analysis to obtain first form information, and registering the first form information to the metadata management end, wherein the first form information is used for constructing metadata information; extracting a target change sentence from the data change sentence, and analyzing the target change sentence by utilizing a user-defined analysis rule to obtain second form information; and sending an information acquisition request carrying the second table information to the metadata management end, and receiving the metadata information fed back by the metadata management end aiming at the information acquisition request.
Specifically, the register table statement is a statement that indicates that a new table is added to the target database, and the statement is used for registering the new table in the target database and storing data to be stored. Correspondingly, the first table information specifically refers to metadata information of a table to be created in the associated target database, and the metadata information is used for being stored at a metadata management end. Correspondingly, the target change statement specifically refers to a statement submitted when data deletion or update is performed; correspondingly, the second table information is the table information carried in the currently obtained target change statement, and is used for acquiring and storing the information of the metadata information of the data table to be processed at the metadata management end.
Based on the above, after receiving the data change statement, the registered form statement may be extracted from the data change statement to analyze, and at this time, the first form information is obtained, and then the first form information is registered to the metadata management end, so that the metadata information can be built at the metadata management end; when the data change processing is needed, extracting a target change sentence from the data change sentence, and analyzing the target change sentence by utilizing a user-defined analysis rule so as to obtain second form information; and then, sending an information acquisition request carrying second form information to the metadata management end, and if the second form information corresponds to the first form information, indicating that the data needing to be operated at the moment is stored in a data storage table corresponding to the first form information, so that the metadata information can be fed back to the data operation end, and the subsequent use is convenient.
For example, after submitting the SQL statement to the relational database A through the Spark SQL client, the registry SQL statement may be parsed first, the table information is lifted and registered to the metadata management end catalyst, so as to realize that new metadata information is added to the metadata management end catalyst. When receiving the SQL statement corresponding to Update or Delete, the statement can be parsed by using a custom parsing rule, after the table information is extracted, the corresponding metadata information is searched from the metadata management end catalyst, so that the metadata information is convenient to be used for obtaining the corresponding data storage table subsequently, and the Update or Delete operation of the data is performed.
In summary, by using the metadata management end to manage metadata information, when data operation is performed on the target database, the corresponding metadata information is acquired from the metadata management end to load the data storage table, so that any type of operation is supported, and the operation is not limited to the operation in which the data result cannot be changed.
Step S206, loading the data storage table associated with the target database according to the metadata information, and extracting the data field to be changed from the data change statement.
Specifically, after the metadata information is obtained from the metadata management end, further, in order to perform any form of operation on the data in the target database, a data storage table for storing the data to be processed can be loaded according to the metadata information, and the data field to be changed is extracted from the data change statement, so that the corresponding data operation can be completed at the data operation end, and finally, the result is written back to the target database, thereby supporting the completion of the data operation processing without creating JDBC.
The data storage table specifically refers to a table for storing data to be processed based on a data change statement, and may be a physical data table stored in a database or a memory data table temporarily stored in a memory. Correspondingly, the data field to be changed is specifically a data field recorded in the data change statement, and the data field is used for determining the data field to be changed in the data storage table and deleting or updating the data field in the data storage table according to the data field, so that flexible data change processing can be completed at the data operation end.
Further, when the data storage table is loaded, considering that the data to be processed by the data change statement may exist in the physical data table or may exist in the memory data table, in order to ensure that the data change statement can be executed successfully, the data in any table can be operated, and the detection of the data table can be performed based on the metadata information, in this embodiment, the specific implementation manner is as follows:
detecting whether a data table to be changed corresponding to the data change statement exists in the target database or not according to the metadata information; if yes, loading a physical data table in the target database as the data storage table; if not, loading the memory data table associated with the target database as the data storage table.
Specifically, the data table to be changed is a data table corresponding to the data to be processed by the data change statement; correspondingly, the physical data table specifically refers to a table for storing data in the target database; correspondingly, the memory data table specifically refers to a data table in the memory of the computer, which has a mapping relationship with the data table in the target database, and the table exists in the memory.
Based on the above, after obtaining metadata information from the metadata management end, in order to ensure that the deletion or update of the data in the target database is completed after the data is operated, whether the data table to be changed corresponding to the data change statement exists in the target database or not can be detected according to the metadata information; if yes, the data which need to be deleted or updated at the moment are indicated to be directly stored in the physical data table in the database, and the physical data table in the target database can be loaded to be used as a data storage table; if not, the metadata information cannot be directly related to the data table in the target database, and in order to complete the deletion or updating of the data, the memory data table related to the target database can be loaded as a data storage table, so that the subsequent data operation processing based on the loaded data storage table is realized.
In summary, in order to achieve that the operation on the data in the target database can be completed in any state, after the metadata information is acquired, the table storing the data may be loaded to the data operation end, and considering that the metadata information may have a problem of directly accessing the database or being unable to directly access the database, the loading of the physical data table or the memory data table may be performed, so as to support the subsequent data operation processing.
In addition, the data operation end and the target database are not necessarily in an interactive relation with each other in a safe access relation, so that the target database can be accessed by combining the connection information before the data operation, and the data operation is ensured to be completed in a safe environment; in this embodiment, the specific implementation manner is as follows:
determining registration view information according to the data change statement, and acquiring connection information of the target database from the metadata management terminal based on the registration view information; accessing the target database based on the connection information, and in case the access passes, executing step S208.
Specifically, the registration view information is information obtained by the data operation end in a temporary table generation mode, and connection information can be obtained from the metadata management end through the information; correspondingly, the connection information specifically refers to information required to be used for accessing the target database, and the information is used for recording information such as the name, access password and the like of the target database.
Based on the above, before the data storage table is acquired, the registration view information can be determined according to the data change statement, and the connection information of the target database is acquired from the metadata management terminal based on the registration view information; thereafter, the target database is accessed based on the connection information, and in the case of passing of the access, it is indicated that the data manipulation process can be performed at this time, and the subsequent step S208 can be performed.
Along the above example, when submitting an SQL statement corresponding to Update or Delete to the relational database A through the Spark SQL client, determining that the SQL statement is a statement which cannot be resolved by the Spark SQL client based on the native grammar resolution rule, and obtaining connection information { ID } of the relational database A from a metadata management end catalyst of Spark according to the registration view information; and (3) a password: * Based on the above, accessing the relational database A to support the subsequent loading of the corresponding data storage table to perform Update or Delete operation.
In conclusion, the access of the target database is performed by combining the connection information, so that the data operation is completed in a safe environment, and meanwhile, the problem of error in accessing the target database can be avoided.
Step S208, a target updating strategy is selected based on the table type of the data storage table, the data storage table is updated by utilizing the data field to be changed according to the target updating strategy, and the updating result is written back to the target database.
Specifically, after the data fields to be changed are loaded into the data storage table and extracted from the data change statement, further, in order to enable the update of the data storage table to support the write-back operation of the database, a target update strategy can be selected according to the table type of the data storage table, different strategies can be selected for different types of data storage tables, the data storage table is updated according to the selected target update strategy by utilizing the data fields to be changed, the data fields to be deleted or the data fields to be updated are updated, after the data fields in the data storage table are deleted or updated, the update result can be written back to the target database, and the deletion and update processing of the data at the data operation end is completed.
When the data storage table is updated, namely, the data field corresponding to the data field to be changed is searched from the data storage table, then the data field is deleted or updated, the operation is completed in the data storage table, and when the deletion or the updating is completed, a new data storage table is obtained and is written back to the target database. To support the data operation end to perform any type of data operation on the relational database.
In addition, considering that different types of data storage tables correspond to different target update policies, and different policies correspond to different update operations, in order to ensure the update accuracy of the data storage tables written back to the database, different policies may be selected according to the table type, and in this embodiment, the specific implementation manner is as follows:
under the condition that the data storage table is a physical data table, selecting an update strategy in a database as a target update strategy; and under the condition that the data storage table is an internal memory data table, selecting an external database updating strategy as a target updating strategy.
Specifically, the in-database updating policy refers to that in the case that the table type of the data storage table is a physical data table, the policy for updating the data storage table in the database can be selected to complete the updating operation. Correspondingly, the external update strategy of the database can be selected to finish the update operation under the condition that the table type of the data storage table is the memory data table, so that different strategies can be selected to finish the update aiming at two scenes of data update, and the accuracy of data update is ensured.
Further, in the case that the data storage table is a physical data table, it is explained that the data storage table read at this time is an actual data table stored in the target database, so that the data operation process can be directly performed in the target database; in this embodiment, the specific implementation manner is as follows:
creating a data change task corresponding to the data field to be changed based on the update strategy in the database; executing the data change statement in the target database by executing the data change task; and updating the physical data table according to the execution result of the data change statement in the target database.
Specifically, the data change task is specifically a JDBC task created when the data storage table is a physical data table, and is used for implementing execution of a data change statement in the target data, and implementing deletion or update of data in the physical data table.
Based on the above, when the data storage table is a physical data table, it is indicated that the update operation needs to be completed in the database, and then the data change task corresponding to the data field to be changed can be created according to the update strategy in the database; at the moment, by executing the data change task, the execution of the data change statement in the target database can be realized; and deleting or updating the data field corresponding to the data field to be changed according to the execution result of the data change statement in the target database by executing the data change statement in the target database, so as to complete updating of the physical data table. Thus, the update or deletion of the data can be completed in the library.
Along the above example, after metadata information is obtained from a metadata management end catalyst, whether an SQL statement corresponding to Update or Delete is submitted to a relational database A or not can be detected, if yes, a JDBC task can be created at the moment to Delete or Update a field 1, after that, the Update or Delete of the field 1 can be directly performed in the relational database A by executing the task, and then the Update or Delete operation of the Spark SQL client to the relational database A is realized.
In summary, when the data storage table is a physical data table, deletion or update of the data field to be changed can be directly performed in the target database based on the data change statement, so that the data operation end is supported to complete the data operation except the insertion selection in the target data.
Furthermore, in the case that the data storage table is a memory data table, the memory data table needs to be loaded to the data operation end, then data is deleted or updated on the basis of the memory data table, and the updated result is written back to the target database after the completion of the data is completed, and in consideration of the fact that the data is deleted or updated into two different operations, the data is completed by combining different operators; in this embodiment, the first aspect: when the data field to be changed is a data field to be deleted, calling a field deletion operator based on the database external update strategy; deleting the data field related to the data field to be deleted in the memory data table by using the field deletion operator, and generating a first data set according to a deletion result; and converting the first data set into a target memory data table as updating of the data storage table.
Specifically, the data field to be deleted is a data field to be deleted when the data change statement is a data delete statement; correspondingly, the field deletion operator specifically refers to an operator capable of deleting the data field in the data memory table based on the data field to be deleted; correspondingly, the first data set is specifically a data set obtained by deleting the data fields in the memory data table, and the data set can be used as the update of the data storage table by converting the data set into the target memory data table, so that the update result can be written back to the target database. In practical application, when the updated result is written back to the target database, the mapping processing of the data is completed based on the target memory data table, so that the final result written back to the target database is the updated result.
Based on the above, in the case that the data field to be changed is the data field to be deleted and the type of the data storage table is the memory data table, the update operation needs to be completed outside the database, so that the field deletion operator can be called based on the outside-database update strategy; and deleting the data fields related to the data fields to be deleted in the memory data table by using a field deletion operator to generate a first data set according to the deletion result, converting the first data set into a target memory data table on the basis, updating the data storage table, and finally writing back an updating result to the target database to complete the deletion processing of the corresponding data fields.
When the SQL statement is the statement for performing Delete operation on the data, the field 2 to be deleted can be extracted from the SQL statement according to the custom SQL parsing rule, then the Filter operator can be called to Delete and Filter the unnecessary fields in the memory data table, a new DataSet can be generated according to the deleting result, and after the new DataSet is converted into a DataFrame, the result of subsequently writing back the deleted field 2 to the relational database A can be realized.
In summary, the field to be deleted can be removed from the memory data table by performing the deletion processing of the data field in the memory data table by using the field deletion operator, so that the final result written back to the target database is ensured to be the result of deleting the field, and the data operation terminal is supported to perform the data deletion operation on the target database.
Second aspect: when the data field to be changed is a data field to be updated, calling a field updating operator based on the external updating strategy of the database; updating the data field associated with the data field to be updated in the memory data table by using the field updating operator, and generating a second data set according to an updating result; and converting the second data set into an updated memory data table as an update to the data storage table.
Specifically, the data field to be updated is a data field required to be updated when the data change statement is a data update statement; correspondingly, the field updating operator specifically refers to an operator capable of updating the data field in the data memory table based on the data field to be updated. Correspondingly, the second data set is specifically a data set obtained by updating the data fields in the memory data table, and the second data set can be used as an update of the data storage table by converting the second data set into the target memory data table, so that an update result can be written back to the target database.
Based on the above, in the case that the data field to be changed is the data field to be updated, the field update operator can be called based on the off-database update policy at this time; updating the data field associated with the data field to be updated in the memory data table by using a field updating operator, so as to generate a second data set according to an updating result; on the basis, the second data set can be converted into an updated memory data table as the update of the data storage table, and finally the update result is written back to the target database, so that the update processing of the corresponding data field can be completed.
Along the above example, when the SQL statement is a statement for performing Update operation on data, a field 3 needing Update can be extracted from the SQL statement according to a custom SQL parsing rule, a map operator can be called to Update the field 3 in the memory data table, a new DataSet can be generated according to an Update result, and after the new DataSet is converted into a DataFrame, a result of subsequently writing the updated field 3 back to the relational database A can be achieved.
In summary, the field to be updated can be updated in the memory data table by using the field updating operator to update the data field in the memory data table, so that the final result written back to the target database is ensured to be the result of updating the field, and the data operation end is supported to perform the data updating operation on the target database.
In addition, when writing back the update result to the target database, the update may be implemented by using a database connector, and in this embodiment, the specific implementation manner is as follows:
determining a target data storage table according to the updating result; and calling a database connector of the target database, and writing the target data storage table back to the target database through the database connector.
The database connector is specifically a node for loading data by the data operation end and connected with the database. Based on this, determining a target data storage table from the update result; at this point, the database connector of the target database may be invoked, and the target data storage table may be written back to the target database through the database connector.
Referring to the schematic diagram shown in fig. 3, the Spark SQL client may obtain different data storage tables from different types of relational databases through a connector, and table1, table2, table3 and …, then perform Update or Delete operation on the data in the Spark SQL client, and obtain a new data storage table after the operation is completed, that is, a resultatable, convert the new data storage table into resultatable 2, and finally, re-write the new data storage table back to each relational database through the connector.
In order to improve the flexibility of data operation in the relational database and complete data operation according to actual scene requirements, the data processing method provided by the embodiment can analyze the data change statement by utilizing a custom analysis rule under the condition that the data change statement submitted for the target database is received, so that metadata information is obtained at a metadata management end according to an analysis result, and the metadata information is used as a basis of the data operation; then, a data storage table corresponding to the data operation requirement in the current scene can be loaded in the target database according to the metadata information, and the data field to be changed is extracted from the data change statement, so that the data operation can be flexibly completed for any data operation scene; after the data storage table and the data field to be changed are obtained, in order to ensure that any type of operation can be completed for the database, a corresponding target updating strategy can be selected according to the table type of the data storage table, the data storage table is updated by the data field to be changed according to the target updating strategy, and the updating result is written back to the target database. When the data in the database is operated, the processing of any requirement can be completed on the database, so that the data can be used more flexibly to meet the actual requirement.
The application of the data processing method provided in the present application in a relational database scenario is taken as an example in the following description with reference to fig. 4, and the data processing method is further described. Fig. 4 shows a process flow chart of a data processing method according to an embodiment of the present application, which specifically includes the following steps:
step S402, receiving a data change statement submitted for a target database.
Step S404, analyzing the data change statement according to the self-defined analysis rule, acquiring metadata information from the metadata management terminal according to the analysis result, and extracting the data field to be changed from the data change statement.
Step S406, detecting whether a data table to be changed corresponding to the data change statement exists in the target database or not according to the metadata information; if yes, go to step S408; if not, go to step S416.
Step S408, loading a physical data table in the target database as a data storage table;
step S410, creating a data change task corresponding to a data field to be changed based on an update strategy in a database;
step S412, executing the data change statement in the target database by executing the data change task;
step S414, the physical data table is updated according to the execution result of the execution data change statement in the target database.
Step S416, the memory data table associated with the target database is loaded as a data storage table, and the data storage table is updated by utilizing the data field to be changed based on the external update strategy of the database.
On the one hand, under the condition that the data field to be changed is the data field to be deleted, a field deletion operator is called based on an external update strategy of the database; deleting the data field associated with the data field to be deleted in the memory data table by using a field deletion operator, and generating a first data set according to a deletion result; and converting the first data set into a target memory data table as updating the data storage table.
On the other hand, under the condition that the data field to be changed is the data field to be updated, a field updating operator is called based on an external updating strategy of the database; updating the data field associated with the data field to be updated in the memory data table by using a field updating operator, and generating a second data set according to an updating result; and converting the second data set into an updated memory data table as an update to the data storage table.
Step S418, determining a target data storage table according to the updating result, calling a database connector of the target database, and writing the target data storage table back to the target database through the database connector.
In order to improve the flexibility of data operation in the relational database and complete data operation according to actual scene requirements, the data processing method provided by the embodiment can analyze the data change statement by utilizing a custom analysis rule under the condition that the data change statement submitted for the target database is received, so that metadata information is obtained at a metadata management end according to an analysis result, and the metadata information is used as a basis of the data operation; then, a data storage table corresponding to the data operation requirement in the current scene can be loaded in the target database according to the metadata information, and the data field to be changed is extracted from the data change statement, so that the data operation can be flexibly completed for any data operation scene; after the data storage table and the data field to be changed are obtained, in order to ensure that any type of operation can be completed for the database, a corresponding target updating strategy can be selected according to the table type of the data storage table, the data storage table is updated by the data field to be changed according to the target updating strategy, and the updating result is written back to the target database. When the data in the database is operated, the processing of any requirement can be completed on the database, so that the data can be used more flexibly to meet the actual requirement.
Corresponding to the method embodiment, the present application further provides an embodiment of a data processing device, and fig. 5 shows a schematic structural diagram of a data processing device according to an embodiment of the present application. As shown in fig. 5, the apparatus is applied to a data operation end, and includes:
a receiving module 502 configured to receive a data change statement submitted for a target database;
the parsing module 504 is configured to parse the data change statement according to a custom parsing rule, and obtain metadata information at a metadata management end according to a parsing result;
a loading module 506 configured to load a data storage table associated with the target database according to the metadata information, and extract a data field to be changed in the data change statement;
and the updating module 508 is configured to select a target updating strategy based on the table type of the data storage table, update the data storage table by utilizing the data field to be changed according to the target updating strategy, and write the updating result back to the target database.
In an alternative embodiment, the parsing module 504 is further configured to:
extracting a registered form statement from the data change statement for analysis to obtain first form information, and registering the first form information to the metadata management end, wherein the first form information is used for constructing metadata information; extracting a target change sentence from the data change sentence, and analyzing the target change sentence by utilizing a user-defined analysis rule to obtain second form information; and sending an information acquisition request carrying the second table information to the metadata management end, and receiving the metadata information fed back by the metadata management end aiming at the information acquisition request.
In an alternative embodiment, the loading module 506 is further configured to:
detecting whether a data table to be changed corresponding to the data change statement exists in the target database or not according to the metadata information; if yes, loading a physical data table in the target database as the data storage table; if not, loading a memory data table associated with the target database as the data storage table;
the update module 508 is further configured to:
under the condition that the data storage table is a physical data table, selecting an update strategy in a database as a target update strategy; and under the condition that the data storage table is an internal memory data table, selecting an external database updating strategy as a target updating strategy.
In an alternative embodiment, in the case where the data storage table is an in-memory data table, the updating module 508 is further configured to:
when the data field to be changed is a data field to be deleted, calling a field deletion operator based on the database external update strategy; deleting the data field related to the data field to be deleted in the memory data table by using the field deletion operator, and generating a first data set according to a deletion result; and converting the first data set into a target memory data table as updating of the data storage table.
In an alternative embodiment, in the case where the data storage table is an in-memory data table, the updating module 508 is further configured to:
when the data field to be changed is a data field to be updated, calling a field updating operator based on the external updating strategy of the database; updating the data field associated with the data field to be updated in the memory data table by using the field updating operator, and generating a second data set according to an updating result; and converting the second data set into an updated memory data table as an update to the data storage table.
In an alternative embodiment, in the case where the data storage table is a physical data table, the update module 508 is further configured to:
creating a data change task corresponding to the data field to be changed based on the update strategy in the database; executing the data change statement in the target database by executing the data change task; and updating the physical data table according to the execution result of the data change statement in the target database.
In an alternative embodiment, the apparatus further comprises:
The access module is configured to determine registration view information according to the data change statement and acquire connection information of the target database from the metadata management terminal based on the registration view information; and accessing the target database based on the connection information, and executing the step of selecting a target update strategy based on the table type of the data storage table if the access is passed.
In an alternative embodiment, the apparatus further comprises:
the execution task module is configured to analyze the data change statement by utilizing a standard analysis rule under the condition that the statement type of the data change statement is a target statement type; generating a change task set according to the analysis result, and executing the change task contained in the change task set according to a logic execution strategy; and updating the target data contained in the target database according to the execution result of the change task.
In an alternative embodiment, the update module 508 is further configured to:
determining a target data storage table according to the updating result; and calling a database connector of the target database, and writing the target data storage table back to the target database through the database connector.
In order to improve the flexibility of data operation in a relational database and complete data operation according to actual scene requirements, the data processing device provided by the embodiment can analyze the data change statement by utilizing a custom analysis rule under the condition of receiving the data change statement submitted for a target database, so as to obtain metadata information at a metadata management end according to an analysis result, and the metadata information is used as a basis of the data operation; then, a data storage table corresponding to the data operation requirement in the current scene can be loaded in the target database according to the metadata information, and the data field to be changed is extracted from the data change statement, so that the data operation can be flexibly completed for any data operation scene; after the data storage table and the data field to be changed are obtained, in order to ensure that any type of operation can be completed for the database, a corresponding target updating strategy can be selected according to the table type of the data storage table, the data storage table is updated by the data field to be changed according to the target updating strategy, and the updating result is written back to the target database. When the data in the database is operated, the processing of any requirement can be completed on the database, so that the data can be used more flexibly to meet the actual requirement.
The above is a schematic solution of a data processing apparatus of the present embodiment. It should be noted that, the technical solution of the data processing apparatus and the technical solution of the data processing method belong to the same conception, and details of the technical solution of the data processing apparatus, which are not described in detail, can be referred to the description of the technical solution of the data processing method. Furthermore, the components in the apparatus embodiments should be understood as functional blocks that must be established to implement the steps of the program flow or the steps of the method, and the functional blocks are not actually functional partitions or separate limitations. The device claims defined by such a set of functional modules should be understood as a functional module architecture for implementing the solution primarily by means of the computer program described in the specification, and not as a physical device for implementing the solution primarily by means of hardware.
Fig. 6 illustrates a block diagram of a computing device 600 provided in accordance with an embodiment of the present application. The components of computing device 600 include, but are not limited to, memory 610 and processor 620. The processor 620 is coupled to the memory 610 via a bus 630 and a database 650 is used to hold data.
Computing device 600 also includes access device 640, access device 640 enabling computing device 600 to communicate via one or more networks 660. Examples of such networks include public switched telephone networks (PSTN, public Switched Telephone Network), local area networks (LAN, local Area Network), wide area networks (WAN, wide Area Network), personal area networks (PAN, personal Area Network), or combinations of communication networks such as the internet. The access device 640 may include one or more of any type of network interface, wired or wireless, such as a network interface card (NIC, network interface controller), such as an IEEE802.11 wireless local area network (WLAN, wireless Local Area Network) wireless interface, a worldwide interoperability for microwave access (Wi-MAX, worldwide Interoperability for Microwave Access) interface, an ethernet interface, a universal serial bus (USB, universal Serial Bus) interface, a cellular network interface, a bluetooth interface, a near field communication (NFC, near Field Communication) interface, and so forth.
In one embodiment of the present application, the above-described components of computing device 600, as well as other components not shown in FIG. 6, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device illustrated in FIG. 6 is for exemplary purposes only and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as desired.
Computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smart phone), wearable computing device (e.g., smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or personal computer (PC, personal Computer). Computing device 600 may also be a mobile or stationary server.
Wherein the processor 620 is configured to execute computer-executable instructions of the data processing method.
The foregoing is a schematic illustration of a computing device of this embodiment. It should be noted that, the technical solution of the computing device and the technical solution of the data processing method belong to the same concept, and details of the technical solution of the computing device, which are not described in detail, can be referred to the description of the technical solution of the data processing method.
An embodiment of the present application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, are used in a data processing method.
The above is an exemplary version of a computer-readable storage medium of the present embodiment. It should be noted that, the technical solution of the storage medium and the technical solution of the data processing method belong to the same concept, and details of the technical solution of the storage medium which are not described in detail can be referred to the description of the technical solution of the data processing method.
The computer instructions include computer program code that may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the content of the computer readable medium can be increased or decreased appropriately according to the requirements of the patent practice, for example, in some areas, according to the patent practice, the computer readable medium does not include an electric carrier signal and a telecommunication signal.
It should be noted that, for the sake of simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the present application is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all necessary for the present application.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.
The above-disclosed preferred embodiments of the present application are provided only as an aid to the elucidation of the present application. Alternative embodiments are not intended to be exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the teaching of this application. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. This application is to be limited only by the claims and the full scope and equivalents thereof.
Claims (11)
1. A data processing method, applied to a data operation end, comprising:
receiving a data change statement submitted for a target database;
analyzing the data change statement according to a user-defined analysis rule, and acquiring metadata information at a metadata management end according to an analysis result;
loading a data storage table associated with the target database according to the metadata information, and extracting a data field to be changed from the data change statement;
and selecting a target updating strategy based on the table type of the data storage table, updating the data storage table by utilizing the data field to be changed according to the target updating strategy, determining a target data storage table according to an updating result, calling a database connector of the target database, and writing the target data storage table back to the target database through the database connector.
2. The data processing method according to claim 1, wherein the parsing the data change statement according to the custom parsing rule, and obtaining metadata information at a metadata management terminal according to a parsing result, includes:
extracting a registered form statement from the data change statement for analysis to obtain first form information, and registering the first form information to the metadata management end, wherein the first form information is used for constructing metadata information;
Extracting a target change sentence from the data change sentence, and analyzing the target change sentence by utilizing a user-defined analysis rule to obtain second form information;
and sending an information acquisition request carrying the second table information to the metadata management end, and receiving the metadata information fed back by the metadata management end aiming at the information acquisition request.
3. The data processing method according to claim 1, wherein said loading the data storage table associated with the target database according to the metadata information comprises:
detecting whether a data table to be changed corresponding to the data change statement exists in the target database or not according to the metadata information;
if yes, loading a physical data table in the target database as the data storage table;
if not, loading a memory data table associated with the target database as the data storage table;
the selecting a target update policy based on a table type of the data storage table includes:
under the condition that the data storage table is a physical data table, selecting an update strategy in a database as a target update strategy;
and under the condition that the data storage table is an internal memory data table, selecting an external database updating strategy as a target updating strategy.
4. A data processing method according to claim 3, wherein, in the case where the data storage table is an in-memory data table, the updating the data storage table with the data field to be changed according to the target update policy includes:
when the data field to be changed is a data field to be deleted, calling a field deletion operator based on the database external update strategy;
deleting the data field related to the data field to be deleted in the memory data table by using the field deletion operator, and generating a first data set according to a deletion result;
and converting the first data set into a target memory data table as updating of the data storage table.
5. A data processing method according to claim 3, wherein, in the case where the data storage table is an in-memory data table, the updating the data storage table with the data field to be changed according to the target update policy includes:
when the data field to be changed is a data field to be updated, calling a field updating operator based on the external updating strategy of the database;
updating the data field associated with the data field to be updated in the memory data table by using the field updating operator, and generating a second data set according to an updating result;
And converting the second data set into an updated memory data table as an update to the data storage table.
6. A data processing method according to claim 3, wherein, in the case where the data storage table is a physical data table, the updating the data storage table with the data field to be changed according to the target update policy includes:
creating a data change task corresponding to the data field to be changed based on the update strategy in the database;
executing the data change statement in the target database by executing the data change task;
and updating the physical data table according to the execution result of the data change statement in the target database.
7. The data processing method according to claim 1, wherein before the step of selecting a target update policy based on a table type of the data storage table is performed, further comprising:
determining registration view information according to the data change statement, and acquiring connection information of the target database from the metadata management terminal based on the registration view information;
and accessing the target database based on the connection information, and executing the step of selecting a target updating strategy based on the table type of the data storage table in the case that the access is passed.
8. The method according to any one of claims 1 to 7, wherein after the step of receiving the data change statement submitted for the target database is performed, further comprising:
analyzing the data change sentence by using a reference analysis rule under the condition that the sentence type of the data change sentence is the target sentence type;
generating a change task set according to the analysis result, and executing the change task contained in the change task set according to a logic execution strategy;
and updating the target data contained in the target database according to the execution result of the change task.
9. A data processing apparatus, for application to a data manipulation terminal, comprising:
a receiving module configured to receive a data change statement submitted for a target database;
the analysis module is configured to analyze the data change statement according to a custom analysis rule and acquire metadata information at a metadata management end according to an analysis result;
the loading module is configured to load a data storage table associated with the target database according to the metadata information and extract a data field to be changed in the data change statement;
And the updating module is configured to select a target updating strategy based on the table type of the data storage table, update the data storage table by utilizing the data field to be changed according to the target updating strategy, determine a target data storage table according to an updating result, call a database connector of the target database, and write the target data storage table back to the target database through the database connector.
10. A computing device, comprising:
a memory and a processor;
the memory is configured to store computer executable instructions and the processor is configured to execute the computer executable instructions to implement the steps of the method of any one of claims 1 to 8.
11. A computer readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311331566.8A CN117093597B (en) | 2023-10-16 | 2023-10-16 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311331566.8A CN117093597B (en) | 2023-10-16 | 2023-10-16 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117093597A CN117093597A (en) | 2023-11-21 |
| CN117093597B true CN117093597B (en) | 2024-01-23 |
Family
ID=88782112
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311331566.8A Active CN117093597B (en) | 2023-10-16 | 2023-10-16 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117093597B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108073625A (en) * | 2016-11-14 | 2018-05-25 | 北京京东尚科信息技术有限公司 | For the system and method for metadata information management |
| CN109857535A (en) * | 2019-02-18 | 2019-06-07 | 国家计算机网络与信息安全管理中心 | The implementation method and device of task priority control towards Spark JDBC |
| CN111291037A (en) * | 2018-11-22 | 2020-06-16 | 中国移动通信集团重庆有限公司 | Data storage and query method, device, equipment and computer storage medium |
| CA3065863A1 (en) * | 2018-12-21 | 2020-06-21 | Christopher Johnson | Systems and methods for implementing a secure database for storing a patient operational longitudinal record |
| CN113377789A (en) * | 2021-05-31 | 2021-09-10 | 上海数禾信息科技有限公司 | Processing method and device for database change data, computer equipment and medium |
| GB202115084D0 (en) * | 2020-11-13 | 2021-12-08 | Ibm | Data synchronization in a data analysis system |
| CN115237903A (en) * | 2021-04-22 | 2022-10-25 | 中国移动通信集团安徽有限公司 | Wide table generation method, device, equipment and storage medium |
| CN116578456A (en) * | 2023-04-04 | 2023-08-11 | 中国工商银行股份有限公司 | Data maintenance rechecking method, device, computer equipment, storage medium and product |
-
2023
- 2023-10-16 CN CN202311331566.8A patent/CN117093597B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108073625A (en) * | 2016-11-14 | 2018-05-25 | 北京京东尚科信息技术有限公司 | For the system and method for metadata information management |
| CN111291037A (en) * | 2018-11-22 | 2020-06-16 | 中国移动通信集团重庆有限公司 | Data storage and query method, device, equipment and computer storage medium |
| CA3065863A1 (en) * | 2018-12-21 | 2020-06-21 | Christopher Johnson | Systems and methods for implementing a secure database for storing a patient operational longitudinal record |
| CN109857535A (en) * | 2019-02-18 | 2019-06-07 | 国家计算机网络与信息安全管理中心 | The implementation method and device of task priority control towards Spark JDBC |
| GB202115084D0 (en) * | 2020-11-13 | 2021-12-08 | Ibm | Data synchronization in a data analysis system |
| CN115237903A (en) * | 2021-04-22 | 2022-10-25 | 中国移动通信集团安徽有限公司 | Wide table generation method, device, equipment and storage medium |
| CN113377789A (en) * | 2021-05-31 | 2021-09-10 | 上海数禾信息科技有限公司 | Processing method and device for database change data, computer equipment and medium |
| CN116578456A (en) * | 2023-04-04 | 2023-08-11 | 中国工商银行股份有限公司 | Data maintenance rechecking method, device, computer equipment, storage medium and product |
Non-Patent Citations (2)
| Title |
|---|
| 基于嵌入式数据库的同步系统的设计与实现;刘谋黎;;自动化应用(第12期);全文 * |
| 基于混合架构的高校数据平台构建研究;张军;王芬芬;;当代教育实践与教学研究(第14期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117093597A (en) | 2023-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109492053B (en) | Method and device for accessing data | |
| CN110851209B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN108334609B (en) | Method, device, equipment and storage medium for realizing JSON format data access in Oracle | |
| CN111259067B (en) | Method, device and equipment for realizing DAO interface based on Spring | |
| CN109669925B (en) | Management method and device of unstructured data | |
| CN113204571B (en) | SQL execution method and device related to write-in operation and storage medium | |
| CN109710220B (en) | Relational database query method, relational database query device, relational database query equipment and storage medium | |
| CN113297320A (en) | Distributed database system and data processing method | |
| CN109815240A (en) | For managing method, apparatus, equipment and the storage medium of index | |
| CN113138781B (en) | CSV configuration updating method and storage medium | |
| CN111813744A (en) | File searching method, device, equipment and storage medium | |
| CN113239063B (en) | Graph database updating method and device | |
| CN117093597B (en) | Data processing method and device | |
| CN116361287A (en) | Path analysis method, device and system | |
| CN115114325B (en) | Data query method and device, electronic equipment and storage medium | |
| CN111125216A (en) | Method and device for importing data into Phoenix | |
| CN116049193A (en) | Data storage method and device | |
| CN116010345A (en) | Method, device and equipment for realizing table service scheme of flow batch integrated data lake | |
| CN110222105B (en) | Data summarization processing method and device | |
| CN114860727A (en) | Zipper watch updating method and device | |
| CN110471708B (en) | Method and device for acquiring configuration items based on reusable components | |
| CN113297230B (en) | Data verification method and device | |
| WO2019153547A1 (en) | Database operation method, apparatus and device, and computer-readable storage medium | |
| CN111061721B (en) | Data processing method and device | |
| CN113297199B (en) | Method and device for using spatiotemporal data engine and Cassandra database system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |