CN114490717A - Data processing method and device, equipment and storage medium - Google Patents

Data processing method and device, equipment and storage medium Download PDF

Info

Publication number
CN114490717A
CN114490717A CN202210122309.2A CN202210122309A CN114490717A CN 114490717 A CN114490717 A CN 114490717A CN 202210122309 A CN202210122309 A CN 202210122309A CN 114490717 A CN114490717 A CN 114490717A
Authority
CN
China
Prior art keywords
data
operation message
syntax
storage system
data operation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210122309.2A
Other languages
Chinese (zh)
Inventor
李超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Wodong Tianjun Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Wodong Tianjun Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Wodong Tianjun Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN202210122309.2A priority Critical patent/CN114490717A/en
Publication of CN114490717A publication Critical patent/CN114490717A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/242—Query formulation
    • G06F16/2433—Query languages
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application discloses a data processing method, which comprises the following steps: sending a received first data operation message to a first data access layer, wherein the first data operation message relates to a writing method, the syntax of the first data operation message is a first syntax, and the first data access layer corresponds to a first data storage system supporting the first syntax; calling a first interface through the first data access layer to send the first data operation message to the first data storage system through the first interface; determining a second data operation message obtained after syntax conversion is carried out on the first data operation message through a data synchronization program, wherein the syntax of the second data operation message is a second syntax; and sending the second data operation message to a second data storage system supporting a second syntax through the called second interface. The application also discloses another data processing method, and a device, equipment and a storage medium for realizing the data processing method.

Description

Data processing method and device, equipment and storage medium
Technical Field
The embodiment of the application relates to the technical field of computers, and relates to but is not limited to a data processing method, a data processing device, data processing equipment and a data processing storage medium.
Background
With the rapid increase of business data volume and the large query volume and computation complexity, a relational database management system (mySQL) has difficulty in supporting the existing business volume, and in order to expand the capacity, the current solution is to divide a library into tables (sharing)) to complete the horizontal expansion, however, the problem brought after the division of the library into tables is that a routing field needs to be carried in the query, so that the cross-library is complex, and the efficiency of linked table query and data statistics is low, and at this time, the problem of low efficiency can be solved by using a full-text search engine (elastic search) with distributed multi-user capability.
However, after the search engine is introduced, the developer needs to relearn an Application Programming Interface (API) of the Elasticsearch, which results in an increase in learning cost and development cost.
Disclosure of Invention
Embodiments of the present application provide a data processing method and apparatus, a device, and a storage medium for solving at least one problem in the related art, which can reduce the development cost of a database.
The technical scheme of the embodiment of the application is realized as follows:
in a first aspect, an embodiment of the present application provides a data processing method, which is applied to a client, and the method includes:
sending a received first data operation message to a first data access layer, wherein the first data operation message relates to a writing method, the syntax of the first data operation message is a first syntax, and the first data access layer corresponds to a first data storage system;
calling a first interface through the first data access layer to send the first data operation message to the first data storage system through the first interface, the first data storage system supporting the first syntax;
determining a second data operation message obtained after syntax conversion is carried out on the first data operation message, wherein the syntax of the second data operation message is a second syntax;
sending the second data operation message to a second data storage system through a called second interface to keep the first data storage system and the second data storage system synchronized, wherein the second data storage system supports the second syntax.
In a second aspect, an embodiment of the present application provides a data processing apparatus, including:
the first sending module is used for sending a received first data operation message to a first data access layer, wherein the first data operation message relates to a writing method, the syntax of the first data operation message is first syntax, and the first data access layer corresponds to a first data storage system;
a first calling module, configured to call a first interface through the first data access layer, so as to send the first data operation message to the first data storage system through the first interface, where the first data storage system supports the first syntax;
the determining module is used for determining a second data operation message obtained after syntax conversion is carried out on the first data operation message, wherein the syntax of the second data operation message is a second syntax;
and the second calling module is used for sending the second data operation message to a second data storage system through a called second interface so as to keep the first data storage system and the second data storage system synchronous, and the second data storage system supports the second syntax.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the steps in the data processing method provided by the client or the server when executing the computer program.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps in the data processing method provided by the client or the server.
In an embodiment of the present application, a data processing method, an apparatus, a device, and a storage medium are provided, including: sending a received first data operation message to a first data access layer, wherein the first data operation message relates to a writing method, the syntax of the first data operation message is a first syntax, and the first data access layer corresponds to a first data storage system; calling a first interface through the first data access layer to send the first data operation message to the first data storage system through the first interface, the first data storage system supporting the first syntax; determining a second data operation message obtained after syntax conversion is carried out on the first data operation message through a data synchronization program, wherein the syntax of the second data operation message is a second syntax; and sending the second data operation message to a second data storage system through a called second interface so as to keep the first data storage system and the second data storage system synchronous, wherein the second data storage system supports the second syntax, and when a first data operation message related to a write operation is executed through the first syntax, the second data operation message different from the syntax of the first data operation message is determined through a data synchronization program, the first data operation message is executed in the first data storage system, and the second data operation message is executed in the second data storage system, so that the processing of data in different data sources by the same operation request is realized, and developers are not required to master the second syntax on the basis of mastering the first syntax, thereby reducing the development cost of the database.
Drawings
FIG. 1 is a schematic diagram of an alternative architecture of a data processing system according to an embodiment of the present application;
FIG. 2 is a schematic diagram of an alternative architecture of a data processing system according to an embodiment of the present application;
fig. 3 is an alternative schematic flow chart of a data processing method provided in an embodiment of the present application;
fig. 4 is an alternative schematic structural diagram of a data processing apparatus provided in an embodiment of the present application;
fig. 5 is a schematic diagram of an alternative structure of a data processing apparatus according to an embodiment of the present application;
fig. 6 is an alternative schematic flow chart of a data processing method provided in an embodiment of the present application;
fig. 7 is an alternative schematic flow chart of a data processing method provided in an embodiment of the present application;
fig. 8 is a schematic diagram of an alternative structure of a data processing apparatus according to an embodiment of the present application;
fig. 9 is an optional structural schematic diagram of an electronic device provided in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will describe the specific technical solutions of the present application in further detail with reference to the accompanying drawings in the embodiments of the present application. The following examples are intended to illustrate the present application but are not intended to limit the scope of the present application.
Embodiments of the present application may provide a data processing method and system, and a storage medium. In practical applications, the data processing method may be implemented by a data processing system, and each functional entity in the data processing system may be cooperatively implemented by hardware resources of an electronic device (such as a terminal device or a server), such as computing resources like a processor, and communication resources (such as for supporting communications in various manners like optical cables and cellular).
Some terms referred to in the embodiments of the present application are described.
1) Data storage systems, systems for storing data, such as: mySQL, Elastic Search (ES).
2) mySQL, which is a relational database management system, uses a syntax of Structured Query Syntax (SQL).
3) An elastic search is a distributed multi-user-capability full-Text search engine, which indexes data by using a JavaScript Object Notation (JSON) through a hypertext Transfer Protocol (HTTP).
4) The Data Access layer, namely a Database Access Objects (DAO) layer, is used for performing Data operations, and the DAO layer has an interface and an implementation class of the interface.
The data processing method according to the embodiment of the present application may be applied to the data processing system 10 shown in fig. 1, and includes: a client 101 and a server 102, wherein, a program capable of displaying pages is run in the client 101, such as: a browser or application program (APP) capable of receiving a user initiated data manipulation message. The server can receive the data operation message sent by the client and process the data in the data storage system inside the server or the data storage system connected with the server. The operation types of the client capable of receiving the data operation message of the user may include operations of adding, modifying, deleting and the like, and at this time, the server performs operations of adding, modifying, deleting and the like on the data stored in the data storage system. Here, the client 101 and the server 102 can communicate with each other through the network 103.
In an embodiment of the present application, a data storage system may include a first data storage system and a second data storage system, where the syntax supported by the first data storage system and the syntax supported by the second data storage system are different, that is, the syntax supported by the first data storage system is different, where the first data storage system supports a first syntax and the second data storage system supports a second syntax. In an example, the first data storage system is mySQL and the second data storage system is ES.
In one example, as shown in FIG. 2, a first data storage system 104 and a second data storage system 105 are connected to the server.
The server 101 can perform the following steps: sending a received first data operation message to a first data access layer, wherein the first data operation message relates to a writing method, the syntax of the first data operation message is a first syntax, and the first data access layer corresponds to a first data storage system; calling a first interface through the first data access layer to send the first data operation message to the first data storage system through the first interface, the first data storage system supporting the first syntax; determining a second data operation message obtained after syntax conversion is carried out on the first data operation message, wherein the syntax of the second data operation message is a second syntax; sending the second data operation message to a second data storage system through a called second interface to keep the first data storage system and the second data storage system synchronized, wherein the second data storage system supports the second syntax.
In an embodiment of the present application, the server may be any electronic device with an information processing capability, and in an embodiment, the electronic device may be an intelligent terminal, for example, a mobile terminal with a wireless communication capability, such as a notebook. In another embodiment, the electronic device may also be a computing-enabled terminal device that is not mobile, such as a desktop computer, a server, etc.
By combining the data processing system, the embodiment of the application provides a data processing method, which can reduce the development cost of a database.
Embodiments of a data processing method, an apparatus, a device, and a storage medium according to the embodiments of the present application are described below with reference to the data processing system shown in fig. 1 or fig. 2.
The embodiment of the application provides a data processing method. The functions implemented by the method can be implemented by calling program code by a processor in an electronic device, and the program code can be stored in a computer storage medium.
Fig. 3 is a schematic diagram of an implementation flow of a data processing method according to an embodiment of the present application, and is applied to a server, as shown in fig. 3, the method may include the following steps:
s301, the server side sends the received first data operation message to a first data access layer, the first data operation message relates to a writing method, the syntax of the first data operation message is first syntax, and the first data access layer corresponds to a first data storage system.
The server side can receive the data operation message sent by the client side and judge whether the received data operation message relates to a writing method, and under the condition that the writing method is related, the server side takes the data operation message as a first data operation message sent to the first data access layer and sends the data operation message to the first data access layer.
Here, the first data operation message relating to the write method is for requesting a write operation to be performed in the first data storage system to modify data in the first data storage system. The operation of the first data transmission operation request in the embodiment of the present application may include: add operations, modify operations, delete operations, etc. The first data storage system may be mySQL, in which case the first syntax is SQL.
S302, the server side calls a first interface through the first data access layer so as to send the first data operation message to the first data storage system through the first interface, and the first data storage system supports the first syntax.
The first data access layer may be a mySQL data access layer. The first interface is an interface provided by the first data access layer for accessing the first data storage system, and after receiving the first data operation message, the first data access layer sends the first data operation message to the first data operation system through the first interface.
In practical application, the first data access layer may send the first data operation message to the data access agent, the data access agent is provided with an API of the first data storage system, at this time, the data access agent sends the first data operation message to the first data storage system through the API of mySQL, and the first data storage system executes an operation corresponding to the first data operation message on the data.
And S303, the server determines a second data operation message obtained by syntax conversion of the first data operation message through a data synchronization program, wherein the syntax of the second data operation message is a second syntax.
The data synchronization program can determine a second data operation message, wherein the second data operation message has an operation type that is different from the first data operation message, and a syntax that is different from the syntax of the first data operation message.
Here, the data synchronization program can execute the log file of the first data operation message based on the first data operation message or the first data storage system, resulting in a second data operation message, wherein the syntax of the second data operation message is different from that of the first data operation message and the operated data storage system, but the characterized operation is the same. The syntax of the second data operation is a second syntax. In an example, the second syntax is an ES syntax.
In an example, the first data operation message is for writing data a in a first data storage system and the second data operation message is for writing data a in a second data storage system.
In an example, the first data operation message is used to delete data C of location B in the first data storage system and the second data operation message is used to delete data C of location B in the second data storage system.
Here, the synchronization of the first data storage system and the second data storage system is achieved based on the operation of the first data operation message on the first data storage system and the operation of the second data operation message on the second data storage system.
The data synchronization program can determine a second data operation message based on a mapping of a first data structure in a first syntax and a second data structure in a second syntax.
In an example, the data synchronization program converts the first data operation message into the second data operation message through a mapping relationship between the first data structure and the second data structure.
In an example, the data synchronization program converts the log file of the first data storage system executing the first data operation message into the second data operation message through a mapping relationship between the first data structure and the second data structure.
In the embodiment of the present application, the first data structure and the second data structure may be a mapping relationship performed between the first data structure and the second data structure, or may be a mapping relationship by using a third statement in a third syntax. In an example, a mapping relationship exists between the first data structure and the third statement, and a mapping relationship exists between the third statement and the second data structure, so that the mapping relationship between the first data structure and the second data structure is realized through the third statement.
S304, the server side sends the second data operation message to a second data storage system through a called second interface so as to keep the first data storage system and the second data storage system synchronous, and the second data storage system supports the second syntax.
The data synchronization program can provide a second interface for accessing the second data storage system, and the server sends the second data operation message converted by the data synchronization program to the second data storage system through the second interface so as to execute the operation corresponding to the first data operation message of the first data storage system in the second data storage system, thereby maintaining the synchronization of the first data storage system and the second data storage system.
The second data storage system may be an ES.
And after determining the second data operation message, the data synchronization program sends the second data operation message to the second data storage system through the API of the second data storage system, so that the second data storage system can identify the second data operation message to execute the same operation as the first data storage system in the second data storage system.
In an example, the first data storage system performs an add operation based on the first data operation message, and the second data storage system performs an add operation based on the second data operation message.
In an example, the first data storage system performs the modify operation based on the first data operation message, and the second data storage system performs the modify operation based on the second data operation message.
In an example, the first data storage system performs a delete operation based on the first data operation message, and the second data storage system performs the delete operation based on the second data operation message.
According to the data processing method provided by the embodiment of the application, the received first data operation message is sent to a first data access layer, the first data operation message relates to a writing method, the syntax of the first data operation message is a first syntax, and the first data access layer corresponds to a first data storage system; calling a first interface through the first data access layer to send the first data operation message to the first data storage system through the first interface, the first data storage system supporting the first syntax; determining a second data operation message obtained after syntax conversion is carried out on the first data operation message through a data synchronization program, wherein the syntax of the second data operation message is a second syntax; sending the second data operation message to a second data storage system through the invoked second interface, to keep the first data storage system synchronized with the second data storage system, the second data storage system supporting the second syntax, where, when a first data operation message involving a write operation is executed in a first syntax, determining, by a data synchronization program, a second data operation message in a different syntax than the first data operation message, and executing the first data operation message in a first data storage system, and executing the second data operation message in the second data storage system, realizing the processing of the data in different data sources by the same operation request, and realizing the processing of the data in different data sources by the same operation request, without a developer mastering the second grammar on the basis of mastering the first grammar, thereby reducing the development cost of the database.
As shown in fig. 4, the server 40 includes a data access layer (DAO)401 and a data synchronization program 402, where the DAO401 includes a first DAO4011 and a second DAO4012, the first DAO4011 corresponds to the first data storage system 41, and the second DAO4012 corresponds to the second data storage system 42 through the data synchronization program 402.
In some embodiments, before S301, the server further performs the following steps:
generating the first data access layer based on a first template that supports the first syntax.
In the embodiment of the application, the server includes a code generator, the code generator includes a first configuration file for generating the first data access layer, and the first configuration file includes first configuration data for generating the first data access layer.
The code generator is also configured with a first template and a code generation engine that generates a first data access layer based on the first configuration data and the first template.
In some embodiments, the step S302 of determining, by the data synchronization program, implementation of the second data operation message obtained by performing syntax transformation on the first data operation message includes:
S3021A, sending the first data operation message to a second data access layer, where the second data access layer corresponds to the second data storage system;
S3022A, sending the first data operation message to the data synchronization program through the second data access layer;
S3023A, converting the first data operation message into the second data operation message through the data synchronization program.
Here, the server sends the first data operation message to the first data access layer and sends the first data operation message to the second data access layer, the second data access layer sends the first data operation message to the data synchronization program, and the data synchronization program directly converts the first data operation message into the second data operation message.
In practical application, the second data access layer may send the first data operation message to a message queue, and the data synchronization program consumes the first data operation message in the message queue and converts the consumed first data operation message into the second data operation message.
Wherein the message may be an asynchronous message queue.
Here, the data synchronization program message may parse the first data operation message, parse table and field information carried by the first data operation message, and convert the parsed table and field information into an entity object of the first data storage system.
In some embodiments, prior to S3021A, the following steps are also performed:
generating the second data access layer based on a second template that supports the second syntax.
In the embodiment of the application, the server includes a code generator, the code generator includes a second configuration file for generating the second data access layer, and the second configuration file includes second configuration data for generating the second data access layer.
The code generator is also configured with a second template and a code generation engine that generates a second data access layer based on the second configuration data and the second template.
In practical applications, the first configuration file and the second configuration file may be the same configuration file, and the configuration file includes a mapping relationship between a first data structure using the first syntax and a second data structure using the second syntax.
In one example, the first syntax is mySQL syntax and the second syntax is ES syntax, and the mapping relationship is the mapping relationship between the columns in the table of mySQL and the fields of ES.
In some embodiments, the S302 determines, by the data synchronization program, implementation of the second data operation message obtained by syntax transformation of the first data operation message, and includes:
S3021B, acquiring a log file of the first data storage system executing the first data operation message;
S3022B, sending the log file to the data synchronization program;
S3023B, converting the log file into the second data operation message through the data synchronization program.
Here, the server sends the log file of the first data storage system executing the first data operation message to the data synchronization program, the data synchronization program analyzes the log file to obtain the table and field information carried by the log file, and the analyzed table and field information converts the entity object of the first data storage system into the entity object of the second data storage system.
In the embodiment of the application, after the first data storage system executes the first data operation message, the second data operation message corresponding to the first data operation message is determined based on the log file executed by the first data operation message, and the same data operation as the first data operation message in the first data storage system is executed on the second data storage system based on the second data operation message, so that asynchronous decoupling of operations of the first data storage system and the second data storage system is realized, and the performance requirement on a server is reduced under the condition that data synchronization of different data storage systems is ensured.
In some embodiments, the S3023B, the performing, by the data synchronization program, to convert the log file into the second data operation message includes: analyzing the log file to obtain parameters in the log file; and determining the second data operation message based on the parameters and the set mapping relation, wherein the mapping relation is the corresponding relation between the data structure of the first syntax and the data structure of the second syntax.
The data synchronization program analyzes the log file to obtain parameters including tables, fields and the like in the log file, determines operation information corresponding to the analyzed parameters based on a mapping relation between a first data structure adopting a first grammar and a second data structure adopting a second grammar, namely a corresponding relation between the data structure of the first grammar and the data structure of the second grammar, and determines a second data operation message based on the determined operation information.
In some embodiments, the server further performs the following steps:
sending a received third data operation message to a second data access layer, wherein the third data operation message is a query operation, the syntax of the third data operation message is the first syntax, and the second data access layer corresponds to the second data storage system;
performing syntax conversion on the third data operation message through the second data access layer to obtain a fourth data operation message, wherein the syntax of the fourth data operation message is the second syntax;
sending the fourth data operation message to the second data storage system through the second interface.
In some embodiments, the server further performs the following steps:
analyzing the configuration file to obtain a mapping relation between the data structure of the first grammar and the data structure of the second grammar;
generating the second data access layer based on the mapping relationship and a second template supporting the second syntax.
Next, a data processing method provided in the embodiment of the present application is further described.
The data processing system provided in the embodiment of the present application may be as shown in fig. 5, including: code generator 501, general data access layer 502, data access proxy (jproxy)503, mySQL504, message queue 505, data sync center 506, elasticserver (es) 507.
The code generator 501 includes: configuration management 5011, template management 5012, code generation engine 5013, and database schema (schema) 5014.
The configuration management 5011 is configured to manage a configuration file, where the configuration management may support the configuration file in xml format, and a piece of configuration file describes a mapping (mapping) relationship between a field of mySQL and a field of an Elasticsearch. The mapping relation comprises the mapping relation between the field of each characterization column in the table of mySQL and the Elasticissearch field. Here, based on the fields of the respective token columns in the table of mySQL included in the mapping relationship, the table structure of mySQL can be determined.
The template management 5012 is used to manage templates of the data access layer, where the templates of the data access layer include: the template for mysql is the first template and the template for ES is the second template.
The code generation engine 5013 is used to generate the generic data access layer 502.
The database schema (schema)5014 is a configuration file managed by the configuration management 5011 and is used for configuring mapping relationships between mySQL and ESs.
The code generator obtains data necessary for generating the data access layer by analyzing the configuration file through the configuration management 5011, and the code generation engine 5013 combines the generated data with the template managed by the template management 5012 to generate the general data access layer 502.
Here, the purpose of parsing the configuration file is to acquire table structure information, and the result of the parsing is column information of each table and mapping information of the ES.
When the addition, deletion and modification operations on the database are completed, the universal data access layer 502 submits the data change to the ES507 in an asynchronous mode of a message queue 505 or a binary log file (binlog), and a fusing mechanism is added to the read operation of the universal data access layer 502, so that switching between an elastic search query and a database query can be realized.
The generic data access layer 502 includes: a mysql data access layer 5021 and an ES data access layer 5022. The mysql data access layer 5021 is generated based on a template of mysql, and the ES data access layer 5022 is generated based on a template of ES.
Here, the mysql data access layer 5021 provides addition, modification and deletion operations related to the write method, and sends data operation messages related to the addition, modification and deletion operations of the write method to Jproxy 503.
For query operations that do not involve a write method, the ES data access layer 5022 analyzes the query conditions of the read operation, which are generated by the retrievable field + template identified in the configuration file; the ES data access layer 5022 also provides the syntax for converting SQL into ES client api by a general SQL query method. For add, modify, or delete operations involving a write method, the ES data access layer 5022 may send an operation request to the message queue 505.
Jproxy503 provides the uniform access of mySQL504, and has the functions of traffic overload protection, automatic data splitting, configurable routing rules, seamless data migration and the like.
The message queue 505 is used to buffer the data operation messages sent by the ES data access layer 5022, and the data synchronization center 506 will consume the buffered operation requests.
The data sync center 506 may consume the data operation messages of the message queue 505 and may also receive a log file of mySQL execution Jproxy503 data operation messages. Wherein the data synchronization center 506 runs a data synchronization program.
The data synchronization center 506 includes: SQL parsing module 5061, ES syntax translation module 5062, and execution module 5063.
The SQL parsing module 5061 is configured to parse the log file of mySQL504 or the data operation message of the message queue 504 to obtain the required entries in the API of the ES. Here, the log file is analyzed to obtain information of the columns before and after the data change.
The ES syntax conversion module 5062 completes the conversion of the log file into ES based on the identification of the SQL table and ES mapping relationship in the configuration file of the code generator 501. Here, the ES syntax conversion module 5062 performs field mapping based on the entry obtained by the SQL parsing module 5061, and converts the MySQL entity object into the entity object of the ES.
The execution module 5063 calls an ES API to save data according to the converted ES statement. Here, the Elasticsearch is provided with a save or update operation.
In conclusion, the data access is realized by the data access layer which is the data access object generated by the code generator, the development cost is reduced, the human efficiency is improved, the generated elastic search data access layer realizes the operations of increasing, searching, modifying and deleting (CRUD) based on the SQL grammar, developers can customize the SQL, and the learning cost of the elastic search grammar is effectively reduced. Wherein the CRUD operation comprises the following operations: add (Create), Retrieve (Retrieve), Update (Update) and Delete (Delete).
The generation method of the code generator for generating the generic data access layer is shown in fig. 6, and comprises the following steps:
s601, the server analyzes the configuration file to obtain the structure information of the table;
s602, configuring a definition object by the server side based on the structure information of the table;
here, the defined object is a JAVA object, where for mySQL, the defined object stores which columns and data types of the columns, etc. under each table; for ES, the definition object holds attribute descriptions of which fields are under each Document (Document), and the data type of the fields and whether they can be retrieved.
S603, loading a template by a server;
s604, the server generates a table building SQL;
here, through the description of the structure information of the table in the configuration file, the SQL statement for creating the table is generated, and the processes of creating the table and creating the ES index are avoided.
S605, the server side generates mapping of MY SQL and ES;
a mapping of MY SQL and ES is generated based on the configured definition object.
S606, the service end generates MY SQL DAO and ES DAO based on the configured definition object and the template.
A method for asynchronously committing operations involving addition, modification, deletion, etc. of write methods to an ED by a data access layer is shown in FIG. 7 and includes:
s701, the data access layer calls a data operation method to generate a data operation message.
S702, the data access layer sends a data operation message to the message queue.
Sending a data operation message to a message queue by the data access layer may be understood as generating a message onto the message queue.
And S703, the message queue carries out message sorting.
Wherein, the message arrangement can comprise the duplication elimination and the filing of the message.
S704, the data synchronization center consumes the data operation messages in the message queue.
Consumption of a data operation message in a message queue by a data synchronization center can be understood as consumption of the message queue.
S705, the data synchronization center analyzes the SQL statement in the message.
S706, the data synchronization center converts the SQL statement into ES grammar.
And S707, the data synchronization center calls an ES API.
S708, determine whether the operation corresponding to the data operation message is successfully executed?
If the execution is not successful, S704 is executed again, and if the execution is successful, it is ended. Here, the re-execution of S704 may be understood as a Message Queue (MQ) retry.
Fig. 8 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application, and as shown in fig. 8, the data processing apparatus 800 includes:
a first sending module 801, configured to send a received first data operation message to a first data access layer, where the first data operation message relates to a write method, a syntax of the first data operation message is a first syntax, and the first data access layer corresponds to a first data storage system;
a first calling module 802, configured to call a first interface through the first data access layer, so as to send the first data operation message to the first data storage system through the first interface, where the first data storage system supports the first syntax;
a determining module 803, configured to determine a second data operation message obtained after syntax conversion is performed on the first data operation message, where syntax of the second data operation message is a second syntax;
a second invoking module 804, configured to send the second data operation message to a second data storage system through an invoked second interface, so as to keep the first data storage system and the second data storage system synchronized, where the second data storage system supports the second syntax.
In some embodiments, the apparatus 800 further comprises:
a first generation module to generate the first data access layer based on a first template that supports the first syntax.
In some embodiments, the determining module 803 is further configured to:
sending the first data operation message to a second data access layer, wherein the second data access layer corresponds to the second data storage system;
sending the first data operation message to the data synchronization program through the second data access layer;
converting the first data operation message into the second data operation message through the data synchronization program.
In some embodiments, the apparatus 800 further comprises:
a second generation module to generate the second data access layer based on a second template that supports the second syntax.
In some embodiments, the determining module 803 is further configured to:
acquiring a log file of the first data storage system executing the first data operation message;
sending the log file to the data synchronization program;
and converting the log file into the second data operation message through the data synchronization program.
In some embodiments, the determining module 803 is further configured to:
analyzing the log file to obtain parameters in the log file;
and determining the second data operation message based on the parameters and the set mapping relation, wherein the mapping relation is the corresponding relation between the data structure of the first syntax and the data structure of the second syntax.
In some embodiments, the apparatus 800 further comprises:
a second sending module, configured to send a received third data operation message to a second data access layer, where the third data operation message is a query operation, a syntax of the third data operation message is the first syntax, and the second data access layer corresponds to the second data storage system;
a conversion module, configured to perform syntax conversion on the third data operation message through the second data access layer to obtain a fourth data operation message, where a syntax of the fourth data operation message is the second syntax;
a third sending module, configured to send the fourth data operation message to the second data storage system through the second interface.
In some embodiments, the apparatus 800 further comprises:
a third generation module to:
analyzing the configuration file to obtain a mapping relation between the data structure of the first grammar and the data structure of the second grammar;
generating the second data access layer based on the mapping relationship and a second template supporting the second syntax.
The above description of the system embodiment is similar to the above description of the method embodiment, with similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the system of the present application, reference is made to the description of the embodiments of the method of the present application for understanding.
It should be noted that, in the embodiment of the present application, if the data processing method is implemented in the form of a software functional module and sold or used as a standalone product, the data processing method may also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application or portions thereof that contribute to the related art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, or an optical disk. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.
The embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program that is stored in the memory and can be run on the processor, and when the processor runs the computer program, the steps in the data processing method of the server are implemented.
Correspondingly, the present application provides a storage medium, that is, a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements the data processing method provided by the server in the foregoing embodiments.
Here, it should be noted that: the above description of the storage medium embodiment is similar to the description of the method embodiment described above, with similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the storage medium of the present application, reference is made to the description of the embodiments of the method of the present application.
It should be noted that fig. 9 is a schematic diagram of a hardware entity (of a server) of an electronic device according to an embodiment of the present application, and as shown in fig. 9, the electronic device 900 includes: a processor 901, at least one communication bus 902, at least one external communication interface 904, and memory 905. Wherein the communication bus 902 is configured to enable connective communication between these components. In an example, the electronic device 900 further includes: a user interface 903, wherein the user interface 903 may comprise a display screen, and the external communication interface 904 may comprise a standard wired interface and a wireless interface.
The Memory 905 is configured to store instructions and applications executable by the processor 901, and may also buffer data (e.g., image data, audio data, voice communication data, and video communication data) to be processed or already processed by the processor 901 and modules in the electronic device, and may be implemented by a FLASH Memory (FLASH) or a Random Access Memory (RAM).
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in some embodiments" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units; can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: a removable storage device, a ROM, a magnetic or optical disk, or other various media that can store program code.
Alternatively, the integrated units described above in the present application may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as independent products. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented or portions thereof contributing to the related art may be embodied in the form of a software product stored in a storage medium, and including several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a removable storage device, a ROM, a magnetic or optical disk, or other various media that can store program code.
The above description is only for the embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (11)

1. A method of data processing, the method comprising:
sending a received first data operation message to a first data access layer, wherein the first data operation message relates to a writing method, the syntax of the first data operation message is a first syntax, and the first data access layer corresponds to a first data storage system;
calling a first interface through the first data access layer to send the first data operation message to the first data storage system through the first interface, the first data storage system supporting the first syntax;
determining a second data operation message obtained after syntax conversion is carried out on the first data operation message through a data synchronization program, wherein the syntax of the second data operation message is a second syntax;
sending the second data operation message to a second data storage system through a called second interface to keep the first data storage system and the second data storage system synchronized, wherein the second data storage system supports the second syntax.
2. The method of claim 1, further comprising:
generating the first data access layer based on a first template that supports the first syntax.
3. The method of claim 1, wherein determining, by the data synchronization procedure, the second data operation message obtained by syntax transformation of the first data operation message comprises:
sending the first data operation message to a second data access layer, wherein the second data access layer corresponds to the second data storage system;
sending the first data operation message to the data synchronization program through the second data access layer;
converting the first data operation message into the second data operation message by the data synchronization program.
4. The method of claim 3, further comprising:
generating the second data access layer based on a second template that supports the second syntax.
5. The method of claim 1, wherein determining, by the data synchronization procedure, the second data operation message obtained by syntax transformation of the first data operation message comprises:
acquiring a log file of the first data storage system executing the first data operation message;
sending the log file to the data synchronization program;
and converting the log file into the second data operation message through the data synchronization program.
6. The method of claim 5, wherein converting the log file into the second data operation message by the data synchronization program comprises:
analyzing the log file to obtain parameters in the log file;
and determining the second data operation message based on the parameters and the set mapping relation, wherein the mapping relation is the corresponding relation between the data structure of the first syntax and the data structure of the second syntax.
7. The method of claim 1, further comprising:
sending a received third data operation message to a second data access layer, wherein the third data operation message is a query operation, the syntax of the third data operation message is the first syntax, and the second data access layer corresponds to the second data storage system;
performing syntax conversion on the third data operation message through the second data access layer to obtain a fourth data operation message, wherein the syntax of the fourth data operation message is the second syntax;
sending the fourth data operation message to the second data storage system through the second interface.
8. The method of claim 7, further comprising:
analyzing the configuration file to obtain a mapping relation between the data structure of the first grammar and the data structure of the second grammar;
generating the second data access layer based on the mapping relationship and a second template supporting the second syntax.
9. A data processing apparatus, characterized in that the apparatus comprises:
the first sending module is used for sending a received first data operation message to a first data access layer, wherein the first data operation message relates to a writing method, the syntax of the first data operation message is first syntax, and the first data access layer corresponds to a first data storage system;
a first calling module, configured to call a first interface through the first data access layer, so as to send the first data operation message to the first data storage system through the first interface, where the first data storage system supports the first syntax;
the determining module is used for determining a second data operation message obtained after syntax conversion is carried out on the first data operation message, wherein the syntax of the second data operation message is a second syntax;
and the second calling module is used for sending the second data operation message to a second data storage system through a called second interface so as to keep the first data storage system and the second data storage system synchronous, and the second data storage system supports the second syntax.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the data processing method according to any one of claims 1 to 8 when executing the computer program.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 8.
CN202210122309.2A 2022-02-09 2022-02-09 Data processing method and device, equipment and storage medium Pending CN114490717A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210122309.2A CN114490717A (en) 2022-02-09 2022-02-09 Data processing method and device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210122309.2A CN114490717A (en) 2022-02-09 2022-02-09 Data processing method and device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN114490717A true CN114490717A (en) 2022-05-13

Family

ID=81479281

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210122309.2A Pending CN114490717A (en) 2022-02-09 2022-02-09 Data processing method and device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114490717A (en)

Similar Documents

Publication Publication Date Title
CN109086409B (en) Microservice data processing method and device, electronic equipment and computer readable medium
WO2020160265A1 (en) Data storage apparatus, translation apparatus, and database access method
CN110019498B (en) Log synchronization method and device, storage medium and electronic equipment
CN106648569B (en) Target serialization realization method and device
CN110489440B (en) Data query method and device
US11687715B2 (en) Summary generation method and apparatus
CN114528044B (en) Interface calling method, device, equipment and medium
WO2021203918A1 (en) Method for processing model parameters, and apparatus
CN111221851A (en) Lucene-based mass data query and storage method and device
CN110941655A (en) Data format conversion method and device
CN111352951A (en) Data export method, device and system
CN110109983B (en) Method and device for operating Redis database
US10866960B2 (en) Dynamic execution of ETL jobs without metadata repository
WO2022184077A1 (en) Document editing method and apparatus, and terminal and non-transitory storage medium
CN108959294B (en) Method and device for accessing search engine
CN113254819B (en) Page rendering method, system, equipment and storage medium
CN114780615A (en) Error code management method and device thereof
CN112307061A (en) Method and device for querying data
CN110888869A (en) Data processing method and device
CN115344614A (en) Data processing method and device, storage medium and electronic equipment
CN114490717A (en) Data processing method and device, equipment and storage medium
CN114968917A (en) Method and device for rapidly importing file data
US10114864B1 (en) List element query support and processing
CN113468258A (en) Heterogeneous data conversion method and device and storage medium
CN113722007A (en) Configuration method, device and system of VPN branch equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination