CN114003614A

CN114003614A - Data synchronization device, method, online transaction system, computer equipment and storage medium

Info

Publication number: CN114003614A
Application number: CN202111296890.1A
Authority: CN
Inventors: 曾文浩; 方锐涛; 卢明
Original assignee: Guangzhou Chuangle Information Technology Co ltd
Current assignee: Guangzhou Chuangle Information Technology Co ltd
Priority date: 2021-11-02
Filing date: 2021-11-02
Publication date: 2022-02-01

Abstract

The invention relates to a data synchronizer, method, online transaction system, computer equipment and storage medium, the apparatus is set up in online transaction system, the system also includes the goods module, the data synchronizer includes synchronous database increment procedure Canal, relational database MySQL and message queue RabbitMQ service, the message queue RabbitMQ service can communicate with goods module, wherein, the apparatus includes: the registration module is used for registering a synchronous database increment program Canal as a Slave server Slave node of a relational database MySQL; the updating module is used for receiving the updated data by the incremental program Canal of the synchronous database when the data in the MySQL of the relational database is updated; and the update information sending module is used for sending the update data to the commodity module through a message queue RabbitMQ service by the synchronous database increment program Canal. The purpose of improving the real-time performance of data synchronization under the conditions of no invasion to the existing codes and frequent bank refreshing is avoided when the data are synchronized is achieved.

Description

Data synchronization device, method, online transaction system, computer equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of computers, in particular to a data synchronization device, a data synchronization method, an online transaction system, computer equipment and a storage medium.

Background

In the current internet environment, the traditional data synchronization from MySQL to ES (unrelated database) is usually realized by combining a database queue and scheduling, a piece of data is inserted into the database queue where the data synchronization needs to be triggered, and the synchronization from the database to the ES is realized by regularly scanning the database queue through a scheduler.

Therefore, it is very urgent to establish a mechanism to avoid code intrusion and frequent library refreshing, and to ensure higher real-time data synchronization.

Disclosure of Invention

The embodiment of the invention provides a data synchronization device and method, an online transaction system, computer equipment and a storage medium, and aims to solve the problem that client equipment possibly carries malicious programs to occupy a large amount of network resources in the data communication process and influence the data security of a user.

In a first aspect, an embodiment of the present invention provides a data synchronization device, where the device is disposed in an online transaction system, the system further includes a commodity module, the data synchronization device includes a synchronization database increment program Canal, a relational database MySQL, and a message queue RabbitMQ service, and the message queue RabbitMQ service may communicate with the commodity module, where the device includes:

the registration module is used for registering a synchronous database increment program Canal as a Slave server Slave node of the relational database MySQL;

the updating module is used for receiving the updated data by the incremental program Canal of the synchronous database when the data in the relational database MySQL is updated;

and the update information sending module is used for sending the update data to the commodity module through the rabbitMQ service of the synchronous database increment program Canal.

In a second aspect, an embodiment of the present invention further provides an online transaction system, which includes a retrieval module, a commodity module, and the data synchronization apparatus in the first aspect, where the retrieval module and the data synchronization apparatus can communicate with each other, the commodity module and the data synchronization apparatus can communicate with each other, the retrieval module and the commodity module include a non-relational database, where,

the retrieval module is configured to:

when an access request for commodity information is received, standardized commodity information of a commodity to be inquired is requested from the non-relational database.

The commodity module is used for:

and the non-relational database acquires the commodity basic information every a preset time period and then assembles the commodity basic information into commodity inventory data.

In a third aspect, an embodiment of the present invention further provides a data synchronization method, where the method is applied to the online transaction system described in the second aspect, and the method includes:

registering the incremental program Canal of the synchronous database as a Slave server slope node of the MySQL of the relational database;

when the data in the relational database MySQL is updated, the incremental program Canal of the synchronous database receives the updated data;

the synchronization database increment program Canal sends the update data to the message queue RabbitMQ service.

In a fourth aspect, an embodiment of the present invention further provides a computer device, where the computer device includes:

one or more processors;

a memory for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the data synchronization method of the third aspect.

In a fifth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data synchronization method according to the third aspect.

In this embodiment, a data synchronization device is provided in the online transaction system, and the system further includes a commodity module. The data synchronization device comprises a synchronization database increment program Canal, a relational database MySQL and a message queue RabbitMQ service, wherein the message queue RabbitMQ service can communicate with the commodity module. Registering a synchronous database incremental program (Canal) as a Slave server Slave node of the relational database MySQL by a registration module in the device; when the data in the relational database MySQL is updated, the incremental program Canal of the synchronous database receives the updated data; and the synchronous database increment program Canal sends the updated data to the commodity module through the message queue RabbitMQ service. The method has the advantages of avoiding code intrusion and frequently refreshing the database, and solving the problem of delay of database data synchronization to ES.

Drawings

Fig. 1 is a block diagram of a data synchronization apparatus according to an embodiment of the present invention;

fig. 2A is a diagram illustrating an example of an application of a data synchronization apparatus according to an embodiment of the present invention;

fig. 2B is a diagram illustrating an application example of a data synchronization apparatus according to an embodiment of the present invention;

fig. 2C is a diagram illustrating an example of an application of the data synchronization apparatus according to the embodiment of the present invention;

fig. 3 is a schematic structural diagram of an online transaction system according to a second embodiment of the present invention;

FIG. 4 is a flowchart of a data synchronization method according to a third embodiment of the present invention

Fig. 5 is a schematic structural diagram of a computer device according to a fourth embodiment of the present invention.

Detailed Description

The noun explains:

ES

the Elasticissearch is a distributed, high-expansion and high-real-time search and data analysis engine. It can conveniently make a large amount of data have the capability of searching, analyzing and exploring. The horizontal flexibility of the elastic search is fully utilized, so that the data becomes more valuable in a production environment. The implementation principle of the Elasticissearch is mainly divided into the following steps, firstly, a user submits data to an Elasticissearch database, then a word controller divides words of corresponding sentences, the weights and word division results are stored into the data, when the user searches data, the results are ranked and scored according to the weights, and then returned results are presented to the user.

WMS

The abbreviation of Warehouse Management System, the Warehouse Management System is a Management System which comprehensively uses the functions of batch Management, material correspondence, inventory checking, quality inspection Management, virtual Warehouse Management, instant inventory Management and the like through the functions of warehousing business, ex-Warehouse business, Warehouse allocation, inventory allocation, virtual Warehouse Management and the like, effectively controls and tracks the whole process of logistics and cost Management of Warehouse business, and realizes or perfects the Management of Warehouse information of enterprises. The system can independently execute inventory operation, can be used in combination with bills and certificates of other systems, and can provide more complete enterprise logistics management flow and financial management information for enterprises.

MySQL

A relational database management system, developed by MySQL AB corporation, sweden, and belonging to Oracle brand products. MySQL is one of the most popular Relational Database Management systems, and in terms of WEB applications, MySQL is one of the best RDBMS (Relational Database Management System) application software.

MySQL is a relational database management system that keeps data in different tables instead of putting all the data in one large repository, which increases speed and flexibility.

The SQL language used by MySQL is the most common standardized language for accessing databases. MySQL software adopts a double-authorization policy and is divided into a community version and a business version, and generally MySQL is selected as a website database for development of small and medium-sized websites due to the characteristics of small volume, high speed, low total ownership cost and particularly open source codes.

Canal

The method is mainly used for providing incremental data subscription and consumption based on MySQL database incremental log analysis. canal simulates an interaction protocol of MySQL slave, pretends to be MySQL slave, and sends dump protocol to MySQL master

The MySQL master receives the dump request and starts pushing the binary log to the slave (i.e. canal), which parses the binary log object (originally the byte stream).

RabbitMQ

Open source message broker software (also known as message oriented middleware) implementing the Advanced Message Queuing Protocol (AMQP). The RabbitMQ server is written in Erlang language, while the cluster and failover are built on an open telecommunications platform framework. All major programming languages have a client library that communicates with the agent interface.

SPU(Standard Product Unit)：

And standardizing product units. Is the smallest unit of information aggregation for a commodity, and is a set of reusable, easily retrievable standardized information that describes the characteristics of a product. In colloquial, a commodity with the same attribute value and property may be referred to as an SPU.

SKU

All of them are called Stock Keeping Unit, i.e. the basic Unit of Stock in-out metering, which can be in units of pieces, boxes, trays, etc. SKU this is a necessary method for logistics management in large chain supermarkets DC (distribution centre). Now, it has been extended to the shorthand of a uniform product number, each product corresponding to a unique SKU number. Single product: a commodity is called a single commodity when any one of its attributes, such as brand, model, configuration, grade, flower color, packaging capacity, unit, production date, expiration date, usage, price, and place of production, is different from that of other commodities.

Master-slave synchronization

A high-precision reference clock is set up at a switching node, the reference clock signal is transmitted to each slave node in the network through a transmission link, and each slave node locks the local clock frequency on the reference clock frequency by using a phase-locked loop technology, so that the clock signal synchronization among the nodes in the network is realized.

The master-slave synchronous network generally adopts a star structure or a tree structure, wherein most nodes are slave nodes. The slave node clock tracks the state of the reference clock, so that the cost can be effectively reduced. Therefore, the master-slave synchronization technology can improve the accuracy of the clock system and reduce the cost.

Mysql Binlog

A log file in binary format, but a binlog file cannot be equated to a specific file under a certain directory of the OS system. Binlog is used for recording the change of Mysql to the database (only the modification operation to the data is recorded), and is mainly used for master-slave copy and increment recovery of the database.

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Example one

Fig. 1 is a block diagram of a data synchronization device provided in an embodiment of the present invention, where the data synchronization device is disposed in an online transaction system, the system further includes a commodity module, the data synchronization device includes a synchronization database increment program Canal, a relational database MySQL, and a message queue RabbitMQ service, and the message queue RabbitMQ service may communicate with the commodity module, where the data synchronization device specifically includes:

and the registering module 101 is configured to register a synchronous database increment program Canal as a Slave server Slave node of the relational database MySQL.

In the embodiment of the present invention, as shown in the example application diagram of the data synchronization apparatus shown in fig. 2A, the data synchronization apparatus includes MySQL service, Canal service, and RabbitMQ service.

First, a Canal (synchronous database incremental program) service is started and registered as the Salve (slave server) node of MySQL.

An updating module 102, configured to receive update data by the synchronous database incremental program Canal when data in the relational database MySQL is updated.

Preferably, in another embodiment of the present invention, the updating module 102 is further configured to:

when the data in the relational database MySQL is updated, the relational database MySQL synchronizes the data updating information to a binary database log;

the synchronous database increment program Canal obtains the data updating information by acquiring the binary database log;

the application program regularly pulls data from the cancer and distributes the data to the related business Queue through MQ (Message Queue)

And the incremental program Canal of the synchronous database receives the updated data through the data updating information.

In the embodiment of the invention, when the data in the relational database MySQL is found to be updated, namely the MySQL data is sent and changed, the data is synchronized to the Canal service through the binlog (binary log file). The sync database increment program Canal service will obtain the update data from binlog.

And the update information sending module 103 is configured to send the update data to the commodity module through the RabbitMQ service of the message queue by the synchronous database increment program Canal.

In the embodiment of the present invention, an application program periodically pulls data from the Canal, and distributes the data to a related business Queue through an MQ (Message Queue), and the data is distributed by the business Queue to a platform commodity Queue consumer in a commodity module as shown in fig. 2B.

In this embodiment, according to a registration module, the Slave server Slave node is configured to register a synchronous database incremental program Canal as the relational database MySQL; the updating module is used for receiving the updated data by the incremental program Canal of the synchronous database when the data in the relational database MySQL is updated; and the update information sending module is used for sending the update data to the commodity module through the rabbitMQ service of the synchronous database increment program Canal. The method and the device achieve the purpose of improving the real-time performance of data synchronization under the conditions of no invasion to the existing codes and frequent bank refreshing prevention when the data are synchronized.

It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.

Example two

Fig. 3 is a block diagram of an online transaction system according to a second embodiment of the present invention, where the system includes a retrieval module 301, a goods module 302, and a data synchronization device 303, the retrieval module 301 and the data synchronization device 303 can communicate with each other, the goods module 302 and the data synchronization device 303 can communicate with each other, the retrieval module 301 and the goods module 302 include a non-relational database 304, where,

the retrieving module 301 is configured to:

In the embodiment of the present invention, as shown in fig. 2B, the retrieval module includes a DC (commodity) service, an ES service, a TC (order) service, and a WMS (warehouse management) service, and when receiving a commodity data request sent by the ES service, the DC (commodity) service returns commodity basic data, and then the ES service assembles the basic data.

The merchandise module 302 is configured to:

In the embodiment of the present invention, as shown in fig. 2B, the ES non-relational database may obtain the commodity basic data from the DC service every preset time period and then encapsulate the commodity basic data, so as to effectively detect the update data.

Optionally, in a further embodiment of the present invention, the system further includes a client, the retrieving module 301 and the client may communicate with each other, and the retrieving module 301 is further configured to:

when receiving an access request of the client to a commodity list, sending the access request to the non-relational database;

the non-relational database obtains stored inventory data and returns the inventory data to the client if the local matching is successful according to the information of the commodities to be inquired in the access request;

otherwise, extracting the attribute information of the commodity to be inquired from the data synchronization device according to the access request;

and assembling the acquired attribute information into standardized commodity information.

In the embodiment of the present invention, as shown in fig. 2C, when a client sends a commodity information retrieval request, the access request is first sent to the non-relational database, and the non-relational database locally queries the corresponding commodity packaging information according to the information to be queried in the access request.

It can be understood that if the package information of the goods to be queried exists in the query local data, the package information is returned to the client. Otherwise, the data to be inquired is the updated data, the attribute information of the corresponding commodity is requested from the data synchronization device, and the acquired attribute information is packaged into the standardized commodity information.

The standardized commodity information, also called SPU (Standard Product Unit standardized Product Unit), is the minimum Unit of commodity information aggregation, and is a set of reusable and easily retrievable standardized information, which describes the characteristics of a Product. In colloquial, a commodity with the same attribute value and property may be referred to as an SPU.

Optionally, in a further embodiment of the present invention, the retrieving module 301 is further configured to:

the non-relational database requests commodity attribute information from the data synchronization device every other preset time period;

and the non-relational database assembles the commodity attribute information into standardized commodity information.

In the embodiment of the present invention, the retrieval module may further request the data synchronization device for the commodity attribute information every preset time period.

It will be appreciated that the merchandise attribute information requested herein may more efficiently discover updated merchandise information from the database for timely packaging of the merchandise information stored as standardized.

In practical applications, the preset interval time is set by a related technician according to a specific use scenario, which is not limited in the embodiment of the present invention.

Optionally, in another embodiment of the present invention, the commodity module further includes a commodity service module, an order service module, and a warehousing management module, and the commodity module is further configured to:

the non-relational database acquires commodity basic information from the commodity service module every other preset time period;

the non-relational database assembles the acquired basic commodity information into basic commodity data;

the non-relational database acquires commodity sales volume information from the order service module;

the non-relational database assembles the commodity basic data and the commodity sales information into commodity sales data;

the non-relational database acquires commodity inventory information from the warehousing management module;

and the non-relational database assembles the commodity inventory information and the commodity sales volume data into commodity inventory data.

In the embodiment of the present invention, as shown in fig. 2B, the commodity service module further includes a commodity service module, an order service module, and a warehousing management module, wherein the non-relational database ES is used for obtaining and packaging relevant data from the commodity service module, the order service module, and the warehousing management module, and finally packaged into commodity inventory data for storage.

Optionally, in a further embodiment of the present invention, the merchandise module is further configured to:

the non-relational database receives the updating data sent by the incremental program Canal of the synchronous database in the data synchronization device;

and the non-relational database assembles the received updating data into the commodity basic data.

In the embodiment of the invention, a non-relational database ES and a message queue RabbitMQ service in a data synchronization device can communicate with each other, when data in a relational database MySQL is updated, a synchronous database increment program Canal receives the updated data and transmits the updated data to the ES through the message queue RabbitMQ service, and the ES is packaged into commodity basic data after receiving the updated data.

In summary, in the embodiment of the present invention, through an online transaction system, the system includes a retrieval module, a commodity module, and a data synchronization apparatus described in one embodiment, where the retrieval module and the data synchronization apparatus can communicate with each other, the commodity module and the data synchronization apparatus can communicate with each other, and the retrieval module and the commodity module include a non-relational database, where the retrieval module is configured to: when an access request for commodity information is received, standardized commodity information of a commodity to be inquired is requested from the non-relational database. The commodity module is used for: and the non-relational database acquires the commodity basic information every a preset time period and then assembles the commodity basic information into commodity inventory data. The purpose of improving the data synchronization real-time performance under the condition of no invasion to background codes in a shopping system is achieved.

EXAMPLE III

Fig. 4 is a data synchronization method provided in the third embodiment of the present invention, where the method is applied to the online transaction system, and the method specifically includes:

step 401, registering the incremental program Canal of the synchronous database as a Slave server Slave node of the MySQL of the relational database;

this step is the same as that described in the registration module 101 in the first embodiment, and will not be described in detail here.

Step 402, when the data in the relational database MySQL is updated, the incremental program Canal of the synchronous database receives the updated data;

this step is the same as that described in the update module 102 of the first embodiment, and will not be described in detail here.

In step 403, the incremental program Canal of the synchronous database sends the update data to the RabbitMQ service of the message queue.

This step is the same as that described in the first embodiment of the update information sending module 103, and will not be described in detail here.

The data protection device provided by the embodiment of the invention can execute the data synchronization method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

Example four

Fig. 5 is a schematic structural diagram of a computer device according to a fourth embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 5 is only an example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention.

As shown in FIG. 5, computer device 12 is in the form of a general purpose computing device. The components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.

Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Computer device 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer device 12 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, and commonly referred to as a "hard drive"). Although not shown in FIG. 4, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.

Computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with computer device 12, and/or with any devices (e.g., network card, modem, etc.) that enable computer device 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, computer device 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with the other modules of computer device 12 via bus 18. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processing unit 16 executes various functional applications and data processing, such as implementing a data synchronization method provided by an embodiment of the present invention, by executing programs stored in the system memory 28.

EXAMPLE five

An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements each process of the data synchronization method, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.

A computer readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data synchronization device disposed in a network transaction system, wherein the system further includes a commodity module, the data synchronization device includes a synchronization database increment program Canal, a relational database MySQL, and a message queue RabbitMQ service, and the message queue RabbitMQ service can communicate with the commodity module, wherein the device includes:

2. The apparatus of claim 1, wherein the update module is further configured to:

3. An online transaction system, comprising a retrieval module, a commodity module, and the data synchronization apparatus of claims 1-2, wherein the retrieval module and the data synchronization apparatus are communicable with each other, the commodity module and the data synchronization apparatus are communicable with each other, the retrieval module and the commodity module comprise a non-relational database, wherein,

the retrieval module is configured to:

when an access request for commodity information is received, requesting standardized commodity information of a commodity to be inquired from the non-relational database;

the commodity module is used for:

4. The system of claim 3, further comprising a client, wherein the retrieval module and the client are in communication with each other, the retrieval module further configured to:

5. The system of claim 4, wherein the retrieval module is further configured to:

6. The system of claim 3, wherein the goods module further comprises a goods services module, an order services module, and a warehousing management module, the goods module further configured to:

7. The system of claim 3, wherein the merchandise module is further configured to:

8. A data synchronization method applied to the online transaction system according to claims 3 to 7, the method comprising:

9. A computer device, characterized in that the computer device comprises:

one or more processors;

a memory for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the data synchronization method as recited in claim 8.

10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the data synchronization method as claimed in claim 8.