CN113177091A - Storage method and device of incremental data, storage medium and electronic device - Google Patents
Storage method and device of incremental data, storage medium and electronic device Download PDFInfo
- Publication number
- CN113177091A CN113177091A CN202110548480.5A CN202110548480A CN113177091A CN 113177091 A CN113177091 A CN 113177091A CN 202110548480 A CN202110548480 A CN 202110548480A CN 113177091 A CN113177091 A CN 113177091A
- Authority
- CN
- China
- Prior art keywords
- incremental data
- determining
- data
- region
- storage node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a storage method and a device of incremental data, a storage medium and an electronic device, wherein the method comprises the following steps: determining a region code corresponding to the acquired incremental data; determining a target storage node corresponding to the incremental data from the determined mapping relation between the region codes and the storage nodes; and storing the incremental data to a target database according to the target storage node. By the method and the device, the problem of the global regional storage of the incremental data in the related technology is solved, and the effect of regional storage of the incremental data is achieved.
Description
Technical Field
The embodiment of the invention relates to the field of communication, in particular to a method and a device for storing incremental data, a storage medium and an electronic device.
Background
In the using process of the database, when the storage structure and the data safety of the database cannot meet the business requirements, the data in the database needs to be migrated and backed up to a new database which can meet the business requirements, which relates to the problem of migrating a large amount of data; while global data storage is made more challenging in the context of increasingly strong global data security protections. More and more countries propose that data in the local area needs to be saved or backed up to the local area.
In the prior art, when incremental data is distributed and stored, only a message distribution strategy is described simply, the message distribution strategy is not introduced, and the problem of global data dynamic distribution is not solved.
Aiming at the problem of incremental data global partitioned storage in the prior art, an effective solution is not provided in the related technology.
Disclosure of Invention
The embodiment of the invention provides a storage method and device of incremental data, a storage medium and an electronic device, which are used for at least solving the problem of global regional storage of the incremental data in the related technology.
According to an embodiment of the present invention, there is provided a storage method of incremental data, including: determining a region code corresponding to the acquired incremental data; determining a target storage node corresponding to the incremental data from the determined mapping relation between the region codes and the storage nodes; and storing the incremental data to a target database according to the target storage node.
According to another embodiment of the present invention, there is provided a storage apparatus of incremental data, including: the first determining module is used for determining the region code corresponding to the acquired incremental data; a second determining module, configured to determine a target storage node corresponding to the incremental data from the determined mapping relationship between the region code and the storage node; and the first storage module is used for storing the incremental data to a target database according to the target storage node.
In an exemplary embodiment, the first determining module includes: a first analyzing unit, configured to analyze user information corresponding to the incremental data from a message queue in which the incremental data is located; a first determining unit, configured to determine an area code corresponding to the incremental data from an area attribute of the user information.
In an exemplary embodiment, the first determining module includes: a second determining unit, configured to determine, when user information corresponding to the incremental data is not resolved from a message queue in which the incremental data is located, a session networking protocol IP address of a client that acquires the incremental data; a second analyzing unit, configured to analyze an area where the incremental data is located according to the IP address; and a third determining unit, configured to determine, by using the area where the incremental data is located, an area code corresponding to the incremental data.
In an exemplary embodiment, the first determining module includes: and the self-defining unit is used for self-defining the area code corresponding to the incremental data under the condition that the user information corresponding to the incremental data is not analyzed from the message queue where the incremental data is located and the IP address of the session networking protocol of the client of the incremental data is not acquired so as to determine the area code corresponding to the acquired incremental data.
In an exemplary embodiment, the customization unit includes: a first dividing unit, configured to divide a global time zone into N coding regions, where N is a natural number greater than 1; a second dividing subunit, configured to divide the incremental data into target encoding regions corresponding to times at which the incremental data are generated among the N encoding regions according to the times at which the incremental data are generated; and a first determining subunit, configured to determine, based on the region code corresponding to the target coding region, the region code corresponding to the incremental data.
In an exemplary embodiment, the apparatus further includes a third determining module, configured to determine a mapping relationship between the region code and the storage node before determining a target storage node corresponding to the incremental data from the determined mapping relationship between the region code and the storage node; and the storage module is used for storing the mapping relation into a remote dictionary service cache.
In an exemplary embodiment, the apparatus further includes: and a changing module, configured to change, in the remote dictionary cache, the mapping relationship between the region code and the storage node when the storage node corresponding to the region code is changed after the mapping relationship is stored in the remote dictionary service cache.
In an exemplary embodiment, the apparatus further includes: a fourth determining module, configured to determine an event attribute of the incremental data when determining a data table corresponding to the incremental data before determining an area code corresponding to the acquired incremental data; the first packaging module is used for packaging the event attribute and the incremental data to obtain packaged data; and the first sending module is used for sending the encapsulated data to a message queue.
According to a further embodiment of the present invention, there is also provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
According to yet another embodiment of the present invention, there is also provided an electronic device, including a memory in which a computer program is stored and a processor configured to execute the computer program to perform the steps in any of the above method embodiments.
According to the invention, the region code corresponding to the acquired incremental data is determined; determining a target storage node corresponding to the incremental data from the determined mapping relation between the region codes and the storage nodes; and storing the incremental data to a target database according to the target storage node. The purpose of determining the storage node corresponding to the incremental data based on the obtained regional code of the incremental data is achieved. Therefore, the problem of incremental data global partitioned storage in the related technology can be solved, and the effect of incremental data partitioned storage is achieved.
Drawings
Fig. 1 is a block diagram of a hardware structure of a mobile terminal of a storage method of incremental data according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method of storing incremental data according to an embodiment of the present invention;
FIG. 3 is an overall flow diagram according to an embodiment of the invention;
FIG. 4 is a flow chart of determining region coding according to an embodiment of the present invention;
FIG. 5 is a flow diagram of dynamic partitioning according to an embodiment of the present invention;
FIG. 6 is a system block diagram according to an embodiment of the invention;
fig. 7 is a block diagram of a structure of a storage device of incremental data according to an embodiment of the present invention.
Detailed Description
Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings in conjunction with the embodiments.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided in the embodiments of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. Taking an example of the present invention running on a mobile terminal, fig. 1 is a block diagram of a hardware structure of the mobile terminal of a storage method of incremental data according to an embodiment of the present invention. As shown in fig. 1, the mobile terminal may include one or more (only one shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data, wherein the mobile terminal may further include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and does not limit the structure of the mobile terminal. For example, the mobile terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used to store a computer program, for example, a software program and a module of an application software, such as a computer program corresponding to the incremental data storage method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, so as to implement the method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a Network adapter (NIC), which can be connected to other Network devices through a base station so as to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
In this embodiment, a storage method of incremental data is provided, and fig. 2 is a flowchart of the storage method of incremental data according to an embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
step S202, determining a region code corresponding to the acquired incremental data;
step S204, determining a target storage node corresponding to the incremental data from the determined mapping relation between the region code and the storage node;
and step S206, storing the incremental data to a target database according to the target storage node.
The execution subject of the above steps may be a terminal, but is not limited thereto.
The embodiment includes but is not limited to being applied to a scenario of storing incremental data in a partitioned mode.
Through the steps, the region code corresponding to the acquired incremental data is determined; determining a target storage node corresponding to the incremental data from the determined mapping relation between the region codes and the storage nodes; and storing the incremental data to a target database according to the target storage node. The purpose of determining the storage node corresponding to the incremental data based on the obtained regional code of the incremental data is achieved. Therefore, the problem of incremental data global partitioned storage in the related technology can be solved, and the effect of incremental data partitioned storage is achieved.
In one exemplary embodiment, determining the region code corresponding to the acquired delta data includes:
s1, analyzing the user information corresponding to the incremental data from the message queue where the incremental data is located;
s2, determining the area code corresponding to the incremental data from the area attribute of the user information.
In this embodiment, the incremental data is set in the message queue, the message queue is analyzed to obtain the incremental data, and the region code corresponding to the data is determined according to the partition matching rule.
In one exemplary embodiment, determining the region code corresponding to the acquired delta data includes:
s1, determining the IP address of the client terminal for obtaining the incremental data under the condition that the user information corresponding to the incremental data is not analyzed from the message queue where the incremental data is located;
s2, analyzing the area of the incremental data according to the IP address;
and S3, determining the area code corresponding to the incremental data by using the area where the incremental data is located.
In the present embodiment, first, the area information thereof is determined from the user information.
In one exemplary embodiment, determining the region code corresponding to the acquired delta data includes:
and S1, when the user information corresponding to the incremental data is not analyzed from the message queue where the incremental data is located and the IP address of the client of the incremental data is not acquired, customizing the area code corresponding to the incremental data to determine the area code corresponding to the acquired incremental data.
In an exemplary embodiment, the region coding corresponding to the custom increment data includes:
s1, dividing the global time zone into N coding regions, wherein N is a natural number larger than 1;
s2, dividing the incremental data into N encoding areas and target encoding areas corresponding to the time of generating the incremental data according to the time of generating the incremental data;
and S3, determining the area code corresponding to the incremental data based on the area code corresponding to the target coding area.
In the present embodiment, for example, the world is divided into three regions by time zone coding codes, and is divided into α (west 12-west 5), β (west 4-east 4), γ (east 5-east 12) centering on 0 time zone. The data entering the custom partition rule is not matched to any region in the previous matching rule. The rule assumes that all records are generated by 8am-4pm operations in the area where the user is located. According to the rule, data of an unknown region is distributed to one of the large region regions. The time intervals corresponding to alpha, beta and gamma are respectively alpha-12, -5, beta-4, 4 and gamma 5, 12.
In an exemplary embodiment, before determining the target storage node corresponding to the incremental data from the determined mapping relationship between the region code and the storage node, the method further includes:
s1, determining the mapping relation between the region codes and the storage nodes;
and S2, storing the mapping relation into a remote dictionary service cache.
In this embodiment, the corresponding relationship between the region encoding code and the storage node region may be written into the redis cache in advance.
In an exemplary embodiment, after storing the mapping relationship in the remote dictionary service cache, the method further comprises:
and S1, changing the mapping relation between the region code and the storage node in the remote dictionary cache when the storage node corresponding to the region code is changed.
In this embodiment, when the storage node region corresponding to the region code is changed, the data partition service may send data to the message queue of the new region only by changing the mapping relationship of the redis and adding the data execution service and the target database of the X region. The function of dynamically storing the incremental data to the target database can be achieved, and other services are not affected.
In an exemplary embodiment, before determining the region code corresponding to the acquired delta data, the method further includes:
s1, determining the event attribute of the incremental data when determining the data table corresponding to the incremental data;
s2, encapsulating the event attribute and the incremental data to obtain encapsulated data;
and S3, sending the encapsulated data to a message queue.
In this embodiment, table names of the partitions to be executed can be preset, binary log Binlog data analyzed by a tool Canal component of the incremental data can be synchronized, when the corresponding tables are matched, events are divided into DELETE, update, INSERT in INSERT, analyze each Column according to EventType, obtain partition index, and assemble sql statements which can be executed finally. And the two parts of data are packed into json objects and are sent to a message queue.
The present invention will be described in detail with reference to the following specific examples:
as shown in fig. 3, it is an overall flowchart in the present embodiment, and includes the following steps:
s301: starting a binary log writing function and setting binlog-format as a ROW mode ROW mode in the source database;
s302: the data preprocessing module is used for presetting table names needing partitioning and Billog data analyzed by a Canal component, dividing events into DELETE, UPADTE and INSERT according to event attributes EventType when the corresponding tables are matched, analyzing each read-only attribute Column, acquiring a partition index (a function is a reference of values or values in a return table or a region, two forms, an array form and a reference form. The two parts of data are packaged into Object Notation (JSON) objects and sent to a message queue.
Example (c): when the resulting binlog is a user table, the json objects are:
the index may be a user ID, an order ID, a device ID, etc.
S303: a data partitioning module: the corresponding relation between the region coding code and the storage node region is written into a Remote Dictionary service (Remote Dictionary Server) redis cache in advance. The module consumes and analyzes the data in the message queue, and determines the region code corresponding to the data according to the partition matching rule. As shown in fig. 4, the matching determination is made by a plurality of dimensions. The method comprises the steps of firstly determining area information according to user information, when the area information cannot be judged through the existing information, analyzing the area where the area information is located through obtaining an Internet Protocol Address (IP) Address when a client operates, and when the area information cannot be determined, distributing the area by adopting a custom strategy.
The self-defining strategy divides the world into three region coding codes according to time zones, and divides the code into alpha (west 12-west 5), beta (west 4-east 4) and gamma (east 5-east 12) by taking a 0 time zone as a center. The data entering the custom partition rule is not matched to any region in the previous matching rule. The rule assumes that all records are generated by 8am-4pm operations in the area where the user is located. According to the rule, data of an unknown region is distributed to one of the large region regions. The time intervals corresponding to alpha, beta and gamma are respectively alpha-12, -5, beta-4, 4 and gamma 5, 12.
Partition formula: r (α, β, γ) ═ 12-t, where t is the number of time integers of UTC.
Example (c): a Universal Time warping (UTC) is used, the Time 03:15 is the data generated, t is 3, and the region code is γ.
And after the region code is acquired, sending the region information of the storage node in the service local cache to message queues of different storage nodes. The service local cache synchronizes the mapping relation of codes and regions from the redis.
As shown in fig. 5, when a storage node region corresponding to a region code changes, the dynamic partition rule only needs to change the mapping relationship of the redis, and adds a data execution service and a target database in the X region, so that the data partition service sends data to a message queue in a new region. The function of dynamically storing the incremental data to the target database can be achieved, and other services are not affected.
Example (c): the original data of the region A is stored in the region a, and when the region A requires that the data of the region is required to be stored in the region x, the region A can be realized only by changing the region corresponding to the region A from a to x.
S304: a data execution module: and consuming the data in the local area message queue and writing the data into the target database.
S305: the system module is shown in fig. 6.
In summary, in the embodiment, the area information is determined according to the user information, when the determination cannot be made through the existing information, the area where the user information is located is analyzed by obtaining the IP address when the client operates, and when none of the information can determine the area, the area is allocated by using the custom policy. The user-defined strategy divides the global area into three large areas by time and distributes the areas according to the time generated by the log. The local cache of the service synchronizes the mapping relation between the codes and the regions from the regions at regular time, when the regions of the storage nodes corresponding to the region codes change, the corresponding relation of the regions only needs to be changed, and the data execution service and the target database of the X region are newly added, so that the function of dynamically storing the incremental data to the target database can be achieved, and other services are not affected.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
In this embodiment, a storage device for incremental data is further provided, and the storage device is used to implement the foregoing embodiments and preferred embodiments, and the description of the storage device that has been already made is omitted. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 7 is a block diagram of a structure of an incremental data storage apparatus according to an embodiment of the present invention, as shown in fig. 7, the apparatus including:
a first determining module 72, configured to determine a region code corresponding to the acquired incremental data;
a second determining module 74, configured to determine, from the determined mapping relationship between the region code and the storage node, a target storage node corresponding to the incremental data;
and a first storage module 76, configured to store the incremental data to the target database according to the target storage node.
In an exemplary embodiment, the first determining module includes:
a first analyzing unit, configured to analyze user information corresponding to the incremental data from a message queue in which the incremental data is located;
a first determining unit, configured to determine an area code corresponding to the incremental data from an area attribute of the user information.
In an exemplary embodiment, the first determining module includes:
a second determining unit, configured to determine, when user information corresponding to the incremental data is not resolved from a message queue in which the incremental data is located, a session networking protocol IP address of a client that acquires the incremental data;
a second analyzing unit, configured to analyze an area where the incremental data is located according to the IP address;
and a third determining unit, configured to determine, by using the area where the incremental data is located, an area code corresponding to the incremental data.
In an exemplary embodiment, the first determining module includes:
and the self-defining unit is used for self-defining the area code corresponding to the incremental data under the condition that the user information corresponding to the incremental data is not analyzed from the message queue where the incremental data is located and the IP address of the session networking protocol of the client of the incremental data is not acquired so as to determine the area code corresponding to the acquired incremental data.
In an exemplary embodiment, the customization unit includes:
a first dividing unit, configured to divide a global time zone into N coding regions, where N is a natural number greater than 1;
a second dividing subunit, configured to divide the incremental data into target encoding regions corresponding to times at which the incremental data are generated among the N encoding regions according to the times at which the incremental data are generated;
and a first determining subunit, configured to determine, based on the region code corresponding to the target coding region, the region code corresponding to the incremental data.
In an exemplary embodiment, the apparatus further includes a third determining module, configured to determine a mapping relationship between the region code and the storage node before determining a target storage node corresponding to the incremental data from the determined mapping relationship between the region code and the storage node;
and the storage module is used for storing the mapping relation into a remote dictionary service cache.
In an exemplary embodiment, the apparatus further includes: and a changing module, configured to change, in the remote dictionary cache, the mapping relationship between the region code and the storage node when the storage node corresponding to the region code is changed after the mapping relationship is stored in the remote dictionary service cache.
In an exemplary embodiment, the apparatus further includes:
a fourth determining module, configured to determine an event attribute of the incremental data when determining a data table corresponding to the incremental data before determining an area code corresponding to the acquired incremental data;
the first packaging module is used for packaging the event attribute and the incremental data to obtain packaged data;
and the first sending module is used for sending the encapsulated data to a message queue.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above-mentioned method embodiments when executed.
In the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:
s1, determining the area code corresponding to the acquired incremental data;
s2, determining a target storage node corresponding to the incremental data from the determined mapping relation between the region code and the storage node;
and S3, storing the incremental data to the target database according to the target storage node.
In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
In an exemplary embodiment, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
In an exemplary embodiment, the processor may be configured to execute the following steps by a computer program:
s1, determining the area code corresponding to the acquired incremental data;
s2, determining a target storage node corresponding to the incremental data from the determined mapping relation between the region code and the storage node;
and S3, storing the incremental data to the target database according to the target storage node.
For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary embodiments, and details of this embodiment are not repeated herein.
It will be apparent to those skilled in the art that the various modules or steps of the invention described above may be implemented using a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and they may be implemented using program code executable by the computing devices, such that they may be stored in a memory device and executed by the computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into various integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (11)
1. A method for storing incremental data, comprising:
determining a region code corresponding to the acquired incremental data;
determining a target storage node corresponding to the incremental data from the determined mapping relation between the region codes and the storage nodes;
and storing the incremental data to a target database according to the target storage node.
2. The method of claim 1, wherein determining the region code corresponding to the acquired delta data comprises:
analyzing user information corresponding to the incremental data from a message queue where the incremental data is located;
and determining the area code corresponding to the incremental data from the area attribute of the user information.
3. The method of claim 1, wherein determining the region code corresponding to the acquired delta data comprises:
determining a session networking protocol (IP) address of a client for acquiring the incremental data under the condition that user information corresponding to the incremental data is not analyzed from a message queue where the incremental data is located;
analyzing the area where the incremental data is located according to the IP address;
and determining the area code corresponding to the incremental data by using the area where the incremental data is located.
4. The method of claim 1, wherein determining the region code corresponding to the acquired delta data comprises:
and under the condition that the user information corresponding to the incremental data is not analyzed from the message queue where the incremental data is located and the IP address of the session networking protocol of the client of the incremental data is not acquired, customizing the area code corresponding to the incremental data to determine the area code corresponding to the acquired incremental data.
5. The method of claim 4, wherein customizing the region code corresponding to the incremental data comprises:
dividing a global time zone into N coding regions, wherein N is a natural number greater than 1;
dividing the incremental data into target encoding regions corresponding to the time for generating the incremental data in the N encoding regions according to the time for generating the incremental data;
and determining the region code corresponding to the incremental data based on the region code corresponding to the target coding region.
6. The method of claim 1, wherein before determining the target storage node corresponding to the incremental data from the determined mapping relationship between the region code and the storage node, the method further comprises:
determining a mapping relation between the region code and the storage node;
and storing the mapping relation into a remote dictionary service cache.
7. The method of claim 6, wherein after storing the mapping relationship in a remote dictionary service cache, the method further comprises:
and when the storage node corresponding to the region code is changed, changing the mapping relation between the region code and the storage node in the remote dictionary cache.
8. The method of claim 1, wherein prior to determining the region code corresponding to the acquired delta data, the method further comprises:
determining an event attribute of the incremental data in the case of determining a data table corresponding to the incremental data;
packaging the event attribute and the incremental data to obtain packaged data;
and sending the encapsulated data to a message queue.
9. An apparatus for storing incremental data, comprising:
the first determining module is used for determining the region code corresponding to the acquired incremental data;
the second determining module is used for determining a target storage node corresponding to the incremental data from the determined mapping relation between the region codes and the storage nodes;
and the first storage module is used for storing the incremental data to a target database according to the target storage node.
10. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the method of any of claims 1 to 8 when executed.
11. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110548480.5A CN113177091B (en) | 2021-05-19 | 2021-05-19 | Incremental data storage method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110548480.5A CN113177091B (en) | 2021-05-19 | 2021-05-19 | Incremental data storage method and device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113177091A true CN113177091A (en) | 2021-07-27 |
| CN113177091B CN113177091B (en) | 2023-10-10 |
Family
ID=76929379
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110548480.5A Active CN113177091B (en) | 2021-05-19 | 2021-05-19 | Incremental data storage method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113177091B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114928580A (en) * | 2022-04-12 | 2022-08-19 | 深圳市优必选科技股份有限公司 | Data increment sending method, receiving method, terminal equipment and storage medium |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040136273A1 (en) * | 2003-01-09 | 2004-07-15 | Peter Chambers | Robust power-on meter and method using a limited-write memory |
| US20130198103A1 (en) * | 2012-01-31 | 2013-08-01 | Sap Ag | Mapping Between Different Delta Handling Patterns |
| CN103995861A (en) * | 2014-05-16 | 2014-08-20 | 罗敬宁 | Distributed data device, method and system based on spatial correlation |
| CN109299088A (en) * | 2018-08-22 | 2019-02-01 | 中国平安人寿保险股份有限公司 | Mass data storage means, device, storage medium and electronic equipment |
| WO2019072103A1 (en) * | 2017-10-11 | 2019-04-18 | 阿里巴巴集团控股有限公司 | Method and apparatus for obtaining target data |
| CN110134755A (en) * | 2019-05-28 | 2019-08-16 | 北京百度网讯科技有限公司 | Map updating method, device, equipment, server and medium |
| US10459918B1 (en) * | 2016-06-28 | 2019-10-29 | Amazon Technologies, Inc. | Generating query results based on data partitions |
| US20200050687A1 (en) * | 2018-08-09 | 2020-02-13 | Netapp Inc. | Resynchronization to a filesystem synchronous replication relationship endpoint |
| CN111723148A (en) * | 2020-04-10 | 2020-09-29 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
| CN111723160A (en) * | 2020-08-24 | 2020-09-29 | 国网浙江省电力有限公司 | Multi-source heterogeneous incremental data synchronization method and system |
| CN112100147A (en) * | 2020-07-27 | 2020-12-18 | 杭州玳数科技有限公司 | Method and system for realizing real-time acquisition from Bilog to HIVE based on Flink |
| CN112307037A (en) * | 2019-07-26 | 2021-02-02 | 北京京东振世信息技术有限公司 | Data synchronization method and device |
-
2021
- 2021-05-19 CN CN202110548480.5A patent/CN113177091B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040136273A1 (en) * | 2003-01-09 | 2004-07-15 | Peter Chambers | Robust power-on meter and method using a limited-write memory |
| US20130198103A1 (en) * | 2012-01-31 | 2013-08-01 | Sap Ag | Mapping Between Different Delta Handling Patterns |
| CN103995861A (en) * | 2014-05-16 | 2014-08-20 | 罗敬宁 | Distributed data device, method and system based on spatial correlation |
| US10459918B1 (en) * | 2016-06-28 | 2019-10-29 | Amazon Technologies, Inc. | Generating query results based on data partitions |
| WO2019072103A1 (en) * | 2017-10-11 | 2019-04-18 | 阿里巴巴集团控股有限公司 | Method and apparatus for obtaining target data |
| US20200050687A1 (en) * | 2018-08-09 | 2020-02-13 | Netapp Inc. | Resynchronization to a filesystem synchronous replication relationship endpoint |
| CN109299088A (en) * | 2018-08-22 | 2019-02-01 | 中国平安人寿保险股份有限公司 | Mass data storage means, device, storage medium and electronic equipment |
| CN110134755A (en) * | 2019-05-28 | 2019-08-16 | 北京百度网讯科技有限公司 | Map updating method, device, equipment, server and medium |
| CN112307037A (en) * | 2019-07-26 | 2021-02-02 | 北京京东振世信息技术有限公司 | Data synchronization method and device |
| CN111723148A (en) * | 2020-04-10 | 2020-09-29 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
| CN112100147A (en) * | 2020-07-27 | 2020-12-18 | 杭州玳数科技有限公司 | Method and system for realizing real-time acquisition from Bilog to HIVE based on Flink |
| CN111723160A (en) * | 2020-08-24 | 2020-09-29 | 国网浙江省电力有限公司 | Multi-source heterogeneous incremental data synchronization method and system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114928580A (en) * | 2022-04-12 | 2022-08-19 | 深圳市优必选科技股份有限公司 | Data increment sending method, receiving method, terminal equipment and storage medium |
| CN114928580B (en) * | 2022-04-12 | 2023-06-02 | 深圳市优必选科技股份有限公司 | Data increment transmitting method, receiving method, terminal device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113177091B (en) | 2023-10-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10831562B2 (en) | Method and system for operating a data center by reducing an amount of data to be processed | |
| CN111723148B (en) | Data storage method and device, storage medium and electronic device | |
| US11379271B2 (en) | Parallel processing on data processing servers through even division of data records | |
| CN111552838B (en) | Data processing method and device, computer equipment and storage medium | |
| CN110427494B (en) | Knowledge graph display method and device, storage medium and electronic device | |
| CN110532493B (en) | Data processing method and device, storage medium and electronic device | |
| CN110727738A (en) | Global routing system based on data fragmentation, electronic equipment and storage medium | |
| US20200159746A1 (en) | Concurrent data imports | |
| CN113177091B (en) | Incremental data storage method and device, storage medium and electronic device | |
| CN111177243B (en) | Data export method and device, storage medium and electronic device | |
| CN110569129A (en) | Resource allocation method and device, storage medium and electronic device | |
| CN109657317B (en) | CPLD pin allocation method, system and equipment | |
| CN108629050B (en) | Service data adjustment method, device, computer equipment and storage medium | |
| CN116805946A (en) | Message request processing method and device, electronic equipment and storage medium | |
| US10942787B2 (en) | Instance mapping engine and tools | |
| CN111008220A (en) | Dynamic identification method and device of data source, storage medium and electronic device | |
| CN112231405A (en) | Data storage device | |
| CN113746918A (en) | Hypertext transfer protocol proxy method and system | |
| US11100077B2 (en) | Event table management using type-dependent portions | |
| CN110276212B (en) | Data processing method and device, storage medium and electronic device | |
| CN112653539A (en) | Method, device and equipment for storing data to be stored | |
| CN110751204A (en) | Data fusion method and device, storage medium and electronic device | |
| CN113434598B (en) | Method, device and electronic device for realizing data double writing | |
| CN113852919B (en) | Method and device for generating early warning message, storage medium and electronic device | |
| CN116909816B (en) | Database recovery method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |