CN113177091B - Incremental data storage method and device, storage medium and electronic device - Google Patents
Incremental data storage method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN113177091B CN113177091B CN202110548480.5A CN202110548480A CN113177091B CN 113177091 B CN113177091 B CN 113177091B CN 202110548480 A CN202110548480 A CN 202110548480A CN 113177091 B CN113177091 B CN 113177091B
- Authority
- CN
- China
- Prior art keywords
- incremental data
- determining
- region code
- data
- region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Abstract
The embodiment of the application provides a storage method and device of incremental data, a storage medium and an electronic device, wherein the method comprises the following steps: determining the region code corresponding to the obtained incremental data; determining a target storage node corresponding to the incremental data from the determined mapping relation between the region code and the storage node; and storing the incremental data to a target database according to the target storage node. The application solves the problem of global zoning storage of the incremental data in the related technology and achieves the effect of zoning storage of the incremental data.
Description
Technical Field
The embodiment of the application relates to the field of communication, in particular to a storage method and device of incremental data, a storage medium and an electronic device.
Background
In the use process of the database, when the storage structure and the data security of the database can not meet the service requirement, the data in the database is required to be migrated and backed up to a new database which can meet the service requirement, and the problem of migration of a large amount of data is solved; while making global data storage more challenging in the context of increasingly strong global data security regionalization protections. More and more countries propose that data in this area needs to be saved or backed up to this area.
In the prior art, when incremental data is distributed and stored, only a message distribution strategy is simply described, the distribution strategy is not introduced by a message, and the problem of globalization data dynamic distribution is not solved.
Aiming at the problem of global zoning storage of incremental data in the prior art, no effective solution has been proposed in the related art.
Disclosure of Invention
The embodiment of the application provides a storage method and device of incremental data, a storage medium and an electronic device, which at least solve the problem of global regional storage of the incremental data in the related technology.
According to an embodiment of the present application, there is provided a method of storing incremental data, including: determining the region code corresponding to the obtained incremental data; determining a target storage node corresponding to the incremental data from the determined mapping relation between the region codes and the storage nodes; and storing the incremental data to a target database according to the target storage node.
According to another embodiment of the present application, there is provided a storage device for incremental data, including: the first determining module is used for determining the region code corresponding to the acquired incremental data; the second determining module is used for determining a target storage node corresponding to the incremental data from the determined mapping relation between the region codes and the storage nodes; and the first storage module is used for storing the incremental data to a target database according to the target storage node.
In an exemplary embodiment, the first determining module includes: a first analyzing unit, configured to analyze user information corresponding to the incremental data from a message queue where the incremental data is located; and a first determining unit configured to determine an area code corresponding to the incremental data from the area attribute of the user information.
In an exemplary embodiment, the first determining module includes: a second determining unit, configured to determine a session IP address of a client that obtains the incremental data, where user information corresponding to the incremental data is not parsed from a message queue in which the incremental data is located; a second analyzing unit, configured to analyze, according to the IP address, an area where the incremental data is located; and a third determining unit, configured to determine an area code corresponding to the incremental data by using an area where the incremental data is located.
In an exemplary embodiment, the first determining module includes: the self-defining unit is used for self-defining the region code corresponding to the incremental data to determine the region code corresponding to the acquired incremental data under the condition that the user information corresponding to the incremental data is not analyzed from the message queue where the incremental data is located and the IP address of the client of the incremental data is not acquired.
In an exemplary embodiment, the customization unit includes: a first dividing subunit configured to divide a global time zone into N encoding regions, where N is a natural number greater than 1; a second dividing subunit configured to divide the incremental data into target code areas corresponding to the time of generating the incremental data, from among the N code areas, according to the time of generating the incremental data; and the first determination subunit is used for determining the region code corresponding to the incremental data based on the region code corresponding to the target coding region.
In an exemplary embodiment, the apparatus further includes a third determining module, configured to determine a mapping relationship between the region code and the storage node before determining a target storage node corresponding to the incremental data from the determined mapping relationship between the region code and the storage node; and the storage module is used for storing the mapping relation into a remote dictionary service cache.
In an exemplary embodiment, the above apparatus further includes: and a changing module configured to change, in the remote dictionary service cache, a mapping relationship between the region code and the storage node when the storage node corresponding to the region code is changed after the mapping relationship is stored in the remote dictionary service cache.
In an exemplary embodiment, the above apparatus further includes: a fourth determining module, configured to determine an event attribute of the incremental data when determining a data table corresponding to the incremental data before determining an area code corresponding to the acquired incremental data; the first encapsulation module is used for encapsulating the event attribute and the increment data to obtain encapsulation data; and the first sending module is used for sending the encapsulated data to a message queue.
According to a further embodiment of the application, there is also provided a computer readable storage medium having stored therein a computer program, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.
According to a further embodiment of the application, there is also provided an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.
According to the application, the region codes corresponding to the acquired incremental data are determined; determining a target storage node corresponding to the incremental data from the determined mapping relation between the region code and the storage node; and storing the incremental data to a target database according to the target storage node. The method and the device achieve the aim of determining the storage node corresponding to the incremental data based on the acquired area code of the incremental data. Therefore, the problem of global zoning storage of the incremental data in the related technology can be solved, and the effect of zoning storage of the incremental data is achieved.
Drawings
FIG. 1 is a block diagram of a mobile terminal in hardware configuration of a method for storing incremental data according to an embodiment of the present application;
FIG. 2 is a flow chart of a method of storing delta data according to an embodiment of the present application;
FIG. 3 is an overall flow chart according to an embodiment of the application;
FIG. 4 is a flow chart of determining region encoding according to an embodiment of the present application;
FIG. 5 is a flow chart of dynamic partitioning according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a system module according to an embodiment of the application;
fig. 7 is a block diagram of a storage device for delta data according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in detail below with reference to the accompanying drawings in conjunction with the embodiments.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided in the embodiments of the present application may be performed in a mobile terminal, a computer terminal or similar computing device. Taking the mobile terminal as an example, fig. 1 is a block diagram of a hardware structure of the mobile terminal according to an embodiment of the present application. As shown in fig. 1, a mobile terminal may include one or more (only one is shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a microprocessor MCU or a processing device such as a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 for communication functions and an input-output device 108. It will be appreciated by those skilled in the art that the structure shown in fig. 1 is merely illustrative and not limiting of the structure of the mobile terminal described above. For example, the mobile terminal may also include more or fewer components than shown in fig. 1, or have a different configuration than shown in fig. 1.
The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a method for storing incremental data in an embodiment of the present application, and the processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, that is, implement the above-mentioned method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, which may be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission means 106 is arranged to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet wirelessly.
In this embodiment, a method for storing incremental data is provided, and fig. 2 is a flowchart of a method for storing incremental data according to an embodiment of the present application, as shown in fig. 2, where the flowchart includes the following steps:
step S202, determining a region code corresponding to the acquired incremental data;
step S204, determining a target storage node corresponding to the incremental data from the mapping relation between the determined region code and the storage node;
step S206, the incremental data is stored in the target database according to the target storage node.
The main execution body of the above steps may be a terminal, but is not limited thereto.
This embodiment includes, but is not limited to, application in scenarios where incremental data is stored in regions.
Through the steps, the region codes corresponding to the acquired incremental data are determined; determining a target storage node corresponding to the incremental data from the determined mapping relation between the region code and the storage node; and storing the incremental data to a target database according to the target storage node. The method and the device achieve the aim of determining the storage node corresponding to the incremental data based on the acquired area code of the incremental data. Therefore, the problem of global zoning storage of the incremental data in the related technology can be solved, and the effect of zoning storage of the incremental data is achieved.
In an exemplary embodiment, determining the region code corresponding to the acquired incremental data includes:
s1, analyzing user information corresponding to incremental data from a message queue where the incremental data is located;
s2, determining the region code corresponding to the incremental data from the region attribute of the user information.
In this embodiment, incremental data is set in a message queue, and the incremental data is obtained by analyzing the message queue, and the region code corresponding to the data is determined according to the partition matching rule.
In an exemplary embodiment, determining the region code corresponding to the acquired incremental data includes:
s1, determining a session networking protocol IP address of a client for acquiring incremental data under the condition that user information corresponding to the incremental data is not analyzed from a message queue in which the incremental data is positioned;
s2, analyzing the area where the incremental data is located according to the IP address;
s3, determining the region code corresponding to the incremental data by using the region where the incremental data is located.
In the present embodiment, the area information thereof is first determined from the user information.
In an exemplary embodiment, determining the region code corresponding to the acquired incremental data includes:
s1, under the condition that user information corresponding to the incremental data is not analyzed from a message queue where the incremental data is located and a session networking protocol IP address of a client of the incremental data is not acquired, the region code corresponding to the incremental data is customized to determine the region code corresponding to the acquired incremental data.
In an exemplary embodiment, the region code corresponding to the custom delta data includes:
s1, dividing a global time zone into N coding regions, wherein N is a natural number greater than 1;
s2, dividing the incremental data into target coding areas corresponding to the time for generating the incremental data in the N coding areas according to the time for generating the incremental data;
s3, determining the region code corresponding to the incremental data based on the region code corresponding to the target coding region.
In this embodiment, for example, the global time domain is divided into three large area codes, and is divided into α (west 12-west 5), β (west 4-east 4), and γ (east 5-east 12) with 0 time zone as the center. The data entering the custom partition rules are not matched to any region in the previous matching rules. The rule assumes that all records are generated at 8am-4pm operation in the zone where the user is located. By this rule, data of an unknown region is assigned to one of the large region regions. The time intervals corresponding to alpha, beta and gamma are alpha-12, -5, beta-4, 4 and gamma-5, 12 respectively.
In an exemplary embodiment, before determining the target storage node corresponding to the incremental data from the determined mapping relationship between the region code and the storage node, the method further includes:
s1, determining a mapping relation between region codes and storage nodes;
s2, storing the mapping relation into a remote dictionary service cache.
In this embodiment, the correspondence between the region code and the storage node region may be written into the redis cache in advance.
In one exemplary embodiment, after storing the mapping relationship in the remote dictionary service cache, the method further comprises:
s1, when the storage node corresponding to the region code is changed, the mapping relation between the region code and the storage node is changed in the remote dictionary cache.
In this embodiment, when the storage node region corresponding to the region code changes, only the mapping relation of redis needs to be changed, and the data execution service and the target database of the X region are newly added, and the data partition service will send data to the message queue of the new region. The function of dynamically storing the incremental data into the target database can be achieved, and other businesses are not affected.
In an exemplary embodiment, before determining the region code corresponding to the acquired incremental data, the method further includes:
s1, determining event attributes of incremental data under the condition that a data table corresponding to the incremental data is determined;
s2, packaging event attribute and incremental data to obtain packaging data;
and S3, sending the encapsulated data to a message queue.
In this embodiment, table names of partitions required are preset, binary log Binlog data analyzed by a tool Canal component for synchronizing incremental data is obtained, when the binary log Binlog data is matched with a corresponding table, events are divided into DELETE, update UPADTE, INSERT INSERT according to EventType, each Column is analyzed, a partition index is obtained, and finally executable sql sentences are assembled. And subpackaging the two parts of data into json objects and sending the json objects to a message queue.
The application will be described in detail with reference to specific examples below:
as shown in fig. 3, the overall flowchart in this embodiment includes the following steps:
s301: a source database, which is to start a binary log writing function and set binlog-format as a ROW mode ROW mode;
s302: the data preprocessing module is used for presetting table names needing partitioning, analyzing Binlog data by the Canal component, dividing an event into DELETE, UPADTE, INSERT according to event attribute EventType when the table names are matched with the corresponding table, analyzing each read-only attribute Column, acquiring a partition index (the function is a value or a reference of the value in a return table or a region, two forms, an array form and a reference form, wherein the array form usually returns a numerical value or a numerical value array, the reference form usually returns a reference), and simultaneously assembling the finally executable sql statement. The two parts of data are split into object numbered musical notation (JavaScript Object Notation, simply JSON) objects and sent to a message queue.
Examples: when the generated binlog is a user table, json objects are:
the index may be a user ID, order ID, device ID, etc.
S303: and a data partitioning module: the corresponding relation between the region code and the storage node region is written into a remote dictionary service (Remote Dictionary Server) redis cache in advance. The module consumes and analyzes the data in the message queue, and determines the region code corresponding to the data according to the partition matching rule. As shown in fig. 4, is determined by matching multiple dimensions. Firstly, determining the area information according to the user information, when the user information cannot be judged through the existing information, analyzing the area where the user information is located through acquiring an Internet protocol address (Internet Protocol Address, abbreviated as IP) address when the client operates, and when the user information cannot determine the area, adopting a self-defined strategy to allocate the area.
The method comprises the steps of customizing a strategy, dividing the global time into three large area coding codes, and dividing the three large area coding codes into alpha (Western 12-Western 5), beta (Western 4-east 4) and gamma (east 5-east 12) by taking a 0 time zone as a center. The data entering the custom partition rules are not matched to any region in the previous matching rules. The rule assumes that all records are generated at 8am-4pm operation in the zone where the user is located. By this rule, data of an unknown region is assigned to one of the large region regions. The time intervals corresponding to alpha, beta and gamma are alpha-12, -5, beta-4, 4 and gamma-5, 12 respectively.
Partition formula: r (α, β, γ) =12-t, where t is the time integer number of UTC.
Examples: a general time coordination (Universal Time Chiming, abbreviated as UTC), the data generated by time 03:15, t is 3, and the region code is γ.
After the region code is obtained, the region information of the storage nodes in the service local cache is sent to the message queues of different storage nodes. The service local cache synchronizes the code and region mapping relation from the redis at regular time.
As shown in FIG. 5, when the storage node region corresponding to the region code changes, the mapping relation of the redis is only required to be changed, and the data execution service and the target database of the X region are newly added, and the data partition service sends data to the message queue of the new region. The function of dynamically storing the incremental data into the target database can be achieved, and other businesses are not affected.
Examples: when the data of the original area A is stored in the region a, and the data of the area A is required to be stored in the region x, the data can be realized only by changing the region corresponding to the area A from a to x.
S304: and the data execution module is used for: and consuming the data in the local area message queue and writing the data into a target database.
S305: the system module is shown in fig. 6.
In summary, the present embodiment determines the area information according to the user information, and when the user information cannot be determined by the existing information, the area is resolved by acquiring the IP address of the client during operation, and when the area cannot be determined by the above information, the user-defined policy is adopted to allocate the area. The custom strategy divides the world into three large areas on time and allocates areas according to the time the log was generated. The service local cache can synchronize the mapping relation of the code and the region from the redis at regular time, when the region of the storage node corresponding to the region code is changed, the corresponding relation of the redis is only required to be changed, the data execution service of the newly added X region and the target database can achieve the function of dynamically storing the incremental data into the target database, and other businesses are not affected.
From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present application.
The embodiment also provides a device for storing incremental data, which is used for implementing the foregoing embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
FIG. 7 is a block diagram of a device for storing incremental data according to an embodiment of the present application, as shown in FIG. 7, the device comprising:
a first determining module 72, configured to determine a region code corresponding to the acquired incremental data;
a second determining module 74, configured to determine a target storage node corresponding to the incremental data from the determined mapping relationship between the region code and the storage node;
a first storage module 76 for storing incremental data to the target database according to the target storage node.
In an exemplary embodiment, the first determining module includes:
a first analyzing unit, configured to analyze user information corresponding to the incremental data from a message queue where the incremental data is located;
and a first determining unit configured to determine an area code corresponding to the incremental data from the area attribute of the user information.
In an exemplary embodiment, the first determining module includes:
a second determining unit, configured to determine a session IP address of a client that obtains the incremental data, where user information corresponding to the incremental data is not parsed from a message queue in which the incremental data is located;
a second analyzing unit, configured to analyze, according to the IP address, an area where the incremental data is located;
and a third determining unit, configured to determine an area code corresponding to the incremental data by using an area where the incremental data is located.
In an exemplary embodiment, the first determining module includes:
the self-defining unit is used for self-defining the region code corresponding to the incremental data to determine the region code corresponding to the acquired incremental data under the condition that the user information corresponding to the incremental data is not analyzed from the message queue where the incremental data is located and the IP address of the client of the incremental data is not acquired.
In an exemplary embodiment, the customization unit includes:
a first dividing subunit configured to divide a global time zone into N encoding regions, where N is a natural number greater than 1;
a second dividing subunit configured to divide the incremental data into target code areas corresponding to the time of generating the incremental data, from among the N code areas, according to the time of generating the incremental data;
and the first determination subunit is used for determining the region code corresponding to the incremental data based on the region code corresponding to the target coding region.
In an exemplary embodiment, the apparatus further includes a third determining module, configured to determine a mapping relationship between the region code and the storage node before determining a target storage node corresponding to the incremental data from the determined mapping relationship between the region code and the storage node;
and the storage module is used for storing the mapping relation into a remote dictionary service cache.
In an exemplary embodiment, the above apparatus further includes: and a changing module configured to change, in the remote dictionary service cache, a mapping relationship between the region code and the storage node when the storage node corresponding to the region code is changed after the mapping relationship is stored in the remote dictionary service cache.
In an exemplary embodiment, the above apparatus further includes:
a fourth determining module, configured to determine an event attribute of the incremental data when determining a data table corresponding to the incremental data before determining an area code corresponding to the acquired incremental data;
the first encapsulation module is used for encapsulating the event attribute and the increment data to obtain encapsulation data;
and the first sending module is used for sending the encapsulated data to a message queue.
It should be noted that each of the above modules may be implemented by software or hardware, and for the latter, it may be implemented by, but not limited to: the modules are all located in the same processor; alternatively, the above modules may be located in different processors in any combination.
Embodiments of the present application also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.
In the present embodiment, the above-described computer-readable storage medium may be configured to store a computer program for performing the steps of:
s1, determining a region code corresponding to the acquired incremental data;
s2, determining a target storage node corresponding to the incremental data from the mapping relation between the determined region code and the storage node;
and S3, storing the incremental data to a target database according to the target storage node.
In one exemplary embodiment, the computer readable storage medium may include, but is not limited to: a usb disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing a computer program.
An embodiment of the application also provides an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.
In an exemplary embodiment, the electronic apparatus may further include a transmission device connected to the processor, and an input/output device connected to the processor.
In an exemplary embodiment, the above-mentioned processor may be arranged to perform the following steps by means of a computer program:
s1, determining a region code corresponding to the acquired incremental data;
s2, determining a target storage node corresponding to the incremental data from the mapping relation between the determined region code and the storage node;
and S3, storing the incremental data to a target database according to the target storage node.
Specific examples in this embodiment may refer to the examples described in the foregoing embodiments and the exemplary implementation, and this embodiment is not described herein.
It will be appreciated by those skilled in the art that the modules or steps of the application described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.
The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, but various modifications and variations can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the principle of the present application should be included in the protection scope of the present application.
Claims (6)
1. A method of storing incremental data, comprising:
determining the region code corresponding to the obtained incremental data;
determining a target storage node corresponding to the incremental data from the determined mapping relation between the region code and the storage node;
storing the incremental data to a target database according to the target storage node;
determining the region code corresponding to the acquired incremental data, wherein the region code comprises one of the following steps:
analyzing user information corresponding to the incremental data from a message queue in which the incremental data is located; determining the region code corresponding to the incremental data from the region attribute of the user information;
under the condition that user information corresponding to the incremental data is not analyzed from a message queue in which the incremental data is located, determining a session networking protocol (IP) address of a client for acquiring the incremental data; analyzing the area where the incremental data is located according to the IP address; determining a region code corresponding to the incremental data by using a region where the incremental data is located;
under the condition that user information corresponding to the incremental data is not analyzed from a message queue in which the incremental data is located and a session networking protocol (IP) address of a client of the incremental data is not acquired, the region code corresponding to the incremental data is customized to determine the region code corresponding to the acquired incremental data;
the customizing the region code corresponding to the incremental data comprises the following steps:
dividing the global time zone into N coding regions, wherein N is a natural number greater than 1;
dividing the incremental data into target coding areas corresponding to the time for generating the incremental data in the N coding areas according to the time for generating the incremental data;
determining the region code corresponding to the incremental data based on the region code corresponding to the target coding region;
before determining the target storage node corresponding to the incremental data from the determined mapping relationship between the region code and the storage node, the method further includes:
determining a mapping relation between the region codes and the storage nodes;
and storing the mapping relation into a remote dictionary service cache.
2. The method of claim 1, wherein after storing the mapping relationship in a remote dictionary service cache, the method further comprises:
and when the storage node corresponding to the region code is changed, changing the mapping relation between the region code and the storage node in the remote dictionary cache.
3. The method of claim 1, wherein prior to determining the region code corresponding to the acquired delta data, the method further comprises:
determining event attributes of the incremental data under the condition of determining a data table corresponding to the incremental data;
encapsulating the event attribute and the incremental data to obtain encapsulated data;
and sending the encapsulated data to a message queue.
4. A storage device for incremental data, comprising:
the first determining module is used for determining the region code corresponding to the acquired incremental data;
the second determining module is used for determining a target storage node corresponding to the incremental data from the determined mapping relation between the region codes and the storage nodes;
the first storage module is used for storing the incremental data to a target database according to the target storage node;
the first determining module determines the region code corresponding to the acquired incremental data by one of the following modes: analyzing user information corresponding to the incremental data from a message queue in which the incremental data is located; determining the region code corresponding to the incremental data from the region attribute of the user information; under the condition that user information corresponding to the incremental data is not analyzed from a message queue in which the incremental data is located, determining a session networking protocol (IP) address of a client for acquiring the incremental data; analyzing the area where the incremental data is located according to the IP address; determining a region code corresponding to the incremental data by using a region where the incremental data is located; under the condition that user information corresponding to the incremental data is not analyzed from a message queue in which the incremental data is located and a session networking protocol (IP) address of a client of the incremental data is not acquired, the region code corresponding to the incremental data is customized to determine the region code corresponding to the acquired incremental data;
the apparatus is further for dividing the global time zone into N encoding regions, wherein N is a natural number greater than 1; dividing the incremental data into target coding areas corresponding to the time for generating the incremental data in the N coding areas according to the time for generating the incremental data; determining the region code corresponding to the incremental data based on the region code corresponding to the target coding region;
the device is further used for determining the mapping relation between the region code and the storage node before determining the target storage node corresponding to the incremental data from the determined mapping relation between the region code and the storage node; and storing the mapping relation into a remote dictionary service cache.
5. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program, wherein the computer program is arranged to perform the method of any of the claims 1 to 3 when run.
6. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to run the computer program to perform the method of any of the claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110548480.5A CN113177091B (en) | 2021-05-19 | 2021-05-19 | Incremental data storage method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110548480.5A CN113177091B (en) | 2021-05-19 | 2021-05-19 | Incremental data storage method and device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113177091A CN113177091A (en) | 2021-07-27 |
| CN113177091B true CN113177091B (en) | 2023-10-10 |
Family
ID=76929379
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110548480.5A Active CN113177091B (en) | 2021-05-19 | 2021-05-19 | Incremental data storage method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113177091B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114928580B (en) * | 2022-04-12 | 2023-06-02 | 深圳市优必选科技股份有限公司 | Data increment transmitting method, receiving method, terminal device and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103995861A (en) * | 2014-05-16 | 2014-08-20 | 罗敬宁 | Distributed data device, method and system based on spatial correlation |
| CN109299088A (en) * | 2018-08-22 | 2019-02-01 | 中国平安人寿保险股份有限公司 | Mass data storage means, device, storage medium and electronic equipment |
| WO2019072103A1 (en) * | 2017-10-11 | 2019-04-18 | 阿里巴巴集团控股有限公司 | Method and apparatus for obtaining target data |
| CN110134755A (en) * | 2019-05-28 | 2019-08-16 | 北京百度网讯科技有限公司 | Map updating method, device, equipment, server and medium |
| US10459918B1 (en) * | 2016-06-28 | 2019-10-29 | Amazon Technologies, Inc. | Generating query results based on data partitions |
| CN111723148A (en) * | 2020-04-10 | 2020-09-29 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
| CN111723160A (en) * | 2020-08-24 | 2020-09-29 | 国网浙江省电力有限公司 | Multi-source heterogeneous incremental data synchronization method and system |
| CN112100147A (en) * | 2020-07-27 | 2020-12-18 | 杭州玳数科技有限公司 | Method and system for realizing real-time acquisition from Bilog to HIVE based on Flink |
| CN112307037A (en) * | 2019-07-26 | 2021-02-02 | 北京京东振世信息技术有限公司 | Data synchronization method and device |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6785191B2 (en) * | 2003-01-09 | 2004-08-31 | Micrel, Inc. | Robust power-on meter and method using a limited-write memory |
| US20130198103A1 (en) * | 2012-01-31 | 2013-08-01 | Sap Ag | Mapping Between Different Delta Handling Patterns |
| US11468014B2 (en) * | 2018-08-09 | 2022-10-11 | Netapp Inc. | Resynchronization to a filesystem synchronous replication relationship endpoint |
-
2021
- 2021-05-19 CN CN202110548480.5A patent/CN113177091B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103995861A (en) * | 2014-05-16 | 2014-08-20 | 罗敬宁 | Distributed data device, method and system based on spatial correlation |
| US10459918B1 (en) * | 2016-06-28 | 2019-10-29 | Amazon Technologies, Inc. | Generating query results based on data partitions |
| WO2019072103A1 (en) * | 2017-10-11 | 2019-04-18 | 阿里巴巴集团控股有限公司 | Method and apparatus for obtaining target data |
| CN109299088A (en) * | 2018-08-22 | 2019-02-01 | 中国平安人寿保险股份有限公司 | Mass data storage means, device, storage medium and electronic equipment |
| CN110134755A (en) * | 2019-05-28 | 2019-08-16 | 北京百度网讯科技有限公司 | Map updating method, device, equipment, server and medium |
| CN112307037A (en) * | 2019-07-26 | 2021-02-02 | 北京京东振世信息技术有限公司 | Data synchronization method and device |
| CN111723148A (en) * | 2020-04-10 | 2020-09-29 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
| CN112100147A (en) * | 2020-07-27 | 2020-12-18 | 杭州玳数科技有限公司 | Method and system for realizing real-time acquisition from Bilog to HIVE based on Flink |
| CN111723160A (en) * | 2020-08-24 | 2020-09-29 | 国网浙江省电力有限公司 | Multi-source heterogeneous incremental data synchronization method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113177091A (en) | 2021-07-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20170278003A1 (en) | Complex Event Processing Method, Apparatus, and System | |
| CN111552838B (en) | Data processing method and device, computer equipment and storage medium | |
| CN111723148B (en) | Data storage method and device, storage medium and electronic device | |
| WO2020248375A1 (en) | Method and system for synchronizing data between databases, computer device and storage medium | |
| CN110427494B (en) | Knowledge graph display method and device, storage medium and electronic device | |
| CN113177091B (en) | Incremental data storage method and device, storage medium and electronic device | |
| CN112188144B (en) | Audio transmission method and device, storage medium and electronic device | |
| CN110569129A (en) | Resource allocation method and device, storage medium and electronic device | |
| CN111177243A (en) | Data export method and device, storage medium and electronic device | |
| CN108629050B (en) | Service data adjustment method, device, computer equipment and storage medium | |
| CN111338787B (en) | Data processing method and device, storage medium and electronic device | |
| CN111367916B (en) | Data storage method and device | |
| CN109862134B (en) | Lease time configuration method and system of IP address and DHCP client | |
| CN110751204A (en) | Data fusion method and device, storage medium and electronic device | |
| CN113746918A (en) | Hypertext transfer protocol proxy method and system | |
| CN112579954A (en) | Method and device for outputting seismic information, storage medium and electronic device | |
| CN113852919B (en) | Method and device for generating early warning message, storage medium and electronic device | |
| CN110647546A (en) | Third-party rule engine generation method and device | |
| CN110276212B (en) | Data processing method and device, storage medium and electronic device | |
| CN110417624B (en) | Request statistical method, device and storage medium | |
| CN112738207B (en) | Keyword data transmission method and device, storage medium and electronic device | |
| CN113434598B (en) | Method, device and electronic device for realizing data double writing | |
| CN114201499B (en) | Data uploading method and device and computer equipment | |
| CN111400385A (en) | Data export method and device, storage medium and electronic device | |
| CN116739599A (en) | Transaction isolation processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |