CN117786000A - Data synchronization method, device, computer equipment and storage medium - Google Patents
Data synchronization method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN117786000A CN117786000A CN202311618209.XA CN202311618209A CN117786000A CN 117786000 A CN117786000 A CN 117786000A CN 202311618209 A CN202311618209 A CN 202311618209A CN 117786000 A CN117786000 A CN 117786000A
- Authority
- CN
- China
- Prior art keywords
- data
- consumer
- preset
- cache
- preset space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000015654 memory Effects 0.000 claims description 26
- 238000012216 screening Methods 0.000 claims description 19
- 238000013507 mapping Methods 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 7
- 238000013500 data storage Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000010845 search algorithm Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of computers and discloses a data synchronization method, a device, computer equipment and a storage medium, wherein the data synchronization method is applied to a data user, and the data user comprises at least one consumer; on the basis, when the data operation instruction sent by at least one consumer is obtained, the data information corresponding to the data operation instruction is screened in the preset space, so that the data obtaining efficiency of at least one consumer in the data using party is improved, and meanwhile, the network congestion problem caused by large data quantity and large number of consumers is avoided.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data synchronization method, a data synchronization device, a computer device, and a storage medium.
Background
In a micro-service architecture, a plurality of separate programs are typically respectively provided to be solely responsible for corresponding services. When different service modules communicate based on service demands, a large amount of network resource overhead is generated under the condition of large base number of consumers, and the network performance requirement is extremely high.
Disclosure of Invention
In view of the above, the present invention provides a data synchronization method, apparatus, computer device and storage medium, so as to solve the problem of large resource overhead of the service communication network.
In a first aspect, the present invention provides a data synchronization method, applied to a data consumer, the data consumer comprising at least one consumer, the method comprising:
obtaining cache data from a data source;
storing the cache data into a preset space;
when a data operation instruction sent by at least one consumer is obtained, screening data information corresponding to the data operation instruction in a preset space.
The data synchronization method has the advantages that the data synchronization method is applied to a data user, the data user comprises at least one consumer, the method comprises the steps of obtaining cache data from a data source, storing the cache data in a preset space, and the problems of data obtaining barriers and the like caused by network delay and the like caused by the fact that the data user obtains data from the data source when the data user needs to obtain corresponding data are avoided; on the basis, when the data operation instruction sent by at least one consumer is obtained, the data information corresponding to the data operation instruction is screened in the preset space, so that the data obtaining efficiency of at least one consumer in the data using party is improved, and meanwhile, the network congestion problem caused by large data quantity and large number of consumers is avoided.
In an alternative embodiment, the method for obtaining the cache data from the data source specifically includes:
and obtaining the cache data from the data source according to the default configuration parameters.
In an alternative embodiment, storing the cached data in a preset space specifically includes:
distributing preset index information for the cache data;
and storing the cache data in a preset space corresponding to the preset index according to the preset index information.
In an optional implementation manner, the data operation instruction includes a data identifier, and the data information corresponding to the data operation instruction is screened in a preset space, which specifically includes:
according to the data identification, determining preset index information corresponding to the data identification according to the mapping relation between the data identification and the preset index information;
screening data information corresponding to preset index information in a preset space.
In an alternative embodiment, the method further comprises:
when the data information is not screened in the preset space, establishing connection with a data source corresponding to the data operation instruction according to a preset mode;
corresponding data information is obtained from a data source.
In an alternative embodiment, the method further comprises:
when the change of the data in the data source is monitored, determining changed data information;
setting an update label for cache data corresponding to the data information in a preset space, wherein the update label is used for acquiring the updated data information according to the update label when at least one consumer needs to acquire the data information corresponding to the update label.
In a second aspect, the present invention provides a data synchronizing device for use with a data consumer, the data consumer comprising at least one consumer, the device comprising:
the data acquisition module is used for acquiring cache data from a data source;
the data storage module is used for storing the cache data to a preset space;
and the screening data module is used for screening data information corresponding to the data operation instruction in a preset space when the data operation instruction sent by at least one consumer is acquired.
In a third aspect, the present invention provides a computer device comprising: the data synchronization method comprises the steps of storing computer instructions in a memory and a processor, wherein the memory and the processor are in communication connection, and the processor executes the computer instructions, so that the data synchronization method of the first aspect or any corresponding implementation mode is executed.
In a fourth aspect, the present invention provides a computer readable storage medium having stored thereon computer instructions for causing a computer to perform the data synchronization method of the first aspect or any of its corresponding embodiments.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method of device fault detection according to an embodiment of the present invention;
FIG. 2 is a block diagram of a data synchronization device according to an embodiment of the present invention;
fig. 3 is a schematic hardware structure of a computer device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In this embodiment, a data synchronization method is provided, and fig. 1 is a flowchart of the data synchronization method according to an embodiment of the present invention. The method is deployed on a data consumer, wherein the data consumer comprises at least one consumer, and each consumer can acquire corresponding data based on the data consumer. As shown in fig. 1, the data synchronization method includes the steps of:
step S101, obtaining cache data from a data source.
Illustratively, the user may obtain certain cached data from the data source in advance.
First, a data source from which the cached data is to be obtained needs to be determined. This may be a database, file, API interface, or any other data source that provides data. Connecting a data source: depending on the type of data source, appropriate methods are used to connect to the data source. For example, if the data source is a database, a database connection library (e.g., sqlite3 or psychrog 2 in Python) may be used to establish the connection. Executing a query or request: once connected to the data source, a query or request may be performed to retrieve the data. This may involve writing SQL queries, sending HTTP requests, or performing other data source specific operations.
Parsing and extracting data: depending on the data format returned by the data source, useful information may need to be parsed and extracted. This may involve parsing JSON, XML, or other markup language, or extracting data from a database result set.
In an actual application scenario, the data sources include a plurality of data sources, and the data user refers to the data sources and includes a plurality of data sources, where corresponding data in all the data sources is cached, and in a preferred embodiment, the method includes obtaining cached data from the data sources specifically: and obtaining the cache data from the data source according to the default configuration parameters.
For example, when data in a data source is cached for the first time, the data to be cached may be determined according to a default configuration parameter in a data user, for example, the default configuration parameter may be data source information, and a data size, a data type, and the like of the cache.
Step S102, storing the cache data to a preset space.
After the cache data is acquired, the cache data is stored in a preset space, so that corresponding consumers in subsequent data users can directly acquire the corresponding data from the preset space when the corresponding consumers need to acquire the corresponding data, the time delay of the consumers for acquiring the data from the data source is reduced, and the data use efficiency is improved.
Correspondingly, the validity period of the data and the like can be set for the cache data in the preset space, so that the safety of the data is improved. After the data storage is finished, the connection with the data source is disconnected, so that the safety of the data is ensured, and the data source party and the data user party are prevented from being attacked by means of network attack and the like.
In a preferred embodiment, the step S102 specifically includes:
distributing preset index information for the cache data;
and storing the cache data in a preset space corresponding to the preset index according to the preset index information.
For example, after the preset index information is allocated to the cache data, the cache data may be stored in the corresponding preset space according to the preset index information. The preset index information may be an identification or tag for quickly finding and locating a particular data item in the cached data. By using the preset index information, a specific area or location for storing the cache data can be created or identified in the preset space.
When the cache data is stored in a preset space corresponding to the preset index, the data item can be placed in a corresponding storage area according to the preset index information. This storage area may be a location on a physical storage device or a particular portion of virtual storage space. By storing the cache data in accordance with the preset index information, it is possible to manage and access the data more efficiently. When the cached data needs to be accessed or retrieved, the corresponding preset index information can be used to quickly locate and find the required data item.
It should be noted that, in the process of allocating the preset index information to the cache data and storing the same in the preset space, factors such as the type, structure and use requirement of the data need to be considered. Different data types and structures may require the use of different indexing policies and storage approaches to optimize cache performance and efficiency.
Step S103, when the data operation instruction sent by at least one consumer is obtained, screening data information corresponding to the data operation instruction in a preset space.
For example, after the data operation instruction sent by at least one consumer is obtained, screening of the data information may be performed in a preset space to find the data information corresponding to the data operation instruction. For example, a consumer may send a data manipulation instruction requesting modification of a particular data item in a particular database. After receiving this instruction, a backup or copy of the database may be found in a preset space and screened to find data items that need to be modified.
The screening process may be automated or may be performed manually. In automated screening, a search algorithm or filter may be used to automatically identify data information that matches the data manipulation instructions. In manual screening, each piece of data information may be manually inspected by a professional to find items that match the instructions. Once the data information corresponding to the data manipulation instruction is found, the instruction may be executed.
In short, screening data information corresponding to data operation instructions in a preset space is an effective method, and data items needing to be operated can be quickly and accurately found and corresponding operations can be executed.
In a preferred embodiment, the step S103 specifically includes:
according to the data identification, determining preset index information corresponding to the data identification according to the mapping relation between the data identification and the preset index information;
screening data information corresponding to preset index information in a preset space.
Illustratively, according to the mapping relationship between the data identifier and the preset index information, the preset index information corresponding to the data identifier may be determined. The mapping relationship may be a predefined correspondence relationship, and is used to map the data identifier to the corresponding preset index information.
Once the preset index information corresponding to the data identifier is determined, the data information corresponding to the preset index information can be screened in a preset space. Through the process, the data information corresponding to the data identifier can be quickly found from the preset space, and corresponding operation is performed. It should be noted that the establishment and maintenance of the mapping relationship is a key link. Ensuring the accuracy and consistency of the mapping is critical to the proper processing of the cached data. Meanwhile, it is very important to select proper cache management strategies and algorithms according to actual demands and conditions.
On the basis of the embodiment, when no data information is screened in the preset space, establishing connection with a data source corresponding to the data operation instruction according to a preset mode; corresponding data information is obtained from a data source.
After the data user acquires the cache data, the data user also needs to continuously monitor the data change of the data source, and after the data in the data source is changed, the changed data information is determined; setting an update label for cache data corresponding to the data information in a preset space, wherein the update label is used for acquiring the updated data information according to the update label when at least one consumer needs to acquire the data information corresponding to the update label.
The data producer can broadcast and inform the data consumer monitoring the fence in a redispub/sub mode, the inside of the fence sdk can mark the unit, and the unit is distinguished according to the mark when the consumer generates the call, so that instant pressure to the producer is avoided because a large number of listeners receive notification at the same time and return to the source immediately.
When a consumer in a data user needs to acquire or use the data, connection is established to acquire corresponding change data from a preset space or from a data source, so that the problems that in the prior art, when the change data appears, whether the consumer uses the data or not, the change data is synchronized to each consumer in the same way, the data quantity is increased sharply, the instantaneous data synchronization content is larger, flow spike is easy to cause, and the data synchronization is slow and failure are avoided.
The embodiment also provides a data synchronization device, which is used for implementing the foregoing embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
The present embodiment provides a data synchronization device, as shown in fig. 2, applied to a data consumer, where the data consumer includes at least one consumer, the device includes:
an acquisition data module 201, configured to acquire cache data from a data source;
a data storage module 202, configured to store the buffered data into a preset space;
and the screening data module 203 is configured to screen, when the data operation instruction sent by at least one consumer is obtained, data information corresponding to the data operation instruction in a preset space.
In some alternative embodiments, the data acquisition module is specifically configured to: and obtaining the cache data from the data source according to the default configuration parameters.
In some alternative embodiments, the data module is stored, in particular for:
distributing preset index information for the cache data;
and storing the cache data in a preset space corresponding to the preset index according to the preset index information.
In some alternative embodiments, the screening data module is specifically configured to:
according to the data identification, determining preset index information corresponding to the data identification according to the mapping relation between the data identification and the preset index information;
screening data information corresponding to preset index information in a preset space.
In some alternative embodiments, the apparatus is further for:
when the data information is not screened in the preset space, establishing connection with a data source corresponding to the data operation instruction according to a preset mode;
corresponding data information is obtained from a data source.
In some alternative embodiments, the apparatus is further for:
when the change of the data in the data source is monitored, determining changed data information;
setting an update label for cache data corresponding to the data information in a preset space, wherein the update label is used for acquiring the updated data information according to the update label when at least one consumer needs to acquire the data information corresponding to the update label.
Further functional descriptions of the above respective modules and units are the same as those of the above corresponding embodiments, and are not repeated here.
The data synchronization means in this embodiment are presented in the form of functional units, here referred to as ASIC (Application Specific Integrated Circuit ) circuits, processors and memories executing one or more software or fixed programs, and/or other devices that can provide the above described functionality.
The embodiment of the invention also provides computer equipment, which is provided with the data synchronization device shown in the figure 2.
Referring to fig. 3, fig. 3 is a schematic structural diagram of a computer device according to an alternative embodiment of the present invention, and as shown in fig. 3, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are communicatively coupled to each other using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the computer device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In some alternative embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple computer devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 10 is illustrated in fig. 3.
The processor 10 may be a central processor, a network processor, or a combination thereof. The processor 10 may further include a hardware chip, among others. The hardware chip may be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general-purpose array logic, or any combination thereof.
Wherein the memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform a method for implementing the embodiments described above.
The memory 20 may include a storage program area that may store an operating system, at least one application program required for functions, and a storage data area; the storage data area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, memory 20 may optionally include memory located remotely from processor 10, which may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk, or solid state disk; the memory 20 may also comprise a combination of the above types of memories.
The computer device also includes a communication interface 30 for the computer device to communicate with other devices or communication networks.
The embodiments of the present invention also provide a computer readable storage medium, and the method according to the embodiments of the present invention described above may be implemented in hardware, firmware, or as a computer code which may be recorded on a storage medium, or as original stored in a remote storage medium or a non-transitory machine readable storage medium downloaded through a network and to be stored in a local storage medium, so that the method described herein may be stored on such software process on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, a solid state disk or the like; further, the storage medium may also comprise a combination of memories of the kind described above. It will be appreciated that a computer, processor, microprocessor controller or programmable hardware includes a storage element that can store or receive software or computer code that, when accessed and executed by the computer, processor or hardware, implements the methods illustrated by the above embodiments. Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope of the invention as defined by the appended claims.
Claims (9)
1. A method of data synchronization, for application to a data consumer, the data consumer comprising at least one consumer, the method comprising:
obtaining cache data from a data source;
storing the cache data into a preset space;
when a data operation instruction sent by at least one consumer is obtained, screening data information corresponding to the data operation instruction in the preset space.
2. The method according to claim 1, wherein the obtaining the cached data from the data source specifically comprises:
and obtaining cache data from the data source according to default configuration parameters.
3. The method according to claim 2, wherein the storing the cache data in a preset space specifically includes:
distributing preset index information for the cache data;
and storing the cache data in the preset space corresponding to the preset index according to the preset index information.
4. The method according to claim 3, wherein the data operation instruction includes a data identifier, and the screening the data information corresponding to the data operation instruction in the preset space specifically includes:
determining preset index information corresponding to the data identifier according to the mapping relation between the data identifier and the preset index information and the data identifier;
screening data information corresponding to the preset index information in the preset space.
5. The method according to any one of claims 1-4, further comprising:
when the data information is not screened in the preset space, establishing connection with a data source corresponding to the data operation instruction according to a preset mode;
and acquiring corresponding data information from the data source.
6. The method of claim 5, wherein the method further comprises:
when the change of the data in the data source is monitored, determining changed data information;
setting an update label for cache data corresponding to the data information in the preset space, wherein the update label is used for acquiring updated data information according to the update label when the at least one consumer needs to acquire the data information corresponding to the update label.
7. A data synchronization device for application to a data consumer, the data consumer comprising at least one consumer, the device comprising:
the data acquisition module is used for acquiring cache data from a data source;
the data storage module is used for storing the cache data to a preset space;
and the screening data module is used for screening data information corresponding to the data operation instruction in the preset space when the data operation instruction sent by at least one consumer is acquired.
8. A computer device, comprising:
a memory and a processor, the memory and the processor being communicatively coupled to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the data synchronization method of any of claims 1 to 6.
9. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the data synchronization method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311618209.XA CN117786000A (en) | 2023-11-29 | 2023-11-29 | Data synchronization method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311618209.XA CN117786000A (en) | 2023-11-29 | 2023-11-29 | Data synchronization method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117786000A true CN117786000A (en) | 2024-03-29 |
Family
ID=90391762
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311618209.XA Pending CN117786000A (en) | 2023-11-29 | 2023-11-29 | Data synchronization method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117786000A (en) |
-
2023
- 2023-11-29 CN CN202311618209.XA patent/CN117786000A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8069224B2 (en) | Method, equipment and system for resource acquisition | |
| CN109542529B (en) | Function name-based point burying method and device, computer equipment and storage medium | |
| US20150378721A1 (en) | Methods for managing applications using semantic modeling and tagging and devices thereof | |
| CN109639750B (en) | Service data processing method and device | |
| CN111460129B (en) | Method, device, electronic equipment and storage medium for generating identification | |
| CN107241242B (en) | Data processing method and device | |
| CN103034735A (en) | Big data distributed file export method | |
| CN109167840B (en) | Task pushing method, node autonomous server and edge cache server | |
| CN110543512B (en) | Information synchronization method, device and system | |
| CN112364110A (en) | Metadata management method, device and equipment and computer storage medium | |
| CN103716384A (en) | Method and device for realizing cloud storage data synchronization in cross-data-center manner | |
| CN107885634B (en) | Method and device for processing abnormal information in monitoring | |
| CN112463549A (en) | Auditing method, device and equipment of cloud platform and computer readable storage medium | |
| CN112835978A (en) | Data storage method and device and computer equipment | |
| CN110955460B (en) | Service process starting method and device, electronic equipment and storage medium | |
| CN117786000A (en) | Data synchronization method, device, computer equipment and storage medium | |
| US10237372B2 (en) | Scalable programming architecture for telecommunications devices | |
| CN110493326B (en) | Zookeeper-based cluster configuration file management system and method | |
| CN113438095B (en) | Method, device and equipment for managing configuration data and storage medium | |
| CN112579247A (en) | Method and device for determining task state | |
| CN111367860A (en) | File refreshing method and device | |
| CN117240825B (en) | Address library construction method, device, equipment and medium applied to CDN | |
| CN116484373B (en) | Abnormal process checking and killing method, system, device, computer equipment and storage medium | |
| CN112118140B (en) | CDN configuration method, CDN configuration device, computer equipment and storage medium | |
| CN112732757B (en) | Method, system, device, equipment and storage medium for processing degraded data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |