CN113010549A - Data processing method based on remote multi-active system, related equipment and storage medium - Google Patents

Data processing method based on remote multi-active system, related equipment and storage medium Download PDF

Info

Publication number
CN113010549A
CN113010549A CN202110134158.8A CN202110134158A CN113010549A CN 113010549 A CN113010549 A CN 113010549A CN 202110134158 A CN202110134158 A CN 202110134158A CN 113010549 A CN113010549 A CN 113010549A
Authority
CN
China
Prior art keywords
target
data
target data
cache
cache space
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110134158.8A
Other languages
Chinese (zh)
Inventor
张哲铭
唐文宁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Cloud Computing Beijing Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN202110134158.8A priority Critical patent/CN113010549A/en
Publication of CN113010549A publication Critical patent/CN113010549A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24552Database cache management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • G06F16/2365Ensuring data consistency and integrity
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/275Synchronous replication

Abstract

The embodiment of the application discloses a data processing method based on a remote multi-active system, related equipment and a storage medium, which are applied to the technical field of computers. The method comprises the following steps: the method comprises the steps of receiving target data to be stored, caching the target data to a target cache space corresponding to target proxy equipment, broadcasting a first notification message to other proxy equipment after the target data are successfully cached to the target cache space, storing the target data to a target database in a plurality of databases, and performing data synchronization processing between the target database and other databases after the target data are successfully stored to the target database. By adopting the embodiment of the application, the data related to the scene of multiple different places and multiple lives can be better processed.

Description

Data processing method based on remote multi-active system, related equipment and storage medium
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method and apparatus, a proxy device, and a storage medium based on a remote multi-session system.
Background
With the development of the internet, the requirements on the stability and the real-time performance of the service are higher and higher; currently, the industry is also gradually beginning to use the remote multi-active technology to provide services to the outside even when a data server carrying service fails. The remote multi-activity service deployment method is a service deployment mode for ensuring high availability of services, and can avoid server risks by storing the services in different regions; the "live" is relative to the cold backup, the cold backup is to backup the full data, the service requirement is not supported at ordinary times, the standby machine room is switched to only when the main machine room (or called as the main server) fails, and the "live" means that the machine rooms (servers) also need to flow in daily service to support the service. At present, in an application scenario involving multi-place and multi-live, how to better perform data storage becomes a research hotspot.
Disclosure of Invention
The embodiment of the application provides a data processing method based on a remote multi-live system, related equipment and a storage medium, which can better process data related to a remote multi-live scene.
On one hand, the embodiment of the application provides a data processing method based on a remote multi-live system, wherein the remote multi-live system comprises a plurality of proxy devices, a cache space corresponding to each proxy device and a plurality of databases; wherein, the areas of any two agent devices are different; the method is performed by a target proxy device of the plurality of proxy devices, the method comprising:
receiving target data to be stored, and caching the target data to a target cache space corresponding to the target proxy equipment;
after the target data is successfully cached in the target cache space, broadcasting a first notification message to other proxy devices, where the first notification message is used to notify: if the target data exists in other cache spaces corresponding to the other proxy devices, deleting the target data in the other cache spaces;
and storing the target data into a target database in the plurality of databases, and after the target data is successfully stored into the target database, performing data synchronization processing between the target database and other databases.
In one aspect, an embodiment of the present application provides a data processing apparatus based on a remote multi-session system, where the remote multi-session system includes a plurality of proxy devices, a cache space corresponding to each proxy device, and a plurality of databases; wherein, the areas of any two agent devices are different; the apparatus operating in a target agent device of the plurality of agent devices, the apparatus comprising:
the receiving module is used for receiving target data to be stored and caching the target data to a target cache space corresponding to the target proxy equipment;
a notification module, configured to broadcast a first notification message to other proxy devices after the target data is successfully cached in the target cache space, where the first notification message is used to notify: if the target data exists in other cache spaces corresponding to the other proxy devices, deleting the target data in the other cache spaces;
and the processing module is used for storing the target data into a target database in the plurality of databases, and after the target data is successfully stored into the target database, executing data synchronization processing between the target database and other databases.
In one aspect, an embodiment of the present application provides a proxy device, where the proxy device includes a processor and a memory, where the processor and the memory are connected to each other, where the memory is used to store computer program instructions, and the processor is configured to execute the computer program instructions to implement part or all of the steps in the above data processing method.
In one aspect, the present application provides a computer-readable storage medium, in which computer program instructions are stored, and when the computer program instructions are executed by a processor, the computer program instructions are used to perform some or all of the steps in the data processing method.
In the embodiment of the application, the proxy device receives target data to be stored, caches the target data to a target cache space corresponding to the target proxy device, broadcasts a first notification message to other proxy devices after the target data is successfully cached to the target cache space, stores the target data to a target database in a plurality of databases, and executes data synchronization processing between the target database and other databases after the target data is successfully stored to the target database. Therefore, by implementing the above scheme, the high availability of the remote multi-live system can be realized, and meanwhile, the uniqueness of the data in the cache and the consistency of the data in the database can be ensured, so that the data involved in the remote multi-live scene can be better processed.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a remote multi-live system according to an embodiment of the present disclosure;
fig. 2 is a schematic flowchart of a data processing method based on a remote multi-session system according to an embodiment of the present application;
fig. 3 is a schematic diagram of a data processing scenario based on a remote multi-session system according to an embodiment of the present application;
fig. 4a is a schematic diagram of a data preview interface according to an embodiment of the present application;
FIG. 4b is a schematic diagram of another data preview interface provided in the embodiments of the present application;
FIG. 4c is a schematic diagram of another data preview interface provided in the embodiment of the present application;
fig. 5 is a schematic flowchart of another data processing method based on a remote multi-session system according to an embodiment of the present application;
fig. 6 is a schematic diagram of another data processing scenario based on a remote multi-session system according to an embodiment of the present application;
fig. 7 is a schematic diagram of a data processing scenario based on a remote multi-live system according to an embodiment of the present application;
FIG. 8 is a diagram illustrating a scenario for resolving data conflicts according to an embodiment of the present application;
fig. 9a is a schematic diagram of a data collision scenario provided in an embodiment of the present application;
fig. 9b is a schematic diagram of another data collision scenario provided in the embodiment of the present application;
fig. 9c is a schematic diagram of another data collision scenario provided by an embodiment of the present application;
FIG. 10 is a diagram illustrating another scenario for resolving data conflicts according to an embodiment of the present application;
fig. 11 is a schematic diagram of a remote multi-active system based fault resolution provided in an embodiment of the present application;
fig. 12 is a schematic structural diagram of a data processing apparatus based on a remote multi-session system according to an embodiment of the present application;
fig. 13 is a schematic structural diagram of a proxy device according to an embodiment of the present application.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
In order to better realize multi-live in different places, the embodiment of the application firstly provides a brand-new multi-live in different places system and a data processing scheme based on the multi-live in different places system; the data processing method and the data processing system have the advantages that high availability of the remote multi-living system is guaranteed, and meanwhile a series of data processing such as data storage and data modification can be better performed on data related in a remote multi-living scene, so that the user can be provided with interactive experience of the data such as the data stored in the local. Referring to fig. 1, the remote multi-active system proposed in the embodiment of the present application may sequentially include, in a top-down order: the system comprises an access gateway layer, a service access layer, a service layer, a data agent layer, a data cache layer and a data storage layer.
The service access layer may include M access gateway devices, where a so-called access gateway device refers to a device that implements network interconnection through protocol conversion, and may specifically be a sparta gateway (unified access gateway developed by Tencent corporation) or other gateways, which is not limited in this regard. The service access layer can comprise M access service devices, and one access service device can be in communication correspondence with one access gateway device in the access gateway layer; the access service device refers to a device that can allow a user terminal to access a service layer through a transmission medium such as an optical fiber, a twisted pair, a coaxial cable, a wireless access technology, and the like. The business service layer can comprise M business service devices, and one business service device is communicated with one business access device in the business access layer; the business service device refers to a device capable of providing one or more online services for the user terminal, such as various online services like account numbers, content, user behaviors, and the like.
The data proxy layer may include M proxy devices (also called DataProxy), the data cache layer may include M cache spaces, and the data storage layer may include N databases (data base, DB). Wherein, the values of M and N are positive integers which are more than or equal to 2, and the values of M and N can be the same or different, which is not limited; moreover, one proxy device corresponds to one cache space, and the areas of any two proxy devices are different from each other. Any proxy device mentioned herein can be used for acting on the cache space of the data cache layer and the database of the data bottom layer, so as to realize the consistency of the data in the cache layer and the consistency of the data bottom layer, and to unify the local operation experience outside. Wherein, the above mentioned cache space may be a non-relational database (NOSQL), such as a Remote Dictionary Server (Remote Dictionary Server); any database in the data store layer may be a relational database (SQL), without limitation.
The remote multi-live system can ensure high service availability and avoid risks, that is, when a cache space of a certain region fails, data in cache spaces of other regions can be synchronized into the cache space of the region, or users in the region can directly access the cache spaces of other regions, and when a certain database fails, data in other databases can be synchronized into the database, or users can directly access other databases. Among them, High Availability (High Availability) means that the system can reduce the downtime during which the service cannot be provided, thereby ensuring High Availability of the service. For example, as shown in fig. 1, when a failure occurs in a database located at D, disaster recovery data in a database located at E may be used, that is, the database located at E is a disaster recovery database of a database located at D, that is, the problem that the failed database cannot provide external services can be solved through remote disaster recovery. The disaster recovery data is backup data used when the server storing the data fails to work and the data is unavailable. Because the remote multi-live system is a cross-city remote multi-live system, the disaster recovery is also in a cross-city disaster recovery level, and further urban disasters such as earthquakes and the like can be effectively dealt with. It can be understood that, in the embodiments of the present application, consistency of data in a plurality of cache spaces and consistency of data in a plurality of databases can be achieved, so that any two cache spaces are disaster backup cache spaces and any two databases are disaster backup databases.
It should be noted that fig. 1 merely illustrates the system architecture of the remote multi-active system proposed in the embodiment of the present application, and does not limit the system architecture. For example, FIG. 1 illustrates, by way of example only, a cache space of A, B, C three places and a database of D, E two places; however, in actual deployment, the number of cache spaces may not be limited to 3, and the number of databases may not be limited to 2. For another example, in other embodiments, the offsite multi-live System may further include an internal Management System, which may be an OMS (Order Management System), a CMS (Content Management System), or a TMS (Transportation Management System). In this case, the number of proxy devices in the remote multi-active system may be greater than the number of service devices, that is, the number of proxy devices may not be limited to M; the system data generated by the internal management system can be selected and cached in a cache space of the data caching layer or stored in a database of the data storage layer by an agent device according to specific business requirements.
Moreover, any of the above-mentioned devices (such as proxy devices, business service devices, etc.) may be a server; the server mentioned here may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a CDN (Content Delivery Network), a big data and artificial intelligence platform, and the like. When any device is a cloud server, the remote multi-active system provided by the embodiment of the present application can also be used as a distributed cloud storage system (storage system) for implementing cloud storage (cloud storage); the storage system refers to a storage system which integrates a large number of storage devices (storage devices are also called storage nodes) of different types in a network through application software or application interfaces to cooperatively work together through functions of cluster application, grid technology, distributed storage file system and the like, and provides data storage and service access functions to the outside.
In addition, the above mentioned agent devices may be deployed in different cities in the same country, that is, the areas where the agent devices are located are divided by taking cities as units; in this case, the placeshifting multi-active system proposed in the embodiment of the present application can be understood as a placeshifting multi-active system at a cross-city level. Alternatively, the above mentioned agent devices may be deployed at different locations in the same city, that is, the areas where the agent devices are located are divided in units of specific locations; in this case, the placeshifting multi-active system proposed in the embodiment of the present application can be understood as a placeshifting multi-active system at the same city level. Still alternatively, the above mentioned proxy devices may be deployed in different countries, that is, the regions where the proxy devices are located are divided by country; in this case, the remote multi-active system proposed in the embodiment of the present application may be understood as a multiactive system at a transnational level. For convenience of illustration, the remote multi-active system mentioned later is described by taking the remote multi-active system at the city-crossing level as an example.
In a specific implementation, the operation principle of the above-mentioned remote multi-active system is as follows: when a user uses online services through terminal equipment, the access gateway layer selects one access gateway equipment for the terminal equipment according to the distance between the terminal equipment and each access gateway equipment, so that the selected access gateway equipment can realize interconnection between the terminal equipment and one access service equipment in the service access layer through protocol conversion. At this time, the access service device may access a certain service device of the service layer according to the online service used by the user, so that the accessed service device provides the online service for the user, and sends the data to be stored, which is generated in the process of providing the online service, to the corresponding proxy device, and the proxy device processes the data based on the data processing scheme provided in the embodiment of the present application. The above mentioned terminal devices may include but are not limited to: smart phones, tablets, laptops, and desktops, among others.
Specifically, the scheme principle of the data processing scheme is roughly as follows: when the proxy equipment receives the target data to be stored, the target data can be directly stored into the cache space corresponding to the proxy equipment by adopting a unique strategy of multiple caches, and then the target data is stored into each database in the data storage layer according to the strategy of DB data consistency, so that the data consistency among the databases is ensured. Or after receiving the target data to be stored, the agent device may also determine whether the target data needs to be stored in the database; if so, then storing the target data into a cache space corresponding to the proxy equipment by adopting a strategy of only caching more than one; otherwise, caching the target data into each cache space in the data cache layer by adopting a cache consistency strategy adaptive to the target data so as to ensure the data consistency among the cache spaces. Further, when the target data is modified at the same time in multiple places, the problem of data conflict in multiple places can be solved in different ways. By processing the data through the technical scheme, the consistency, the real-time performance and the high availability of the data can be realized, so that the user experience is improved.
It should be noted that the data processing scheme proposed in the embodiment of the present application can be applied to various service scenarios. For example, the data processing scheme may be applied to any distributed service scenario, where the distributed service scenario may be any read-write-more-less service scenario, that is, a service scenario in which the number of read requests of a user is greater than the number of write requests of the user, such as service scenarios of knowledge science popularization, information, and shopping guide; for another example, the data processing scheme can be applied to any flow document type business scene, such as business scenes of e-commerce transactions and bill flow. For another example, the data processing scheme can also be applied to any state-dependent business scenario, such as a business scenario of a bank account class, and the like. The application scenario is not limited in the embodiment of the present application, and with the evolution of the system architecture and the appearance of a new service scenario, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.
Based on the above-mentioned related description of the remote multi-active system and the data processing scheme, the embodiment of the present application provides a data processing method based on the remote multi-active system. As can be seen from the foregoing, the remote multi-live system may include a plurality of agent devices, a cache space corresponding to each agent device, and a plurality of databases; in a particular implementation, the data processing method may be performed by a target agent device of a plurality of agent devices. Referring to fig. 2, the data processing method based on the remote multi-live system may include the following steps S201 to S203:
s201, receiving target data to be stored, and caching the target data to a target cache space corresponding to the target proxy equipment.
The target data to be stored can be data generated by performing write operation through the terminal equipment in response to user operation; for example, a user issues comments through a terminal to generate data. Alternatively, the target data to be stored may also be data generated by an internal management system; for example, in a news information service scenario, data corresponding to news entries, hot spots, and the like, which are recommended and displayed for a user, and the like. The terminal devices mentioned in the embodiments of the present application may also be different according to different target data. Specifically, when the target data is data generated by performing write operation through the terminal device, the target proxy device mentioned in the embodiment of the present application may be a proxy device that is accessed according to a distance; for example, a proxy device in a region closest to the terminal is selected as a proxy device for near access, based on the distance between the terminal and the region in which the plurality of proxy devices are located. When the target data is data generated by the internal management system, the target agent device may be a designated agent device.
After receiving the target data to be stored, the target proxy device may cache the target data into a target cache space corresponding to the target proxy device, where the target cache space may be a cache space corresponding to the target proxy device. However, it should be noted that when the cache space corresponding to the target proxy device fails and cannot provide service, the target cache space may also be another designated cache space; for example, if the target proxy device corresponds to the cache space a and the other cache spaces are cache spaces B, the target cache space of the target proxy device is defined as the cache space a, and if the cache space a fails, the cache space B may be designated as the target cache space of the target proxy device.
In one embodiment, the cache mode adopted for the target data may be a direct storage mode; the specific implementation of caching the target data into the target cache space corresponding to the target proxy device may be: and directly storing the target data into a target cache space corresponding to the target proxy equipment. In one embodiment, the caching mode adopted for the target data may be key-value caching (key-value caching); the specific implementation of caching the target data into the target cache space corresponding to the target proxy device may be: analyzing the target data to obtain an identifier and data content contained in the target data; and taking the analyzed identifier as a key and the data content as a value corresponding to the key, and caching the key and the corresponding value into a target cache space. For example, the target data is data generated by performing a registration operation, an identifier obtained by analyzing the target data is a user ID, and the data content is a user password, user information, and the like; the user ID may be regarded as a key, and the user password, user information, etc. may be regarded as a value, and then the key and the corresponding value may be cached in the target cache space.
S202, after the target data are successfully cached in the target cache space, broadcasting a first notification message to other proxy equipment.
Wherein, other agent equipment means: proxy equipment in the long-distance multi-living system except the target proxy equipment; the number of other proxy devices may be one or more, and is not limited thereto. The first notification message is obtained according to the target data, and is used for notifying: and if the target data exists in other cache spaces corresponding to other proxy equipment, deleting the target data in the other cache spaces. The target data can be unique in the cache regions of a plurality of regions by broadcasting the first notification message to other agent devices, namely, the target data can be included in the target cache region corresponding to the target agent device in the target region; therefore, the uniqueness of the data storage operation can be ensured when the target data is stored in the database subsequently. Optionally, the first notification message may be generated after the target data is successfully cached in the target cache space, or may be generated asynchronously when the target data is cached in the target cache space. The optional first notification message may be broadcast through a message queue, which may be CKafka, Kafka, etc.
S203, storing the target data into a target database in the plurality of databases, and after the target data is successfully stored into the target database, executing data synchronization processing between the target database and other databases.
Wherein the target database may be any one of a plurality of databases. Alternatively, the target database may be a designated database of a plurality of databases; for example, when there is a master-slave relationship between multiple databases, the target database may be a master database of the multiple databases. In one embodiment, the target data may be stored directly to the target database by the target proxy device. In another embodiment, the target data may also be stored in the target database by the target agent device through a target data management component configured for the target agent device; the target data management component may be an asynchronous writer, such as flushroute (an asynchronous writer developed by tengchin corporation), and latency may be reduced to some extent by writing data asynchronously. It is to be understood that the target data management component may be located inside the target agent device or outside the target agent device, which is not limited thereto. In this case, the specific implementation of storing the target data into the target database in the plurality of databases may be: the target agent device may generate a second notification message and send the second notification message to the target data management component to notify the target data management component to store the target data in the target database.
The step of storing the target data in the target database of the multiple databases may be performed before broadcasting the first notification message, may be performed after broadcasting the first notification message, or may be performed while broadcasting the first notification message, and the embodiment of the present application is not limited herein. Also, when other agent devices are also configured with other data management components, the aforementioned "broadcast of the first notification message to other agent devices" in step S202 may also refer to: and broadcasting the first notification message to other data management components corresponding to other proxy equipment through the message queue, so that the other data management components delete the target data of other corresponding cache spaces when receiving the first notification message aiming at the target data. The second notification message may be generated simultaneously with the first notification message, may be generated before the first notification message, or may be generated after the first notification message, which is not limited herein in this embodiment of the application.
After the target data is successfully stored in the target database, the target agent device can perform data synchronization processing between the target database and other databases; the other databases refer to databases other than the target database in the multiple databases, and the number of the other databases may be one or more, which is not limited. Data between the target database and other databases are synchronized based on binlog (database write execution log), so that uniqueness of data writing and data consistency between databases are realized. Because the database in which the data is written is the target database, and the target database synchronizes the data to other databases, the other databases can be used as disaster recovery databases of the target database, when the target database fails, the target proxy device can switch the flow to the other databases, that is, when the target proxy device receives the request, the target proxy device accesses the other databases.
In one embodiment, in order to verify that the synchronized data of other databases is available and correct, the read traffic of other regions may be introduced into the other databases, and when the proxy devices of other regions receive the read request, the proxy devices of other regions may access the corresponding other databases, thereby implementing the verification of whether the data of other databases is correct at the time of service on the cable. For example, as shown in fig. 3, when the proxy device B receives written target data, the target data is cached in the cache space B, and a first notification message and a second notification message are asynchronously generated, after the target data is cached successfully, the first notification message is broadcast to the data management component in other regions, i.e., the data management component A, C, through the message queue, and the second notification message is sent to the data management component B, when the data management component A, C receives the first notification message, the target data in the respective cache space, i.e., the cache space A, C, is deleted, when the data management component B receives the second notification message, the target data is stored in the database a, and then the target data is synchronized from the target database to the database B. Optionally, the read traffic of the region a may be introduced into the database B, that is, when the agent device a receives the read request of the region a, the agent device a may access the database B and return the result. At this time, the database a is a read-write database, and the database B is a read database.
In an embodiment, because the database synchronization has a certain delay, after a write operation is performed through the terminal, before generated target data is successfully cached in a target cache space or successfully stored in the target database, in order to avoid reading dirty data, that is, data that is not updated, through the terminal, the technical scheme of the present application can perform a terminal false display operation, that is, after the write operation is performed through the terminal, the terminal can immediately display a result of the write operation, meanwhile, the proxy device receives the data generated by the write operation and processes the generated data, and after the proxy device processes the generated data, other areas can query the result of the write operation through a read request. For example, as shown in fig. 4a, after a comment is posted in a comment area by a user a through a terminal a, the terminal a may immediately display the comment, and data corresponding to the comment is not processed by a proxy device, at this time, the user B cannot view the comment in the comment area through a terminal B, as shown in fig. 4B, after the proxy device completes processing the data, the user B may view the comment in the comment area through the terminal B, as shown in fig. 4 c.
In the embodiment of the application, the proxy device receives target data to be stored, caches the target data to a target cache space corresponding to the target proxy device, broadcasts a first notification message to other proxy devices after the target data is successfully cached to the target cache space, so that other proxy devices can delete the target data of other corresponding cache spaces, thereby ensuring the multi-place uniqueness of the data stored in the database during caching, the target proxy device stores the target data to a target database of a plurality of databases, and performs data synchronization processing between the target database and other databases after the target data is successfully stored to the target database, so as to ensure the data consistency among the plurality of databases. Therefore, the data processing method can better process the data related to the long-distance multi-live scenes.
Please refer to fig. 5, which is another data processing method based on a remote multi-active system according to an embodiment of the present application, and the method may be executed by the above-mentioned target proxy device. As shown in fig. 5, the data processing method based on the remote multi-active system may include the following steps S501 to S506:
s501, receiving target data to be stored.
S502, acquiring a storage identifier of the target data.
In a specific implementation process, the target agent device may receive a data storage request about target data, where the data storage request may carry a storage identifier of the target data; the target agent device may then parse the storage identification from the data storage request. The storage identification is used for indicating whether the target data needs to be stored in the database or not; if the storage identifier indicates that the target data needs to be stored in the database, steps S503-S505 are performed, and if the storage identifier indicates that the target data does not need to be stored in the database, step S506 is performed. The storage identifier may be determined according to any one of the following manners:
in one embodiment, the storage identity may be determined based on characteristics of the target data. Since the database of the embodiment of the application can use the SQL database, and the cache space can use the NOSQL database, whether the target data is stored in the database can be determined according to whether the target data can satisfy the relational characteristics of SQL. For example, data of one post is stored, the data includes a main key ID, an object aid of the post, post content, a user ID, and the like, and the post content is a large text field and is not suitable for being queried in the database by "where content", so that the data of the main key ID, the object aid of the post, the user ID, and the like can be stored in the database, and the post content can be cached in the cache space.
In one embodiment, the storage identity may be determined based on the number of requests for the target data. For example, when there are a large number of write requests for the target data, i.e., when there are frequent update operations for the target data, the target data may not need to be stored in the database. Or, when there are a large number of read requests for the target data, that is, when there are frequent query operations for the target data, it is not necessary to store the target data in the database. Alternatively, the stored identity of the target data may be specified by a technician according to the actual business scenario.
S503, if the storage identifier indicates that the target data needs to be stored in the database, caching the target data in a target cache space corresponding to the target proxy equipment.
In the embodiment of the present invention, reference may be made to the related description of step S201 for specific implementation of step S503, which is not described herein again. In an embodiment, after the target data is successfully cached in the target cache space corresponding to the target proxy device, step S504 may be directly performed.
In one embodiment, the space of the target cache space is limited, and the target data may be some unimportant data irrelevant to the user, so that the target data is cached for a long time, and the space resource is occupied and wasted. Therefore, when the target data is not related to the user, an expiration time policy can be applied to the target data to enable subsequent automatic cleaning of the target data when the target data is cached. Based on this, after determining that the target data needs to be stored in the database, the target agent device may further obtain a target data category of the target data, where the target data category may include a user data category or a non-user data category. The proxy equipment can determine the target data type of the target data by analyzing whether the target data carries the user ID, namely, if the target data carries the user ID, the target data type of the target data is the user data type; and if the target data carries the user ID, indicating that the target data type of the target data is a non-user data type.
If the target data category is the user data category, the target proxy device may perform steps S504-S405. It can be understood that, at this time, the target data of the user data category has uniqueness in multi-place cache, and as more users are in a read-write-more-less state and more are in a fixed area, and rarely perform write operation frequently in different areas, it is ensured that the target data is cached in the cache space of the area where the user is located, and the target data in the off-site cache space is deleted, so that the memory and the power consumption of the cache space can be reduced. If the target data type is a non-user data type, setting survival time (namely expiration time) for the target data, counting the cache time of the target data in the target cache space after the target data is successfully cached in the target cache space, and deleting the target data in the target cache space when the cache time is determined to be equal to or greater than the survival time (namely the target data is expired). It should be noted that, the setting of the survival time of the target data may be performed when the target data is cached in the target cache space, or may be performed after the target data is cached in the target cache space, which is not limited in this respect. When the target data type of the target data is the non-user data type, the target agent device still stores the target data in the target database, i.e., the target agent device still performs step S505.
Optionally, deleting the target data in the target cache space may be that the proxy device periodically traverses all data in the cache space, generates a third notification message when determining that the cache duration of the target data is equal to or greater than the survival duration, and sends the third notification message to the corresponding data management component through the message queue, so that the data management component deletes the target data in the target cache space. Or, when the proxy device receives a read request or a write request for target data, it may check whether the cache duration of the target data is equal to or greater than the survival duration, and if the target data is expired, notify the corresponding data management component to delete the target data.
S504, after the target data are successfully cached in the target cache space, a first notification message is broadcasted to other proxy equipment.
And S505, storing the target data into a target database in the plurality of databases, and after the target data is successfully stored into the target database, executing data synchronization processing between the target database and other databases.
S506, if the storage identification indicates that the target data does not need to be stored in the database, obtaining a cache consistency strategy adapted to the target data, and caching the target data into a target cache space and other cache spaces by adopting the cache consistency strategy.
The cache consistency policy adapted to the target data may include: a strong consistency policy in place, or a final consistency policy. The multi-ground strong consistency strategy is as follows: when the target data is successfully cached in the target cache space and other cache spaces, the target data is allowed to access, that is, any node at any time, and the results returned by the read requests for the target data in all the cache spaces are consistent. The final consistency policy is: as long as the target data is successfully cached in the target cache space, the target data can be allowed to be accessed without waiting for the policy of other proxy devices for successfully caching the target data, that is, any node at any time, the results returned by the read requests for the target data in all the cache spaces may be consistent or inconsistent, but the final results are consistent. In the specific implementation process, a multi-place strong consistency strategy or a final consistency strategy can be selected as a cache consistency strategy adapted to target data according to an actual service scene, so that multi-place multi-activity is realized.
Illustratively, when the target data is production data from inside to outside, for example, system data produced by an internal management system, such as articles, news information, disease entries, etc., a strong consistency policy may be selected as the cache consistency policy adapted to the target data. When the target data is data with tolerance, namely short-term data inconsistency can be tolerated, and when waiting for a period of time, latest updated data can be finally inquired, for example, a user updates a head portrait, the user can immediately view the updated head portrait, other users in different places may view the head portrait which is not updated at the moment and can view the updated head portrait after waiting for a period of time, or when the target data is data with low requirement on service real-time, for example, a commodity popularity list is viewed, the data is generally updated regularly without being updated in real time, and a final consistency policy can be selected as a cache consistency policy adapted to the target data.
In an embodiment, when the cache consistency policy is a strong consistency policy, a specific implementation manner of caching the target data into the target cache space and the other cache spaces by using the cache consistency policy may be: caching the target data into a target cache space of the target proxy device, setting the state of the target data to be a use-prohibited state in the target cache space, sending a first cache request about the target data to other proxy devices to request the other proxy devices to cache the target data into other cache spaces corresponding to the other proxy devices, and updating the state of the target data from the use-prohibited state to a usable state in the target cache space when receiving a cache completion notification of the other proxy devices. The first cache request may carry target data. The first cache request may be issued after the target data is successfully cached in the target cache space of the target proxy device, or may be issued while the target data is cached in the target cache space of the target proxy device, where a time for sending the first cache request is not limited.
It can be understood that the target data is in a prohibited use state in the target cache space, that is, in order to prohibit access, when a user issues a read request for the target data, a corresponding result cannot be returned, or the returned result is still an un-updated result. For example, the target data includes key1 and corresponding value1, and key1 does not exist in the target cache space before, when the target data is in the use prohibition state, if the user sends a read request for key1, a null value is returned; or, the target data includes key1 and corresponding value2, and key1 and corresponding value1 exist in the target cache space before, when the target data is in the use disabled state, if the user issues a read request for key1, value1 is returned, and only when the target data is in the use enabled state, the user issues a read request for key1, and value2 is returned.
For example, as shown in fig. 6, when the proxy device B receives the written target data, the target data is cached to the cache space B, the state of the target data in the cache space B is the use disabled state, and a first cache request for the target data is sent to the proxy device A, C, the proxy device A, C writes the target data in the corresponding cache space A, C upon receiving the first cache request, and sends a notification to the proxy device B to notify the proxy device a that the caching of the target data is completed, and the proxy device a updates the state of the target data from the use disabled state to the use enabled state in the cache space B upon receiving the cache completion notification of the proxy device A, C. Therefore, the strong consistency of the data cache is realized, namely when the target data is cached in the target cache space, the confirmation of other cache spaces is waited, the other cache spaces are ensured to finish the caching of the target data, and the target data is only available when the confirmation of other cache spaces is received. .
In an embodiment, when the cache consistency policy is the final consistency policy, the specific implementation of caching the target data into the target cache space and the other cache spaces by using the cache consistency policy may be: caching the target data to a target cache space of the target proxy equipment, asynchronously sending a second cache request related to the target data to other proxy equipment to request other proxy equipment to cache the target data to other cache spaces corresponding to other proxy equipment, if the target data is successfully written in the target cache space, the state equipment of the target data in the target cache space is in a usable state, if the target data is cached in the target cache space in a failure mode, caching the target data to a redo (redo) queue, after a re-caching event related to the target data is detected, reading the target data from the redo queue, and caching the target data to the target cache space of the target proxy equipment again. The second cache request may carry target data. The second cache request may be issued after the target data is successfully cached in the target cache space of the target proxy device, or may be sent while the target data is cached in the target cache space of the target proxy device.
In an embodiment, the specific manner in which the other proxy device caches the target data in the other cache space corresponding to the other proxy device may be: and the target agent equipment sends the second cache request to other data management components corresponding to other agent equipment through the message queue, and when the other data management components receive the second cache request, the target data is cached to other cache spaces corresponding to other agent equipment. The remote synchronization of the cache is realized through the message queue, and the target data can be ensured not to be lost even if a fault occurs when the target data is cached to other cache spaces.
It can be understood that, when a local user sends a read request for target data, the returned result is a result included in the target data, and if the cache space in another place does not complete cache synchronization of the target data at this time, the user in another place cannot return a corresponding result when sending the read request for the target data, or the returned result is still an un-updated result. For example, the cache space a caches target data, and the target data includes key1 and a corresponding value1, if a user in the region a issues a read request for key1, the cache space a is accessed and a value1 is returned, if the cache space B in the region B does not complete the caching of the target data at this time and there is no key1 in the cache space B before, a null value is returned when the user B in the region B issues a read request for key1, or if the cache space B has a key1 and a corresponding value2 before, a value2 is returned when the user B in the region B issues a read request for key1, and only when the cache space B in the region B completes the caching of the target data, the user B in the region B issues a read request for key1, a value2 is returned.
In one embodiment, when the target data is not cached to the target data, the proxy device may first cache the target data to the redo queue, so as to prevent data loss. The reasons for the cache failure may be that a cache space fails, data is already cached when the target data is cached in the target cache space, a technician indicates that the target data is not cached at this time, and the like. And when the target agent equipment detects the re-caching event, acquiring the target data from the redo queue and caching the target data again. The re-caching event may be detection of failure release, completion of caching of data to be cached before the target data is cached in the target cache space, instruction of a technician to continue caching of the target data at the time, and the like.
For example, as shown in fig. 7, when the proxy device B receives the written target data, the target data is buffered to the buffer space B, and a second buffer request is sent to the data management component A, C corresponding to the proxy device A, C through the message queue, and when the data management component A, C receives the second buffer request, the target data is buffered to the corresponding buffer space A, C. If the target data is cached to the cache space B and fails to be cached, caching the target data to a redo queue, after the cache event is detected, reading the target data from the redo queue by the proxy equipment B, and caching the target data to the cache space B again. Therefore, the final consistency of the data cache is realized, namely when the target data is cached, the target data is cached in a local cache space firstly, the target data is available locally at the moment and then is synchronized to other cache spaces, and when the target data is cached in other cache spaces, the target data is available in other areas.
It should be noted that, the data types of any data mentioned in the embodiments of the present application may be classified into a conflict-free copy data type and a non-conflict-free copy data type; the Conflict-Free copied Data Type may be a CRDT (conflicted-Free copied Data Type), and the Data of the CRDT Type may be used for copying and merging Data across networks, and may automatically resolve Data conflicts and finally achieve Data consistency. Accordingly, the target data type of the target data mentioned in the embodiments of the present application may be a conflict-free copy data type or a non-conflict-free copy data type. In addition, after the target data is cached in the target cache space and other cache spaces by adopting the cache consistency strategy, when modification operation is performed on the target data in many ways, a data conflict situation is generated in many ways. Therefore, different schemes may be used to address this situation depending on the target data type of the target data.
In a possible implementation, the target data type of the target data is a conflict-free copy data type, and a specific implementation manner for resolving a multiple data conflict situation may be: the target proxy equipment responds to modification operation aiming at target data, modifies the target data in a target cache space according to the modification operation to obtain modified first data, if modification notification about the target data sent by other proxy equipment is received, second data obtained by modifying the target data in other cache spaces by other proxy equipment is obtained, the target proxy equipment combines the first data and the second data to obtain combined data, and the combined data is adopted in the target cache space to update the first data.
For example, comment like approve, that is, many comment like approve at the same time, like approve data, then data conflict situation occurs. For such data, the CRDT type can be used for caching, the data generated in many places meets the switching law, the combination law and the power law, that is, the data generated in many places is effective regardless of the sequence, and the data generated in many places are combined to finally realize data consistency. For example, for data of the count class, the data type is the CRDT type, the data processing mode adopted is the final quantity consistency, as shown in fig. 8, the original value of key3 is 0, that is, target data key3 is modified at time point a, the first data obtained by adding one to the value corresponding to key3, target data key3 is modified at time point C, the second data obtained by adding one to the value corresponding to key3, and adding one to the value corresponding to key3 at time point t2, so that the first data and the second data are combined to obtain combined data of value 3 corresponding to key3, so the value corresponding to key3 after the data is combined by A, B, C is 3, and the original value of key3 is updated by using the combined data, before the update, the value corresponding to key 4645 read a is read by a, the value corresponding to key B is read by B8236, the value corresponding to key 892 is read by C, but finally the data of three places are consistent, namely the value values corresponding to the key3 of three places are all 3.
For example, as shown in fig. 9a, at this time, the praise number of the comment is 4, the user a praise the comment through the terminal a, and at the same time, the user B praise the comment through the terminal B, at this time, the praise data of the two places are not merged and synchronized, because the technical solution of the present application executes the terminal ghost operation, the praise number displayed on the terminal a plus one is 5, and the praise number displayed on the terminal B plus one is 5, as shown in fig. 9B, after the praise data of the two places are merged and synchronized, the praise numbers displayed on the terminal a and the terminal B are 6, and as shown in fig. 9c, the final coincidence of the conflict data is achieved.
In one embodiment, the target data type of the target data is not a conflict-free copy data type, and a specific embodiment for resolving a multiple data conflict situation may be: when receiving second data sent by other proxy equipment, target proxy equipment acquires first storage information about the target data and second storage information about the target data from other proxy equipment, wherein the first storage information is a first version number generated when the target proxy equipment modifies the target data, the second storage information is a second version number generated when other proxy equipment modifies the target data, the target proxy equipment compares the first version number with the second version number, if the first version number is larger than the second version number, the first data is kept unchanged in a target cache space, and if the first version number is smaller than or equal to the second version number, the first data is updated to the second data in the target cache space. When the target proxy device modifies the target data in the target cache space in response to the modification operation, the first version number corresponding to the target data is incremented by one, for example, the version number corresponding to key1 included in the target data in the target cache space is 1, and when the target proxy device modifies the target data to obtain the first data, the corresponding version number is changed to 2. When the first data in the target cache space is updated to the second data, the first version number is also updated to the second version number, for example, the version number corresponding to the key1 contained in the first data in the target cache space is 1, and the version number corresponding to the key1 contained in the second data is 2, and after the first data is updated, the version number corresponding to the key1 contained in the target data in the target cache space is changed to 2.
Optionally, in a possible implementation, the first storage information may further include a first timestamp generated when the target proxy device modifies the target data, and the second storage information may further include a second timestamp generated when other proxy devices modify the target data, and if the first version number is smaller than the second version number, the first data in the target cache space is updated to the second data, and the target data in the target cache space is updated to the target data cached by other proxy devices; if the first version number is equal to the second version number, the first timestamp and the second timestamp are compared. Specifically, if the first timestamp is greater than or equal to the second timestamp, the first data is kept unchanged in the target cache space, and the target data is kept unchanged in the target cache space; and if the first timestamp is smaller than the second timestamp, updating the first data into second data in the target cache space.
Optionally, when multiple write data conflicts occur, the data conflict can be resolved by using a version number or a version number + a timestamp, or by using only a timestamp. Namely, lww (last Write wins) strategy, which is a distributed data consistency synchronization strategy, when distributed Write data conflict, comparison is performed according to a specified mode to determine the most recently written data, and the most recently written data is used as the latest data to update the existing data. For example, a user at a location a posts a post in the community, a management end of the community post is at a location B, the user at the location a can modify the post posted by the user, and a manager at the location B can modify the post of the user through the management end, so that the same target data can be modified at the same time, at this time, data conflict can be solved according to the LWW policy, that is, the latest modified data is selected, and the modified data generated before is invalid.
For example, for data that can be generated by behavior defined by key value pairs, the data type is a non-CRDT type, the data processing mode adopted is the LWW policy, as shown in fig. 10, that is, a user in a place modifies the value corresponding to key2 to be value1, the generated first storage information is version number v1, the timestamp is ts1, a user in C place modifies the value corresponding to key2 to be value2, and the second storage information is generated to be version number v1, the timestamp is ts2, the target data written according to the timestamp and version number C is the latest data, and the target data written according to C place is synchronized with A, B, C three times, so that the data in three places are consistent, that is, the value values corresponding to key2 in three places are all value 2.
Through a large number of tests on the technical scheme of the embodiment of the present application, it can be known that the technical scheme can synchronize disaster recovery data from a remote cache space to a local new cache space when a local cache space fails, and ensure that the disaster recovery data is correctly available, thereby effectively improving user experience, and the technical scheme of the embodiment of the present application can make an online service closer to a user, when the user uses the online service, the delay of data transmission is reduced to a certain extent, fig. 11 is a schematic diagram of failure solution based on a remote multi-active system, as shown in fig. 11, when the cache space of a location C fails and cannot be recovered, a proxy device in the technical scheme of the present application can synchronize a copy of data from the cache space of a location B to the new cache space of the location C through a DTS (database transfer service, data is transferred from one server to another server), and the proxy equipment switches the service of the place C to a new cache space, and the user has no perception in the whole using process.
In the embodiment of the application, the proxy equipment receives target data to be stored, acquires a storage identifier of the target data, caches the target data to a target cache space corresponding to the target proxy equipment if the storage identifier indicates that the target data needs to be stored in a database, broadcasts a first notification message to other proxy equipment after the target data is successfully cached to the target cache space, stores the target data to a target database in a plurality of databases, and performs data synchronization processing between the target database and other databases after the target data is successfully stored to the target database so as to ensure data consistency in the plurality of databases; if the storage identifier indicates that the target data does not need to be stored in the database, a cache consistency strategy adapted to the target data is obtained, and the target data is cached in the target cache space and other cache spaces by adopting the cache consistency strategy, so that the data consistency in the multiple cache spaces is ensured. Therefore, by implementing the method, the data can be processed differently according to different service scenes of the application, so that the data processing of the data related to the different-place multi-living scenes can be better realized, and the data such as the interactive experience stored in the local place can be provided for the user.
Referring to fig. 12, a schematic structural diagram of a data processing apparatus based on a remote multi-active system according to the present application is shown, where the related remote multi-active system includes a plurality of proxy devices, a cache space corresponding to each proxy device, and a plurality of databases; wherein, the areas of any two agent devices are different; the apparatus is operated in a target agent device of a plurality of agent devices, and is configured to perform the method according to the embodiment shown in fig. 2 and 5, for convenience of description, only the portion related to the embodiment of the present application is shown, and details of the technology are not disclosed, and reference may be made to the embodiment shown in fig. 2 and 5 of the present application. The data processing apparatus 1200 based on the remote multi-live system may include: a receiving module 1201, a notification module 1202, and a processing module 1203.
A receiving module 1201, configured to receive target data to be stored, and cache the target data in a target cache space corresponding to the target proxy device;
a notification module 1202, configured to broadcast a first notification message to other proxy devices after the target data is successfully cached in the target cache space, where the first notification message is used to notify: if the target data exists in other cache spaces corresponding to the other proxy devices, deleting the target data in the other cache spaces;
a processing module 1203, configured to store the target data in a target database of the multiple databases, and after the target data is successfully stored in the target database, perform data synchronization processing between the target database and another database.
In a possible implementation, after the receiving module 1201 is configured to receive target data to be stored, the receiving module is further configured to:
acquiring a storage identifier of the target data, wherein the storage identifier is used for indicating whether the target data needs to be stored in a database or not;
if the storage identification indicates that the target data needs to be stored in a database, caching the target data in a cache space corresponding to the target proxy equipment;
if the storage identification indicates that the target data does not need to be stored in a database, a cache consistency strategy adapted to the target data is obtained, and the target data is cached in the target cache space and the other cache spaces by adopting the cache consistency strategy.
In a possible implementation manner, the receiving module 1201, when the storage identifier indicates that the target data needs to be stored in the database, is further configured to:
obtaining a target data category of the target data, wherein the target data category comprises: a user data category or a non-user data category;
if the target data type is the user data type, after the target data is successfully cached in the target cache space, a step of broadcasting a first notification message to other agent equipment is executed;
if the target data type is the non-user data type, setting survival time length for the target data, counting the cache time length of the target data in the target cache space after the target data is successfully cached in the target cache space, and deleting the target data in the target cache space when the cache time length is determined to be equal to or larger than the survival time length.
In a possible implementation manner, when the receiving module 1201 is configured to cache the target data into the target cache space and the other cache spaces by using the cache consistency policy, specifically, the receiving module is configured to:
caching the target data into a target cache space of the target proxy equipment, and setting the state of the target data in the target cache space to be a use forbidding state;
sending a first caching request about the target data to the other proxy equipment to request the other proxy equipment to cache the target data into other caching spaces corresponding to the other proxy equipment;
when the cache completion notification of the other proxy equipment is received, updating the state of the target data from the use prohibition state to a usable state in the target cache space.
In a possible implementation manner, when the receiving module 1201 is configured to cache the target data into the target cache space and the other cache spaces by using the cache consistency policy, specifically, the receiving module is configured to:
caching the target data to a target cache space of the target proxy equipment, and asynchronously sending a second cache request about the target data to other proxy equipment to request the other proxy equipment to cache the target data to other cache spaces corresponding to the other proxy equipment;
if the target data is successfully written into the target cache space, setting the state of the target data to be a usable state in the target cache space;
if the target data is not cached in the target cache space, caching the target data into a redo queue, reading the target data from the redo queue after a redo event related to the target data is detected, and caching the target data into the target cache space of the target proxy equipment again.
In one possible implementation, after the receiving module 1201 is configured to cache the target data into the target cache space and the other cache spaces using the cache consistency policy, the receiving module is further configured to:
responding to modification operation aiming at target data, and modifying the target data in the target cache space according to the modification operation to obtain modified first data;
if receiving a modification notification about the target data sent by the other proxy equipment, acquiring second data obtained by modifying the target data in the other cache space by the other proxy equipment;
if the target data type of the target data is a conflict-free replication data type, merging the first data and the second data to obtain merged data; and updating the first data in the target cache space by adopting the merged data.
In a possible implementation, the receiving module 1201 is further configured to:
if the target data type is not the conflict-free copy data type, acquiring first storage information about the target data, wherein the first storage information at least comprises: a first version number generated when the target agent device modifies the target data;
obtaining second storage information about the target data from the other proxy device, the second storage information including at least: a second version number generated when the other proxy device modifies the target data;
if the first version number is larger than the second version number, keeping the first data unchanged in the target cache space; and if the first version number is less than or equal to the second version number, updating the first data into the second data in the target cache space.
In one possible implementation, the first storage information further includes: the target agent device generates a first timestamp when modifying the target data, and the second storage information further includes: a second timestamp generated when the other proxy device modifies the target data; the processing module 1203 is specifically configured to:
if the first version number is smaller than the second version number, updating the first data into the second data in the target cache space;
comparing the first timestamp with the second timestamp if the first version number is equal to the second version number;
when the first timestamp is greater than or equal to the second timestamp, keeping the first data unchanged in the target cache space; when the first timestamp is less than the second timestamp, updating the first data to the second data in the target cache space.
In the embodiment of the application, a receiving module receives target data to be stored, the target data are cached in a target cache space corresponding to target proxy equipment, a notification module broadcasts a first notification message to other proxy equipment after the target data are successfully cached in the target cache space, so that other proxy equipment can delete the target data in other corresponding cache spaces, a processing module stores the target data in a target database in a plurality of databases, and after the target data are successfully stored in the target database, data synchronization processing is performed between the target database and other databases, so that data consistency among the databases is ensured. Therefore, by implementing the process, the data involved in the long-distance multi-live scene can be better processed.
The functional modules in the embodiments of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module may be implemented in a form of hardware, or may be implemented in a form of software functional module, which is not limited in this application.
Please refer to fig. 13, which is a schematic structural diagram of a proxy device according to an embodiment of the present application. As shown in fig. 13, the proxy apparatus 1310 includes: at least one processor 1301, memory 1302. Optionally, the electronic device may further include a network interface 1303. Data can be interacted among the processor 1301, the memory 1302 and the network interface 1303, the network interface 1303 is controlled by the processor to receive and transmit messages, the memory 1302 is used for storing computer programs, the computer programs comprise program instructions, and the processor 1301 is used for executing the program instructions stored in the memory 1302. Wherein the processor 1301 is configured to call the program instructions to perform the method described above.
The memory 1302 may include volatile memory (volatile memory), such as random-access memory (RAM); the memory 1302 may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a solid-state drive (SSD), etc.; the memory 1302 may also include a combination of the above types of memory.
The processor 1301 may be a Central Processing Unit (CPU) 1301. In one embodiment, the processor 1301 may also be a Graphics Processing Unit (GPU) 1301. The processor 1301 may also be a combination of a CPU and a GPU.
In one embodiment, the memory 1302 is used to store program instructions. The processor 1301 can call the program instructions to perform the following steps:
receiving target data to be stored, and caching the target data to a target cache space corresponding to the target proxy equipment;
after the target data is successfully cached in the target cache space, broadcasting a first notification message to other proxy devices, where the first notification message is used to notify: if the target data exists in other cache spaces corresponding to the other proxy devices, deleting the target data in the other cache spaces;
and storing the target data into a target database in the plurality of databases, and after the target data is successfully stored into the target database, performing data synchronization processing between the target database and other databases.
In a possible implementation, the processor 1301, after being configured to receive the target data to be stored, is further configured to perform:
acquiring a storage identifier of the target data, wherein the storage identifier is used for indicating whether the target data needs to be stored in a database or not;
if the storage identification indicates that the target data needs to be stored in a database, caching the target data in a cache space corresponding to the target proxy equipment;
if the storage identification indicates that the target data does not need to be stored in a database, a cache consistency strategy adapted to the target data is obtained, and the target data is cached in the target cache space and the other cache spaces by adopting the cache consistency strategy.
In a possible implementation manner, the processor 1301 is further configured to, when the storage identifier indicates that the target data needs to be stored in the database, perform:
obtaining a target data category of the target data, wherein the target data category comprises: a user data category or a non-user data category;
if the target data type is the user data type, after the target data is successfully cached in the target cache space, a step of broadcasting a first notification message to other agent equipment is executed;
if the target data type is the non-user data type, setting survival time length for the target data, counting the cache time length of the target data in the target cache space after the target data is successfully cached in the target cache space, and deleting the target data in the target cache space when the cache time length is determined to be equal to or larger than the survival time length.
In one possible implementation, the processor 1301, when configured to cache the target data into the target cache space and the other cache space using the cache coherency policy, is further configured to:
caching the target data into a target cache space of the target proxy equipment, and setting the state of the target data in the target cache space to be a use forbidding state;
sending a first caching request about the target data to the other proxy equipment to request the other proxy equipment to cache the target data into other caching spaces corresponding to the other proxy equipment;
when the cache completion notification of the other proxy equipment is received, updating the state of the target data from the use prohibition state to a usable state in the target cache space.
In one possible implementation, the processor 1301, when configured to cache the target data into the target cache space and the other cache space using the cache coherency policy, is further configured to:
caching the target data to a target cache space of the target proxy equipment, and asynchronously sending a second cache request about the target data to other proxy equipment to request the other proxy equipment to cache the target data to other cache spaces corresponding to the other proxy equipment;
if the target data is successfully written into the target cache space, setting the state of the target data to be a usable state in the target cache space;
if the target data is not cached in the target cache space, caching the target data into a redo queue, reading the target data from the redo queue after a redo event related to the target data is detected, and caching the target data into the target cache space of the target proxy equipment again.
In one possible implementation, the processor 1301, after being configured to cache the target data into the target cache space and the other cache spaces by using the cache coherency policy, is further configured to:
responding to modification operation aiming at target data, and modifying the target data in the target cache space according to the modification operation to obtain modified first data;
if receiving a modification notification about the target data sent by the other proxy equipment, acquiring second data obtained by modifying the target data in the other cache space by the other proxy equipment;
if the target data type of the target data is a conflict-free replication data type, merging the first data and the second data to obtain merged data; and updating the first data in the target cache space by adopting the merged data.
In one possible implementation, the processor 1301 is further configured to perform:
if the target data type is not the conflict-free copy data type, acquiring first storage information about the target data, wherein the first storage information at least comprises: a first version number generated when the target agent device modifies the target data;
obtaining second storage information about the target data from the other proxy device, the second storage information including at least: a second version number generated when the other proxy device modifies the target data;
if the first version number is larger than the second version number, keeping the first data unchanged in the target cache space; and if the first version number is less than or equal to the second version number, updating the first data into the second data in the target cache space.
In one possible implementation, the first storage information further includes: the target agent device generates a first timestamp when modifying the target data, and the second storage information further includes: a second timestamp generated when the other proxy device modifies the target data; the processor 1301 is also operable to perform:
if the first version number is smaller than the second version number, updating the first data into the second data in the target cache space;
comparing the first timestamp with the second timestamp if the first version number is equal to the second version number;
when the first timestamp is greater than or equal to the second timestamp, keeping the first data unchanged in the target cache space; when the first timestamp is less than the second timestamp, updating the first data to the second data in the target cache space.
In specific implementation, the apparatus, the processor 1301, the memory 1302 and the like described in this embodiment of the present application may perform the implementation described in the above method embodiment, and may also perform the implementation described in this embodiment of the present application, which is not described herein again.
Also provided in embodiments of the present application is a computer (readable) storage medium storing a computer program comprising program instructions that, when executed by a processor, perform some or all of the steps performed in the above-described method embodiments. Alternatively, the computer storage media may be volatile or nonvolatile.
It should be noted that according to an aspect of the present application, a computer program product or a computer program is also provided, and the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer readable storage medium. The computer instructions are read by a processor of a computer device from a computer-readable storage medium, and the computer instructions are executed by the processor to cause the computer device to perform the method provided in the various alternatives in the aspect of the embodiment of the placeshifting multi-active system-based data processing method shown in fig. 2 or the placeshifting multi-active system-based data processing method shown in fig. 5.
Reference herein to "a plurality" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
It will be understood by those skilled in the art that all or part of the processes of the methods of the above embodiments may be implemented by a computer program, which may be stored in a computer storage medium, and the computer storage medium may be a computer readable storage medium, and when executed, the programs may include the processes of the above embodiments of the methods. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
While the present disclosure has been described with reference to particular embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure.

Claims (10)

1. A data processing method based on a remote multi-active system is characterized in that the remote multi-active system comprises a plurality of proxy devices, a cache space corresponding to each proxy device and a plurality of databases; wherein, the areas of any two agent devices are different; the method is performed by a target proxy device of the plurality of proxy devices, the method comprising:
receiving target data to be stored, and caching the target data to a target cache space corresponding to the target proxy equipment;
after the target data is successfully cached in the target cache space, broadcasting a first notification message to other proxy devices, where the first notification message is used to notify: if the target data exists in other cache spaces corresponding to the other proxy devices, deleting the target data in the other cache spaces;
and storing the target data into a target database in the plurality of databases, and after the target data is successfully stored into the target database, performing data synchronization processing between the target database and other databases.
2. The method of claim 1, wherein after receiving the target data to be stored, further comprising:
acquiring a storage identifier of the target data, wherein the storage identifier is used for indicating whether the target data needs to be stored in a database or not;
if the storage identification indicates that the target data needs to be stored in a database, caching the target data in a cache space corresponding to the target proxy equipment;
if the storage identification indicates that the target data does not need to be stored in a database, a cache consistency strategy adapted to the target data is obtained, and the target data is cached in the target cache space and the other cache spaces by adopting the cache consistency strategy.
3. The method of claim 2, wherein when the storage identity indicates that the target data is to be stored in a database, the method further comprises:
obtaining a target data category of the target data, wherein the target data category comprises: a user data category or a non-user data category;
if the target data type is the user data type, after the target data is successfully cached in the target cache space, a step of broadcasting a first notification message to other agent equipment is executed;
if the target data type is the non-user data type, setting survival time length for the target data, counting the cache time length of the target data in the target cache space after the target data is successfully cached in the target cache space, and deleting the target data in the target cache space when the cache time length is determined to be equal to or larger than the survival time length.
4. The method of claim 2, wherein said caching the target data into the target cache space and the other cache space using the cache coherency policy comprises:
caching the target data into a target cache space of the target proxy equipment, and setting the state of the target data in the target cache space to be a use forbidding state;
sending a first caching request about the target data to the other proxy equipment to request the other proxy equipment to cache the target data into other caching spaces corresponding to the other proxy equipment;
when the cache completion notification of the other proxy equipment is received, updating the state of the target data from the use prohibition state to a usable state in the target cache space.
5. The method of claim 2, wherein said caching the target data into the target cache space and the other cache space using the cache coherency policy comprises:
caching the target data to a target cache space of the target proxy equipment, and asynchronously sending a second cache request about the target data to other proxy equipment to request the other proxy equipment to cache the target data to other cache spaces corresponding to the other proxy equipment;
if the target data is successfully written into the target cache space, setting the state of the target data to be a usable state in the target cache space;
if the target data is not cached in the target cache space, caching the target data into a redo queue, reading the target data from the redo queue after a redo event related to the target data is detected, and caching the target data into the target cache space of the target proxy equipment again.
6. The method of claim 2, wherein after the caching the target data in the target cache space and the other cache spaces using the cache coherency policy, further comprising:
responding to modification operation aiming at target data, and modifying the target data in the target cache space according to the modification operation to obtain modified first data;
if receiving a modification notification about the target data sent by the other proxy equipment, acquiring second data obtained by modifying the target data in the other cache space by the other proxy equipment;
if the target data type of the target data is a conflict-free replication data type, merging the first data and the second data to obtain merged data; and updating the first data in the target cache space by adopting the merged data.
7. The method of claim 6, wherein the method further comprises:
if the target data type is not the conflict-free copy data type, acquiring first storage information about the target data, wherein the first storage information at least comprises: a first version number generated when the target agent device modifies the target data;
obtaining second storage information about the target data from the other proxy device, the second storage information including at least: a second version number generated when the other proxy device modifies the target data;
if the first version number is larger than the second version number, keeping the first data unchanged in the target cache space; and if the first version number is less than or equal to the second version number, updating the first data into the second data in the target cache space.
8. The method of claim 7, wherein the first stored information further comprises: the target agent device generates a first timestamp when modifying the target data, and the second storage information further includes: a second timestamp generated when the other proxy device modifies the target data;
if the first version number is less than or equal to the second version number, updating the first data to the second data in the target cache space, including:
if the first version number is smaller than the second version number, updating the first data into the second data in the target cache space;
comparing the first timestamp with the second timestamp if the first version number is equal to the second version number;
when the first timestamp is greater than or equal to the second timestamp, keeping the first data unchanged in the target cache space; when the first timestamp is less than the second timestamp, updating the first data to the second data in the target cache space.
9. A data processing device based on a remote multi-live system is characterized in that the remote multi-live system comprises a plurality of proxy devices, a cache space corresponding to each proxy device and a plurality of databases; wherein, the areas of any two agent devices are different; the apparatus operating in a target agent device of the plurality of agent devices, the apparatus comprising:
the receiving module is used for receiving target data to be stored and caching the target data to a target cache space corresponding to the target proxy equipment;
a notification module, configured to broadcast a first notification message to other proxy devices after the target data is successfully cached in the target cache space, where the first notification message is used to notify: if the target data exists in other cache spaces corresponding to the other proxy devices, deleting the target data in the other cache spaces;
and the processing module is used for storing the target data into a target database in the plurality of databases, and after the target data is successfully stored into the target database, executing data synchronization processing between the target database and other databases.
10. A proxy device, characterized in that it comprises a processor and a memory, said processor and memory being interconnected, wherein said memory is adapted to store a computer program comprising program instructions, said processor being configured to invoke said program instructions to perform the method according to any one of claims 1-8.
CN202110134158.8A 2021-01-29 2021-01-29 Data processing method based on remote multi-active system, related equipment and storage medium Pending CN113010549A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110134158.8A CN113010549A (en) 2021-01-29 2021-01-29 Data processing method based on remote multi-active system, related equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110134158.8A CN113010549A (en) 2021-01-29 2021-01-29 Data processing method based on remote multi-active system, related equipment and storage medium

Publications (1)

Publication Number Publication Date
CN113010549A true CN113010549A (en) 2021-06-22

Family

ID=76385478

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110134158.8A Pending CN113010549A (en) 2021-01-29 2021-01-29 Data processing method based on remote multi-active system, related equipment and storage medium

Country Status (1)

Country Link
CN (1) CN113010549A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112187738A (en) * 2020-09-11 2021-01-05 中国银联股份有限公司 Service data access control method, device and computer readable storage medium
CN114328618A (en) * 2021-11-30 2022-04-12 苏州浪潮智能科技有限公司 Cache data synchronization method, device, equipment and storage medium
CN116226093A (en) * 2023-04-25 2023-06-06 北京庚顿数据科技有限公司 Real-time database system based on dual-activity high-availability architecture

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102752381A (en) * 2012-06-28 2012-10-24 北京邮电大学 Multi-movable-duplicate mechanism applied to distributed storage and access method thereof
US20120290595A1 (en) * 2009-11-25 2012-11-15 Jack Kreindler Super-records
CN104601720A (en) * 2015-01-30 2015-05-06 乐视网信息技术(北京)股份有限公司 Cache access control method and device
CN106341454A (en) * 2016-08-23 2017-01-18 世纪龙信息网络有限责任公司 Across-room multiple-active distributed database management system and across-room multiple-active distributed database management method
CN109074226A (en) * 2016-09-28 2018-12-21 华为技术有限公司 Data de-duplication method, storage system and controller in a kind of storage system
CN110348826A (en) * 2018-04-08 2019-10-18 财付通支付科技有限公司 Strange land disaster recovery method, system, equipment and readable storage medium storing program for executing mostly living
CN110941666A (en) * 2019-11-01 2020-03-31 网联清算有限公司 Database multi-activity method and device

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120290595A1 (en) * 2009-11-25 2012-11-15 Jack Kreindler Super-records
CN102752381A (en) * 2012-06-28 2012-10-24 北京邮电大学 Multi-movable-duplicate mechanism applied to distributed storage and access method thereof
CN104601720A (en) * 2015-01-30 2015-05-06 乐视网信息技术(北京)股份有限公司 Cache access control method and device
CN106341454A (en) * 2016-08-23 2017-01-18 世纪龙信息网络有限责任公司 Across-room multiple-active distributed database management system and across-room multiple-active distributed database management method
CN109074226A (en) * 2016-09-28 2018-12-21 华为技术有限公司 Data de-duplication method, storage system and controller in a kind of storage system
CN110348826A (en) * 2018-04-08 2019-10-18 财付通支付科技有限公司 Strange land disaster recovery method, system, equipment and readable storage medium storing program for executing mostly living
CN110941666A (en) * 2019-11-01 2020-03-31 网联清算有限公司 Database multi-activity method and device

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112187738A (en) * 2020-09-11 2021-01-05 中国银联股份有限公司 Service data access control method, device and computer readable storage medium
CN114328618A (en) * 2021-11-30 2022-04-12 苏州浪潮智能科技有限公司 Cache data synchronization method, device, equipment and storage medium
CN114328618B (en) * 2021-11-30 2024-02-09 苏州浪潮智能科技有限公司 Cache data synchronization method, device, equipment and storage medium
CN116226093A (en) * 2023-04-25 2023-06-06 北京庚顿数据科技有限公司 Real-time database system based on dual-activity high-availability architecture
CN116226093B (en) * 2023-04-25 2023-08-08 北京庚顿数据科技有限公司 Real-time database system based on dual-activity high-availability architecture

Similar Documents

Publication Publication Date Title
US11520770B2 (en) System and method for providing high availability data
US10713275B2 (en) System and method for augmenting consensus election in a distributed database
US9965364B2 (en) Fault tolerant listener registration in the presence of node crashes in a data grid
US7693882B2 (en) Replicating data across the nodes in a cluster environment
CN113010549A (en) Data processing method based on remote multi-active system, related equipment and storage medium
US10366106B2 (en) Quorum-based replication of data records
US9753954B2 (en) Data node fencing in a distributed file system
CN110045912B (en) Data processing method and device
CN106815218B (en) Database access method and device and database system
CN109547512B (en) NoSQL-based distributed Session management method and device
CN112084258A (en) Data synchronization method and device
CN104935654A (en) Caching method, write point client and read client in server cluster system
CN112162846B (en) Transaction processing method, device and computer readable storage medium
EP4213038A1 (en) Data processing method and apparatus based on distributed storage, device, and medium
CN113094430B (en) Data processing method, device, equipment and storage medium
CN112384906A (en) MVCC-based database system asynchronous cache consistency
CN109726211B (en) Distributed time sequence database
US11288237B2 (en) Distributed file system with thin arbiter node
CN115495495A (en) Transaction processing method, distributed database system, cluster and medium
CN110196788B (en) Data reading method, device and system and storage medium
CN115238006A (en) Retrieval data synchronization method, device, equipment and computer storage medium
CN116821232A (en) Data synchronization method and related device
JP4286857B2 (en) Internode shared file control method
CN115698955A (en) Fault tolerance of transaction images
CN110442573A (en) A kind of method and device of distributed fault-tolerance key assignments storage

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 40046015

Country of ref document: HK

TA01 Transfer of patent application right

Effective date of registration: 20230922

Address after: 100190 Beijing Haidian District Zhichun Road 49 No. 3 West 309

Applicant after: TENCENT CLOUD COMPUTING (BEIJING) Co.,Ltd.

Address before: 518057 Tencent Building, No. 1 High-tech Zone, Nanshan District, Shenzhen City, Guangdong Province, 35 floors

Applicant before: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd.

TA01 Transfer of patent application right