CN114564318A

CN114564318A - Data processing method, device and system, electronic equipment and storage medium

Info

Publication number: CN114564318A
Application number: CN202210129518.XA
Authority: CN
Inventors: 任鸿翔; 朱超
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-02-11
Filing date: 2022-02-11
Publication date: 2022-05-31

Abstract

The disclosure provides a data processing method, a data processing device, a data processing system, an electronic device and a storage medium, and relates to the field of data processing, in particular to the technical field of big data. The specific implementation scheme is as follows: receiving resource content data sent by an RCMC receiving module, wherein the NID of the resource content data has a mapping relation with an RCMC processing module; constructing a mutual exclusion lock for the NID of the received resource content data, and sequentially writing the resource content data carrying the same NID into a database based on the mutual exclusion lock; adding the resource content data into a local queue every time the writing of one piece of resource content data is completed; and pushing the resource content data in the local queue to a first message middleware corresponding to the local queue based on the exclusive lock of the NID of each piece of resource content data in the local queue, so that the caching service acquires and caches the resource content data from the first message middleware. The caching service may be prevented from caching incorrect versions of the resource content data.

Description

Data processing method, device and system, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of data processing technology, and in particular, to the field of big data technology.

Background

In the distributed cache system, when the receiving module in the distributed cache system receives the updated resource content data, the updated resource content data can be written into a message middleware. The writing module may obtain the updated resource content data from the message middleware, write the obtained resource content data into the database, and write the obtained resource content data into another message middleware, so that the cache layer obtains the updated resource content data.

Disclosure of Invention

The disclosure provides a data processing method, device, system, electronic equipment and storage medium.

According to an aspect of the present disclosure, there is provided a data processing method, which is applied to a resource content management center RCMC processing module, and the method includes:

receiving resource content data sent by an RCMC receiving module, wherein the resource content data carries a message identifier NID, and the NID of the resource content data has a mapping relation with the RCMC processing module;

constructing a mutual exclusion lock for the NID of the received resource content data, and sequentially writing the resource content data carrying the same NID into a database based on the mutual exclusion lock;

adding the resource content data into a local queue every time the writing of one piece of resource content data is completed;

and pushing the resource content data in the local queue to a first message middleware corresponding to the local queue based on the exclusive lock of the NID of each piece of resource content data in the local queue, so that the caching service acquires and caches the resource content data from the first message middleware.

According to a second aspect of the present disclosure, there is provided a data processing apparatus, which is applied to a resource content management center RCMC processing module, and includes:

a receiving unit, configured to receive resource content data sent by an RCMC receiving module, where the resource content data carries a message identifier NID, and the NID of the resource content data has a mapping relationship with the RCMC processing module;

the write-in unit is used for constructing a mutual exclusion lock for the NID of the received resource content data and sequentially writing the resource content data carrying the same NID into a database based on the mutual exclusion lock;

an adding unit, configured to add, to a local queue, one piece of resource content data each time writing of the resource content data is completed;

and the pushing unit is used for pushing the resource content data in the local queue to a first message middleware corresponding to the local queue based on the exclusive lock of the NID of each piece of resource content data in the local queue, so that the caching service acquires and caches the resource content data from the first message middleware.

According to a third aspect of the present disclosure, there is provided a data processing system comprising: the system comprises a plurality of RCMC receiving modules, a plurality of RCMC processing modules, a database, first message middleware and a cache service;

each RCMC receiving module is used for sending the resource content data to the RCMC processing module which has a mapping relation with the NID of the resource content data;

each RCMC processing module configured to perform the method steps of the first aspect;

the caching service is used for acquiring and caching resource content data from the first message middleware.

According to a fourth aspect of the present disclosure, there is provided an electronic device comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the first aspect.

According to a fifth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of the first aspect.

According to a sixth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method of the first aspect described above.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

fig. 1 is a schematic diagram of a write module according to an embodiment of the disclosure;

fig. 2 is a flowchart of a data processing method according to an embodiment of the present disclosure;

fig. 3 is an exemplary diagram of a method for writing data to a database by an RCMC processing module according to an embodiment of the present disclosure;

fig. 4 is an exemplary diagram of a method for an RCMC receiving module to send data to an RCMC processing module according to an embodiment of the present disclosure;

fig. 5 is an exemplary diagram of a method for an RCMC processing module to push data to first message middleware according to an embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 7 is a block diagram of a data processing system provided by an embodiment of the present disclosure;

fig. 8 is a block diagram of an electronic device for implementing a data processing method of an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the related art, after the write module acquires the resource content data from one message middleware, the write module may write the acquired resource content data into the database, and then write the resource content data written into the database into another message middleware, so that the cache service acquires and caches the resource content data from the message middleware.

Because the resource content data does not necessarily carry the version number, when the update frequency of the resource content data of the same resource is high, or when a writing module has a fault such as machine jitter, the sequence of the resource content data written into another message middleware is inconsistent with the actual update sequence of the resource content data, so that the caching service caches the resource content data of an incorrect version.

For example, if the resource content data of the same resource is frequently updated, assuming that the resource is an article, the article is modified by an author once and then is immediately masked by an administrator, the writing module may obtain two pieces of resource content data, the first piece is the modification data of the article by the author, and the second piece is the data masked by the administrator.

The writing module writes a first piece of data into the database and then writes a second piece of data into the database. Because the data volume of the second piece of data is small, it may happen that the second piece of data is processed first and is pushed to the message middleware first, and the first piece of data is processed last and is then pushed to the message middleware, so that the caching service acquires the second piece of data first and then acquires the first piece of data. Further, after the service obtains the second piece of data from the cache service, the article will be displayed as an effect modified by the author according to the second piece of data, so that the article is not masked and a service error occurs.

In order to solve the above problem, an embodiment of the present disclosure provides a data processing method, which is applied to a Resource Content Management Center (RCMC) processing module, where the RCMC processing module may be any RCMC processing module in a data processing system. The data processing system comprises a plurality of RCMC receiving modules, a plurality of RCMC processing modules, a database, a first message middleware and a cache service.

As shown in fig. 1, the embodiment of the present disclosure splits a write module in the prior art into an RCMC receive module and an RCMC process module. Fig. 1 exemplarily shows 3 RCMC receiving modules and 3 RCMC processing modules, and in an actual implementation, the number of RCMC receiving modules and RCMC processing modules included in the data processing system is not limited thereto.

The RCMC receiving module is used for receiving the resource flow and sending the resource flow to the RCMC processing module. The resource traffic includes a plurality of pieces of resource content data. The resource content data may be data generated by articles, videos, comment messages and the like published by the user, data generated by various possible operations such as modification operation, forwarding operation or approval operation of the published content by the user, and data generated by operations such as shielding or deleting of the content published by the user by an administrator.

The RCMC processing module is used for writing the received resource content data into a database and then pushing the resource content data to the first message middleware, so that the caching service acquires the resource content data from the first message middleware and caches the resource content data in a caching layer.

In the embodiment of the present disclosure, the RCMC receiving module may be Pre-RCMC, and the RCMC processing module may be an RCMC instance.

The following describes the data processing method provided by the embodiments of the present disclosure in detail.

As shown in fig. 2, the data processing method provided by the embodiment of the present disclosure includes:

s201, receiving resource content data sent by an RCMC receiving module, wherein the resource content data carries a message Identity (News Identity, NID), and the NID of the resource content data has a mapping relation with an RCMC processing module.

Wherein all resource content data of the same resource carries the same NID.

In the embodiment of the present disclosure, the NID of the resource content data has a preset mapping relationship with the RCMC processing module, and further, the resource content data having the same NID may be mapped to the same RCMC processing module.

S202, constructing a mutual exclusion lock for the NID of the received resource content data, and sequentially writing the resource content data carrying the same NID into a database based on the mutual exclusion lock.

Optionally, the RCMC processing module has a cache, and in the embodiment of the present disclosure, a mutual exclusion lock may be constructed for the NID by the cache. The Licache is a cache used externally and has a function of constructing a mutual exclusion lock.

After constructing a mutual exclusion lock for the NID of the received resource content data, it can be ensured that only one thread can process the resource content data with the NID at the same time. That is, at the same time, the RCMC processing module can process only one of the resource content data pieces with the same NID, and after completing the writing of one resource content data piece, can continue to process the next resource content data piece with the same NID, so that the resource content data pieces with the same NID can be written into the database in the order of reception.

As an example, as shown in fig. 3, the RCMC processing module 1 receives three pieces of resource content data belonging to the same resource in sequence, which are respectively referred to as resource 1 ", resource 1', and resource 1 for short. Since the three pieces of resource content data belong to the same resource, they have the same NID.

The RCMC processing module 1 firstly receives the resource 1 ', then constructs a mutual exclusion lock for the NID of the resource 1', synchronizes the resource 1 'to the database read-write module 1, and then writes the resource' into the database through the database read-write module 1. In this process, the RCMC processing module 1 receives the resource 1 ', and since the NID of the resource 1 ' and the NID of the resource 1 ″ have the same mutual exclusive lock, the RCMC processing module 1 can write the resource 1 ' into the database after the resource 1 ″ is written into the database. Similarly, after the RCMC processing module 1 writes the resource 1' into the database, the resource 1 can be written into the database, thereby avoiding the disorder of the sequence.

Similarly, the RCMC processing module 2 may sequentially write the resource 2 ", the resource 2 ', and the resource 2 into the database through the database read-write module 2 based on the exclusive locks of the NIDs of the resource 2", the resource 2', and the resource 2. The RCMC processing module 3 may sequentially write the resource 3 ", the resource 3 ', and the resource 3 into the database through the database read-write module 3 based on the exclusive locks of the NIDs of the resource 3", the resource 3', and the resource 3.

Optionally, the database read/write module in the embodiment of the present disclosure may be a Global Content Manager Service (GCMS).

And S203, adding the resource content data to the local queue every time the writing of one piece of resource content data is completed.

S204, based on the mutual exclusion lock of the NID of each piece of resource content data in the local queue, pushing the resource content data in the local queue to a first message middleware corresponding to the local queue, so that the caching service acquires and caches the resource content data from the first message middleware.

In the embodiment of the present disclosure, the RCMC processing module may generate one local queue for each first message middleware, and the resource content data that needs to be pushed into the same message middleware may be added to the same local queue.

By adopting the technical scheme, the RCMC processing module can receive the resource content data corresponding to the NID with the mapping relation with the RCMC processing module, so that the resource content data with the same NID can be received by the same RCMC processing module. The RCMC processing module writes the resource content databases carrying the same NID in sequence based on the mutual exclusion lock of the NID, and is limited by the mutual exclusion lock, the RCMC processing module can only process one resource content data in the resource content data with the same NID at the same time, and the resource content data updated after the same resource can be prevented from being written in first, so that the resource content data of the same resource are ensured to be written in the databases and the local queue in sequence. Because the RCMC processing module also utilizes the exclusive lock of the NID when pushing the resource content data in the local queue to the first message middleware, the resource content data of the same resource is ensured to be sequentially pushed to the first message middleware, so that the caching service can acquire and cache the resource content data of the same resource according to a correct sequence, the inconsistency between the caching layer and the data version in the database is avoided, and the caching layer can cache the resource content data of the correct version.

In addition, if the data version cached by the cache layer is wrong, only the data cached by the cache layer can be updated manually, and if the error is not found in time and the manual update is performed, the cached data can be wrong for a long time.

In this embodiment of the present disclosure, the resource content data acquired by the RCMC receiving module comes from the second message middleware, and the second message middleware is configured to cache the updated resource content data.

When updated resource content data exists, the updated resource content data may be issued to the second message middleware, the plurality of RCMC receiving modules in the embodiment of the present disclosure may be used as consumers to consume the resource content data in the second message middleware, the same resource content data in the second message middleware may only be consumed by one RCMC receiving module, and when one resource content data in the second message middleware is acquired by one RCMC receiving module, the resource content data may be deleted from the second message middleware.

Optionally, after the RCMC receiving module obtains the resource content data, basic verification and validity verification may be performed on the resource content data, and after the verification is passed, the resource content data is sent to the RCMC processing module. The method for verifying the resource content data is the same as the method for verifying the resource content data by the write-in module in the prior art, and is not described herein again.

By adopting the method, the RCMC receiving module can acquire the resource content data from the second message middleware, and the message middleware adopts a sequential storage mechanism, so that the RCMC receiving module can acquire the resource content data according to the sequence of the resource content data updating, and the RCMC receiving module can subsequently transmit the resource content data to the RCMC processing module according to the sequence of the resource content data updating.

In another embodiment of the present disclosure, the resource content data received by the RCMC processing module in S201 is: and after the RCMC receiving module carries out consistent hash operation on the NID of the acquired resource content data to obtain a hash value, the determined resource content data corresponding to the hash value with the preset mapping relation with the RCMC processing module is determined.

When the RCMC receiving module acquires a piece of resource content data, the consistent hash operation may be performed on the NID of the resource content data to obtain the hash value of the NID. The RCMC receiving module prestores preset mapping relationships between the hash values and the RCMC processing modules, and the RCMC receiving module can search the RCMC processing modules corresponding to the calculated hash values from the preset mapping relationships, and then synchronize the resource content data to the RCMC processing modules based on the synchronization interfaces. The RCMC receiving module can classify the received resource content data according to the hash value of NID, so that the resource content data of the same resource is processed by the same RCMC processing module all the time.

For example, as shown in fig. 4, all 3 RCMC receiving modules in fig. 4 may receive the resource content data of resource 1, and all the 3 RCMC receiving modules may send the resource content data belonging to resource 1 to the same RCMC processing module.

The consistent hash operation adopted in the embodiment of the present disclosure may be any consistent hash algorithm, for example, a remainder division and remaining method may be used to perform remainder taking (mod) on the NID, and an obtained remainder is used as a hash value of the NID.

When the number of RCMC processing modules in the system changes (for example, the number of RCMC processing modules changes due to the increase or migration of RCMC instances), the mapping relationship between each hash value and the RCMC processing modules may be re-determined, so that subsequent resource content data with the same NID may still be sent to the same RCMC processing module after the mapping relationship is transformed.

By adopting the method, the receiving function and the processing function are separated in the embodiment of the disclosure, the RCMC receiving module can send the resource content processing module with the same NID to the same RCMC processing module based on the mapping relation between the Hash value of the NID and the RCMC processing module, so that different RCMC processing modules can be prevented from processing different resource content data of the same resource, and further, the problem of time sequence disorder caused by writing the later updated resource content data into the cache layer first is avoided, and the resource content data of the same resource is finally written into the cache layer in sequence.

In an embodiment of the present disclosure, the resource content data that needs to be written into the cache layer and the resource content data that does not need to be written into the cache layer exist in the resource content data received by the RCMC processing module, and on this basis, the above S203 may specifically be implemented as:

when the writing of one piece of resource content data is finished, under the condition that the resource content data needs to be written into a cache layer, the resource content data is added into a local queue corresponding to a first message middleware which needs to be pushed by the resource content data, wherein the plurality of first message middleware and the plurality of local queues in the RCMC processing module have one-to-one correspondence.

After the RCMC writes the resource content data into the database, it can also determine whether the resource content data needs to be written into the cache layer. The determining method may be flexibly set according to actual requirements, and the resource content data that does not need to be distributed through the cache layer does not need to be written into the cache layer, for example, the resource content of the specified NID may be set not to be written into the cache layer, or the resource content data of the specified type does not need to be written into the cache layer, which is not specifically limited in the embodiment of the present disclosure.

In the related art, the writing module may determine which first message to push the resource content data to based on the actual content included in the resource content data. In the embodiment of the present disclosure, a local queue corresponding to each first message middleware exists in the RCMC processing module, and further, the RCMC writing module may write the resource content data that needs to be pushed to the same message middleware into the same first message middleware.

By the method, each time one piece of resource content data is written into the database, the resource data is added into the local queue corresponding to the first message middleware which needs to push the resource content data, so that the resource content data which needs to be pushed to the same message middleware can be ensured to be written into the local queue in sequence, and then the resource content data can be sequentially pushed to the first message middleware subsequently, and the resource content data pushed to the first message middleware is prevented from being disordered in time sequence.

On the basis of the foregoing embodiment, S204 may be specifically implemented as:

and acquiring each piece of resource content data in a preset window of the local queue through a preset number of threads, wherein the preset window comprises a previous preset number of pieces of resource content data in the local queue.

And pushing the acquired resource content data to a first message middleware corresponding to the local queue through a mutual exclusion lock of the NID of each thread based on the acquired resource content data.

The size of the preset window can be flexibly configured, and the number of the resource content data included in the preset window of the local queue can be equal to the number of threads consuming the resource content data in the local queue in the RCMC processing module.

Exemplarily, as shown in fig. 5, it is assumed that the RCMC processing module receives three pieces of resource content data, respectively referred to as resource 1', and resource 2 in sequence, and the three pieces of resource content data need to be pushed to the same message middleware.

After the RCMC processing module writes resource 1 "into the database, resource 1" may be added to the local queue shown in fig. 5. Resource 1 'is then added to the local queue after resource 1' is written to the database. After resource 2 is written to the database, resource 2 is added to the local queue.

Assuming that the preset window is 3, the preset window of the local queue sequentially includes resource 1 ', resource 1' and resource 2, and the RCMC module can push the resource content data in the preset window to the first message middleware through 3 threads.

For example, thread 1 acquires resource 1 ", thread 2 acquires resource 1', and thread 3 acquires resource 2. Since resources 1 "and 1 'have the same NID, thread 1 uses the exclusive lock of the NID in writing resource 1" to the first message middleware, at which point thread 2 cannot process resource 1'. But thread 3 can write resource 2 into the first message middleware at this time. After the resource 1 ' of the thread 1 is written into the first message middleware, the thread 2 can write the resource 1 into the first message middleware by using the exclusive lock of the NID, so that the resource content data in the first message middleware are the resource 1 ', the resource 2 and the resource 1 ' in sequence.

Due to the existence of the mutual exclusion lock, the pushing sequence between the resource 1 'and the resource 1' belonging to the same resource is not disordered. Therefore, the method can realize the storage and the caching of the resource content data in the global locking at the resource granularity, namely the resource content data with the same NID are sequentially written into the database and sequentially pushed to the first message middleware by using the exclusive lock of the NID in the whole process, so that the sequence of the cache layer acquiring the resource content data with the same NID from the first message middleware is consistent with the storage sequence in the database, and the consistency of the cache layer and the data version stored in the database is further ensured.

Corresponding to the foregoing method embodiment, an embodiment of the present disclosure further provides a data processing apparatus, where the apparatus is applied to an RCMC processing module, and as shown in fig. 6, the apparatus includes:

a receiving unit 601, configured to receive resource content data sent by an RCMC receiving module, where the resource content data carries a message identifier NID, and the NID of the resource content data has a mapping relationship with an RCMC processing module;

a write-in unit 602, configured to construct a mutual exclusion lock for the NID of the received resource content data, and sequentially write resource content data carrying the same NID into a database based on the mutual exclusion lock;

an adding unit 603, configured to add, to the local queue, one piece of resource content data each time writing of the resource content data is completed;

a pushing unit 604, configured to push the resource content data in the local queue to a first message middleware corresponding to the local queue based on a mutual exclusion lock of the NID of each piece of resource content data in the local queue, so that the caching service acquires and caches the resource content data from the first message middleware.

In another embodiment of the present disclosure, the pushing unit 604 is specifically configured to:

acquiring each piece of resource content data in a preset window of a local queue through a preset number of threads, wherein the preset window comprises a previous preset number of pieces of resource content data in the local queue;

In another embodiment of the present disclosure, the adding unit 603 is specifically configured to:

In another embodiment of the present disclosure, the resource content data received by the RCMC processing module is: and after the RCMC receiving module carries out consistent hash operation on the NID of the acquired resource content data to obtain a hash value, the determined resource content data corresponding to the hash value with the preset mapping relation with the RCMC processing module is determined.

In another embodiment of the present disclosure, the resource content data acquired by the RCMC receiving module comes from a second message middleware, and the second message middleware is configured to cache the updated resource content data.

Corresponding to the foregoing method embodiment, an embodiment of the present disclosure further provides a data processing system, as shown in fig. 7, where the system includes: the system comprises a plurality of RCMC receiving modules, a plurality of RCMC processing modules, a database, first message middleware and a cache service.

Fig. 7 exemplarily shows 3 RCMC receiving modules and 3 RCMC processing modules, and in an actual implementation, the number of the modules included in the data processing system is not limited to this.

Each RCMC receiving module is configured to send the resource content data to the RCMC processing module having a mapping relationship with the NID of the resource content data.

Each RCMC processing module is configured to implement the method steps implemented by the RCMC processing module in the foregoing method embodiment.

And the cache service is used for acquiring and caching the resource content data from the first message middleware.

The cache service may receive a data query request of the service, the cache service preferentially trusts locally cached data, and if the locally cached data is missing, for example, resource content data corresponding to a certain resource is not locally cached, the cache service may perform a penetrating access to the database, thereby obtaining latest resource content data of the resource. If the data version of the local cache has errors, the cache service returns the data of the error version to the business service, and a process of penetrating access to the database is not triggered, so that the business service acquires the data of the error version, and further the business execution is wrong.

By adopting the system provided by the embodiment of the disclosure, the resource content data of the same resource can be sent to the same RCMC processing module, and the RCMC processing module sequentially writes the resource content data of the same resource into the database and sequentially pushes the resource content data to the first message middleware based on the mutual exclusion lock, that is, for a plurality of resource content data of the same resource, the resource content data of the wrong version of the cache due to the wrong sequence of the resource content data acquired by the cache service from the first message middleware is strictly pushed according to the receiving sequence, so that the resource content data of the wrong version of the cache is prevented from being cached, and the consistency of the stored data of the cache layer and the database in the system is improved on the premise of increasing the consumption of system resources as low as possible.

In this disclosure, each RCMC receiving module is specifically configured to perform consistent hash operation on the NID of each piece of acquired resource content data to obtain a hash value, and send the resource content data to the RCMC processing module having a preset mapping relationship with the hash value.

Optionally, as shown in fig. 7, the system further comprises a second message middleware.

The second message middleware is used for caching the updated resource content data;

and the RCMC receiving modules are used for acquiring the resource content data from the second message middleware.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good customs of the public order.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 8 illustrates a schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 8, the electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)802 or a computer program loaded from a storage unit 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic apparatus 800 can also be stored. The calculation unit 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.

A number of components in the electronic device 800 are connected to the I/O interface 805, including: an input unit 806, such as a keyboard, a mouse, or the like; an output unit 807 such as various types of displays, speakers, and the like; a storage unit 808, such as a magnetic disk, optical disk, or the like; and a communication unit 809 such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

Computing unit 801 may be a variety of general and/or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 801 executes the respective methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and/or installed onto device 800 via ROM 802 and/or communications unit 809. When loaded into RAM 803 and executed by the computing unit 801, a computer program may perform one or more steps of the data processing method described above. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel or sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A data processing method is applied to a Resource Content Management Center (RCMC) processing module, and comprises the following steps:

2. The method of claim 1, wherein pushing the resource content data in the local queue to the first message middleware corresponding to the local queue based on a mutually exclusive lock of the NID of each resource content data in the local queue comprises:

acquiring each piece of resource content data in a preset window of the local queue through a preset number of threads, wherein the preset window comprises a previous preset number of pieces of resource content data in the local queue;

3. The method of claim 1 or 2, wherein adding a piece of resource content data to a local queue each time a write of the resource content data is completed comprises:

4. The method of claim 1, wherein the resource content data received by the RCMC processing module is: and after the RCMC receiving module carries out consistent hash operation on NID of the acquired resource content data to obtain a hash value, the determined resource content data corresponding to the hash value with a preset mapping relation with the RCMC processing module.

5. The method of claim 4, wherein the resource content data obtained by the RCMC receiving module is from a second message middleware, and the second message middleware is configured to cache updated resource content data.

6. A data processing device, the device is applied to a Resource Content Management Center (RCMC) processing module, and the device comprises:

7. The apparatus according to claim 6, wherein the pushing unit is specifically configured to:

8. The apparatus according to claim 6 or 7, wherein the adding unit is specifically configured to:

when the resource content data needs to be written into the cache layer, the resource content data is added into a local queue corresponding to a first message middleware which needs to be pushed by the resource content data, wherein the plurality of first message middleware and the plurality of local queues in the RCMC processing module have a one-to-one correspondence relationship.

9. The apparatus of claim 6, wherein the RCMC processing module receives resource content data as: and after the RCMC receiving module carries out consistent hash operation on NID of the acquired resource content data to obtain a hash value, the determined resource content data corresponding to the hash value with a preset mapping relation with the RCMC processing module.

10. The apparatus of claim 9, wherein the resource content data obtained by the RCMC receiving module is from second message middleware, and the second message middleware is configured to cache updated resource content data.

11. A data processing system, comprising: the system comprises a plurality of RCMC receiving modules, a plurality of RCMC processing modules, a database, a first message middleware and a cache service;

each RCMC processing module for performing the method of any one of claims 1-5;

12. The system of claim 11, wherein,

each RCMC receiving module is specifically configured to perform consistent hash operation on the NID of each piece of acquired resource content data to obtain a hash value, and send the resource content data to the RCMC processing module having a preset mapping relationship with the hash value.

13. The system of claim 12, further comprising second message middleware;

the plurality of RCMC receiving modules are configured to obtain resource content data from the second message middleware.

14. An electronic device, comprising:

at least one processor; and

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

15. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-5.

16. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-5.