CN116150273A - Data processing method, device, computer equipment and storage medium - Google Patents

Data processing method, device, computer equipment and storage medium Download PDF

Info

Publication number
CN116150273A
CN116150273A CN202310150640.XA CN202310150640A CN116150273A CN 116150273 A CN116150273 A CN 116150273A CN 202310150640 A CN202310150640 A CN 202310150640A CN 116150273 A CN116150273 A CN 116150273A
Authority
CN
China
Prior art keywords
data
configuration
resource pool
service
synchronized
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310150640.XA
Other languages
Chinese (zh)
Inventor
张辉
于洋
李全松
任耀耀
张廉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Mobile Communications Group Co Ltd
China Mobile Information Technology Co Ltd
Original Assignee
China Mobile Communications Group Co Ltd
China Mobile Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Mobile Communications Group Co Ltd, China Mobile Information Technology Co Ltd filed Critical China Mobile Communications Group Co Ltd
Priority to CN202310150640.XA priority Critical patent/CN116150273A/en
Publication of CN116150273A publication Critical patent/CN116150273A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/275—Synchronous replication
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2379—Updates performed during online database operations; commit processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The application discloses a data processing method, a data processing device, computer equipment and a storage medium. Receiving service configuration information comprising a target plug-in identifier, which is sent by a configuration resource center, and acquiring a target data configuration plug-in corresponding to the target plug-in identifier according to the plug-in identifier and the associated information of the data configuration plug-in; determining to-be-synchronized configuration relation data corresponding to service configuration information through a target data configuration plug-in, and sending a data synchronization instruction corresponding to the to-be-synchronized configuration relation data to a configuration data center service of at least one second resource pool in a peer-to-peer network where the first resource pool is located, wherein the data synchronization instruction comprises description event data, and the data synchronization instruction is used for indicating the configuration data center service of the second resource pool to update the description event data. Therefore, the data center service is configured through distributed multipoint deployment, so that real-time synchronization and updating of data are completed, waste of labor resources during data scheduling is avoided, and data scheduling efficiency is improved.

Description

Data processing method, device, computer equipment and storage medium
Technical Field
The application belongs to the technical field of computers, and particularly relates to a data processing method, a data processing device, computer equipment and a storage medium.
Background
In the related art, the data scheduling mode is to store all configuration relation data in a central database established by a configuration resource center, establish a node database in a resource pool cluster of each branch, and send the data relation data to each node database by the central database. When configuration relation data changes, the central database needs to be updated firstly, and after the update is completed, the snapshot synchronization function of the database is scheduled, and the updated data of the central database is synchronized into each node database, so that the application program of each branched resource pool acquires the changed configuration relation data by querying the local node database.
However, the data synchronization between the central database and the node database is based on the snapshot synchronization function, when the refresh function of the database corresponding to the snapshot synchronization function is lost, the data synchronization is failed, and when the data synchronization is failed, the data refresh needs to be performed again, otherwise, the situation that the service transaction is failed due to inconsistent configuration related data can occur, in addition, special personnel are required to operate the refresh and synchronization snapshot in the maintenance of the configuration related data, so that the data scheduling efficiency is reduced while the manual resources are wasted.
Disclosure of Invention
The embodiment of the application provides a data processing method, a device, computer equipment and a storage medium, which can solve the problems of manual resource waste and low data scheduling efficiency of a data scheduling mode in the prior art.
In a first aspect, an embodiment of the present application provides a data processing method, where the method may include:
receiving service configuration information sent by a configuration resource center, wherein the service configuration information comprises a target plug-in identification, and the service configuration information is determined by a data synchronization request received by the configuration resource center;
acquiring a target data configuration plugin corresponding to the target plugin identifier according to the plugin identifier and the associated information of the data configuration plugin;
determining to-be-synchronized configuration relation data corresponding to the service configuration information through a target data configuration plug-in;
transmitting a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located; the data synchronization instruction is used for indicating the configuration data center service of the second resource pool to update the description event data, and the description event data is obtained by carrying out content serialization processing on the configuration relationship data to be synchronized.
In a second aspect, embodiments of the present application provide a data processing apparatus, which may include:
the receiving module is used for receiving service configuration information sent by the configuration resource center, wherein the service configuration information comprises a target plug-in identification, and the service configuration information is determined by a data synchronization request received by the configuration resource center;
the acquisition module is used for acquiring a target data configuration plugin corresponding to the target plugin identifier according to the plugin identifier and the associated information of the data configuration plugin;
the determining module is used for determining to-be-synchronized configuration relation data corresponding to the service configuration information through the target data configuration plug-in;
a sending module, configured to send a data synchronization instruction to a configuration data center service of at least one second resource pool in a peer-to-peer network where the first resource pool is located; the data synchronization instruction is used for indicating the configuration data center service of the second resource pool to update the description event data, and the description event data is obtained by carrying out content serialization processing on the configuration relationship data to be synchronized.
In a third aspect, embodiments of the present application provide a data processing system, the system may include: configuring a resource center and at least one resource pool, each of the at least one resource pool including a configuration data center service; wherein, the liquid crystal display device comprises a liquid crystal display device,
The resource allocation center is used for receiving the data synchronization request and generating service allocation information based on the data synchronization request, wherein the service allocation information comprises a target plug-in identifier corresponding to the data synchronization request; and transmitting service configuration information to at least one resource pool, the at least one resource pool including a first resource pool;
the first resource pool is used for acquiring a target data configuration plugin corresponding to the target plugin identifier according to the plugin identifier and the associated information of the data configuration plugin under the condition of receiving service configuration information sent by the configuration resource center; determining to-be-synchronized configuration relation data corresponding to the service configuration information through a target data configuration plug-in; transmitting a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located;
the data synchronization instruction is used for indicating the configuration data center service of the second resource pool to update the description event data, and the description event data is obtained by carrying out content serialization processing on the configuration relationship data to be synchronized.
In a fourth aspect, embodiments of the present application provide a computing device comprising: a processor and a memory storing computer program instructions;
The processor when executing the computer program instructions implements the data processing method as shown in the first aspect.
In a fifth aspect, embodiments of the present application provide a computer storage medium having stored thereon computer program instructions which, when executed by a processor, implement a data processing method as described in the first aspect.
In a sixth aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to execute a program or instructions to implement a data processing method as shown in the first aspect.
In a seventh aspect, embodiments of the present application provide a computer program product stored in a storage medium, the program product being executable by at least one processor to implement the data processing method as shown in the first aspect.
The data processing method, the device, the computer equipment and the storage medium of the embodiment of the application receive service configuration information including a target plug-in identifier, which is sent by a configuration resource center, acquire a target data configuration plug-in corresponding to the target plug-in identifier according to the plug-in identifier and the associated information of the data configuration plug-in, then determine configuration relationship data to be synchronized corresponding to the service configuration information through the target data configuration plug-in, and send a data synchronization instruction to a configuration data center service of at least one second resource pool in a peer-to-peer network where the first resource pool is located, wherein the data synchronization instruction comprises description event data, the data synchronization instruction is used for indicating the configuration data center service of the second resource pool to update the description event data, and the description event data is obtained by carrying out content serialization processing on the configuration relationship data to be synchronized. In this way, configuration data center services are deployed in each resource pool to provide the association information of plug-in identification and data configuration plug-in, so that the target data configuration plug-in is obtained according to the multiple types of target plug-in identification, the target data configuration plug-in determines the relationship data to be synchronized in the resource pools, and performs content serialization processing on the relationship data to be synchronized to obtain standardized description event data, so that in a peer-to-peer network, each resource pool can read the description event data through the configuration data center services of each resource pool, real-time synchronization and updating of the data are completed, distributed multi-point deployment service is realized, and the problem of dropping points in the data scheduling process is avoided. In the embodiment of the application, each resource pool configuration center service in the peer-to-peer network where the first resource pool is located is regarded as a peer, task flow and data synchronization can be performed among the resource pool configuration center services of each resource pool, and the data synchronization among the resource pool configuration center services does not depend on the configuration resource center, so that the overall scheduling and availability of data are not affected when a single or partial cluster service fails, thereby ensuring that required updated data can be acquired through each resource pool, and improving the data scheduling efficiency. In addition, the configuration data center service in each resource pool can automatically realize the automatic expansion and the elastic capacity expansion of the configuration data center service in real time and concurrently with the call quantity, thereby saving the labor cost of data synchronization and maintenance and improving the accuracy, the integrity and the safety of the synchronization of the configuration data center service among the cross-resource pools.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described, and it is possible for a person skilled in the art to obtain other drawings according to these drawings without inventive effort.
FIG. 1 is a schematic diagram of a data scheduling architecture;
FIG. 2 is a schematic diagram of a data processing system according to an embodiment of the present application;
FIG. 3 is a second schematic diagram of a data processing system according to an embodiment of the present disclosure;
FIG. 4 is a flowchart of a data processing method according to an embodiment of the present application;
FIG. 5 is a schematic diagram of an interface in a data processing method according to an embodiment of the present application;
FIG. 6 is a schematic diagram of an interface in a data processing method according to an embodiment of the present application;
FIG. 7 is a schematic diagram of a data processing apparatus according to one embodiment of the present application;
fig. 8 is a schematic structural diagram of a computer device according to an embodiment of the present application.
Detailed Description
Features and exemplary embodiments of various aspects of the present application are described in detail below to make the objects, technical solutions and advantages of the present application more apparent, and to further describe the present application in conjunction with the accompanying drawings and the detailed embodiments. It should be understood that the specific embodiments described herein are intended to be illustrative of the application and are not intended to be limiting. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present application by showing examples of the present application.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
In the related art, as shown in fig. 1, a configuration resource center establishes a central database for storing all configuration relation data, resource pools (such as a large-area resource pool and a resource-saving pool cluster) of each branch establish a node database, and data contents are issued to the node databases of each branch from the central database. When configuration data changes (new addition, deletion and replacement), the central database needs to be updated first, after the update is completed, the snapshot synchronization function of the database is scheduled or scripts are prepared according to the update conditions, and the updated data of the central database is synchronized to each node database, so that a user end (such as an application program) corresponding to each branched resource pool queries the corresponding node database to obtain the changed configuration relation data.
However, the data synchronization method between the central database and the node database as described above has the following problems: the lazy is strong, the data synchronization between the central database and the node database is based on the snapshot synchronization function, and when the refreshing function of the database corresponding to the snapshot synchronization function is lost, the data synchronization is failed; the fault tolerance is poor, after the data synchronization fails, the data refreshing is needed to be carried out again, at this time, manual intervention is needed, otherwise, the situation that service transaction fails due to inconsistent configuration associated data can occur, the data consistency is difficult to ensure, and the data scheduling efficiency is reduced; the data integrity is poor, because the application of each resource pool depends on the local node database corresponding to the resource pool respectively, if the script cannot accurately synchronize configuration data or the script input is difficult to carry out integrity check on the synchronous configuration data, the problem that the business transaction is abnormal because of the data of the node database can occur, so that the data integrity is poor, and the business efficiency can be influenced; the maintainability is poor, with more and more machines and containerization, the maintenance complexity of the snapshot synchronization function and the script is high, and special personnel are required to operate refreshing and synchronization of the snapshot, so that the cost of data synchronization is increased while the labor resource is wasted; the expansibility is weak, because databases of different types cannot interact with each other, configuration relation data can only be stored in a fixed storage medium, so that the storage or data type of a replacement center database or a node database is complex, and the data synchronization difficulty is increased.
Based on this, in order to solve the above-mentioned problems, the embodiments of the present application provide a data processing method for automatically synchronizing data across resource pools based on a distributed and micro-service technology, in which a decentralizing architecture and a peer-to-peer network are adopted, and highly available clustered configuration data center services are deployed in platform environments (Platform as a Service, paaS) of each resource pool, such as a large-area resource pool and a resource-saving pool, and data synchronization and interaction are performed between the configuration data center services of each resource pool through the peer-to-peer network. In the peer-to-peer network, each resource pool configuration center service is regarded as a peer, and task flow and data synchronization between each configuration data center service are independent of the configuration resource center. Therefore, when the service of the configuration data center of a single or partial resource pool fails, the overall scheduling and availability of the configuration relation data are not affected, so that the required data can be obtained through each resource pool, and the service can be provided for the associated user side. In addition, after the configuration data center service is deployed, under the condition that the configuration resource center receives a data synchronization request, the configuration resource center can send service configuration information to each resource pool, wherein the service configuration information can comprise information such as service identification, resource pool information, service copy number, synchronization time, plug-in name, plug-in configuration, persistence configuration and the like, the resource pool receiving the service configuration information is a first resource pool, a target data configuration plug-in is started according to the service configuration information, the target data configuration plug-in finds a data source of relationship data to be configured to be synchronized to perform data loading, the relationship data to be configured to be synchronized after loading is subjected to content serialization processing to obtain description event data, whether the description event data meets the persistence data condition is determined through a persistence module in the first resource pool, and if the description event data meets the persistence data condition, the description event data is stored in a classified mode. Finally, the user side corresponding to the resource pool, such as the first resource pool, can acquire the description event data related to the resource pool through a connection interface (such as a Restful interface) provided by a management center accessing the first resource pool or a navigation-based interface (Sidecar). Of course, the data synchronization instruction may also be sent to the configuration data center service of at least one second resource pool in the peer-to-peer network through the peer-to-peer network where the first resource pool is located, so as to synchronize the data updated by the center database into each node database.
Based on the foregoing, first, with reference to fig. 2, a description is given of a data processing system provided in an embodiment of the present application.
As shown in fig. 2, data processing system 20 may include a configuration resource center 201 and at least one resource pool (e.g., resource pool 1, resource pools 2, … …, resource pool N, N being an integer greater than 1). Wherein each of the at least one resource pool includes a configuration data center service, and the at least one resource pool may include a first resource pool.
The resource allocation center is used for receiving a data synchronization request (such as an Http request) and generating service allocation information based on the data synchronization request, wherein the service allocation information comprises a target plug-in identifier corresponding to the data synchronization request; and, referring to the dashed line portion in fig. 3, transmitting service configuration information to at least one resource pool, the at least one resource pool including the first resource pool. The user may configure the data synchronization request through the interface shown in fig. 4, and at this time, the user may select a resource pool for receiving service configuration information corresponding to the data synchronization request from the configuration data synchronization request.
Before sending service configuration information to at least one resource pool, the configuration resource center 201 sends a service configuration instruction to each resource pool, where the service configuration instruction is used to instruct the resource pool to construct a configuration data center service according to the service configuration information carried by the service configuration instruction, where the configuration data center service may provide multiple types of plug-in identifiers and association information of data configuration plug-ins, and interact with at least one resource pool in the peer-to-peer network through the peer-to-peer network, where the content of interaction may include description event data corresponding to configuration relationship data to be synchronized. Illustratively, a worker managing the service may configure the configuration data center service in each resource pool through an interface as shown in fig. 5, and browse the state of the configuration data center service in each resource pool from its corresponding interface.
The first resource pool is used for acquiring a target data configuration plugin corresponding to the target plugin identifier according to the plugin identifier and the associated information of the data configuration plugin under the condition of receiving service configuration information sent by the configuration resource center; acquiring data sources (such as a data source A, a data source B and a data source C) of the to-be-synchronized configuration relation data through a target data configuration plug-in, and determining the to-be-synchronized configuration relation data corresponding to the service configuration information based on the data sources; and, referring to the solid line portion of fig. 3, sending a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located. The data synchronization instruction is used for indicating the configuration data center service of the second resource pool to update the description event data, and the description event data is obtained by carrying out content serialization processing on the configuration relationship data to be synchronized. Here, the second resource pool included in the peer-to-peer network where the first resource pool is located may be a resource pool associated with the configuration resource center 201, that is, at least one resource pool includes the second resource pool, and of course, a resource pool not associated with the configuration resource center 201 may also be used.
It should be noted that, the first resource pool may further include at least one subscription interface and a unified configuration data scheduler (Unified configuration data scheduling, UCDS), and the UCDS program may provide a UCDS Sidecar interface. Based on this, in addition to sending a data synchronization instruction to a configuration data center service of at least one second resource pool in a peer-to-peer network where the first resource pool is located, after determining configuration relation data to be synchronized, content serialization processing may be performed on the configuration relation data to be synchronized through UCDS to obtain description event data, and then the description event data is added to a data object list in the first resource pool, so that, when a data subscription request sent by at least one user terminal associated with the first resource pool is received, the description event data in the data object list is sent to the user terminal through a subscription interface or UCDS Sidecar interface, so as to provide services to the associated user terminal.
Based on this, the above data processing system has the advantages that firstly, compatibility of multiple data types does not need to consider whether different data sources can interact or whether the data of the configuration relation to be synchronized is persistent, and the data processing system can uniformly manage the different data sources and the data types; secondly, the coupling degree between different modules in the service is extremely low, and one block can be independently used or replaced by other services; thirdly, stateless service is started only by slightly configuring, and meanwhile, the efficient horizontal expansion is convenient, and the load pressure of an upstream data source is reduced; fourthly, the data comparison is carried out after the MD5 calculation, each time of transmission is MD5 content, the actual configuration data communication is carried out only after the actual change occurs, the content of each time of data synchronization is reduced, and the communication efficiency between services is improved; fifthly, the high readable data is convenient to transplant, the data is serialized into a standard JSON format and has corresponding name naming, so that the structural property of the data is enhanced, the configuration data is convenient to read, and the unified JSON format is convenient for the relocation of the configuration data.
Based on the data processing system, the embodiment of the application provides a data processing method, a device, equipment and a storage medium. The data processing method, apparatus, device and storage medium according to embodiments of the present application will be described in detail below with reference to fig. 6, and it should be noted that these embodiments are not intended to limit the scope of the disclosure of the present application.
The data processing method provided in the embodiment of the present application is described in detail below with reference to fig. 6.
Fig. 6 is a flowchart of a data processing method according to an embodiment of the present application.
As shown in fig. 6, the data processing method may be applied to any resource pool in the data processing system as shown in fig. 2 or fig. 3, specifically, may be applied to the first resource pool, and the data processing method may specifically include the following steps:
step 610, receiving service configuration information sent by a configuration resource center, where the service configuration information includes a target plug-in identifier, and the service configuration information is determined by a data synchronization request received by the configuration resource center; step 620, obtaining a target data configuration plugin corresponding to the target plugin identifier according to the plugin identifier and the associated information of the data configuration plugin; step 630, determining configuration relationship data to be synchronized corresponding to the service configuration information through the target data configuration plug-in; step 640, sending a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located; the data synchronization instruction is used for indicating the configuration data center service of the second resource pool to update the description event data, and the description event data is obtained by carrying out content serialization processing on the configuration relationship data to be synchronized.
Therefore, configuration data center services are deployed in each resource pool to provide association information of plug-in identification and data configuration plug-in, so that target data configuration plug-in is obtained according to multiple types of target plug-in identification, the target data configuration plug-in determines configuration relation data to be synchronized in the resource pools and performs content serialization processing on the configuration relation data to be synchronized to obtain standardized description event data, and therefore, in a peer-to-peer network, each resource pool can read the description event data through the configuration data center services of each resource pool, real-time synchronization and updating of the data are completed, distributed multi-point deployment service is achieved, and the problem of dropping points in a data scheduling process is avoided. In the embodiment of the application, each resource pool configuration center service in the peer-to-peer network where the first resource pool is located is regarded as a peer, task flow and data synchronization can be performed among the resource pool configuration center services of each resource pool, and the data synchronization among the resource pool configuration center services does not depend on the configuration resource center, so that the overall scheduling and availability of data are not affected when a single or partial cluster service fails, thereby ensuring that required updated data can be acquired through each resource pool, and improving the data scheduling efficiency. In addition, the configuration data center service in each resource pool can automatically realize the automatic expansion and the elastic capacity expansion of the configuration data center service in real time and concurrently with the call quantity, thereby saving the labor cost of data synchronization and maintenance and improving the accuracy, the integrity and the safety of the synchronization of the configuration data center service among the cross-resource pools.
The above steps are described in detail below, and are specifically described below.
First, referring to step 610, in one or more possible embodiments, in the event that a data synchronization request is received by a configuration resource center, service configuration information is generated and sent to any resource pool associated with the configuration resource center based on the data synchronization request. Based on this, any resource pool, such as the first resource pool, may receive service configuration information sent by the configuration resource center. Specifically, the service configuration information may include a target plug-in identification and at least one of the following: service identification, resource pool information, number of service copies, synchronization time, plug-in configuration, and persistence configuration.
Next, referring to step 620, in one or more possible embodiments, a preconfigured configuration data center service is included in the first resource pool, based on which, a plug-in identifier corresponding to the preconfigured multiple types of data configuration plug-ins and associated information of the data configuration plug-ins can be invoked by the configuration data center service, and based on the plug-in identifier and the associated information of the data configuration plug-ins, a target plug-in identifier is obtained. Wherein the target plugin identification is used to uniquely identify the plugin.
Furthermore, referring to step 630, in one or more possible embodiments, step 630 may specifically include:
step 6301, configuring plug-in units through target data, and reading plug-in unit configuration information in service configuration information, wherein the plug-in unit configuration information comprises a data source of configuration relation data to be synchronized corresponding to the service configuration information;
step 6302, loading configuration data of the data source to obtain configuration relationship data to be synchronized.
The method comprises the steps that a target data configuration plug-in can be started through configuration data center service of a first resource pool, a data source of corresponding configuration relation data to be synchronized is found through service configuration information read by the target data configuration plug-in, and initial loading operation is conducted on the configuration relation data to be synchronized in the data source, so that the configuration relation data to be synchronized is obtained. The target data configuration plug-in supports two modes of reading file configuration data and calling a data source api interface to acquire a data source.
Based on this, in one example, after step 6301, the data processing method may further include:
in the event that the plug-in configuration information in the service configuration information cannot be read by the target data configuration plug-in, the configuration data center service of the first resource pool sends the abnormal state information of the data source to the target group, and the target group comprises an administrator account for processing the abnormal state information.
When the data source fails or the plug-in configuration information in the service configuration information cannot be read through the target data configuration plug-in, the first resource pool can automatically send the abnormal state information of the data source to the alarm inspection robot in a command mode, so that the alarm inspection robot can send the abnormal state information of the data source to a target group related to a service after receiving the command, and an administrator of an administrator account for processing the abnormal state information in the target group can process the abnormal state information.
Then, referring to step 640, in one or more possible embodiments, prior to step 640, the data processing method may further include:
and carrying out content serialization processing on the relationship data to be synchronized through the target data configuration plug-in to obtain standardized packaged description event data, wherein the description event data comprises the relationship data to be synchronized after serialization, the data type of the relationship data to be synchronized and tracing information.
Illustratively, after the primary loading is finished, the content serialization processing is performed on the to-be-synchronized configuration relationship data through the target data configuration plug-in, that is, the to-be-synchronized configuration relationship data is packaged into standardized description event data (i.e., cloudEvent data), and the CloudEvent data structure comprises the serialized to-be-synchronized configuration relationship data, the data type (including non-critical and critical types) and the traceability information (including resource pool information, data format, data id, data type, sha256 values and the like).
Based on this, in another possible embodiment or embodiments, before step 640, the data processing method may further include:
adding descriptive event data to a list of data objects in a first resource pool;
and sending descriptive event data in a data object list to the user terminal under the condition that a data subscription request of at least one user terminal associated with the first resource pool is received.
The standard CloudEvent data is pushed to the data object list in the first resource pool by the target data configuration plug-in, so that the internal service (such as the configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located) and the external user terminal acquire the related description time data from the data object list.
In yet another or more possible embodiments, the data object list to be synchronously configured with relationship data contained in the first resource pool, based on which, prior to step 640, the data processing method may further comprise:
step 6501, obtaining the data type of the relationship data to be synchronously configured in the data object list;
step 6502, determining whether the storage duration of the description event data corresponding to the configuration relationship data to be synchronized in the data object list exceeds a preset threshold value when the data type is characterized in that the configuration relationship data to be synchronized is non-key type data;
Step 6503, marking the description event data through a preset persistent tag when the storage duration of the description event data in the data object list exceeds a preset threshold and the compression mode of the relationship data to be synchronously configured serves a preset compression mode;
step 6504, storing the marked descriptive event data to a data object list.
Further, after step 6501, the data processing method may further include:
step 6505, under the condition that the data type is characterized in that the relationship data to be synchronized is key data, encrypting the description event data corresponding to the relationship data to be synchronized through at least one symmetrical and asymmetrical encryption algorithm in the first resource pool;
step 6506, obtaining a persistent label corresponding to the encrypted descriptive event data, wherein the persistent label is UCDSID code;
step 6507, storing the encrypted descriptive event data with UCDSID encoding to a data object list.
Illustratively, the foregoing steps 6501 through 6507 are still illustratively described with reference to the foregoing step 640, and are specifically described as follows: the first resource pool is provided with a persistence module, the persistence module is used for acquiring newly added CloudEvent data from a data object list, judging according to data types in the CloudEvent data, constructing corresponding persistence models according to different data types, managing data storage time length, compression mode and cleaning mechanism according to persistence processing modes of non-key type data in the persistence model when the persistence module reads that the data types of the CloudEvent data are non-key type data, namely marking CloudEvent data meeting persistence conditions, and storing marked description event data into the data object list. On the contrary, when the data type of the read CloudEvent data is key class data, the key class data model also provides a plurality of symmetric and asymmetric encryption algorithms (or Sha256 algorithm) on the basis of a basic persistence processing mode to encrypt the key data, and simultaneously, a unique UCDSID code is automatically generated for key class configuration data to prevent data tampering.
Based on this, the first resource pool includes synchronization configuration information including at least one of: the step 640 may specifically include:
generating a synchronous instruction according to description event data with a lasting label in a data object list;
and sending a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located according to the synchronization configuration information.
For example, the configuration resource center may create synchronization configuration information (Custom resources) in advance for each resource pool, where the Custom resources include information such as data synchronization time, synchronization operation type, synchronization manner, and the like, based on which the resource pool may send a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located through the configuration data center service according to the created Custom resources information, that is, synchronizing description event data meeting persistence conditions in the first resource pool to configuration data center services of other resource pools in the peer-to-peer network.
In yet another or more possible embodiments, the data processing method may further comprise, prior to step 640:
Receiving a GRPC request sent by a configuration data center service of at least one second resource pool in the peer-to-peer network;
according to the synchronization configuration information, sending a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located, including:
and sending a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located according to the synchronization configuration information in a broadcasting mode.
In the peer-to-peer network, for example, the configuration data center services of each resource pool can perform data communication in a gRPC request mode, and when new description event data needs to be synchronized, a data broadcasting mode is adopted to complete the data synchronization update of the current resource pool service.
The embodiment of the application provides a data processing method based on configuration data center service in each resource pool, wherein the configuration data center service is deployed in each resource pool by constructing a peer-to-peer network, and a plurality of types of data configuration plugins are provided to form plugin identification and associated information of the data configuration plugins. The configuration resource center finds out a corresponding target data configuration plug-in according to the service configuration data, the target data configuration plug-in completes loading of multi-data source configuration data, the configuration data is serialized into standard CloudEvent data, the CloudEvent data is pushed into a data object list in real time, the persistence module classifies the to-be-synchronized configuration relation data, and different persistence flows are established for non-critical data and critical data. In order to ensure the safety and the integrity of the key class data, the persistence module encrypts the key class configuration data based on the Sha256 algorithm and generates UCDSID codes as unique identifiers of the configuration data. By creating Custom resources for the resource pools, standardized data synchronize configuration relation data to be synchronized, and the fact that each resource pool finishes real-time synchronization and updating of data by reading the Custom resources in a peer-to-peer network is guaranteed. The API interface and the injection Sidecar are used for providing a configuration data sharing mode for the user side, so that data multi-channel acquisition is ensured, meanwhile, high availability of configuration data center service is ensured, data content is distinguished by adopting configuration numbers, the configuration data center service automatically realizes automatic expansion and elastic capacity expansion of the service according to real-time concurrent call of the user side, labor cost of data synchronization and maintenance is saved, accuracy, integrity and safety of data synchronization of configuration relation to be synchronized among cross-resource pools are improved, and a short board of the existing data synchronization mode is avoided.
In summary, in order to better describe the data processing method described above, the embodiments of the present application will be described below with a specific example, and the following details are described.
First, in step S001, the first resource pool receives service configuration information sent by the configuration resource center, where the service configuration information may include information such as service identifier, resource pool information, service copy number, synchronization time, target plug-in identifier, plug-in configuration, and persistent configuration.
Step S002 is received, and the target data configuration plugin corresponding matching with the target plugin identification is searched through the configuration data center service in the first resource pool according to the plugin identification and the association information of the data configuration plugin.
Next, step S003, through the configuration data center service in the first resource pool, the target data configuration plug-in matched in step S002 is started, the target data configuration plug-in reads information in the service configuration information, finds a data source of the configuration relationship data to be synchronized corresponding to the service configuration information, and loads the configuration data of the data source to obtain the configuration relationship data to be synchronized. The target data configuration plug-in supports two modes of reading file configuration data and calling a data source api interface to search a data source. When the data source fails or cannot be acquired, the configuration data center service in the first resource pool can automatically send the abnormal state information of the data source to the alarm inspection robot in a command mode, and the alarm inspection robot can distribute the abnormal state information of the data source to the target group after receiving the command.
In addition, step S004, after obtaining the configuration relationship data to be synchronized, the target data configuration plug-in performs content serialization operation on the configuration relationship data to be synchronized to obtain standardized encapsulated description event data, that is, the configuration relationship data to be synchronized is encapsulated into standardized CloudEvent data, where the CloudEvent data structure includes the serialized configuration relationship data to be synchronized, data types (including non-critical and critical types), and tracing information (including resource pool information, data format, data id, data type, sha256 values, etc.).
Subsequently, in step S005, the target data configuration plug-in pushes the standard CloudEvent data to the data object list, so as to subscribe the data of the internal service and the external user terminal.
Then, step S006, obtaining newly added CloudEvent data from the data object list in real time through the persistence module in the first resource pool, judging according to the data types in the CloudEvent data, constructing corresponding persistence models by the persistence module according to different data types, and managing the data storage duration, compression mode and cleaning mechanism according to the persistence processing mode of the non-critical data in the persistence model when the persistence module reads that the data types of the CloudEvent data are the non-critical data. When the data type of the read CloudEvent data is key data, the key data model also provides a plurality of symmetric and asymmetric encryption algorithms on the basis of a basic persistence processing mode to encrypt the key data, and simultaneously, a unique UCDSID code is automatically generated for key configuration data to prevent data tampering.
Subsequently, in step S007, in conjunction with fig. 4, an administrator configuring the resource center creates a Custom resource for each resource pool, where the Custom resource contains information such as data synchronization time, synchronization operation type, synchronization mode, etc., and the configuration data center service synchronizes the description event data after the local persistence in S006 to other resource pool services in the peer-to-peer network according to the created Custom resource information. In the peer-to-peer network, the data communication is carried out among the resource pool services in a gRPC request mode, and when new description event data needs to be synchronized, the data broadcasting mode is adopted to finish the synchronous update of the configuration data of the current resource pool service.
Then, in step S008-1, after data synchronization is completed through the configuration data center service in the first resource pool, when the user side performs source pasting access through the API interface provided by the configuration data center service in the resource pool, the configuration data center service performs quick search or comparison of data from the data object list, returns a matching result to the user side application, and the user side acquires corresponding configuration relationship data to be synchronized in a data subscription manner. Meanwhile, in step S008-2, according to the actions information (including the application id and the application name) of the application pod of the user side, the configuration data center service in the first resource pool automatically adds Sidecar container to the application pod of the user side, and the user side can access the data nearby through the Sidecar service injected by the configuration data center service in the first resource pool. Meanwhile, in step S008-3, the configuration data center service in the first resource pool is distributed and deployed in multiple resource pools, the data content is distinguished through the configuration number, the configuration data center service in the first resource pool automatically increases and decreases the number of service copies according to the real-time concurrent call quantity of the user side, and further automatic expansion and elastic capacity expansion of the service are realized.
Based on this, the data processing method provided in the embodiment of the application has the following effects: firstly, data are automatically synchronized, timeliness of the data can be guaranteed through self-defining of timing loading time of configuration data, manual execution by special personnel of an administrator is not needed in configurable automatic timing execution, a snapshot refreshing function of the database is not needed by the administrator manually, and a refreshing synchronization result is checked; secondly, the method has expandability, and the embodiment of the application provides a plug-in type configurable database which does not need to consider a data source or a data format and does not need to rely on a data snapshot refreshing function; thirdly, clustered, containerized, micro-service and managed service are adopted in the embodiment of the application, so that the artificial uncertain factors are avoided and stopped, the manager is not required to manually execute, and a plurality of artificial uncertain factors are reduced; fourth, the fault tolerance capability, in the embodiment of the application, the clustered, containerized, micro-serviced and managed services can effectively reduce manual operation, avoid waste of human resources and reduce data processing cost; fifthly, load balancing is performed, the data source is effectively subjected to shunt processing, and various software and hardware load balancers can be arranged in front of the service, so that dependence on database performance is avoided; the method has the advantages that the fault tolerance is high, the resource pools provided by the embodiment of the application are in low coupling, the data reading and the data access are independent and do not affect each other, the clustered management provides high performance for the service, the independence of the configuration data is achieved, the configuration data are independent and do not affect each other, and the situation that the data base is completely relied on is avoided; seventhly, the access diversity is achieved, the Http request mode in the embodiment of the application is easier to use than SQL, the REST protocol can be directly exported at the same time, reading is convenient, and the condition of SQL query function only depending on data is avoided; eighth, data integrity, in the embodiment of the application, a sha256 algorithm is adopted to generate a unique hash value for each configuration data, so that the data integrity is ensured, and the lack of integrity check is avoided; ninth, the data security, the application not only provides a plurality of symmetric and asymmetric encryption algorithms for the data security, but also generates UCDSID codes for each configuration data, thereby preventing the data from being tampered and avoiding the lack of security guarantee; tenth, the service of the new configuration data center is stateless, and the service can be distributed for multi-point deployment, so that the problem of dropping is avoided, the cluster can be completely managed, and the high availability is ensured; eleventh, the data transmission strength is weakened, the access amount of data transmission can be reduced to the minimum by comparing the MD5 content, the whole data is not required to be synchronized each time and compared one by one, and whether the data change occurs can be known only by comparing the MD5 value of the designated configuration name; twelve, load reducing, if all applications access the data source to carry out a large amount of high-speed requests on the data source service, the configuration data source is too large, so that clients of the resource pool cannot normally synchronize configuration data to cause failure of service processing, the resource pool accesses the configuration resource center, and the configuration resource center manages the relationship data to be synchronized, so that the relationship data to be synchronized can be effectively subjected to load reducing.
In addition, after the unified configuration data scheduling method across the resource pools is adopted, the data synchronization mode is changed from manual intervention to automatic processing, so that the complexity of data synchronization operation is reduced, and the labor cost and the time cost input in data synchronization are saved. Meanwhile, the distributed configuration data center service has the advantages of high availability, strong portability, high data synchronization efficiency and high data security, and can be popularized and used in a service support system. Therefore, the problems of inconsistent, incomplete and low accuracy of data synchronization of configuration data among the cross-resource pools are solved, and the flexibility, the rapidity and the simplicity of the data synchronization linkage among the cross-resource pools are improved. In addition, the data processing method provided by the embodiment of the application not only can ensure the rapidity, the integrity and the accuracy of data synchronization by providing configuration data source management, configuration data serialization, configuration data persistence, data automatic timing synchronization capability and data query pluralism, but also ensures the high availability of configuration center services, the unification of configuration content, the consistency and the unification management and control of configuration and the like through the resources such as dynamic expansion capacity of Kubernetes, PVC, custom Resource Definition and the like.
Based on the same inventive concept, the application also provides a data processing device. This is described in detail with reference to fig. 7.
Fig. 7 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.
In some embodiments of the present application, the data processing apparatus shown in fig. 7 may be disposed in any resource pool in the data processing system shown in fig. 2 or fig. 3, and may specifically be the first resource pool.
As shown in fig. 7, the data processing apparatus 70 may specifically include:
a receiving module 701, configured to receive service configuration information sent by a configuration resource center, where the service configuration information includes a target plug-in identifier, and the service configuration information is determined by a data synchronization request received by the configuration resource center;
the obtaining module 702 is configured to obtain a target data configuration plugin corresponding to the target plugin identifier according to the plugin identifier and the association information of the data configuration plugin;
a determining module 703, configured to determine, through the target data configuration plug-in, configuration relationship data to be synchronized corresponding to the service configuration information;
a sending module 704, configured to send a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located; the data synchronization instruction is used for indicating the configuration data center service of the second resource pool to update the description event data, and the description event data is obtained by carrying out content serialization processing on the configuration relationship data to be synchronized.
The data processing device 70 in the embodiment of the present application will be described in detail below.
In one or more alternative embodiments, the data processing apparatus 70 in embodiments of the present application further includes a read module and a load module; wherein, the liquid crystal display device comprises a liquid crystal display device,
the reading module is used for configuring plug-in units through target data, and reading plug-in unit configuration information in the service configuration information, wherein the plug-in unit configuration information comprises a data source of configuration relation data to be synchronized, which corresponds to the service configuration information;
and the loading module is used for loading the configuration data of the data source to obtain the configuration relation data to be synchronized.
In another or more alternative embodiments, the sending module 704 may be further configured to send, by way of a configuration data center service of the first resource pool, abnormal state information of the data source to a target group, where the target group includes an administrator account that handles the abnormal state information, in a case where the plug-in configuration information in the service configuration information is not readable by the target data configuration plug-in.
In yet another or more alternative embodiments, the data processing apparatus 70 in embodiments of the present application further includes a processing module; wherein, the liquid crystal display device comprises a liquid crystal display device,
the processing module is used for carrying out content serialization processing on the relationship data to be synchronized through the target data configuration plug-in, so as to obtain standardized packaged description event data, wherein the description event data comprises the relationship data to be synchronized after serialization, the data type of the relationship data to be synchronized and tracing information.
In yet another alternative embodiment or alternative embodiments, the data processing apparatus 70 in the embodiments of the present application further includes an adding module; wherein, the liquid crystal display device comprises a liquid crystal display device,
an adding module for adding descriptive event data to a list of data objects in a first resource pool;
the sending module 704 may be further configured to send, to the user terminal, descriptive event data in the data object list in case of receiving a data subscription request of at least one user terminal associated with the first resource pool.
In yet another or more alternative embodiments, the data processing apparatus 70 in embodiments of the present application further includes a marking module and a first storage module; wherein, the liquid crystal display device comprises a liquid crystal display device,
the obtaining module 702 may be specifically configured to obtain, in a case where the description event data includes a data object list in the first resource pool, a data type of relationship data to be synchronously configured in the data object list;
the determining module 703 may be further configured to determine, when the data type is characterized in that the configuration relationship data to be synchronized is non-critical data, whether a storage duration of description event data corresponding to the configuration relationship data to be synchronized in the data object list exceeds a preset threshold;
the marking module is used for marking the description event data through a preset lasting tag under the condition that the storage time length of the description event data in the data object list exceeds a preset threshold value and the compression mode of the relationship data to be synchronously configured serves a preset compression mode;
And the storage module is used for storing the marked descriptive event data to the data object list.
In yet another or more alternative embodiments, the data processing apparatus 70 in embodiments of the present application further includes an encryption module and a second storage module; wherein, the liquid crystal display device comprises a liquid crystal display device,
the encryption module is used for encrypting the description event data corresponding to the to-be-synchronized configuration relation data through at least one symmetrical and asymmetrical encryption algorithm in the first resource pool under the condition that the data type is characterized in that the to-be-synchronized configuration relation data is key type data;
the obtaining module 702 is further configured to obtain a persistent tag corresponding to the encrypted descriptive event data, where the persistent tag is UCDSID code;
and the second storage module is used for storing the encrypted descriptive event data with UCDSID codes into a data object list.
In still another or more alternative embodiments, the data processing apparatus 70 in this embodiment of the present application further includes a generating module and a third sending module; wherein, the liquid crystal display device comprises a liquid crystal display device,
the generating module is configured to include synchronization configuration information in the first resource pool, where the synchronization configuration information includes at least one of the following: under the conditions of data synchronization time, synchronization operation type and synchronization mode, generating a synchronization instruction according to description event data with a lasting label in a data object list;
And the third sending module is used for sending a data synchronization instruction to the configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located according to the synchronization configuration information.
In yet another or more alternative embodiments, the receiving module 701 is further configured to receive a GRPC request sent by a configuration data center service of at least one second resource pool in the peer-to-peer network;
the data processing device 70 in this embodiment of the present application further includes a fourth sending module; wherein, the liquid crystal display device comprises a liquid crystal display device,
and the fourth sending module is used for sending a data synchronization instruction to the configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located according to the synchronization configuration information in a broadcasting mode.
The method comprises the steps of receiving service configuration information comprising a target plug-in identifier sent by a configuration resource center, acquiring a target data configuration plug-in corresponding to the target plug-in identifier according to the plug-in identifier and the association information of the data configuration plug-in, determining configuration relation data to be synchronized corresponding to the service configuration information through the target data configuration plug-in, and sending a data synchronization instruction to configuration data center service of at least one second resource pool in a peer-to-peer network where a first resource pool is located, wherein the data synchronization instruction comprises description event data, the data synchronization instruction is used for indicating the configuration data center service of the second resource pool to update description event data, and the description event data is obtained by carrying out content serialization processing on the configuration relation data to be synchronized. In this way, configuration data center services are deployed in each resource pool to provide the association information of plug-in identification and data configuration plug-in, so that the target data configuration plug-in is obtained according to the multiple types of target plug-in identification, the target data configuration plug-in determines the relationship data to be synchronized in the resource pools, and performs content serialization processing on the relationship data to be synchronized to obtain standardized description event data, so that in a peer-to-peer network, each resource pool can read the description event data through the configuration data center services of each resource pool, real-time synchronization and updating of the data are completed, distributed multi-point deployment service is realized, and the problem of dropping points in the data scheduling process is avoided. In the embodiment of the application, each resource pool configuration center service in the peer-to-peer network where the first resource pool is located is regarded as a peer, task flow and data synchronization can be performed among the resource pool configuration center services of each resource pool, and the data synchronization among the resource pool configuration center services does not depend on the configuration resource center, so that the overall scheduling and availability of data are not affected when a single or partial cluster service fails, thereby ensuring that required updated data can be acquired through each resource pool, and improving the data scheduling efficiency. In addition, the configuration data center service in each resource pool can automatically realize the automatic expansion and the elastic capacity expansion of the configuration data center service in real time and concurrently with the call quantity, thereby saving the labor cost of data synchronization and maintenance and improving the accuracy, the integrity and the safety of the synchronization of the configuration data center service among the cross-resource pools.
Based on the same inventive concept, the application also provides a computer device. This is described in detail with reference to fig. 8.
Fig. 8 is a schematic structural diagram of a computer device according to an embodiment of the present application.
As shown in fig. 8, the computer device may comprise an electronic device or a server. The computer device may include, among other things, a processor 801 and a memory 802 storing computer program instructions.
In particular, the processor 801 may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured to implement one or more integrated circuits of embodiments of the present application.
Memory 802 may include mass storage for data or instructions. By way of example, and not limitation, memory 802 may include a Hard Disk Drive (HDD), floppy Disk Drive, flash memory, optical Disk, magneto-optical Disk, magnetic tape, or universal serial bus (Universal Serial Bus, USB) Drive, or a combination of two or more of the above. Memory 802 may include removable or non-removable (or fixed) media, where appropriate. Memory 802 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In a particular embodiment, the memory 802 is a non-volatile solid-state memory. In a particular embodiment, the memory 802 includes solid state storage (ROM). The ROM may be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these, where appropriate.
The processor 801 implements any of the data processing methods of the above embodiments by reading and executing computer program instructions stored in the memory 802.
In one example, the data processing device may also include a communication interface 803 and a bus 810. As shown in fig. 8, the processor 801, the memory 802, and the communication interface 803 are connected to each other via a bus 810 and perform communication with each other.
The communication interface 803 is mainly used to implement communication between each module, apparatus, unit and/or device in the embodiments of the present application.
Bus 810 includes hardware, software, or both, coupling components of the flow control device to one another. By way of example, and not limitation, the buses may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a micro channel architecture (MCa) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus, or a combination of two or more of the above. Bus 810 may include one or more buses, where appropriate. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.
The data processing device may perform the data processing method in the embodiments of the present application, thereby implementing the data processing method and apparatus described in connection with fig. 2 to 6.
In addition, in combination with the data processing method in the above embodiment, the embodiment of the application may be implemented by providing a computer readable storage medium. The computer readable storage medium has stored thereon computer program instructions; which when executed by a processor, implement any of the data processing methods of the above embodiments.
It should be clear that the present application is not limited to the particular arrangements and processes described above and illustrated in the drawings. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps, after appreciating the spirit of the present application.
The functional blocks shown in the above block diagrams may be implemented in hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the present application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine readable medium or transmitted over transmission media or communication links by a data signal carried in a carrier wave. A "machine-readable medium" may include any medium that can store or transfer information. Examples of machine-readable media include electronic circuitry, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio Frequency (RF) links, and the like. The code segments may be downloaded via computer networks such as the internet, intranets, etc.
It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be different from the order in the embodiments, or several steps may be performed simultaneously.
In the foregoing, only the specific embodiments of the present application are described, and it will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the systems, modules and units described above may refer to the corresponding processes in the foregoing method embodiments, which are not repeated herein. It should be understood that the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, which are intended to be included in the scope of the present application.

Claims (13)

1. A data processing method, applied to a first resource pool, comprising:
receiving service configuration information sent by a configuration resource center, wherein the service configuration information comprises a target plug-in identification, and the service configuration information is determined by a data synchronization request received by the configuration resource center;
Acquiring a target data configuration plugin corresponding to the target plugin identifier according to the plugin identifier and the associated information of the data configuration plugin;
determining configuration relation data to be synchronized corresponding to the service configuration information through the target data configuration plug-in;
transmitting a data synchronization instruction to a configuration data center service of at least one second resource pool in a peer-to-peer network where the first resource pool is located; the data synchronization instruction comprises description event data, and the data synchronization instruction is used for indicating configuration data center service of the second resource pool to update the description event data, wherein the description event data is obtained by carrying out content serialization processing on the configuration relationship data to be synchronized.
2. The method according to claim 1, wherein the determining, by the target data configuration plug-in, configuration relationship data to be synchronized corresponding to the service configuration information includes:
reading plug-in configuration information in the service configuration information through the target data configuration plug-in, wherein the plug-in configuration information comprises a data source of configuration relationship data to be synchronized, which corresponds to the service configuration information;
And loading the configuration data of the data source to obtain the configuration relation data to be synchronized.
3. The method according to claim 2, wherein after the reading of the plug-in configuration information in the service configuration information by the target data configuration plug-in, the method further comprises:
and under the condition that the plug-in configuration information in the service configuration information cannot be read through the target data configuration plug-in, the configuration data center service of the first resource pool sends the abnormal state information of the data source to a target group, wherein the target group comprises an administrator account for processing the abnormal state information.
4. The method of claim 1, wherein prior to sending the data synchronization instruction to a configuration data center service of at least one second resource pool in a peer-to-peer network with which the first resource pool is located, the method further comprises:
and carrying out content serialization processing on the to-be-synchronized configuration relation data through the target data configuration plug-in to obtain standardized packaged description event data, wherein the description event data comprises the serialized to-be-synchronized configuration relation data, the data type of the to-be-synchronized configuration relation data and tracing information.
5. The method of claim 4, wherein prior to sending the data synchronization instruction to a configuration data center service of at least one second resource pool in a peer-to-peer network with which the first resource pool is located, the method further comprises:
adding the descriptive event data to a list of data objects in the first resource pool;
and under the condition that a data subscription request of at least one user side associated with the first resource pool is received, transmitting descriptive event data in the data object list to the user side.
6. The method of claim 1, wherein the description event data is stored in a list of data objects in the first resource pool;
before the data synchronization instruction is sent to the configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located, the method further includes:
acquiring the data type of the configuration relation data to be synchronized in the data object list;
determining whether the storage time length of the description event data corresponding to the configuration relation data to be synchronized in the data object list exceeds a preset threshold value or not under the condition that the data type is characterized in that the configuration relation data to be synchronized is non-key type data;
Marking the description event data through a preset lasting tag under the condition that the storage time length of the description event data in the data object list exceeds a preset threshold value and the compression mode of the to-be-synchronized configuration relation data serves a preset compression mode;
and storing the marked description event data into the data object list.
7. The method of claim 6, wherein after the obtaining the data type of the configuration relationship data to be synchronized in the data object list, the method further comprises:
encrypting description event data corresponding to the configuration relation data to be synchronized through at least one symmetrical and asymmetrical encryption algorithm in the first resource pool under the condition that the data type is characterized in that the configuration relation data to be synchronized is key type data;
obtaining a persistent tag corresponding to the encrypted descriptive event data, wherein the persistent tag is UCDSID code;
storing the encrypted descriptive event data with the UCDSID code to the data object list.
8. The method of claim 6 or 7, wherein the first resource pool comprises synchronization configuration information comprising at least one of: data synchronization time, synchronization operation type and synchronization mode;
The sending a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located, including:
generating a synchronous instruction according to description event data with a lasting label in the data object list;
and sending a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located according to the synchronization configuration information.
9. The method of claim 8, wherein before sending a data synchronization instruction to a configuration data center service of at least one second resource pool in a peer-to-peer network where the first resource pool is located according to the synchronization configuration information, the method further comprises:
receiving a GRPC request sent by a configuration data center service of at least one second resource pool in the peer-to-peer network;
the sending a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located according to the synchronization configuration information, including:
and sending a data synchronization instruction to a configuration data center service of at least one second resource pool in the peer-to-peer network where the first resource pool is located according to the synchronization configuration information in a broadcasting mode.
10. A data processing system, comprising: configuring a resource center and at least one resource pool, each of the at least one resource pool comprising a configuration data center service; wherein, the liquid crystal display device comprises a liquid crystal display device,
the configuration resource center is used for receiving a data synchronization request and generating service configuration information based on the data synchronization request, wherein the service configuration information comprises a target plug-in identifier corresponding to the data synchronization request; and transmitting the service configuration information to the at least one resource pool, the at least one resource pool including a first resource pool;
the first resource pool is used for acquiring a target data configuration plugin corresponding to the target plugin identifier according to the plugin identifier and the associated information of the data configuration plugin under the condition of receiving the service configuration information sent by the configuration resource center; determining configuration relation data to be synchronized corresponding to the service configuration information through the target data configuration plug-in; transmitting a data synchronization instruction to a configuration data center service of at least one second resource pool in a peer-to-peer network where the first resource pool is located;
the data synchronization instruction comprises description event data, and the data synchronization instruction is used for indicating configuration data center service of the second resource pool to update the description event data, wherein the description event data is obtained by carrying out content serialization processing on the configuration relationship data to be synchronized.
11. A data processing apparatus, the apparatus comprising:
the receiving module is used for receiving service configuration information sent by a configuration resource center, wherein the service configuration information comprises a target plug-in identification, and the service configuration information is determined by a data synchronization request received by the configuration resource center;
the acquisition module is used for acquiring a target data configuration plugin corresponding to the target plugin identifier according to the plugin identifier and the associated information of the data configuration plugin;
the determining module is used for determining to-be-synchronized configuration relation data corresponding to the service configuration information through the target data configuration plug-in;
a sending module, configured to send a data synchronization instruction to a configuration data center service of at least one second resource pool in a peer-to-peer network where the first resource pool is located; the data synchronization instruction comprises description event data, and the data synchronization instruction is used for indicating configuration data center service of the second resource pool to update the description event data, wherein the description event data is obtained by carrying out content serialization processing on the configuration relationship data to be synchronized.
12. A computer device, the computing device comprising: a processor and a memory storing computer program instructions;
The processor, when executing the computer program instructions, implements a data processing method as claimed in any one of claims 1-9.
13. A storage medium having stored thereon computer program instructions which, when executed by a processor, implement the data processing method of any of claims 1-9.
CN202310150640.XA 2023-02-16 2023-02-16 Data processing method, device, computer equipment and storage medium Pending CN116150273A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310150640.XA CN116150273A (en) 2023-02-16 2023-02-16 Data processing method, device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310150640.XA CN116150273A (en) 2023-02-16 2023-02-16 Data processing method, device, computer equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116150273A true CN116150273A (en) 2023-05-23

Family

ID=86355994

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310150640.XA Pending CN116150273A (en) 2023-02-16 2023-02-16 Data processing method, device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116150273A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117041044A (en) * 2023-08-23 2023-11-10 中电金信软件有限公司 Gateway configuration method, device and system

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117041044A (en) * 2023-08-23 2023-11-10 中电金信软件有限公司 Gateway configuration method, device and system

Similar Documents

Publication Publication Date Title
US11374826B2 (en) Systems and methods for enhanced monitoring of a distributed computing system
CN107431651B (en) Life cycle management method and equipment for network service
CN108228814B (en) Data synchronization method and device
RU2683630C2 (en) Method for update of nsd network service descriptor and device
CN107800565B (en) Inspection method, inspection device, inspection system, computer equipment and storage medium
WO2018130902A1 (en) Bulk creation of managed functions in a network that includes virtualized network function
CN110166562B (en) Data synchronization method and device, storage medium and electronic equipment
CN110289965B (en) Application program service management method and device
CN110677453A (en) ZooKeeper-based distributed lock service implementation method, device, equipment and storage medium
CN111510326A (en) Master-slave equipment configuration information synchronization method, system, terminal equipment and storage medium
CN116150273A (en) Data processing method, device, computer equipment and storage medium
CN115562757A (en) Data processing method, configuration center system, electronic device, and storage medium
CN108733477B (en) Method, device and equipment for data clustering processing
CN110798358B (en) Distributed service identification method and device, computer readable medium and electronic equipment
CN108370329B (en) Management method and device for management function object
CN112448909A (en) Electronic lock management method, device, system and storage medium
CN115756877A (en) Resource synchronization method, device, equipment and storage medium
US20230048343A1 (en) Correlation of a virtual machine to a host within a virtual domain
CN109542841A (en) The method and terminal device of data snapshot are created in cluster
CN112416980B (en) Data service processing method, device and equipment
CN115277707A (en) Service processing method, device, electronic equipment and storage medium
CN114020453A (en) Service bus message processing method, service bus system, electronic device, and medium
CN109472124B (en) Method, device, equipment and medium for matching interface use permission
CN110935168A (en) Distributed ID generation method, device and equipment for global uniform architecture
CN112579247A (en) Method and device for determining task state

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination