CN111046057A - Data processing method and device for server cluster, computer equipment and medium - Google Patents
Data processing method and device for server cluster, computer equipment and medium Download PDFInfo
- Publication number
- CN111046057A CN111046057A CN201911373509.XA CN201911373509A CN111046057A CN 111046057 A CN111046057 A CN 111046057A CN 201911373509 A CN201911373509 A CN 201911373509A CN 111046057 A CN111046057 A CN 111046057A
- Authority
- CN
- China
- Prior art keywords
- sequence
- self
- increment
- identification data
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2272—Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure provides a data processing method for a server cluster, including: an identification request about specified data from a first service deployed in a server cluster is received, and identification data for the identification request is generated. Wherein the identification data comprises: the system comprises a time identifier, a first self-increment sequence and a second self-increment sequence, wherein the first self-increment sequence is generated based on the deployment information of a first service in a server cluster. The generated identification data is then sent to the first service for storing the specified data in the distributed database by the first service based on the identification data. The present disclosure also provides a data processing apparatus of a server cluster, a computer device and a computer readable storage medium.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and apparatus for a server cluster, a computer device, and a medium.
Background
In the prior art, an identification data scheme of a distributed system is generated, and the identification data includes a timestamp, a machine code, and a serial number. For services deployed in the same machine/instance, when time adjustment, clock callback and other situations occur, the same timestamp may be generated, so that the generated identification data has a repeated risk and cannot meet the data processing requirements of a distributed system.
Disclosure of Invention
In view of the above, the present disclosure provides a data processing method, apparatus, computer device, and medium for a server cluster.
One aspect of the present disclosure provides a data processing method for a server cluster, including: an identification request about specified data from a first service deployed in a server cluster is received, and identification data for the identification request is generated. Wherein the identification data comprises: the system comprises a time identifier, a first self-increment sequence and a second self-increment sequence, wherein the first self-increment sequence is generated based on the deployment information of a first service in a server cluster. The generated identification data is then sent to the first service for storing the specified data in the distributed database by the first service based on the identification data.
According to an embodiment of the present disclosure, the generating identification data for the identification request includes: in one aspect, a time stamp is generated based on time information of the received stamp request. On the other hand, deployment information of the first service in the server cluster is obtained, and then a first self-increment sequence is generated based on the deployment information. Wherein the deployment information comprises: deployment path and machine IP address. Next, a second incremental sequence is generated based on the time stamp, the first incremental sequence, and the amount of generated stamp data. The time stamp, the first incrementing sequence, and the second incrementing sequence are combined to obtain the stamp data.
According to an embodiment of the present disclosure, the generating a first increasing sequence based on the deployment information includes: a primary key for the first service is generated based on a deployment path of the first service in the server cluster and the machine IP address. Then, it is determined whether a sequence value for the primary key exists in the first mapping table. If so, the sequence value is taken as a first self-increasing sequence. If not, acquiring the used maximum first auto-increment value, and generating a first auto-increment sequence based on the used maximum first auto-increment value. Wherein, the first mapping table includes: mapping from primary key to sequence value.
According to an embodiment of the present disclosure, the generating the first self-increment sequence based on the maximum first self-increment value used includes: adding 1 to the maximum first auto-increment value to obtain an updated maximum first auto-increment value. Then, it is determined whether a primary key of a maximum first self-increment value for the update exists in the second mapping table. If so, the above operations are repeatedly performed. And if not, taking the updated maximum first self-increment value as a first self-increment sequence, and recording the mapping relation between the primary key and the first self-increment sequence in the first mapping table and the second mapping table. Wherein the second mapping table includes: mapping from sequence values to primary keys.
According to an embodiment of the present disclosure, the generating the second incrementing sequence based on the time identification, the first incrementing sequence, and the number of generated identification data includes: and acquiring a used maximum second auto-increment value aiming at the time identifier and the first auto-increment sequence, and then adding 1 to the maximum second auto-increment value to obtain a second auto-increment sequence.
According to an embodiment of the present disclosure, the method further includes: after the identification data is sent to the first service, an exception message is received from the first service regarding the identification data. And deleting the mapping relation between the primary key and the first self-increment sequence aiming at the identification data from the first mapping table and the second mapping table. Then, a used maximum first auto-increment value is obtained, and an updated first auto-increment sequence is generated based on the used maximum first auto-increment value.
According to an embodiment of the present disclosure, the generating an updated first auto-increment sequence based on the maximum first auto-increment value already used includes: adding 1 to the maximum first auto-increment value to obtain an updated maximum first auto-increment value. It is determined whether a primary key of a maximum first self-increment value exists for the update in the second mapping table. If so, the above operations are repeatedly performed. And if not, taking the updated maximum first self-increment value as an updated first self-increment sequence, and recording the mapping relation between the primary key and the updated first self-increment sequence in the first mapping table and the second mapping table. Wherein the second mapping table includes: mapping from sequence values to primary keys.
According to an embodiment of the present disclosure, the identification data further comprises a sign bit identification. The combining the time stamp, the first self-increment sequence and the second self-increment sequence to obtain the stamp data includes: and splicing the symbol bit identifier, the time identifier, the first self-increment sequence and the second self-increment sequence in sequence to obtain identifier data.
According to an embodiment of the present disclosure, the method further includes: for any historical unit of time, an identification data capacity for that historical unit of time is determined. Unused identification data for the any one of the historical units of time is determined based on the identification data capacity and the identification data actually generated within the any one of the historical units of time. The unused identification data is then cached.
According to an embodiment of the present disclosure, the generating the identification data for the identification request includes: one unused identification data is selected from the cached unused identification data as the identification data.
Another aspect of the present disclosure provides a data processing apparatus of a server cluster, including: the device comprises a receiving module, a generating module and a sending module. The receiving module is used for receiving an identification request about specified data from a first service deployed in a server cluster. The generation module is to generate identification data for the identification request. Wherein the identification data comprises: the system comprises a time identifier, a first self-increment sequence and a second self-increment sequence, wherein the first self-increment sequence is generated based on the deployment information of a first service in a server cluster. The sending module is used for sending the identification data to the first service so that the first service stores the specified data into the distributed database based on the identification data.
Another aspect of the present disclosure provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method as described above when executing the program.
Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method as described above when executed.
Another aspect of the disclosure provides a computer program comprising computer executable instructions for implementing the method as described above when executed.
According to the embodiment of the disclosure, when any service in the server cluster updates the data of the distributed database, the identification data is acquired from the shared storage space, and then the data updating operation is performed on the distributed database based on the acquired identification data, so that each data updating operation received by the distributed database corresponds to one identification data, the data stored in the distributed database can be globally and uniquely identified and indexed, and the data in the distributed database can be conveniently searched, used, managed and processed through the identification parameters corresponding to the data updating operations. The identification data corresponding to each data comprises a time identification, a first self-increment sequence and a second self-increment sequence, and the time identification, the first self-increment sequence and the second self-increment sequence ensure the global uniqueness of the identification data corresponding to any data in the identification data generated by the server cluster. Even if the same time identification is generated due to time adjustment and clock callback, the dual change of the first self-increment sequence and the second self-increment sequence has enough margin to ensure the global uniqueness of the identification data, so that the ordered and unique identification data is provided for data configuration, and the subsequent service processing can be efficiently, accurately, quickly and flexibly performed.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
fig. 1 schematically illustrates an exemplary system architecture of a data processing method and apparatus for a cluster of application servers according to an embodiment of the present disclosure;
FIG. 2 schematically illustrates an example data structure of identification data generated by one process;
FIG. 3 schematically shows a flow chart of a method of data processing of a server cluster according to an embodiment of the present disclosure;
FIG. 4A schematically illustrates an example data structure of identification data according to an embodiment of the present disclosure;
FIG. 4B schematically illustrates an example flow diagram of a data processing process for a cluster of servers in accordance with an embodiment of this disclosure;
FIG. 5 schematically shows a block diagram of a data processing apparatus of a server cluster according to an embodiment of the present disclosure; and
FIG. 6 schematically shows a block diagram of a computer device according to an embodiment of the disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
The embodiment of the disclosure provides a data processing method and device for a server cluster. The method may include a receiving process, a generating process, and a transmitting process. In a receiving process, an identification request for specified data from a first service deployed on a cluster of servers is received. Then, a generation process is performed to generate identification data for the identification request. Wherein the identification data comprises: the system comprises a time identifier, a first self-increment sequence and a second self-increment sequence, wherein the first self-increment sequence is generated based on the deployment information of a first service in a server cluster. Then, a sending process is performed to send the generated identification data to the first service, so that the first service stores the specified data into the distributed database based on the identification data.
Fig. 1 schematically illustrates an exemplary system architecture 100 in which data processing methods and apparatus for a cluster of servers may be applied, according to an embodiment of the disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.
As shown in fig. 1, the System architecture 100 according to the embodiment shows a structure of a Distributed Database System, where the Distributed Database System (DDBMS) includes servers 110, 120, and 130, each of which may deploy one or more services, each of which has its own Database Management System (DMS) corresponding to different databases 111, 121, and 131, respectively, the Database Management System is a large-scale software for managing and managing databases, which is used to build, use, and maintain the databases and performs unified Management and control on the databases to ensure security and integrity of the databases, users access data in the databases through the DBMS, and Database administrators perform maintenance work on the databases through the DBMS. It enables multiple applications and users to build, modify and query databases in different ways, either at the same time or at different times. The three servers 110, 120 and 130 are connected via a network 140, each server has its own client (not shown), and a user can perform some operations on the database in the server corresponding to the client through the client, and can also perform some operations on the database in other servers in the distributed database system through the client.
The data processing method and device for the server cluster provided by the embodiment of the disclosure can be applied to the distributed database system shown in fig. 1, and perform an improved data updating operation on the distributed database system shown in fig. 1, so that the data stored in the distributed database has globally unique identification and index.
It should be understood that the number of servers and databases in fig. 1 is merely illustrative. There may be any number of servers and databases, as desired for implementation.
In one approach, the identification data applicable to the distributed system is generated using, for example, a snowflake (snowflake) algorithm. FIG. 2 schematically illustrates an example data structure of identification data generated by one process. As shown in fig. 2, from left to right, the 1 st bit is the sign bit. The next 41 bits are the time stamp, which is used to hold millisecond time. Then 10 bits of machine code, each node in the server cluster has a fixed machine code, in this example, 10 bits long machine code supports deployment of 1024 nodes at most. The last 12 bits are the self-incrementing sequence number, which characterizes the count in milliseconds. In theory each node can produce 409.6 million identification data per millisecond.
In such a processing method, if time adjustment, clock callback, or the like is encountered, the same timestamp may be generated, and if the 10-bit machine code is the same (for example, if the service is deployed in the same machine/instance), the generated identification data has the same self-increment serial number, and the identification data may be duplicated, which may not satisfy the data processing requirements of the distributed system.
According to the embodiments of the present disclosure, a data processing method for a server cluster is provided, which can cope with special situations such as time adjustment, and the method is described below with an illustration. It should be noted that the sequence numbers of the respective operations in the following methods are merely used as representations of the operations for description, and should not be construed as representing the execution order of the respective operations. The method need not be performed in the exact order shown, unless explicitly stated.
Fig. 3 schematically shows a flow chart of a data processing method of a server cluster according to an embodiment of the present disclosure. The method can be applied to a shared storage space of a server cluster, when one or more services in the server cluster need to perform data updating operation on a distributed database, an identification request can be sent to the shared storage space, and the shared storage space responds to the identification request to generate and feed back globally unique identification data.
As shown in fig. 3, the method may include operations S301 to S303 as follows.
In operation S301, an identification request for specified data from a first service deployed in a server cluster is received.
The first service may be any service deployed in the server cluster. Illustratively, the designated data may be, for example, business data, the business data is data related to a specific business project, and an important part in the business data processing process is to generate identification data corresponding to the business data, and then store the business data into a database by using the identification data as an index, and on the basis, business processing such as backtracking search, problem location, statistical analysis and the like may be subsequently performed on the business data to assist smooth performance of the corresponding business project. In this operation S301, when any service needs to acquire identification data about specified data, it needs to send an identification request to the shared storage space.
Then, in operation S302, identification data for the identification request is generated.
The identification data generated in operation S302 includes a time identification, a first incrementing sequence, and a second incrementing sequence. Wherein the time identifier is used for characterizing the time of generating the identification data or for characterizing the time of receiving the identification request. The first self-increment sequence is generated based on the deployment information of the first service in the server cluster and is used for characterizing the deployment characteristics of the first service. Due to the adoption of the mode of the self-increment sequence, the same service can correspond to different first self-increment sequences under different conditions, and compared with a fixed machine code, the change margin of the identification data is increased. The second incrementing sequence is used to characterize the count of identification data for the same service at the same time period.
Next, in operation S303, the generated identification data is transmitted to the first service to store the specified data in the distributed database by the first service based on the identification data.
As can be understood by those skilled in the art, according to the data processing method of the server cluster disclosed by the present disclosure, when any service in the server cluster updates data of the distributed database, the identification data is obtained from the shared storage space, and then the data updating operation is performed on the distributed database based on the obtained identification data, so that each data updating operation received by the distributed database corresponds to one identification data, so that the data stored in the distributed database can be globally and uniquely identified and indexed, and the data in the distributed database can be conveniently searched, used, managed and processed through the identification parameter corresponding to each data updating operation. The identification data corresponding to each data comprises a time identification, a first self-increment sequence and a second self-increment sequence, and the time identification, the first self-increment sequence and the second self-increment sequence ensure the global uniqueness of the identification data corresponding to any data in the identification data generated by the server cluster. Even if the same time identification is generated due to time adjustment and clock callback, the dual change of the first self-increment sequence and the second self-increment sequence has enough margin to ensure the global uniqueness of the identification data, so that the ordered and unique identification data is provided for data configuration, and the subsequent service processing can be efficiently, accurately, quickly and flexibly performed.
According to an embodiment of the present disclosure, the generating of the identification data for the identification request may include: and respectively generating a time identifier, a first self-increment sequence and a second self-increment sequence. The time stamp, the first incrementing sequence, and the second incrementing sequence are then combined to obtain the stamp data.
The process of generating the time stamp may include, for example: a time stamp is generated based on the time information of the received stamp request. The process of generating the first increasing sequence may include, for example: the method comprises the steps of obtaining deployment information of a first service in a server cluster, and then generating a first self-increment sequence based on the deployment information. Illustratively, the deployment information includes: a deployment path and a machine IP (internet Protocol) address. The deployment path may be a deployment path of the first service in the server cluster, and the machine IP address may be an IP address of a machine where the first service is deployed. The process of generating the second increasing sequence may include, for example: a second incremental sequence is generated based on the time identification, the first incremental sequence, and the amount of generated identification data.
The process of generating the first incrementing sequence is exemplified below.
For example, the shared memory space may be configured with a first mapping table and a second mapping table. The first mapping table includes a mapping from a primary key to a sequence value, and the second mapping table includes a mapping from a sequence value to a primary key.
On this basis, the process of generating the first self-increment sequence based on the deployment information may include: first, a primary key for a first service is generated based on a deployment path of the first service in a server cluster and a machine IP address. For example, the deployment path and the machine IP address may be spliced in a certain order to obtain a primary key for the first service, or the deployment path and/or the machine IP address may be operated by a predetermined algorithm (e.g., a hash algorithm, etc., which is not limited herein), and then the operation result is spliced to obtain the primary key for the first service.
Then, it is determined whether a sequence value for the primary key exists in the first mapping table. If so, indicating that a first self-increment sequence already exists for the first service, the sequence value is taken as the first self-increment sequence to continue to be used. If not, the first self-increment sequence aiming at the first service does not exist currently, the maximum first self-increment value currently used by the shared storage space is obtained, and the first self-increment sequence is generated based on the maximum first self-increment value used.
For example, the above-mentioned process of generating the first auto-increment sequence based on the maximum first auto-increment value used may be performed as follows: adding 1 to the maximum first auto-increment value to obtain an updated maximum first auto-increment value. Then, it is determined whether a primary key of a maximum first self-increment value for the update exists in the second mapping table. If so, the above operations are repeated until the updated maximum first self-increment value that is not currently used is found. If not, the updated maximum first self-increment value is used as a first self-increment sequence, and the mapping relation between the main key and the first self-increment sequence is recorded in the first mapping table and the second mapping table.
It will be appreciated that the above-described process of checking in the second mapping table may avoid reuse for the same auto-increment value. For example, the self-increment value of the shared storage space is cyclically valued in an interval of 1-1024, and when the current maximum first self-increment value is 1024, 1 is added to the maximum first sub-increment value to obtain an updated maximum first self-increment value of 1. The updated maximum first increment value is likely to be used, so that a search confirmation in the second mapping table is required until the updated maximum first increment value of the unrecorded mapping relation is found as the first increment sequence.
After generating the time stamp and the first incrementing sequence, according to an embodiment of the present disclosure, the above process of generating the second incrementing sequence based on the time stamp, the first incrementing sequence, and the number of generated stamp data may include: and acquiring a used maximum second auto-increment value aiming at the time identifier and the first auto-increment sequence, and then adding 1 to the maximum second auto-increment value to obtain a second auto-increment sequence.
After the identification data is generated and fed back, the first service performs storage processing on the specified data based on the identification data, and needs to check whether historical identification data identical to the identification data already exists in the distributed database, so as to avoid damage to global uniqueness of the identification data in the distributed database caused by some abnormal factors. And when the same historical identification data exists, sending an exception message about the identification data to the shared storage space so that the shared storage space updates the identification data. According to an embodiment of the present disclosure, the method may further include: after the identification data is sent to the first service, an exception message is received from the first service regarding the identification data. And deleting the mapping relation between the primary key and the first self-increment sequence aiming at the identification data from the first mapping table and the second mapping table. Then, the maximum first auto-increment value currently used by the shared storage space is obtained, and an updated first auto-increment sequence is generated based on the maximum first auto-increment value used.
Illustratively, the above process of generating the updated first self-increment sequence based on the used maximum first self-increment value may include: adding 1 to the maximum first auto-increment value to obtain an updated maximum first auto-increment value. It is determined whether a primary key of a maximum first self-increment value exists for the update in the second mapping table. If so, the above operations are repeatedly performed. And if not, taking the updated maximum first self-increment value as an updated first self-increment sequence, and recording the mapping relation between the primary key and the updated first self-increment sequence in the first mapping table and the second mapping table. Wherein the second mapping table includes: mapping from sequence values to primary keys. The process is the same as the process for generating the first self-increment sequence, and is described in detail above, and is not described herein again.
According to an embodiment of the present disclosure, the identification data further comprises a sign bit identification. The combining the time stamp, the first self-increment sequence and the second self-increment sequence to obtain the stamp data includes: and splicing the symbol bit identifier, the time identifier, the first self-increment sequence and the second self-increment sequence in sequence to obtain identifier data.
According to an embodiment of the present disclosure, the method further includes: for any historical unit of time, an identification data capacity for that historical unit of time is determined. Unused identification data for the any one of the historical units of time is determined based on the identification data capacity and the identification data actually generated within the any one of the historical units of time. The unused identification data is then cached. On this basis, the generating of the identification data for the identification request includes: one unused identification data is selected from the cached unused identification data as the identification data.
The above embodiments are exemplified with reference to fig. 4A and 4B in conjunction with specific examples.
Fig. 4A schematically illustrates an example data structure of identification data according to an embodiment of the present disclosure.
As shown in fig. 4A, the identification data may include, from left to right, a sign bit, a time identification, a first incrementing sequence, and a second incrementing sequence. In the example shown in fig. 4A, the sign bit takes 1 bit, and the time stamp takes 41 bits, which can be used to store millisecond time. The shared storage space for generating the annotation data is, for example, a Redis cluster, and the first self-increment sequence is, for example, a Redis self-increment sequence, and occupies 10 bits. The second incrementing sequence takes 12 bits. In other examples, the sequence relationship of each part in the identification data, the data length of each part, and the like may be set according to actual needs, and are not limited herein.
In a server cluster, when any service needs to perform an update operation for a distributed database, an identification request may be sent to a Redis cluster. In response to the identification request, the Redis cluster may generate identification data as shown in FIG. 4A. In generating the identification data, unlike the related art in which a fixed machine code is generated for a service, the data processing method according to the embodiment of the present disclosure needs to generate a first self-increment sequence.
Fig. 4B schematically shows an example flow diagram of a data processing procedure of a server cluster according to an embodiment of the present disclosure.
In the example shown in fig. 4B, first, logical writing of the process of generating identification data is performed through the InitializingBean interface that implements the Spring framework. For example, an afterpropertieset method implementing initializibbean is called when a Spring container is initialized at server startup, and in this method, generation logic writing of identification data may be performed.
The following operations S401 to S407 are then performed.
In operation S401, a deployment path and a machine IP address of a service are acquired.
Illustratively, the deployment path of the service is obtained by a method system. Getlocalhost () is used to obtain the machine IP address of the server where the service is located.
In operation S402, a primary KEY is generated according to the currently acquired deployment path and IP address of the service.
In operation S403, it is queried in the first mapping table of the Redis cluster whether a value corresponding to KEY exists, where the value is the sequence value in the foregoing. If not, operation S404 is performed. If so, operation S407 is performed.
In operation S404, a maximum auto-increment value X that is currently used in the Redis cluster is obtained, and the X is incremented by 1 to obtain an updated maximum auto-increment value X'.
In operation S405, it is queried in a second mapping table of the Redis cluster whether a primary key corresponding to X' exists. If not, operation S406 is performed. If so, operation S404 is repeatedly performed.
In operation S406, with X ' as a first self-increment sequence, a mapping relationship from KEY to X ' is constructed in a first mapping table, and a mapping relationship from X ' to KEY is constructed in a second mapping table.
In operation S407, the value continues to be used as the first auto-increment sequence.
Identification data, which may be referred to as a distributed id (identification), may thus be generated as shown in fig. 4A based on the first and second augmented sequences.
The incremental values in the above description can be circularly valued in increments in 1-1024 intervals by using a Redis incremental sequence.
It is understood that in order to avoid operation conflicts, it should be ensured that the above operations are performed within one atomicity in the above process. For example, a Redis cluster may use setnx distributed locks and lua scripts to guarantee sequential execution of multiple servers. In addition, in the above process, the first mapping table and the second mapping table may be stored in various types of storage spaces such as Redis, mysql, zookeeper, and the like, which is not limited herein.
Further, based on the data processing method shown in fig. 4B, when a clock dial-back situation occurs, 4096 pieces of identification data can be generated per unit time (for example, 1 millisecond) in the example shown in fig. 4A to 4B, and if the generated identification data is not repeated at the time of warehousing, it can be continuously used. If a conflict of the identification data occurs, the exception may be captured, the mapping keys (IP + path, new self-increment value) and (new self-increment value, IP + path) used in the first mapping table and the second mapping table at this time are deleted, and the above operation S404 is continuously performed, so as to change the first self-increment sequence in the identification data and start generating a new distributed ID.
In another embodiment of the present disclosure, for a case where the clock callback is particularly large, the distributed ID may be stored in a cache after being generated for the distributed ID that is not used per unit time in the morning of each day or in a time period when the traffic is relatively small, and one day or a sufficient amount of the distributed ID may be cached for use by the system in a special case by observing the usage of the distributed ID per day.
It can be understood that according to the data processing method of the server cluster disclosed by the embodiment of the disclosure, the situation of distributed ID repetition can be rapidly processed.
Fig. 5 schematically shows a block diagram of a data processing device of a server cluster according to an embodiment of the present disclosure.
As shown in fig. 5, the data processing apparatus 500 of the server cluster may include: a receiving module 510, a generating module 520, and a transmitting module 530.
The receiving module 510 is configured to receive an identification request regarding specified data from a first service deployed on a server cluster.
The generation module 520 is used to generate identification data for the identification request. Wherein the identification data comprises: the system comprises a time identifier, a first self-increment sequence and a second self-increment sequence, wherein the first self-increment sequence is generated based on the deployment information of a first service in a server cluster.
The sending module 530 is configured to send the identification data to the first service for storing the specified data in the distributed database by the first service based on the identification data.
It should be noted that the implementation, solved technical problems, implemented functions, and achieved technical effects of each module/unit/subunit and the like in the apparatus part embodiment are respectively the same as or similar to the implementation, solved technical problems, implemented functions, and achieved technical effects of each corresponding step in the method part embodiment, and are not described herein again.
Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.
For example, any of the receiving module 510, the generating module 520, and the transmitting module 530 may be combined and implemented in one module, or any one of them may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the receiving module 510, the generating module 520, and the transmitting module 530 may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware by any other reasonable manner of integrating or packaging a circuit, or in any one of three implementations of software, hardware, and firmware, or in a suitable combination of any of them. Alternatively, at least one of the receiving module 510, the generating module 520 and the transmitting module 530 may be at least partially implemented as a computer program module, which when executed may perform a corresponding function.
Fig. 6 schematically shows a block diagram of a computer device adapted to implement the above described method according to an embodiment of the present disclosure. The computer device shown in fig. 6 is only an example and should not bring any limitation to the function and scope of use of the embodiments of the present disclosure.
As shown in fig. 6, a computer apparatus 600 according to an embodiment of the present disclosure includes a processor 601 which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. Processor 601 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 601 may also include onboard memory for caching purposes. Processor 601 may include a single processing unit or multiple processing units for performing different actions of a method flow according to embodiments of the disclosure.
In the RAM 603, various programs and data necessary for the operation of the apparatus 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 602 and/or RAM 603. It is to be noted that the programs may also be stored in one or more memories other than the ROM 602 and RAM 603. The processor 601 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to an embodiment of the present disclosure, device 600 may also include an input/output (I/O) interface 605, input/output (I/O) interface 605 also being connected to bus 604. The device 600 may also include one or more of the following components connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program, when executed by the processor 601, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 602 and/or RAM 603 described above and/or one or more memories other than the ROM 602 and RAM 603.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.
The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.
Claims (13)
1. A data processing method of a server cluster comprises the following steps:
receiving an identification request about specified data from a first service deployed on the server cluster;
generating identification data for the identification request, wherein the identification data comprises: the first self-increment sequence is generated based on the deployment information of the first service in the server cluster; and
sending the identification data to the first service for storing the specified data in a distributed database by the first service based on the identification data.
2. The method of claim 1, wherein the generating identification data for the identification request comprises:
generating the time identifier based on the time information of the identifier request;
obtaining the deployment information of the first service in the server cluster, wherein the deployment information includes: a deployment path and a machine IP address;
generating the first self-increment sequence based on the deployment information;
generating the second increasing sequence based on the time identification, the first increasing sequence, and the amount of identification data that has been generated; and
combining the time stamp, the first incremental sequence, and the second incremental sequence to obtain the stamp data.
3. The method of claim 2, wherein the generating the first increasing sequence based on the deployment information comprises:
generating a primary key for the first service based on the deployment path and the machine IP address;
determining whether a sequence value for the primary key exists in a first mapping table, the first mapping table comprising: mapping relation from the primary key to the sequence value;
if so, taking the sequence value as the first self-increment sequence;
if not, acquiring the used maximum first self-increment value; and
generating the first auto-increment sequence based on the used maximum first auto-increment value.
4. The method of claim 3, wherein the generating the first self-increment sequence based on the used maximum first self-increment value comprises:
adding 1 to the maximum first auto-increment value to obtain an updated maximum first auto-increment value;
determining whether a primary key for the updated maximum first self-increment value exists in a second mapping table, the second mapping table comprising: a mapping from sequence values to primary keys;
if yes, repeatedly executing the operation; and
and if not, taking the updated maximum first self-increment value as the first self-increment sequence, and recording the mapping relation between the primary key and the first self-increment sequence in the first mapping table and the second mapping table.
5. The method of claim 2, wherein the generating the second increasing sequence based on the time identification, the first increasing sequence, and the amount of identification data that has been generated comprises:
obtaining a used maximum second auto-increment value for the time identification and the first auto-increment sequence; and
adding 1 to the maximum second auto-increment value to obtain the second auto-increment sequence.
6. The method of claim 1, further comprising:
after said sending said identification data to said first service, receiving an exception message from said first service regarding said identification data;
deleting the mapping relation between the primary key and the first self-increment sequence aiming at the identification data from a first mapping table and a second mapping table;
acquiring a used maximum first self-increment value;
generating an updated first auto-increment sequence based on the used maximum first auto-increment value.
7. The method of claim 6, wherein the generating an updated first self-increment sequence based on the used maximum first self-increment value comprises:
adding 1 to the maximum first auto-increment value to obtain an updated maximum first auto-increment value;
determining whether a primary key for the updated maximum first self-increment value exists in a second mapping table, the second mapping table comprising: a mapping from sequence values to primary keys;
if yes, repeatedly executing the operation; and
if not, taking the updated maximum first self-increment value as the updated first self-increment sequence, and recording the mapping relation between the primary key and the updated first self-increment sequence in the first mapping table and the second mapping table.
8. The method of claim 2, wherein the identification data further comprises a sign bit identification;
the combining the time stamp, the first incrementing sequence, and the second incrementing sequence to obtain the stamp data includes: and sequentially splicing the symbol bit identifier, the time identifier, the first self-increment sequence and the second self-increment sequence to obtain the identifier data.
9. The method of claim 1, further comprising:
for any historical unit time, determining an identification data capacity for the any historical unit time;
determining unused identification data for the any historical unit time based on the identification data capacity and the identification data actually generated in the any historical unit time; and
caching the unused identification data.
10. The method of claim 9, wherein the generating identification data for the identification request comprises:
selecting one unused identification data from the cached unused identification data as the identification data.
11. A data processing apparatus of a server cluster, comprising:
a receiving module, configured to receive an identification request regarding specified data from a first service deployed in the server cluster;
a generating module, configured to generate identification data for the identification request, where the identification data includes: the first self-increment sequence is generated based on the deployment information of the first service in the server cluster; and
a sending module, configured to send the identification data to the first service, so that the first service stores the specified data in a distributed database based on the identification data.
12. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor when executing the program implementing:
the method of any one of claims 1 to 10.
13. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform:
the method of any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911373509.XA CN111046057A (en) | 2019-12-26 | 2019-12-26 | Data processing method and device for server cluster, computer equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911373509.XA CN111046057A (en) | 2019-12-26 | 2019-12-26 | Data processing method and device for server cluster, computer equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111046057A true CN111046057A (en) | 2020-04-21 |
Family
ID=70240476
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911373509.XA Pending CN111046057A (en) | 2019-12-26 | 2019-12-26 | Data processing method and device for server cluster, computer equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111046057A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111651459A (en) * | 2020-06-08 | 2020-09-11 | 北京首汽智行科技有限公司 | Service ID generation method based on Redis |
| CN111694792A (en) * | 2020-05-29 | 2020-09-22 | 中国建设银行股份有限公司 | Identification generation method and device based on snowfly |
| CN111694845A (en) * | 2020-05-29 | 2020-09-22 | 中国建设银行股份有限公司 | Redis-based identifier generation method and device |
| CN111708775A (en) * | 2020-05-21 | 2020-09-25 | 四川虹美智能科技有限公司 | Self-adding ID generation method, device and system |
| CN111813559A (en) * | 2020-07-23 | 2020-10-23 | 济南大陆机电股份有限公司 | Industrial Internet distributed identification coding method and system |
| CN111953772A (en) * | 2020-08-11 | 2020-11-17 | 北京达佳互联信息技术有限公司 | Request processing method, device, server and storage medium |
| CN113220682A (en) * | 2021-05-08 | 2021-08-06 | 北京猿力未来科技有限公司 | Business data analysis method, data processing method, data analysis system and storage medium |
| CN113297327A (en) * | 2021-05-24 | 2021-08-24 | 建信金融科技有限责任公司 | System and method for generating distributed ID |
| CN113360172A (en) * | 2021-06-30 | 2021-09-07 | 平安普惠企业管理有限公司 | Application deployment method and device, computer equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020268A (en) * | 2012-12-26 | 2013-04-03 | 大唐软件技术股份有限公司 | Serial number applying method and system of relational database |
| CN108984639A (en) * | 2018-06-22 | 2018-12-11 | 联想(北京)有限公司 | The data processing method and device of server cluster |
| CN109344162A (en) * | 2018-09-14 | 2019-02-15 | 北京京东金融科技控股有限公司 | Business data processing method and device |
| CN109697133A (en) * | 2018-12-28 | 2019-04-30 | 广州华多网络科技有限公司 | ID generation method, apparatus and system |
| CN109951541A (en) * | 2019-03-11 | 2019-06-28 | 中国银联股份有限公司 | A kind of serial number generation method and server |
| CN110263565A (en) * | 2019-06-28 | 2019-09-20 | 北京百度网讯科技有限公司 | Method and apparatus for calling service |
-
2019
- 2019-12-26 CN CN201911373509.XA patent/CN111046057A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020268A (en) * | 2012-12-26 | 2013-04-03 | 大唐软件技术股份有限公司 | Serial number applying method and system of relational database |
| CN108984639A (en) * | 2018-06-22 | 2018-12-11 | 联想(北京)有限公司 | The data processing method and device of server cluster |
| CN109344162A (en) * | 2018-09-14 | 2019-02-15 | 北京京东金融科技控股有限公司 | Business data processing method and device |
| CN109697133A (en) * | 2018-12-28 | 2019-04-30 | 广州华多网络科技有限公司 | ID generation method, apparatus and system |
| CN109951541A (en) * | 2019-03-11 | 2019-06-28 | 中国银联股份有限公司 | A kind of serial number generation method and server |
| CN110263565A (en) * | 2019-06-28 | 2019-09-20 | 北京百度网讯科技有限公司 | Method and apparatus for calling service |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111708775A (en) * | 2020-05-21 | 2020-09-25 | 四川虹美智能科技有限公司 | Self-adding ID generation method, device and system |
| CN111694792A (en) * | 2020-05-29 | 2020-09-22 | 中国建设银行股份有限公司 | Identification generation method and device based on snowfly |
| CN111694845A (en) * | 2020-05-29 | 2020-09-22 | 中国建设银行股份有限公司 | Redis-based identifier generation method and device |
| CN111651459A (en) * | 2020-06-08 | 2020-09-11 | 北京首汽智行科技有限公司 | Service ID generation method based on Redis |
| CN111813559A (en) * | 2020-07-23 | 2020-10-23 | 济南大陆机电股份有限公司 | Industrial Internet distributed identification coding method and system |
| CN111953772A (en) * | 2020-08-11 | 2020-11-17 | 北京达佳互联信息技术有限公司 | Request processing method, device, server and storage medium |
| CN111953772B (en) * | 2020-08-11 | 2022-11-22 | 北京达佳互联信息技术有限公司 | Request processing method, device, server and storage medium |
| CN113220682A (en) * | 2021-05-08 | 2021-08-06 | 北京猿力未来科技有限公司 | Business data analysis method, data processing method, data analysis system and storage medium |
| CN113297327A (en) * | 2021-05-24 | 2021-08-24 | 建信金融科技有限责任公司 | System and method for generating distributed ID |
| CN113360172A (en) * | 2021-06-30 | 2021-09-07 | 平安普惠企业管理有限公司 | Application deployment method and device, computer equipment and storage medium |
| CN113360172B (en) * | 2021-06-30 | 2024-02-27 | 李建 | Application deployment method, device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111046057A (en) | Data processing method and device for server cluster, computer equipment and medium | |
| US10506043B2 (en) | Rapid client-side component processing based on component relationships | |
| US8069224B2 (en) | Method, equipment and system for resource acquisition | |
| US10887190B2 (en) | System for simultaneous viewing and editing of multiple network device configurations | |
| CN107800565B (en) | Inspection method, inspection device, inspection system, computer equipment and storage medium | |
| CN109474456B (en) | Configuration data processing method, device, computer equipment and storage medium | |
| US9135454B2 (en) | Systems and methods for enabling searchable encryption | |
| US7974218B2 (en) | Intersection-based configuration management | |
| US11082293B2 (en) | System and method for validating correctness of changes to network device configurations | |
| CN110083627B (en) | Data processing method, system, computer device and storage medium | |
| US10474185B2 (en) | Timestamp alignment across a plurality of computing devices | |
| CN103701913A (en) | Data synchronization method and device | |
| US20210168207A1 (en) | Method, electronic device and computer program product for storing and accessing data | |
| US9026493B1 (en) | Multi-master RDBMS improvements for distributed computing environment | |
| CN103716384A (en) | Method and device for realizing cloud storage data synchronization in cross-data-center manner | |
| CN113760847A (en) | Log data processing method, device, equipment and storage medium | |
| CN110046062A (en) | Distributed data processing method and system | |
| CN111104250B (en) | Method, apparatus and computer readable medium for data processing | |
| CN103905512A (en) | Data processing method and equipment | |
| CN111031126B (en) | Cluster cache sharing method, system, equipment and storage medium | |
| US10185735B2 (en) | Distributed database system and a non-transitory computer readable medium | |
| CN112035413A (en) | Metadata information query method and device and storage medium | |
| CN112860746B (en) | Cache reduction-based method, equipment and system | |
| CN110935168B (en) | Distributed ID generation method, device and equipment for global uniform architecture | |
| JP5703405B2 (en) | Network device search system and search method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant after: Jingdong Technology Holding Co.,Ltd. Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant before: JINGDONG DIGITAL TECHNOLOGY HOLDINGS Co.,Ltd. |
|
| CB02 | Change of applicant information |