CN113342851B - Sequence capacity expansion method - Google Patents
Sequence capacity expansion method Download PDFInfo
- Publication number
- CN113342851B CN113342851B CN202110595188.9A CN202110595188A CN113342851B CN 113342851 B CN113342851 B CN 113342851B CN 202110595188 A CN202110595188 A CN 202110595188A CN 113342851 B CN113342851 B CN 113342851B
- Authority
- CN
- China
- Prior art keywords
- sequence
- value
- preset threshold
- number segment
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000011084 recovery Methods 0.000 abstract description 8
- 230000002688 persistence Effects 0.000 description 16
- 230000006870 function Effects 0.000 description 8
- 230000001960 triggered effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000008602 contraction Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000003780 insertion Methods 0.000 description 2
- 230000037431 insertion Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 241000282326 Felis catus Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a sequence expansion method. Wherein the method comprises the following steps: determining the maximum available sequence value in the current number segment in a first database; judging whether the current sequence value in the current number segment is larger than the maximum available sequence value; under the condition that the current sequence value is smaller than the maximum available sequence value, determining the ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment, wherein the ratio is a dynamic value; and judging whether the ratio reaches a preset threshold, and triggering a sequence capacity expansion operation which is used for requesting a new target number segment from the second database under the condition that the ratio reaches the preset threshold. The method and the device solve the technical problems that in the related art, when the Redis is used for generating the global unique ID, the ID is repeated and the data recovery time is too long after the Redis is restarted.
Description
Technical Field
The application relates to the field of identification information, in particular to a sequence capacity expansion method.
Background
In complex distributed systems, it is often necessary to uniquely identify large amounts of data and messages. In the system of products such as finance, payment, catering, hotels, cat eye movies and the like which are commented by the beauty group, the data are gradually increased, a unique identifier ID is needed to identify a piece of data or information after the data are divided into tables, and the self-increasing identification ID of the database obviously cannot meet the requirements under a certain scene due to performance reasons; such as orders, riders, coupons, etc., also need to be identified by a unique ID, a system that can generate a globally unique ID is highly desirable. In the related art, an ID is often generated by using a cache database, but the cache database is prone to cache loss, and in the example of Redis, there are two persistence modes RDB and AOF, and the following two problems may occur: (1) The RDB can generate a snapshot regularly for persistence, if the snapshot is continuously self-increased but the Redis is not timely persistence, and at the moment, the Redis service is not available, partial data loss can exist after the Redis is restarted, and the situation of ID repetition occurs. (2) The AOF can persist each write command, and even if Redis service is not available, ID repetition does not occur, but due to the particularity of the incr command, the Redis restarting and recovering data can be caused to be too long. (3) When the Redis cooperates with the database, the frequency of synchronization with the database needs to be precisely controlled, and the balance of usability and performance is sought. With fixed time interval synchronization, if the call traffic is large, data loss is likely to occur, and if the time interval is reduced, unnecessary synchronization overhead is generated at idle. Therefore, in the related art, when the Redis is used to generate the global unique ID, after the Redis is down and restarted, there may be technical problems such as ID repetition and too long time for recovering data.
In view of the above problems, no effective solution has been proposed at present.
Disclosure of Invention
The embodiment of the application provides a sequence capacity expansion method, which at least solves the technical problems of ID repetition and overlong data recovery time caused by the fact that Redis is down and restarted when the Redis is used for generating a global unique ID in the related technology.
According to an aspect of the embodiments of the present application, there is provided a sequence expansion method, including: determining the maximum available sequence value in the current number segment in the first database, wherein the current number segment comprises a plurality of sequence values, the sequence values are sequentially arranged in the current number segment according to a preset sequence, the sequence values correspond to the identification information one by one, and the maximum available sequence value is the sequence value with the maximum value in the current number segment; judging whether the current sequence value in the current number segment is larger than the maximum available sequence value; under the condition that the current sequence value is smaller than the maximum available sequence value, determining the ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment, wherein the ratio is a dynamic value; judging whether the ratio reaches a preset threshold, and triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold, wherein the sequence capacity expansion operation is used for requesting a new target number segment from a second database, the first database is a non-relational database, and the second database is a relational database.
Optionally, the initial sequence length of the target number segment is determined by: obtaining the total number of sequence generation in a preset time length, and determining the corresponding average sequence number in a unit time length; and determining the initial sequence length of the target mark segment according to the average sequence number and the preset multiple.
Optionally, determining the initial sequence length of the target label segment according to the average sequence number and the preset multiple includes: and determining the initial sequence length of the target label segment according to the product of the average sequence number and the preset multiple.
Optionally, the preset threshold includes: the method comprises the steps of a first preset threshold, a second preset threshold, a third preset threshold and a fourth preset threshold, wherein the values corresponding to the first preset threshold, the second preset threshold, the third preset threshold and the fourth preset threshold are sequentially reduced, judging whether the ratio reaches the preset threshold, and triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold, wherein the method comprises the following steps:
judging whether the ratio reaches a first preset threshold value, and triggering a first sequence capacity expansion operation under the condition that the ratio reaches the first preset threshold value; after determining that the first sequence expansion operation fails, judging whether the ratio reaches a second preset threshold value, and triggering a second sequence expansion operation under the condition that the ratio reaches the second preset threshold value; after determining that the second-order-column capacity expansion operation fails, judging whether the ratio reaches a third preset threshold value, and triggering the third-order-column capacity expansion operation under the condition that the ratio reaches the third preset threshold value; after the failure of the third sequence expansion operation is determined, judging whether the ratio reaches a fourth preset threshold value, and triggering the fourth sequence expansion operation under the condition that the ratio reaches the fourth preset threshold value.
Optionally, after triggering the sequence expansion operation in the case that the ratio reaches the preset threshold, the method further includes: the maximum available value in the first database in the second database is updated.
Optionally, the non-relational database comprises: redis database, relational database includes: oracle database.
Optionally, after determining the initial sequence length of the target label segment according to the average sequence number and the preset multiple, the method further includes: judging the initial sequence length of the target mark segment and the size of a preset step length; under the condition that the initial sequence length of the target number segment is larger than a preset step length, taking the initial sequence length of the target number segment as the sequence length of the target number segment; and taking the preset step length as the sequence length of the target number segment under the condition that the initial sequence length of the target number segment is smaller than the preset step length.
According to an aspect of an embodiment of the present application, there is provided a sequential capacity expansion device, including: the first determining module is used for determining the maximum available sequence value in the current number segment in the first database, wherein the current number segment comprises a plurality of sequence values, the sequence values are sequentially arranged in the current number segment according to a preset sequence, the sequence values are in one-to-one correspondence with the identification information, and the maximum available sequence value is the sequence value with the maximum value in the current number segment; the judging module is used for judging whether the current sequence value in the current number segment is larger than the maximum available sequence value or not; the second determining module is used for determining the ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment under the condition that the current sequence value is smaller than the maximum available sequence value, wherein the ratio is a dynamic value; and the capacity expansion module is used for judging whether the ratio reaches a preset threshold value, and triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold value, wherein the sequence capacity expansion operation is used for requesting a new target number segment from a second database, the first database is a non-relational database, and the second database is a relational database.
According to an aspect of the embodiments of the present application, there is further provided a nonvolatile storage medium, where the nonvolatile storage medium includes a stored program, and when the program runs, the device in which the nonvolatile storage medium is controlled to execute any one of the sequence expansion methods.
According to an aspect of the embodiments of the present application, there is further provided a processor, configured to execute a program, where the program executes any one of the sequence expansion methods.
In the embodiment of the application, the method for generating the identification information by adopting two different types of databases includes the steps that the maximum available sequence value in a current number segment is determined in a first database, wherein the current number segment comprises a plurality of sequence values, the sequence values are sequentially arranged in the current number segment according to a preset sequence, the sequence values are in one-to-one correspondence with the identification information, and the maximum available sequence value is the sequence value with the maximum value in the current number segment; judging whether the current sequence value in the current number segment is larger than the maximum available sequence value; under the condition that the current sequence value is smaller than the maximum available sequence value, determining the ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment, wherein the ratio is a dynamic value; judging whether the ratio reaches a preset threshold value, triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold value, wherein the sequence capacity expansion operation is used for requesting a new target number segment from a second database, the first database is a non-relational database, the second database is a relational database, the purposes of combining the first database (cache database) with the second database (traditional relational database) and expanding the sequence based on the traditional relational database are achieved, the cache persistence function of the cache database is realized, the cache database is combined with the traditional relational database, the cache database is responsible for generating the sequence, the traditional relational database is responsible for persistence of the sequence basic information, and the cache database can recover and generate new identification information based on the persistence of the sequence basic information in the traditional relational database, so that the technical problems of ID repetition and recovery data time caused after the re-shutdown are solved when the Redis is used for generating a global unique ID in the related technology.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:
FIG. 1 is a flow diagram of an alternative sequential expansion method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of an alternative workflow principle of the present application;
fig. 3 is a schematic structural diagram of an alternative sequential capacity expansion device according to an embodiment of the present application.
Detailed Description
In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In order to facilitate a better understanding of the related embodiments of the present application, technical terms or partial terms that may be involved in the related embodiments of the present application are explained below:
redis database: an open source, support network, memory-based, distributed, persistent key-value pair storage database written using ANSIC.
The RDB is a snapshot storage persistence mode, specifically, memory data of a Redis at a certain moment is stored in a file of hardware, the default stored file name is dump.rdb, and when the Redis server is started, the data of the dump.rdb file is reloaded into the memory to restore the data.
And in an AOF persistence mode, each write operation command of the client to the server is recorded, the write operations are additionally stored to the end of a file with a suffix of AOF by using a Redis protocol, and when the Redis server is restarted, the command of AOF files is loaded and operated, so that the purpose of recovering data is achieved.
In accordance with the embodiments of the present application, there is provided a method embodiment of sequence expansion, it being noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system, such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
The unique system ID is a problem frequently encountered when designing a real-time service system, and the method for generating the ID is quite large and is suitable for different scenes, requirements and performance requirements. Some more complex systems have multiple ID generation strategies. Some common ID generation strategies in the related art are listed below:
1. the database grows sequences or fields from itself, most commonly. With the database, the full database is unique.
Uuid: in a common manner. Either database or program generation may be utilized, generally globally unique.
Redis generation ID: when the performance of using a database to generate an ID is not sufficient, an attempt may be made to use dis to generate an ID. This relies primarily on Redis being single threaded, so generating globally unique IDs can also be used. Can be implemented with atomic operations INCR and INCRBY of Redis.
The swflag algorithm of twitter: the swflag is a distributed ID generation algorithm with Twitter open source, resulting in a long type ID. The core idea is as follows: using 41 bits as the number of milliseconds, 10 bits as the machine ID (5 bits are the data center, 5 bits of machine ID), 12 bits as the running water number within milliseconds (meaning that each node can generate 4096 IDs per millisecond), and finally one sign bit, always 0.
5. Unique ID is generated using zookeeper: the zookeeper generates a sequence number primarily from its znode data version, can generate 32-bit and 64-bit data version numbers, which can be used by clients as unique sequence numbers.
However, to take cache downtime into consideration when generating an ID with a cache system, redis, which has two persistence methods RDB and AOF, may generate the following two types of problems: (1) RDB will make a snapshot to persist regularly, if it is continuously self-increasing but Redis is not persisted in time, and this will hang up the Redis, and there may be partial data loss after restarting the Redis, resulting in the situation that ID is repeated. (2) The AOF can persist each write command, and even if the Redis is hung up, the situation that the ID is repeated does not occur, but due to the particularity of the incr command, the data recovery time of the Redis restarting is too long. (3) When the Redis cooperates with the database, the frequency of synchronization with the database needs to be precisely controlled, and the balance of usability and performance is sought. With fixed time interval synchronization, if the call traffic is large, data loss is likely to occur, and if the time interval is reduced, unnecessary synchronization overhead is generated at idle.
The method and the device mainly solve the problems that the restarting is performed after the Redis is down, the ID is repeated and the recovery time is too long, and the like.
Fig. 1 is a sequence expansion method according to an embodiment of the present application, as shown in fig. 1, the method includes the following steps:
step S102, determining the maximum available sequence value in the current number segment in the first database, wherein the current number segment comprises a plurality of sequence values, the sequence values are sequentially arranged in the current number segment according to a preset sequence, the sequence values are in one-to-one correspondence with the identification information, and the maximum available sequence value is the sequence value with the maximum value in the current number segment;
step S104, judging whether the current sequence value in the current number segment is larger than the maximum available sequence value;
step S106, under the condition that the current sequence value is smaller than the maximum available sequence value, determining the ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment, wherein the ratio is a dynamic value;
step S108, judging whether the ratio reaches a preset threshold, and triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold, wherein the sequence capacity expansion operation is used for requesting a new target number segment from a second database, the first database is a non-relational database, and the second database is a relational database.
In the sequence capacity expansion method, the maximum available sequence value in the current number segment in the first database can be determined, wherein the current number segment comprises a plurality of sequence values, the sequence values are sequentially arranged in the current number segment according to a preset sequence, the sequence values correspond to the identification information one by one, and the maximum available sequence value is the sequence value with the maximum value in the current number segment; judging whether the current sequence value in the current number segment is larger than the maximum available sequence value; under the condition that the current sequence value is smaller than the maximum available sequence value, determining the ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment, wherein the ratio is a dynamic value; judging whether the ratio reaches a preset threshold value, triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold value, wherein the sequence capacity expansion operation is used for requesting a new target number segment from a second database, the first database is a non-relational database, the second database is a relational database, the purposes of combining the first database (cache database) with the second database (traditional relational database) and expanding the sequence based on the traditional relational database are achieved, the cache persistence function of the cache database is realized, the cache database is combined with the traditional relational database, the cache database is responsible for generating the sequence, the traditional relational database is responsible for persistence of the sequence basic information, and the cache database can recover and generate new identification information based on the persistence of the sequence basic information in the traditional relational database, so that the technical problems of ID repetition and recovery data time caused after the re-shutdown are solved when the Redis is used for generating a global unique ID in the related technology.
It should be noted that, the initial sequence length of the target number segment may be determined by: obtaining the total number of sequence generation in a preset time length, and determining the corresponding average sequence number in a unit time length; and determining the initial sequence length of the target mark segment according to the average sequence number and the preset multiple.
Specifically, determining the initial sequence length of the target number segment according to the average sequence number and the preset multiple includes: and determining the initial sequence length of the target label segment according to the product of the average sequence number and the preset multiple.
It should be noted that, the preset threshold includes: the first preset threshold, the second preset threshold, the third preset threshold and the fourth preset threshold, wherein the values corresponding to the first preset threshold, the second preset threshold, the third preset threshold and the fourth preset threshold are sequentially reduced, so that whether the ratio reaches the preset threshold is judged, and when the ratio reaches the preset threshold, the sequence capacity expansion operation is triggered, which comprises the following steps:
judging whether the ratio reaches a first preset threshold value, and triggering a first sequence capacity expansion operation under the condition that the ratio reaches the first preset threshold value; after determining that the first sequence expansion operation fails, judging whether the ratio reaches a second preset threshold value, and triggering a second sequence expansion operation under the condition that the ratio reaches the second preset threshold value; after determining that the second-order-column capacity expansion operation fails, judging whether the ratio reaches a third preset threshold value, and triggering the third-order-column capacity expansion operation under the condition that the ratio reaches the third preset threshold value; after the failure of the third sequence expansion operation is determined, judging whether the ratio reaches a fourth preset threshold value, and triggering the fourth sequence expansion operation under the condition that the ratio reaches the fourth preset threshold value. It should be noted that, in the extreme case of the system running in ultra-high concurrency, the probability that all the four preset thresholds cannot be triggered is very low.
In some embodiments of the present application, after triggering the sequence expansion operation in the case that the ratio reaches the preset threshold, the method further includes: the maximum available value in the first database in the second database is updated.
It should be noted that the non-relational database includes, but is not limited to: redis databases, relational databases including, but not limited to: oracle database.
In some embodiments of the present application, after determining the initial sequence length of the reference mark segment according to the average sequence number and the preset multiple, the initial sequence length of the reference mark segment and the preset step size may be determined; under the condition that the initial sequence length of the target number segment is larger than a preset step length, taking the initial sequence length of the target number segment as the sequence length of the target number segment; and taking the preset step length as the sequence length of the target number segment under the condition that the initial sequence length of the target number segment is smaller than the preset step length.
FIG. 2 is a schematic diagram of an alternative workflow principle of the present application, as shown in FIG. 2, the flow includes:
the user calls the Redis database, acquires the maximum available value in the current sequence from the Redis database, judges whether the maximum value of the sequence exists, if so, throws out the maximum ID of the abnormal prompt index, crosses the boundary, if not, judges whether to trigger the sequence expansion according to an algorithm, if so, updates the maximum available value of the Redis sequence stored in the Oracle database, updates the Redis insertion sequence information, generates a sequence return, and if not, generates the sequence return.
The user calls a Redis database, acquires the maximum available value in the current sequence from the Redis database, judges whether the maximum value of the sequence exists, acquires the Redis sequence information from the Oracle database if the maximum value of the sequence does not exist, then judges whether the Redis sequence information exists in the Oracle database, if yes, updates the maximum available value of the Redis sequence stored in the Oracle database, updates the Redis insertion sequence information, and generates a sequence return; if not, the Oracle database inserts the Redis sequence data, the Redis inserts the sequence information, and a sequence return is generated.
It is easy to note that in the embodiment of the present application, a conventional relational database (e.g. Oracle) is used as a sequence base value, an increment sequence is taken from a distributed cache (e.g. Redis), if the increment reaches a threshold value, a step size to be applied in the next step is dynamically predicted according to inertia time, and a new base value is taken from the Oracle. If the sequence service is restarted, the number segment from the basic value to the next compensation interval can be abandoned, and the problem of repeated running water is avoided.
The sequence model uses the following concepts:
(1) Sequence key: caching names used by the acquisition sequences;
(2) maxID: a current maximum available sequence value;
(3) Expansion: the maxID is about to reach the upper limit, the maxID is dynamically increased according to a formula, and the capacity expansion ratio can be set as follows: {0.4,0.3,0.2,0.1}, i.e. the first preset threshold, the second preset threshold, the third preset threshold and the fourth preset threshold may be set to 0.4,0.3,0.2,0.1, respectively, and when the capacity expansion ratio is reached, the maxID capacity expansion is triggered;
(4) Step size: the usable sequence length of each expansion increase is 5000, and the step length can be dynamically changed according to the step expansion function;
(5) Step size expansion and contraction capacity formula: step size = total number of sequence generation of last 5 minutes/5*2, the step size cannot be smaller than the default step size, namely the initial sequence length (step size) is determined by obtaining the total number of sequence generation (total number of sequence generation of last 5 minutes) in a preset time period, determining the corresponding average sequence number (total number of sequence generation of last 5 minutes/5) in a unit time period (in per minute) in the preset time period, and determining the initial sequence length of the target number segment according to the average sequence number (total number of sequence generation of last 5 minutes/5) and a preset multiple (2).
The working principle of the tool is as follows:
the tool will perform an initialization Redis sequence operation when started, and the Redis sequence relates to main data items including: sequence key, maxID, sequence step size, etc. After the application is started, maxID is obtained from Redis, the maxID is not null, the comparison is carried out with the current sequence value, whether the expansion is triggered or not is carried out, and the sequence value is returned if the expansion is not triggered. If Redis sequence acquisition maxID is null (used herein in connection with Redis and Oracle), then Redis sequence persistence information is acquired in the Oracle database. Initializing the Redis sequence when the information exists and generating the Redis sequence information to be put in storage and initializing the Redis sequence when the information does not exist.
Meanwhile, a dynamic step length prediction function is added, the program can independently calculate the total number of sequences generated per minute of each sequence, and the optimal step length is calculated according to every 5 minutes, if the lowest step length of the initial value of the program is 1000, the service grows in a certain 5 minutes, the average sequence value generated number in the last 5 minutes becomes 2000, and the current step length value is 4000 according to the existing step length expansion and contraction formula. Otherwise, the traffic volume is reduced, the step value is also reduced, and the minimum step value cannot be lower than the default step value. The predicted value of the step size can be achieved through a variety of prediction algorithms.
It should be noted that, the above embodiment is illustrated by Oracle and Redis, and other conventional relational databases and distributed cache components may be used for processing in the same manner as above. Furthermore, the above definitions of the elements and methods are not limited to the specific structures, shapes or modes mentioned in the embodiments, and may be modified or replaced simply by one skilled in the art, for example: oracle may be replaced with a relational database such as Mysql.
It is readily noted that the present application provides a tool that is capable of generating a fast distributed globally unique sequence; the method and the device solve the problems that in global sequence service, the ID is repeated after the buffer service is down and the recovery time is too long, and the like.
Fig. 3 is a sequence expansion device according to an embodiment of the present application, as shown in fig. 3, including:
a first determining module 40, configured to determine a maximum available sequence value in a current number segment in the first database, where the current number segment includes a plurality of sequence values, the sequence values are sequentially arranged in a predetermined order in the current number segment, the sequence values correspond to the identification information one by one, and the maximum available sequence value is a sequence value with a maximum value in the current number segment;
a judging module 42, configured to judge whether the current sequence value in the current number segment is greater than the maximum available sequence value;
a second determining module 44, configured to determine, in the case that the current sequence value is smaller than the maximum available sequence value, a ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment, where the ratio is a dynamic value;
the capacity expansion module 46 is configured to determine whether the ratio reaches a preset threshold, and trigger a sequence capacity expansion operation when the ratio reaches the preset threshold, where the sequence capacity expansion operation is used to request a new target number segment from a second database, and the first database is a non-relational database, and the second database is a relational database.
In the sequence expansion device, a first determining module 40 is configured to determine a maximum available sequence value in a current number segment in a first database, where the current number segment includes a plurality of sequence values, the sequence values are sequentially arranged in a predetermined order in the current number segment, the sequence values correspond to identification information one by one, and the maximum available sequence value is a sequence value with a maximum value in the current number segment; a judging module 42, configured to judge whether the current sequence value in the current number segment is greater than the maximum available sequence value; a second determining module 44, configured to determine, in the case that the current sequence value is smaller than the maximum available sequence value, a ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment, where the ratio is a dynamic value; the capacity expansion module 46 is configured to determine whether the ratio reaches a preset threshold, trigger a sequence capacity expansion operation when the ratio reaches the preset threshold, and the sequence capacity expansion operation is configured to request a new target number segment from a second database, where the first database is a non-relational database, and the second database is a relational database, so as to achieve the purposes of using the first database (cache database) in combination with the second database (traditional relational database) and expanding the sequence based on the traditional relational database, thereby implementing a cache persistence function of the cache database, changing a mode of combining the cache database with the traditional relational database, wherein the cache database is responsible for generating the sequence, the traditional relational database is responsible for persistence of sequence basic information, and the cache database can recover and generate new identification information based on the persistence of sequence basic information in the traditional relational database, so as to solve the technical problems of ID duplication and recovery time after the redes are restarted due to the use of the redes to generate globally unique IDs in the related art.
According to an aspect of the embodiments of the present application, there is further provided a nonvolatile storage medium, where the nonvolatile storage medium includes a stored program, and when the program runs, the device in which the nonvolatile storage medium is controlled to execute any one of the sequence expansion methods.
The storage medium is used for storing program instructions for executing the functions of determining the maximum available sequence value in the current number segment in the first database, wherein the current number segment comprises a plurality of sequence values, the sequence values are sequentially arranged in the current number segment according to a preset sequence, the sequence values are in one-to-one correspondence with the identification information, and the maximum available sequence value is the sequence value with the maximum value in the current number segment; judging whether the current sequence value in the current number segment is larger than the maximum available sequence value; under the condition that the current sequence value is smaller than the maximum available sequence value, determining the ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment, wherein the ratio is a dynamic value; judging whether the ratio reaches a preset threshold, and triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold, wherein the sequence capacity expansion operation is used for requesting a new target number segment from a second database, the first database is a non-relational database, and the second database is a relational database.
According to an aspect of the embodiments of the present application, there is further provided a processor, configured to execute a program, where the program executes any one of the sequence expansion methods.
Specifically, the above processor is configured to call program instructions in the memory, and implement the following functions: determining the maximum available sequence value in the current number segment in the first database, wherein the current number segment comprises a plurality of sequence values, the sequence values are sequentially arranged in the current number segment according to a preset sequence, the sequence values correspond to the identification information one by one, and the maximum available sequence value is the sequence value with the maximum value in the current number segment; judging whether the current sequence value in the current number segment is larger than the maximum available sequence value; under the condition that the current sequence value is smaller than the maximum available sequence value, determining the ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment, wherein the ratio is a dynamic value; judging whether the ratio reaches a preset threshold, and triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold, wherein the sequence capacity expansion operation is used for requesting a new target number segment from a second database, the first database is a non-relational database, and the second database is a relational database.
The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.
In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed technology content may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, for example, may be a logic function division, and may be implemented in another manner, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application and are intended to be comprehended within the scope of the present application.
Claims (9)
1. A method of sequence expansion, comprising:
determining the maximum available sequence value in a current number segment in a first database, wherein the current number segment comprises a plurality of sequence values, the sequence values are sequentially arranged in the current number segment according to a preset sequence, the sequence values are in one-to-one correspondence with identification information, and the maximum available sequence value is the sequence value with the maximum value in the current number segment;
judging whether the current sequence value in the current number segment is larger than the maximum available sequence value or not;
determining a ratio of the remaining unused sequence values to the maximum available sequence value in the current number segment, wherein the ratio is a dynamic value, under the condition that the current sequence value is smaller than the maximum available sequence value;
judging whether the ratio reaches a preset threshold value, and triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold value, wherein the sequence capacity expansion operation is used for requesting a new target number segment from a second database, the first database is a non-relational database, and the second database is a relational database;
wherein the initial sequence length of the target number segment is determined by:
obtaining the total number of sequence generation in a preset time length, and determining the corresponding average sequence number in a unit time length;
and determining the initial sequence length of the target number segment according to the average sequence number and a preset multiple.
2. The method of claim 1, wherein determining the initial sequence length of the target number segment from the average sequence number and a predetermined multiple comprises:
and determining the initial sequence length of the target number segment according to the product of the average sequence number and the preset multiple.
3. The method of claim 1, wherein the preset threshold comprises: the method comprises the steps of a first preset threshold, a second preset threshold, a third preset threshold and a fourth preset threshold, wherein the values corresponding to the first preset threshold, the second preset threshold, the third preset threshold and the fourth preset threshold are sequentially reduced, judging whether the ratio reaches the preset threshold or not, and triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold, wherein the method comprises the following steps:
judging whether the ratio reaches a first preset threshold value or not, and triggering a first sequence capacity expansion operation under the condition that the ratio reaches the first preset threshold value;
after determining that the first sequence expansion operation fails, judging whether the ratio reaches the second preset threshold value, and triggering a second sequence expansion operation under the condition that the ratio reaches the second preset threshold value;
after determining that the second-order-column capacity expansion operation fails, judging whether the ratio reaches the third preset threshold value or not, and triggering a third-order-column capacity expansion operation under the condition that the ratio reaches the third preset threshold value;
and after determining that the third sequence expansion operation fails, judging whether the ratio reaches the fourth preset threshold, and triggering a fourth sequence expansion operation under the condition that the ratio reaches the fourth preset threshold.
4. The method of claim 1, wherein, in the event that the ratio reaches a preset threshold, after triggering a sequence expansion operation, the method further comprises:
updating the maximum available value in the first database in the second database.
5. The method of claim 1, wherein the non-relational database comprises: a Redis database, the relational database comprising: oracle database.
6. The method of claim 1, wherein after determining the initial sequence length of the target number segment from the average sequence number and a preset multiple, the method further comprises:
judging the initial sequence length of the target number segment and the size of a preset step length;
taking the initial sequence length of the target number segment as the sequence length of the target number segment under the condition that the initial sequence length of the target number segment is larger than the preset step length;
and taking the preset step length as the sequence length of the target number segment when the initial sequence length of the target number segment is smaller than the preset step length.
7. A sequence expansion device, comprising:
the first determining module is used for determining the maximum available sequence value in the current number segment in the first database, wherein the current number segment comprises a plurality of sequence values, the sequence values are sequentially arranged in the current number segment according to a preset sequence, the sequence values correspond to the identification information one by one, and the maximum available sequence value is the sequence value with the maximum value in the current number segment;
the judging module is used for judging whether the current sequence value in the current number segment is larger than the maximum available sequence value or not;
the second determining module is used for determining the ratio of the remaining unused sequence value to the maximum available sequence value in the current number segment under the condition that the current sequence value is smaller than the maximum available sequence value, wherein the ratio is a dynamic value;
the capacity expansion module is used for judging whether the ratio reaches a preset threshold value, and triggering a sequence capacity expansion operation under the condition that the ratio reaches the preset threshold value, wherein the sequence capacity expansion operation is used for requesting a new target number segment from a second database, the first database is a non-relational database, and the second database is a relational database;
wherein the initial sequence length of the target number segment is determined by:
obtaining the total number of sequence generation in a preset time length, and determining the corresponding average sequence number in a unit time length;
and determining the initial sequence length of the target number segment according to the average sequence number and a preset multiple.
8. A non-volatile storage medium, characterized in that the non-volatile storage medium comprises a stored program, wherein the program, when run, controls a device in which the non-volatile storage medium is located to perform the sequence expansion method of any of claims 1 to 6.
9. A processor, characterized in that the processor is configured to run a program, wherein the program when run performs the sequence expansion method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110595188.9A CN113342851B (en) | 2021-05-28 | 2021-05-28 | Sequence capacity expansion method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110595188.9A CN113342851B (en) | 2021-05-28 | 2021-05-28 | Sequence capacity expansion method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113342851A CN113342851A (en) | 2021-09-03 |
| CN113342851B true CN113342851B (en) | 2024-04-02 |
Family
ID=77472122
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110595188.9A Active CN113342851B (en) | 2021-05-28 | 2021-05-28 | Sequence capacity expansion method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113342851B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292314B (en) * | 2022-07-28 | 2023-11-24 | 上海数禾信息科技有限公司 | Sequence number generation method, device, computer equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999525A (en) * | 2011-09-16 | 2013-03-27 | 深圳市金蝶中间件有限公司 | Data-table processing method and system |
| CN106899654A (en) * | 2016-08-10 | 2017-06-27 | 阿里巴巴集团控股有限公司 | A kind of sequence value generation method, apparatus and system |
| CN111930765A (en) * | 2020-07-31 | 2020-11-13 | 银盛支付服务股份有限公司 | Method for generating unique serial number of distributed system |
| CN111949730A (en) * | 2020-08-12 | 2020-11-17 | 中国银行股份有限公司 | Distributed global sequence number implementation method and sequence distributor |
| CN112035460A (en) * | 2019-06-04 | 2020-12-04 | 北京京东尚科信息技术有限公司 | Identification distribution method, device, equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10108630B2 (en) * | 2011-04-07 | 2018-10-23 | Microsoft Technology Licensing, Llc | Cluster unique identifier |
-
2021
- 2021-05-28 CN CN202110595188.9A patent/CN113342851B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999525A (en) * | 2011-09-16 | 2013-03-27 | 深圳市金蝶中间件有限公司 | Data-table processing method and system |
| CN106899654A (en) * | 2016-08-10 | 2017-06-27 | 阿里巴巴集团控股有限公司 | A kind of sequence value generation method, apparatus and system |
| CN112035460A (en) * | 2019-06-04 | 2020-12-04 | 北京京东尚科信息技术有限公司 | Identification distribution method, device, equipment and storage medium |
| CN111930765A (en) * | 2020-07-31 | 2020-11-13 | 银盛支付服务股份有限公司 | Method for generating unique serial number of distributed system |
| CN111949730A (en) * | 2020-08-12 | 2020-11-17 | 中国银行股份有限公司 | Distributed global sequence number implementation method and sequence distributor |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113342851A (en) | 2021-09-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111460023B (en) | Method, device, equipment and storage medium for processing service data based on elastic search | |
| CN102521269B (en) | Index-based computer continuous data protection method | |
| US9092475B2 (en) | Database log parallelization | |
| US9672244B2 (en) | Efficient undo-processing during data redistribution | |
| US9155320B2 (en) | Prefix-based leaf node storage for database system | |
| US5642503A (en) | Method and computer system for implementing concurrent accesses of a database record by multiple users | |
| US9418094B2 (en) | Method and apparatus for performing multi-stage table updates | |
| US11269902B2 (en) | Time series data management method, device, and apparatus | |
| CN104572845B (en) | Document distribution method, device, equipment and system | |
| US11468011B2 (en) | Database management system | |
| AU2010250536A1 (en) | Meta-information sharing distributed database system in virtual single memory storage | |
| CN111309720A (en) | Time sequence data storage method, time sequence data reading method, time sequence data storage device, time sequence data reading device, electronic equipment and storage medium | |
| CN109445702A (en) | A kind of piece of grade data deduplication storage | |
| CN109241004B (en) | Metadata file size recovery method, system, device and readable storage medium | |
| CN113342851B (en) | Sequence capacity expansion method | |
| WO2013091167A1 (en) | Log storage method and system | |
| CN112286903A (en) | Containerization-based relational database optimization method and device | |
| CN106649722A (en) | Storing and inquiring method for high-frequency data of monitoring system | |
| CN111316255A (en) | Data storage system and method for providing a data storage system | |
| CN114297198A (en) | Method and device for generating serial number in distributed system | |
| CN107302569A (en) | A kind of security monitoring Data acquisition and storage method of facing cloud platform | |
| CN114936188A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111143323A (en) | MPP database management method, device and system | |
| US11048728B2 (en) | Dependent object analysis | |
| CN113032408B (en) | Data processing method, system and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |