CN109697187B

CN109697187B - Sequential message-based capacity expansion and contraction method and device and electronic equipment

Info

Publication number: CN109697187B
Application number: CN201711002835.0A
Authority: CN
Inventors: 冯嘉; 王小瑞; 周唯
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2017-10-24
Filing date: 2017-10-24
Publication date: 2022-10-21
Anticipated expiration: 2037-10-24
Also published as: CN109697187A

Abstract

The embodiment of the invention provides a sequential message-based capacity expansion and reduction method and device and electronic equipment. The capacity expansion method comprises the following steps: respectively recording the maximum site snapshot of each directory file to be expanded, wherein the maximum site of each directory file to be expanded is used for pointing to the latest written sequence message corresponding to the maximum site; meanwhile, controlling a server to be expanded to start a write mode; and when consumption sites in each directory file to be expanded are larger than or equal to the maximum site snapshot, controlling the server to be expanded to start a read mode, wherein the consumption sites are used for pointing to the sequential messages currently read by the consumer. The embodiment of the invention controls the read-write mode of the server to be expanded and contracted by effectively matching the site pointers in the directory file, thereby realizing the smooth expansion and contraction of the sequence message.

Description

Sequential message-based capacity expansion and contraction method and device and electronic equipment

Technical Field

The invention relates to the technical field of computers, in particular to a sequential message-based capacity expansion and contraction method and device and electronic equipment.

Background

In a distributed messaging system, message data has a specific Topic (Topic) attribute, which has the characteristics of sequential addition and global limitation, and is called sequential message.

When the storage or calculation resources of the system are insufficient, a new server needs to be added, namely, the system capacity expansion is carried out; when a certain server is abnormally down or partial resources are reduced due to system planning requirements, the corresponding server needs to be removed, that is, system capacity reduction is performed. When system expansion and capacity expansion are carried out, data (sequence messages) stored in each server of the original topological structure are redistributed, so that resources are evenly distributed in the new topological structure.

The traditional expansion and contraction capacity of the distributed system mainly comprises the following three aspects: 1. full replication, i.e., redistributing the original data into a new topology; 2. incremental synchronization, i.e., redistributing the newly added data into a new topology; 3. and (4) stopping writing and switching, namely, selecting a proper time to perform writing stopping operation on the original server when the increment is synchronized after the full replication is finished, and then switching the original data routing topology to a new routing topology.

In the process of implementing the invention, the inventor finds that the prior art has at least the following problems: the traditional capacity expansion and reduction technology has limitations in three aspects of full-scale copying, incremental synchronization and stop-write switching. For full replication, there is a conflict between the data replication speed and the write pressure on the database; for incremental synchronization, when multiple servers consume the same change log at the same time, the update may be out of order; for the stop-write switching, the switching time is difficult to grasp, and the capacity expansion and contraction without data loss are difficult to achieve.

Disclosure of Invention

The embodiment of the invention provides a method and a device for expanding and reducing capacity based on sequence information and electronic equipment, which are used for overcoming the defects of the existing capacity expanding and reducing technology and realizing continuous writing of the capacity expanding and reducing capacity based on the sequence information.

In order to achieve the above object, an embodiment of the present invention provides a capacity expansion method based on a sequence message, where the sequence message is sequentially stored in a data file, an address of the sequence message is stored in a plurality of directory files, and each directory file corresponds to a consumer, and the method includes: respectively recording the maximum site snapshot of each directory file to be expanded, wherein the maximum site of each directory file to be expanded is used for pointing to the latest written sequence message corresponding to the maximum site; meanwhile, controlling a server to be expanded to start a write mode; and when consumption sites in each directory file to be expanded are larger than or equal to the maximum site snapshot, controlling the server to be expanded to start a read mode, wherein the consumption sites are used for pointing to the sequential messages currently read by the consumer.

The embodiment of the invention also provides a capacity reduction method based on the sequence message, the sequence message is sequentially stored in a data file, the address of the sequence message is stored in a plurality of directory files, each directory file corresponds to a consumer, and the method comprises the following steps: respectively recording the maximum site snapshot of each directory file to be reduced, wherein the maximum site of each directory file to be reduced is used for pointing to the latest written sequence message corresponding to the maximum site; meanwhile, controlling the server to be reduced to close the writing mode; and when the consumption site in each directory file to be reduced is larger than or equal to the maximum site snapshot, controlling the server to be reduced to close a reading mode, wherein the consumption site is used for pointing to the sequential message currently read by the consumer.

An embodiment of the present invention further provides a capacity expansion device based on a sequence message, where the sequence message is sequentially stored in a data file, addresses of the sequence message are stored in a plurality of directory files, and each directory file corresponds to a consumer, and the device includes: the first recording module is used for respectively recording the maximum site snapshots of the directory files to be expanded, and the maximum site of each directory file to be expanded is used for pointing to the latest written sequence message corresponding to the maximum site; the first control module is used for controlling the server to be expanded to start a write mode while the first recording module records the maximum site snapshot; and the second control module is used for controlling the server to be expanded to start a read mode when consumption sites in each directory file to be expanded are larger than or equal to the maximum site snapshot, wherein the consumption sites are used for pointing to the sequential messages currently read by the consumer.

An embodiment of the present invention further provides a capacity reduction device based on a sequence message, where the sequence message is sequentially stored in a data file, addresses of the sequence message are stored in a plurality of directory files, and each directory file corresponds to a consumer, and the device includes: the second recording module is used for respectively recording the maximum locus snapshot of each directory file to be contracted, and the maximum locus of each directory file to be contracted is used for pointing to the latest written sequence message corresponding to the maximum locus; the fourth control module is used for controlling the server to be reduced in capacity to close a write mode while the second recording module records the maximum locus snapshot; and a fifth control module, configured to control the server to be reduced to close a read mode when consumption sites in each directory file to be reduced are greater than or equal to the maximum site snapshot, where the consumption sites are used to point to the sequential messages currently read by the consumer.

An embodiment of the present invention further provides an electronic device, including: a memory for storing a program; a processor for executing the program stored in the memory for: respectively recording the maximum site snapshot of each directory file to be expanded, wherein the maximum site of each directory file to be expanded is used for pointing to the latest written sequence message corresponding to the maximum site, the sequence message is sequentially stored in a data file, the address of the sequence message is stored in a plurality of directory files, and each directory file corresponds to a consumer; meanwhile, controlling a server to be expanded to start a write mode; and when the consumption site in each directory file to be expanded is greater than or equal to the maximum site snapshot, controlling the server to be expanded to start a read mode, wherein the consumption site is used for pointing to the sequential message currently read by the consumer.

An embodiment of the present invention further provides an electronic device, including: a memory for storing a program; a processor for executing the program stored in the memory for: respectively recording the maximum site snapshots of the directory files to be reduced, wherein the maximum site of each directory file to be reduced is used for pointing to the latest written sequence message corresponding to the maximum site, the sequence message is sequentially stored in a data file, the addresses of the sequence message are stored in a plurality of directory files, and each directory file corresponds to a consumer; meanwhile, controlling the server to be reduced to close the writing mode; and when the consumption site in each directory file to be reduced is larger than or equal to the maximum site snapshot, controlling the server to be reduced to close a reading mode, wherein the consumption site is used for pointing to the sequential message currently read by the consumer.

The method and the device for expanding and contracting the capacity based on the sequence message and the electronic equipment provided by the embodiment of the invention control the read-write mode of the server to be expanded and contracted through the effective cooperation of the site pointers in the directory file, avoid the limitation brought by adopting the traditional distributed system capacity expanding and contracting technology, and realize the smooth capacity expansion and contraction of the sequence message under the condition of not migrating data through the design idea of separating the data file and the directory file.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

fig. 1 is a schematic diagram illustrating a method for scaling a content based on a sequence message according to an embodiment of the present invention;

fig. 2a is a first schematic structural diagram of a service system according to an embodiment of the present invention;

fig. 2b is a schematic structural diagram of a service system according to an embodiment of the present invention;

FIG. 3 is a flowchart illustrating an exemplary embodiment of a method for capacity expansion based on sequence messages;

FIG. 4 is a flowchart illustrating another embodiment of a method for capacity expansion based on sequence messages according to the present invention;

FIG. 5 is a flowchart of an embodiment of a method for sequential message-based capacity reduction according to the present invention;

FIG. 6 is a flowchart of another embodiment of a method for sequential message-based capacity reduction according to the present invention;

fig. 7 is a schematic structural diagram of an embodiment of a capacity expansion apparatus based on a sequence message according to the present invention;

FIG. 8 is a schematic structural diagram of an alternative embodiment of a capacity expansion apparatus based on sequence messages according to the present invention;

FIG. 9 is a schematic structural diagram of an embodiment of a sequential message-based capacity reduction apparatus according to the present invention;

FIG. 10 is a schematic structural diagram of another embodiment of a sequential-message-based capacity reduction apparatus according to the present invention;

FIG. 11 is a schematic structural diagram of an embodiment of an electronic device provided in the present invention;

fig. 12 is a schematic structural diagram of another embodiment of an electronic device provided in the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

In order to overcome the defects of the prior art, the application provides a solution, wherein data files and directory files are respectively stored in each server of a topological structure, sequence messages are sequentially stored in the data files, addresses of the sequence messages are stored in a plurality of directory files, and each directory file corresponds to a consumer. The main principle of the solution is: when the capacity expansion/reduction operation is started, firstly, the maximum site snapshot of each directory file to be expanded/reduced in an original server (a server not in the column of the capacity expansion/reduction) of the topological structure is recorded respectively, and meanwhile, the write mode of the server to be expanded/reduced is controlled to be opened/closed (the server is opened, namely, the server is allowed to write data; the server is closed, namely, the server is forbidden to write data). That is, the corresponding position of the latest written sequence message in the directory file at present (the moment when the write mode is turned on/off by the server to be extended/reduced) is recorded. In the subsequent process, with the continuous writing of data, the maximum location point of the directory file will continue to increase, but still the snapshot of the maximum location point (the moment when the write mode is turned on/off by the server to be expanded/reduced) is used as the judgment basis. When recording the maximum site snapshot, the consumption site of each directory file to be expanded/reduced is less than or equal to the maximum site snapshot, the consumption site of each directory file to be expanded/reduced is continuously increased along with the continuous reading of data by a consumer, and when the consumption site of one directory file to be expanded/reduced is increased to be the same as or larger than the previously recorded maximum site snapshot, it means that the consumer corresponding to the directory file to be expanded/reduced has read all data before the maximum site snapshot. When all the files of the directory to be expanded/reduced satisfy the condition, that is, when the consumption site in each file of the directory to be expanded/reduced is greater than or equal to the maximum site snapshot, it means that all the sequential messages before the open/close write mode of the server to be expanded/reduced have been read by the consumer, and at this time, the open/close read mode of each server to be expanded/reduced is controlled (the server opens the read mode, that is, the server allows the consumer to read the data stored therein; the server closes the read mode, that is, the server prohibits the consumer from reading the data stored therein). In this process, the original server (server not in the scaling/capacity column) read-write mode is kept on all the time. The capacity expansion/reduction scheme provided by the embodiment of the invention can elegantly and smoothly realize the capacity expansion and reduction based on the sequence message under the condition of not migrating data.

Fig. 1 is a schematic diagram illustrating a schematic principle of a method for scaling a content based on a sequence message according to an embodiment of the present invention. As shown in fig. 1, in the origin server, the sequence messages are stored in sequence in a data file, and the addresses of the sequence messages are stored in a plurality of directory files, each directory file corresponding to a consumer. Taking two directory files to be expanded/contracted as an example, the addresses of

sequence messages

1, 3, 5 and 6 corresponding to the consumer A are respectively stored in four sites of 1, 2, 3 and 4 of the directory file 1; the addresses of

sequential messages

2, 4, 7, 8 corresponding to consumer B are stored in four

locations

1, 2, 3, 4 of directory file 2, respectively. When the capacity expansion/reduction operation is started, firstly, the maximum site snapshot of each directory file to be expanded/reduced (the maximum site of each directory file to be reduced is used for pointing to the latest written sequence message corresponding to the maximum site) is recorded respectively, and meanwhile, the write mode of the server to be expanded/reduced is controlled to be opened/closed. The following description will be made taking the directory file 1 as an example. In directory file 1, maximum site 4 points to the currently most recently written sequential message 6 corresponding to consumer a, minimum site 1 points to the oldest written sequential message 1 corresponding to consumer a, and consumption site 3 points to the sequential message 5 currently read by consumer a. Firstly, recording a snapshot (site 4) of the maximum site of the directory file 1 (at the moment when the write mode of the server to be expanded/reduced is turned on/off), and as the consumer A continuously reads data, the consumption site in the directory file 1 will continue to increase. When the consumption site in the directory file 1 is greater than or equal to the maximum site snapshot (site 4), it means that all the related sequential messages before the write mode is turned on/off by the server to be expanded/reduced are read by the consumer a. And when the consumption sites in all the directory files to be expanded/reduced are greater than or equal to the maximum site snapshot, controlling each server to be expanded/reduced to open/close a reading mode. In this process, the read-write mode of the original server (server not in the scaling/capacity column) is always kept on, so that scaling based on the sequence message is realized gracefully and smoothly without migrating data.

The method provided by the embodiment of the invention can be applied to any distributed service system for storing the sequential messages. Fig. 2a is a schematic structural diagram of a service system according to an embodiment of the present invention. As shown in fig. 2a, the service system is configured to expand a topology, and includes:

original servers

1, 2, n (servers originally existing in the topology), servers to be expanded (a plurality of servers may be used, one is taken as an example in the figure), and expansion devices. The capacity expansion device may be configured to execute the processing flows shown in fig. 3 and fig. 4, which are mainly configured to record the snapshot of the maximum location point of each directory file to be expanded in the original server, where the maximum location point is used to point to the latest written sequence message corresponding to each directory file to be expanded; meanwhile, the server to be expanded is controlled to start a write mode, and a part of newly generated data flows into the server to be expanded; and when the consumption site in each directory file to be expanded is larger than or equal to the maximum site snapshot, controlling the server to be expanded to start a reading mode, wherein the consumption site is used for pointing to the current read sequential message of the consumer. At the moment, the system capacity expansion is completed, and the server to be subjected to capacity expansion keeps an open state in the same read-write mode as the original server. Fig. 2b is a schematic structural diagram of a service system according to an embodiment of the present invention. As shown in fig. 2b, the service system is used for capacity reduction of a topology structure, and includes:

original servers

1, 2, 1., (servers that originally existed in the topology), servers to be debugged (servers that originally existed in the topology but were to be removed, may be multiple, one is taken as an example in the figure), and debuggers. The capacity reduction device may be configured to execute the processing flows shown in fig. 5 and fig. 6, which are mainly used to record the maximum location snapshot of each directory file to be reduced in the original server; meanwhile, the server to be reduced is controlled to close the writing mode, and newly generated data can not flow into the server to be reduced any more; and when the consumption site in each directory file to be reduced is greater than or equal to the snapshot of the maximum site, controlling the server to be reduced to close the read mode. At this time, the system capacity reduction is completed, and the server to be capacity reduced exits the topological structure. By effectively matching the point pointers in the directory file, the read-write mode of the capacity expansion and reduction server is controlled, and the steps of full replication, increment synchronization and write stop switching in the traditional distributed system capacity expansion mode are skipped in a phase-changing manner under the scene that the original server does not stop writing, so that the elegant capacity expansion of the sequence message is realized.

Example one

Fig. 3 is a flowchart of an embodiment of a sequential message-based capacity expansion method provided in the present invention, where an execution subject of the method may be the service system described in the foregoing embodiment. In the embodiment of the invention, the sequence message is sequentially stored in the data file, the address of the sequence message is stored in a plurality of directory files, and each directory file corresponds to one consumer. As shown in fig. 3, the capacity expansion method based on sequence messages includes the following steps:

and S301, respectively recording the maximum site snapshot of each directory file to be expanded, wherein the maximum site is used for pointing to the latest written sequence message corresponding to each directory file to be expanded.

S302, controlling the server to be expanded to start a write mode.

In this embodiment of the present invention, the steps S301 and S302 are performed simultaneously, that is, when the capacity expansion operation is started, the operations of recording the snapshot at the maximum location and starting the write mode of the server to be subjected to capacity expansion are performed simultaneously. To ensure the sequentiality of the messages, the capacity can be expanded according to a multiple relation, for example, if there are 2 servers, 2 servers are added. After the write mode of the server to be expanded is started, half of newly generated data will flow into the newly added server (the server to be expanded).

And S303, when the consumption sites in each directory file to be expanded are larger than or equal to the maximum site snapshot, controlling the server to be expanded to start a read mode, wherein the consumption sites are used for pointing to the current read sequential messages of the consumer.

When all the sequence messages before the servers to be expanded start the write mode are read by the consumers, the service system controls the servers to be expanded to start the read mode, thereby completing the expansion of the system. In this process, the original read-write mode of the server is always kept on.

The capacity expansion method based on the sequence message provided by the embodiment of the invention controls the read-write mode of the server to be expanded by effectively matching the consumption site and the maximum site pointer in the directory file, avoids the limitation caused by adopting the traditional distributed system capacity expansion technology, and realizes the elegant and smooth capacity expansion of the sequence message under the condition of not transferring data through the design idea of separating the data file and the directory file.

Example two

Fig. 4 is a flowchart of another embodiment of a method for capacity expansion based on sequence messages according to the present invention. As shown in fig. 4, on the basis of the embodiment shown in fig. 3, the capacity expansion method based on sequence messages provided in this embodiment may further include the following steps:

s401, in each directory file, comparing the size of the consumption site with the size of the minimum site, wherein the minimum site is used for pointing to the earliest written sequence message.

S402, when the consumption site is larger than or equal to the minimum site, determining the directory file as the directory file to be expanded.

In the embodiment of the invention, before the capacity expansion operation is carried out, the consumption list of the message body is firstly obtained, and whether the capacity expansion condition is met by a consumer is checked. The message data has a Time To Live (TTL), and as Time increases, the earliest written data is continuously discarded, so the current minimum point gradually increases. If the consumption site of the consumer is smaller than the minimum site, the sequential message which is currently read by the consumer is discarded, the sequential message belongs to invalid consumption, and the consumer cannot be included in the final expansion list. Only the directory file with the consumption site greater than or equal to the minimum site is determined as the directory file to be expanded.

And S403, recording the maximum site snapshot of each directory file to be expanded.

S404, controlling the server to be expanded to start a write mode.

S405, start a timer.

In the embodiment of the present invention, the above steps S403, S404 and S405 are performed simultaneously.

S406, when the consumption sites in each directory file to be expanded are larger than or equal to the maximum site snapshot, controlling the server to be expanded to start a read mode.

And S407, when the timing time of the timer reaches a preset time threshold, controlling the server to be expanded to start a read mode.

Before the timing time reaches a preset time threshold, if all consumers finish the progress (as the consumers continuously read the sequence information, consumption sites of the consumers gradually increase to a maximum site snapshot), starting a reading mode by the server to be expanded, and completing the expansion operation; if the timing time reaches the preset time threshold, it indicates that the consumer cannot catch up with the progress within the specified time, and in order to smoothly expand the capacity, the server to be expanded may be directly controlled to start the read mode to complete the expansion, or may be manually intervened to perform manual operation.

According to the capacity expansion method based on the sequence message, provided by the embodiment of the invention, the directory file meeting the capacity expansion condition is selected through effective matching among the consumption site, the minimum site and the maximum site pointer in the directory file, and the read-write mode of the server to be expanded is controlled, so that the limitation caused by the adoption of the traditional distributed system capacity expansion technology is avoided, and the elegant and smooth capacity expansion of the sequence message is realized through the design idea of separating the data file from the directory file under the condition of not transferring data.

EXAMPLE III

Fig. 5 is a flowchart of an embodiment of a method for capacity reduction based on sequence messages, where an execution subject of the method may be a service system described in the above embodiment. In the embodiment of the invention, the sequence message is sequentially stored in the data file, the address of the sequence message is stored in a plurality of directory files, and each directory file corresponds to one consumer. As shown in fig. 5, the method for capacity reduction based on sequence messages includes the following steps:

s501, respectively recording the maximum position point snapshot of each directory file to be reduced, wherein the maximum position point is used for pointing to the latest written sequence information corresponding to each directory file to be reduced.

And S502, controlling the server to be condensed to close the writing mode.

In the embodiment of the present invention, the above steps S501 and S502 are performed simultaneously, that is, when the capacity reduction operation is started, the operations of recording the snapshot of the maximum location and closing the write mode of the server to be reduced are performed simultaneously. In order to ensure the sequentiality of the messages, the capacity reduction can be performed according to a multiple relation, for example, a double relation is taken as an example, 4 servers are originally provided, and 2 servers are reduced. After closing the write mode of the server to be condensed, the newly generated data will flow entirely into the remaining servers (i.e., the origin servers).

S503, when the consumption sites in each directory file to be reduced are larger than or equal to the maximum site snapshot, controlling the server to be reduced to close the reading mode, wherein the consumption sites are used for pointing to the current reading sequence information of the consumer.

When all the sequential messages before the writing mode of the servers to be reduced is closed are read by the consumers, the service system controls the servers to be reduced to close the reading mode, and the servers to be reduced are offline, so that the reduction of the system is completed. In this process, the original server's read-write mode remains on all the time.

According to the capacity reduction method based on the sequence message, the read-write mode of the server to be reduced is controlled through effective cooperation between the consumption site and the maximum site pointer in the directory file, the limitation caused by the adoption of the traditional distributed system capacity reduction technology is avoided, and through the design idea of separating the data file from the directory file, elegant and smooth capacity reduction of the sequence message is achieved under the condition that data are not migrated.

Example four

Fig. 6 is a flowchart of another embodiment of a method for capacity reduction based on sequence messages according to the present invention. As shown in fig. 6, on the basis of the embodiment shown in fig. 5, the method for capacity reduction based on sequence messages provided in this embodiment may further include the following steps:

s601, comparing the size of the consumption site with the size of the minimum site in each directory file, where the minimum site is used to point to the earliest written sequence message.

S602, when the consumption site is larger than or equal to the minimum site, determining the directory file as the directory file to be contracted.

In the embodiment of the invention, before the capacity reduction operation is carried out, the consumption list of the message body is firstly obtained, and whether the consumer meets the capacity reduction condition is checked. Message data has TTL, and as time increases, the oldest written data is continuously discarded, and thus, the current minimum position gradually increases. If the consumption site of the consumer is smaller than the minimum site, the sequential message which is currently read by the consumer is discarded and belongs to invalid consumption, and the consumer cannot be included in the final abbreviated list. Only directory files whose consumption sites are greater than or equal to the minimum site are determined as directory files to be contracted.

S603, recording the maximum locus snapshot of each directory file to be contracted.

And S604, controlling the server to be condensed to close the writing mode.

S605 starts a timer.

In the embodiment of the present invention, the above steps S603, S604 and S605 are performed simultaneously.

S606, when the consumption sites in the directory files to be reduced are larger than or equal to the snapshot of the maximum site, the server to be reduced is controlled to close the reading mode.

And S607, controlling the server to be contracted to close the reading mode when the timing time of the timer reaches the preset time threshold.

Before the timing time reaches a preset time threshold, if all consumers finish the progress (as the consumers continuously read the sequence messages, consumption sites of the consumers gradually increase to the snapshot of the maximum site), the server to be subjected to capacity reduction closes the reading mode, and the capacity reduction operation is finished; if the timing time reaches the preset time threshold, the fact that the consumer cannot catch up with the progress within the specified time is indicated, in order to smoothly reduce the capacity, the server to be reduced can be directly controlled to close the reading mode, the capacity reduction is completed, and manual intervention and manual operation can be carried out.

According to the capacity reduction method based on the sequence message, provided by the embodiment of the invention, the directory file meeting the capacity reduction condition is selected through effective matching among the consumption site, the minimum site and the maximum site pointer in the directory file, and the read-write mode of the server to be subjected to capacity reduction is controlled, so that the limitation caused by the adoption of the traditional distributed system capacity reduction technology is avoided, and the elegant and smooth capacity reduction of the sequence message is realized through the design idea of separating the data file from the directory file under the condition of not transferring data.

EXAMPLE five

Fig. 7 is a schematic structural diagram of an embodiment of a capacity expansion apparatus based on sequence messages, which may be used to execute the method steps shown in fig. 3. In the embodiment of the invention, the sequence message is sequentially stored in the data file, the address of the sequence message is stored in a plurality of directory files, and each directory file corresponds to one consumer. As shown in fig. 7, the apparatus may include: a first recording module 71, a first control module 72 and a second control module 73.

The first recording module 71 is configured to record a snapshot of a maximum location point of each directory file to be expanded, where the maximum location point is used to point to a sequence message written latest in the directory file to be expanded; the first control module 72 is configured to control the server to be expanded to start a write mode while the first recording module 71 records the snapshot of the maximum site; the second control module 73 is configured to control the server to be expanded to start a read mode when consumption sites in each directory file to be expanded are greater than or equal to the maximum site snapshot, where the consumption sites are used to point to a sequential message currently read by a consumer.

In the embodiment of the present invention, the first recording module 71 and the first control module 72 operate simultaneously. Namely, when the capacity expansion operation is started, the operations of recording the snapshot at the maximum site and starting the write mode of the server to be subjected to capacity expansion are executed at the same time. In order to ensure the sequentiality of the messages, the capacity expansion should be performed according to the multiple relation. After the first control module 72 starts the write mode of the server to be expanded, half of the newly generated data will flow into the newly added server (the server to be expanded). When all the sequence messages before the servers to be expanded start the write mode are read by the consumers, the second control module 73 controls each server to be expanded to start the read mode, thereby completing the expansion of the system. In this process, the original server's read-write mode remains on all the time.

The capacity expansion device based on the sequence message provided by the embodiment of the invention controls the read-write mode of the server to be expanded by effectively matching the consumption site pointer and the maximum site pointer in the directory file, avoids the limitation caused by adopting the traditional distributed system capacity expansion technology, and realizes the elegant and smooth capacity expansion of the sequence message under the condition of not transferring data through the design idea of separating the data file and the directory file.

EXAMPLE six

Fig. 8 is a schematic structural diagram of another embodiment of a capacity expansion apparatus based on sequence messages, which may be used to execute the method steps shown in fig. 4. As shown in fig. 8, on the basis of the embodiment shown in fig. 7, the capacity expansion device based on sequence messages according to an embodiment of the present invention may further include: a first comparison block 81 and a first processing block 82.

The first comparing module 81 is configured to compare the size of a consumption site with the size of a minimum site in each directory file before the first recording module 71 records the snapshot of the maximum site of each directory file to be expanded, where the minimum site is used to point to the earliest written sequence message; the first processing module 82 is configured to determine the directory file as a directory file to be expanded when the consumption location is greater than or equal to the minimum location.

In the embodiment of the present invention, before performing the capacity expansion operation, the consumption list of the message body is first obtained, and the first comparison module 81 checks whether the consumer meets the capacity expansion condition. Message data has TTL, and as time increases, the oldest written data is discarded continuously, and thus the current minimum point gradually increases. If the consumption site of the consumer is smaller than the minimum site, the sequential message which is currently read by the consumer is discarded, the sequential message belongs to invalid consumption, and the consumer cannot be included in the final expansion list. Only when the consumption site is greater than or equal to the minimum site, the first processing module 82 determines the directory file as the directory file to be expanded.

Further, the capacity expansion device based on the sequence message provided in the embodiment of the present invention may further include: a first timing module 83 and a third control module 84.

The first timing module 83 is configured to start a timer when the first control module 72 controls the server to be expanded to start a write mode; the third control module 84 is configured to control the server to be expanded to start the read mode when the timing time of the timer reaches a preset time threshold.

Before the timing time reaches the preset time threshold, if all consumers finish the progress (the consumption sites of the consumers gradually increase to the snapshot of the maximum site along with the continuous reading of the sequence messages by the consumers), the second control module 73 controls the server to be expanded to start the reading mode, and the expansion operation is finished; if the timing time reaches the preset time threshold, it indicates that the consumer cannot catch up with the progress within the specified time, and in order to smoothly expand the capacity, the third control module 84 may directly control the server to be expanded to start the read mode to complete the expansion, or may manually intervene to manually operate.

The capacity expansion device based on the sequence message selects the directory file meeting the capacity expansion condition through the effective matching among the consumption site, the minimum site and the maximum site pointer in the directory file, and controls the read-write mode of the server to be subjected to capacity expansion, thereby avoiding the limitation caused by adopting the traditional distributed system capacity expansion technology, and realizing the elegant and smooth capacity expansion of the sequence message under the condition of not migrating data through the design idea of separating the data file and the directory file.

EXAMPLE seven

Fig. 9 is a schematic structural diagram of an embodiment of a sequential message-based capacity reduction apparatus according to the present invention, which can be used to execute the method steps shown in fig. 5. In the embodiment of the invention, the sequence message is sequentially stored in the data file, the address of the sequence message is stored in a plurality of directory files, and each directory file corresponds to one consumer. As shown in fig. 9, the apparatus may include: a second recording module 91, a fourth control module 92 and a fifth control module 93.

The second recording module 91 is configured to record a snapshot of a maximum location point of each directory file to be reduced, where the maximum location point is used to point to a latest written sequence message of the directory file to be reduced; the fourth control module 92 is configured to control the server to be reduced to close the write mode while the second recording module 91 records the snapshot of the maximum location; the fifth control module 93 is configured to control the server to be reduced to close the read mode when the consumption sites in each directory file to be reduced are greater than or equal to the maximum site snapshot, where the consumption sites are used to point to the sequential message currently read by the consumer.

In the embodiment of the present invention, the second recording module 91 and the fourth control module 92 operate simultaneously. That is, when the capacity reduction operation is started, the operations of recording the snapshot of the maximum location and closing the write mode of the server to be capacity reduced are executed simultaneously. In order to ensure the sequentiality of the messages, the reduction capacity should be carried out according to the multiple relation. After the fourth control module 92 closes the write mode of the server to be condensed, the newly generated data will all flow into the remaining servers (origin servers). When all the sequential messages before the servers to be reduced close the write mode have been read by the consumer, the fifth control module 93 controls each server to be reduced to close the read mode, thereby completing the reduction of the system. In this process, the original server's read-write mode remains on all the time.

The capacity reduction device based on the sequence message provided by the embodiment of the invention controls the read-write mode of the server to be reduced through the effective matching between the consumption site and the maximum site pointer in the directory file, avoids the limitation caused by adopting the traditional distributed system capacity reduction technology, and realizes the elegant and smooth capacity reduction of the sequence message under the condition of not transferring data through the design idea of separating the data file and the directory file.

Example eight

Fig. 10 is a schematic structural diagram of another embodiment of a sequence-message-based capacity reduction apparatus according to the present invention, which can be used to execute the steps of the method shown in fig. 6. As shown in fig. 10, on the basis of the embodiment shown in fig. 9, the apparatus for scaling based on sequence messages according to the embodiment of the present invention may further include: a second comparing module 101 and a second processing module 102.

The second comparing module 101 is configured to compare the size of a consumption site with the size of a minimum site in each directory file before the second recording module 91 records the snapshot of the maximum site of each directory file to be contracted, respectively, where the minimum site is used to point to the earliest written sequence message; the second processing module 102 is configured to determine the directory file as a directory file to be condensed when the consumption location is greater than or equal to the minimum location.

In the embodiment of the present invention, before performing a capacity reduction operation, a consumption list of a message body is first obtained, and the second comparison module 101 checks whether a consumer meets a capacity reduction condition. Message data has TTL, and as time increases, the oldest written data is continuously discarded, and thus, the current minimum position gradually increases. If the consumption site of the consumer is smaller than the minimum site, the sequential message which is currently read by the consumer is discarded and belongs to invalid consumption, and the consumer cannot be included in the final abbreviated list. The second processing module 102 determines the directory file as the directory file to be contracted only if the consumption location is greater than or equal to the minimum location.

Further, the apparatus for scaling based on sequence message provided in the embodiment of the present invention may further include: a second timing module 103 and a sixth control module 104.

The second timing module 103 is configured to start a timer when the fourth control module 92 controls the server to be reduced to close the write mode; the sixth control module 104 is configured to control the server to be reduced to close the reading mode when the timing time of the timer reaches a preset time threshold.

Before the timing time reaches the preset time threshold, if all consumers finish the progress (as the consumers continuously read the sequence messages, consumption sites of the consumers gradually increase to the snapshot of the maximum site), the fifth control module 93 controls the server to be reduced to close the reading mode, and the reduction operation is finished; if the timing time reaches the preset time threshold, it indicates that the consumer cannot catch up with the progress within the specified time, and in order to smoothly reduce the capacity, the sixth control module 104 may directly control the server to be reduced to close the reading mode, so as to complete the reduction, or may manually intervene, and perform the operation manually.

According to the capacity reduction device based on the sequence message, provided by the embodiment of the invention, the directory file meeting the capacity reduction condition is selected through effective matching among the consumption site, the minimum site and the maximum site pointer in the directory file, and the read-write mode of the server to be subjected to capacity reduction is controlled, so that the limitation caused by the adoption of the traditional distributed system capacity reduction technology is avoided, and through the design idea of separating the data file from the directory file, the elegant and smooth capacity reduction of the sequence message is realized under the condition of not transferring data.

Example nine

The internal functions and structures of the sequential message-based capacity expansion device, which may be implemented as an electronic device, are described above. Fig. 11 is a schematic structural diagram of an embodiment of an electronic device provided in the present invention. As shown in fig. 11, the electronic device includes a memory 111 and a processor 112.

The memory 111 stores programs. In addition to the above programs, the memory 111 may also be configured to store other various data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and the like.

The memory 111 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

A processor 112, coupled to the memory 111, for executing the program stored in the memory 111 to:

respectively recording the maximum site snapshot of each directory file to be expanded, wherein the maximum site is used for pointing to the latest written sequence message of the directory file to be expanded, the sequence message is sequentially stored in a data file, the address of the sequence message is stored in a plurality of directory files, and each directory file corresponds to a consumer; meanwhile, controlling a server to be expanded to start a write mode; and when the consumption site in each directory file to be expanded is greater than or equal to the maximum site snapshot, controlling the server to be expanded to start a read mode, wherein the consumption site is used for pointing to the current read sequential message of the consumer.

The above specific processing operations have been described in detail in the foregoing embodiments, and are not described again here.

Further, as shown in fig. 11, the electronic device may further include: communication components 113, power components 114, audio components 115, display 116, and other components. Only some of the components are schematically shown in fig. 11, and it is not meant that the electronic device includes only the components shown in fig. 11.

The communication component 113 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 113 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 113 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

Based on the communication component 113, the processor 112 may store all consumption data in an external database through the communication component 113.

A power supply component 114 that provides power to the various components of the electronic device. The power components 114 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for an electronic device.

Audio component 115 is configured to output and/or input audio signals. For example, audio component 115 may include a Microphone (MIC) configured to receive external audio signals when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the memory 111 or transmitted via the communication component 113. In some embodiments, audio component 115 also includes a speaker for outputting audio signals.

The display 116 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

Example ten

The internal functions and structures of the sequential message-based capacity reduction apparatus, which can be implemented as an electronic device, are described above. Fig. 12 is a schematic structural diagram of another embodiment of an electronic device provided in the present invention. As shown in fig. 12, the electronic device includes a memory 121 and a processor 122.

The memory 121 stores programs. In addition to the above-described programs, the memory 121 may also be configured to store other various data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and the like.

The memory 121 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

A processor 122, coupled to the memory 121, for executing the program stored in the memory 121 to:

respectively recording the maximum site snapshot of each directory file to be reduced, wherein the maximum site is used for pointing to the latest written sequence message of the directory file to be reduced, the sequence message is sequentially stored in a data file, the address of the sequence message is stored in a plurality of directory files, and each directory file corresponds to a consumer; meanwhile, controlling the server to be contracted to close the writing mode; and when the consumption sites in the directory files to be reduced are larger than or equal to the maximum site snapshot, controlling the server to be reduced to close the reading mode, wherein the consumption sites are used for pointing to the current reading sequence message of the consumer.

Further, as shown in fig. 12, the electronic device may further include: communication components 123, power components 124, audio components 125, display 126, and other components. Only some of the components are schematically shown in fig. 12, and the electronic device is not meant to include only the components shown in fig. 12.

The communication component 123 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 123 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 123 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

Based on the communication component 123, the processor 122 can store all consumption data in an external database through the communication component 123.

A power supply component 124 that provides power to the various components of the electronic device. The power components 124 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for an electronic device.

Audio component 125 is configured to output and/or input audio signals. For example, the audio component 125 includes a Microphone (MIC) configured to receive external audio signals when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the memory 121 or transmitted via the communication component 123. In some embodiments, audio component 125 also includes a speaker for outputting audio signals.

The display 126 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and these modifications or substitutions do not depart from the spirit of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A method for capacity expansion based on sequential messages, wherein the sequential messages are sequentially stored in data files, addresses of the sequential messages are stored in a plurality of directory files, and each directory file corresponds to a consumer, the method comprising:

respectively recording the maximum site snapshot of each directory file to be expanded, wherein the maximum site of each directory file to be expanded is used for pointing to the latest written sequence message corresponding to the maximum site;

meanwhile, controlling a server to be expanded to start a write mode;

and when consumption sites in each directory file to be expanded are larger than or equal to the maximum site snapshot, controlling the server to be expanded to start a read mode, wherein the consumption sites are used for pointing to the sequential messages currently read by the consumer.

2. The method according to claim 1, further comprising, before the step of recording the snapshot of the maximum location of each directory file to be expanded, respectively:

in each of said directory files, comparing the size of said consumption site with a minimum site, said minimum site for pointing to said sequential message written earliest;

and when the consumption site is greater than or equal to the minimum site, determining the directory file as the directory file to be expanded.

3. The method for capacity expansion based on sequence message as claimed in claim 1, further comprising:

the method comprises the steps that a timer is started while a server to be expanded is controlled to start a write mode;

and when the timing time of the timer reaches a preset time threshold, controlling the server to be expanded to start a read mode.

4. A method for capacity reduction based on sequential messages, wherein the sequential messages are sequentially stored in data files, and addresses of the sequential messages are stored in a plurality of directory files, each directory file corresponding to a consumer, the method comprising:

respectively recording the maximum site snapshot of each directory file to be reduced, wherein the maximum site of each directory file to be reduced is used for pointing to the latest written sequence message corresponding to the maximum site;

meanwhile, controlling the server to be reduced to close the writing mode;

and when the consumption site in each directory file to be reduced is greater than or equal to the maximum site snapshot, controlling the server to be reduced to close the reading mode, wherein the consumption site is used for pointing to the sequential message currently read by the consumer.

5. The method according to claim 4, further comprising, before said respectively recording the snapshot of the maximum location of each directory file to be condensed:

and when the consumption site is greater than or equal to the minimum site, determining the directory file as the directory file to be contracted.

6. The method of claim 4, further comprising:

the method comprises the steps that a timer is started while a server to be subjected to capacity reduction is controlled to close a writing mode;

and when the timing time of the timer reaches a preset time threshold value, controlling the server to be contracted to close the reading mode.

7. A capacity expansion apparatus based on sequential messages, wherein the sequential messages are sequentially stored in data files, addresses of the sequential messages are stored in a plurality of directory files, and each directory file corresponds to a consumer, the apparatus comprising:

the first recording module is used for respectively recording the maximum site snapshots of the directory files to be expanded, and the maximum site of each directory file to be expanded is used for pointing to the latest written sequence message corresponding to the maximum site;

the first control module is used for controlling the server to be expanded to start a write mode while the first recording module records the maximum site snapshot;

and the second control module is used for controlling the server to be expanded to start a reading mode when consumption sites in each directory file to be expanded are larger than or equal to the maximum site snapshot, wherein the consumption sites are used for pointing to the sequential messages currently read by the consumer.

8. A sequential message based capacity reduction apparatus, wherein the sequential message is stored in a data file in sequence, and wherein addresses of the sequential message are stored in a plurality of directory files, each of the directory files corresponding to a consumer, the apparatus comprising:

the second recording module is used for respectively recording the maximum site snapshots of the directory files to be reduced, and the maximum site of each directory file to be reduced is used for pointing to the latest written sequence message corresponding to the maximum site;

the fourth control module is used for controlling the server to be subjected to capacity reduction to close a write mode while the second recording module records the maximum site snapshot;

and a fifth control module, configured to control the server to be reduced to close a read mode when consumption sites in each directory file to be reduced are greater than or equal to the maximum site snapshot, where the consumption sites are used to point to the sequential messages currently read by the consumer.

9. An electronic device, comprising:

a memory for storing a program;

a processor for executing the program stored in the memory for:

respectively recording the maximum site snapshot of each directory file to be expanded, wherein the maximum site of each directory file to be expanded is used for pointing to the latest written sequence message corresponding to the maximum site, the sequence message is sequentially stored in a data file, the address of the sequence message is stored in a plurality of directory files, and each directory file corresponds to a consumer;

meanwhile, controlling a server to be expanded to start a write mode;

10. An electronic device, comprising:

a memory for storing a program;

a processor for executing the program stored in the memory for:

respectively recording the maximum site snapshots of the directory files to be reduced, wherein the maximum site of each directory file to be reduced is used for pointing to the latest written sequence message corresponding to the maximum site, the sequence message is sequentially stored in a data file, the addresses of the sequence message are stored in a plurality of directory files, and each directory file corresponds to a consumer;

meanwhile, controlling the server to be reduced to close the writing mode;

and when the consumption site in each directory file to be reduced is larger than or equal to the maximum site snapshot, controlling the server to be reduced to close a reading mode, wherein the consumption site is used for pointing to the sequential message currently read by the consumer.