WO2024000987A1 - Data storage method, server and storage medium - Google Patents

Abstract

Provided in the present application are a data storage method, a server and a storage medium. The method comprises: a server being able to acquire the number of system users, an estimated number of participating users for the current activity, and the number of items, in which users are allowed to participate, of the current activity; the server being able to calculate the ratio of the number of system users to the estimated number of participating users; when the ratio is greater than a preset threshold value, the server being able to store participation information regarding the current activity by means of a bitmap; the server being able to perform calculation according to the number of system users, the estimated number of participating users, and the number of items in which users are allowed to participate, so as to obtain the number of target fragments of each bitmap; and the server being able to fragment each bitmap according to the number of target fragments, and to record participation information of the system users in the current activity by using the fragmented bitmaps. By means of the method of the present application, the utilization rate of a storage space is improved, thereby improving the data storage and reading efficiency.

Description

Data storage methods, servers and storage media

This application claims priority to the Chinese patent application filed with the China Patent Office on June 28, 2022, with the application number CN202210740249.0 and the application name "Data storage method, server and storage medium", the entire content of which is incorporated by reference in in this application.

Technical field

The present application relates to the field of computers, and in particular, to a data storage method, server and storage medium.

Background technique

With the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech), and marketing activities are no exception. However, due to the huge number of system users in the financial industry, higher requirements are placed on technology. In marketing activities, electronic rights replace original paper rights and can be promoted and used more conveniently. At the same time, the data management background can also record the number of participants in marketing activities and the distribution of rights and interests.

In the existing technology, the data management background can use bitmap algorithms to record marketing activity information. For example, the data management backend can create a bitmap for each interest. The data management background can set the size of each bitmap according to the number of system users. Each system user can correspond to one bit in a bitmap. When user A in the system acquires equity B, the data management background can set the value of the bit corresponding to user A in the bitmap corresponding to equity B to 1 to realize the record of user A acquiring equity B.

However, the size of each bitmap in this bitmap algorithm usually needs to be set according to the number of system users. When the number of system users is very large, but the number of users actually participating in the marketing activity is relatively small, it is easy to waste a large amount of data space in each bitmap, and there is a problem of low data storage efficiency.

Contents of the invention

This application provides a data storage method, server and storage medium to solve the problem that the number of participating users actually participating in the marketing activity is relatively small, which easily leads to the waste of a large amount of data space in each bitmap and low data storage efficiency. question.

In the first aspect, this application provides a data storage method, including:

Obtain the number of system users, the estimated number of participating users in the current activity, and the number of projects allowed to participate in the current activity; among them, each project in the current activity corresponds to a bitmap;

When the ratio between the number of system users and the estimated number of participating users is greater than the preset threshold, the target number of shards for each bitmap is determined based on the number of system users, the estimated number of participating users, and the number of allowed participating projects. ;

The bitmap is fragmented according to the target number of fragments, and the fragmented bitmap is used to record the participation information of the system user in the current activity.

Optionally, determining the target number of shards for each bitmap based on the number of system users, the estimated number of participating users, and the number of allowed participating projects, specifically includes:

Determine the range of the number of shards based on the estimated number of participating users;

According to the range of the number of fragments, calculate the bitmap space occupation corresponding to each number of fragments within the range;

Compare the bitmap space occupancy corresponding to each of the number of fragments, and determine the number of fragments corresponding to the minimum value of the bitmap space occupancy as the target number of fragments.

Optionally, calculating the bitmap space occupancy corresponding to each number of fragments within the range specifically includes:

According to the number of system users, the estimated number of participating users, the number of allowed participation items, and the number of slices, the bitmap space occupation corresponding to the number of slices is determined.

Optionally, determining the bitmap space occupancy corresponding to the number of slices based on the number of system users, the estimated number of participating users, the number of allowed participation projects, and the number of slices, specifically includes: : Determine the total length of keywords for each bitmap according to the number of items allowed to participate, the number of fragments and the fixed length of keywords; wherein each fragment corresponds to one keyword;

According to the average fragment length of each fragment, the probability value of each fragment occupying fifty percent of the storage space, the average storage space value saved by each fragment and the number of fragments , determine the space occupied by all the fragments of each bitmap;

The bitmap space occupancy is determined based on the total length of the keywords of each bitmap, the space occupancy of all the slices of each bitmap, and the number of allowed participating items.

Optionally, recording the system user's participation information in the current activity specifically includes:

Determine the number of participating projects of the system user according to the value of the system user in each bitmap slice;

When the number of participating projects is less than the number of allowed participating projects, the value of the system user in the slice of the bitmap is modified from the first data to the second data; when the number of participating projects is greater than or equal to the number of allowed participating projects when, denying the system user to continue to participate in the current activity.

Optionally, the bitmap is fragmented according to the target number of fragments, and the fragmented bitmap is used to record the participation information of the system user in the current activity. Methods also include:

Determine the bitmap space occupancy according to the number of system users, the estimated number of participating users, the number of allowed participating projects, and the number of target shards;

Determine the space occupied by key-value pairs based on the number of system users, the estimated number of participating users, and the number of allowed participating projects;

When the space occupied by the key-value pair is less than the space occupied by the bitmap, the key-value pair is used to record the participation information of the system user in the current activity.

Optionally, the method also includes:

Obtain the number of participating projects of the system user recorded in the key-value pair corresponding to the system user;

When the number of participating projects is less than the number of allowed participating projects, increase the number of participating projects of the system user by a unit value;

When the number of participating projects is greater than or equal to the number of allowed participating projects, the system user is refused to continue to participate in the current activity.

In a second aspect, this application provides a data storage device, including:

The acquisition module is used to obtain the number of system users, the estimated number of participating users in the current activity, and the number of projects allowed to participate in the current activity; where each project in the current activity corresponds to a bitmap;

A processing module configured to determine each bitmap based on the number of system users, the estimated number of participating users, and the number of allowed participating projects when the ratio between the number of system users and the estimated number of participating users is greater than a preset threshold. The target number of fragments; fragment the bitmap according to the target number of fragments, and use the fragmented bitmap to record the participation information of system users in the current activity.

Optionally, the processing module is specifically used for:

Determine the range of the number of shards based on the estimated number of participating users;

According to the range of the number of fragments, calculate the bitmap space occupation corresponding to each number of fragments within the range;

Compare the bitmap space occupancy corresponding to each of the number of fragments, and determine the number of fragments corresponding to the minimum value of the bitmap space occupancy as the target number of fragments.

Optionally, the processing module is specifically used for:

According to the number of system users, the estimated number of participating users, the number of allowed participation items, and the number of slices, the bitmap space occupation corresponding to the number of slices is determined.

Optionally, the processing module is specifically used for:

Determine the total length of keywords for each bitmap according to the number of items allowed to participate, the number of fragments and the fixed length of keywords; wherein each fragment corresponds to one keyword;

According to the average fragment length of each fragment, the probability value of each fragment occupying fifty percent of the storage space, the average storage space value saved by each fragment and the number of fragments , determine the space occupied by all the fragments of each bitmap;

The bitmap space occupancy is determined based on the total length of the keywords of each bitmap, the space occupancy of all the slices of each bitmap, and the number of allowed participating items.

Optionally, the processing module is specifically used for:

Determine the number of participating projects of the system user according to the value of the system user in each bitmap slice;

When the number of participating projects is less than the number of allowed participating projects, modify the value of the system user in the slice of the bitmap from the first data to the second data;

When the number of participating projects is greater than or equal to the number of allowed participating projects, the system user is refused to continue to participate in the current activity.

Optionally, the bitmap is fragmented according to the target number of fragments, and the fragmented bitmap is used to record the participation information of the system user in the current activity. Processing module, also used for:

Determine the bitmap space occupancy according to the number of system users, the estimated number of participating users, the number of allowed participating projects, and the number of target shards;

Determine the space occupied by key-value pairs based on the number of system users, the estimated number of participating users, and the number of allowed participating projects;

When the space occupied by the key-value pair is less than the space occupied by the bitmap, the key-value pair is used to record the participation information of the system user in the current activity.

Optionally, the processing module is also used to:

Obtain the number of participating projects of the system user recorded in the key-value pair corresponding to the system user;

When the number of participating projects is less than the number of allowed participating projects, increase the number of participating projects of the system user by a unit value;

When the number of participating projects is greater than or equal to the number of allowed participating projects, the system user is refused to continue to participate in the current activity.

In a third aspect, this application provides a server, including: a memory and a processor;

The memory is used to store a computer program; the processor is used to execute the data storage method in the first aspect and any possible design of the first aspect according to the computer program stored in the memory.

In a fourth aspect, the present application provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium. When at least one processor of the server executes the computer program, the server executes the first aspect and any one of the first aspects. possible data storage methods in the design.

In a fifth aspect, the present application provides a computer program product. The computer program product includes a computer program. When at least one processor of the server executes the computer program, the server executes the first aspect and any possible design of the first aspect. The data storage method in .

The data storage method provided by this application is to obtain the number of system users; determine the estimated number of participating users based on the current activity; obtain the number of projects allowed to participate in the current activity, each project in the current activity corresponds to a bitmap; calculate the number of system users The ratio to the estimated number of participating users; use this ratio to compare with the preset threshold; when the ratio is greater than the preset threshold, use a bitmap to store the participation information of the current activity; according to the number of system users, the estimated number of participating users and the number of projects allowed to participate in, calculate the target number of shards for each bitmap; fragment the bitmap according to the number of target shards, and use the fragmented bitmap to record the system user's participation information in the current activity , to achieve the effect of improving the utilization of storage space, avoiding too concentrated access to storage space, and improving the efficiency of data storage and reading.

Description of drawings

In order to explain the technical solutions in this application or the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are of the present invention. For some embodiments of the application, those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of a bitmap provided by an embodiment of the present application;

Figure 2 is a schematic bitmap diagram of a right provided by an embodiment of the present application;

Figure 3 is a schematic diagram of a scenario for issuing activity rights according to an embodiment of the present application;

Figure 4 is a flow chart of a data storage method provided by an embodiment of the present application;

Figure 5 is a flow chart of a data storage method provided by an embodiment of the present application;

Figure 6 is a flow chart of a data storage method provided by an embodiment of the present application;

Figure 7 is a flow chart of a data storage process provided by an embodiment of the present application;

Figure 8 is a schematic structural diagram of a data storage device provided by an embodiment of the present application;

Figure 9 is a schematic diagram of the hardware structure of a server provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application. , not all examples. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

The terms "first", "second", "third", "fourth", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe specific objects. Sequence or sequence. It should be understood that data so used are interchangeable where appropriate. For example, without departing from the scope of this article, the first information may also be called second information, and similarly, the second information may also be called first information.

Depending on the context, the word "if" as used herein may be interpreted as "when" or "when" or "in response to determining."

Furthermore, as used herein, the singular forms "a," "an," and "the" may also be used to describe the plural forms unless the context dictates otherwise.

It should be further understood that the terms "comprising" and "including" indicate the presence of features, steps, operations, elements, components, items, categories, and/or groups, but do not exclude one or more other features, steps, operations, elements, The presence, occurrence, or addition of components, items, categories, and/or groups.

The terms "or" and "and/or" as used herein are to be construed as inclusive or to mean any one or any combination. Therefore, "A, B or C" or "A, B and/or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C" . Exceptions to this definition occur only when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some manner.

With the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech), and marketing activities are no exception. However, due to the huge number of system users in the financial industry, higher requirements are placed on technology. In marketing activities, using electronic rights and interests instead of original paper rights can not only be more conveniently promoted and used, but also more accurately control the distribution of rights, thereby accurately controlling marketing costs. Among them, benefits can include coupons, discount coupons, membership card discounts, etc. For example, a data management backend can be set up in the server. The data management backend can record information on the number of users participating in the activity, the rights issuance of the activity, the number of rights obtained by each user, and other dimensions. Among them, the record of the number of rights and interests obtained by each user can also be verified to verify the maximum number of rights and interests that each user can obtain in this activity, thereby ensuring that the number of rights and interests obtained by the user in this activity is less than the preset upper limit.

In the existing technology, in order to accurately control the number of rights issued, the server can use a redis cluster to store the number of rights each user has obtained. At each time the rights are issued, the server can query the number of rights the user has obtained stored in redis, and perform verification based on the number of rights. If the number of rights is less than the preset upper limit, the server can send the rights to the user. Otherwise, the user will no longer be able to obtain this benefit. The server can use key-value pairs to store the number of rights and interests that each user has obtained in redis. In the key-value data structure, each user needs a key storage space and a value storage space. The key may include the user's user ID and other information used to uniquely identify the user. The length of Key is usually fixed. Usually a key occupies about 50 bytes of storage space. The number of users participating in an activity is usually proportional to the space required to store user participation information. When there are a large number of users, assuming there are 100 million users, and using the key-value data structure, the space occupied by the keys of all users can reach:

100,000,000×50(byte)÷1024÷1024≈4768(M)

Therefore, compared to using key-value data structures, bitmap bitmap can better save storage space when facing a large number of users. In the bitmap, each user needs to correspond to one bit (1bit) of storage space. The 1-bit storage space can be set with a value of 0 or 1. The 1-bit storage space can be used to indicate whether the user has obtained the right. Assuming there are 100 million users, the server can allocate 100 million bits of storage space to store whether the 100 million users have obtained the rights. As shown in Figure 1, each grid can correspond to 1 bit. Among them, the last grid corresponds to the 100 millionth bit. The storage space occupied by this bitmap is:

100,000,000(bit)÷8÷1024÷1024≈12(M)

That is, when there are 100 million users, the storage space required for a bitmap is 12 MB. When the preset upper limit of rights obtained by each user in this marketing activity is N, N bitmaps can be set in the server. That is, its space occupation is 12N trillion. For example, as shown in Figure 2, the first coupon, the second coupon and the third coupon respectively correspond to 3 bitmaps of three rights. When a user with an offset value of 1 obtains three benefits, the values of the grids with an offset value of 1 in the three bitmaps are all 1. When a user with an offset value of 4 obtains 2 interests, two of the three grids with an offset value of 4 in the three bitmaps have a value of 1.

However, when there are a large number of system users but the number of users actually participating in the activity is small, the key-value data structure can save more space. Because in the bitmap, once a user with a large ID participates in an activity, the size of the bitmap needs to be set accordingly according to the user ID. That is, the memory size occupied by the bitmap is determined by the maximum offset. Even if the bitmap only stores information about one user, if the user ID is large, it will still take up a lot of space. For example, when the 100 millionth user participates in the activity, as shown in Figure 1, the size of the bitmap is 12M. In extreme cases, assuming that only the 100 millionth user participates in the activity, a large amount of the 12M storage space will be wasted. The key-value data structure is different. The server can create a key-value pair corresponding to a user when he or she participates in an activity. That is, when only the 100 millionth user participates in the activity, the server only needs to use 50 bytes of storage space. It can be seen that the number of users in the system is very large, but the number of users who actually participated in the marketing activity is relatively small, which easily leads to the waste of a large amount of data space in each bitmap and the problem of low data storage efficiency.

In response to the above problems, this application proposes a data storage method based on bitmap sharding. The server can divide a bitmap into L slices, and the offset of each slice is the maximum offset/L. Moreover, the participation information of users from the 1st to the maximum offset/L will be stored on the first shard, and the participation information of the users from the 1st to the maximum offset/L+1 to 2 (maximum offset/L) will be stored on the second shard, and so on. For example, when the number of system users is 100 million and only the 100 millionth user participates in the activity, only the bit corresponding to the maximum offset of the last shard is used, and the memory of other shards will not be occupied. Therefore, the actual storage space used is only 12/L(M). In this way, even if there is a large offset, it will only waste the space of one fragment. The use of this sharding method greatly improves memory utilization. In addition, the use of this sharding method can also achieve load balancing, thereby avoiding excessive concentration of requests to access storage space, resulting in reduced data processing efficiency.

After implementing sharding on the bitmap, the server also needs to set a key in redis for each shard. The use of this key allows the server to quickly find the corresponding shards. The value corresponding to the key stores a slice of a bitmap. Therefore, the more shards there are, the more redis keys will be, and the keys will occupy more memory. In the existing technology, there is no specific and reliable algorithm for calculating the number of shards. The existing limit on the number of shards is only related to the number of redis cluster nodes. As long as the number of server shards exceeds the number of redis cluster nodes so that requests are not concentrated on a certain node. Obviously, the setting of this number of shards cannot be optimal. Therefore, in this application, the server can determine the target number of shards for each bitmap based on the number of system users, the estimated number of participating users, and the number of projects allowed to participate. The target number of shards is the optimal number of shards. The server can fragment the bitmap according to the target number of fragments, thereby recording the user's participation information.

For example, assuming there are 100 million users, the user ID increases from 1 to 100 million. After the bitmap is fragmented, each fragment can store 2^20≈105w user information. The server can divide a bitmap into 95 slices.

The server can use the redis command SETBIT(key, offset, value) to set the activity participation information when the system user's activity participation information changes.

The server can calculate the slice number (sliceNo) where the system user is located. The calculation formula is:

sliceNo=userId/(2^20)

That is, when user ID (userID)<2^20, sliceNo=0. When the user ID is between 2^20 and 2^21, sliceNo=1, and so on.

The server can calculate the offset of the system user on its corresponding shard. The calculation formula is:

offset=userId%(2^20)

The server can determine the fragment sequence number and offset based on the above two formulas. According to the slice sequence number and offset, the above redis instruction can become: SETBIT(key+sliceNo,offset,value).

If you need to set the 100 millionth user, the command is: SETBIT(key+95,385280,1). This command can control the server to set the status of the 100 millionth user at position 385280 in the 95th shard. When the server only has the 100 millionth user participating in the current activity and the other 94 shards are not used, the amount of storage space occupied can be:

385280/8/1024/1024≈43.37kb

It can be seen that slicing the bitmap can effectively save the storage space of the bitmap when the number of event participants is small.

The technical solution of the present application will be described in detail below with specific examples. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

Figure 3 shows a schematic diagram of a scenario for issuing activity rights according to an embodiment of the present application. As shown in Figure 3, the server can be associated with third-party channels, the application's product trading functions, and the application's customer behavior. When the server detects that a user clicks to view marketing activities on a third-party channel, the server can issue benefits to the user. Alternatively, when the server detects that a user completes a product transaction in the application, the server can issue benefits to the user. Or, when the server detects that the user performs corresponding behavioral operations in the application, the server can issue rights to the user. After the server determines that it needs to issue rights to the user, it calls the coupon issuance interface in MES-SERVICE to realize the issuance of the rights. Afterwards, the server can query the redis cluster for the number of rights and interests that the user has obtained. The server can call MES-SERVICE to verify the amount of equity. If the user's number of rights is less than the preset upper limit, the server can issue the rights to the user. At the same time, the server can also write the rights issuance record into the flow table of the mes database (mesdb). The server can update the number of rights and interests that the user has obtained in redis. At the same time, the service can also use redis synchronization rules to synchronize this information in the redis cluster.

In this application, the server is used as the execution subject to execute the data storage method in the following embodiment. Specifically, the execution subject may be a hardware device of the server, or a software application in the server that implements the following embodiments, or a computer-readable storage medium installed with a software application that implements the following embodiments, or a computer-readable storage medium that implements the following embodiments. Code for the software application of the embodiment.

Figure 4 shows a flow chart of a data storage method provided by an embodiment of the present application. Based on the embodiment shown in Figure 3, as shown in Figure 4, with the server as the execution subject, the method of this embodiment may include the following steps:

S101. Obtain the number of system users, the estimated number of participating users in the current activity, and the number of projects allowed to participate in the current activity. Among them, each currently active item corresponds to a bitmap.

In this embodiment, the server can obtain the number of system users. For example, in the financial field such as banks, the number of users of the system can be the number of users who have opened cards in the bank. For another example, in a certain application, the number of system users can be the number of users who have completed registration in the application.

The server can also determine the estimated number of participating users based on current activity. Optionally, the estimated number of participating users may be a number estimated by the server based on the historical number of participants of the event. Alternatively, the estimated number of participating users may also be the number of users who meet the participation rules of the current activity and are filtered by the server based on the participation conditions of the current activity.

The server can also obtain the number of currently active projects allowed to participate. The number of projects allowed to participate in is the number of projects that each user participating in the activity is allowed to participate in in the current activity. For example, the current activity may include 20 activities. During the activity, each user can participate in up to 6 activities, so the number of allowed participation projects is 6. For another example, the current activity includes 5 rights issuance channels, and each rights issuance channel distributes rights in 6 times. During the event, in order to control event costs, each user can obtain up to 8 rights. At this time, the number of participating projects allowed is 8.

S102. When the ratio between the number of system users and the estimated number of participating users is greater than the preset threshold, determine the target number of shards for each bitmap based on the number of system users, the estimated number of participating users, and the number of projects allowed to participate.

In this embodiment, during the data storage process, if the number of system users is large but the estimated number of participating users is small, using key-value pairs to store the participation information of the current activity can more effectively save storage space. As the estimated number of participating users increases, the storage space consumed by key-value pairs will increase linearly. The storage space required for bitmaps is relatively fixed. Therefore, when the number of system users remains unchanged but the estimated number of participating users is large, using a bitmap algorithm to store the participation information of the current activity can save more storage space. Therefore, the server can determine the ratio of the number of system users to the estimated number of participating users by calculating the quotient of the estimated number of participating users divided by the number of system users. When the number of system users remains unchanged, the greater the estimated number of participating users, the greater the ratio. For this feature, the server can use this ratio to compare with a preset threshold. When the ratio is greater than the preset threshold, it means that the estimated number of participating users is large, and the server can use a bitmap to store the participation information of the current activity.

In one example, when the ratio is less than or equal to the preset threshold, it means that the estimated number of participating users is small, and the server can store the participation information of the current activity in the form of key-value pairs. This process may be specifically shown in step S304 in Figure 6 .

After it is determined that the server uses bitmaps to store participation information, the estimated number of participating users is smaller than the number of system users because the number of system users is usually too large. Therefore, when the server determines to use a bitmap algorithm to store participation information for the current activity, there are usually more unused bits in the bitmap. In order to further improve the utilization of bitmap storage space, the server can calculate the target number of shards for each bitmap based on the number of system users, the estimated number of participating users, and the number of allowed participating projects. The server can fragment each bitmap according to the target number of fragments, thereby saving storage space. In this process, the specific process of the server calculating the number of target shards can be shown in Figure 5.

S103. Segment the bitmap according to the target number of shards, and use the segmented bitmap to record the system user's participation information in the current activity.

In this embodiment, the server may fragment each bitmap according to the target number of fragments calculated in step S102. The server can store the fragmented bitmap using key-value pairs. Among them, key is used to store the sharding information of the shard. The fragmentation information includes at least a bitmap sequence number and a fragmentation sequence number. For example, when three bitmaps are included and the target number of fragments is 10, the key value with bitmap number 1 and fragment number 5 is used to store the bitmap information of the fifth fragment in the first bitmap. Among them, value is used to store the bitmap information of the fragment. The maximum storage space occupied by this shard is the ratio of the number of system users to the number of target shards. Then, the bitmap information of each shard includes the participation information of the number of system users/the number of target shards (bits).

In an example, when the server needs to record a system user's participation information in the current activity, the specific steps may include:

Step 1. The server determines the number of participating projects of the system user based on the system user's value in each bitmap slice.

In this step, the server can determine the fragment sequence number and offset corresponding to the system user in each bitmap based on the user ID of the system user. Among them, the user ID is a numerical code used to uniquely identify a system user. For example, when including 100 million system users, the user ID of the 100 millionth system user could be 100000000. When the target number of shards is 100, each shard is used to record the participation information of 1 million system users. Therefore, each slice can contain 1 million bits. The shard number of the 100 millionth user is 10, and the offset is 1 million. That is, the 100 millionth user can be stored in the 1 millionth bit of the 10th shard. According to the fragment sequence number and the offset, the server can obtain the value corresponding to the system user in each bitmap. The server can determine the number of projects in which the system user participates based on this value. Since in the bitmap, when the user participates in the activity, its value is set to the second data. Therefore, the server can directly count the number of the second data. Because the value of each bit only includes 0 and 1. The second data is 1. Therefore, the server can directly calculate the number of participating projects of the system users through accumulation.

Step 2: When the number of participating projects is less than the number of allowed participating projects, the server modifies the system user's value in the slice of the bitmap from the first data to the second data. When the number of participating projects is equal to the number of allowed participating projects, the server refuses the system user to participate in the project.

In this step, the server can compare the number of projects the system user participates in and the number of projects allowed to participate. Since the number of participating projects allowed is the same as the number of bitmaps. Therefore, the calculated maximum number of participating projects is equal to the number of allowed participating projects. When the number of participating projects is equal to the number of projects allowed to participate, it means that the user of the system has obtained all the rights and interests allowed and participated in all the activities that can be participated in the current activity. Therefore, when the number of participating projects is equal to the number of allowed participating projects, the server refuses the system user to continue participating in the current activity. The server will no longer issue new rights to users of this system. Otherwise, when the number of participating projects is less than the number of projects allowed to participate, it means that the user of the system has not obtained all the rights allowed. Alternatively, system users can continue to participate in other currently active projects. Therefore, when the number of participating projects is less than the number of allowed participating projects, the server can modify the value in one bitmap from the first data to the second data. For example, when 5 bitmaps are included and the user has obtained 2 benefits, the value in 2 of the 5 bitmaps is the second data. At this time, the server can randomly select a bitmap from three other bitmaps whose value is the first data, and modify the value corresponding to the system user from the first data to the second data. Because the value of each bit in the bitmap includes two types: 0 and 1. Normally the first data is 0 and the second data is 1. The first data is used to mean that the user has not obtained the rights. The second data is used to indicate that the user has obtained the rights.

With the data storage method provided by this application, the server can obtain the number of system users. The server can also determine the estimated number of participating users based on current activity. The server can also obtain the number of currently active projects allowed to participate. Each item in the current activity corresponds to a bitmap. The server can calculate the ratio of the number of system users to the estimated number of participating users. The server can use this ratio to compare to a preset threshold. When the ratio is greater than the preset threshold, the server can use a bitmap to store the participation information of the current activity. The server can calculate the target number of shards for each bitmap based on the number of system users, the estimated number of participating users, and the number of projects allowed to participate. The server can fragment the bitmap according to the target number of fragments, and use the fragmented bitmap to record the participation information of system users in the current activity. In this application, by calculating the number of shards, the sharded storage of the bitmap is realized, which improves the utilization of storage space, avoids overly concentrated access to the storage space, and improves the efficiency of data storage and reading.

Figure 5 shows a flow chart of a data storage method provided by an embodiment of the present application. Based on the embodiments shown in Figures 3 and 4, as shown in Figure 5, with the server as the execution subject, the calculation process of the target number of shards in this embodiment may include the following steps:

S201. Determine the range of the number of shards based on the estimated number of participating users.

In this embodiment, the server can determine the maximum number of shards based on the estimated number of participating users. The maximum number of shards is the estimated number of participating users. For example, when there are 100 system users and the estimated number of participating users is 10, the maximum number of shards is 10. The minimum number of shards is 1. When a bitmap includes 1 fragment, the bitmap is not fragmented. When including 100 system users and the estimated number of participating users is 10, in the worst case, the system users participating in the current activity are evenly distributed among all system users. At this time, when there are 10 shards and there is 1 user in each shard, all 10 shards will be used. When there are 20 shards, since there are only 10 users in total, only 10 shards are used. Since each fragment, in addition to the storage space occupied by the bitmap fragment, each bitmap fragment also corresponds to a key, and the space occupied by the key will increase as the number of bitmap fragments increases. Therefore, the lower limit of the shard number range is 1, and the upper limit is the estimated number of participating users.

S202. According to the range of the number of fragments, calculate the bitmap space occupation corresponding to each number of fragments within the range.

In this step, after determining the range of the number of shards, the server can use the number of each shard in the range, together with the number of system users, the estimated number of participating users, and the number of allowed participating projects, to calculate the number of each shard. The corresponding bitmap space occupied. For example, when the lower limit of the number of shards is 1 and the upper limit is 1000, the server can calculate the bitmap space usage when the number of shards is 1, 2,..., 1000 respectively. The calculated data can be shown in Table 1. Table 1 enumerates the bitmap space usage (Mem) corresponding to each number of shards when the number of system users remains unchanged. The unit of the bitmap space occupied (Mem) is byte.

Table 1

In one example, in order to improve calculation efficiency, the server can also pre-calculate a shard number comparison table based on the number of different system users, different estimated number of participating users, and different number of shards. When the number of system users remains unchanged, the comparison table can be as shown in Table 2. Among them, U represents the estimated number of participating users, t represents the number of shards, and Mem (byte) represents the bitmap space occupied in byte. Table 2

In one example, after the server determines the number of system users, the estimated number of participating users, the number of projects allowed to participate, and the number of shards, the specific process of calculating the bitmap space usage may include:

Step 1. The server determines the total length of keywords for each bitmap based on the number of projects allowed to participate, the number of shards, and the fixed length of keywords. Among them, each fragment corresponds to a keyword.

In this step, the fragmented bitmap needs to store key and value parts in Redis. The storage space occupied by these two parts of data is divided into two parts: Mem (key) and Mem (value). Among them, the bitmap information of each fragment is stored in value. And each shard can correspond to a key. Therefore, when the number of shards increases, the storage space occupied will increase linearly with the increase in the number of shards. That is, the number of shards in Mem(key) is proportional to the occupied storage space.

The storage space occupied by key is:

Mem(key)=storage space occupied by each key×number of shards

The key components in the redis bitmap structure are mainly the system number, activity number, bitmap serial number and fragmentation serial number. Among them, the bitmap serial number can be determined based on the number of projects allowed to participate. The fragment sequence number is determined based on the number of target fragments. The system number is determined based on the server's system number. The activity number is determined based on the activity number of the current activity. This activity number uniquely identifies the current activity.

The storage space occupied by a single key is specifically:

Mem(key)＝SDS(9)+system number+activity number+bitmap serial number+fragment serial number

Its unit can be byte. Since the system number and activity number are fixed for the current activity, the three parts of SDS(9), system number and activity number in the above formula can be calculated as constants. Among them, SDS(9) requires fixed storage space for this key. The constant value can be written as:

Mem _key =SDS(9)+system number+activity number

The storage space occupied by the keys of all slices of all bitmaps can be expressed as:

Among them, Mem(Ct _key ) represents the total length of keys of all slices of each bitmap. Mem _key represents a constant value of the storage space required by a shard. t represents the number of shards.

Indicates the storage space occupied by all slice numbers in a bitmap. For example, when 10 shards are included, t=10. The fragment numbers to be used can include 1, 2,...10. Since data can be stored in binary, the server can use log ₂ (i)+1 to determine the number of digits required for the shard number with number i. For example, when i=3, log ₂ (i)+1=2.58. After rounding, it can be determined that 2 bits are needed to store the fragment number with a value of 3. For another example, when i=9, log ₂ (i)+1=4.17. After rounding, it can be determined that 4 bits are needed to store the fragment number with a value of 9. After calculating and accumulating each fragment number, it can be obtained that when t=10, the storage space occupied by the fragment number is 29 bits. In this calculation, the calculation of the space occupied by the bitmap sequence number is actually omitted. In actual use, in order to control costs, the number of projects allowed to participate is usually a smaller value. For example, the number of projects allowed to participate can be 3, 5, 6, etc. Therefore, the storage space occupied by the bitmap numbers of all bitmaps has a very small impact compared to the total amount of storage space occupied by all bitmaps. For example, when 100 million system users are included, the storage space occupied by a bitmap is 12M. When 10 bitmaps are included, the bitmap numbers of the 10 bitmaps occupy only 29 bits of storage space. Therefore, in order to facilitate calculation, the calculation of the bitmap number is directly omitted in this application.

Step 2. The server determines each fragment based on the average fragment length of each fragment, the probability value of each fragment occupying fifty percent of the storage space, the average storage space saved by each fragment, and the number of fragments. The space occupied by all slices of the bitmap.

In this step, the storage space occupied by value can be estimated using probability. For example, the probability that one user occupies half of the storage space of a bitmap of size m is 50%. When the number of users is m/2, the probability of occupying half of the bitmap memory is 100%. It can be seen that the actual number of users is directly proportional to the probability of occupying 50% of the storage space of the shard. That is, the number of shards in Mem(value) is inversely proportional to the occupied storage space. The value in the redis bitmap is used to store the number of issued rights for each user. Each position in the bitmap represents a user, with 1 representing issued rights and 0 representing unissued rights. The storage space occupied by this value is:

Among them, the storage space occupied by a single value can be expressed as:

Mem(value)=SDS(25)+bitmap maximum offset

Its unit can be byte. Among them, SDS(25) requires fixed storage space for this key. Among them, the maximum offset of the bitmap is the offset of the largest number of system users that may appear in the shard. The storage space occupied by the value of all slices of all bitmaps can be expressed as:

Among them, Mem(Ct _value ) is the space occupied by all slices of each bitmap. S is the number of system users. U is the estimated number of participating users. t is the number of shards. Among them, e is used to represent the maximum length of each fragment. The calculation formula of e can be:

The server can determine the average maximum memory space occupied by each shard based on the value of e. The maximum memory space occupied by each fragment can be expressed as Mem(e). For example, when the maximum length of each fragment is 100, the Mem(e) is 100bit. in,

Used to represent the probability value of occupying 50% of the storage space of the shard. For example, the probability that a user appears in each position of a shard is 1/e. Taking the middle position of the shard as the boundary, the probability that a user appears in the first 50% of the storage space of the shard and the probability of appearing in the last 50% of the storage space of the shard are both 50%. When the number of users participating in the current activity is e/2, the probability of occupying 50% of the storage space of the shard is 100%.

in,

For each additional shard, the amount of wasted space is reduced on average. In the worst case of storage space usage, U system users are evenly distributed among S system users. At this time, the average distance between two system users is

Therefore, every time a shard is added, the new shard will most likely be empty.

of storage space. Alternatively, it can be considered that every time a shard is added, the size of the storage space that may be available on the new shard ranges from 0 to

between. In order to cover the possibility of extreme situations, the median is taken in this example

as possible free storage space. For example, take the bitmap shown in Table 3 as an example. This bitmap includes 20 bits, which can correspond to 20 system users. Among them, the 1st, 3rd, 9th, 12th, and 19th system users have obtained their rights and interests, while the remaining system users have not yet obtained their rights and interests. After dividing the bitmap into four slices, the four slices of the bitmap shown in Table 4 can be obtained.

table 3

Table 4

The gray part in Table 4 shows the space that can be saved by each shard. Since the 1-bit storage space corresponding to the last dark gray grid in the bitmap shown in Table 3 can be saved in the bitmap. Therefore, in the fourth shard shown in Table 4, the 1-bit storage space corresponding to the dark gray grid does not belong to the optimization of storage space brought about by sharding. Therefore, in actual calculations, the average storage space of each fragment will be multiplied by t-1, thereby calculating the optimization of storage space brought by the bitmap fragmentation to the first t-1 fragments.

Step 3. The server determines the bitmap space occupancy based on the total length of the keywords of each bitmap, the space occupied by all slices of each bitmap, and the number of allowed participating projects.

In this step, the server can determine the total length of the keyword Mem (Ct _key ) of each bitmap and the space occupied by all slices of each bitmap Mem (Ct _value ) based on the above steps. The length Mem (Ct _key ), the space occupied by all slices Mem (Ct _value ) and the number of allowed participating projects determine the bitmap space occupied. Its formula can be:

Mem(Ct)=[Mem(Ct _key )+Mem(Ct _value )]×N

Among them, N is the number of bitmaps, that is, the number of projects allowed to participate.

S203. Compare the bitmap space occupancy corresponding to the number of each shard, and determine the number of shards corresponding to the minimum bitmap space occupancy as the target number of shards.

In this step, the server can sort according to the bitmap space occupancy after determining the bitmap space occupancy corresponding to each number of fragments according to the above steps. The server can select the number of shards corresponding to the minimum bitmap space occupation as the target number of shards. The server can use this number of fragments to complete fragmentation of the bitmap. The server can complete the recording of the participation information of the current activity in the fragmented bitmap.

With the data storage method provided by this application, the server can determine the maximum number of shards based on the estimated number of participating users, thereby determining the range of the number of shards. The server can use the number of each shard in this range, together with the number of system users, the estimated number of participating users, and the number of allowed participating projects, to calculate the bitmap space occupation corresponding to each shard number. The server can sort the bitmaps based on their space usage. The server can select the number of shards corresponding to the minimum bitmap space occupation as the target number of shards. In this application, the optimal number of fragments is calculated by calculating the bitmap space occupancy corresponding to each possible number of fragments, thereby improving the space utilization when storing the participation information of the current activity, and at the same time Through sharding, the storage efficiency of the participation information of the current activity is also improved.

Figure 6 shows a flow chart of a data storage method provided by an embodiment of the present application. Based on the embodiments shown in Figures 3 to 5, as shown in Figure 6, with the server as the execution subject, the method of this embodiment may include the following steps:

S301. Obtain the number of system users, the estimated number of participating users in the current activity, and the number of projects allowed to participate in the current activity. Among them, each currently active item corresponds to a bitmap.

S302. Determine the target number of shards for each bitmap based on the number of system users, the estimated number of participating users, and the number of projects allowed to participate.

In this embodiment, steps S301 and S302 are implemented similarly to steps S101 and S102 in the embodiment of FIG. 2, and will not be described again in this embodiment. As shown in Figure 7, this step is equivalent to S401. The server can obtain the estimated number of participating users U in the current activity and the number N of allowed participating projects in the current activity.

S303. Determine the bitmap space occupation based on the number of system users, the estimated number of participating users, the number of projects allowed to participate, and the number of target shards.

In this embodiment, the server can calculate the minimum number of bits based on the number of system users, the estimated number of participating users, the number of allowed participating projects, and the target number of shards in the embodiment shown in Figure 5 after determining the target number of shards. The amount of image space occupied. This step is equivalent to S403 in Figure 7.

S304. Determine the space occupied by key-value pairs based on the number of system users, the estimated number of participating users, and the number of projects allowed to participate.

In this embodiment, the server can use the structural information in the key-value structure to calculate the space occupied by the key-value pair. This step is equivalent to S402 in Figure 7 . Among them, the execution order of S303 and S304 and the execution order of S402 and S403 are not limited by the flow chart and can be exchanged arbitrarily.

In the key-value pair, the key includes the system number, activity number and user ID. In the key-value storage method, the server can create a key-value pair when a system user participates in the current activity, and use the key-value pair to record the system user's participation information in the current activity. Therefore, in the key, in addition to the system number used to identify the server and the activity number identifying the current activity, only the user ID needs to be stored in the key. The user ID is a code used by the server to uniquely identify the system. For example, when including 100 million system users, the user ID of the 100 millionth user could be 100000000. In the key-value pair, value is used to store the number of benefits the user has obtained or the number of projects he has participated in. For example, when the user has obtained 3 interests, the value of this value is 3. For another example, when the user has participated in 6 projects, the value of this value is 6.

For example, the key-value data can be:

key=6069:act01:openIdA; value=1

key=6069:act01:openIdB; value=3

key=6069:act01:openIdC; value=2

Similar to the method of calculating the bitmap space occupancy, the calculation process of the space occupancy of the key-value pair also needs to calculate the storage space of the key and the storage space of the value separately. Among them, the calculation method of the storage space occupied by the key can be:

Mem(key)＝SDS(9)+system number+activity number+user serial number

Its unit is byte. When the number of system users is 100 million, the storage space for a user's key value is approximately 20 bytes. For each user, the storage space usage of the key is relatively fixed. Therefore, the server can determine the total amount of storage space required for the key in the current activity based on the estimated number of participating users and the usage of the key's storage space.

Among them, the storage space occupied by value can be calculated as:

Mem(value)＝SDS(25)+log ₂ (n)

Its unit is byte. Among them, n is the number of projects each system user is allowed to participate in or the number of rights and interests that can be obtained in the activity participation rules. log ₂ (n) is used to count the number of digits in a number. Because in value, the value is cumulative, that is, only one value is stored in value at each time. Therefore, when the value in value is n, the value requires the largest storage space. log ₂ (n) is the number of bits needed to store the value n. The number of projects allowed to participate is basically in single digits, so the value of log ₂ (n) in value can be directly simplified to 1byte. The calculation formula for the storage space occupied by this value can be simplified as:

Mem(value)＝SDS(25)+1byte

Based on the above Mem(key) and Mem(value), the server can calculate the space occupied by the key-value pair Mem(kv). The calculation formula can be:

Mem(kv)＝[Mem(key)+Mem(value)]*U

Among them, U is the estimated number of participating users, which is the maximum number of participants in the event. SDS(25) is a fixed constant value that redis takes up memory when using the key-value structure to store data. SDS(25) corresponds to 25byte.

S305. When the space occupied by the key-value pair is less than the space occupied by the bitmap, use the key-value pair to record the system user's participation information in the current activity.

In this embodiment, the server may compare the bitmap space occupancy calculated in step S303 with the key-value pair space occupancy calculated in step S304. This step is equivalent to S404 in Figure 7. When the space occupied by the key-value pair is less than the space occupied by the bitmap, the server determines to use the key-value pair to record the system user's participation information in the current activity. That is, when the space occupied by the key-value pair is less than the space occupied by the bitmap, the server will continue to execute S414 in Figure 7. Otherwise, when the space occupied by the key-value pair is greater than or equal to the space occupied by the bitmap, the server can continue to execute step S306. Alternatively, the server may continue to execute S405 in Figure 7.

In one example, when the server determines that it is necessary to use key values to pair the Geely system user's participation information in the current activity, the specific recording process of the participation information may include the following steps:

Step 1. The server obtains the number of participating projects of the system user recorded in the key-value pair corresponding to the system user.

In this step, the server can directly search for the key corresponding to the system user in redis based on the user ID of the system user. The server can determine the value in the key-value pair after determining the key. This value can include the number of projects the system user participates in. The number of participating projects is the number of rights and interests that have been obtained. This step may be shown in step S415 in FIG. 7 . Among them, the instruction for the server to obtain the number of participating projects of the system user can be:

Value=GET(6069:act01:openIdA);

Step 2. When the number of participating projects is less than the number of allowed participating projects, the server increases the number of participating projects of the system user by the unit value.

In this step, the server can compare the number of participating projects with the number of allowed participating projects. If the number of participating projects that the system user has participated in is less than the number of participating projects, the server can allow the system user to participate in the project and increase the number of participating projects by the system user by the unit value. The unit value can be 1. For example, when a system user has obtained 3 rights and interests, and the maximum number of rights and interests that the system allows the user to obtain is 6, the server can determine that the user can continue to obtain rights and interests. The server can issue rights and interests to users and modify the record in redis of the number of rights and interests that the user has obtained. This step is equivalent to S417 in Figure 7. Among them, the instruction for the server to write the number of participating projects (number of equity) back to redis can be:

SET(6069:act01:openIdA,value+1);

Step 3. When the number of participating projects is greater than or equal to the number of allowed participating projects, the server refuses the system user to participate in the project.

In this step, if the number of participating projects that the system user has participated in reaches the number of participating projects, the server determines that the system user has completed participation in the current activity. The server can deny the user further participation in the current activity. For example, when a system user has acquired 6 rights and interests, and the maximum number of rights that the system allows a user to acquire is 6, the server can determine that the user has acquired all rights and interests. The server can refuse the user to obtain the rights when the user requests the rights again. This step is equivalent to the operation when S416 in FIG. 7 is determined to be negative.

S306. Segment the bitmap according to the target number of shards, and use the segmented bitmap to record the system user's participation information in the current activity.

Among them, step S306 is implemented similarly to step S103 in the embodiment of FIG. 2, and will not be described again in this embodiment. The specific implementation process of S306 can be shown as steps S405 to S413 in Figure 7 . The server can determine the fragment sequence number and offset of the system user based on the user ID of the system user. The server can determine the value corresponding to the system user in each bitmap based on the fragment sequence number and offset. The server can AND these values. When the operation result is 1, it means that the values in these bitmaps are all 1. When the value of the system user in each bitmap is 1, it means that the system user has obtained all the rights and interests that can be obtained, or has participated in all the activities that can be participated in. Otherwise, when the operation result is 0, it means that there is at least one value in these bitmaps that is 0. This result shows that users of this system can continue to obtain rights and interests or participate in active projects. When a user obtains benefits or participates in an active project, the server can set the value in a bitmap with a value of 0 to 1.

Among them, the storage structure of the fragmented bitmap can be:

Bitmap 1:

Bitmap slice 1:key＝6069:bitmap:act01:1:sliceNo;value＝1 0 1 0 0 0 0 0…

Bitmap slice 2:key＝6069:bitmap:act01:1:sliceNo;value＝1 0 1 0 0 0 0 0…

…

Bitmap slice M:key＝6069:bitmap:act01:1:sliceNo; value＝1 0 1 0 0 0 0 0…

Bitmap 2:

Bitmap slice 1:key＝6069:bitmap:act01:2:sliceNo;value＝1 0 1 0 0 0 0 0…

Bitmap slice 2:key＝6069:bitmap:act01:2:sliceNo;value＝1 0 1 0 0 0 0 0…

…

Bitmap slice M:key＝6069:bitmap:act01:2:sliceNo; value＝1 0 1 0 0 0 0 0…

When the user ID is X, the calculation formula of SLICE _NO can be:

The calculation formula for the offset of the system user on the shard can be:

offset=X%e

After the server determines the fragment sequence number and offset, the server can obtain the value of the system user in the N bitmap fragments. The acquisition instruction can be:

value1＝GETBIT("6069:bitmap:act01:1:(X/e)",(X%e));

value2＝GETBIT("6069:bitmap:act01:2:(X/e)",(X%e));

value3＝GETBIT("6069:bitmap:act01:3:(X/e)",(X%e);

…

valueN＝GETBIT("6069:bitmap:act01:N:(X/e)",(X%e));

The server can calculate the number of participating projects of the system user based on the value in each bitmap. Its calculation formula can be:

count＝value1+value2+value3…+valueN

The server can also perform an AND operation on all value values to more quickly determine whether the user's number of participating projects has reached the upper limit. Its calculation formula can be:

result＝value1&value2&value3&….&valueN

When the value of result is 1, it means that the number of participating projects for this system user has reached the upper limit. Otherwise, it means that the number of participating projects for users of the system has not reached the upper limit. The judgment logic of this process can be specifically as follows:

With the data storage method provided by this application, the server can obtain the number of system users, the estimated number of participating users in the current activity, and the number of projects allowed to participate in the current activity. The server can determine the target number of shards for each bitmap based on the number of system users, the estimated number of participating users, and the number of projects allowed to participate. After determining the target number of shards, the server can calculate the minimum bitmap space occupation based on the number of system users, the estimated number of participating users, the number of projects allowed to participate, and the target number of shards. The server can use the structural information in the key-value structure to calculate the space occupied by the key-value pair. The server can compare the space occupied by the key-value pair to be smaller than the space occupied by the bitmap. When the space occupied by the key-value pair is less than the space occupied by the bitmap, the server determines to use the key-value pair to record the system user's participation information in the current activity. When the space occupied by the key-value pair is greater than or equal to the space occupied by the bitmap, the server can fragment the bitmap according to the number of target fragments, and use the fragmented bitmap to record the system user's participation information in the current activity. In this application, by comparing the key-value pair space occupancy and the bitmap space occupancy, it is possible to select the optimal storage method for the participation information of the current activity based on the number of different system users, the estimated number of participating users, and the number of projects allowed to participate. Record and improve the storage efficiency and storage space utilization of participation information.

Figure 8 shows a schematic structural diagram of a data storage device provided by an embodiment of the present application. As shown in Figure 8, the data storage device 10 of this embodiment is used to implement operations corresponding to the server in any of the above method embodiments. , the data storage device 10 of this embodiment includes:

The acquisition module 11 is used to obtain the number of system users, the estimated number of participating users in the current activity, and the number of allowed participating projects in the current activity. Among them, each currently active item corresponds to a bitmap.

The processing module 12 is configured to determine the target number of shards for each bitmap based on the number of system users, the estimated number of participating users, and the number of allowed participating projects when the ratio between the number of system users and the estimated number of participating users is greater than the preset threshold. . The bitmap is fragmented according to the target number of fragments, and the fragmented bitmap is used to record the system user's participation information in the current activity.

Optionally, the processing module 12 is specifically used for:

Determine the range of the number of shards based on the estimated number of participating users.

According to the range of the number of fragments, calculate the bitmap space occupation corresponding to each number of fragments within the range.

Compare the bitmap space occupancy corresponding to each number of shards, and determine the number of shards corresponding to the minimum bitmap space occupancy as the target number of shards.

Optionally, the processing module 12 is specifically used for:

Based on the number of system users, the estimated number of participating users, the number of projects allowed to participate, and the number of shards, determine the bitmap space occupation corresponding to the number of shards.

Optionally, the processing module 12 is specifically used for:

The total keyword length of each bitmap is determined based on the number of projects allowed to participate, the number of shards, and the fixed length of the keyword. Among them, each fragment corresponds to a keyword.

Based on the average fragment length of each fragment, the probability value of each fragment occupying 50% of the storage space, the average storage space saved by each fragment and the number of fragments, determine all the bitmaps The space occupied by the shard.

The bitmap space occupancy is determined based on the total length of the keywords of each bitmap, the space occupied by all slices of each bitmap, and the number of allowed participating projects.

Optionally, the processing module 12 is specifically used for:

The number of participating projects of the system user is determined based on the value of the system user in each bitmap fragment.

When the number of participating projects is less than the number of allowed participating projects, the value of the system user in the slice of the bitmap is modified from the first data to the second data.

When the number of participating projects is greater than or equal to the number of allowed participating projects, the system user will be denied continued participation in the current activity.

Optionally, the bitmap is fragmented according to the target number of fragments, and the fragmented bitmap is used to record the system user's participation information in the current activity. The processing module 12 is also used to:

Determine the bitmap space occupation based on the number of system users, the estimated number of participating users, the number of projects allowed to participate, and the number of target shards.

Determine the space occupied by key-value pairs based on the number of system users, the estimated number of participating users, and the number of projects allowed to participate.

When the space occupied by the key-value pair is less than the space occupied by the bitmap, the key-value pair is used to record the system user's participation information in the current activity.

Optionally, the processing module 12 is also used to:

Get the number of participating projects of the system user recorded in the key-value pair corresponding to the system user.

When the number of participating projects is less than the number of allowed participating projects, increase the number of participating projects for the system user by a unit value.

When the number of participating projects is greater than or equal to the number of allowed participating projects, the system user will be denied continued participation in the current activity.

The data storage device 10 provided by the embodiments of the present application can execute the above method embodiments. For its specific implementation principles and technical effects, please refer to the above method embodiments. This embodiment will not be described again here.

Figure 9 shows a schematic diagram of the hardware structure of a server provided by an embodiment of the present application. As shown in FIG. 9 , the server 20 is used to implement operations corresponding to the server in any of the above method embodiments. The server 20 in this embodiment may include: a memory 21 and a processor 22 .

Memory 21 is used to store computer programs. The memory 21 may include high-speed random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory, and may also be a U disk or a mobile hard disk. , read-only memory, magnetic disk or optical disk, etc.

The processor 22 is configured to execute the computer program stored in the memory to implement the data storage method in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments. The processor 22 can be a central processing unit (Central Processing Unit, CPU), or other general-purpose processor, digital signal processor (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), etc. . A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in conjunction with the invention can be directly embodied and executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

Optionally, the memory 21 can be independent or integrated with the processor 22 .

When the memory 21 is a device independent of the processor 22, the server 20 may also include a bus 23. The bus 23 is used to connect the memory 21 and the processor 22 . The bus 23 may be an Industry Standard Architecture (Industry Standard Architecture, ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

The server provided in this embodiment can be used to execute the above-mentioned data storage method. Its implementation method and technical effects are similar, and will not be described again in this embodiment.

This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it is used to implement the methods provided by the above-mentioned various embodiments.

The computer-readable storage medium may be a computer storage medium or a communication medium. Communication media includes any medium that facilitates transfer of a computer program from one place to another. Computer storage media can be any available media that can be accessed by a general purpose or special purpose computer. For example, a computer-readable storage medium is coupled to a processor such that the processor can read information from the computer-readable storage medium and write information to the computer-readable storage medium. Of course, the computer-readable storage medium may also be an integral part of the processor. The processor and computer-readable storage medium may be located in Application Specific Integrated Circuits (ASICs). Additionally, the ASIC can be located in the user equipment. Of course, the processor and the computer-readable storage medium may also exist as discrete components in the communication device.

Specifically, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (Static Random-Access Memory, SRAM), electrically erasable memory In addition to programmable read-only memory (Electrically-Erasable Programmable Read-Only Memory, EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), programmable read-only memory (Programmable read-only memory, PROM) ), read-only memory (Read-Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk. Storage media can be any available media that can be accessed by a general purpose or special purpose computer.

The application also provides a computer program product. The computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium. At least one processor of the device can read the computer program from the computer-readable storage medium, and at least one processor executes the computer program so that the device implements the methods provided by the various embodiments described above.

An embodiment of the present application also provides a chip. The chip includes a memory and a processor. The memory is used to store a computer program. The processor is used to call and run the computer program from the memory, so that the device equipped with the chip can perform the above various possible implementations. method within the method.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another unit. A system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or modules, and may be in electrical, mechanical or other forms.

Each module may be physically separated, for example, installed in different locations of one device, or installed on different devices, or distributed to multiple network units, or distributed to multiple processors. Individual modules can also be integrated together, for example, installed in the same device, or integrated in a set of codes. Each module can exist in the form of hardware, or can exist in the form of software, or can also be implemented in the form of software plus hardware. This application can select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment.

When each module is implemented in the form of a software function module, the integrated module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods of various embodiments of the present application.

It should be understood that although each step in the flow chart in the above embodiment is shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least some of the steps in the figure may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and their execution order is not necessarily sequential. may be performed in turn or alternately with other steps or sub-steps of other steps or at least part of stages.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application, but are not intended to limit it. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features. . However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the scope of the technical solutions of the embodiments of this application.

Claims (10)

1. A data storage method, characterized in that the method includes:

   Obtain the number of system users, the estimated number of participating users in the current activity, and the number of projects allowed to participate in the current activity; among them, each project in the current activity corresponds to a bitmap;

   When the ratio between the number of system users and the estimated number of participating users is greater than the preset threshold, the target number of shards for each bitmap is determined based on the number of system users, the estimated number of participating users, and the number of allowed participating projects. ;

   The bitmap is fragmented according to the target number of fragments, and the fragmented bitmap is used to record the participation information of the system user in the current activity.
2. The method according to claim 1, characterized in that determining the target number of shards for each bitmap based on the number of system users, the estimated number of participating users and the number of allowed participating projects, specifically includes: :

   Determine the range of the number of shards based on the estimated number of participating users;

   According to the range of the number of fragments, calculate the bitmap space occupation corresponding to each number of fragments within the range;

   Compare the bitmap space occupancy corresponding to each of the number of fragments, and determine the number of fragments corresponding to the minimum value of the bitmap space occupancy as the target number of fragments.
3. The method according to claim 2, characterized in that the calculation of the bitmap space occupation corresponding to each number of fragments in the range specifically includes:

   According to the number of system users, the estimated number of participating users, the number of allowed participation items, and the number of slices, the bitmap space occupation corresponding to the number of slices is determined.
4. The method according to claim 3, characterized in that, based on the number of system users, the estimated number of participating users, the number of allowed participating projects and the number of shards, determining the number of shards corresponding to The amount of bitmap space occupied, specifically includes:

   Determine the total length of keywords for each bitmap according to the number of items allowed to participate, the number of fragments and the fixed length of keywords; wherein each fragment corresponds to one keyword;

   According to the average fragment length of each fragment, the probability value of each fragment occupying fifty percent of the storage space, the average storage space value saved by each fragment and the number of fragments , determine the space occupied by all the fragments of each bitmap;

   The bitmap space occupancy is determined based on the total length of the keywords of each bitmap, the space occupancy of all the slices of each bitmap, and the number of allowed participating items.
5. The method according to any one of claims 1 to 4, characterized in that recording the system user's participation information in the current activity specifically includes:

   Determine the number of participating projects of the system user according to the value of the system user in each bitmap slice;

   When the number of participating projects is less than the number of allowed participating projects, modify the value of the system user in the slice of the bitmap from the first data to the second data;

   When the number of participating projects is greater than or equal to the number of allowed participating projects, the system user is refused to continue to participate in the current activity.
6. The method according to any one of claims 1 to 4, characterized in that the bitmap is fragmented according to the target number of fragments, and the fragmented bitmap is used to record all the bitmaps. Before obtaining the participation information of the system user in the current activity, the method further includes:

   Determine the bitmap space occupancy according to the number of system users, the estimated number of participating users, the number of allowed participating projects, and the number of target shards;

   Determine the space occupied by key-value pairs based on the number of system users, the estimated number of participating users, and the number of allowed participating projects;

   When the space occupied by the key-value pair is less than the space occupied by the bitmap, the key-value pair is used to record the participation information of the system user in the current activity.
7. The method according to claim 6, characterized in that the method further includes:

   Obtain the number of participating projects of the system user recorded in the key-value pair corresponding to the system user;

   When the number of participating projects is less than the number of allowed participating projects, increase the number of participating projects of the system user by a unit value;

   When the number of participating projects is greater than or equal to the number of allowed participating projects, the system user is refused to continue to participate in the current activity.
8. A server, characterized in that the server includes: a memory and a processor;

   The memory is used to store computer programs; the processor is used to implement the data storage method according to any one of claims 1 to 6 according to the computer program stored in the memory.
9. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it is used to implement the data as claimed in any one of claims 1 to 6 Storage method.
10. A computer program product, characterized in that the computer program product includes a computer program, and when the computer program is executed by a processor, the data storage method according to any one of claims 1 to 6 is implemented.

Images