CN113986921A

CN113986921A - Blacklist query method, system, electronic equipment and storage medium

Info

Publication number: CN113986921A
Application number: CN202111259250.3A
Authority: CN
Inventors: 黄代荣; 卢飞; 谷达旺; 巫北辰
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2021-10-27
Filing date: 2021-10-27
Publication date: 2022-01-28

Abstract

The disclosure provides a blacklist query method, which is applied to the field of financial science and technology or other fields, and comprises the following steps: constructing a BitMap array to store blacklist data; calculating a hash value according to the element to be queried, and calculating to obtain a corresponding BitMap array address according to the hash value; acquiring a first result corresponding to the corresponding BitMap array address through Redis, and judging whether the first result is matched; if the query result is matched with the table entry, comparing the element to be queried with the table entry according to a database table of Redis to obtain a second result; if the second result also matches, the element to be queried is in the blacklist. The present disclosure also provides a blacklist query system, an electronic device, a computer-readable storage medium and a program product.

Description

Blacklist query method, system, electronic equipment and storage medium

Technical Field

The disclosure relates to the technical field of financial technology, in particular to a blacklist query method, a blacklist query system, electronic equipment and a storage medium.

Background

Blacklist collision all relates to in each field of internet, through blacklist collision of different dimensions, relevant authority can be fine control, protection system safety etc.. In the financial industry, the blacklist collision use scene is also wider, and technical support can be provided in the aspects of account number protection, fund safety protection and the like through financial blacklist collision with different dimensions. However, with the development of the financial industry, the dimension and the number of various lists are rapidly increased, and the development of the internet also has higher and higher requirements on the processing efficiency of blacklist collision.

The prior conventional method has the following defects:

the method comprises the following steps: using a relational database, lists are maintained through database tables. SQL query statements are sent to a database through a program, whether the data exist in the database table is checked, and the purpose of list collision is achieved. The disadvantages are as follows: in this way, each time of list collision, the database is required to be connected to execute related query, and under the condition of high concurrency, the response efficiency of the database is low, and the database becomes a weak link of the whole system and even goes down. High-concurrency use scenarios cannot be well satisfied.

The second method comprises the following steps: and (3) using the local cache to import the list data maintained by the database into the local cache of the application when the application is started, synchronizing the list data of the database to the local cache at regular time, and directly performing list collision through the local cache. The disadvantages are as follows: in this way, in the same cluster, a list of data is maintained in different nodes, and the data consumes a large amount of memory space; meanwhile, different nodes need to update the list data independently according to the homonymous data at regular time, so that the timeliness and the consistency of the list are not high.

The third method comprises the following steps: and (3) using a memory database, caching a relational database list by the memory database, storing list data by common character strings, writing the full list data into a memory, and performing list collision by the memory database. The disadvantages are as follows: each list is stored in a character string mode, the consumption of the memory is increased sharply along with the increase of the number of the lists, the dimension of the list which can be maintained in a single memory database cluster is limited, and the use cost is high.

Disclosure of Invention

Technical problem to be solved

In view of the above problems, the present disclosure provides a blacklist query method, a system, an electronic device and a storage medium, which are used to at least partially solve the technical problems of low response efficiency, low timeliness and the like of the conventional query method.

(II) technical scheme

One aspect of the present disclosure provides a blacklist query method, including: constructing a BitMap array to store blacklist data; calculating a hash value according to the element to be queried, and calculating to obtain a corresponding BitMap array address according to the hash value; acquiring a first result corresponding to the corresponding BitMap array address through Redis, and judging whether the first result is matched; if the query result is matched with the table entry, comparing the element to be queried with the table entry according to a database table of Redis to obtain a second result; if the second result also matches, the element to be queried is in the blacklist.

Further, constructing the BitMap array to store the blacklist data comprises the following steps: determining the BitMap length and the Hash times required by storing the blacklist data according to the data volume of the blacklist data and the estimated misjudgment rate; establishing an array by taking the BitMap length as the capacity of the array elements; and storing the blacklist data into the array elements of the array.

Further, determining the length of the BitMap and the number of times of hashing required for storing the blacklist data according to the data amount of the blacklist data and the estimated misjudgment rate includes:

the BitMap length and the number of hashes are calculated according to the following formula:

k＝-log₂ p

wherein m represents the length of the bitmap array, k represents the hash times, n represents the list element prediction amount, and p represents the prediction error rate.

Further, calculating a hash value of the element to be queried according to the element to be queried, and calculating a corresponding BitMap array address according to the hash value includes: calculating the hash value of the element to be inquired according to the configured hash function and the hash times; and obtaining a corresponding BitMap array address according to the Hash value to the BitMap length module.

Further, acquiring a first result corresponding to the corresponding BitMap array address through Redis includes: acquiring a bit value corresponding to a corresponding BitMap array address through an array operation method provided by Redis; if the value of the bit is not 1, the element to be inquired is not in the blacklist; if all the bit values are 1, the comparison is continued.

Further, comparing the element to be queried with the table entry according to the database table of Redis includes: querying in a database table of Redis according to the unique identifier of the element to be queried; if the effective state corresponding to the element to be queried is effective, the element to be queried is in the blacklist; otherwise, the element to be queried is not in the blacklist.

Further, constructing the BitMap array to store the blacklist data further includes updating a Redis database table, including: reading elements to be updated in batches, and creating a Redis database temporary table; carrying out BitMap processing on elements to be stored in sequence, and storing the elements into array elements of a BitMap array; updating the effective state of the element to be stored in the temporary table of the Redis database; and replacing the original Redis database table by using the updated temporary Redis database table.

Further, performing BitMap processing on the elements to be stored in sequence, wherein the storing into the array elements of the BitMap array comprises: carrying out BitMap processing on elements to be stored in sequence to obtain corresponding BitMap array addresses; setting the value of a bit corresponding to the corresponding BitMap array address to be 1 by an array operation method provided by Redis; the corresponding validation state in the Redis database table is set to validate.

Further, setting the respective validation states in the Redis database tables to validate further includes retries, including: reading an element of which the effective state is effective and the value of a bit corresponding to the element is 0 in a Redis database table; and carrying out the bitmap processing process again on the elements.

Further, constructing the BitMap array to store the blacklist data further includes deleting elements, including: and setting the effective state corresponding to the element to be deleted in the Redis database table as invalid.

In another aspect, the present disclosure provides a blacklist query system, including: the building module is used for building a BitMap array to store blacklist data; the computing module is used for computing the hash value of the element to be queried according to the element to be queried and computing a corresponding BitMap array address according to the hash value; the matching module is used for acquiring a first result corresponding to the corresponding BitMap array address through Redis and judging whether the first result is matched or not; the comparison module is used for comparing the element to be queried with the table entry according to the database table of Redis to obtain a second result if the element to be queried is matched with the table entry; if the second result also matches, the element to be queried is in the blacklist.

Yet another aspect of the present disclosure provides an electronic device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program when executed by the processor causes the processor to perform the blacklist lookup method as described above.

Yet another aspect of the present disclosure provides a computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the blacklist query method as described above.

Yet another aspect of the present disclosure provides a computer program product comprising a computer program which, when executed by a processor, implements a blacklist query method as aforementioned.

(III) advantageous effects

According to the blacklist query method, the system, the electronic device and the storage medium, the Redis memory database is adopted, high concurrency is supported, the response time is short, and a large number of blacklist elements which cannot be queried and hit can be filtered out quickly; by adopting a binary BitMap data structure, the consumption of the memory can be obviously reduced, more blacklist and blacklist elements can be supported under a single cluster, and the hardware cost of the system is saved; and by introducing a secondary comparison mechanism, secondary comparison is carried out through a database table under the condition that a large number of missed elements are filtered, and blacklist query can still be accurately carried out under the characteristics of quick Redis response and small usage amount of a BitMap memory.

Drawings

For a more complete understanding of the present disclosure and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

fig. 1 schematically illustrates an application scenario of a blacklist query method according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow diagram of a blacklist query method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a Bitmap rationale diagram according to an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart of a method of constructing a BitMap array storage blacklist data according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow chart of a method of computing a BitMap array address according to an embodiment of the disclosure;

FIG. 6 schematically illustrates a flowchart of a method of constructing an update of a Redis database table in BitMap array storage blacklist data according to an embodiment of the present disclosure;

FIG. 7 schematically illustrates a flow chart of a method of storing elements into array elements of a BitMap array, in accordance with an embodiment of the present disclosure;

FIG. 8 schematically illustrates a process flow diagram for roster creation and redo according to an embodiment of the present disclosure;

FIG. 9 schematically illustrates a process flow diagram for a roster collision according to an embodiment of the disclosure;

FIG. 10 schematically illustrates a process flow diagram for roster element addition according to an embodiment of the present disclosure;

FIG. 11 schematically illustrates a process flow diagram for the deletion of a list element according to an embodiment of the disclosure;

FIG. 12 schematically illustrates a block diagram of a blacklist query system according to an embodiment of the present disclosure;

fig. 13 schematically shows a block diagram of an electronic device adapted to implement the above described method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

Some block diagrams and/or flow diagrams are shown in the figures. It will be understood that some blocks of the block diagrams and/or flowchart illustrations, or combinations thereof, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the instructions, which execute via the processor, create means for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks. The techniques of this disclosure may be implemented in hardware and/or software (including firmware, microcode, etc.). In addition, the techniques of this disclosure may take the form of a computer program product on a computer-readable storage medium having instructions stored thereon for use by or in connection with an instruction execution system.

Based on the disadvantages of the conventional blacklist query method, embodiments of the present disclosure provide a blacklist query method, a system, an electronic device, and a storage medium, which retain the advantages of different conventional methods and avoid some of the disadvantages that have occurred; the method can provide higher concurrency, better timeliness and better data consistency, can obviously reduce the memory consumption of the memory database, can increase the list dimensionality maintained in a single memory database, and reduces the construction cost of the system.

Fig. 1 schematically illustrates an exemplary system architecture 100 that may be applied to a blacklist query method according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1, the system architecture 100 according to this embodiment may include

terminal devices

101, 102, 103, a network 104 and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may have installed thereon various communication client applications, such as a web browser application, a search-type application, an instant messaging tool, social platform software, etc. (by way of example only).

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by users using the

terminal devices

101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the blacklist query method provided by the embodiments of the present disclosure may be generally executed by the server 105. Accordingly, the system of the blacklist query method provided by the embodiment of the present disclosure may be generally disposed in the server 105. The blacklist query method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. Correspondingly, the system of the blacklist query method provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster that is different from the server 105 and is capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Typically, the number of users and the access amount faced by us are huge, such as millions, tens of millions of users, or tens of millions or even hundreds of millions of access information. When the magnitude of hundreds of millions is reached and the speed is increased in the future, the traditional method for querying the database is difficult to support, and the storage capacity and the query efficiency are tested. Therefore, we have to select a collection type that is very efficient in counting large amounts of data (e.g., in billions).

In the disclosure, a BitMap array is adopted for data storage. The BitMap array marks the value corresponding to an element with a bit, and the key is the element. Because bit is used as a unit to store data, the storage space can be greatly saved. In the BitMap array, each array element is "0" or "1", indicating that its corresponding array element is absent or present. For example: on a 32-bit machine, a shaping int a occupies 32 bits in memory, and the corresponding 32 bits can be used to represent 0-31 decimal numbers. The representation of the BitMap array in memory may be a list of binary tables consisting of 0 and 1, such as 0101001. When a bit in the BitMap array is in an occupied state, setting the value of the bit to be 1; when a bit is in an idle state, setting the value of the bit to be 0. And the number of each bit is the offset of the bit from the starting point in the BitMap array.

Fig. 2 schematically shows a flow chart of a blacklist query method according to an embodiment of the present disclosure.

As shown in FIG. 2, the blacklist query method 200 may include operations S210-S240.

In operation S210, a BitMap array is constructed to store blacklist data.

The Bitmap bottom layer structure is a binary array, fig. 3 is a Bitmap basic principle diagram, each bit stores binary '0' or '1', a fixed-length binary array needs to be determined during initialization, and all data items are assigned to '0'. If the blacklist data is stored by adopting the character string, the unique identifier corresponding to each element is stored as int, which occupies 4 bytes, namely 32 bits, and the single identifier only occupies one bit in the Bitmap, so that the memory is saved by 32 times. That is, the bit corresponding to the element item is stored in the final Bitmap, so that the storage space is greatly saved.

In operation S220, a hash value of the element to be queried is calculated according to the element to be queried, and a corresponding BitMap array address is obtained according to the hash value.

If an element is to be determined to be in a collection, all elements are typically saved and then compared to determine the presence of the element. Linked lists, trees, etc. data structures are all in this manner. But as the number of elements in the set increases, the storage space required by the elements is larger and larger, and the retrieval speed is slower and slower. However, there is also a data structure called Hash table (Hash table) which can map an element to a point in a bit array (BitArray) by means of a Hash function. Thus, it can be known whether the element exists in the set or not by judging whether the point is 1 or not. I.e., the hash table accesses the record by mapping the key value to a location in the table to speed up the lookup.

And calculating a plurality of hash values of the elements by a formulated hash function and required calculation times, obtaining a single position of the element item stored in the array according to the length modulus of the Bitmap array, and carrying out assignment or matching operation ('1' represents current bit matching). The judgment of whether the numerical value exists is carried out by using the hash calculation for many times, so that the hash collision condition is greatly reduced.

In operation S230, a first result corresponding to the corresponding BitMap array address is obtained through Redis, and whether the first result is matched is determined.

The Bitmap record in the Redis is used for storing the blacklist data, the bottom layer data structure of the Bitmap uses a String type SDS data structure to store the bit array, the Redis utilizes 8 bit bits of each byte array, and each bit represents the binary state of one element (either 0 or 1).

Bitmaps provide get bit, SETBIT operations that read and write bits at the offset position of the bit array by an offset value, note that the offset starts at 0. For example, when determining the user login status, only one key _ status is needed to indicate that the user login status set data is stored, the user ID is set as offset, and the online status is set to 1 and the offline status is set to 0. And judging whether the corresponding user is online or not through GETBIT. The 50000 universal subscriber only needs 6MB of space. GETBIT takes the value of the bit at offset of the value of a key, and returns 0 when a key is not present.

Reading a binary value corresponding to the subscript of the corresponding name unit array through a bit array operation method GETBIT provided by Redis, and according to the binary value.

In operation S240, if the matching is successful, comparing the element to be queried with the table entry according to the database table of Redis to obtain a second result; if the second result also matches, the element to be queried is in the blacklist.

Because the hash function and the hash value have theoretical conflict to the array length (the probability of the conflict is calculated by the following formula and is related to the array length and the hash times), the accurate collision needs to be avoided by secondary collision (the method adopts a database table). Therefore, when the matching is judged according to the binary value, the element to be queried cannot be completely judged to be in the blacklist, for example, under the condition that the financial industry needs to completely ensure accuracy, secondary confirmation can be performed through a database table (the hit number is filtered by the BitMap, the residual amount is small), and if the element to be queried is also matched with the table entry, the element to be queried is considered to be in the blacklist.

In the present disclosure, Redis itself acts as an in-memory database, providing a higher amount of concurrency. Meanwhile, since the String type key of Redis is realized by an SDS data structure, the String type key is binary safe, supports the String type key as a bit array to operate, and provides basic support.

Fig. 4 schematically shows a flowchart of a method for constructing a BitMap array storage blacklist data according to an embodiment of the present disclosure.

As shown in fig. 4, the method for storing blacklist data for constructing the BitMap array includes:

in operation S211, a BitMap length and a hash number required for storing the blacklist data are determined according to the data amount of the blacklist data and the estimated misjudgment rate.

The judgment of whether the numerical value exists is carried out by using the hash calculation for many times, although the hash conflict situation is greatly reduced, a certain defect still exists, and the situation that the misjudgment is easy to occur is easily caused. If a certain number is judged to be absent through the bloom filter, the number is judged to be absent really, and misjudgment cannot occur; if a certain number is judged to exist through the bloom filter, the judgment is possibly misjudged at the moment, and the certain number may not exist. However, by adjusting the ratio of the number of hash functions, the size of the bitmap and the number of digits to be stored, the probability of such a false positive can be reduced to a very low value.

k＝-log₂ p

wherein m represents the length of the BitMap array, k represents the hash times, n represents the list element pre-estimation amount, and p represents the pre-estimation error rate.

Murmur3_128 is used in this disclosure as the base hash function. The BitMap array is initialized in Redis by the determined BitMap length.

In operation S212, an array is established with the BitMap length as the capacity of the array element.

Under the condition that the number k of the hash functions is constant: the larger the BitMap array length m is, the lower the false judgment rate is; the larger the number n of inserted elements, the higher the false positive rate. And obtaining the BitMap length according to the estimated misjudgment rate and the element estimation amount.

In operation S213, the blacklist data is stored into array elements of the array.

After the length of the BitMap array is determined, dividing each array element in the BitMap array by the capacity of the array element, and setting subscripts for the array elements according to the offset positions of the array elements in the array. For example, a bitmap with a length of 10, each bit stores ten shaping digits from 0 to 9, and the bit corresponding to each bitmap is 0. When the number stored is 3, the position of the bitmap 3 becomes 1.

Fig. 5 schematically shows a flowchart of a method for calculating a hash value according to an element to be queried and obtaining a corresponding BitMap array address according to the hash value according to an embodiment of the present disclosure.

As shown in fig. 5, the method for calculating the hash value according to the element to be queried and obtaining the corresponding BitMap array address according to the hash value includes:

in operation S221, a hash value of the element to be queried is calculated according to the configured hash function and the number of hashes.

The hash value of the element to be queried is first calculated. The method comprises the steps of giving a table M, having a function f (key), substituting a function into any given key value key, and if an address recorded in the table and containing the key can be obtained, calling the table M as a hash table, and using the function f (key) as a hash function.

In an optional embodiment of the present disclosure, the predetermined hash algorithm is a murmurur hash algorithm. Of course, the use of the murmur hash algorithm is only an example description, and those skilled in the art can flexibly adjust which hash algorithm is used according to the requirement. It is within the scope of the protection concept of the present disclosure to implement the calculation of the BitMap array address by using any hash algorithm with low collision rate and/or high efficiency.

In operation S222, a BitMap array address is obtained by modulo the BitMap length according to the hash value.

And calculating to obtain the corresponding subscript of the bitmap binary number group by using hash (1ength-1) according to the length of the bitmap binary number group in the list.

On the basis of the above embodiment, obtaining a first result corresponding to the corresponding BitMap array address through Redis includes: acquiring a bit value corresponding to a corresponding BitMap array address through an array operation method provided by Redis; if the value of the bit is not 1, the element to be inquired is not in the blacklist; if all the bit values are 1, the comparison is continued.

And reading a binary value corresponding to the subscript of the corresponding name unit array by a bit array operation method getbit provided by Redis. If the values of the corresponding positions are not totally '1', the elements are not in the list, the list is not hit, the result is directly returned, if the subscripts of the digit group are totally '1', the elements cannot be completely judged to be in the list under the bitmap structure, and secondary confirmation can be carried out through the database table under the condition that the financial industry needs to completely ensure accuracy.

On the basis of the above embodiment, comparing the element to be queried with the table entry according to the database table of Redis includes: querying in a database table of Redis according to the unique identifier of the element to be queried; if the effective state corresponding to the element to be queried is effective, the element to be queried is in the blacklist; otherwise, the element to be queried is not in the blacklist.

Table 1 is a database table structure used by the method.

TABLE 1

And performing secondary confirmation through the database table, inquiring the database table according to the name of the list element and the effective state field, judging whether the secondary confirmation result hits the list, and returning a corresponding result.

Fig. 6 schematically shows a flowchart of a method for constructing an update of a Redis database table in a BitMap array storage blacklist data according to an embodiment of the present disclosure.

As shown in fig. 6, the update method of the Redis database table includes:

in operation S601, the elements to be updated are read in batch, and a Redis database temporary table is created.

In operation S602, BitMap processing is sequentially performed on the elements to be stored, and the elements are stored in array elements of the BitMap array.

In operation S603, the validation status of the element to be stored in the temporary table of the Redis database is updated.

In operation S604, the original Redis database table is replaced with the updated Redis database temporary table.

In order to eliminate the elements of the failed part in the bitmap and keep the tidiness of the bitmap structure, a timing program (for example, 7 days) finishes the redoing of the list, a temporary bitmap is generated in the redoing process, and the original bitmap is replaced after the redoing.

Fig. 7 schematically shows a flowchart of a method for performing BitMap processing on elements to be stored in sequence and storing the elements into array elements of a BitMap array according to an embodiment of the present disclosure.

As shown in fig. 7, the method for performing BitMap processing on elements to be stored in sequence and storing the elements to be stored in the array element of the BitMap array includes:

in operation S701, BitMap processing is sequentially performed on the elements to be stored to obtain a corresponding BitMap array address.

In operation S702, the value of the bit corresponding to the BitMap array address is set to 1 by the array operation method provided by Redis.

In operation S703, the corresponding validation state in the Redis database table is set to validate.

Firstly, carrying out BitMap processing on an element to be stored to obtain an array address, and setting a corresponding binary value under the array address to be '1'; after the operation is successful, the 'Bitmap state' field of the update database is updated to be '1', which indicates that the Bitmap structure is placed.

On the basis of the above embodiment, setting the corresponding validation state in the Redis database table to validate further includes retrying, including: reading an element of which the effective state is effective and the value of a bit corresponding to the element is 0 in a Redis database table; and carrying out the BitMap processing process again on the elements.

In order to ensure that errors in the adding process need to be retried, the background periodically reads the elements of which the effective state field is '1' and the Bitmap state field is '0' through batch tasks, and re-executes the Bitmap entering process.

On the basis of the above embodiment, constructing the BitMap array to store the blacklist data further includes deleting elements, including: and setting the effective state corresponding to the element to be deleted in the Redis database table as invalid.

The specified element can not be directly deleted from the bitmap in the operation of deleting the list element, and the setting is modified to be in the invalid state by updating the 'effective state' field corresponding to the updated element to '0' in the database table.

The steps of the method are further described below with a specific example.

The method and the device combine the Redis memory database, the bitmap data structure and introduce a secondary collision mechanism, can support higher concurrency, reduce the consumption of the memory, and can perform accurate place name list collision. The list collision in this embodiment is the aforementioned blacklist query, and the purpose of list collision is achieved by checking whether data exists in a database table.

1. A list creation process. FIG. 8 illustrates a process flow diagram for roster creation and redo. The list creation and redo step 800 includes: step 801, batch reading of MySQL table effective list elements. At step 802, Redis creates a new temporary name. And step 803, circularly processing the single list element, wherein the processing process of the single list element comprises steps 804-806. Step 804, calculating the multiple HashCode of the list element by the following formula: m represents the length of a bitmap array, k represents the hash times, n represents the list element prediction amount, and p represents the prediction error rate;

k＝-log₂ p

the array length of the bitmap and the number of hashes can be determined, and the method uses murmur3_128 as a basic hash function. Step 805, element multi-bitmap subscript index calculation. In step 806, the value of the temporary list bitmap index is set to binary 1. And step 807, updating the batch state of the MySQL table list, and initializing a bitmap array in the Redis according to the determined bitmap length. And reading the list element with the effective state of 1 in the database table through a batch timing program, and circularly finishing bitmap processing of the list element in batches. And 808, processing the full elements of the list, setting the name of the Redis rename bitmap temporary list as the original name single name, in order to remove the failed elements in the bitmap, keeping the tidiness of the bitmap structure, finishing the redo of the list by the timing program (7 days), generating a temporary bitmap in the redo process, and replacing the original bitmap after the redo.

2. And (5) list collision process. FIG. 9 shows a flowchart of the process of list collision. The list collision step 900 includes: step 901, calculating element multiple HashCode; and calculating the times of the added elements according to the hash function configured by the list and the element hash to obtain a plurality of hash codes of the elements. Step 902, calculating the index of the element multi-bitmap subscript; and calculating to obtain the subscript of the bitmap binary number group by using hash code (length-1) according to the length of the bitmap binary number group in the list. And step 903, acquiring the subscript value of the element multi-bitmap, and reading a binary value corresponding to the subscript of the corresponding name unit array through a digit array operation method getbit provided by Redis. Step 904, determine whether all binary values are binary 1. Step 906, if the values of the corresponding positions are not all '1', it indicates that the element is not in the list, and the element misses the list, and the result is directly returned. Step 905, if the subscripts of the digit group are all '1', under the bitmap structure, the elements cannot be completely judged to be in the list, under the condition that the accuracy needs to be completely guaranteed in the financial industry, secondary confirmation can be performed through the database table (the hit number is bitmap filtered, the residual amount is small), the database table is inquired through the fields of the name of the list elements and the effective state, and whether the result of the secondary confirmation hits the list is determined. And step 907, if the result of the secondary confirmation is yes, the collision is successful, and the element returns a corresponding result in the list.

3. List element addition process. FIG. 10 illustrates a process flow diagram for roster element addition. The add list element step 1000 includes: step 1001, the list element is written into MySQL. Step 1002, calculating multiple HashCode of the list elements; and calculating the times of the added elements according to the hash function configured by the list and the element hash to obtain a plurality of hash codes of the elements. When the list is abnormal, the list bitmap is updated to be abnormal 1003; step 1004, the MySQL queries a failure name list item according to the status bit; step 1005, circularly processing the single list element, and then entering a HashCode calculation flow added with the list element, wherein the subsequent steps are the same as the list element adding processing flow. Step 1006, calculating the index of the element multi-bitmap subscript; and calculating to obtain the subscript of the bitmap binary number group by using hash code (length-1) according to the length of the bitmap binary number group in the list. Step 1007, setting the value of the list bitmap index as binary 1; and setting the binary value corresponding to the subscript of the corresponding name unit array to be '1' by using a digit array operation method setbit provided by Redis. Step 1008, after the operation is successful, the update database "Bitmap status" field is updated to '1', indicating that the Bitmap structure has been placed. In order to ensure that errors in the adding process need to be retried, the background periodically reads the elements of which the effective state field is '1' and the Bitmap state field is '0' through batch tasks, and re-executes the Bitmap entering process.

4. And (5) deleting the list element. FIG. 11 illustrates a process flow diagram for list element deletion. The delete list element step 1100 includes: step 1101, updating the state of the element MySQL table to be invalid; due to the data structure of the bitmap, the specified element cannot be directly deleted from the bitmap in the list element deletion operation, the '0' failure state is updated by updating the field of the 'effective state' corresponding to the element in the database table, and when the element is subjected to secondary verification through the database, the list element is deleted due to the failure of the element.

The method achieves the effect of integrating the advantages of each method through the combination of various data structures and technologies, avoids various defects in a single method, and has good practicability. The method has the following advantages:

the method has the advantages that: lower response time. The Redis memory database is used, higher concurrency is supported, response time is short, a large number of elements which cannot collide with a hit list can be filtered out quickly, and better user experience is achieved.

The advantages are two: and the memory resource consumption of the memory database is reduced. The binary bitmap data structure of the method can obviously reduce the consumption of the memory, can support more lists and list elements under a single cluster, and saves the hardware cost among systems.

The advantages are three: accurate list collision. Through the introduced secondary collision mechanism, secondary collision is carried out through the database table under the condition that a large number of missed elements are filtered, and list collision can be accurately carried out under the characteristics of rapidness in Redis and small using amount of bitmap memory is reserved.

FIG. 12 schematically illustrates a block diagram of a blacklist query system according to an embodiment of the present disclosure.

As shown in fig. 12, the blacklist query system 1200 includes: a construction module 1210, a calculation module 1220, a matching module 1230, and a comparison module 1240.

The building module 1210 is used for building a BitMap array to store blacklist data; according to an embodiment of the present disclosure, the building module 1210 may be configured to perform the step S210 described above with reference to fig. 2, for example, and is not described herein again.

The calculating module 1220 is configured to calculate a hash value of the element to be queried according to the element to be queried, and calculate a corresponding BitMap array address according to the hash value; according to an embodiment of the present disclosure, the calculating module 1220 may be configured to perform the step S220 described above with reference to fig. 2, for example, and is not described herein again.

The matching module 1230 is configured to obtain a first result corresponding to the corresponding BitMap array address through Redis, and determine whether the first result is matched; according to an embodiment of the present disclosure, the matching module 1230 may be configured to perform the step S230 described above with reference to fig. 2, for example, and is not described herein again.

A comparison module 1240, configured to, if the matching is performed, further compare the to-be-queried element with the table entry according to a database table of Redis, to obtain a second result; if the second result is also matched, the element to be inquired is in the blacklist; according to an embodiment of the present disclosure, the comparison module 1240 may be used to perform the step S240 described above with reference to fig. 2, for example, and is not described herein again.

It should be noted that any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

For example, any of the building module 1210, the computing module 1220, the matching module 1230, and the comparing module 1240 may be combined and implemented in one module, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the disclosure, at least one of the building module 1210, the calculating module 1220, the matching module 1230, and the comparing module 1240 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or may be implemented in any one of three implementations of software, hardware, and firmware, or in any suitable combination of any of them. Alternatively, at least one of the construction module 1210, the calculation module 1220, the matching module 1230, the comparison module 1240 may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

The blacklist query system and the blacklist query method can be used in the fields of financial science and technology and the like, and provide a combined Redis memory database, a bitmap data structure and a secondary collision mechanism, so that higher concurrency can be supported, the consumption of a memory is reduced, and accurate place name list collision can be performed.

Fig. 13 schematically shows a block diagram of an electronic device adapted to implement the above described method according to an embodiment of the present disclosure. The electronic device shown in fig. 13 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 13, the electronic apparatus 1300 described in this embodiment includes: a processor 1301, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)1302 or a program loaded from a storage section 1308 into a Random Access Memory (RAM) 1303. The processor 1301 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 1301 may also include onboard memory for caching purposes. Processor 1301 may include a single processing unit or multiple processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 1303, various programs and data necessary for the operation of the system 1300 are stored. The processor 1301, the ROM1302, and the RAM 1303 are connected to each other via a bus 1304. The processor 1301 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM1302 and/or the RAM 1303. Note that the programs may also be stored in one or more memories other than the ROM1302 and RAM 1303. The processor 1301 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 1300 may also include input/output (I/O) interface 1305, which is also connected to bus 1304, according to an embodiment of the present disclosure. The system 1300 may also include one or more of the following components connected to the I/O interface 1305: an input portion 1306 including a keyboard, a mouse, and the like; an output section 1307 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 1308 including a hard disk and the like; and a communication section 1309 including a network interface card such as a LAN card, a modem, or the like. The communication section 1309 performs communication processing via a network such as the internet. A drive 1310 is also connected to the I/O interface 1305 as needed. A removable medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1310 as necessary, so that a computer program read out therefrom is mounted into the storage portion 1308 as necessary.

According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via communications component 1309 and/or installed from removable media 1311. The computer program, when executed by the processor 1301, performs the functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

The embodiments of the present disclosure also provide a computer-readable storage medium, which may be included in the device/apparatus/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement a blacklist query method according to an embodiment of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include one or more memories other than the ROM1302 and/or the RAM 1303 and/or the ROM1302 and the RAM 1303 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method illustrated in the flow chart. When the computer program product runs in a computer system, the program code is used for causing the computer system to realize the blacklist query method provided by the embodiment of the disclosure.

The computer program performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure when executed by the processor 1301. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted in the form of a signal on a network medium, distributed, downloaded and installed via communications component 1309, and/or installed from removable media 1311. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such embodiments, the computer program may be downloaded and installed from a network via communications component 1309 and/or installed from removable media 1311. The computer program, when executed by the processor 1301, performs the functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

It should be noted that each functional module in each embodiment of the present disclosure may be integrated into one processing module, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a separate product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present disclosure may be embodied in the form of software products, in part or in whole, which substantially contributes to the prior art.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

While the disclosure has been shown and described with reference to certain exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the appended claims and their equivalents. Accordingly, the scope of the present disclosure should not be limited to the above-described embodiments, but should be defined not only by the appended claims, but also by equivalents thereof.

Claims

1. A blacklist query method is characterized by comprising the following steps:

constructing a BitMap array to store blacklist data;

calculating a hash value according to the element to be inquired, and calculating to obtain a corresponding BitMap array address according to the hash value;

acquiring a first result corresponding to the corresponding BitMap array address through Redis, and judging whether the first result is matched;

if the query is matched with the table entry, comparing the element to be queried with the table entry according to the database table of Redis to obtain a second result; if the second result also matches, the element to be queried is in the blacklist.

2. The blacklist querying method according to claim 1, wherein the constructing the BitMap array to store blacklist data comprises:

determining the length of the BitMap and the number of times of Hash required by storing the blacklist data according to the data volume of the blacklist data and the estimated misjudgment rate;

establishing an array by taking the BitMap length as the capacity of an array element;

and storing the blacklist data into array elements of the array.

3. The method of claim 2, wherein the determining the BitMap length and the hash number required for storing the blacklist data according to the data amount of the blacklist data and the estimated misjudgment rate comprises:

calculating the BitMap length and the hash times according to the following formula:

k＝-log₂ p

4. The blacklist query method according to claim 2, wherein the calculating a hash value according to an element to be queried and obtaining a corresponding BitMap array address according to the hash value includes:

calculating the hash value of the element to be inquired according to the configured hash function and the hash times;

and obtaining a corresponding BitMap array address according to the modulus of the BitMap length of the hash value.

5. The blacklist querying method according to claim 1, wherein the obtaining a first result corresponding to the corresponding BitMap array address by Redis includes:

acquiring a bit value corresponding to the corresponding BitMap array address through an array operation method provided by Redis;

if the value of the bit is not all 1, the element to be inquired is not in the blacklist;

and if the bit values are all 1, continuing to perform the comparison.

6. The blacklist query method according to claim 5, wherein said comparing the element to be queried with an entry according to the database table of Redis comprises:

querying in a database table of the Redis according to the unique identifier of the element to be queried;

if the effective state corresponding to the element to be queried is effective, the element to be queried is in the blacklist; otherwise, the element to be queried is not in the blacklist.

7. The blacklist querying method according to claim 1, wherein said constructing a BitMap array to store blacklist data further comprises updating a Redis database table, comprising:

reading elements to be updated in batches, and creating a Redis database temporary table;

carrying out BitMap processing on the elements to be stored in sequence, and storing the elements into array elements of the BitMap array;

updating the effective state of the element to be stored in the temporary table of the Redis database;

and replacing the original Redis database table by using the updated temporary Redis database table.

8. The blacklist query method according to claim 7, wherein the sequentially performing the BitMap processing on the elements to be stored, and storing into the array elements of the BitMap array comprises:

carrying out BitMap processing on the elements to be stored in sequence to obtain the corresponding BitMap array address;

setting the value of a bit corresponding to the corresponding BitMap array address to be 1 by an array operation method provided by Redis;

the corresponding validation state in the Redis database table is set to validate.

9. The blacklist querying method according to claim 8, wherein said setting the respective validation status in the Redis database table to validate further comprises retrying, including:

reading an element of which the effective state is effective and the value of a corresponding bit is 0 in the Redis database table;

and carrying out the BitMap processing process again on the elements.

10. The blacklist querying method according to claim 8, wherein said constructing a BitMap array to store blacklist data further comprises deleting the element, including:

and setting the effective state corresponding to the element to be deleted in the Redis database table as invalid.

11. A blacklist query system, comprising:

the building module is used for building a BitMap array to store blacklist data;

the computing module is used for computing a hash value of the element to be queried according to the element to be queried and computing a corresponding BitMap array address according to the hash value;

the matching module is used for acquiring a first result corresponding to the corresponding BitMap array address through Redis and judging whether the first result is matched or not;

the comparison module is used for comparing the element to be queried with the table entry according to the database table of Redis to obtain a second result if the element to be queried is matched with the table entry; if the second result also matches, the element to be queried is in the blacklist.

12. An electronic device, comprising:

a processor;

a memory storing a computer executable program which, when executed by the processor, causes the processor to perform the blacklist query method as claimed in any one of claims 1 to 10.

13. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the blacklist querying method as claimed in any one of claims 1 to 10.

14. A computer program product comprising a computer program which, when executed by a processor, implements a blacklist query method as claimed in any one of claims 1 to 10.