CN113392110A

CN113392110A - Data processing method and device, electronic equipment and storage medium

Info

Publication number: CN113392110A
Application number: CN202110654382.XA
Authority: CN
Inventors: 王杰颖
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Priority date: 2021-06-11
Filing date: 2021-06-11
Publication date: 2021-09-14

Abstract

The embodiment of the invention is suitable for the technical field of data processing, and provides a data processing method, a data processing device, electronic equipment and a storage medium, wherein the data processing method comprises the following steps: under the condition that a data request is received, generating a corresponding first hash value based on the data request; acquiring a first number corresponding to the data request from a set number pool based on the first hash value; setting at least two different numbers stored in the number pool; and generating a key corresponding to the data request based on the first number.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.

Background

Redis is a key-value storage system, and when a large amount of traffic accesses Redis, for example, when thousands of millions of requests access a certain key on Redis, the traffic is too concentrated, which causes a shutdown of a Redis server. In the related technology, the number of corresponding keys is increased on the Redis side, and the increased keys are used for sharing the access pressure of the original keys. The related art generates a new key by random number combination, but the random number combination mode has no traceability, and another key is generated next time based on the request.

Disclosure of Invention

In order to solve the above problem, embodiments of the present invention provide a data processing method, an apparatus, an electronic device, and a storage medium, so as to at least solve the problem that keys generated by combining random numbers in the related art do not have traceability.

The technical scheme of the invention is realized as follows:

in a first aspect, an embodiment of the present invention provides a data processing method, where the method includes:

under the condition that a data request is received, generating a corresponding first hash value based on the data request;

acquiring a first number corresponding to the data request from a set number pool based on the first hash value; at least two different numbers are stored in the set number pool;

and generating a key corresponding to the data request based on the first number.

In the above aspect, when generating at least two digits in the set digit pool based on set parameters, the method includes:

generating the at least two different numbers based on a two-layer nested for-loop traversal setting parameter; each number of the at least two different numbers is distributed in the set number pool in the order from small to large; the difference of all adjacent numbers of the at least two different numbers is equal.

In the above aspect, the method further includes:

and setting the input custom parameters as the setting parameters.

In the foregoing solution, the obtaining a first number corresponding to the data request from a set number pool based on the first hash value includes:

and acquiring a first number corresponding to the data request from the set number pool by using a tailMap method of TreeMap based on the first hash value.

In the foregoing solution, the generating a key corresponding to the data request based on the first number includes:

determining a first parameter corresponding to the data request; the first parameter represents the identity of the corresponding key;

and combining the first number and the first parameter to obtain a key corresponding to the data request.

In the foregoing solution, the data processing method further includes:

and determining a first number in a set number pool as a first number corresponding to the data request when the first number corresponding to the data request cannot be acquired from the set number pool based on the first hash value.

In the foregoing solution, the generating a corresponding first hash value based on the data request includes:

and performing hash calculation on the request parameters of the data request to obtain the first hash value.

In a second aspect, an embodiment of the present invention provides a data processing apparatus, including:

the device comprises a first generation module, a second generation module and a third generation module, wherein the first generation module is used for generating a corresponding first hash value based on a data request under the condition that the data request is received;

the acquisition module is used for acquiring a first number corresponding to the data request from a set number pool based on the first hash value; at least two different numbers are stored in the set number pool;

and the second generation module is used for generating the key corresponding to the data request based on the first number.

In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, where the processor and the memory are connected to each other, where the memory is used to store a computer program, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the steps of the data processing method provided in the first aspect of the embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including: the computer-readable storage medium stores a computer program. Which when executed by a processor performs the steps of the data processing method as provided by the first aspect of an embodiment of the invention.

In the embodiment of the invention, under the condition of receiving the data request, a corresponding first hash value is generated based on the data request, a first number corresponding to the data request is acquired from a set number pool based on the first hash value, and a key corresponding to the data request is generated based on the first number. Wherein, at least two different numbers are stored in the set number pool. The embodiment of the invention generates different keys based on different data requests, the generated keys have traceability, the same data request always corresponds to the same key, the method is very suitable for a data statistics scene, can be used for data deduplication, ensures that repeated statistics cannot occur, and ensures the accuracy of data statistics results.

Drawings

Fig. 1 is a schematic flow chart illustrating an implementation of a data processing method according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of another implementation of a data processing method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of another data processing flow provided by an embodiment of the invention;

FIG. 4 is a diagram of a data processing apparatus according to an embodiment of the present invention;

fig. 5 is a schematic diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the process of counting the total data by using the Redis cache, under the condition of large-flow access to the Redis, for example, when thousands of millions of requests access to a certain key on the Redis, the flow is excessively concentrated and reaches the upper limit of a physical network card, so that the shutdown of the Redis server is caused. In order to solve such problems, in the related art, the number of corresponding keys is increased on the Redis side, and a currently common mode is to generate a new key through random number combination and share the access pressure of the original key through the new key.

But the keys generated by the random number combination are not traceable and a different key is generated next time based on the request. This makes the same user's access at different time periods fall on different keys when using Redis statistical users, resulting in inaccurate data statistics.

In view of the above disadvantages of the related art, embodiments of the present invention provide a data processing method, which at least enables generated keys to have traceability, and the same data request always corresponds to the same key, thereby ensuring accuracy of a data statistics result. In order to explain the technical means of the present invention, the following description will be given by way of specific examples.

Fig. 1 is a schematic flow chart illustrating an implementation process of a data processing method according to an embodiment of the present invention, where an execution main body of the data processing method may be an electronic device, and the electronic device may be a desktop computer, a notebook computer, a server, and the like. Referring to fig. 1, the data processing method includes:

s101, when a data request is received, generating a corresponding first hash value based on the data request.

Here, the data request may be any request for Redis, for example, the data request may be to access a certain key in the Redis.

In an embodiment, the generating a corresponding first hash value based on the data request includes:

Here, the request parameters of the data request include a user name, a device number, and the like. And carrying out Hash calculation on the request parameters to obtain a first Hash value. In practical applications, the first hash value of the request parameter may be calculated using the MurMurHash algorithm. The MurmurHash is a non-encryption type hash function and is suitable for general hash retrieval operation.

The first hash value generated based on the same data request is unique, and the first hash value generated based on different data requests is necessarily different.

S102, acquiring a first number corresponding to the data request from a set number pool based on the first hash value; at least two different numbers are stored in the set number pool.

The set number pool is a pre-created data set for storing numbers, and the number of numbers in the set number pool is determined by the service request volume of Redis.

Before the service is loaded, the number pool needs to be initialized, and the numbers in the number pool are generated based on the parameters customized by a developer.

In one embodiment, in generating at least two digits in the set digit pool based on set parameters, the method includes:

Here, the custom parameters input by the developer are set as the setting parameters. In practical application, at least two different numbers can be calculated by using a MurMurHash algorithm for the set parameters.

In the process of generating at least two numbers, the parameters are set by using a two-layer nested for loop traversal, so that the generated at least two different numbers are uniformly and smoothly distributed in the set number pool. For example, 10 numbers are generated in total, the distribution of the generated 10 numbers in the set number pool is (1, 3, 5, 7, 9, 11, 13, 15, 17, 19), wherein each number is distributed in the order from small to large, and the difference value of all adjacent numbers is 2.

In practical application, a two-layer for loop mode is used, and Long type numbers generated by calling the MurMurHash algorithm are stored in the TreeMap. The setting parameter may be i + "1" + n, the generated digital value is Long type, and all the finally generated digits are stored in TreeMap < Long, String >, where Long is the Hash value of the algorithm, String is description information, and keys refers to the digit pool.

In an embodiment, the obtaining a first number corresponding to the data request from a set number pool based on the first hash value includes:

For example, assuming that the first hash value corresponding to the data request is a, based on the set number pool, a method of tailmap (a).

In one embodiment, the data processing method further comprises:

When the method for calling the tailMap is abnormal, namely the first number corresponding to the data request is not obtained from the set number pool, the TreeMap is used for obtaining the firstKey (), namely the first number in the set number pool is used as the first number corresponding to the data request.

S103, generating a key corresponding to the data request based on the first number.

Here, the acquired first number may be directly used as a key corresponding to the data request, or the first number may be used as a part of the key.

Referring to fig. 2, in an embodiment, the generating a key corresponding to the data request based on the first number includes:

s201, determining a first parameter corresponding to the data request; the first parameter characterizes an identity of the corresponding key.

The first parameter is used for distinguishing different keys, so that the developer can know some information through the first parameter. For example, the first parameter is "APP + page 1", and through the first parameter, the developer can know that the role of the corresponding data request of the key is to access page 1 of the APP.

S202, combining the first number and the first parameter to obtain a key corresponding to the data request.

Through combining the first number and the first parameter, developers can conveniently understand the keys, conflict with other keys is avoided, and distinguishing between the keys and other keys is facilitated.

In practical applications, the generated key may be stored in Map < String, Object >, where String is the key and Object is corresponding information, for example, if the key is a statistical number, the Object is the statistical number.

Therefore, the same key can be obtained by calculating the same data request every time, and the key is traceable, efficient and available, and is very suitable for the duplicate removal scene of people or equipment. For example, when the number of the access users is counted, the access of the same user in different time periods always corresponds to the same key, so that the situation of repeatedly counting the same user does not occur, and the accuracy of the counting result is ensured. And the same data request always corresponds to the same key, and other data requests cannot request the key, so that the problem of hot keys caused by a single key cannot occur.

Referring to fig. 3, fig. 3 is a schematic diagram of a data processing flow provided by an application embodiment of the present invention. First, a number pool is initialized. The total number of the numbers in the digital pool is controlled by traversing the set parameters through a two-layer nested for loop, so that the numbers are uniformly and stably distributed in the digital pool. For example, 10 numbers are generated in total, the distribution of the generated 10 numbers in the set number pool is (1, 3, 5, 7, 9, 11, 13, 15, 17, 19), wherein each number is distributed in the order from small to large, and the difference value of all adjacent numbers is 2. And performing hash calculation on the set parameters by adopting a MurMurHash algorithm, and storing the hash value obtained by calculation in TreeMap. The setting parameters are parameters defined by developers.

Then, based on request parameters of the data request, a MurMurHash algorithm is used for generating a Hash value A, based on the Hash value A, the current segmentation is intercepted by using a tailMap method of TreeMap, a firstKey () is obtained, a number B in a number pool is obtained, and the number B is used as a key corresponding to the data request or a part of the key. And in the case that the number in the number pool cannot be obtained abnormally, using TreeMap to obtain the firstKey () as the key corresponding to the data request or part of the key. The request parameters of the data request include a user name, a device number and the like. The MurmurHash is a non-encryption type hash function and is suitable for general hash retrieval operation.

And finally, storing the generated key in Map < String, Object >, wherein the generated key has traceability, the same data request always corresponds to the same key, and the method is very suitable for a data statistics scene, can be used for data deduplication, ensures that repeated statistics cannot occur, and ensures the accuracy of a data statistics result. And the same data request always corresponds to the same key, and other data requests cannot request the key, so that the problem of hot keys caused by a single key cannot occur.

It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The technical means described in the embodiments of the present invention may be arbitrarily combined without conflict.

In addition, in the embodiments of the present invention, "first", "second", and the like are used for distinguishing similar objects, and are not necessarily used for describing a specific order or a sequential order.

Referring to fig. 4, fig. 4 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 5, the apparatus includes: the device comprises a first generation module, an acquisition module and a second generation module.

In one embodiment, the obtaining module, when generating at least two digits in the set digit pool based on the set parameter, is configured to:

In one embodiment, the apparatus further comprises:

and the setting module is used for setting the input custom parameters as the setting parameters.

In an embodiment, the obtaining module, when obtaining the first number corresponding to the data request from the set number pool based on the first hash value, is configured to:

In an embodiment, the second generating module, when generating the key corresponding to the data request based on the first number, is configured to:

The first parameter is used for distinguishing different keys, so that the developer can know some information through the first parameter. For example, the first parameter is "APP + page 1", and through the first parameter, the developer can know that the role of the corresponding data request of the key is to access page 1 of the APP. Through combining the first number and the first parameter, developers can conveniently understand the keys, conflict with other keys is avoided, and distinguishing between the keys and other keys is facilitated.

In one embodiment, the data processing apparatus further comprises:

and the determining module is used for determining a first number in a set number pool as the first number corresponding to the data request under the condition that the first number corresponding to the data request cannot be acquired from the set number pool based on the first hash value.

In an embodiment, the first generating module, when generating the corresponding first hash value based on the data request, is configured to:

The first hash value of the request parameter may be calculated using the MurMurHash algorithm. The MurmurHash is a non-encryption type hash function and is suitable for general hash retrieval operation.

In practical applications, the first generating module, the obtaining module and the second generating module may be implemented by a Processor in an electronic device, such as a Central Processing Unit (CPU), a Digital Signal Processor (DSP), a Micro Control Unit (MCU), or a Programmable Gate Array (FPGA).

It should be noted that: in the data processing apparatus provided in the above embodiment, when performing data processing, only the division of the above modules is exemplified, and in practical applications, the processing may be distributed to different modules as needed, that is, the internal structure of the apparatus may be divided into different modules to complete all or part of the processing described above. In addition, the data processing apparatus and the data processing method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments for details, which are not described herein again.

Based on the hardware implementation of the program module, in order to implement the method of the embodiment of the present application, an embodiment of the present application further provides an electronic device. Fig. 5 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application, and as shown in fig. 5, the electronic device includes:

the communication interface can carry out information interaction with other equipment such as network equipment and the like;

and the processor is connected with the communication interface to realize information interaction with other equipment, and is used for executing the method provided by one or more technical schemes on the electronic equipment side when running a computer program. And the computer program is stored on the memory.

Of course, in practice, the various components in an electronic device are coupled together by a bus system. It will be appreciated that a bus system is used to enable communications among the components. The bus system includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as a bus system in fig. 5.

The memory in the embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program for operating on an electronic device.

It will be appreciated that the memory can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile Memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a magnetic random access Memory (FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical disk, or a Compact Disc Read-Only Memory (CD-ROM); the magnetic surface storage may be disk storage or tape storage. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced DRAM), Synchronous Dynamic Random Access Memory (SLDRAM), Direct Memory (DRmb Access), and Random Access Memory (DRAM). The memories described in the embodiments of the present application are intended to comprise, without being limited to, these and any other suitable types of memory.

The method disclosed in the embodiments of the present application may be applied to a processor, or may be implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The processor described above may be a general purpose processor, a DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The processor may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium located in a memory where a processor reads the programs in the memory and in combination with its hardware performs the steps of the method as previously described.

Optionally, when the processor executes the program, the corresponding process implemented by the electronic device in each method of the embodiment of the present application is implemented, and for brevity, no further description is given here.

In an exemplary embodiment, the present application further provides a storage medium, specifically a computer storage medium, for example, a first memory storing a computer program, where the computer program is executable by a processor of an electronic device to perform the steps of the foregoing method. The computer readable storage medium may be Memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface Memory, optical disk, or CD-ROM.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus, electronic device and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.

Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code.

Alternatively, the integrated units described above in the present application may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as independent products. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented or portions thereof contributing to the prior art may be embodied in the form of a software product stored in a storage medium, and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code.

The technical means described in the embodiments of the present application may be arbitrarily combined without conflict.

In addition, in the examples of the present application, "first", "second", and the like are used for distinguishing similar objects, and are not necessarily used for describing a specific order or a sequential order.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data processing method, comprising:

2. The method of claim 1, wherein in generating at least two digits in the set digit pool based on set parameters, the method comprises:

3. The method of claim 2, further comprising:

and setting the input custom parameters as the setting parameters.

4. The method of claim 1, wherein obtaining the first number corresponding to the data request from a pool of set numbers based on the first hash value comprises:

5. The method of claim 1, wherein generating the key corresponding to the data request based on the first number comprises:

6. The method of claim 1, wherein the data processing method further comprises:

7. The method of claim 1, wherein generating the corresponding first hash value based on the data request comprises:

8. A data processing apparatus, comprising:

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions that, when executed by a processor, cause the processor to carry out the data processing method according to any one of claims 1 to 7.