CN116775669A - Data processing method, device, electronic equipment and computer storage medium - Google Patents
Data processing method, device, electronic equipment and computer storage medium Download PDFInfo
- Publication number
- CN116775669A CN116775669A CN202211430884.5A CN202211430884A CN116775669A CN 116775669 A CN116775669 A CN 116775669A CN 202211430884 A CN202211430884 A CN 202211430884A CN 116775669 A CN116775669 A CN 116775669A
- Authority
- CN
- China
- Prior art keywords
- data
- lock
- target
- release time
- time period
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 22
- 238000000034 method Methods 0.000 claims abstract description 49
- 238000004590 computer program Methods 0.000 claims description 15
- 238000012544 monitoring process Methods 0.000 claims description 15
- 230000008602 contraction Effects 0.000 abstract description 4
- 238000012217 deletion Methods 0.000 description 14
- 230000037430 deletion Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 6
- KLDZYURQCUYZBL-UHFFFAOYSA-N 2-[3-[(2-hydroxyphenyl)methylideneamino]propyliminomethyl]phenol Chemical compound OC1=CC=CC=C1C=NCCCN=CC1=CC=CC=C1O KLDZYURQCUYZBL-UHFFFAOYSA-N 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 3
- 201000001098 delayed sleep phase syndrome Diseases 0.000 description 3
- 208000033921 delayed sleep phase type circadian rhythm sleep disease Diseases 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a data processing method, a device, electronic equipment and a computer storage medium, wherein the method comprises the following steps: intercepting a plurality of data operation requests received by a server in a current time period; the data operation request is used for performing writing operation or deleting operation on operation data; counting the data inflow rates corresponding to the operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deleting rate; determining the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate; therefore, the elastic expansion and contraction of the release time of the lock are realized according to the data inflow rate, and the data processing efficiency in a high concurrency scene is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method, a data processing device, an electronic device, and a computer storage medium.
Background
The storage of data often presents such problems: how to maintain consistency of data erasure; relational database management systems (MYSQL, my Structured Query Language), structured query language (Structured Query Language Sever, sql server), oracle and other relational databases can be solved by key association operation modes; however, when the user has strict requirements on the read-write speed of the data, a non-relational database is needed, and the Redis database is considered first.
Redis databases typically use the manner in which a lock mechanism operates, i.e., redis locks, simply locks, to acquire a lock for certain object data when one thread operates on the object data, and wait for execution if the lock is not available when another thread accesses the object data. Although the operation mode can ensure the consistency of the hierarchical association data erasure, the operation mode has the problems of thread waiting and the like, so that the data processing efficiency in a high concurrency scene is lower.
Disclosure of Invention
The embodiment of the application provides a data processing method, a data processing device, electronic equipment and a computer storage medium, which can improve the data processing efficiency in a high concurrency scene.
The technical scheme of the embodiment of the application is realized as follows:
the embodiment of the application provides a data processing method, which comprises the following steps:
intercepting a plurality of data operation requests received by a server in a current time period; the data operation request is used for performing writing operation or deleting operation on operation data;
counting the data inflow rates corresponding to the operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deleting rate;
Determining the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate;
and executing the operation corresponding to each target data operation request based on the determined release time of the lock.
In some embodiments, in a case where the data inflow rate includes a data write rate and a data delete rate, the determining, based on the data inflow rate, a release time of a lock corresponding to each target data operation request received in a next period of time includes:
acquiring write release time and delete release time of the lock in the current time period;
determining a first ratio of the data writing rate to the writing release time of the lock in the current time period and a second ratio of the data deleting rate to the deleting release time of the lock in the current time period;
and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the first ratio and the second ratio.
In some embodiments, the determining, based on the first ratio and the second ratio, a release time of the lock corresponding to each target data operation request received in the next period of time includes:
Adjusting the first ratio and the second ratio so that the first ratio and the second ratio are equal;
and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the adjusted first ratio and second ratio.
In some embodiments, the performing an operation corresponding to each of the target data operation requests based on the determined release time of the lock includes:
for each target data operation request, acquiring a lock of target operation data; the target data operation request is used for performing writing operation or deleting operation on the target operation data;
and locking the target operation data based on the determined release time of the lock, and executing corresponding operation on the target operation data according to the target data operation request.
In some embodiments, the lock for obtaining target operation data includes:
acquiring a lock of the target operation data from a Redis database; the lock of the target operation data is in a key-value form, the key value is the identification of the target operation data, and the value is the release time of the lock of the target operation data.
In some embodiments, the method further comprises:
monitoring the execution process corresponding to each target data operation request until the monitoring time length reaches the determined lock release time, and releasing the locks of the target operation data.
In some embodiments, the method further comprises:
and executing rollback operation on the execution of the target operation data under the condition that the monitoring time length reaches the determined release time of the lock and the lock of the target operation data is detected not to be released.
The embodiment of the application provides a data processing device, which comprises an interception module, a statistics module, a determination module and an execution module, wherein,
the intercepting module is used for intercepting a plurality of data operation requests received by the server side in the current time period; the data operation request is used for performing writing operation or deleting operation on operation data;
the statistics module is used for counting the data inflow rates corresponding to the operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deleting rate;
the determining module is used for determining the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate;
And the execution module is used for executing the operation corresponding to each target data operation request based on the determined release time of the lock.
The embodiment of the application provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the data processing method provided by one or more of the technical schemes when executing the program.
Embodiments of the present application provide a computer storage medium storing a computer program; the computer program, when executed, is capable of implementing the data processing method provided by one or more of the foregoing technical solutions.
The embodiment of the application provides a data processing method, a device, electronic equipment and a computer storage medium, wherein the method comprises the following steps: intercepting a plurality of data operation requests received by a server in a current time period; the data operation request is used for performing writing operation or deleting operation on operation data; counting the data inflow rates corresponding to the operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deleting rate; determining the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate; and executing the operation corresponding to each target data operation request based on the determined release time of the lock.
It can be seen that, in the embodiment of the present application, each data operation request in the current time period is intercepted, so as to count the data inflow rate in the time period, and further dynamically adjust the release time of the lock corresponding to each target data operation request in the next time period, that is, the elastic expansion and contraction of the release time of the lock can be realized; for the high concurrency scene, because the release time of the lock is dynamically adjusted according to the data inflow rate, namely, different data inflow rates correspond to different release times of the lock, the data deleting operation can be ensured to be completed as far as possible within the release time of the lock, the problem of thread waiting is optimized while the consistency of the data deleting operation is ensured, and the data processing efficiency under the high concurrency scene is improved.
Drawings
FIG. 1 is a schematic flow chart of a data processing method according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of writing and deleting operation of operation data according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating another data processing method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings.
The present application will be described in further detail with reference to the drawings and examples. It is to be understood that the examples provided herein are for the purpose of illustration only and are not intended to limit the application. The following examples are provided for carrying out the present application, but not for all examples, and the embodiments described in the present application can be carried out in any combination without any conflict.
It should be noted that, in the present application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a method or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such method or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other related elements in a method or apparatus comprising the element (e.g., a step in a method or an element in an apparatus, e.g., an element may be part of a processor, part of a program or software, etc.).
The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., C and/or D, may represent: the three cases of C alone, C and D together and D alone exist. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of C, D, E, and may mean including any one or more elements selected from the group consisting of C, D and E.
For example, the data processing method provided by the present application includes a series of steps, but the data processing method provided by the present application is not limited to the described steps, and similarly, the data processing apparatus provided by the present application includes a series of modules, but the data processing apparatus provided by the present application is not limited to include the explicitly described modules, and may include modules that are required to be set for acquiring related information or performing processing based on the information.
In the related art, for the meaning of a lock, in brief, each object data stored in the dis database by a user has a lock, and the lock can only be acquired and consumed by a single thread; when the lock has been acquired, other threads can only suspend waiting. In general, the release time of the lock is fixed, and in order to ensure the accuracy of task operation performed by each thread, the release time of the lock is set to be relatively long, so that when more object data need to be processed, a problem of thread waiting exists, and the data processing efficiency in a high concurrency scene is low; in addition, when the number of object data to be processed in the first time period is small, and the number of object data to be processed in the second time period is large, if the release time of the lock set in the first time period is continuously adopted in the second time period, some object data cannot be operated after exceeding the release time of the lock, and further, the risk of data loss exists, and the data processing requirement under a high concurrency scene is difficult to meet.
Wherein the implementation of the Redis lock is related to the atomicity characteristics of the Redis itself, which is single threaded; redis lock corresponds to a tag of an object data, and key-value is usually represented by a key value, which is usually a universally unique identification code (universally unique identifier, uuid) of the object, and value is usually 1 to indicate that the Redis lock of the object data has been acquired, and 0 to indicate that the Redis lock of the object data can be acquired. The benefits of Redis lock are that atomicity of single data operation can be perfectly guaranteed, but when the stored object data is a plurality of data with hierarchy, namely, dependency relationship, the consistency of data deletion and writing is difficult to guarantee by the method.
In order to solve the above problems, the following embodiments are proposed.
In some embodiments of the application, the data processing method may be implemented by a processor in the data processing apparatus, where the processor may be at least one of an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing apparatus (Digital Signal Processing Device, DSPD), a programmable logic device (Programmable Logic Device, PLD), a field programmable gate array (Field Programmable Gate Array, FPGA), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, and a microprocessor.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present application, as shown in fig. 1, the flow may include:
step 100: and intercepting a plurality of data operation requests received by the server side in the current time period.
For example, a user may initiate a data operation request at a client, where the client sends the data operation request initiated by the user to a server, and the server may receive the data operation request sent by the client; here, for each data operation request received by the server, the interceptor may perform an interception operation, and further intercept a plurality of data operation requests received by the server in the current time period.
Here, the current period may be a first period in which the interceptor performs the intercepting operation; the time length of the current time period can be determined according to an actual scene, and the embodiment of the application is not limited to the determination; for example, in a high concurrency scenario, the length of the current time period is typically less than 1 second.
In the embodiment of the application, the data operation request is used for writing or deleting operation on operation data; the operation data may be data with a hierarchical association relationship, and the operation data has corresponding identification information, where the identification information may be uuid of the operation data.
Illustratively, writing or deleting operation on the operation data is an operation for the database; namely, the operation data is written into a certain database, or the operation data in a certain database is deleted; here, the type of the database is not limited, and may be, for example, a Redis database or another type of database.
Here, taking the dis database as an example, the meaning of operation data having a hierarchical association is described with reference to fig. 2, it is assumed that one operation data b currently needs to be written into the dis database, and the writing of the data is divided into two parts: a first section for writing an operation data b into the dis database (corresponding to step (2) in the figure); the second part places an identification (e.g., uuid) of the operation data b into the object data a (step (1) in the corresponding diagram), which is a list that may contain a plurality of uuids similar to the operation data b. Similarly, assuming that an operation data b needs to be deleted from the Redis database, the deletion of the data is also divided into two parts: a first section that deletes an operation data b from the dis database (corresponding to step (4) in the figure); and a second section for deleting uuid of the operation data b from the object data a (step (3) in the correspondence map). At this time, the operation data b and the object data a have a hierarchical association relationship, that is, the operation data b is data having a hierarchical association relationship.
The plurality of data operation requests correspond to the plurality of operation data, and the operation data corresponding to the different data operation requests may be the same or different. For example, in the case where a certain data operation request is for writing operation data b, another data operation request may be for deleting operation data b or deleting operation data c.
Step 101: and counting the data inflow rates corresponding to the operation data completed in the current time period.
In the embodiment of the application, after intercepting a plurality of data operation requests aiming at a plurality of operation data in a current time period according to the steps, further counting the data inflow rates corresponding to the plurality of operation data completed in the time period; here, the data inflow rate may include at least one of a data writing rate and a data deleting rate.
For example, for a plurality of intercepted data operation requests, the following three cases may be included: 1) The plurality of data operation requests are requests for writing operation to the plurality of operation data; 2) The plurality of data operation requests are all requests for deleting operation on the plurality of operation data; 3) The plurality of data operation requests include both a request for writing operation to a plurality of operation data and a request for deleting operation to a plurality of operation data.
It can be understood that, when the plurality of operation data corresponding to the 1 st case are counted, the obtained data inflow rate only includes the data writing rate; when the operation data corresponding to the 2 nd case are counted, the obtained data inflow rate only comprises the data deletion rate; when the plurality of operation data corresponding to the 3 rd case are counted, the obtained data inflow rate includes a data writing rate and a data deleting rate.
Step 102: based on the data inflow rate, a release time of a lock corresponding to each target data operation request received in the next time period is determined.
Here, the next period of time represents a certain period of time after the current period of time, and the period of time is the same as the period of time of the current period of time.
Illustratively, the target data operation request is used for performing a write operation or a delete operation on target operation data, wherein the target operation data is data with a hierarchical association relationship; it should be noted that, the target data operation request and the data operation request are both requests received by the server, and the functions of the two requests are the same, and the difference is only that the time period received by the server is different, and the target operation data is similar to the operation data, which is not repeated here.
In the embodiment of the application, the release time of the lock is also called as the locking time of the lock, and the release time can comprise at least one of the write release time and the delete release time of the lock; it will be appreciated that the release time of the lock includes content that is correspondingly determined based on the content that is included in the data inflow rate.
For example, if the data inflow rate includes only the data write rate, based on the data write rate, a write release time of a lock corresponding to each target data operation request for performing a write operation received in a next period of time may be determined. If the data inflow rate includes only the data deletion rate, a deletion release time of the lock corresponding to each target data operation request for performing a deletion operation received in the next period of time may be determined based on the data deletion rate. If the data inflow rate includes a data write rate and a data delete rate, a write release time of a lock corresponding to each target data operation request for performing a write operation received in a next period of time and a delete release time of a lock corresponding to each target data operation request for performing a delete operation may be determined based on the data write rate and the data delete rate. Because in the case where the data inflow rate includes the data write rate and the data delete rate, the write release time and the delete release time of the lock corresponding to the next period can be determined simultaneously; next, a description will be given of a manner of determining the write release time and the delete release time of the lock, taking this case as an example.
In one embodiment, in a case where the data inflow rate includes a data write rate and a data delete rate, determining, based on the data inflow rate, a release time of a lock corresponding to each target data operation request received in a next period of time may include: acquiring write release time and delete release time of a lock in a current time period; determining a first ratio of a data writing rate to a writing release time of a lock in a current time period and a second ratio of a data deleting rate to a deleting release time of the lock in the current time period; based on the first ratio and the second ratio, a release time of the lock corresponding to each target data operation request received in the next time period is determined.
Here, the write release time and the delete release time of the lock in the current time period may be preset according to actual conditions, and the values of the write release time and the delete release time may be the same or different; the embodiment of the present application is not limited thereto, and for example, the value of both may be set to 5 minutes, or one may be set to 3 minutes, and the other may be set to 5 minutes.
Illustratively, the write release time t of the lock is obtained during the current time period write And delete release time t delete After that, the data writing rate v is further determined write And write release time t of lock in current time period delete Is a first ratio of (2)Data deletion rate v delete And the deletion release time t of the lock in the current time period delete Is a second ratio of (2)Then, based on the two ratios, the release time of the lock corresponding to each target data operation request received in the next time period is determined.
In some embodiments, determining a release time of the lock corresponding to each target data operation request received in a next time period based on the first ratio and the second ratio may include: adjusting the first ratio and the second ratio to make the first ratio and the second ratio equal; and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the adjusted first ratio and second ratio.
In the embodiment of the application, the first ratio and the second ratio are adjusted to be equal, that is, the relation shown in the formula (1) is satisfied:
here, the data writing rate v write And data deletion rate v delete In kilobytes per second (KB/s); write release time t of lock write And delete release time t delete In seconds(s).
Illustratively, during the adjustment of equation (1), the following conditions are satisfied: the values of the first ratio and the second ratio are both larger than 1, and the write release time of the lock after adjustment and the delete release time of the lock are both larger than set values; here, the value of the set value may be set according to the actual situation, and may be set to 30s, for example.
Illustratively, as can be seen from equation (1), if the data write rate v in the current period of time write Increasing the write release time t of the lock write Equal proportion increase, namely, the write release time of the lock after adjustment can be obtained; similarly, if the data deletion rate v in the current time period delete Increasing the delete release time t of the lock delete Equal proportion is increased, namely, the deleting and releasing time of the adjusted lock can be obtained; and determining the write release time of the adjusted lock as the write release time of the lock corresponding to each target data operation request for performing the write operation received in the next time period, and determining the delete release time of the adjusted lock as the delete release time of the lock corresponding to each target data operation request for performing the delete operation received in the next time period.
After intercepting a plurality of target data operation requests in a next time period, continuously counting data inflow rates corresponding to the plurality of target operation data to determine release time of locks corresponding to the data operation requests in the next time period, and so on until the server side does not receive the data operation requests sent by the client side.
It can be seen that, in the embodiment of the application, the dynamic expansion and contraction of the lock release time corresponding to each target data operation request in the next time period of a certain time period are controlled by counting the data inflow rate in the time period; specifically, when the data inflow speed in a certain time period is high, the release time of the lock in the time period is adjusted, so that the release time of the lock in the next time period is relatively prolonged, each operation data is ensured to be written in or deleted in the release time of the lock, the risk of data loss is greatly reduced, and further, the data processing requirement under a high concurrency scene is met. On the contrary, when the data inflow speed in a certain time period is smaller, the release time of the lock in the time period is adjusted, so that the release time of the lock in the next time period is relatively shortened, each operation data can be ensured to be written or deleted as far as possible in the release time of the lock, the thread waiting problem can be solved, and the data processing efficiency in a high concurrency scene is improved.
Step 103: and executing an operation corresponding to each target data operation request based on the determined release time of the lock.
In some embodiments, performing an operation corresponding to each target data operation request based on the determined release time of the lock may include: for each target data operation request, acquiring a lock of target operation data; and locking the target operation data based on the determined release time of the lock, and executing corresponding operation on the target operation data according to each target data operation request.
Illustratively, a lock that obtains target operational data may include: acquiring a lock of target operation data from a Redis database; it should be noted that, locks involved in the embodiments of the present application are all locks in key-value form; wherein, for the lock of the operation data, the key value is the identification of the operation data, and the value is the release time of the lock of the operation data; for the lock of the target operation data, the key value is the identification of the target operation data, and the value is the release time of the lock of the target operation data.
For example, after the lock of the target operation data is obtained, the lock may be locked according to the release time of the lock determined in the above steps, and the corresponding writing or deleting operation may be performed on the target operation data according to the target data operation request during the release time of the lock.
In some embodiments, the above method further comprises: and monitoring the execution process corresponding to each target data operation request until the monitoring time reaches the determined release time of the lock, and releasing the lock of the target operation data.
For example, the execution process corresponding to each target data operation request may be monitored by a monitoring function or a monitoring program, and when the monitoring duration reaches the determined release time of the lock, the release operation is performed on the lock of the target operation data; generally, if the execution of the target operation data is not abnormal or interrupted, the lock is actively released.
Here, after the lock of the target operation data is released, the released lock returns to the dis database.
In some embodiments, the above method may further comprise: and executing rollback operation on the target operation data under the condition that the monitoring time length reaches the determined release time of the lock and the fact that the lock of the target operation data is not released is detected.
For example, if the listening period reaches the determined release time of the lock, and the lock of the target operation data is not released, it is indicated that there may be some abnormality in the execution process of the target operation data, at this time, the execution of the target operation data may be performed by performing a rollback operation, that is, performing the relevant operation again after the execution of the target operation data is rolled back to the state before the execution, so that the accuracy of the execution process may be ensured. Here, the rollback operation may be implemented by adding a transaction operation to the data execution process, or may be implemented by other means.
If the rollback operation is performed, or the lock of the target operation data is detected not to be released, the exception needs to be captured and a log is printed to remind the user to process the exception in time.
The embodiment of the application provides a data processing method, which comprises the following steps: intercepting a plurality of data operation requests received by a server in a current time period; the data operation request is used for performing a writing operation or a deleting operation on the operation data; counting the data inflow rates corresponding to the completion of a plurality of operation data in the current time period; the data inflow rate includes at least one of a data writing rate and a data deleting rate; determining the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate; and executing an operation corresponding to each target data operation request based on the determined release time of the lock. It can be seen that, in the embodiment of the present application, each data operation request in the current time period is intercepted, so as to count the data inflow rate in the time period, and further dynamically adjust the release time of the lock corresponding to each target data operation request in the next time period, that is, the elastic expansion and contraction of the release time of the lock can be realized; for the high concurrency scene, because the release time of the lock is dynamically adjusted according to the data inflow rate, namely, different data inflow rates correspond to different release times of the lock, the data deleting operation can be ensured to be completed as far as possible within the release time of the lock, the problem of thread waiting is optimized while the consistency of the data deleting operation is ensured, and the data processing efficiency under the high concurrency scene is improved.
In order to further embody the object of the present application, the embodiments of the present application are further described on the basis of the above embodiments.
In a high concurrency scenario, a large number of access requests (corresponding to the data operation requests or the target data operation requests) are generally received, and data writing operation and deleting operation need to be continuously performed on the database; here, taking the above-described fig. 2 as an example, there may be various execution orders regarding the steps (1) (2) (3) (4) shown in fig. 2, and ideally, when one writing operation and deleting operation for operation data are simultaneously received (assuming that the execution of step (1) is prioritized over the step (2), the execution of step (3) is prioritized over the step (4)), the optimal execution order of steps is (1) (2) (3) (4) or (3) (4) (1) (2), so that dirty data is not seen when the user performs a reading operation during the execution of any step; however, in a practical scenario, such a step execution sequence (1) (3) (4) (2) or (3) (1) (2) (4) may occur; it can be seen that in these cases, abnormal situations such as deletion of operation data to be written or deletion of operation data not to be deleted occur, and these situations are difficult to check and modify online, which often causes network accidents.
In order to overcome the above problems, the embodiment of the present application uses an interceptor to acquire a plurality of data operation requests received by a server in a current time period, and optimizes a dis lock mechanism in the related art based on the intercepted data operation requests.
Illustratively, the optimization procedure for the Redis lock mechanism is as follows, and the key-value is also used to maintain the lock of the object data (corresponding to the above-described operation data or the target operation data), unlike the related art, the value stores not 0 and 1, but the release time of the lock of the object data. A significant risk with a conventional Redis lock is that the lock may not be released and the object data may never be acquired. Here, the Redis lock is optimized to prevent the lock from being actively released.
Illustratively, the situation where the lock is not actively released may include two of the following: one is that during the execution of intermediate steps (1) (2) (3) (4), an exception occurs without being trapped, resulting in the lock release procedure not being executed; another is that the intermediate steps (1) (2) (3) (4) are performed too long, resulting in a large number of tasks piling up and overflowing the memory. Therefore, time is used herein to achieve automatic release, also known as passive release, of locks of object data. The process of releasing the lock will be described with reference to fig. 3.
Fig. 3 is a schematic flow chart of another data processing method according to an embodiment of the present application, where, as shown in fig. 3, the graph includes an interceptor and a Redis database, and the flow may include: each time a deleting operation or a writing operation of the object data is received by the server, automatically reading a lock of the object data from the Redis database, and if the lock of the object data is obtained, locking the object data, namely, locking the object data, and setting a value of the lock of the object data to be a system. When the release time t of the lock is set to 5 minutes, it means that the lock will be automatically released after 5 minutes. If no lock is acquired after 5 minutes, wait for a time period t and request again to acquire a lock. When the smooth execution of the steps (1), 2), 3 and 4 is completed, the system will release the lock, i.e. actively release the current lock, and in general, no exception or interruption occurs in the program execution process, the lock will be actively released, and the lock will return to the Redis database after being released.
For example, in the case where the above two locks cannot be actively released, the following targeted processing may be performed. An exception may be captured and journaled for the first scenario; and (3) monitoring the duration of the intermediate execution steps (1) (2) (3) (4) in the second situation, and throwing out an abnormality and capturing if the release time t of the lock is exceeded. Here, the steps (1) and (2) and the steps (3) and (4) may be added simultaneously with each other, and when an abnormality occurs, the operations such as writing deletion and the like performed on the target data may be rolled back.
It can be seen that the data processing method provided by the embodiment of the application optimizes parameter setting, realizes dynamic adjustment of the release time of the lock, saves the cost of manual intervention, and reduces the redundancy work of parameter adjustment. In addition, the storage flow of Redis-level associated data is realized based on the dynamic lock, so that a universal storage scheme can be provided for more service scenes, and the method has a certain use prospect.
Fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application, as shown in fig. 4, where the apparatus includes: an interception module 400, a statistics module 401, a determination module 402, and an execution module 403, wherein:
the interception module 400 is configured to intercept a plurality of data operation requests received by the server in a current time period; the data operation request is used for performing writing operation or deleting operation on operation data;
a statistics module 401, configured to count data inflow rates corresponding to a plurality of operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deleting rate;
a determining module 402, configured to determine, based on the data inflow rate, a release time of a lock corresponding to each target data operation request received in a next period of time;
And an execution module 403, configured to execute an operation corresponding to the each target data operation request based on the determined release time of the lock.
In some embodiments, in a case where the data inflow rate includes a data write rate and a data delete rate, the determining module 402 is configured to determine, based on the data inflow rate, a release time of a lock corresponding to each target data operation request received in a next period of time, including:
acquiring write release time and delete release time of the lock in the current time period;
determining a first ratio of the data writing rate to the writing release time of the lock in the current time period and a second ratio of the data deleting rate to the deleting release time of the lock in the current time period;
and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the first ratio and the second ratio.
In some embodiments, the determining module 402, configured to determine, based on the first ratio and the second ratio, a release time of a lock corresponding to each target data operation request received in the next period of time, includes:
Adjusting the first ratio and the second ratio so that the first ratio and the second ratio are equal;
and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the adjusted first ratio and second ratio.
In some embodiments, the executing module 403 is configured to execute, based on the determined release time of the lock, an operation corresponding to each target data operation request, including:
for each target data operation request, acquiring a lock of target operation data; the target data operation request is used for performing writing operation or deleting operation on the target operation data;
and locking the target operation data based on the determined release time of the lock, and executing corresponding operation on the target operation data according to the target data operation request.
In some embodiments, the executing module 403, configured to obtain the lock of the target operation data, includes:
acquiring a lock of the target operation data from a Redis database; the lock of the target operation data is in a key-value form, the key value is the identification of the target operation data, and the value is the release time of the lock of the target operation data.
In some embodiments, the apparatus further comprises a listening module for:
monitoring the execution process corresponding to each target data operation request until the monitoring time length reaches the determined lock release time, and releasing the locks of the target operation data.
In some embodiments, the apparatus further comprises a rollback module to:
and executing rollback operation on the execution of the target operation data under the condition that the monitoring time length reaches the determined release time of the lock and the lock of the target operation data is detected not to be released.
In practical applications, the interception module 400, the statistics module 401, the determination module 402, the execution module 403, the listening module, and the rollback module may be implemented by a processor located in the electronic device, where the processor may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, a controller, a microcontroller, and a microprocessor.
In addition, each functional module in the present embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional modules.
The integrated units, if implemented in the form of software functional modules, may be stored in a computer-readable storage medium, if not sold or used as separate products, and based on such understanding, the technical solution of the present embodiment may be embodied essentially or partly in the form of a software product, which is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or processor (processor) to perform all or part of the steps of the method of the present embodiment. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk or an optical disk, or the like, which can store program codes.
Specifically, the computer program instructions corresponding to one data processing method in the present embodiment may be stored on a storage medium such as an optical disc, a hard disc, or a usb disk, and when the computer program instructions corresponding to one data processing method in the storage medium are read or executed by an electronic device, any one of the data processing methods in the foregoing embodiments is implemented.
Based on the same technical concept as the foregoing embodiments, referring to fig. 5, an electronic device 500 provided by an embodiment of the present application may include: a memory 501 and a processor 502; wherein, the liquid crystal display device comprises a liquid crystal display device,
a memory 501 for storing computer programs and data;
a processor 502 for executing a computer program stored in a memory to implement any of the data processing methods of the previous embodiments.
In practical applications, the memory 501 may be a volatile memory (RAM); or a non-volatile memory (non-volatile memory), such as ROM, flash memory (flash memory), hard Disk (HDD), or Solid State Drive (SSD); or a combination of the above types of memory and provides instructions and data to the processor 502.
The processor 502 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, a controller, a microcontroller, and a microprocessor. It will be appreciated that the electronics for implementing the above-described processor functions may be other for different data processing systems, and embodiments of the present application are not particularly limited.
In some embodiments, the functions or modules included in the apparatus provided by the embodiments of the present application may be used to perform the methods described in the foregoing method embodiments, and specific implementations thereof may refer to descriptions of the foregoing method embodiments, which are not repeated herein for brevity.
The foregoing description of various embodiments is intended to highlight differences between the various embodiments, which may be the same or similar to each other by reference, and is not repeated herein for the sake of brevity.
The methods disclosed in the method embodiments provided by the application can be arbitrarily combined under the condition of no conflict to obtain a new method embodiment.
The features disclosed in the embodiments of the products provided by the application can be combined arbitrarily under the condition of no conflict to obtain new embodiments of the products.
The features disclosed in the embodiments of the method or the device provided by the application can be arbitrarily combined under the condition of no conflict to obtain a new embodiment of the method or the device.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above is only a preferred embodiment of the present application, and is not intended to limit the scope of the present application.
Claims (10)
1. A method of data processing, the method comprising:
intercepting a plurality of data operation requests received by a server in a current time period; the data operation request is used for performing writing operation or deleting operation on operation data;
counting the data inflow rates corresponding to the operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deleting rate;
determining the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate;
and executing the operation corresponding to each target data operation request based on the determined release time of the lock.
2. The method of claim 1, wherein, in the case where the data inflow rate includes a data write rate and a data delete rate, the determining, based on the data inflow rate, a release time of a lock corresponding to each target data operation request received in a next period of time includes:
acquiring write release time and delete release time of the lock in the current time period;
Determining a first ratio of the data writing rate to the writing release time of the lock in the current time period and a second ratio of the data deleting rate to the deleting release time of the lock in the current time period;
and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the first ratio and the second ratio.
3. The method of claim 2, the determining, based on the first ratio and the second ratio, a release time of a lock corresponding to each target data operation request received in the next time period, comprising:
adjusting the first ratio and the second ratio so that the first ratio and the second ratio are equal;
and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the adjusted first ratio and second ratio.
4. The method of claim 1, wherein the performing an operation corresponding to each of the target data operation requests based on the determined release time of the lock comprises:
for each target data operation request, acquiring a lock of target operation data; the target data operation request is used for performing writing operation or deleting operation on the target operation data;
And locking the target operation data based on the determined release time of the lock, and executing corresponding operation on the target operation data according to the target data operation request.
5. The method of claim 4, wherein the acquiring the lock of the target operational data comprises:
acquiring a lock of the target operation data from a Redis database; the lock of the target operation data is in a key-value form, the key value is the identification of the target operation data, and the value is the release time of the lock of the target operation data.
6. The method according to claim 4, wherein the method further comprises:
monitoring the execution process corresponding to each target data operation request until the monitoring time length reaches the determined lock release time, and releasing the locks of the target operation data.
7. The method of claim 6, wherein the method further comprises:
and executing rollback operation on the execution of the target operation data under the condition that the monitoring time length reaches the determined release time of the lock and the lock of the target operation data is detected not to be released.
8. A data processing apparatus, the apparatus comprising:
the intercepting module is used for intercepting a plurality of data operation requests received by the server side in the current time period; the data operation request is used for performing writing operation or deleting operation on operation data;
the statistics module is used for counting the data inflow rates corresponding to the operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deleting rate;
the determining module is used for determining the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate;
and the execution module is used for executing the operation corresponding to each target data operation request based on the determined release time of the lock.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of claims 1 to 7 when the program is executed.
10. A computer storage medium having stored thereon a computer program, which when executed by a processor implements the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211430884.5A CN116775669A (en) | 2022-11-15 | 2022-11-15 | Data processing method, device, electronic equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211430884.5A CN116775669A (en) | 2022-11-15 | 2022-11-15 | Data processing method, device, electronic equipment and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116775669A true CN116775669A (en) | 2023-09-19 |
Family
ID=87986638
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211430884.5A Pending CN116775669A (en) | 2022-11-15 | 2022-11-15 | Data processing method, device, electronic equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116775669A (en) |
-
2022
- 2022-11-15 CN CN202211430884.5A patent/CN116775669A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11157370B2 (en) | Consistent backup of a distributed database system | |
| JP5129245B2 (en) | Preserving shadow copy data during replication | |
| US11409652B2 (en) | Estimating worker nodes needed for performing garbage collection operations | |
| US8583608B2 (en) | Maximum allowable runtime query governor | |
| US20090125572A1 (en) | Method for managing retention of data on worm disk media based on event notification | |
| US20120221811A1 (en) | Retention management in a worm storage system | |
| US8020046B2 (en) | Transaction log management | |
| US8479046B1 (en) | Systems, methods, and computer readable media for tracking pool storage space reservations | |
| US20090240742A1 (en) | Transaction Log Management | |
| US11765099B2 (en) | Resource allocation using distributed segment processing credits | |
| US20200310965A1 (en) | Deleting data in storage systems that perform garbage collection | |
| US11093290B1 (en) | Backup server resource-aware discovery of client application resources | |
| US20200026428A1 (en) | Smart auto-backup of virtual machines using a virtual proxy | |
| US9785516B2 (en) | Method and system to remove a service from a distributed database system | |
| US20090100434A1 (en) | Transaction management | |
| CN113157600A (en) | Space allocation method of shingled hard disk, file storage system and server | |
| US11106539B2 (en) | Rule book based retention management engine | |
| US11068375B2 (en) | System and method for providing machine learning based memory resiliency | |
| CN116775669A (en) | Data processing method, device, electronic equipment and computer storage medium | |
| US10289721B2 (en) | Query management based on amount of data change | |
| CN108376104B (en) | Node scheduling method and device and computer readable storage medium | |
| US20210326302A1 (en) | Determining shared blocks among snapshots in a storage system | |
| US20180046691A1 (en) | Query governor rules for data replication | |
| CN107239474B (en) | Data recording method and device | |
| US11422970B2 (en) | Handling of data archiving events in a replication system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |