CN110262756B - Method and device for caching data - Google Patents
Method and device for caching data Download PDFInfo
- Publication number
- CN110262756B CN110262756B CN201910547977.8A CN201910547977A CN110262756B CN 110262756 B CN110262756 B CN 110262756B CN 201910547977 A CN201910547977 A CN 201910547977A CN 110262756 B CN110262756 B CN 110262756B
- Authority
- CN
- China
- Prior art keywords
- disk
- data
- value
- parameter
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the disclosure discloses a method and a device for caching data. One embodiment of the method comprises: generating a data operation task according to a data operation request sent by a first user; in response to the fact that memory data overflows to a disk serving as a temporary space in the process of executing the data operation task, determining whether the utilization rate of the disk is limited or not according to the value of a preset disk utilization rate parameter; in response to determining that the disk has a usage limit and that the value of the disk space full flag of the disk is false, performing the following operational steps: determining whether the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data; if so, determining whether the predicted utilization rate of the disk is greater than the value of the disk utilization rate parameter; and in response to determining that the predicted utilization rate of the disk is not greater than the value of the disk utilization rate parameter, caching the overflowed memory data to the disk. This embodiment improves the stability of the database.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to a method and a device for caching data.
Background
The disk is the most important storage medium for the database, and when the disk space is severely insufficient, the task performed by the database may be failed, and even the whole database may be down. The traditional solution to the problem of using space of a disk is to add a monitoring alarm for the use condition of the disk: when the disk space is insufficient and the utilization rate is too high, a corresponding alarm is sent to inform operation and maintenance personnel, and the operation and maintenance personnel carry out manual intervention processing.
Generally, the causes of the disk space shortage alarm are mainly classified into 2 types: 1) The real data volume and log storage are out of limit; 2) Operations such as aggregation, table association, sorting and the like are involved, and when the memory is insufficient, the data can be cached to the disk, so that the utilization rate of the disk can reach 100% instantly, and the whole cluster is crashed. The warning for insufficient disk space caused by the first kind of reasons can be predicted in advance, so that the warning can be avoided by operations such as capacity expansion, historical data deletion and the like. The warning for insufficient disk space caused by the second type of reasons is often unpredictable in advance, so that the problem is difficult to effectively solve.
Disclosure of Invention
The embodiment of the disclosure provides a method and a device for caching data.
In a first aspect, an embodiment of the present disclosure provides a method for caching data, where the method includes: generating a data operation task according to a data operation request sent by a first user; in response to the fact that memory data overflows to a disk serving as a temporary space in the process of executing the data operation task, determining whether the disk has a utilization rate limit or not according to a preset value of a disk utilization rate parameter; in response to determining that the disk usage limit exists and the value of the disk space full flag of the disk is false, performing the following steps: determining whether the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data; if the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data, determining whether the predicted utilization rate of the disk is larger than the value of the disk utilization rate parameter, wherein the predicted utilization rate of the disk is determined according to the data volume of the overflowed memory data, the disk used space of the disk and the total disk space; and in response to determining that the predicted utilization rate of the disk is not greater than the value of the disk utilization rate parameter, caching the overflowed memory data to the disk.
In some embodiments, the above operation further comprises: and in response to determining that the predicted utilization rate of the disk is greater than the value of the disk utilization rate parameter, executing a task rollback operation for the data operation task, and setting the value of the disk space full flag of the disk to true.
In some embodiments, the above operation further comprises: and if the capacity of the residual space of the disk is not larger than the data volume of the overflowed memory data, executing task rollback operation aiming at the data operation task.
In some embodiments, the above method further comprises: and executing a task rollback operation on the data operation task in response to determining that the utilization rate limit of the disk exists and the value of the disk space full flag of the disk is true.
In some embodiments, the above method further comprises: and sending error prompt information to the terminal used by the first user in response to the task rollback operation executed for the data operation task.
In some embodiments, the above method further comprises: receiving a disk utilization rate parameter setting request sent by a second user, wherein the disk utilization rate parameter setting request comprises user information and a parameter value of the second user; determining whether the second user has a parameter setting authority or not according to the user information of the second user; in response to determining that the second user has a parameter setting authority, determining whether the parameter value is a valid value; and taking the parameter value as the value of the disk utilization rate parameter in response to the fact that the parameter value is determined to be the effective value.
In a second aspect, an embodiment of the present disclosure provides an apparatus for caching data, where the apparatus includes: the generating unit is configured to generate a data operation task according to a data operation request sent by a first user; the determining unit is configured to respond to the determination that memory data overflows to a disk serving as a temporary space in the process of executing the data operation task, and determine whether the utilization rate of the disk is limited or not according to a preset value of a disk utilization rate parameter; an execution unit configured to, in response to determining that the disk presence usage limit and the value of the disk space full flag of the disk are false, execute the following operation steps: determining whether the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data or not; if the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data, determining whether the predicted utilization rate of the disk is larger than the value of the disk utilization rate parameter, wherein the predicted utilization rate of the disk is determined according to the data volume of the overflowed memory data, the disk used space of the disk and the total disk space; and in response to determining that the predicted utilization rate of the disk is not greater than the value of the disk utilization rate parameter, caching the overflowed memory data to the disk.
In some embodiments, the above operation further comprises: and in response to determining that the predicted utilization rate of the disk is greater than the value of the disk utilization rate parameter, executing a task rollback operation for the data operation task, and setting the value of the disk space full flag of the disk to true.
In some embodiments, the above operation further comprises: and if the capacity of the residual space of the disk is not larger than the data volume of the overflowed memory data, executing task rollback operation aiming at the data operation task.
In some embodiments, the above apparatus further comprises: a rollback operation execution unit configured to execute a task rollback operation for the data operation task in response to determining that the disk presence usage limit and a value of a disk space full flag of the disk are true.
In some embodiments, the above apparatus further comprises: and the sending unit is configured to respond to the task rollback operation executed for the data operation task and send error prompt information to the terminal used by the first user.
In some embodiments, the above apparatus further comprises: the receiving unit is configured to receive a disk utilization rate parameter setting request sent by a second user, wherein the disk utilization rate parameter setting request comprises user information and a parameter value of the second user; an authority determining unit configured to determine whether the second user has a parameter setting authority according to user information of the second user; a valid value determining unit configured to determine whether the parameter value is a valid value in response to a determination that the second user has a parameter setting authority; and the assignment unit is configured to take the parameter value as the value of the disk utilization rate parameter in response to the fact that the parameter value is determined to be the effective value.
In a third aspect, an embodiment of the present disclosure provides a server, where the server includes: one or more processors; a storage device, on which one or more programs are stored, which, when executed by the one or more processors, cause the one or more processors to implement the method as described in any implementation manner of the first aspect.
In a fourth aspect, the disclosed embodiments provide a computer-readable medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the method as described in any implementation manner of the first aspect.
According to the method and the device for caching data, a data operation task is generated according to a data operation request sent by a first user, and if it is determined that memory data overflows to a disk serving as a temporary space in the process of executing the data operation task, whether the utilization rate limit exists in the disk is determined according to a preset value of a disk utilization rate parameter. If the utilization rate limit exists in the disk and the value of the disk space full flag of the disk is false, executing the following operation steps: first, it is determined whether the capacity of the remaining space of the disk is larger than the data amount of the overflowed memory data. And if the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data, further determining whether the predicted utilization rate of the disk is larger than the value of the disk utilization rate parameter. If the data is not larger than the preset threshold value, the overflowed memory data is cached to the disk, so that the disk utilization rate of the disk cannot be larger than the value of the disk utilization rate parameter after the memory data overflowed by the disk cache is ensured by setting the value of the disk utilization rate parameter, the problem that the space of the disk is insufficient due to the overflow of the memory data, and then the database is crashed is solved, and the stability of the database is improved.
Drawings
Other features, objects and advantages of the disclosure will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present disclosure may be applied;
FIG. 2 is a flow diagram for one embodiment of a method for caching data, according to the present disclosure;
FIG. 3 is a schematic diagram of one application scenario of a method for caching data according to the present disclosure;
FIG. 4 is a flow diagram of yet another embodiment of a method for caching data according to the present disclosure;
FIG. 5 is a schematic diagram illustrating one embodiment of an apparatus for caching data according to the present disclosure;
FIG. 6 is a schematic block diagram of a computer system suitable for use with a server embodying embodiments of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the figures and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that, in the present disclosure, the embodiments and features of the embodiments may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates an exemplary system architecture 100 to which a method for caching data or an apparatus for caching data of embodiments of the present disclosure may be applied.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have various communication client applications installed thereon, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.
The terminal apparatuses 101, 102, and 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices having a display screen and inputting information, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like. When the terminal apparatuses 101, 102, 103 are software, they can be installed in the electronic apparatuses listed above. It may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.
The server 105 may be a server that provides various services, such as a database server. The database server may receive a data operation request from a user, execute a corresponding data operation task according to the data operation request, and feed back an execution result to the terminal devices 101, 102, and 103. A disk may be included in the server 105.
The server 105 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster composed of a plurality of servers, or may be implemented as a single server. When the server 105 is software, it may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
It should be noted that the method for caching data provided by the embodiment of the present disclosure is generally performed by the server 105, and accordingly, the apparatus for caching data is generally disposed in the server 105.
With continued reference to FIG. 2, a flow 200 of one embodiment of a method for caching data in accordance with the present disclosure is shown. The method for caching data comprises the following steps:
In this embodiment, an execution subject (for example, the server 105 shown in fig. 1) of the method for caching data may receive a data operation request from a terminal (for example, the terminal devices 101, 102, 103 shown in fig. 1) with which a first user performs information input, through a wired connection manner or a wireless connection manner. By way of example, the data operation request may include a data addition request, a data query request, a data update request, and the like. The execution subject may generate a data operation task according to the received data operation request, where the data operation task is used to respond to the data operation request. Taking the data operation request as an example, the data operation task may query data from the database and return a query result.
In this embodiment, during the execution of the data operation task, if operations with large data size such as aggregation (aggregation), table association (join), sorting (sort) and the like are involved, the memory may be insufficient, and at this time, the memory data may overflow to the disk as the temporary space. Here, the temporary space may refer to a space for buffering data when the memory is insufficient. If it is determined that the memory data overflows to the disk serving as the temporary space in the process of executing the data operation task, the execution main body can judge whether the utilization rate of the disk is limited according to a preset value of a disk utilization rate parameter. Here, a disk usage rate parameter may be preset inside the execution main body, and a value range of the disk usage rate parameter is as follows: 0 to 100. When the disk utilization rate parameter is 0 or 100, the disk does not have utilization rate limitation; when the disk usage parameter is larger than 0 and smaller than 100, it indicates that there is a usage limit for the disk. Therefore, when the memory data overflows to the disk in the process of executing the data operation task, the execution main body can judge whether the disk has the utilization rate limit or not according to the value of the disk utilization rate parameter.
Here, if it is determined that there is no usage limit for the disk, the execution main may directly write the overflowed memory data to the disk.
In response to determining that the disk usage limit exists and the value of the disk space full flag of the disk is false, step 203, performs the following operations 2031 to 2033.
In this embodiment, a disk space full flag may be set for the disk, and the disk space full flag may be used to indicate whether the disk space is full. Specifically, the disk space full flag may be set to true (true) or false (false), and when the disk space full flag is set to true, it indicates that the disk space is full; when the disk space full flag is set to false, it indicates that the disk space is not full. If it is determined that there is a usage limit on the disk and the disk space full flag of the disk is false, the execution agent may perform the following steps 2031 to 2033.
In this embodiment, the execution main body may determine whether the capacity of the remaining space of the disk is larger than the data amount of the overflowed memory data.
In this embodiment, if it is determined that the capacity of the remaining space of the disk is greater than the data amount of the overflowed memory data, the execution main body may further determine whether the predicted utilization rate of the disk is greater than the value of the disk utilization rate parameter. The predicted utilization rate of the disk is determined according to the data volume of the overflowed memory data, the used disk space of the disk and the total disk space. As an example, the execution subject may be represented by the formula: the predicted usage rate of the disk = (disk used space + amount of data of the overflowed memory data)/total disk space, and the predicted usage rate of the disk is calculated. Here, the predicted usage rate of the disk may refer to a usage rate expected to be achieved by the disk after adding the overflowed memory data to be cached.
In this embodiment, if the predicted usage rate of the disk is not greater than the value of the disk usage parameter, it indicates that even if the overflowed memory data is cached to the disk, the expected usage rate of the disk does not exceed the value of the disk usage parameter. At this time, the execution subject may cache the overflowed memory data to disk.
In some optional implementation manners of this embodiment, the foregoing operation steps may further include: and if the capacity of the residual space of the disk is not larger than the data volume of the overflowed memory data, executing task rollback operation aiming at the data operation task.
In this implementation, if the capacity of the remaining space of the disk is not greater than the data amount of the overflowed memory data, the execution main body may execute the task rollback operation with respect to the data operation task.
In some optional implementation manners of this embodiment, the method for caching data may further include: in response to determining that the disk has a usage limit and that the value of the disk space full flag of the disk is true, a task rollback operation is performed for the data manipulation task.
In this implementation, if it is determined that the usage limit exists in the disk and the value of the disk space full flag of the disk is true, the execution subject may execute a task rollback operation for the data operation task.
In some optional implementation manners of this embodiment, the method for caching data may further include: and sending error prompt information to the first user in response to executing the task rollback operation on the data operation task.
In this implementation, if the task rollback operation is executed for the data operation task, the execution main body may send an error prompt message to the terminal used by the first user to prompt the first user that the usage of the disk space is over-limit.
In some optional implementation manners of this embodiment, the method for caching data may further include the following steps:
firstly, a disk usage rate parameter setting request sent by a second user is received.
In this implementation manner, the execution main body may receive a disk usage rate parameter setting request sent by a second user through a used terminal, where the disk usage rate parameter setting request may include user information and a parameter value of the second user. Here, the user information of the second user may include, but is not limited to, a user name, a password, and the like.
Secondly, determining whether the second user has the parameter setting authority or not according to the user information of the second user.
In this implementation manner, the execution body may store user information of a manager in advance, where the manager may refer to a user having a disk usage parameter setting authority. In this way, the execution principal may match the user information of the second user with the pre-stored user information of the administrator, and if the user information of the second user matches (e.g., is the same as) the pre-stored user information of the administrator, the execution principal may determine that the second user has the parameter setting authority.
Then, in response to determining that the second user has the parameter setting authority, it is determined whether the parameter value is a valid value.
In this implementation, if it is determined that the second user has the parameter setting authority, the execution main body may further determine whether a parameter value in the disk usage parameter setting request is a valid value. Here, the range of values of the disk usage rate parameter is: 0 to 100. If the parameter value in the disk utilization rate parameter setting request meets the condition: if the parameter value is greater than or equal to 0 and less than or equal to 100, the parameter value can be determined to be an effective value.
And finally, in response to the fact that the parameter value is determined to be an effective value, taking the parameter value as a value of the disk utilization rate parameter.
In this implementation manner, if it is determined that the parameter value is an effective value, the execution main body may use the parameter value in the disk usage parameter setting request as the value of the disk usage parameter.
With continued reference to fig. 3, fig. 3 is a schematic diagram of an application scenario of the method for caching data according to the present embodiment. In the application scenario of fig. 3, a first user sends a data operation request to the database server 302 through the terminal device 301, and the database server 302 generates a data operation task according to the data operation request. If the memory data overflows to a disk (not shown in the figure) as a temporary space in the process of executing the data operation task, the database server 302 may determine whether the disk has a usage limit according to a preset value of 95% of a disk usage parameter. In response to determining that there is a usage limit for the disk and that the value of the disk space full flag for the disk is false, database server 302 may perform the following: first, it is determined whether the capacity of the remaining space of the disk is larger than the data amount of the overflowed memory data. And if the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data, further determining whether the predicted utilization rate of the disk is larger than the value of the disk utilization rate parameter by 95%. And in response to determining that the predicted utilization rate of the disk is not greater than 95% of the value of the disk utilization rate parameter, caching the overflowed memory data to the disk.
According to the method provided by the embodiment of the disclosure, by setting the value of the disk utilization rate parameter, after the memory data overflowing from the disk cache is ensured, the disk utilization rate of the disk is not greater than the value of the disk utilization rate parameter, so that the problem that the database is down due to insufficient disk space caused by the overflow of the memory data is avoided, and the stability of the database is improved.
With further reference to FIG. 4, a flow 400 of yet another embodiment of a method for caching data is illustrated. The process 400 of the method for caching data includes the following steps:
In this embodiment, step 401 is similar to step 201 of the embodiment shown in fig. 2, and is not described here again.
In this embodiment, step 402 is similar to step 202 of the embodiment shown in fig. 2, and is not described herein again.
In response to determining that the usage limit exists for the disk and that the value of the disk space full flag of the disk is false, in step 403, the following operation 4031-4034 is performed.
In this embodiment, if it is determined that the disk has a usage limitation and the value of the disk space full flag of the disk is false, the execution subject may perform the following operation steps 4031 to 4034.
In this embodiment, step 4031 is similar to step 2031 in the embodiment shown in fig. 2, and is not described here again.
In this embodiment, step 4032 is similar to step 2032 in the embodiment shown in fig. 2, and is not repeated herein.
In this embodiment, step 4033 is similar to step 2033 of the embodiment shown in fig. 2, and is not described here again.
In this implementation manner, the predicted utilization of the disk is greater than the value of the disk utilization parameter, which indicates that if the overflowed memory data is cached to the disk, the expected utilization of the disk is greater than the value of the disk utilization parameter, which may cause a serious shortage of disk space. Thus, the execution principal may perform a task rollback operation for the data manipulation task. At the same time, the execution principal may set the value of the disk space full flag of the disk to true.
As can be seen from fig. 4, compared with the embodiment corresponding to fig. 2, the process 400 of the method for caching data in this embodiment highlights steps of performing a task rollback operation on a data operation task and setting a value of a disk space full flag of a disk to true when the predicted usage rate of the disk is greater than a value of a disk usage rate parameter. Therefore, the scheme described in the embodiment can avoid the problem that the database is down due to insufficient disk space caused by overflow of the memory data through the task rollback operation, so that the stability of the database is improved. Meanwhile, the value of the disk space full flag of the disk is set to be true, so that the time waste caused by another caching attempt when the data operation task is executed next time can be avoided.
With further reference to fig. 5, as an implementation of the methods shown in the above-mentioned figures, the present disclosure provides an embodiment of an apparatus for caching data, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be specifically applied to various electronic devices.
As shown in fig. 5, the apparatus 500 for caching data of the present embodiment includes: a generating unit 501, a determining unit 502 and an executing unit 503. The generating unit 501 is configured to generate a data operation task according to a data operation request sent by a first user; the determining unit 502 is configured to determine whether a usage limit exists in a disk according to a preset value of a disk usage parameter in response to determining that memory data overflows to the disk as a temporary space in the process of executing the data operation task; the execution unit 503 is configured to, in response to determining that the disk presence usage limit and the value of the disk space full flag of the disk are false, perform the following operation steps: determining whether the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data or not; if the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data, determining whether the predicted utilization rate of the disk is larger than the value of the disk utilization rate parameter, wherein the predicted utilization rate of the disk is determined according to the data volume of the overflowed memory data, the disk used space of the disk and the total disk space; and in response to determining that the predicted utilization rate of the disk is not greater than the value of the disk utilization rate parameter, caching the overflowed memory data to the disk.
In this embodiment, for specific processing of the generating unit 501, the determining unit 502, and the executing unit 503 of the apparatus 500 for caching data and technical effects brought by the specific processing, reference may be made to relevant descriptions of step 201, step 202, and step 203 in the corresponding embodiment of fig. 2, and details are not repeated here.
In some optional implementations of this embodiment, the foregoing operation step further includes: and in response to determining that the predicted utilization rate of the disk is greater than the value of the disk utilization rate parameter, executing a task rollback operation for the data operation task, and setting the value of the disk space full flag of the disk to true.
In some optional implementations of this embodiment, the foregoing operation step further includes: and if the capacity of the residual space of the disk is not larger than the data volume of the overflowed memory data, executing task rollback operation aiming at the data operation task.
In some optional implementations of this embodiment, the apparatus 500 further includes: a rollback operation execution unit (not shown) configured to execute a task rollback operation on the data operation task in response to determining that the disk usage limit exists and that a value of a disk space full flag of the disk is true.
In some optional implementations of this embodiment, the apparatus 500 further includes: and a sending unit (not shown in the figure) configured to send error prompt information to the terminal used by the first user in response to the task rollback operation being performed for the data operation task.
In some optional implementations of this embodiment, the apparatus 500 further includes: a receiving unit (not shown in the figure) configured to receive a disk usage rate parameter setting request sent by a second user, wherein the disk usage rate parameter setting request includes user information and a parameter value of the second user; an authority determination unit (not shown in the figure) configured to determine whether the second user has a parameter setting authority according to the user information of the second user; a valid value determining unit (not shown in the figure) configured to determine whether the parameter value is a valid value in response to a determination that the second user has a parameter setting authority; and an assignment unit (not shown in the figure) configured to take the parameter value as the value of the disk usage parameter in response to determining that the parameter value is a valid value.
Referring now to FIG. 6, a schematic diagram of an electronic device (e.g., the server of FIG. 1) 600 suitable for use in implementing embodiments of the present disclosure is shown. The server shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 6, the electronic device 600 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, or the like; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may be alternatively implemented or provided. Each block shown in fig. 6 may represent one device or may represent multiple devices as desired.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 609, or may be installed from the storage means 608, or may be installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of embodiments of the present disclosure.
It should be noted that the computer readable medium described in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In embodiments of the present disclosure, however, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: generating a data operation task according to a data operation request sent by a first user; in response to the fact that memory data overflows to a disk serving as a temporary space in the process of executing the data operation task, determining whether the disk has a utilization rate limit according to a preset value of a disk utilization rate parameter; in response to determining that the disk availability constraint exists and the value of the disk space full flag of the disk is false, executing the following operation steps: determining whether the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data; if the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data, determining whether the predicted utilization rate of the disk is larger than the value of the disk utilization rate parameter, wherein the predicted utilization rate of the disk is determined according to the data volume of the overflowed memory data, the disk used space of the disk and the total disk space; and in response to the fact that the predicted utilization rate of the disk is not larger than the value of the disk utilization rate parameter, caching the overflowed memory data to the disk.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a generation unit, a determination unit, and an execution unit. The names of the units do not form a limitation on the units themselves in some cases, and for example, the generating unit may also be described as a "unit that generates a data operation task according to a data operation request sent by a first user".
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combinations of the above-mentioned features, and other embodiments in which the above-mentioned features or their equivalents are combined arbitrarily without departing from the spirit of the invention are also encompassed. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (10)
1. A method for caching data, comprising:
generating a data operation task according to a data operation request sent by a first user;
in response to determining that the part of the memory data exceeding the maximum capacity of the memory overflows to a disk serving as a temporary space in the process of executing the data operation task, determining whether the disk has a utilization limit according to a preset value of a disk utilization parameter, wherein the data operation task comprises at least one of aggregation, table association and sorting operation;
in response to determining that the disk usage limit exists and the value of the disk space full flag of the disk is false, performing the following operational steps: determining whether the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data; if the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data, determining whether the predicted utilization rate of the disk is larger than the value of the disk utilization rate parameter, wherein the predicted utilization rate of the disk is determined according to the data volume of the overflowed memory data, the used disk space of the disk and the total disk space; in response to determining that the predicted utilization of the disk is not greater than the value of the disk utilization parameter, caching the overflowed memory data to the disk;
in response to determining that the predicted utilization of the disk is greater than the value of the disk utilization parameter, performing a task rollback operation on the data operation task, and setting a value of a disk space full flag of the disk to true;
receiving a disk utilization rate parameter setting request sent by a second user, wherein the disk utilization rate parameter setting request comprises user information and parameter values of the second user; determining whether the second user has a parameter setting authority or not according to the user information of the second user; in response to determining that the second user has parameter setting permissions, determining whether the parameter value is a valid value; and taking the parameter value as the value of the disk utilization rate parameter in response to the fact that the parameter value is determined to be the effective value.
2. The method of claim 1, wherein the operating step further comprises:
and if the capacity of the residual space of the disk is not larger than the data volume of the overflowed memory data, executing task rollback operation aiming at the data operation task.
3. The method of claim 1, wherein the method further comprises:
and executing a task rollback operation for the data operation task in response to determining that the utilization limit exists for the disk and that the value of the disk space full flag of the disk is true.
4. The method according to any one of claims 2-3, wherein the method further comprises:
and responding to the task rollback operation executed aiming at the data operation task, and sending error prompt information to the terminal used by the first user.
5. An apparatus for caching data, comprising:
the generating unit is configured to generate a data operation task according to a data operation request sent by a first user;
the determining unit is configured to determine whether a utilization limit exists in a disk according to a preset value of a disk utilization parameter in response to determining that part of memory data exceeding the maximum memory capacity overflows to the disk serving as a temporary space in the process of executing the data operation task, wherein the data operation task comprises at least one of aggregation, table association and sorting operation;
an execution unit configured to, in response to determining that the disk presence usage limit and a value of a disk space full flag of the disk are false, perform the following operational steps: determining whether the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data; if the capacity of the residual space of the disk is larger than the data volume of the overflowed memory data, determining whether the predicted utilization rate of the disk is larger than the value of the disk utilization rate parameter, wherein the predicted utilization rate of the disk is determined according to the data volume of the overflowed memory data, the used disk space of the disk and the total disk space; in response to determining that the predicted utilization of the disk is not greater than the value of the disk utilization parameter, caching the overflowed memory data to the disk;
the operating steps further include: in response to determining that the predicted utilization of the disk is greater than the value of the disk utilization parameter, performing a task rollback operation on the data operation task, and setting a value of a disk space full flag of the disk to true;
the receiving unit is configured to receive a disk usage rate parameter setting request sent by a second user, wherein the disk usage rate parameter setting request comprises user information and a parameter value of the second user;
an authority determination unit configured to determine whether the second user has a parameter setting authority according to user information of the second user;
a valid value determining unit configured to determine whether the parameter value is a valid value in response to determining that the second user has a parameter setting authority;
and the assignment unit is configured to respond to the determination that the parameter value is a valid value and take the parameter value as the value of the disk utilization rate parameter.
6. The apparatus of claim 5, wherein the operating step further comprises:
and if the capacity of the residual space of the disk is not larger than the data volume of the overflowed memory data, executing task rollback operation aiming at the data operation task.
7. The apparatus of claim 5, wherein the apparatus further comprises:
a rollback operation execution unit configured to perform a task rollback operation for the data operation task in response to determining that the disk presence usage limit and a value of a disk space full flag of the disk are true.
8. The apparatus of any of claims 5-7, wherein the apparatus further comprises:
a sending unit configured to send error prompt information to a terminal used by the first user in response to performing a task rollback operation for the data operation task.
9. A server, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-4.
10. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910547977.8A CN110262756B (en) | 2019-06-24 | 2019-06-24 | Method and device for caching data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910547977.8A CN110262756B (en) | 2019-06-24 | 2019-06-24 | Method and device for caching data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110262756A CN110262756A (en) | 2019-09-20 |
| CN110262756B true CN110262756B (en) | 2023-03-28 |
Family
ID=67920714
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910547977.8A Active CN110262756B (en) | 2019-06-24 | 2019-06-24 | Method and device for caching data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110262756B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105868025A (en) * | 2016-03-30 | 2016-08-17 | 华中科技大学 | System for settling fierce competition of memory resources in big data processing system |
| CN107155081A (en) * | 2017-05-15 | 2017-09-12 | 青岛海信电器股份有限公司 | Method of program recording and device |
| CN107992432A (en) * | 2017-11-28 | 2018-05-04 | 福建中金在线信息科技有限公司 | The method and terminal device of a kind of data buffer storage |
| CN109831508A (en) * | 2019-02-15 | 2019-05-31 | 网宿科技股份有限公司 | A kind of caching method and equipment, storage medium |
| CN109916022A (en) * | 2017-12-13 | 2019-06-21 | 广东美的制冷设备有限公司 | Control method, device and the computer readable storage medium of air regulator |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10572462B2 (en) * | 2016-04-10 | 2020-02-25 | International Business Machines Corporation | Efficient handling of sort payload in a column organized relational database |
-
2019
- 2019-06-24 CN CN201910547977.8A patent/CN110262756B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105868025A (en) * | 2016-03-30 | 2016-08-17 | 华中科技大学 | System for settling fierce competition of memory resources in big data processing system |
| CN107155081A (en) * | 2017-05-15 | 2017-09-12 | 青岛海信电器股份有限公司 | Method of program recording and device |
| CN107992432A (en) * | 2017-11-28 | 2018-05-04 | 福建中金在线信息科技有限公司 | The method and terminal device of a kind of data buffer storage |
| CN109916022A (en) * | 2017-12-13 | 2019-06-21 | 广东美的制冷设备有限公司 | Control method, device and the computer readable storage medium of air regulator |
| CN109831508A (en) * | 2019-02-15 | 2019-05-31 | 网宿科技股份有限公司 | A kind of caching method and equipment, storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 广播机制解决Shuffle过程数据倾斜的方法;吴恩慈;《计算机系统应用》;20190615(第06期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110262756A (en) | 2019-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111427701A (en) | Workflow engine system and business processing method | |
| CN108011949B (en) | Method and apparatus for acquiring data | |
| CN111460129A (en) | Method and device for generating identification, electronic equipment and storage medium | |
| CN110795315A (en) | Method and device for monitoring service | |
| CN112416632B (en) | Event communication method and device, electronic equipment and computer readable medium | |
| CN111338834B (en) | Data storage method and device | |
| CN111126948A (en) | Processing method and device for approval process | |
| CN113282589A (en) | Data acquisition method and device | |
| CN112948138A (en) | Method and device for processing message | |
| CN110262756B (en) | Method and device for caching data | |
| CN111460020B (en) | Method, device, electronic equipment and medium for resolving message | |
| CN113946816A (en) | Cloud service-based authentication method and device, electronic equipment and storage medium | |
| CN109087097B (en) | Method and device for updating same identifier of chain code | |
| CN113283891A (en) | Information processing method and device and electronic equipment | |
| CN113726885A (en) | Method and device for adjusting flow quota | |
| CN113282455A (en) | Monitoring processing method and device | |
| CN112463616A (en) | Chaos testing method and device for Kubernetes container platform | |
| CN109213815B (en) | Method, device, server terminal and readable medium for controlling execution times | |
| CN113760927A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| CN112559001A (en) | Method and device for updating application | |
| CN115484149B (en) | Network switching method, network switching device, electronic equipment and storage medium | |
| CN113568936B (en) | Real-time stream data storage method, device and terminal equipment | |
| CN110633324B (en) | Method, apparatus, electronic device and computer readable medium for synchronizing data | |
| CN113722316A (en) | Model processing method and device, electronic equipment and computer readable medium | |
| CN114064314A (en) | Service message processing method, system, electronic device and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |