CN104965882B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN104965882B CN104965882B CN201510325610.3A CN201510325610A CN104965882B CN 104965882 B CN104965882 B CN 104965882B CN 201510325610 A CN201510325610 A CN 201510325610A CN 104965882 B CN104965882 B CN 104965882B
- Authority
- CN
- China
- Prior art keywords
- data
- tables
- data line
- batch
- export
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure discloses a kind of data processing method and device, belongs to database field.This method includes：Pending data line is determined in tables of data；N batches are divided to export pending data line；For data line derived from every batch of, deleted from tables of data；Alternatively, after filing to another tables of data, deleted from tables of data.The disclosure exports operation by using a small amount of data of multiple batches of, every batch of, derived data line is deleted, solve the problems, such as that longer data time of deleting can influence the availability of database in the related technology, reach when deleting or filing large-scale data, tables of data will not be locked on a large scale, unit interval pressure caused by tables of data is smaller, does not interfere with the performance of tables of data, to improve tables of data availability effect.
Description
Technical field
This disclosure relates to database field, more particularly to a kind of data processing method and device.
Background technology
During the use of database, it is frequently encountered to delete in the tables of data for needing to magnify capacity from certain and specifies data
Operative scenario.Specified data can no longer be needed data to be used or need to achieve to the data of other tables of data.
In the data processing method that the relevant technologies provide, need to lock relevant data when deleting data, but
It is since the data for needing to delete are usually more, for example, deleting 100,000 row data, it is necessary to lock 100,000 row data.Longer
Deletion data during, the I/O of database can be caused to rise, performance decline, to influence the availability of database.
Invention content
In order to overcome the problems, such as present in the relevant technologies, a kind of data processing method of disclosure offer and device.The technology
Scheme is as follows：
According to the first aspect of the embodiments of the present disclosure, a kind of data processing method is provided, this method includes：
Pending data line is determined in tables of data；
N batches are divided to export the pending data line；
For data line derived from every batch of, deleted from the tables of data；Alternatively, after filing to another tables of data, from this
It is deleted in tables of data.
In a possible embodiment, which is divided n batches to export by this, including：
It obtains this batch and needs derived data line quantity, and obtain the dormancy time after this batch export data line；
The data line is exported from the tables of data according to the data line quantity；
If there is also the pending data line after export process of this batch, according to the dormancy time into
After row suspend mode scheduled duration, into the export process of next batch；
It mutually holds in the mouth the end address of the initial address of data line derived from the next batch and data line derived from this batch
It connects.
In a possible embodiment, this method further includes：
After the export process of this batch, detect whether the connection number in the tables of data is more than preset value；Connection
Number is the current connection quantity of database where operation system accesses tables of data；
If being more than the preset value, dormancy time is extended into preset duration, or, dormancy time is extended into the first duration, the
One duration with to connect number more than preset value be more than degree correlation.
In a possible embodiment, this method further includes：
After the export process of this batch, the leader follower replication of the tables of data is detected with the presence or absence of delay；
If there is delay, dormancy time is extended into preset duration, or, dormancy time is extended the second duration, when second
The long delay size correlation with delay.
In a possible embodiment, pending data line should be determined in tables of data, including：
The data line that key assignments meets treatment conditions, the data line pending as this are calculated from the tables of data.
According to the second aspect of the embodiment of the present disclosure, a kind of data processing equipment is provided, which includes：
Determining module is configured as determining pending data line in tables of data；
Export module is configured as dividing n batches to export the pending data line；
Removing module is configured as, for data line derived from every batch of, deleting from the tables of data；Alternatively, filing is extremely
After another tables of data, deleted from the tables of data.
In a possible embodiment, the export module, including：
Acquisition submodule is configured as obtaining the derived data line quantity of this batch needs, and obtains this batch export number
According to the dormancy time after row；
Submodule is exported, is configured as exporting the data line from the tables of data according to the data line quantity；
Suspend mode submodule, if being configured as after the export process of this batch, there is also the pending data
Row, then after carrying out suspend mode scheduled duration according to the dormancy time, into the export process of next batch；
It mutually holds in the mouth the end address of the initial address of data line derived from the next batch and data line derived from this batch
It connects.
In a possible embodiment, the export module further includes：
First detection sub-module is configured as after the export process of this batch, detecting the company in the tables of data
Connect whether number is more than preset value；Connection number is the current connection of database where operation system accesses tables of data or accesses tables of data
Quantity；
First delay submodule, is configured as when more than preset value, dormancy time is extended preset duration, or, will stop
The first duration of dormancy time lengthening, the first duration with to connect number more than preset value be more than degree correlation.
In a possible embodiment, which further includes：
Second detection sub-module, the principal and subordinate for being configured as detecting the tables of data after export process of this batch are multiple
System is with the presence or absence of delay；
Be delayed submodule, is additionally configured to, when there is delay, dormancy time be extended preset duration, or, when by suspend mode
Between extend the second duration, the delay size correlation of the second duration and delay.
In a possible embodiment, the determining module, is additionally configured to calculate key assignments from the tables of data and meets
The data line for the treatment of conditions, the data line pending as this.
According to the third aspect of the embodiment of the present disclosure, a kind of data processing equipment is provided, which is characterized in that the device packet
It includes：
Processor；
Memory for storing the processor-executable instruction；
Wherein, which is configured as：
Pending data line is determined in tables of data；
N batches are divided to export the pending data line；
For data line derived from every batch of, deleted from the tables of data；Alternatively, after filing to another tables of data, from this
It is deleted in tables of data.
The technical scheme provided by this disclosed embodiment can include the following benefits：
Operation is exported by using a small amount of data of multiple batches of, every batch of, derived data line is deleted, is solved
The problem of longer deletion data time can influence the availability of database in the related technology has reached and has deleted or filing greatly
When the data of scale, tables of data will not be locked on a large scale, unit interval pressure caused by tables of data is smaller, Bu Huiying
Ring tables of data performance, to improve tables of data availability effect.
It should be understood that above general description and following detailed description is merely exemplary, this can not be limited
It is open.
Description of the drawings
The drawings herein are incorporated into the specification and forms part of this specification, and shows the implementation for meeting the disclosure
Example, and consistent with the instructions for explaining the principles of this disclosure.
Fig. 1 is a kind of flow chart of data processing method shown according to an exemplary embodiment；
Fig. 2 is a kind of flow chart of the data processing method shown according to another exemplary embodiment；
Fig. 3 is a kind of flow chart of data processing method shown according to another exemplary embodiment；
Fig. 4 is the block diagram according to the data processing equipment shown in an exemplary embodiment；
Fig. 5 is the block diagram of the data processing equipment shown according to another exemplary embodiment；
Fig. 6 is a kind of block diagram of the data processing equipment shown according to another exemplary embodiment.
Specific implementation mode
Example embodiments are described in detail here, and the example is illustrated in the accompanying drawings.Following description is related to
When attached drawing, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements.Following exemplary embodiment
Described in embodiment do not represent all implementations consistent with this disclosure.On the contrary, they be only with it is such as appended
The example of the consistent device and method of some aspects be described in detail in claims, the disclosure.
It is shown in Figure 1, it is a kind of flow chart of data processing method shown according to an exemplary embodiment, packet
Include following steps.
In a step 101, pending data line is determined in tables of data.
In a step 102, n batches are divided to export pending data line.
In step 103, it for data line derived from every batch of, is deleted from tables of data；Alternatively, filing to another data
After table, deleted from tables of data.
In conclusion the data processing method provided in the embodiment of the present disclosure, a small amount of by using multiple batches of, every batch of
Data export operation, derived data line is deleted, solving longer deletion data time in the related technology can influence
The problem of availability of database, has reached when deleting or filing large-scale data, will not greatly be advised to tables of data
Mode locking, unit interval pressure caused by tables of data is smaller, does not interfere with the performance of tables of data, to improve tables of data can
With the effect of property.
Two embodiments are used to be illustrated respectively to deleting data and filing data below.
It is shown in Figure 2, it is a kind of flow chart of the data processing method shown according to another exemplary embodiment,
Include the following steps.
In step 201, the data line that key assignments meets treatment conditions is calculated from tables of data, as pending data
Row.
Key assignments is the value of a certain item data item of data line in tables of data.
When needing to handle the data in tables of data, server can calculate pending according to treatment conditions
Data line.
For example, when needing to delete the data line before April 3 as hash, treatment conditions are " date data item
Key assignments be April 3 before data line ",
For another example, treatment conditions be " data line that the value after 2 modulus of major key pair is 1 ", then for table 1 shown in data
Table, since the value after 1,3,5,7 pair of 2 modulus is 1, server is according to the calculated pending data line for the treatment of conditions point
It Wei not the first row data line, the third line data line, fifth line data line, the 7th row data line.
| Major key | Date | User | Unit price | Quantity |
| 1 | April 1 | Xiao Wang | 10 | 30 |
| 2 | April 1 | It is small red | 20 | 5 |
| 3 | April 2 | Xiao Wang | 10 | 10 |
| 4 | April 2 | Xiao Li | 10 | 20 |
| 5 | April 3 | Xiao Zhang | 20 | 10 |
| 6 | April 3 | Xiao Li | 30 | 10 |
| 7 | April 3 | It is small red | 30 | 20 |
| 8 | April 4 | Xiao Zhang | 30 | 30 |
Table 1
In step 202, it obtains this batch and needs derived data line quantity.
During daily database use, it is usually hash that data line, which needs the derived scene from tables of data,
Cleaning or data filing.
Wherein, stale data cleaning is that the very low data line of expired in tables of data or frequency of use is deleted；Data
Filing is then that the very low data line of expired or frequency of use is transferred to another Zhang Buhui externally to provide access from tables of data
File in tables of data.The filing tables of data can be the slave table of current data table, which can be with current database
Belong to same database, disparate databases can also be belonged to.
Server needs derived data line quantity usually very when needing to export the data line in tables of data
Greatly, these data lines are exported and need the longer data export time.In longer data export the time, the I/O meetings of tables of data
Rise, operational data-handling capacity is declined so as to cause tables of data.
In the data processing method that the present embodiment proposes, server can divide pending data line to n batch, lead to
It crosses and is exported a large amount of data line from tables of data using a small amount of data export operation of multiple batches of, every batch of, to reduce clothes
The data volume that business device single is written and read tables of data.
Optionally, which is fixed value, such as：100 rows, 200 rows.
For example, server is by 50000 pending data rows point, 500 export, then server calculates this batch needs
Derived data line quantity is 100.
Optionally, which voluntarily can also dynamically set according to the current data processing frequency of tables of data.
If the current data processing frequency of tables of data is more frequent, this batch needs derived data line quantity to reduce；If tables of data
Current data processing frequency it is relatively low when, this batch needs derived data line quantity to increase.
In step 203, the dormancy time after this batch export data line is obtained.
Optionally, which is fixed value, for example, setting dormancy time to 5s.
It should be noted that the execution sequencing of step 202 and step 203 is unlimited, only with step 202 in the present embodiment
It is executed before step 203 to illustrate.Step 202 can also be performed simultaneously with step 203, alternatively, step 203 is in step
It is executed before 202.
In step 204, data line is exported from tables of data according to data line quantity.
In one possible implementation, server is according to the key assignments size of pending data line, from tables of data
Export the corresponding data line of data line quantity.
For example, pending data line is respectively " the 1st row data line to the 50000th row data line ", this batch needs are led
The data line quantity gone out is 100 rows, then server selects the 1st row data line to the 100th line number from small to large according to key assignments size
It is exported from tables of data according to row.
In step 205, if there is also pending data lines after the export process of this batch, according to suspend mode
After time carries out suspend mode scheduled duration, into the export process of next batch.
Server is after the export process for completing this batch, the end address of minute book batch derived data line.
Wherein, due to the end address of data line derived from the initial address of data line derived from next batch and this batch
Mutually it is connected, server can determine that next batch needs derived data line according to the end address of the data line of record.
It should be noted that the initial address and end address are defined relative to data line to be treated, and
It is not limited to that major key is continuous or storage address is continuous.
For example, the end address of pending data row derived from this batch is the data line that key assignments is 100, not derived
In pending data row, the address for the data line that address and the key assignments of data line that key assignments is 101 are 100 is mutually connected, then services
Device determines that the initial address of data line derived from next batch is the data line that key assignments is 101.
In step 206, after the export process of this batch, whether the connection number of detection data table is more than default
Value.
The connection number of tables of data is the current connection quantity of database where operation system accesses tables of data.In tables of data
Connection number reflects the task amount that current data table receives data processing operation, when the connection number of tables of data is larger, data
The ability that table can currently bear other data processing operations is smaller, if at this point, server continues to lead tables of data progress data line
Go out operation, it is easy to impact to the business that tables of data normally provides.
Optionally, in the embodiments of the present disclosure, server, can be with detection data table after one batch export process of every completion
Connection number, if it is detected that connection number is more than preset value, server determines the currently received data processing operation amount of tables of data
It is larger, it is impacted if continuing to execute the business that data line export operation will provide tables of data.
In step 207, if being more than preset value, dormancy time is extended into scheduled duration, or, dormancy time is extended the
One duration, the first duration with to connect number more than preset value be more than degree correlation.
Detect connection number be more than preset value when, server determine the currently received data processing operation amount of tables of data compared with
Greatly, then dynamic extends dormancy time, for example, scheduled duration when suspend mode is 5 seconds, is then extended for the scheduled duration 2 seconds.
The extended time can be fixed value or dynamic value.In other words, server can extend dormancy time predetermined
Duration, for example, 2 seconds；Or, dormancy time is extended the first duration, the first duration with to connect number more than preset value be more than degree
Correlation.For example, connection number is more than degree at the section (A, B), extension 1 second；Connection number be more than degree (B,
C) when section, extend 2 seconds；It is more than degree at the section (C, D) to connect number, extension 3 seconds.
It should be noted that whether server can also be by surpassing every the connection number of prefixed time interval detection data table
The connection number crossed in preset value or real-time detector data table whether be more than preset value method, connection number is monitored, if
When the duration of server suspend mode reaches scheduled duration, still detects that the connection number in tables of data is more than preset value, then service
Device continues to extend dormancy time.
When server detects that the connection number in tables of data is less than preset value, it is determined that carry out data line to tables of data and lead
The influence made to tables of data offer business of going out for drill is little, and the data line that next group can be carried out to tables of data exports operation.
In a step 208, it for data line derived from every batch of, is deleted from tables of data.
When server export data line is for the scene of deletion data from tables of data, for data derived from every batch of
Row, server can line by line delete data from tables of data.In other words, server successively advances to derived data line
Row write locks, then the data line is deleted from tables of data, avoids and is needed greatly when deleting the data line in tables of data
The case where area locks the data line in tables of data into row write, to reduce the influence of the business provided tables of data.
In conclusion the data processing method provided in the embodiment of the present disclosure, a small amount of by using multiple batches of, every batch of
Data export operation, derived data line is deleted, solving longer deletion data time in the related technology can influence
The problem of availability of database, has reached when deleting large-scale data, will not on a large scale have been locked to tables of data, single
Position time pressure caused by tables of data it is smaller, do not interfere with the performance of tables of data, to improve tables of data availability effect
Fruit.
The data processing method provided in the embodiment of the present disclosure passes through the pre- timing of suspend mode in adjacent export process twice
It is long, and according to current connection number dynamic adjustment sleep time, further decrease the unit interval caused by tables of data pressure compared with
It is small, do not interfere with the performance of tables of data, to improve tables of data availability effect.
It is shown in Figure 3, it is a kind of flow chart of data processing method shown according to another exemplary embodiment,
Include the following steps.
In step 301, the data line that key assignments meets treatment conditions is calculated from tables of data, as pending data
Row.
In step 302, it obtains this batch and needs derived data line quantity.
In step 303, the dormancy time after this batch export data line is obtained.
In step 304, data line is exported from tables of data according to data line quantity.
In step 305, if there is also pending data lines after the export process of this batch, according to suspend mode
After time carries out suspend mode scheduled duration, into the export process of next batch.
Within step 306, after the export process of this batch, whether the connection number of detection data table is more than default
Value.
In step 307, if being more than preset value, dormancy time is extended into scheduled duration, or, dormancy time is extended the
One duration, the first duration with to connect number more than preset value be more than degree correlation.
In step 308, after the export process of this batch, the leader follower replication of detection data table, which whether there is, to be prolonged
Late.
When server export data line is used for the scene of data filing, the data for ensureing the data line for filing are needed
Consistency.
Server usually carries out leader follower replication to tables of data, to file the data in tables of data.
In general, the main table of tables of data is used for backup data management, from table for providing a user business.As the master of tables of data
From when replicating in the presence of delay, there are data inconsistence problems in the data line that may result in duplication.
Server can be after the export process of each batch, and the leader follower replication of detection data table, which whether there is, to be prolonged
Late.
It should be noted that the present embodiment is only the duplication between current data table and filing tables of data with leader follower replication
It illustrates.But the leader follower replication can also be the duplication between current data table and other tables of data.
In a step 309, if there is delay, dormancy time is extended into preset duration, or, dormancy time is extended second
The delay size correlation of duration, the second duration and delay.
When testing result is the presence of delay, server is determined temporarily to export without the data line of next batch and be operated,
Server extends dormant event dynamic, for example, scheduled duration when current hibernation is 5s, is then extended for the scheduled duration
2s。
The extended time can be fixed value or dynamic value.In other words, server can extend dormancy time predetermined
Duration, for example, 2 seconds；Or, dormancy time is extended the first duration, the first duration with to connect number more than preset value be more than degree
Correlation.For example, connection number is more than degree at the section (A, B), extension 1 second；Connection number be more than degree (B,
C) when section, extend 2 seconds；It is more than degree at the section (C, D) to connect number, extension 3 seconds.
It should be noted is that server can also pass through the leader follower replication every prefixed time interval detection data table
With the presence or absence of the leader follower replication of delay or real-time detector data table with the presence or absence of the method for delay, tables of data is monitored.
If still being detected when the duration of server suspend mode reaches the scheduled duration after extending there is delay in the leader follower replication of tables of data,
Then server continues to extend dormancy time.
It needs to illustrate on the other hand, the sleep time of server is by step 306 and step 307, step 308 and step
309 joint effects, that is, after the dormancy time of server reaches sleep time, connection number in tables of data be more than preset value or
There is any one establishment postponed in the two conditions in the leader follower replication of person's tables of data, then the sleep time of server can dynamic
Extend.Step 308 and step 309 can also execute in the embodiment depicted in figure 2.
In the step 310, data line derived from every batch of is deleted after filing to another tables of data from tables of data.
For data line derived from every batch of, server is filed by each data line to another tables of data, line by line will
Data line derived from this batch is deleted.That is, server can successively lock derived data line traveling row write, then should
Data line is deleted from tables of data, is avoided the when of filing to the data line in tables of data and is needed large area in tables of data
The case where data line is locked into row write, to reduce the influence of the business provided tables of data.
For example, filing tables of data is table 2.
| Key assignments | Date | User | Unit price | Quantity |
| 6 | April 2 | Xiao Li | 20 | 30 |
| 10 | April 3 | Xiao Zhang | 20 | 10 |
| 11 | April 4 | Xiao Wang | 20 | 10 |
Table 2
In the tables of data being shown in Table 1, if key assignments is respectively 1,5 data behavior, this batch needs the expired number filed
According to row, then the data line that key assignments is respectively 1,5 is directed into after the export of table 1 in table 2 by server, then is deleted line by line from table 1
This 2 row data line.
At this point, filing tables of data is as shown in table 3, the tables of data deleted after data line is then as shown in table 4.
| Key assignments | Date | User | Unit price | Quantity |
| 6 | April 2 | Xiao Li | 20 | 30 |
| 10 | April 3 | Xiao Zhang | 20 | 10 |
| 11 | April 4 | Xiao Wang | 20 | 10 |
| 1 | April 1 | Xiao Wang | 10 | 30 |
| 5 | April 3 | Xiao Zhang | 20 | 10 |
Table 3
| Key assignments | Date | User | Unit price | Quantity |
| 2 | April 1 | It is small red | 20 | 5 |
| 3 | April 2 | Xiao Wang | 10 | 10 |
| 4 | April 2 | Xiao Li | 10 | 20 |
| 6 | April 3 | Xiao Li | 30 | 10 |
| 7 | April 3 | It is small red | 30 | 20 |
| 8 | April 4 | Xiao Zhang | 30 | 30 |
Table 4
In conclusion the data processing method provided in the embodiment of the present disclosure, a small amount of by using multiple batches of, every batch of
Data export operation, derived data line is deleted, data can be influenced by solving longer deletion data time in the related technology
The problem of availability in library, has reached when filing large-scale data, will not on a large scale have been locked to tables of data, when unit
Between the pressure caused by tables of data it is smaller, do not interfere with the performance of tables of data, to improve tables of data availability effect.
The data processing method provided in the embodiment of the present disclosure passes through the pre- timing of suspend mode in adjacent export process twice
It is long, and according to current connection number and/or leader follower replication delay dynamic adjustment sleep time, further decrease unit interval logarithm
It is smaller according to pressure caused by table, do not interfere with the performance of tables of data, to improve tables of data availability effect.
Following is embodiment of the present disclosure, can be used for executing embodiments of the present disclosure.It is real for disclosure device
Undisclosed details in example is applied, embodiments of the present disclosure is please referred to.
It is shown in Figure 4, it is the device packet according to the block diagram of the data processing equipment shown in an exemplary embodiment
It includes.
Determining module 410 is configured as determining pending data line in tables of data.
Export module 420 is configured as dividing n batches to export the pending data line.
Removing module 430 is configured as, for data line derived from every batch of, deleting from the tables of data；Alternatively, filing
After to another tables of data, deleted from the tables of data.
In conclusion the data processing equipment provided in the embodiment of the present disclosure, a small amount of by using multiple batches of, every batch of
Data export operation, derived data line is deleted line by line, solving longer deletion data time in the related technology can influence
The problem of availability of database, has reached when deleting or filing large-scale data, will not greatly be advised to tables of data
Mode locking, unit interval pressure caused by tables of data is smaller, does not interfere with the performance of tables of data, to improve tables of data can
With the effect of property.
It is shown in Figure 5, it is the block diagram of the data processing equipment shown according to another exemplary embodiment, the device
Including.
Determining module 410 is configured as determining pending data line in tables of data.
Export module 420 is configured as dividing n batches to export the pending data line.
Removing module 430 is configured as, for data line derived from every batch of, deleting from the tables of data；Alternatively, filing
After to another tables of data, deleted from the tables of data.
Optionally, the export module 420, including：
Acquisition submodule 421 is configured as obtaining the derived data line quantity of this batch needs, and obtains the export of this batch
Dormancy time after data line；
Submodule 422 is exported, is configured as exporting the data line from the tables of data according to the data line quantity；
Suspend mode submodule 423, is configured as after the export process of this batch that there is also the pending data
When row, after carrying out suspend mode scheduled duration according to the dormancy time, into the export process of next batch；
It mutually holds in the mouth the end address of the initial address of data line derived from the next batch and data line derived from this batch
It connects.
Optionally, the export module 420 further includes：
First detection sub-module 424 is configured as after the export process of this batch, detecting in the tables of data
Connect whether number is more than preset value；Connection number is the current connection quantity of database where operation system accesses tables of data.
First delay submodule 425, is configured as when more than preset value, dormancy time is extended preset duration, or, will
Dormancy time extends the first duration, the first duration with to connect number more than preset value be more than degree correlation.
Optionally, which further includes：
Second detection sub-module 426 is configured as after the export process of this batch, detecting the principal and subordinate of the tables of data
It replicates with the presence or absence of delay；
Second delay submodule 427 is additionally configured to, when there is delay, dormancy time be extended preset duration, or, will
Dormancy time extends the second duration, the delay size correlation of the second duration and delay.
Optionally, the determining module 410 is additionally configured to calculate the number that key assignments meets treatment conditions from the tables of data
According to row, the data line pending as this.
In conclusion the data processing equipment provided in the embodiment of the present disclosure, a small amount of by using multiple batches of, every batch of
Data export operation, derived data line is deleted, solving longer deletion data time in the related technology can influence
The problem of availability of database, has reached when deleting or filing large-scale data, will not greatly be advised to tables of data
Mode locking, unit interval pressure caused by tables of data is smaller, does not interfere with the performance of tables of data, to improve tables of data can
With the effect of property.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method
Embodiment in be described in detail, explanation will be not set forth in detail herein.
One exemplary embodiment of the disclosure provides a kind of data processing equipment, at the data that can realize disclosure offer
Reason method, the data processing equipment include：Processor, the memory for storing processor-executable instruction；
Wherein, processor is configured as：
Pending data line is determined in tables of data；
N batches are divided to export the pending data line；
For data line derived from every batch of, deleted from the tables of data；Alternatively, after filing to another tables of data, from this
It is deleted in tables of data.
Fig. 6 is a kind of block diagram of the device of the table structure of the modification tables of data shown according to another exemplary embodiment.Example
Such as, device 600 may be provided as a network side equipment.With reference to Fig. 6, device 600 includes processing component 602, is further wrapped
One or more processors are included, and by the memory resource representated by memory 604, it can be by processing component 602 for storing
Execution instruction, such as application program.The application program stored in memory 604 may include one or more every
One module for corresponding to one group of instruction.In addition, processing component 602 is configured as executing instruction, to execute above-mentioned data processing
Method.
Device 600 can also include the power management that a power supply module 606 is configured as executive device 600, and one has
Line or radio network interface 608 are configured as device 600 being connected to network and input and output (I/O) interface 610.Dress
Setting 600 can operate based on the operating system for being stored in memory 604, such as Windows ServerTM, Mac OS XTM,
UnixTM, LinuxTM, FreeBSDTM or similar.
Above-mentioned memory is also stored with for executing the instruction operated as follows：
Pending data line is determined in tables of data；
N batches are divided to export pending data line；
For data line derived from every batch of, deleted from tables of data；Alternatively, after filing to another tables of data, from data
It is deleted in table.
Optionally, above-mentioned memory is also stored with for executing the instruction operated as follows：
It obtains this batch and needs derived data line quantity, and obtain the dormancy time after this batch export data line；
Data line is exported from tables of data according to data line quantity；
If there is also pending data lines after the export process of this batch, suspend mode is carried out according to dormancy time
After scheduled duration, into the export process of next batch；
The initial address of data line derived from next batch is mutually connected with the end address of data line derived from this batch.
Optionally, above-mentioned memory is also stored with for executing the instruction operated as follows：
After the export process of this batch, whether the connection number of detection data table is more than preset value, and connection number is industry
The current connection quantity of database where business system accesses tables of data；
If being more than preset value, dormancy time is extended into preset duration, or, by dormancy time the second duration of extension, second
The delay size correlation of duration and delay.
Optionally, above-mentioned memory is also stored with for executing the instruction operated as follows：
After the export process of this batch, the leader follower replication of detection data table is with the presence or absence of delay；
If there is delay, dynamic extends dormancy time.
Optionally, above-mentioned memory is also stored with for executing the instruction operated as follows：
The data line that key assignments meets treatment conditions is calculated from tables of data, as pending data line.
Those skilled in the art after considering the specification and implementing the invention disclosed here, will readily occur to its of the disclosure
Its embodiment.This application is intended to cover any variations, uses, or adaptations of the disclosure, these modifications, purposes or
Person's adaptive change follows the general principles of this disclosure and includes the undocumented common knowledge in the art of the disclosure
Or conventional techniques.The description and examples are only to be considered as illustrative, and the true scope and spirit of the disclosure are by following
Claim is pointed out.
It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and
And various modifications and changes may be made without departing from the scope thereof.The scope of the present disclosure is only limited by the accompanying claims.
Claims (7)
1. a kind of data processing method, which is characterized in that the method includes：
Pending data line is determined in tables of data；
N batches are divided to export the pending data line；
For data line derived from every batch of, deleted from the tables of data；Alternatively, filing is to after another tables of data, from described
It is deleted in another tables of data,
It is described to divide n batches to export the pending data line, including：
It obtains this batch and needs derived data line quantity；And obtain the dormancy time after this batch export data line；
The data line is exported from the tables of data according to the data line quantity；
If there is also the pending data lines after export process of described batch, according to the dormancy time
After carrying out suspend mode scheduled duration, into the export process of next batch；
It mutually holds in the mouth the end address of the initial address of data line derived from the next batch and data line derived from described batch
It connects,
The method further includes：
After the export process of described batch, whether the connection number for detecting the tables of data is more than preset value；The company
Connect the current connection quantity that number is database where operation system accesses the tables of data；
If being more than the preset value, the dormancy time is extended into preset duration, or, when the dormancy time is extended first
Long, first duration is more than degree correlation more than the preset value with the connection number.
2. according to the method described in claim 1, it is characterized in that, the method further includes：
After the export process of described batch, the leader follower replication of the tables of data is detected with the presence or absence of delay；
If there is delay, the dormancy time is extended into preset duration, or, the dormancy time is extended the second duration, institute
State the delay size correlation of the second duration and the delay.
3. according to any method of claim 1 and 2, which is characterized in that it is described determined in tables of data it is pending
Data line, including：
The data line that key assignments meets treatment conditions is calculated from the tables of data, as the pending data line.
4. a kind of data processing equipment, which is characterized in that described device includes：
Determining module is configured as determining pending data line in tables of data；
Export module is configured as dividing n batches to export the pending data line；
Removing module is configured as, for data line derived from every batch of, deleting from the tables of data；Alternatively, filing to another
After one tables of data, deleted from the tables of data,
The export module, including：
Acquisition submodule is configured as obtaining the derived data line quantity of this batch needs, and obtains this batch export data line
Dormancy time afterwards；
Submodule is exported, is configured as exporting the data line from the tables of data according to the data line quantity；
Suspend mode submodule, if being configured as after the export process of described batch, there is also the pending data
Row, then after carrying out suspend mode scheduled duration according to the dormancy time, into the export process of next batch；
It mutually holds in the mouth the end address of the initial address of data line derived from the next batch and data line derived from described batch
It connects,
The export module further includes：
First detection sub-module is configured as after the export process of described batch, detecting the connection of the tables of data
Whether number is more than preset value；The connection number is the current connection quantity of database where operation system accesses the tables of data；
First delay submodule, is configured as when more than the preset value, and the dormancy time is extended preset duration, or,
The dormancy time is extended into the first duration, first duration is more than that degree is in more than the preset value with the connection number
Positive correlation.
5. device according to claim 4, which is characterized in that the export module further includes：
Second detection sub-module is configured as after the export process of described batch, detecting the principal and subordinate of the tables of data
It replicates with the presence or absence of delay；
Second delay submodule, is additionally configured to when there is delay, the dormancy time is extended preset duration, or, by institute
It states dormancy time and extends the second duration, the delay size correlation of second duration and the delay.
6. according to any device of claim 4 and 5, which is characterized in that the determining module is additionally configured to from described
The data line that key assignments meets treatment conditions is calculated in tables of data, as the pending data line.
7. a kind of data processing equipment, which is characterized in that described device includes：
Processor；
Memory for storing the processor-executable instruction；
Wherein, the processor is configured as：
Pending data line is determined in tables of data；
N batches are divided to export the pending data line；
For data line derived from every batch of, deleted from the tables of data；Alternatively, filing is to after another tables of data, from described
It is deleted in tables of data,
It is described to divide n batches to export the pending data line, including：
It obtains this batch and needs derived data line quantity；And obtain the dormancy time after this batch export data line；
The data line is exported from the tables of data according to the data line quantity；
If there is also the pending data lines after export process of described batch, according to the dormancy time
After carrying out suspend mode scheduled duration, into the export process of next batch；
It mutually holds in the mouth the end address of the initial address of data line derived from the next batch and data line derived from described batch
It connects,
The processor is additionally configured to：
After the export process of described batch, whether the connection number for detecting the tables of data is more than preset value；The company
Connect the current connection quantity that number is database where operation system accesses the tables of data；
If being more than the preset value, the dormancy time is extended into preset duration, or, when the dormancy time is extended first
Long, first duration is more than degree correlation more than the preset value with the connection number.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510325610.3A CN104965882B (en) | 2015-06-12 | 2015-06-12 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510325610.3A CN104965882B (en) | 2015-06-12 | 2015-06-12 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104965882A CN104965882A (en) | 2015-10-07 |
| CN104965882B true CN104965882B (en) | 2018-09-04 |
Family
ID=54219919
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510325610.3A Active CN104965882B (en) | 2015-06-12 | 2015-06-12 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104965882B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107979653B (en) * | 2018-01-16 | 2021-02-09 | 北京小米移动软件有限公司 | Load balancing method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1516835A (en) * | 2002-04-15 | 2004-07-28 | 索尼公司 | Data storage device |
| CN101197016A (en) * | 2006-12-08 | 2008-06-11 | 鸿富锦精密工业（深圳）有限公司 | Batching concurrent job processing equipment and method |
-
2015
- 2015-06-12 CN CN201510325610.3A patent/CN104965882B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1516835A (en) * | 2002-04-15 | 2004-07-28 | 索尼公司 | Data storage device |
| CN101197016A (en) * | 2006-12-08 | 2008-06-11 | 鸿富锦精密工业（深圳）有限公司 | Batching concurrent job processing equipment and method |
Non-Patent Citations (1)
| Title |
|---|
| mysql批量删除大量数据;https://www.2cto.com/database/201303/196534.html;《红黑联盟》;20130320;第1页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104965882A (en) | 2015-10-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9167028B1 (en) | Monitoring distributed web application transactions | |
| TWI730043B (en) | Association analysis method and device | |
| JP5664098B2 (en) | Composite event distribution apparatus, composite event distribution method, and composite event distribution program | |
| US9612641B2 (en) | Adjusting the connection idle timeout in connection pools | |
| US8521692B1 (en) | Storage system and method for controlling storage system | |
| Tang et al. | Dynamic job ordering and slot configurations for MapReduce workloads | |
| US9612873B2 (en) | Dynamically scalable data collection and analysis for target device | |
| EP2199915A1 (en) | Monitoring memory consumption | |
| CN103744808B (en) | A kind of method and apparatus for being used to control I/O to ask | |
| WO2004063928A1 (en) | Database load reducing system and load reducing program | |
| WO2022257615A1 (en) | Information processing method and apparatus, and storage medium | |
| WO2017005115A1 (en) | Adaptive optimization method and device for distributed dag system | |
| JP2005251048A (en) | System monitoring method | |
| Maroulis et al. | A holistic energy-efficient real-time scheduler for mixed stream and batch processing workloads | |
| CN104965882B (en) | Data processing method and device | |
| US11283697B1 (en) | Scalable real time metrics management | |
| WO2022016845A1 (en) | Multi-node monitoring method and apparatus, electronic device, and storage medium | |
| CN108932241A (en) | Daily record data statistical method, device and node | |
| CN112528327A (en) | Data desensitization method and device and data restoration method and device | |
| CN105067994A (en) | Method and application for positioning system-on-chip temporal logic error, and error rate calculation method | |
| CN106547517B (en) | A kind of method and device controlling the waiting time | |
| US11593318B2 (en) | Techniques for asynchronous snapshot invalidation | |
| CN109558210A (en) | A kind of method and system of virtual machine applied host machine GPU equipment | |
| CN103324542A (en) | Method and device for inter-module calls | |
| US10713103B2 (en) | Lightweight application programming interface (API) creation and management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |