CN113515497A - Database data processing method, device and system - Google Patents
Database data processing method, device and system Download PDFInfo
- Publication number
- CN113515497A CN113515497A CN202010272752.9A CN202010272752A CN113515497A CN 113515497 A CN113515497 A CN 113515497A CN 202010272752 A CN202010272752 A CN 202010272752A CN 113515497 A CN113515497 A CN 113515497A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- time
- target database
- cold
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 38
- 230000004083 survival effect Effects 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 15
- 238000004140 cleaning Methods 0.000 claims description 8
- 238000010926 purge Methods 0.000 claims 1
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000003203 everyday effect Effects 0.000 description 2
- 230000010365 information processing Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
Abstract
The invention discloses a data processing method, device and system of a database, and relates to the technical field of databases. The data processing method of the database comprises the following steps: periodically traversing the data in the target database, and identifying cold data in the target database according to the survival time and/or the access frequency and/or the last access time of the data; the cold data is purged from the target database. The method can effectively maintain the data volume of the database, has small influence on the data query function of the target database, improves the performance of the database, and can meet the requirements of data reporting and data report table query in a large-scale network environment.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a method, an apparatus, and a system for processing data in a database.
Background
With the increasing demand of large enterprises and institutions on the scale of local area networks, the number of terminals in a local area network is more and more, and in a large local area network system, the number of terminals can usually reach hundreds of thousands or even millions, and under the scene of millions of terminals, a large number of terminals simultaneously initiate data reporting operation, and a console initiates operation of viewing a data report in real time. Without any processing, the query speed of the database is decreased as the database usage time increases. If the console is a web server, the huge amount of data may cause the web server to respond over time, resulting in a crash of the entire web server.
Under the condition, how to effectively maintain the database system can meet the requirements of data reporting and data report query in a large-scale network environment becomes a difficult problem in the front of technicians.
Disclosure of Invention
In view of this, the present application provides a method, an apparatus, and a system for processing data of a database, and mainly aims to solve the technical problem of performance degradation of the database with the increase of the usage time.
According to a first aspect of the present invention, there is provided a data processing method of a database, the method comprising:
periodically traversing data in a target database, and identifying cold data in the target database according to the survival time and/or the access frequency and/or the last access time of the data;
the cold data is purged from the target database.
In one embodiment, prior to identifying cold data in the target database, the method further comprises: and setting a survival time threshold value and/or an access frequency threshold value and/or a last access time threshold value of the data according to the newly added data amount of the target database in a preset time interval.
In one embodiment, identifying cold data in the target database based on the time-to-live and/or frequency of access and/or last time of access of the data comprises: determining the survival time of the data according to the uploading time of the data and the current time of the target database; comparing the survival time of the data with a preset survival time threshold; and if the survival time of the data is greater than the preset survival time threshold, determining the data to be cold data.
In one embodiment, identifying cold data in the target database based on the time-to-live and/or frequency of access and/or last time of access of the data comprises: determining the access frequency of the data according to the number of the access identifications of the data; comparing the access frequency of the data with a preset access frequency threshold; and if the access frequency of the data is less than a preset access frequency threshold value, determining the data to be cold data.
In one embodiment, identifying cold data in the target database based on the time-to-live and/or frequency of access and/or last time of access of the data comprises: comparing the last access time of the data with a preset last access time threshold; and if the last access time of the data is earlier than a preset last access time threshold, determining that the data is cold data.
In one embodiment, the method further comprises: and uploading the cold data to a cloud database.
In one embodiment, the method further comprises: receiving a data query request; inquiring data to be inquired in a target database and/or inquiring cold data to be inquired in a cloud database; and performing data statistics on the data to be queried and/or the cold data to be queried to obtain a data statistical result and issuing the data statistical result.
According to a second aspect of the present invention, there is provided a data processing apparatus for a database, the apparatus comprising:
the data classification module is used for periodically traversing the data in the target database and identifying cold data in the target database according to the survival time and/or the access frequency and/or the last access time of the data;
and the data cleaning module is used for cleaning cold data from the target database.
In one embodiment, the apparatus further includes a threshold setting module, configured to set a lifetime threshold and/or an access frequency threshold and/or a last access time threshold of the data according to an amount of data newly added to the target database in a predetermined time interval.
In one embodiment, the data classification module is specifically configured to determine a survival time of the data according to an upload time of the data and a current time of the target database; comparing the survival time of the data with a preset survival time threshold; and if the survival time of the data is greater than the preset survival time threshold, determining the data to be cold data.
In an embodiment, the data classification module is further specifically configured to determine an access frequency of the data according to the number of the access identifiers of the data; comparing the access frequency of the data with a preset access frequency threshold; and if the access frequency of the data is less than a preset access frequency threshold value, determining the data to be cold data.
In one embodiment, the data classification module is further configured to compare a last access time of the data with a preset last access time threshold; and if the last access time of the data is earlier than a preset last access time threshold, determining that the data is cold data.
In one embodiment, the apparatus further comprises a data synchronization module, which is configured to upload the cold data to a cloud database.
In one embodiment, the apparatus further comprises a request receiving module, a data query module, and a data statistics module, wherein:
the request receiving module is used for receiving a data query request;
the data query module is used for querying data to be queried in the target database and/or querying cold data to be queried in the cloud database;
and the data statistics module is used for performing data statistics on the data to be queried and/or the cold data to be queried to obtain a data statistics result and issuing the data statistics result.
According to a third aspect of the present invention, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the data processing method of the above-described database.
According to a fourth aspect of the present invention, there is provided a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the data processing method of the database when executing the program.
According to a fifth aspect of the present invention, there is provided a data processing system for a database, the system comprising: the system comprises a terminal, a database server, a target database and a cloud database; the database server is stored with a computer program, and the program realizes the data processing method of the database when being executed by a processor, wherein:
the terminal is used for reporting data to the database server;
the database server is used for periodically traversing the data in the target database and identifying cold data in the target database according to the survival time and/or the access frequency and/or the last access time of the data;
the database server is also used for clearing cold data from the target database.
According to a sixth aspect of the present invention, there is provided a computer program product comprising computer executable instructions for implementing the data processing method of the above database when executed.
According to the data processing method, the device and the system of the database, cold data in the target database are identified according to the survival time and/or the access frequency and/or the last access time of the data by periodically traversing the data in the target database, and then the cold data are cleared from the target database, so that the data volume of the database can be effectively maintained, the influence on the data query function of the target database is small, the performance of the database is improved, and the requirements of data reporting and data report table query under a large-scale network environment can be met.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a schematic flow chart illustrating a data processing method for a database according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating another data processing method for a database according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a data processing apparatus of a database according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a data processing apparatus of another database provided in an embodiment of the present invention;
FIG. 5 is a block diagram of a database data processing system according to an embodiment of the present invention;
fig. 6 shows a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
Embodiments of the invention are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the computer system/server include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set top boxes, programmable consumer electronics, network pcs, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above systems, and the like.
The computer system/server may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.
The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
In one embodiment, as shown in fig. 1, a data processing method for a database is provided, which is applied to a database server as an example, and includes the following steps:
101. and periodically traversing the data in the target database, and identifying cold data in the target database according to the survival time and/or the access frequency and/or the last access time of the data.
The cold data refers to data with low access frequency or long data updating time in the database, and relatively speaking, the data with high access frequency or long data updating time in the database has higher application value than the cold data.
Specifically, the method provided in this embodiment may be applied to a database server, an operation object of the method is a target database, and further, the database server may periodically traverse the target server, where a traversal period may be automatically set according to an update speed of data in the target database, if data update is fast or data amount is large, the target server may be traversed once at the same time every day, and if data update is slow or data amount is small, the target server may be traversed once every 3 days, 5 days or a week, and further, after traversing the target server, the database server may identify cold data in the target database according to parameters such as an access frequency of the data, a data update time, or a last access time of the data.
102. The cold data is purged from the target database.
Specifically, the database server may remove the cold data from the target database after recognizing the cold data, so as to save a storage space of the target database and improve performance of the target database.
According to the data processing method of the database provided by the embodiment, the cold data in the target database is identified by periodically traversing the data in the target database according to the survival time and/or the access frequency and/or the last access time of the data, and then the cold data is removed from the target database, so that the data volume of the database is effectively maintained, meanwhile, the influence on the data query function of the target database is small, the performance of the database is improved, and the data processing method can meet the requirements of data reporting and data report table query in a large-scale network environment.
Further, as a refinement and an extension of the specific implementation of the above embodiment, in order to fully illustrate the implementation process of the embodiment, a data processing method of a database is provided, as shown in fig. 2, the method includes the following steps:
201. and setting a survival time threshold value and/or an access frequency threshold value and/or a last access time threshold value of the data according to the newly added data amount of the target database in a preset time interval.
Specifically, the database server may automatically adjust the division standard of the cold data according to the data increment of the target database within a predetermined time, for example, if the data increment exceeds a preset range, for example, 100 ten thousand pieces of data, within a predetermined time interval, for example, within one week, the database server may determine that the data increment of the target database is larger, thereby expanding the determination range of the cold data, that is, the database server may correspondingly shorten the lifetime threshold of the data and/or increase the access frequency threshold and/or approach the last access time threshold, for example, adjust the lifetime threshold from 3 days to 1 day and/or adjust the access frequency threshold from 100 times a week to 200 times a week and/or adjust the last access time threshold from 48 hours to 24 hours when the data increment is larger; or for another example, if the data increment is small within the predetermined time interval, the database server may determine that the data increment of the target database is small and narrow the determination range of the cold data, i.e., the database server may adjust the lifetime threshold of the data to be longer and/or the access frequency threshold to be lower and/or the last access time threshold to be earlier, for example, when the data increment is small, the lifetime threshold may be adjusted from 3 days to 7 days and/or the access frequency threshold may be adjusted from 200 times a week to 100 times a week and/or the last access time threshold may be adjusted from 48 hours to 72 hours, thereby achieving the goal of dynamically adjusting the identification range of the cold data according to the data change amount.
202. And periodically traversing the data in the target database, and identifying cold data in the target database according to the survival time and/or the access frequency and/or the last access time of the data.
Specifically, the traversal period may be automatically set according to the update speed of the data in the target database, if the data update is fast or the data amount is large, the target server may be traversed once at the same time every day, and if the data update is slow or the data amount is small, the target server may be traversed once every 3 days, 5 days or a week, and further, after the database server traverses the target server, the cold data in the target database may be identified according to the parameters such as the access frequency of the data, the data lifetime, and/or the last access time of the data.
For example, the database server may identify cold data based on the lifetime of the data. Specifically, the database server may determine the lifetime of the data according to the uploading time of the data and the current time of the target database, compare the lifetime of the data with a preset lifetime threshold, and determine that the data is cold data if the lifetime of the data is greater than the preset lifetime threshold, for example, the preset lifetime threshold is 7 days, and determine that the data is cold data if the lifetime of the data is greater than 7 days.
For another example, the database server may identify cold data based on the frequency of access to the data. Specifically, the database server may determine an access frequency of the data according to the number of the access identifiers of the data, compare the access frequency of the data with a preset access frequency threshold, and determine that the data is cold data if the access frequency of the data is less than the preset access frequency threshold, for example, the preset access frequency threshold is 100 times a week, and determine that the data is cold data if the access frequency of the data is less than 100 times.
As another example, the database server may identify cold data based on the last access time of the data. Specifically, the database server may compare the last access time of the data with a preset last access time threshold, and if the last access time of a certain data is earlier than the preset last access time threshold, determine that the data is cold data, for example, the preset last access time threshold is 72 hours, and if the last access time of a certain data is earlier than 72 hours, determine that the data is cold data.
For another example, the database server may determine the cold data according to two or three parameters of the access frequency of the data, the lifetime of the data, and the last access of the data, for example, the database server may compare the access frequency of the data with a preset access frequency threshold, and at the same time, compare the lifetime of the data with a preset lifetime threshold, if the lifetime of a certain data is greater than the preset lifetime threshold and the access frequency of the data is less than the preset access frequency threshold, determine that the data is the cold data, for example, the preset lifetime threshold is 7 days, and the preset access frequency threshold is 100 times a week, and if the lifetime of a certain data is greater than 7 days and the access frequency is less than 100 times, determine that the data is the cold data. It will be appreciated that the manner in which cold data is determined based on other parameters is similar.
203. And uploading the cold data to a cloud database.
The cloud database is a database constructed on a cloud cluster, and the reliability of the database can be higher, complicated maintenance work can be omitted, and hardware cost is saved.
Specifically, the database server can upload the divided cold data to the cloud database, so that the cold data can be inquired through the cloud server at any time, if the data volume of the cold data is large, the cold data can be divided into a plurality of batches, then the cold data are uploaded to the cloud database in batches, and then the cold data which are successfully uploaded are marked as being successfully uploaded.
204. The cold data is purged from the target database.
Specifically, after cold data are uploaded to the cloud service, the database server can record index information of the cold data marked as successful uploading in the target database into the data cleaning sequence, and then clear data corresponding to the index information in the data cleaning sequence, so that the storage space of the target database is saved, and the performance of the target database is improved.
205. A data query request is received.
Specifically, the database server may directly receive the data query request sent by the terminal, or may receive the data query request forwarded by the console.
206. And inquiring data to be inquired in the target database and/or inquiring cold data to be inquired in the cloud database.
Specifically, after receiving the data query request, the database server first extracts the name of the data to be queried and the query time range from the data query request, and then queries the data to be queried with the same name in the target database and/or queries the cold data to be queried in the cloud database according to the name and time of the data to be queried.
207. And performing data statistics on the data to be queried and/or the cold data to be queried to obtain a data statistical result and issuing the data statistical result.
Specifically, the database server may perform real-time data statistics on the queried data after querying the data to be queried and/or the cold data to be queried, to obtain a data statistics result, integrate the data statistics result in a report according to a time sequence, and issue the integrated report to the console and the terminal.
According to the data processing method of the database, cold data in a target database is recognized according to various bases, the cold data are uploaded to a cloud database, and the cold data are removed from the target database, so that a database server can query data to be queried from the target database and/or the cloud database at any time when receiving a data query request, and perform data statistics and issuing. By the method, the data volume of the target database can be effectively maintained, the data query function of the database is not influenced, and the performance of the database and the bearing capacity of the database in response to large-scale data are further improved.
Further, as a specific implementation of the method shown in fig. 1 and fig. 2, this embodiment provides a data processing apparatus for a database, as shown in fig. 3, the apparatus includes: a data classification module 31 and a data cleaning module 32.
The data classification module 31 is configured to periodically traverse the data in the target database, and identify cold data in the target database according to the survival time and/or the access frequency and/or the last access time of the data;
and a data cleaning module 32, operable to clean cold data from the target database.
In a specific application scenario, the apparatus further includes a threshold setting module 33, configured to set a lifetime threshold and/or an access frequency threshold and/or a last access time threshold of the data according to a newly added data amount of the target database in a predetermined time interval.
In a specific application scenario, the data classification module 31 may be specifically configured to determine a survival time of the data according to an upload time of the data and a current time of the target database; comparing the survival time of the data with a preset survival time threshold; and if the survival time of the data is greater than the preset survival time threshold, determining the data to be cold data.
In a specific application scenario, the data classification module 31 is further specifically configured to determine an access frequency of the data according to the number of the access identifiers of the data; comparing the access frequency of the data with a preset access frequency threshold; and if the access frequency of the data is less than a preset access frequency threshold value, determining the data to be cold data.
In a specific application scenario, the data classification module 31 is further configured to compare the last access time of the data with a preset last access time threshold; if the last access time of the data is earlier than a preset last access time threshold value, determining that the data is cold data
In a specific application scenario, as shown in fig. 4, the apparatus further includes a data synchronization module 34, where the data synchronization module may be configured to upload cold data to a cloud database.
In a specific application scenario, as shown in fig. 4, the apparatus further includes a request receiving module 35, a data querying module 36 and a data statistics module 37, where:
a request receiving module 35, configured to receive a data query request;
the data query module 36 may be configured to query the target database for the data to be queried, and/or query the cloud database for the cold data to be queried;
and the data statistics module 37 is configured to perform data statistics on the data to be queried and/or the cold data to be queried, obtain a data statistics result, and issue the data statistics result.
It should be noted that other corresponding descriptions of the functional units related to the data processing apparatus of the database provided in this embodiment may refer to the corresponding descriptions in fig. 1 and fig. 2, and are not described herein again.
Further, as shown in fig. 4, this embodiment further provides a data processing system of a database, where the system includes a terminal, a database server, a target database, and a cloud database, where the database server stores a computer program, and the computer program, when executed by a processor, may implement the data processing method of the database provided in the above embodiment, where:
the terminal is used for reporting data to the database server;
the database server is used for storing the data reported by the terminal into the target database, periodically traversing the data in the target database, and identifying cold data in the target database according to the survival time and/or the access frequency and/or the last access time of the data;
and the database server is also used for clearing cold data from the target database.
Based on the methods shown in fig. 1 and fig. 2, correspondingly, the present embodiment further provides a storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the data processing method of the database shown in fig. 1 and fig. 2.
Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, and the software product to be identified may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, or the like), and include several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the method according to the implementation scenarios of the present application.
Based on the above methods shown in fig. 1 and fig. 2, the data processing apparatus embodiments of the databases shown in fig. 3 and fig. 4, and the data processing system embodiment of the database shown in fig. 5, to achieve the above object, this embodiment further provides an entity electronic device for data processing of the database, as shown in fig. 6, where the entity electronic device may specifically be a personal computer, a server, a smart phone, a tablet computer, a smart watch, or other network devices, and the entity device includes a storage medium and a processor; a storage medium for storing a computer program; a processor for executing the computer program to implement the above-mentioned methods as shown in fig. 1 and fig. 2.
Optionally, the entity device may further include a user interface, a network interface, a camera, a Radio Frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, and the like. The user interface may include a Display screen (Display), an input unit such as a keypad (Keyboard), etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), etc.
Those skilled in the art will appreciate that the physical device structure of the data processing of the database provided in the present embodiment does not constitute a limitation to the physical device, and may include more or less components, or combine some components, or arrange different components.
The storage medium may further include an operating system and a network communication module. The operating system is a program for managing the hardware of the above-mentioned entity device and the software resources to be identified, and supports the operation of the information processing program and other software and/or programs to be identified. The network communication module is used for realizing communication among components in the storage medium and communication with other hardware and software in the information processing entity device.
Through the above description of the embodiments, those skilled in the art will clearly understand that the present application can be implemented by software plus a necessary general hardware platform, and can also be implemented by hardware. By applying the technical scheme of the application, the data volume of the database can be effectively maintained, the data query function of the database cannot be influenced, the performance of the database is improved, and the requirements of data reporting and data report query in a large-scale network environment can be met.
In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts in the embodiments are referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The method and system of the present invention may be implemented in a number of ways. For example, the methods and systems of the present invention may be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustrative purposes only, and the steps of the method of the present invention are not limited to the order specifically described above unless specifically indicated otherwise. Furthermore, in some embodiments, the present invention may also be embodied as a program recorded in a recording medium, the program including machine-readable instructions for implementing a method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.
The description of the present invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to practitioners skilled in this art. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
Claims (11)
1. A method of data processing of a database, the method comprising:
periodically traversing data in a target database, and identifying cold data in the target database according to the survival time and/or the access frequency and/or the last access time of the data;
purging the cold data from the target database.
2. The method of claim 1, wherein prior to identifying cold data in the target database, further comprising:
and setting a survival time threshold value and/or an access frequency threshold value and/or a last access time threshold value of the data according to the newly added data amount of the target database in a preset time interval.
3. The method of claim 2, wherein identifying cold data in the target database according to the time-to-live and/or the access frequency and/or the last access time of the data comprises:
determining the survival time of the data according to the uploading time of the data and the current time of the target database;
comparing the survival time of the data with a preset survival time threshold;
and if the survival time of the data is greater than the preset survival time threshold, determining that the data is cold data.
4. The method of claim 2, wherein identifying cold data in the target database according to the time-to-live and/or the access frequency and/or the last access time of the data comprises:
determining the access frequency of the data according to the number of the access identifications of the data;
comparing the access frequency of the data with a preset access frequency threshold;
and if the access frequency of the data is less than the preset access frequency threshold, determining that the data is cold data.
5. The method of claim 2, wherein identifying cold data in the target database according to the time-to-live and/or the access frequency and/or the last access time of the data comprises:
comparing the last access time of the data with a preset last access time threshold;
and if the last access time of the data is earlier than the preset last access time threshold, determining that the data is cold data.
6. The method of claim 1, further comprising:
and uploading the cold data to a cloud database.
7. The method of claim 6, further comprising:
receiving a data query request;
inquiring data to be inquired in the target database and/or inquiring cold data to be inquired in the cloud database;
and performing data statistics on the data to be queried and/or the cold data to be queried to obtain a data statistical result and issuing the data statistical result.
8. A data processing apparatus of a database, the apparatus comprising:
the data classification module is used for periodically traversing the data in the target database and identifying cold data in the target database according to the survival time and/or the access frequency and/or the last access time of the data;
and the data cleaning module is used for cleaning the cold data from the target database.
9. A storage medium having a computer program stored thereon, the computer program, when being executed by a processor, realizing the steps of the method of any one of claims 1 to 7.
10. A computer device comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, characterized in that, when the program instructions are executed by a computer, the computer is caused to perform the steps of the method of any of the preceding claims 1 to 7.
11. A computer program product comprising computer executable instructions, wherein the instructions when executed are for implementing a method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010272752.9A CN113515497A (en) | 2020-04-09 | 2020-04-09 | Database data processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010272752.9A CN113515497A (en) | 2020-04-09 | 2020-04-09 | Database data processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113515497A true CN113515497A (en) | 2021-10-19 |
Family
ID=78060058
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010272752.9A Pending CN113515497A (en) | 2020-04-09 | 2020-04-09 | Database data processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113515497A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817425A (en) * | 2022-06-28 | 2022-07-29 | 成都交大大数据科技有限公司 | Method, device and equipment for classifying cold and hot data and readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103688259A (en) * | 2011-05-19 | 2014-03-26 | 甲骨文国际公司 | Techniques for automatic data placement with compression and columnar storage |
| CN104035925A (en) * | 2013-03-04 | 2014-09-10 | 深圳市腾讯计算机系统有限公司 | Data storage method and device and storage system |
| CN109857737A (en) * | 2019-01-03 | 2019-06-07 | 平安科技（深圳）有限公司 | A kind of cold and hot date storage method and device, electronic equipment |
| CN110287160A (en) * | 2019-05-31 | 2019-09-27 | 广东睿江云计算股份有限公司 | A kind of spatial cache method for cleaning and device |
-
2020
- 2020-04-09 CN CN202010272752.9A patent/CN113515497A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103688259A (en) * | 2011-05-19 | 2014-03-26 | 甲骨文国际公司 | Techniques for automatic data placement with compression and columnar storage |
| CN104035925A (en) * | 2013-03-04 | 2014-09-10 | 深圳市腾讯计算机系统有限公司 | Data storage method and device and storage system |
| CN109857737A (en) * | 2019-01-03 | 2019-06-07 | 平安科技（深圳）有限公司 | A kind of cold and hot date storage method and device, electronic equipment |
| CN110287160A (en) * | 2019-05-31 | 2019-09-27 | 广东睿江云计算股份有限公司 | A kind of spatial cache method for cleaning and device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817425A (en) * | 2022-06-28 | 2022-07-29 | 成都交大大数据科技有限公司 | Method, device and equipment for classifying cold and hot data and readable storage medium |
| CN114817425B (en) * | 2022-06-28 | 2022-09-02 | 成都交大大数据科技有限公司 | Method, device and equipment for classifying cold and hot data and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102511271B1 (en) | Method and device for storing and querying time series data, and server and storage medium therefor | |
| CN112434039A (en) | Data storage method, device, storage medium and electronic device | |
| CN104965873A (en) | Paging inquiring method and apparatus | |
| CN110727727A (en) | Statistical method and device for database | |
| CN110830551A (en) | Service request processing method, device and system | |
| CN108664492B (en) | Method and device for pushing content to user, electronic equipment and storage medium | |
| CN111030888A (en) | Domain name system DNS capacity measuring method, device, equipment and medium | |
| CN111158926B (en) | Service request analysis method, device and equipment | |
| CN116737482A (en) | Method and device for collecting chip test data in real time and electronic equipment | |
| CN113515497A (en) | Database data processing method, device and system | |
| CN114398520A (en) | Data retrieval method, system, device, electronic equipment and storage medium | |
| CN108875035B (en) | Data storage method of distributed file system and related equipment | |
| CN111159131A (en) | Performance optimization method, device, equipment and computer readable storage medium | |
| CN110909263B (en) | Method and device for determining companion relationship of identity characteristics | |
| CN112822289A (en) | Data uploading and reading method, system, equipment and storage medium | |
| CN113515541A (en) | Data query method, device and system of database | |
| CN104834728B (en) | A kind of method for pushing and device for subscribing to video | |
| CN106681894B (en) | Monitoring method and monitoring device of intelligent equipment | |
| CN110134547B (en) | Middleware-based repeated data deleting method and related device | |
| CN110119337B (en) | Data analysis method and device and server | |
| CN109982143B (en) | Method, device, medium and equipment for determining video playing time delay | |
| US10019248B2 (en) | System and method for service matching of instant message software | |
| CN111506613A (en) | Method, system, device and equipment for querying incidence relation of data record | |
| CN111291017A (en) | Mirror image storage and extraction method and device of mirror image warehouse | |
| CN105808905A (en) | Balance calculation method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |