CN115080605A - Data processing method, device, equipment and storage medium - Google Patents

Data processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN115080605A
CN115080605A CN202210625497.0A CN202210625497A CN115080605A CN 115080605 A CN115080605 A CN 115080605A CN 202210625497 A CN202210625497 A CN 202210625497A CN 115080605 A CN115080605 A CN 115080605A
Authority
CN
China
Prior art keywords
data
database
cold
hot
file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210625497.0A
Other languages
Chinese (zh)
Inventor
曾奕锟
傅克标
沈成宇
刘建永
侯敏
曹辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Ruipu Energy Co Ltd
Rept Battero Energy Co Ltd
Original Assignee
Shanghai Ruipu Energy Co Ltd
Rept Battero Energy Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Ruipu Energy Co Ltd, Rept Battero Energy Co Ltd filed Critical Shanghai Ruipu Energy Co Ltd
Priority to CN202210625497.0A priority Critical patent/CN115080605A/en
Publication of CN115080605A publication Critical patent/CN115080605A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2453—Query optimisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/24569—Query processing with adaptation to specific hardware, e.g. adapted for using GPUs or SSDs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0643—Management of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G06F3/0676—Magnetic disk device

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: judging whether the data in the database is cold data or hot data according to a set threshold value; if the data is cold data, storing the key query field of the cold data in a database, compressing the service data of the cold data into a file, storing the file in a disk and deleting the file from the database; if the hot data is the hot data, continuously storing the key query field and the service data of the hot data in the database. According to the method and the device, the storage space can be saved under the condition of acquiring the full amount of data, so that the cost is saved, the target data can be rapidly inquired, and the data retrieval speed is increased.

Description

Data processing method, device, equipment and storage medium
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
As the data amount is increased and accumulated, different types of data can present different access heat degrees, and through analysis, the data are considered to be hot data when initially written in the process of time, and the hot data are defined as cold data along with the gradual decrease of the access frequency or the overlong access time. At present, a partition table and compressed data are mainly used, the partition table still occupies the space of a database, and the compressed data is stored in a disk, so that the data query is slow.
Disclosure of Invention
The invention mainly aims to provide a data processing method, a data processing device, data processing equipment and a data processing storage medium, which can save storage space under the condition of acquiring full data, thereby saving cost, quickly inquiring target data and improving data retrieval speed.
In a first aspect, the present application provides a data processing method, including:
judging whether the data in the database is cold data or hot data according to a set threshold value;
if the data is cold data, storing the key query field of the cold data in a database, compressing the service data of the cold data into a file, storing the file in a disk and deleting the file from the database;
if the hot data is the hot data, continuously storing the key query field and the service data of the hot data in the database.
In one possible implementation, whether the data in the data is cold data or hot data is judged according to the access times.
In one possible embodiment, the data in the database is judged to be cold data or hot data according to the generation time.
In one possible implementation, cold data is read from disk and hot data is read from the database.
In one possible embodiment, the service data of the cold data and the hot data are written into a database as modified or added service data.
In one possible implementation manner, the metadata of the service data to be deleted is inquired in the database, and whether the service data to be deleted is cold data or hot data is judged;
if the data is cold data, inquiring and deleting the compressed file in the disk;
if the data is hot data, the data is directly deleted from the database.
In a possible embodiment, the set time period is a time period of a preset time length from 0 o' clock per day.
In a second aspect, the present application provides a data processing apparatus comprising:
the judging module is used for judging whether the data in the database is cold data or hot data according to a set threshold value;
the first storage module is used for storing the key query field of the cold data in a database if the cold data is the cold data, compressing the service data of the cold data into a file, storing the file in a disk and deleting the file from the database;
and the second storage module is used for continuously storing the key query fields and the service data of the hot data in the database if the hot data are hot data.
In a third aspect, the present application further provides an electronic device, including: a processor; a memory having computer readable instructions stored thereon which, when executed by the processor, implement the method of any one of the first aspects.
In a fourth aspect, the present application also provides a computer readable storage medium storing computer program instructions which, when executed by a computer, cause the computer to perform the method of any of the first aspects.
According to the data processing method, the data processing device, the data processing equipment and the storage medium, whether the data in the database are cold data or hot data is judged according to the set threshold; if the data is cold data, storing the key query field of the cold data in a database, compressing the service data of the cold data into a file, storing the file in a disk and deleting the file from the database; if the data is hot data, the key query fields and the service data of the hot data are continuously stored in the database, so that the storage space can be saved under the condition of acquiring the full amount of data, the cost is saved, the target data can be rapidly queried, and the data retrieval speed is increased.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is a flowchart of a data processing method provided in the present embodiment;
fig. 2 is a schematic diagram of a data processing apparatus according to the present embodiment;
FIG. 3 is a flow chart of data writing provided in the present embodiment;
fig. 4 is a flow chart of data deletion provided in the present embodiment;
FIG. 5 is a schematic diagram of an electronic device provided in an embodiment of the present application;
fig. 6 is a schematic diagram of a computer-readable program medium provided in an embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities.
Referring to fig. 1, fig. 1 is a flowchart of a data processing method according to the present invention, and as shown in fig. 1, a flowchart of data processing includes the steps of:
and S101, judging whether the data in the database is cold data or hot data according to a set threshold value.
Specifically, when data is queried, it is necessary to determine whether the data is cold data or hot data, and different data processing manners are different, and it is determined whether the data in the database is cold data or hot data according to a set threshold.
In one embodiment, when data is queried, whether the data in the database is cold data or hot data is determined according to the number of times of data access, and it can be understood that when data is queried, if the number of times of data query is greater than a set value, hot data is determined, and if the number of times of data query is less than the set value, cold data is determined. When the data in the database is stored for a long time and the number of accesses is greater than a set value, it is determined to be hot data.
In an embodiment, when data is queried, it may be further determined whether the data in the database is cold data or hot data according to the data generation time of the database, optionally, when the data is queried, the data generation time in the database exceeds one year, it is determined that the data in the database exceeding one year is cold data, and when the data is queried, the data generation time in the database does not exceed one year, it is determined that the data in the database not exceeding one year is hot data.
Optionally, when data is queried, whether the data in the database is cold data or hot data is determined according to the last use time of the data in the database, when the last use time of the data in the database exceeds a set value during data query, the data is determined to be cold data, and when the last use time of the data in the database does not exceed the set value during data query, the data is determined to be hot data.
Optionally, the cold data and the hot data in the database are processed at 0 point every day, and it can be understood that, when 0 point every day starts, the data in the database is judged to be the cold data or the hot data according to a set threshold, if the data is determined to be the cold data, the key query field of the cold data is stored in the database, the liquid level time of the cold data is compressed into a file and stored in a disk, and the compressed packet is deleted from the database. When the hot data is determined, the hot data key query field and the business data are continuously saved in the database.
And S102, if the data is cold data, storing the key query field of the cold data in a database, compressing the service data of the cold data into a file, storing the file in a disk, and deleting the file from the database.
Specifically, after the data in the database is confirmed to be cold data, the key query field of the cold data is stored in the database, the service data of the cold data is compressed into a file and stored in a disk, and the compressed packet is deleted from the database, so that the target data can be queried quickly, and meanwhile, the memory space is saved.
In one embodiment, when business data is queried, a query API of a server is called, the server queries metadata of the business data in a database, and determines data generation time according to the metadata of the business data, where the data metadata includes a data generation timestamp, the data generation time is determined according to the data generation timestamp, when the business data in the query database is determined to be cold data, a storage path is constructed according to the data metadata through the cold data, and data is read from a disk, where it is to be noted that the metadata of the data includes data access times and data generation time.
And S103, if the hot data exist, continuously storing the key query fields and the service data of the hot data in a database.
Optionally, after the data in the database is determined to be the hot data, the key query field and the service data of the hot data are continuously stored in the database, so that the target data can be quickly found when the data is queried.
Optionally, the data generation time is judged according to the metadata of the service data, where the data metadata includes a data generation timestamp, the judgment is performed according to the data generation timestamp, and when it is determined that the service data in the query database is hot data, the hot data is directly read from the database.
In one embodiment, when data is queried, if the data generation time exceeds one year, the data is determined to be cold data, the cold data is read from a disk, and after the cold data is read, meta information of the data is updated, wherein the meta information of the data comprises access times and access time.
Optionally, when data is queried, if the data generation time does not exceed one year, determining that the data is hot data, reading the hot data from the database, and updating the meta information of the data after the hot data is read.
In one embodiment, when data is queried and the number of times of access to queried target data is greater than a set value, the data is determined to be cold data, the cold data is read from a disk, and after the cold data is read, the meta information of the data is updated.
Optionally, when data is queried and the number of times of access to the queried target data is smaller than a set value, it is determined as hot data, cold data is read from the disk, and after the cold data is read, the meta information of the data is updated.
In one embodiment, when data is queried and the last access time of the queried target data is greater than a set value, the data is determined to be cold data, the cold data is read from a disk, and after the cold data is read, the meta information of the data is updated.
Optionally, when data is queried and the last access time of the queried target data is less than a set value, the data is determined to be hot data, cold data is read from the disk, and after the cold data is read, the meta information of the data is updated.
According to the data processing method, the data processing device, the data processing equipment and the storage medium, whether the data in the database are cold data or hot data is judged according to the set threshold; if the data is cold data, storing the key query field of the cold data in a database, compressing the service data of the cold data into a file, storing the file in a disk and deleting the file from the database; if the data is hot data, the key query fields and the service data of the hot data are continuously stored in the database, so that the storage space can be saved under the condition of acquiring the full amount of data, the cost is saved, the target data can be rapidly queried, and the data retrieval speed is increased.
Referring to fig. 2, fig. 2 is a schematic diagram of a data processing apparatus according to the present invention, and as shown in fig. 2, the data processing apparatus includes:
a judging module 201, configured to judge whether data in the database is cold data or hot data according to a set threshold;
the first storage module 202 is configured to, if the data is cold data, store a key query field of the cold data in a database, compress a service data of the cold data into a file, store the file in a disk, and delete the file from the database;
and the second storage module 203 is used for continuously storing the key query fields and the service data of the hot data in the database if the hot data is the hot data.
Further, in an embodiment, the determining module 201 is further configured to determine whether data in the data is cold data or hot data according to the number of accesses.
Further, in an embodiment, the determining module 201 is further configured to determine whether the data in the database is cold data or hot data according to the generation time.
Referring to fig. 3, fig. 3 is a data writing flow chart provided by the present invention, and as shown in fig. 3, the data writing flow chart includes:
step S301: and calling a server write API.
Step S302: the data is written to the database.
Step S303: the data metadata is updated.
Specifically, when data in the database needs to be modified or newly added with service data, a write-in API of the server is called, all the service data needing to be modified or newly added is written into the database, it should be noted that, when data is written, the service data needing to be modified or newly added is written into the database without distinguishing cold data from hot data, and after the modified or newly added service data is written into the database, the modification time or creation time in the service data metadata is updated and stored.
Referring to fig. 4, fig. 4 is a flowchart illustrating data deletion provided by the present invention, and as shown in fig. 4, the flowchart illustrating data deletion includes:
step 401: and inquiring the meta information of the service data to be deleted in the database.
Step 402: and judging whether the service data to be deleted is cold data or hot data.
Step 403: if the data is cold data, the compressed file in the disk is inquired and deleted, and if the data is hot data, the compressed file is directly deleted from the database.
Specifically, when deleting the service data, calling a deletion API of the server, firstly inquiring service data metadata information in the database, judging whether the deleted data generation time exceeds one year according to the inquired service data metadata information, if the data generation time in the inquired service data metadata information exceeds one year, determining the data to be cold data, deleting a data compression packet on a disk, if the data generation time does not exceed one year, determining that the hot data is directly deleted from the database, and after deleting the service data, updating the metadata of the service data.
Optionally, the deletion state of the data metadata of the service data is changed to 1.
Optionally, when deleting the service data, a deletion API of the server is called, the service data metadata information is queried in the database, whether the deleted data access times are greater than a set value is judged according to the queried service data metadata information, if the data access times in the queried service data metadata information are greater than the set value, the data are determined to be cold data, the data compression packet on the disk is deleted, if the data access times in the data metadata information are less than the set value, the data are determined to be hot data, the hot data are directly deleted from the database, and after deleting the service data, the metadata of the service data are updated.
Optionally, when deleting the service data, a deletion API of the server is called, the service data metadata information is queried in the database, whether the last access time of the deleted data is greater than a set value is judged according to the query service data metadata information, if the last access time of the data in the query service data metadata information is greater than the set value, the data is determined to be cold data, the data compression packet on the disk is deleted, if the last access time of the data in the data metadata information is less than the set value, the hot data is determined to be directly deleted from the database, and after deleting the service data, the metadata of the service data is updated.
Further, in an embodiment, after data is processed, that is, read, written, and deleted, and metadata of the data is stored, a task is processed at regular time, it can be understood that, starting at 0 point every day, data metadata in the data is queried, data whose data generation time exceeds one year, that is, cold data, service time of the cold data is compressed and packed, and the compressed packet of the cold data is moved to a preset path for storage, and after storage is completed, service data of the cold data which has been compressed and packed in a database is deleted, where the preset path is a path of yyyy/MM/id for storage, where yyy refers to year, MM refers to month, and id refers to primary key id in a data metadata table.
Optionally, after processing the data, that is, reading, writing, and deleting the data, and storing the metadata of the data, the task is processed at regular time, it can be understood that starting at 0 point every day, querying the metadata of the data, selecting the data with the data access times greater than a set value, that is, the cold data, compressing and packaging the service time of the cold data, and moving the compressed and packaged cold data to a preset path for storage, and after the storage is completed, deleting the service data of the cold data compressed and packaged in the database, where the preset path is a path of yyyy/MM/id for storage, where yyyy refers to the year, MM refers to the month, and id refers to the primary key id in the data metadata table.
Optionally, after processing the data, that is, reading, writing, and deleting the data, and storing the metadata of the data, the task is processed at regular time, it can be understood that starting at 0 point every day, querying the metadata of the data, selecting the data whose last access time is greater than a set value, that is, cold data, compressing and packaging the service time of the cold data, and moving the compressed packet of the cold data to a preset path for storage, and after the storage is completed, deleting the service data of the cold data that has been compressed and packaged in the database, where the preset path is a path of yyyy/MM/id for storage, where yyy refers to year, MM refers to month, and id refers to a primary key id in the data metadata table.
An electronic device 500 according to this embodiment of the invention is described below with reference to fig. 5. The electronic device 500 shown in fig. 5 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the electronic device 500 is embodied in the form of a general purpose computing device. The components of the electronic device 500 may include, but are not limited to: the at least one processing unit 510, the at least one memory unit 520, and a bus 530 that couples various system components including the memory unit 520 and the processing unit 510.
Wherein the storage unit stores program code that is executable by the processing unit 510 to cause the processing unit 510 to perform steps according to various exemplary embodiments of the present invention as described in the section "example methods" above in this specification.
The storage unit 520 may include readable media in the form of volatile storage units, such as a random access memory unit (RAM)521 and/or a cache memory unit 522, and may further include a read only memory unit (ROM) 523.
The storage unit 520 may also include a program/utility 524 having a set (at least one) of program modules 525, such program modules 525 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
Bus 530 may be one or more of any of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 500 may also communicate with one or more external devices 600 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 500, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 500 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 550. Also, the electronic device 500 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 560. As shown, the network adapter 560 communicates with the other modules of the electronic device 500 over the bus 530. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
According to an aspect of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above section "exemplary methods" of the present description, when said program product is run on the terminal device.
Referring to fig. 6, a program product 600 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In situations involving remote computing devices, the remote computing devices may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to external computing devices (e.g., through the internet using an internet service provider).
Furthermore, the above-described figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
In summary, according to the data processing method, apparatus, device and storage medium provided by the present application, whether data in a database is cold data or hot data is determined according to a set threshold; if the data is cold data, storing the key query field of the cold data in a database, compressing the service data of the cold data into a file, storing the file in a disk and deleting the file from the database; if the data is hot data, the key query fields and the service data of the hot data are continuously stored in the database, so that the storage space can be saved under the condition of acquiring the full amount of data, the cost is saved, the target data can be rapidly queried, and the data retrieval speed is increased.
The foregoing are merely exemplary embodiments of the present application and no attempt is made to show structural details of the invention in more detail than is necessary for the fundamental understanding of the art, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice with the teachings of the invention. It should be noted that, for those skilled in the art, without departing from the structure of the present invention, several changes and modifications can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicability of the patent. The scope of the claims of the present application shall be determined by the contents of the claims, and the description of the embodiments and the like in the specification shall be used to explain the contents of the claims.

Claims (10)

1. A data processing method, comprising:
judging whether the data in the database is cold data or hot data according to a set threshold value;
if the data is cold data, storing the key query field of the cold data in a database, compressing the service data of the cold data into a file, storing the file in a disk and deleting the file from the database;
if the hot data is the hot data, continuously storing the key query field and the service data of the hot data in the database.
2. The method of claim 1, wherein the determining whether the data in the database is cold data or hot data according to the set threshold comprises:
and judging whether the data in the data is cold data or hot data according to the access times.
3. The method of claim 1, wherein the determining whether the data in the database is cold data or hot data according to the set threshold comprises:
and judging whether the data in the database is cold data or hot data according to the generation time.
4. The method of claim 1, wherein:
cold data is read from the disk and hot data is read from the database.
5. The method of claim 1, wherein:
and writing the cold data and the hot data into a database as modified or newly added service data.
6. The method of claim 1, wherein: the method also comprises a service data deleting step which comprises the following steps:
inquiring the meta information of the service data to be deleted in a database, and judging whether the service data to be deleted is cold data or hot data;
if the data is cold data, inquiring and deleting the compressed file in the disk;
if the data is hot data, the data is directly deleted from the database.
7. The method of claim 1, wherein:
the set time period is a time period of a preset time length from 0 point every day.
8. A data processing apparatus, comprising:
the judging module is used for judging whether the data in the database is cold data or hot data according to a set threshold value;
the first storage module is used for storing the key query field of the cold data in a database if the cold data is the cold data, compressing the service data of the cold data into a file, storing the file in a disk and deleting the file from the database;
and the second storage module is used for continuously storing the key query fields and the service data of the hot data in the database if the hot data are hot data.
9. An electronic device, characterized in that the electronic device comprises:
a processor;
a memory having stored thereon computer readable instructions which, when executed by the processor, implement the method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that it stores computer program instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.
CN202210625497.0A 2022-06-02 2022-06-02 Data processing method, device, equipment and storage medium Pending CN115080605A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210625497.0A CN115080605A (en) 2022-06-02 2022-06-02 Data processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210625497.0A CN115080605A (en) 2022-06-02 2022-06-02 Data processing method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN115080605A true CN115080605A (en) 2022-09-20

Family

ID=83248900

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210625497.0A Pending CN115080605A (en) 2022-06-02 2022-06-02 Data processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115080605A (en)

Similar Documents

Publication Publication Date Title
US9811577B2 (en) Asynchronous data replication using an external buffer table
CN108319654B (en) Computing system, cold and hot data separation method and device, and computer readable storage medium
CN111949710B (en) Data storage method, device, server and storage medium
CN110825694A (en) Data processing method, device, equipment and storage medium
CN111209257B (en) File system fragmentation method and device
US10929347B2 (en) Defragmenting files having file blocks in multiple point-in-time copies
CN114443722A (en) Cache management method and device, storage medium and electronic equipment
CN106156038B (en) Date storage method and device
US10430115B2 (en) System and method for optimizing multiple packaging operations in a storage system
US10311026B2 (en) Compressed data layout for optimizing data transactions
US6510499B1 (en) Method, apparatus, and article of manufacture for providing access to data stored in compressed files
US10387267B2 (en) Fast copy using file system block mappings
US12118245B2 (en) File system improvements for zoned storage device operations
CN116610636A (en) Data processing method and device of file system, electronic equipment and storage medium
CN112394876A (en) Large file storage/reading method, storage/reading device and computer equipment
CN115080605A (en) Data processing method, device, equipment and storage medium
US7949632B2 (en) Database-rearranging program, database-rearranging method, and database-rearranging apparatus
CN111221817B (en) Service information data storage method, device, computer equipment and storage medium
CN108984719B (en) Data deleting method and device based on column storage, server and storage medium
US9256626B2 (en) Maintaining multiple copy versions of component values in a system
CN111782588A (en) File reading method, device, equipment and medium
CN112084141A (en) Full-text retrieval system capacity expansion method, device, equipment and medium
WO2024108668A1 (en) Pure-columnar update method and apparatus for htap and mass data
US11567671B2 (en) Method, electronic device, and computer program product for storage management
US20230385240A1 (en) Optimizations for data deduplication operations

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination