CN116185301A - Data storage processing method, platform, server, system, medium and product - Google Patents

Data storage processing method, platform, server, system, medium and product Download PDF

Info

Publication number
CN116185301A
CN116185301A CN202310225321.0A CN202310225321A CN116185301A CN 116185301 A CN116185301 A CN 116185301A CN 202310225321 A CN202310225321 A CN 202310225321A CN 116185301 A CN116185301 A CN 116185301A
Authority
CN
China
Prior art keywords
data
storage medium
performance storage
stored
low
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310225321.0A
Other languages
Chinese (zh)
Inventor
洪毅清
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
CCB Finetech Co Ltd
Original Assignee
China Construction Bank Corp
CCB Finetech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp, CCB Finetech Co Ltd filed Critical China Construction Bank Corp
Priority to CN202310225321.0A priority Critical patent/CN116185301A/en
Publication of CN116185301A publication Critical patent/CN116185301A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G06F3/0611—Improving I/O performance in relation to response time
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The application provides a data storage processing method, a platform, a server, a system, a medium and a product, and relates to the technical field of data storage, wherein the method comprises the following steps: monitoring and obtaining the use frequency of data in a data table or a data table partition in a Hive database; if the frequency of use of the data has periodicity and the data is currently stored in a low-performance storage medium, the data is transferred to a high-performance storage medium before periodic operation so as to perform periodic operation in the high-performance storage medium. According to the method, before periodic operation, the data can be automatically transferred to a high-performance storage medium, so that the operation efficiency is improved, the data reading and writing speed is improved, the automatic transfer of the data is realized, and the heat scene and the storage cost can be effectively balanced.

Description

Data storage processing method, platform, server, system, medium and product
Technical Field
The present disclosure relates to the field of data storage technologies, and in particular, to a data storage processing method, a platform, a server, a system, a medium, and a product.
Background
Hive is an open source, data warehouse framework built on Hadoop. The Hive stores massive data, and the stored data can reach PB to 10PB level. In particular, hive will store data on high cost high performance storage media or on low cost low performance storage media.
However, if the data is stored on a high cost high performance storage medium, the data storage cost is high; if the data are stored on the low-cost and low-performance storage medium, the data storage cost is low, but the data can be reduced in reading and writing speed to influence the calculation efficiency and the current higher use heat is not matched with the continuous change of the data use scene and the use condition.
Therefore, there is a need for a storage method that can effectively balance the heat scene and the storage cost.
Disclosure of Invention
The application provides a data storage processing method, a platform, a server, a system, a medium and a product, which are used for solving the problem that the conventional data storage mode cannot effectively balance the use heat and the storage cost.
In a first aspect, the present application provides a data storage processing method, including:
monitoring and obtaining the use frequency of data in a data table or a data table partition in a Hive database;
if the frequency of use of the data has periodicity and the data is currently stored in a low-performance storage medium, the data is transferred to a high-performance storage medium before periodic operation so as to perform periodic operation in the high-performance storage medium.
In a second aspect, the present application provides a data storage processing platform, comprising:
the data use frequency monitoring unit is used for monitoring and acquiring the use frequency of the data in the data table or the data table partition in the Hive database;
and the automatic data transfer unit is used for transferring the data to a high-performance storage medium before the periodical operation so as to perform the periodical operation in the high-performance storage medium if the use frequency of the data has periodicity and the data is currently stored in the low-performance storage medium.
In a third aspect, the present application provides a server comprising: a processor, a memory, and a transceiver;
a processor, memory, and transceiver circuitry interconnect;
the memory stores computer-executable instructions;
a transceiver for transceiving data;
the processor executes computer-executable instructions stored in the memory to cause the processor to perform the method as described in the first aspect.
In a fourth aspect, the present application provides a data storage processing system, the processing system comprising: the system comprises a data storage processing platform, a Hive database, a high-performance storage medium and a low-performance storage medium;
the data storage processing platform is respectively in communication connection with the Hive database, the high-performance storage medium and the low-performance storage medium.
In a fifth aspect, the present application provides a computer-readable storage medium having stored therein computer-executable instructions for performing the method according to the first aspect when executed by a processor.
In a sixth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the method of the first aspect.
The data storage processing method, the platform, the server, the system, the medium and the product provided by the application monitor and acquire the use frequency of data in a data table or a data table partition in the Hive database; if the use frequency of the data has periodicity, and the data is currently stored in a low-performance storage medium, the data is transferred to a high-performance storage medium before periodic operation, so that the periodic operation is performed in the high-performance storage medium, the data can be automatically transferred to the high-performance storage medium before the periodic operation, the operation efficiency is improved, the data reading and writing speed is improved, the automatic transfer of the data in the storage media with different performances is realized, and the heat scene and the storage cost can be effectively balanced.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic diagram of a network architecture of a data storage processing method provided in the present application;
FIG. 2 is a schematic flow chart of a data storage processing method provided in the present application;
FIG. 3 is a flow chart of another method for data storage processing provided in the present application;
fig. 4 is a schematic structural diagram of a mail detecting device provided in the present application;
FIG. 5 is a first block diagram of an electronic device for implementing the data storage processing method of the present application;
fig. 6 is a second block diagram of an electronic device for implementing the data storage processing method of the present application.
Specific embodiments thereof have been shown by way of example in the drawings and will herein be described in more detail. These drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.
In the technical scheme of the application, the related information such as financial data or user data is collected, stored, used, processed, transmitted, provided, disclosed and the like, which accords with the regulations of related laws and regulations and does not violate the popular regulations of the public order.
Hive is an open source, data warehouse framework built on Hadoop. The Hive stores massive data, the stored data volume can reach PB to 10PB level, the use frequency of the data is related to the service attribute of the application, and the use conditions of the data of different applications are different. In particular, hive will store data on high cost high performance storage media or on low cost low performance storage media.
However, if the data is stored on a high-cost high-performance storage medium, such as a distributed file system HDFS, the data storage cost is high; if the data are stored on a low-cost and low-performance storage medium, such as object storage, the data storage cost is low, but the data can be reduced in reading and writing speed to affect the calculation efficiency and not match with the current higher use heat as the use scene and the use condition of the data are changed continuously. How to effectively balance the use of heat scenes and storage cost becomes a difficult problem of the current Hive.
Therefore, aiming at the problem that the data storage mode in the prior art cannot effectively balance the use heat and the storage cost, the inventor finds out in the research that the use frequency of the data in the data table or the data table partition in the Hive database is monitored and obtained; if the frequency of use of the data is periodic and the data is currently stored in the low-performance storage medium, the data is transferred to the high-performance storage medium before periodic operation, so that the periodic operation is performed in the high-performance storage medium, the data can be automatically transferred to the high-performance storage medium before the periodic operation, the operation efficiency is improved, the data reading and writing speed is improved, the automatic transfer of the data in the storage medium with different cost and performance is realized, and the heat scene and the storage cost can be effectively balanced.
The inventor proposes the technical solution of the embodiments of the present application based on the inventive findings described above. The following describes a network architecture and an application scenario of the data storage processing method provided in the embodiments of the present application.
As shown in fig. 1, a network architecture corresponding to a data storage processing method provided in an embodiment of the present application includes: a data storage processing platform 1, a hive database 2, a high-performance storage medium 3 and a low-performance storage medium 3. The data storage processing platform 1 is respectively in communication connection with the Hive database 2, the high-performance storage medium 3 and the low-performance storage medium 4, wherein the high-performance storage medium 3 can be arranged in the data storage processing platform 1, the data storage processing platform 1 comprises a data use frequency monitoring unit and a data automatic transfer unit, and the data storage processing platform 1 monitors and acquires the use frequency of data in a data table or a data table partition in the Hive database by adopting the data use frequency monitoring unit; the automatic data transfer unit of the data storage processing platform 1 is used for transferring data to the high-performance storage medium 3 before periodic operation if the use frequency of the data is periodic and the data is currently stored in the low-performance storage medium 4, so that the periodic operation is performed in the high-performance storage medium, the data can be automatically transferred to the high-performance storage medium before the periodic operation, the operation efficiency is improved, the data reading and writing speed is improved, the automatic transfer of the data in the storage media with different performances is realized, and the heat scene and the storage cost can be effectively balanced.
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
Fig. 2 is a flow chart of a data storage processing method provided in the present application, where the method is applied to a server. Wherein the server may be a digital computer representing various forms. Such as cellular telephones, smart phones, laptop computers, desktop computers, workstations, personal digital assistants, blade servers, mainframe computers, and other suitable computers. As shown in fig. 2, the method includes:
in step 201, the frequency of use of data in the data table or the data table partition in the Hive database is monitored and obtained.
Hive is a data warehouse tool based on Hadoop, and can map a structured data file into a database table.
Step 202, if the frequency of use of the data has periodicity, and the data is currently stored in the low-performance storage medium, the data is transferred to the high-performance storage medium before the periodic operation, so as to perform the periodic operation in the high-performance storage medium.
Alternatively, the low-performance storage medium may be an object storage, which is an object-oriented and massive internet storage, and may also be directly referred to as "cloud storage", where the object storage has the characteristics of remote access, wireless capacity expansion, low cost, and the like, and the object storage mainly uses a type of magnetic disk with a low IO speed and low cost, such as a high-density SATA disk.
Alternatively, the high-performance storage medium may be an HDFS, where HDFS (Hadoop Distributed File System) is a file system, and HDFS is an important component of the hadoop ecosystem and is a storage component in hadoop, and HDFS mainly uses a type of disk with a higher IO speed and a higher cost, such as an SSD disk and an SAS disk. Among them, a high-performance storage medium has a relatively high read/write speed and a relatively high cost compared to a low-performance storage medium.
In this embodiment, the frequency of use of the data is periodic, and the data with a fixed settlement date, for example, data with a definite periodicity in monthly settlement or data with periodicity in annual settlement, may be transferred to a high-performance storage medium before calculation, in order to improve the calculation efficiency of such periodic data.
Specifically, if the frequency of use of the data is periodic, and the data is currently stored in a low-performance storage medium, in order to improve the calculation efficiency of the data, the data is transferred to the high-performance storage medium before the periodic operation, so that the periodic operation is performed in the high-performance storage medium.
If the frequency of use of the data is periodic, and the data is currently stored in the high-performance storage medium, the data does not need to be transferred before the periodic operation is completed.
In this embodiment, the frequency of use of the data in the data table or the data table partition in the Hive database is monitored and obtained, if the frequency of use of the data has periodicity, and the data is currently stored in a low-performance storage medium, the data is transferred to a high-performance storage medium before periodic operation, so that the periodic operation is performed in the high-performance storage medium, the data can be automatically transferred to the high-performance storage medium before the periodic operation, the operation efficiency is improved, the data reading and writing speed is improved, the automatic transfer of the data in the storage media with different performances is realized, and the heat scene and the storage cost can be effectively balanced.
Fig. 3 is a flow chart of another data storage processing method provided in the present application, where the method is applied to a server, as shown in fig. 3, and the method includes:
step 301, monitoring and obtaining the use frequency of data in a data table or a data table partition in the Hive database.
In one possible implementation, monitoring the frequency of use of acquiring data in a data table or a data partition in a Hive database includes:
and if the current time is the data transfer time, running a preset script to monitor and acquire the use frequency of the data in the data table or the data partition in the Hive database.
In this embodiment, whether the current time is the data transfer time is monitored, where the data transfer time may be set based on actual conditions, for example, 00:01 minutes per day is set as the data transfer time, if the current time is the data transfer time, a preset script is obtained, and the preset script is run, so that the frequency of use of the data in the data table or the data partition in the Hive database is monitored and obtained.
Optionally, the method further comprises:
inquiring a frequency table in the Hive database, and acquiring the use frequency of data in a data table or a data table partition; wherein the frequency table includes: the identification of each data table in the Hive database, the name of the data table and the time of each access to the data in the data table; alternatively, the identification of each data table in the Hive database, the data table name, the data table partition identification, the data table partition name, and the time at which the data of the data table partition under the data table is accessed each time.
In this embodiment, a frequency table in the Hive database is queried, and the frequency of use of data in the data table or the data table partition is obtained, where the frequency table includes each data table identifier in the Hive database and the time of accessing the data in the data table each time. The data table may include data table partitions, so the frequency table also includes an identification of each data table in the Hive database, a data table name, a data table partition identification, a data table partition name, and a time for each access to the data in the data table partition under the data table.
Specifically, a universal trigger is created, whether the last access time of the data table and the data table partition changes when new inserted data or updated data exists is monitored by the trigger, and if the last access time of certain data in the data table is monitored to change, the data identification of the data and the time of the data access are recorded in the frequency table; if the last access time of certain data in the data table partition is monitored, the identification of the data table, the name of the data table, the identification of the data table partition, the name of the data table partition and the time of accessing the data of the data table partition under the data table each time are recorded in the frequency table.
In step 302, if the frequency of use of the data has periodicity, and the data is currently stored in the low-performance storage medium, the data is transferred to the high-performance storage medium before the periodic operation, so as to perform the periodic operation in the high-performance storage medium.
In this embodiment, the step 302 and the step 202 have the same technical features, and the detailed description will refer to the step 202 and will not be repeated here.
If the frequency of use of the data does not have periodicity, step 303, the data stored in the high-performance storage medium is transferred to the low-performance storage medium, or the data stored in the low-performance storage medium is transferred to the high-performance storage medium according to the frequency of use of the data and the preset frequency of use corresponding to the data.
In this embodiment, the frequency of use of the data does not show periodicity, and there is no class of data with fixed settlement date, and further, the preset frequency of use corresponding to the data is obtained, and the comparison result of the frequency of use of the data and the preset frequency of use corresponding to the data is used to determine whether to transfer the data, specifically, transfer the data stored in the high-performance storage medium to the low-performance storage medium, or transfer the data stored in the low-performance storage medium to the high-performance storage medium.
In one possible implementation manner, according to the frequency of use of the data and the preset frequency of use corresponding to the data, the method includes the steps of:
if the use frequency of the data is smaller than or equal to the corresponding preset use frequency and the data is currently stored in the high-performance storage medium, the data is transferred to the low-performance storage medium;
or alternatively, the process may be performed,
and if the use frequency of the data is larger than the corresponding preset use frequency and the data is currently stored in the storage medium with low performance, the data is transferred to the storage medium with high performance.
In this embodiment, the preset usage frequencies corresponding to different data may be the same or different, and may be configured in advance by the user, where the usage frequency of the data is less than or equal to the corresponding preset usage frequency, which indicates that the usage rate of the user is not high, for example, the preset usage frequency of certain data is set to be 3 times on 3 days continuously, and if the usage frequency of the data is set to be 1 time on 3 days continuously, the usage frequency of the data is less than the corresponding preset usage frequency, which indicates that the data is not accessed very often, and the data is currently stored in a high-performance storage medium, and is transferred to a low-performance storage medium, so that the data can be automatically transferred to the low-performance storage medium in a service low-peak period. Or, for example, the preset use frequency of a certain data is M times of continuous X days, and the preset use frequency can be configured according to actual situations.
It should be noted that, if the frequency of use of the data is less than or equal to the corresponding preset frequency of use, and the data is currently stored in the low-performance storage medium, no data transfer processing is required at this time.
Optionally, the transferring the data to a low performance storage medium includes:
and (3) adding a partition in the low-performance storage medium, and transferring the data to the added partition in the low-performance storage medium.
In this embodiment, a partition is newly added to a low-performance storage medium, and specifically, a script file is designed in advance to automatically construct a new partition, the script file: the source end is a high-performance storage medium such as an HDFS, and the target end is a low-performance storage medium such as an object storage, and a partition is newly added on the target end. The data is further transferred from the high-performance storage medium to the newly added partition in the low-performance storage medium, so that the automatic transfer of the data is realized, and the heat scene and the storage cost can be effectively balanced.
Or, the use frequency of the data is greater than the corresponding preset use frequency, which indicates that the use rate of the user is higher, for example, the preset use frequency of certain data is set to be 3 times in 3 days in average, if the use frequency of the data is greater than the corresponding preset use frequency of the data, which indicates that the data is frequently accessed, the data is currently stored in a low-performance storage medium, the data is transferred to a high-performance storage medium, and the data can be automatically transferred to the high-performance storage medium in a traffic peak period.
It should be noted that, if the frequency of use of the data is greater than the corresponding preset frequency of use, and the data is currently stored in the high-performance storage medium, no data transfer processing is required at this time.
Optionally, the transferring the data to the high performance storage medium includes:
and (3) adding a partition in the high-performance storage medium, and transferring the data to the added partition in the high-performance storage medium.
In this embodiment, a partition is newly added to a high-performance storage medium, and specifically, a script file is designed in advance to automatically construct a new partition, the script file: the source end is a low-performance storage medium such as object storage, and the target end is a high-performance storage medium such as HDFS, a partition is newly added on the target end. The data is transferred from the low-performance storage medium to the newly added partition in the high-performance storage medium, so that the automatic transfer of the data is realized, and the heat scene and the storage cost can be effectively balanced.
Optionally, the data storage processing method further includes:
if the newly added data to be stored in the Hive database is monitored and the corresponding appointed storage path exists in the newly added data to be stored, the newly added data to be stored is stored in a storage medium corresponding to the appointed path;
or alternatively, the process may be performed,
if the newly added data to be stored in the Hive database is monitored, and the corresponding designated storage path does not exist in the newly added data to be stored, the newly added data to be stored is stored in a high-performance storage medium.
In this embodiment, whether newly added data to be stored exists in the Hive database is monitored, if it is monitored that newly added data to be stored exists in the Hive database and corresponding designated storage paths exist in the newly added data to be stored, the designated storage paths are preset storage paths by a user, the newly added data to be stored are stored in storage media corresponding to the designated paths, if the user uses certain data every day, the designated storage paths are high-performance storage media, and the newly added data to be stored are stored in the high-performance storage media.
Or if it is detected that the new data to be stored exists in the Hiv database and the new data to be stored does not exist in the corresponding designated storage path, the user may use the new data to store the new data in a default storage medium, where the default storage medium may be a high-performance storage medium, and store the new data to be stored in the high-performance storage medium.
Optionally, the data storage processing method further includes:
and if the data is monitored to finish the periodical operation, the data is transferred to a storage medium with low performance.
In this embodiment, if the frequency of use of the data is periodic, and the data is currently stored in a low-performance storage medium, in order to improve the calculation efficiency of the data, the data is transferred to the high-performance storage medium before the periodic operation, so that the periodic operation is performed in the high-performance storage medium. If the data is monitored to complete the periodic operation, the use frequency of the data is periodic, and the data can be generally used only after the next period, so that the data can be transferred from a high-performance storage medium to a low-performance storage medium.
In the embodiment, the heat scene and the storage cost can be effectively balanced and used through automatic data transfer between the high-performance storage medium and the low-performance storage medium.
Fig. 4 is a schematic structural diagram of a data storage processing platform provided in the present application, and as shown in fig. 4, a data storage processing platform 400 provided in this embodiment includes a data frequency monitoring unit 401 and a data automatic transfer unit 402.
The data usage frequency monitoring unit 401 is configured to monitor and obtain a usage frequency of data in a data table or a data table partition in the Hive database. The automatic data transfer unit 402 is configured to transfer the data to the high-performance storage medium before the periodic operation if the frequency of use of the data is periodic and the data is currently stored in the low-performance storage medium, so as to perform the periodic operation in the high-performance storage medium.
Optionally, the data storage processing platform further comprises: and a processing unit.
The processing unit queries a frequency table in the Hive database and obtains the use frequency of data in the data table or a data table partition; wherein the frequency table includes: the identification of each data table in the Hive database, the name of the data table and the time of each access to the data in the data table; alternatively, the identification of each data table in the Hive database, the data table name, the data table partition identification, the data table partition name, and the time at which the data of the data table partition under the data table is accessed each time.
Optionally, the automatic data transfer unit is further configured to transfer the data stored in the high-performance storage medium to the low-performance storage medium or transfer the data stored in the low-performance storage medium to the high-performance storage medium according to the use frequency of the data and the preset use frequency corresponding to the data if the use frequency of the data does not have periodicity.
Optionally, the automatic data transfer unit is further configured to transfer the data to a low-performance storage medium if the frequency of use of the data is less than or equal to a corresponding preset frequency of use and the data is currently stored in the high-performance storage medium; or if the use frequency of the data is greater than the corresponding preset use frequency and the data is currently stored in the storage medium with low performance, the data is transferred to the storage medium with high performance.
Optionally, the processing unit is further configured to add a partition to the low-performance storage medium, and transfer the data to the added partition in the low-performance storage medium; and the processing unit is also used for adding a partition in the high-performance storage medium and transferring the data to the added partition in the high-performance storage medium.
Optionally, the automatic data transfer unit is further configured to store the newly added data to be stored in a storage medium corresponding to the designated path if it is detected that the newly added data to be stored exists in the Hive database and the newly added data to be stored exists in the corresponding designated storage path; the automatic data transfer unit is further configured to store the newly added data to be stored in the high-performance storage medium if it is detected that the newly added data to be stored exists in the Hive database and no corresponding designated storage path exists in the newly added data to be stored.
Optionally, the automatic data transfer unit is further configured to transfer the data to a low-performance storage medium if the data is monitored to complete the periodic operation.
Optionally, the data use frequency monitoring unit is further configured to run a preset script to monitor and obtain a use frequency of the data in the data table or the data partition in the Hive database if the current time is monitored as the data transfer time.
Fig. 5 is a first block diagram of a server for implementing the data storage processing method of the present application, and as shown in fig. 5, the server 500 includes: a memory 501, a processor 502 and a transceiver 503.
Processor 502, memory 501 and transceiver 503 are electrically interconnected;
a transceiver 503 for transmitting and receiving data;
memory 501 stores computer-executable instructions;
processor 502 executes computer-executable instructions stored in memory 501, causing processor 502 to perform the methods provided by any of the embodiments described above.
Fig. 6 is a second block diagram of a server for implementing the data storage processing method of the present application, as shown in fig. 6.
Server 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and a communication component 816.
The processing component 802 generally controls overall operation of the server 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interactions between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.
The memory 804 is configured to store various types of data to support operations at the server 800. Examples of such data include instructions for any application or method operating on server 800, contact data, phonebook data, messages, pictures, video, and the like. The memory 804 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
The power supply component 806 provides power to the various components of the server 800. The power components 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the server 800.
The multimedia component 808 includes a screen between the server 800 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or sliding action, but also the duration and pressure associated with the touch or sliding operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the server 800 is in an operation mode, such as a photographing mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the server 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 further includes a speaker for outputting audio signals.
The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.
The sensor assembly 814 includes one or more sensors for providing status assessment of various aspects for the server 800. For example, the sensor component 814 may detect an on/off state of the server 800, a relative positioning of components, such as a display and keypad of the server 800, the sensor component 814 may also detect a change in position of the server 800 or a component of the server 800, the presence or absence of a user's contact with the server 800, an orientation or acceleration/deceleration of the server 800, and a change in temperature of the server 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 816 is configured to facilitate communication between the server 800 and other devices, either wired or wireless. The server 800 may access a wireless network based on a communication standard, such as WiFi,2G, or 3G, or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the server 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.
In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 804 including instructions executable by processor 820 of server 800 to perform the above-described method. For example, the non-transitory computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
Optionally, there is also provided a data storage processing system, the processing system comprising: the system comprises a data storage processing platform, a Hive database, a high-performance storage medium and a low-performance storage medium;
the data storage processing platform is respectively in communication connection with the Hive database, the high-performance storage medium and the low-performance storage medium.
In an exemplary embodiment, there is also provided a computer-readable storage medium having stored therein computer-executable instructions for performing the method of any one of the above embodiments by a processor.
In an exemplary embodiment, a computer program product is also provided, comprising a computer program for executing the method of any of the above embodiments by a processor.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (13)

1. A data storage processing method, the method comprising:
monitoring and obtaining the use frequency of data in a data table or a data table partition in a Hive database;
if the frequency of use of the data has periodicity and the data is currently stored in a low-performance storage medium, the data is transferred to a high-performance storage medium before periodic operation so as to perform periodic operation in the high-performance storage medium.
2. The method as recited in claim 1, further comprising:
inquiring a frequency table in the Hive database, and acquiring the use frequency of data in the data table or a data table partition; wherein the frequency table includes: the identification of each data table in the Hive database, the name of the data table and the time of accessing the data in the data table each time; or, the identification of each data table, the name of the data table, the identification of the partition of the data table, the name of the partition of the data table and the time of accessing the data of the partition of the data table under the data table in the Hive database.
3. The method as recited in claim 1, further comprising:
and if the use frequency of the data does not have periodicity, the data stored in the high-performance storage medium is transferred to the low-performance storage medium or the data stored in the low-performance storage medium is transferred to the high-performance storage medium according to the use frequency of the data and the preset use frequency corresponding to the data.
4. A method according to claim 3, wherein the transferring the data stored in the high-performance storage medium to the low-performance storage medium or transferring the data stored in the low-performance storage medium to the high-performance storage medium according to the frequency of use of the data and the preset frequency of use corresponding to the data comprises:
if the use frequency of the data is smaller than or equal to the corresponding preset use frequency and the data is currently stored in a high-performance storage medium, the data is transferred to a low-performance storage medium;
or alternatively, the process may be performed,
and if the use frequency of the data is larger than the corresponding preset use frequency and the data is currently stored in the low-performance storage medium, the data is transferred to the high-performance storage medium.
5. The method of claim 4, wherein the transferring the data to a low performance storage medium comprises:
adding a partition in the low-performance storage medium, and transferring the data to the newly added partition in the low-performance storage medium;
or alternatively, the process may be performed,
the transferring the data to a high performance storage medium comprises:
and adding a partition in the high-performance storage medium, and transferring the data to the added partition in the high-performance storage medium.
6. The method as recited in claim 1, further comprising:
if the fact that newly added data to be stored exist in the Hive database and the newly added data to be stored exist corresponding appointed storage paths is monitored, the newly added data to be stored are stored in storage media corresponding to the appointed paths;
or alternatively, the process may be performed,
and if the fact that the newly added data to be stored exist in the Hive database and the corresponding appointed storage path does not exist in the newly added data to be stored is monitored, the newly added data to be stored are stored in a high-performance storage medium.
7. The method according to any one of claims 1 to 6, further comprising:
and if the data is monitored to finish the periodical operation, the data is transferred to a storage medium with low performance.
8. The method of claim 1, wherein the monitoring obtains a frequency of use of data in a data table or a data partition in the Hive database, comprising:
and if the current time is the data transfer time, running a preset script to monitor and acquire the use frequency of the data in the data table or the data partition in the Hive database.
9. A data storage processing platform, the platform comprising:
the data use frequency monitoring unit is used for monitoring and acquiring the use frequency of the data in the data table or the data table partition in the Hive database;
and the automatic data transfer unit is used for transferring the data to a high-performance storage medium before the periodical operation so as to perform the periodical operation in the high-performance storage medium if the use frequency of the data has periodicity and the data is currently stored in the low-performance storage medium.
10. A server, comprising: a processor, a memory, and a transceiver;
a processor, memory, and transceiver circuitry interconnect;
a transceiver for transceiving data;
the memory stores computer-executable instructions;
the processor executing computer-executable instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 8.
11. A data storage processing system, the processing system comprising: the system comprises a data storage processing platform, a Hive database, a high-performance storage medium and a low-performance storage medium;
the data storage processing platform is respectively in communication connection with the Hive database, the high-performance storage medium and the low-performance storage medium.
12. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to carry out the method of any one of claims 1 to 8.
13. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 8.
CN202310225321.0A 2023-03-01 2023-03-01 Data storage processing method, platform, server, system, medium and product Pending CN116185301A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310225321.0A CN116185301A (en) 2023-03-01 2023-03-01 Data storage processing method, platform, server, system, medium and product

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310225321.0A CN116185301A (en) 2023-03-01 2023-03-01 Data storage processing method, platform, server, system, medium and product

Publications (1)

Publication Number Publication Date
CN116185301A true CN116185301A (en) 2023-05-30

Family

ID=86444317

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310225321.0A Pending CN116185301A (en) 2023-03-01 2023-03-01 Data storage processing method, platform, server, system, medium and product

Country Status (1)

Country Link
CN (1) CN116185301A (en)

Similar Documents

Publication Publication Date Title
US9942690B2 (en) Method and device for information push
JP6062608B2 (en) Web page access method, apparatus, server, terminal, program, and recording medium
CN106201734B (en) File sharing method and device
CN104462296B (en) File management method and device and terminal
CN109388625B (en) Method and device for processing configuration file in multi-distributed file system
US10324910B2 (en) Contact record processing method and apparatus
CN110826697B (en) Method and device for acquiring sample, electronic equipment and storage medium
CN107463419B (en) Application restarting method and device and computer readable storage medium
CN110213062B (en) Method and device for processing message
CN111767006B (en) Data processing method and device
CN109522286B (en) Processing method and device of file system
CN106851418B (en) Video recommendation method and device
CN116185301A (en) Data storage processing method, platform, server, system, medium and product
CN111290882B (en) Data file backup method, data file backup device and electronic equipment
CN108509641B (en) File backup method, device, server and system
CN112988822A (en) Data query method, device, equipment, readable storage medium and product
CN113378022A (en) In-station search platform, search method and related device
CN112182027B (en) Information query method, device, electronic equipment and storage medium
CN111241097B (en) Method for processing object, device for processing object and storage medium
CN114238728B (en) Vehicle data processing method, device and equipment
CN111767249B (en) Method and device for determining self-running time of function
CN115225716B (en) Data processing method and device and electronic equipment
CN110311968B (en) Method and device for loading file in streaming mode and intelligent equipment
CN111625536B (en) Data access method and device
US20220174364A1 (en) Method and device for information processing

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination