WO2022257615A1

WO2022257615A1 - Information processing method and apparatus, and storage medium

Info

Publication number: WO2022257615A1
Application number: PCT/CN2022/088113
Authority: WO
Inventors: 郑思城; 吴怡燃; 王英杰
Original assignee: 北京沃东天骏信息技术有限公司; 北京京东世纪贸易有限公司
Priority date: 2021-06-09
Filing date: 2022-04-21
Publication date: 2022-12-15
Also published as: CN113391764A

Abstract

Embodiments of the present application disclose an information processing method and apparatus, and a storage medium. The method comprises: when it is detected that the remaining storage capacity in a memory is less than or equal to a preset capacity lower limit threshold, detecting a data access frequency corresponding to storage data in the memory (S101); dividing the storage data into at least one type of data on the basis of the data access frequency, each of the at least one type of data corresponding to one storage category (S102); and determining, on the basis of at least one storage category corresponding to the at least one type of data, at least one storage system corresponding to the at least one type of data, and respectively transferring the at least one type of data to the at least one storage system, wherein each type of data corresponds to one storage system (S103).

Description

An information processing method and device, storage medium

Cross References to Related Applications

This application claims the priority of the Chinese patent application with the application number 202110642322.6 and the application title "An information processing method and device, storage medium" submitted to the China Patent Office on June 9, 2021, the entire contents of which are incorporated by reference in In this application.

technical field

The present application relates to the technical field of information processing, and in particular to an information processing method and device, and a storage medium.

Background technique

With the development of Internet technology, there are more and more data on the network, and the amount of data that needs to be stored is gradually increasing, and many problems in data storage have also arisen.

In the prior art, two sets of clusters are used: a primary cluster with high storage performance (such as solid state disk (Solid State Disk, SSD)) and a standby cluster with low storage performance (such as hard disk drive (Hard Disk Drive, HDD) )). Store the data in the primary cluster first, and then manually copy the data from the primary cluster to the standby cluster for storage when it is manually determined that the heat of the data is reduced. Storage location, as such, reduces intelligence in storing data.

Contents of the invention

In order to solve the above technical problems, the embodiments of the present application expect to provide an information processing method and device, and a storage medium, which can improve the intelligence when storing stored data.

The technical scheme of the present application is realized like this:

An embodiment of the present application provides an information processing method, including:

When it is detected that the remaining storage capacity in the internal memory is less than or equal to the preset capacity lower limit threshold, detecting the data access frequency corresponding to the stored data in the internal memory;

Divide the stored data into at least one type of data based on the data access frequency; each type of data in the at least one type of data corresponds to a storage category;

Based on at least one storage category corresponding to the at least one type of data, determine at least one storage system corresponding to the at least one type of data, and transfer the at least one type of data to the at least one storage system respectively; wherein, each A type of data corresponds to a storage system.

An embodiment of the present application provides an information processing device, and the device includes:

The detecting part is configured to detect the data access frequency corresponding to the stored data in the internal memory when it is detected that the remaining storage capacity in the internal memory is less than or equal to the preset capacity lower limit threshold;

The dividing part is configured to divide the stored data into at least one type of data based on the data access frequency; each type of data in the at least one type of data corresponds to a storage category;

The determining part is configured to determine at least one storage system corresponding to the at least one type of data based on at least one storage class corresponding to the at least one type of data;

The dumping part is configured to dump the at least one type of data to the at least one storage system respectively; wherein, each type of data corresponds to a storage system.

A memory, a processor, and a communication bus, the memory communicates with the processor through the communication bus, the memory stores an information processing program executable by the processor, and when the information processing program is executed , executing the information processing method described above by the processor.

An embodiment of the present application provides a storage medium on which a computer program is stored and applied to an information processing device. When the computer program is executed by a processor, the information processing method described above is implemented.

An embodiment of the present application provides an information processing method and device, and a storage medium. The information processing method includes: detecting that the stored data in the memory corresponds to data access frequency; based on the data access frequency, divide the stored data into at least one type of data; each type of data in at least one type of data corresponds to a storage category; based on at least one storage category corresponding to at least one type of data, determine at least one At least one storage system corresponding to the type of data, and at least one type of data is transferred to at least one storage system; wherein, each type of data corresponds to a storage system. By adopting the implementation scheme of the above method, the information processing device detects the data access frequency corresponding to the stored data in the memory, so that the information processing device can divide the stored data into at least one type of data based on the data access frequency, based on at least one type of data corresponding to at least one type of data A storage category, to determine at least one storage system corresponding to at least one type of data, so as to transfer at least one type of data to at least one storage system; wherein, each type of data corresponds to a storage system, and there is no need to manually determine the storage data The heat of the data and the corresponding storage location do not need to manually dump the stored data, which improves the intelligence when storing the stored data.

Description of drawings

FIG. 1 is a flow chart of an information processing method provided by an embodiment of the present application;

Fig. 2 is a schematic diagram 1 of an exemplary information processing device provided by an embodiment of the present application;

Fig. 3 is an exemplary architecture diagram 2 of an information processing device provided by an embodiment of the present application;

FIG. 4 is a schematic diagram 3 of an exemplary information processing device architecture provided by an embodiment of the present application;

FIG. 5 is a schematic diagram 4 of an exemplary information processing device architecture provided by an embodiment of the present application;

FIG. 6 is a first structural schematic diagram of an information processing device provided by an embodiment of the present application;

FIG. 7 is a second schematic diagram of the composition and structure of an information processing device provided by an embodiment of the present application.

Detailed ways

In order to understand the characteristics and technical contents of the embodiments of the present application in more detail, the implementation of the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. The attached drawings are only for reference and description, and are not intended to limit the embodiments of the present application.

Embodiment one

The embodiment of the present application provides an information processing method. FIG. 1 is a flow chart of an information processing method provided in the embodiment of the present application. As shown in FIG. 1, the information processing method may include:

S101. When it is detected that the remaining storage capacity in the internal memory is less than or equal to a preset capacity lower limit threshold, detect the data access frequency corresponding to the stored data in the internal memory.

An information processing method provided in an embodiment of the present application is applicable to a scenario of dumping stored data.

In the embodiment of the present application, the information processing apparatus may be implemented in various forms. For example, the information processing devices described in this application may include mobile phones, cameras, tablet computers, notebook computers, palmtop computers, personal digital assistants (Personal Digital Assistant, PDA), portable media players (Portable Media Player, PMP), Devices such as navigation devices, wearable devices, smart bracelets, pedometers, and devices such as digital TVs, desktop computers, servers, etc.

In the embodiment of the present application, the stored data may be log data, order data of the product or other data information of the product, which can be determined according to the actual situation, which is not limited in the embodiment of the present application.

In the embodiment of the present application, the amount of stored data may be determined according to actual conditions, which is not limited in the embodiment of the present application.

In the embodiment of the present application, the quantity of stored data corresponds to the quantity of data access frequencies one by one, that is, one stored data corresponds to one data access frequency. The number of data access frequencies can be one, the number of data access frequencies can also be two, and the number of data access frequencies can also be multiple. The specific number of data access frequencies can be determined according to the actual situation. This is not limited.

In the embodiment of the present application, when the information processing device receives the stored data, the information processing device first stores the stored data in the internal memory, and then the information processing device will store the stored data when the remaining storage capacity of the internal memory is less than the lower limit of the preset capacity. In the case of a threshold, the data access frequency corresponding to the stored data in the memory is detected.

It should be noted that the information processing device can detect the data access frequency corresponding to the stored data in the internal memory when the remaining storage capacity of the internal memory is less than the preset lower limit threshold; The device detects the data access frequency corresponding to the stored data in the internal memory; the specific condition for the information processing device to detect the data access frequency corresponding to the stored data in the internal memory can be determined according to the actual situation, which is not limited in this embodiment of the present application.

It should also be noted that the preset capacity lower limit threshold can be the threshold configured in the information processing device; the preset capacity lower limit threshold can also be the threshold obtained by the information processing device before the information processing device receives the stored data; it can also be the information processing device The threshold value acquired by the processing device in other ways; specifically, it may be determined according to the actual situation, which is not limited in this embodiment of the present application.

Exemplarily, the preset capacity lower limit threshold may be 0, that is, the memory is full.

It should also be noted that the preset time period may be the time period configured in the information processing device; the preset time period may also be the time period obtained by the information processing device before the information processing device receives the stored data; The time period acquired by the processing device in other ways; the specific time period may be determined according to the actual situation, which is not limited in this embodiment of the present application.

In this embodiment of the application, the data access frequency may be the access frequency range that identifies the stored data category. For example, if the data category of the stored data is a hot data category, the data access frequency may be greater than or equal to the preset access frequency threshold frequency band; if the data category of the stored data is cold data category, the data access frequency may be a frequency band less than the preset access frequency threshold.

It should be noted that the preset access frequency threshold can be the frequency threshold configured in the information processing device, or it can be the frequency threshold obtained by the information processing device in other ways, which can be determined according to the actual situation. There is no limit to this.

In this embodiment of the application, when the information processing device detects that the remaining storage capacity in the internal memory is less than or equal to the preset capacity lower limit threshold, before detecting the data access frequency corresponding to the stored data in the internal memory, the information processing device will also receive The data identification corresponding to the stored data; correspondingly, at least one type of data includes the first data and/or the second data; the process of the information processing device detecting the data access frequency corresponding to the stored data in the internal memory includes: the information processing device detects from the data identification Filter out the hot data identifiers from the hot data identifiers, and use the hot data identifiers to identify the first access frequency corresponding to the first data; and/or, the information processing device filters out the cold data identifiers from the data identifiers, and uses the cold data identifiers to identify the second data corresponding to The second access frequency; the information processing device uses the first access frequency and/or the second access frequency as the data access frequency.

In the embodiment of the present application, the data identification includes hot data identification and cold data identification. The hot data identifier is used to identify the first data as a hot data category, and the cold data identifier is used to identify the second data as a cold data category.

In this embodiment of the present application, the first access frequency may be a part of the data access frequencies, and the first access frequency may also be the data access frequency. The second access frequency may be a part of the data access frequencies, and the second access frequency may also be a data access frequency. Wherein, if the first access frequency is a part of the data access frequency, and the second access frequency is also a part of the data access frequency, then the first access frequency and the second access frequency constitute the data access frequency.

It should be noted that the first access frequency may be a frequency segment greater than or equal to a preset access frequency threshold; the second access frequency may be a frequency segment less than the preset access frequency threshold.

It should be noted that, when the information processing device determines that the data identifier of the first data in the stored data is a hot data identifier, the information processing device uses the hot data identifier as the data identifier of the first data; When it is determined that the data identifier of the second data in the stored data is a cold data identifier, the information processing device uses the cold data identifier as the data identifier of the second data.

In this embodiment of the present application, the first access frequency may be the total number of times the first data is accessed within a period of time, and the first access frequency may also be the number of times the first data is accessed per second. The second access frequency may be the total access times of the second data within a period of time, or the second access frequency may be the access times of the second data per second.

It should be noted that the period of time can be 3 months, 6 months, 1 year, etc., and the specific period of time can be determined according to the actual situation, which is not limited in the embodiment of the present application.

Exemplarily, if the first access frequency may be the total access times of the first data within a period of time, then the first access frequency may be the total access times of the first data within 6 months (for example, 1000qps). If the first access frequency is the number of times the first data is accessed per second, the first access frequency may be 1000qps/sec (one thousand requests per second); or the first access frequency may be 1000qps/sec and may last for 5 minutes. If the second access frequency can be the total access times of the second data within a period of time, then the second access frequency can be the total access times of the second data within 6 months (for example, 1000qps). If the second access frequency is the number of times the second data is accessed per second, the second access frequency may be 1000qps/sec (one thousand requests per second); or the second access frequency may be 1000qps/sec and last for 5 minutes.

In this embodiment of the application, the information processing device may directly receive the data identifier transmitted by the user; the information processing device may also create a hot and cold table according to the instruction transmitted by the user, and determine the data identifier corresponding to the stored data from the hot and cold table.

It should be noted that, if the information processing device creates the hot and cold tables according to the instructions transmitted by the user, the way for the information processing device to create the hot and cold tables can be as follows:

create'cold_table',{NAME=>'f1',DATA_STORAGE=>'CFS'};

create'hot_table',{NAME=>'f1'} or create'hot_table',{NAME=>'f1',DATA_STORAGE=>'HDFS'}.

S102. Based on the data access frequency, divide the stored data into at least one type of data; each type of data in the at least one type of data corresponds to a storage category.

In the embodiment of the present application, after the information processing device detects the data access frequency corresponding to the stored data in the internal memory, the information processing device may divide the stored data into at least one type of data based on the data access frequency.

In this embodiment of the present application, at least one type of data includes first data of a hot data type and/or second data of a cold data type.

In the embodiment of the present application, the information processing device can be a distributed non-relational (NoSQL) system, and the HDFS address of the distributed file system (Hadoop Distribution FileSystem, HDFS) and the cloud file system (Cloud File System) are stored in the distributed NoSQL system. Service, CFS) CFS address, the distributed NoSQL system can load the HDFS system to the distributed NoSQL system through the HDFS address, and load the CFS system to the distributed NoSQL system through the CFS address, that is, the distributed NoSQL system is set with HDFS system and the CFS system. When the distributed NoSQL system starts, it automatically loads the HDFS system and the CFS system.

In the embodiment of the present application, a management node (HMaster) is set in the distributed NoSQL system, which is responsible for table management (addition, deletion, modification, query), region management, file system initialization, and so on. The distributed NoSQL system is also equipped with a sorting module (Compact), which is used to monitor the access frequency of files and the formulation of rules. The life cycle of data meets the rules set by the business to realize the flow of data (periodic inspection analysis files, statistical files frequency of access, checking for compliance with rules, automating data migration).

It should be noted that HDFS system and CFS sting are used to store different types of storage data respectively. Among them, the HDFS system uses full SSD disks, and the CFS system uses cloud disks.

It should be noted that the information processing device is provided with a file system interface (FileSystemInterface) for routing and forwarding, and the information processing device can use the file system interface to select a storage system (HDFS system or CFS system) that matches the stored data identifier, thereby Transfer storage data to HDFS system or CFS system.

In this embodiment of the application, the storage category includes a hot data storage category and a cold data storage category, where the storage system corresponding to the hot data storage category is a high-frequency access storage system, and the storage system corresponding to the cold data storage category may be a low-frequency access storage system. system. It should be noted that the high-frequency access storage system may be an HDFS system, and the low-frequency access storage system may be a CFS system.

In the embodiment of the present application, each type of data in at least one type of data corresponds to a storage type, including: the first data of the hot data type corresponds to the hot data storage type, and the second data of the cold data type corresponds to the cold data storage type.

In the embodiment of the present application, if the information processing device receives the data identifier corresponding to the stored data, and the information processing device filters out the hot data identifier from the data identifier, and uses the hot data identifier to identify the first access frequency corresponding to the first data; And/or, the information processing device screens out the cold data identifier from the data identifier, and uses the cold data identifier to identify the second access frequency corresponding to the second data; using the first access frequency and/or the second access frequency as the data access frequency, Then, based on the data access frequency, the information processing device divides the stored data into at least one type of data. The information processing device can filter out the first access frequency corresponding to the hot data identifier from the data access frequency, and determine from the stored data The first data corresponding to the first access frequency; and/or, the information processing device screens out the second access frequency corresponding to the cold data identifier from the data access frequency, and determines the second data corresponding to the second access frequency from the stored data .

In the embodiment of the present application, the storage tag of the data is identified in the hot and cold table, and the information processing device may also determine the storage system of the data from the storage tag.

Exemplarily, the storage label may be: DATA_STORAGE=>'CFS' or DATA_STORAGE=>'HDFS'.

In the embodiment of the present application, the information processing device divides the stored data into at least one type of data based on the data access frequency. The information processing device can also filter out the access frequency greater than or equal to the preset access frequency threshold from the data access frequency. the first access frequency, and determine the first data corresponding to the first access frequency in the stored data; and/or, the information processing device screens out the second access frequency whose access frequency is less than the preset access frequency threshold from the data access frequencies , and determine the second data corresponding to the second access frequency in the stored data.

S103. Based on at least one storage category corresponding to at least one type of data, determine at least one storage system corresponding to at least one type of data, and transfer at least one type of data to at least one storage system; wherein, each type of data corresponds to a storage system system.

In this embodiment of the application, after the information processing device divides the stored data into at least one type of data based on the data access frequency, the information processing device can determine the corresponding at least one storage system, and transfer at least one type of data to at least one storage system respectively.

In this embodiment of the present application, at least one type of data is in one-to-one correspondence with at least one storage class, that is, one type of data corresponds to one storage class. For example: the first data of the hot data category corresponds to the hot data storage category, and the second data of the cold data category corresponds to the cold data storage category.

Exemplarily, the correspondence between at least one storage class corresponding to at least one type of data may be: DATA_STORAGE=>'CFS' or DATA_STORAGE=>'HDFS'.

In this embodiment of the present application, at least one storage system includes a high-frequency access storage system and/or a low-frequency access storage system.

It should be noted that the HDFS system is used to store the first data identified by the hot data; the CFS is used to store the second data identified by the cold data.

In this embodiment of the application, after the information processing device transfers at least one type of data to at least one storage system, the information processing device stores the first data in the high-frequency access storage system, and the storage time of the first data meets the predetermined When the storage duration is set, the information processing device detects the third access frequency of the first data; when the third access frequency is less than the preset access frequency threshold, the information processing device determines that the first data is marked as a cold data mark The information processing device transfers the first data from the high-frequency access storage system to the low-frequency access storage system corresponding to the cold data identifier.

In this embodiment of the present application, the third access frequency may be the total number of accesses within the preset storage duration of the stored data, and the third access frequency may also be the number of accesses per second of the stored data.

Exemplarily, if the third access frequency can be the total access times of the stored data within the preset storage period, the third access frequency can be the total access times of the stored data within 6 months (for example, 1000qps). If the third access frequency is the number of accesses per second of the stored data, the third access frequency may be 1000qps/sec (one thousand requests per second); or the third access frequency may be 1000qps/sec and may last for 5 minutes.

In this embodiment of the application, the preset storage duration may be the duration configured in the information processing device; the preset storage duration may also be the information processing device obtains before the information processing device compares the storage duration of the first data with the preset storage duration. The duration obtained; the preset storage duration may also be the duration obtained by the information processing device in other ways, which may be determined according to actual conditions, and is not limited in this embodiment of the present application.

In the embodiment of the present application, the process of the information processing device transferring the first data from the high-frequency access storage system to the low-frequency access storage system corresponding to the cold data identifier includes that the information processing device first compresses the first data to obtain the compressed the compressed first data; the information processing device transfers the compressed first data to the low-frequency access storage system.

In the embodiment of the present application, the information processing device can use the lossless compression algorithm (LZ4) compression format to compress the stored data to obtain the compressed first data; the information processing device can also use other data compression methods to compress the stored data , to obtain the compressed first data, which may be specifically determined according to actual conditions, which is not limited in this embodiment of the present application.

In the embodiment of the present application, the information processing device further includes a data flow (compact) component, the data flow component is used to organize the module to be responsible for monitoring the access frequency of the stored data and formulating rules, and the life cycle of the stored data meets the rules set by the business Data flow can be realized (periodically check and analyze files, count file access frequency, check whether rules are met, and automate data migration).

It can be understood that, when the information processing device determines that the access frequency of the first data stored in the HDFS system is less than the preset access frequency threshold (or the identifier of the first data becomes a cold data identifier), the information processing device will The data transfer component can be used to transfer the first data to the CFS system, which eliminates the need to manually transfer the first data, realizes the automatic transfer process of the first data, and improves the intelligence of the first data transfer .

In this embodiment of the application, after the information processing device transfers at least one type of data to at least one storage system, the information processing device stores the second data in the low-frequency access storage system, and the storage time of the second data meets the predetermined When the storage duration is assumed, the information processing device detects the fourth access frequency of the second data; when the fourth access frequency is greater than or equal to the preset access frequency threshold, the information processing device determines that the identifier of the second data is a hot data identifier ; The information processing device transfers the second data from the low-frequency access storage system to the high-frequency access storage system corresponding to the hot data identifier.

In the embodiment of the present application, the fourth access frequency may be the total number of access times of the second data within the preset storage duration, and the fourth access frequency may also be the number of access times of the second data per second.

Exemplarily, if the fourth access frequency can be the total access times of the second data within the preset storage period, the fourth access frequency can be the total access times of the second data within 6 months (for example, 1000qps). If the fourth access frequency is the number of times the second data is accessed per second, the fourth access frequency may be 1000qps/sec (one thousand requests per second); or the fourth access frequency may be 1000qps/sec and may last for 5 minutes.

In the embodiment of the present application, the information processing device transfers the second data from the low-frequency access storage system to the high-frequency access storage system corresponding to the hot data identifier, including that the information processing device first compresses the second data to obtain Compressed second data: the information processing device transfers the compressed second data to the high-frequency access storage system corresponding to the hot data identifier.

In the embodiment of the present application, the information processing device can use the data compression component (Zstandard, ZSTD) compression format to compress the second data to obtain the compressed second data; the information processing device can also use other data compression methods to compress the second data. The second data is compressed to obtain the compressed second data, which may be specifically determined according to actual conditions, which is not limited in this embodiment of the present application.

It can be understood that when the information processing device determines that the access frequency of the second data stored in the CFS system is greater than or equal to the preset access frequency threshold (or the identifier of the second data becomes a hot data identifier), the information processing The device can use the data transfer component to transfer the second data to the HDFS system, no longer need to manually transfer the second data, realize the automatic transfer process of the second data, and improve the efficiency of the second data transfer intelligence.

Exemplarily, as shown in FIG. 2 : the information processing device may be NoSQL, and NoSQL includes Regionserver. The specific Regionserver includes memory, file system interface and data flow components (data flow); NoSQL also includes management nodes. The information processing device may first use the management node to receive the data identifier corresponding to the stored data (create a table), and then, when the information processing device receives the stored data written by the client, the information processing device directly writes the stored data into Internal memory, when the information processing device detects that the remaining storage capacity in the internal memory is less than the preset capacity lower limit threshold, the information processing device detects the data access frequency corresponding to the stored data in the internal memory, and the information processing device filters out from the data access frequency The hot data identifies the corresponding first access frequency, and determines the first data corresponding to the first access frequency from the stored data; the information processing device selects a storage medium based on at least one storage category corresponding to at least one type of data, and determines the first data The corresponding storage system (the type is HDFS) is a high-frequency access storage system (HDFS client); the information processing device transfers the first data to the high-frequency access storage system through the file system interface. The information processing device screens out the second access frequency corresponding to the cold data identifier from the data access frequency, and determines the second data corresponding to the second access frequency from the stored data; the information processing device based on at least one storage corresponding to at least one type of data Select the storage medium by category, and determine that the storage system (type is CFS) corresponding to the second data is a low-frequency access storage system (CFS client); the information processing device transfers the second data to the low-frequency access storage system through the file system interface. Afterwards, when the information processing device stores the first data in the high-frequency access storage system and the storage duration of the first data meets the preset storage duration, the information processing device periodically detects the third access frequency of the first data (Periodically check and analyze files, check file access frequency, check whether the automatic data migration meets the rules); when the third access frequency is less than the preset access frequency threshold, the information processing device determines that the identification of the first data is a cold data identification; the information processing device The first data is transferred from the high-frequency access storage system to the low-frequency access storage system corresponding to the cold data identifier through the data transfer component. When the information processing device stores the second data in the low-frequency access storage system, and the storage duration of the second data satisfies the preset storage duration, the information processing device periodically detects the fourth access frequency of the second data (periodic inspection Analyze the file, check the file access frequency, check whether the rule is satisfied (automated data migration); in the case that the fourth access frequency is greater than or equal to the preset access frequency threshold, the information processing device determines that the identification of the second data is a hot data identification; the information processing device passes The data transfer component transfers the second data from the low-frequency access storage system to the high-frequency access storage system corresponding to the hot data identifier. Wherein, the HDFS system is a system obtained from multiple (possibly three) SSDs.

Exemplarily, the way for the information processing device to create the hot and cold tables may be:

create'cold:table',{NAME=>'f1',METADATA=>{'DATA_STORAGE'=>'cfs'}}

create'chsTable',{NAME=>'f',COLD_BOUNDARY=>'86400'}.

Exemplary, as shown in Figure 3: the cluster management node (management node) includes metadata management, area management and initialization file system (initialization HDFS system and CFS system), under the situation that the information processing device starts, the cluster management node just Initialize the HDFS system and CFS system. When the Regionserver receives the stored data written by the client, the Regionserver writes the stored data into the memory. When the Regionserver detects that the remaining storage capacity in the memory is less than the preset capacity lower limit threshold, and When the Regionserver determines that the identifier of the first data is a hot data identifier, the Regionserver transfers the first data to the HDFS system; when the Regionserver detects that the remaining storage capacity in the memory is less than the preset capacity lower limit threshold, and the Regionserver determines that the When the second data is marked as cold data, the Regionserver transfers the second data to the CFS system.

Exemplarily, as shown in Figure 4: when the information processing device receives the stored data written by the client through the cluster management node, the information processing device directly writes the stored data into the memory, and when the information processing device detects When the remaining storage capacity in the internal memory is less than the preset capacity lower limit threshold, the information processing device detects the data access frequency corresponding to the stored data in the internal memory, and the information processing device screens out the first access corresponding to the hot data identifier from the data access frequency. Frequency, and determine the first data corresponding to the first access frequency from the stored data; the information processing device determines that the storage system corresponding to the first data is high based on at least one storage category (type is HDFS) corresponding to at least one type of data A frequent access storage system (CFS system); the information processing device transfers the first data to the high frequency access storage system through a file system interface. The information processing device screens out the second access frequency corresponding to the cold data identifier from the data access frequency, and determines the second data corresponding to the second access frequency from the stored data; the information processing device based on at least one storage corresponding to at least one type of data The category (the type is CFS), determines that the storage system corresponding to the second data is a low-frequency access storage system (CFS system); the information processing device transfers the second data to the low-frequency access storage system through the file system interface. Afterwards, when the information processing device stores the first data in the high-frequency access storage system, and the storage duration of the first data meets the preset storage duration, the information processing device detects the third access frequency of the first data; 3. When the access frequency is less than the preset access frequency threshold, the information processing device determines that the identifier of the first data is a cold data identifier; the information processing device uses the data transfer component (data transfer) to transfer the first data from the high-frequency access storage system Save to the low-frequency access storage system corresponding to the cold data ID. When the information processing device stores the second data in the low-frequency access storage system, and the storage duration of the second data satisfies the preset storage duration, the information processing device detects a fourth access frequency of the second data; If it is greater than or equal to the preset access frequency threshold, the information processing device determines that the identification of the second data is a hot data identification; the information processing device uses the data flow component to transfer the second data from the low-frequency access storage system to the hot data Identify the corresponding high-frequency access storage system.

Exemplarily, as shown in FIG. 5: the information processing device stores the stored data written by the client in real time into the internal memory, and when the information processing device determines that the identifier of the first data in the stored data is a hot data identifier, The information processing device transfers (refreshes) the first data to the HDFS system (hot smoke layer). Determine that the storage location corresponding to the cold data mark is the CFS system (cold layer); and utilize the LZ4 compression method to compress the first data (data compression), and transfer the compressed first data to the cold layer; after that, in the information When the processing device determines that the identifier of the first data in the cold layer has changed to a hot data identifier, the information processing device determines that the storage location corresponding to the hot data identifier is the hot smoke layer; and uses the ZSTD compression method to compress the compressed first data The data is compressed (data compression), the first compressed data is obtained, and the first compressed data is transferred to the hot smoke layer. Among them, the hot smoke layer is the local file system in the file system layer, including (HDD, SSD, NVM, AEP storage class memory); the cold layer is the storage in the file system layer, specifically including cloud storage or other cloud storage.

It can be understood that the information processing device detects the data access frequency corresponding to the stored data in the memory, so that the information processing device can divide the stored data into at least one type of data based on the data access frequency, based on at least one corresponding to the at least one type of data Storage category, determining at least one storage system corresponding to at least one type of data, so as to transfer at least one type of data to at least one storage system; wherein, each type of data corresponds to a storage system, and there is no need to manually determine the storage data Data popularity and corresponding storage locations do not require manual dumping of stored data, which improves intelligence when storing stored data.

Embodiment two

Based on the same inventive concept of Embodiment 1, the embodiment of the present application provides an information processing device 1 corresponding to an information processing method; FIG. 6 is a schematic diagram of the composition and structure of an information processing device provided in the embodiment of the present application. Information processing device 1 may include:

The detection part 11 is configured to detect the data access frequency corresponding to the stored data in the internal memory when it is detected that the remaining storage capacity in the internal memory is less than or equal to the preset capacity lower limit threshold;

The division part 12 is configured to divide the stored data into at least one type of data based on the data access frequency; each type of data in the at least one type of data corresponds to a storage category;

The determining part 13 is configured to determine at least one storage system corresponding to the at least one type of data based on at least one storage class corresponding to the at least one type of data;

The dumping part 14 is configured to dump the at least one type of data to the at least one storage system respectively; wherein, each type of data corresponds to a storage system.

In some embodiments of the present application, the device further includes a receiving part;

The receiving part is configured to receive a data identifier corresponding to the stored data; the data identifier includes a hot data identifier and a cold data identifier;

Correspondingly, the at least one type of data includes first data and/or second data; the device further includes a screening part and an identification part;

The screening part is configured to filter out the hot data tags from the data tags; and/or, filter out the cold data tags from the data tags;

The identification part is configured to use the hot data identifier to identify the first access frequency corresponding to the first data; and/or use the cold data identifier to identify the second access frequency corresponding to the second data; The first access frequency and/or the second access frequency are used as the data access frequency. .

In some embodiments of the present application, the screening part is configured to filter out the first access frequency corresponding to the hot data identifier from the data access frequency, and/or, from the data access frequency Filter out the second access frequency corresponding to the cold data identifier;

The determining part 13 is configured to determine from the stored data the first data corresponding to the first access frequency; and/or, determine from the stored data that the second access frequency corresponds to The second data of .

In some embodiments of the present application, the determination part 13 is configured to filter out the first access frequency whose access frequency is greater than or equal to a preset access frequency threshold value from the data access frequencies, and determine the first data corresponding to the first access frequency; and/or, filter out the second access frequency whose access frequency is less than the preset access frequency threshold from the data access frequency, and determine in the stored data second data corresponding to the second access frequency. .

In some embodiments of the present application, the at least one storage system includes a high-frequency access storage system and/or a low-frequency access storage system;

The detection part 11 is configured to detect the third data of the first data when the first data is stored in the high-frequency access storage system and the storage duration of the first data satisfies a preset storage duration. frequency of visits;

The determining part 13 is configured to determine that the identifier of the first data is a cold data identifier when the third access frequency is less than a preset access frequency threshold;

The dumping part 14 is configured to dump the first data from the high-frequency access storage system to the low-frequency access storage system corresponding to the cold data identifier.

In some embodiments of the present application, the detection part 11 is configured to detect the a fourth access frequency of the second data;

The determining part 13 is configured to determine that the identifier of the second data is a hot data identifier when the fourth access frequency is greater than or equal to a preset access frequency threshold;

The dumping part 14 is configured to dump the second data from the low-frequency access storage system to the high-frequency access storage system corresponding to the hot data identifier.

It should be noted that, in practical applications, the above-mentioned detection part 11, division part 12, determination part 13 and dump part 14 can be realized by the processor 15 on the information processing device 1, specifically a CPU (Central Processing Unit, central processing unit ), MPU (Microprocessor Unit, microprocessor), DSP (Digital Signal Processing, digital signal processor) or field programmable gate array (FPGA, Field Programmable Gate Array) etc.; The memory 16 implements.

The embodiment of the present invention also provides an information processing device 1. As shown in FIG. The processor 15 communicates, and the memory 16 stores a program executable by the processor 15. When the program is executed, the processor 15 executes the information processing method as described above.

In practical applications, the above-mentioned memory 16 can be a volatile memory (volatile memory), such as a random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), flash memory (flash memory), hard disk (Hard Disk Drive, HDD) or solid-state drive (Solid-State Drive, SSD); Provide instructions and data.

An embodiment of the present invention provides a computer-readable storage medium, on which there is a computer program, and when the program is executed by the processor 15, the information processing method as described above is realized.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the protection scope of the present application.

Industrial Applicability

Claims

An information processing method, the method comprising:

When it is detected that the remaining storage capacity in the internal memory is less than or equal to the preset capacity lower limit threshold, detecting the data access frequency corresponding to the stored data in the internal memory;

Divide the stored data into at least one type of data based on the data access frequency; each type of data in the at least one type of data corresponds to a storage category;

Based on at least one storage category corresponding to the at least one type of data, determine at least one storage system corresponding to the at least one type of data, and transfer the at least one type of data to the at least one storage system respectively; wherein, each A type of data corresponds to a storage system.
The method according to claim 1, wherein, when it is detected that the remaining storage capacity in the internal memory is less than or equal to the preset capacity lower limit threshold, before detecting the data access frequency corresponding to the stored data in the internal memory, the The method also includes:

Receive a data identifier corresponding to the stored data; the data identifier includes a hot data identifier and a cold data identifier;

Correspondingly, the at least one type of data includes first data and/or second data; the detection of the data access frequency corresponding to the stored data in the memory includes:

Filter out the hot data identifier from the data identifiers, and use the hot data identifier to identify a first access frequency corresponding to the first data;

And/or, filter out the cold data identifier from the data identifiers, and use the cold data identifier to identify a second access frequency corresponding to the second data;

Use the first access frequency and/or the second access frequency as the data access frequency.
The method according to claim 2, wherein said dividing said stored data into at least one type of data based on said data access frequency comprises:

selecting the first access frequency corresponding to the hot data identifier from the data access frequency, and determining the first data corresponding to the first access frequency from the stored data;

And/or, filter out the second access frequency corresponding to the cold data identifier from the data access frequency, and determine the second data corresponding to the second access frequency from the stored data.
The method according to claim 1, wherein said dividing said stored data into at least one type of data based on said data access frequency comprises:

Screening out a first access frequency whose access frequency is greater than or equal to a preset access frequency threshold from the data access frequencies, and determining first data corresponding to the first access frequency in the stored data;

And/or, filter out a second access frequency whose access frequency is less than the preset access frequency threshold from the data access frequencies, and determine second data corresponding to the second access frequency in the stored data.
The method according to claim 1, wherein said at least one storage system includes a high-frequency access storage system and/or a low-frequency access storage system; said transferring said at least one type of data to said at least one storage system respectively Afterwards, the method also includes:

When the first data is stored in the high-frequency access storage system and the storage duration of the first data satisfies a preset storage duration, detecting a third access frequency of the first data;

When the third access frequency is less than a preset access frequency threshold, determine that the first data identifier is a cold data identifier;

Dumping the first data from the high-frequency access storage system to the low-frequency access storage system corresponding to the cold data identifier.
The method according to claim 1, wherein, after transferring the at least one type of data to the at least one storage system, the method further comprises:

When the second data is stored in the low-frequency access storage system, and the storage duration of the second data satisfies a preset storage duration, detecting a fourth access frequency of the second data;

When the fourth access frequency is greater than or equal to a preset access frequency threshold, determining that the identifier of the second data is a hot data identifier;

Transferring the second data from the low-frequency access storage system to the high-frequency access storage system corresponding to the hot data identifier.
An information processing device, the device comprising:

The detecting part is configured to detect the data access frequency corresponding to the stored data in the internal memory when it is detected that the remaining storage capacity in the internal memory is less than or equal to the preset capacity lower limit threshold;

The dividing part is configured to divide the stored data into at least one type of data based on the data access frequency; each type of data in the at least one type of data corresponds to a storage category;

The determining part is configured to determine at least one storage system corresponding to the at least one type of data based on at least one storage class corresponding to the at least one type of data;

The dumping part is configured to dump the at least one type of data to the at least one storage system respectively; wherein, each type of data corresponds to a storage system.
The apparatus according to claim 7, further comprising a receiving portion;

The receiving part is configured to receive a data identifier corresponding to the stored data; the data identifier includes a hot data identifier and a cold data identifier;

Correspondingly, the at least one type of data includes first data and/or second data; the device further includes a screening part and an identification part;

The screening part is configured to filter out the hot data tags from the data tags; and/or, filter out the cold data tags from the data tags;

The identification part is configured to use the hot data identifier to identify the first access frequency corresponding to the first data; and/or use the cold data identifier to identify the second access frequency corresponding to the second data; The first access frequency and/or the second access frequency are used as the data access frequency.
An information processing device, the device comprising:

A memory, a processor, and a communication bus, the memory communicates with the processor through the communication bus, the memory stores an information processing program executable by the processor, and when the information processing program is executed , executing the method according to any one of claims 1 to 6 by the processor.
A storage medium, on which a computer program is stored, applied to an information processing device, when the computer program is executed by a processor, the method described in any one of claims 1 to 6 is realized.