CN109976905B

CN109976905B - Memory management method and device and electronic equipment

Info

Publication number: CN109976905B
Application number: CN201910159040.3A
Authority: CN
Inventors: 于连宇; 李栋
Original assignee: Lenovo Beijing Ltd
Current assignee: Lenovo Beijing Ltd
Priority date: 2019-03-01
Filing date: 2019-03-01
Publication date: 2021-10-22
Anticipated expiration: 2039-03-01
Also published as: CN109976905A

Abstract

The present disclosure provides a memory management method, which includes obtaining data characteristics of a plurality of sets of data, predicting a usage frequency of the plurality of sets of data based on the data characteristics, and writing part of the data in the plurality of sets of data into a memory based on the usage frequency of the plurality of sets of data. The disclosure also provides a memory management device, a server cluster and a computer readable storage medium.

Description

Memory management method and device and electronic equipment

Technical Field

The disclosure relates to a memory management method and device and an electronic device.

Background

The query speed is a core problem of a computing system, and in data query, compared with disk reading and writing, the performance improvement of a memory has an influence on the query speed in a magnitude order. Currently, memory management generally adopts a runtime loading policy or a static caching policy in advance. The loading during the operation is to apply for the memory immediately when the query is carried out, the memory is insufficient and needs to overflow to a disk, and the speed of the stress reaction method cannot be guaranteed. The pre-static policy is to write specific data into the memory in advance, which is relatively wasteful of memory and is not suitable for applications with a tight memory or large data volume.

Disclosure of Invention

An aspect of the present disclosure provides a memory management method, including obtaining data characteristics of a plurality of sets of data, predicting a frequency of use of the plurality of sets of data based on the data characteristics, and writing a part of the plurality of sets of data into a memory based on the frequency of use of the plurality of sets of data.

Optionally, the method further includes training a machine learning model based on data features and query behavior, and predicting the frequency of use of the plurality of sets of data based on the data features includes: predicting, based on the machine learning model, a frequency of use of a plurality of sets of data corresponding to the data features.

Optionally, the method further includes determining, based on the usage frequency of the multiple sets of data, a portion of data to be written into the memory as a set of data to be cached, and removing data that does not belong to the set of data to be cached from the memory.

Optionally, the writing, based on the usage frequency of the multiple sets of data, part of the data in the multiple sets of data into a memory includes writing, into the memory, part of the data in the data set to be cached that is not stored in the memory.

Optionally, the writing, based on the usage frequency of the multiple sets of data, part of the data in the multiple sets of data into a memory includes obtaining a planning capacity, determining, based on the usage frequency of the multiple sets of data and the planning capacity, that part of the data to be written into the memory is used as a data set to be cached, and writing part or all of the data in the data set to be cached into the memory.

Optionally, in a case that the data includes data in a data table, the partial data includes data in a data slice of the data table.

Another aspect of the disclosure provides a memory management apparatus including an obtaining module, a predicting module, and a writing module. And the obtaining module is used for obtaining the data characteristics of the multiple groups of data. And the prediction module is used for predicting the use frequency of the multiple groups of data based on the data characteristics. And the writing module is used for writing part of data in the multiple groups of data into a memory based on the use frequency of the multiple groups of data.

Optionally, the apparatus further comprises a determining module and a removing module. And the determining module is used for determining partial data to be written into the memory as a data set to be cached based on the use frequency of the multiple groups of data. And the removing module is used for removing the data which does not belong to the data set to be cached from the memory.

Optionally, the writing module is configured to write a part of data, which is not stored in the memory, in the data set to be cached into the memory.

Optionally, the apparatus further comprises a training module for training the machine learning model based on the data features and the query behavior. The prediction module is configured to predict a frequency of use of a plurality of sets of data corresponding to the data features based on the machine learning model.

Optionally, the writing module includes an obtaining submodule, a determining submodule, and a writing submodule. An obtaining submodule for obtaining the planned capacity. And the determining submodule is used for determining partial data to be written into the memory as a data set to be cached based on the use frequency of the multiple groups of data and the planning capacity. And the writing submodule is used for writing part or all of the data in the data set to be cached into a memory.

Another aspect of the disclosure provides a server cluster including at least one processor and at least one memory. The memory has stored thereon a computer program which, when executed by the processor, causes the processor to perform the method as described above.

Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method as described above when executed.

Another aspect of the disclosure provides a computer program comprising computer executable instructions for implementing the method as described above when executed.

Drawings

For a more complete understanding of the present disclosure and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

fig. 1 schematically illustrates an application scenario of a memory management method according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a memory management method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a memory management method according to another embodiment of the present disclosure;

FIG. 4 is a flow chart schematically illustrating writing a portion of the plurality of sets of data to memory based on a frequency of use of the plurality of sets of data, in accordance with an embodiment of the present disclosure;

fig. 5 schematically shows a block diagram of a memory management device according to an embodiment of the present disclosure;

fig. 6 schematically shows a block diagram of a memory management device according to another embodiment of the present disclosure;

FIG. 7 schematically illustrates a block diagram of a write module according to an embodiment of the disclosure; and

FIG. 8 schematically shows a block diagram of a server cluster according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

Some block diagrams and/or flow diagrams are shown in the figures. It will be understood that some blocks of the block diagrams and/or flowchart illustrations, or combinations thereof, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the instructions, which execute via the processor, create means for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks. The techniques of this disclosure may be implemented in hardware and/or software (including firmware, microcode, etc.). In addition, the techniques of this disclosure may take the form of a computer program product on a computer-readable storage medium having instructions stored thereon for use by or in connection with an instruction execution system.

The embodiment of the disclosure provides a memory management method, which includes obtaining data characteristics of multiple groups of data, predicting the use frequency of the multiple groups of data based on the data characteristics, and writing part of the data in the multiple groups of data into a memory based on the use frequency of the multiple groups of data.

Fig. 1 schematically illustrates an application scenario of a memory management method according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a scenario in which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1, the distributed system 100 may include a plurality of nodes 110 and a network 120. Network 120 serves as a medium for providing communication links between multiple nodes 110. Network 120 may include various connection types, such as wired, wireless communication links, and so forth. The node 110 may be, for example, a dedicated computing or storage device, or a terminal device of multiple users, etc. The distributed system 100 may implement the memory management method of the embodiments of the present disclosure.

In the distributed system 100, common data is stored in a distributed memory, and reading and writing in the memory can effectively improve the calculation efficiency. The disclosed embodiments focus on how to reasonably utilize memory and speed up data query. The distributed system generally plans a data cache region, and the data cache region occupies a data memory with a considerable scale.

Fig. 2 schematically shows a flow chart of a memory management method according to an embodiment of the present disclosure.

As shown in fig. 2, the method includes operations S210 to S230.

In operation S210, data characteristics of a plurality of sets of data are obtained.

According to embodiments of the present disclosure, data is typically stored in a database in the form of tables. The plurality of sets of data may be a plurality of data tables. The data characteristics may be, for example, a data field repetition rate, a null rate in the data table (a ratio of null data to all data in the table), a range interval data rate, and the like.

The above may be referred to as static characteristics of the data. The embodiments of the present disclosure may also obtain dynamic characteristics of data, for example, by processing the query log, query time, query subject, query condition, and the like of each data query may be obtained. The disclosed embodiments may integrate the static and dynamic characteristics described above to predict the frequency of use of the data.

In operation S220, a frequency of use of the plurality of sets of data is predicted based on the data characteristics. The frequency of use of the data is also referred to as the heat and cold of the data.

According to the embodiment of the disclosure, a machine learning model can be trained based on data features and query behaviors, and the use frequency of multiple groups of data corresponding to the data features can be predicted based on the machine learning model. The machine learning model may include, for example, a decision tree, a support vector machine, an artificial neural network, and so forth. By collecting data for a certain period of time, processing the data as training data, taking data characteristics as input, inquiring behavior as output label, and training the machine learning model, the use frequency of the data in a subsequent period of time can be predicted under the condition of obtaining new data characteristics.

In operation S230, a portion of the data in the plurality of sets of data is written into a memory based on the frequency of use of the plurality of sets of data. For example, a threshold value may be set, and when the frequency of use is greater than the threshold value, the data may be written in the memory in advance. Alternatively, all the use frequencies may be clustered, and the data in a set with a higher use frequency may be written in the memory in advance. Or, all the data can be sorted according to the use frequency and sequentially written into the memory until the space occupied by the written data reaches a preset value.

According to an embodiment of the present disclosure, in a case where the data includes data in a data table, the partial data includes data in a data slice of the data table. For example, for a data table, a portion of the data table with a higher frequency of use may be further or simultaneously determined, and only the portion of the data may be written to the memory instead of the entire data table.

Fig. 3 schematically shows a flow chart of a memory management method according to another embodiment of the present disclosure.

As shown in fig. 3, the method further includes operations S310 and S320 on the basis of the foregoing embodiment.

In operation S310, based on the usage frequency of the multiple sets of data, a portion of data to be written into the memory is determined as a set of data to be cached.

In operation S320, data not belonging to the data set to be cached is removed from the memory.

According to the method, after the predicted use frequency is obtained, the data which are not commonly used are removed from the memory, the memory space is dynamically managed, and the use rate of system resources is improved.

According to the embodiment of the present disclosure, as shown in fig. 3, operation S230 may be specifically implemented as operation S330, where part of data, which is not stored in the memory, in the data set to be cached is written into the memory. At the time of writing, it may be checked whether data to be stored in the memory already exists in the memory, and if so, the writing is not repeated.

Fig. 4 schematically shows a flowchart of writing part of the data in the plurality of data groups into the memory based on the usage frequency of the plurality of data groups according to an embodiment of the present disclosure.

As shown in fig. 4, the method includes operations S410 to S430.

In operation S410, a planned capacity is obtained. According to the embodiment of the present disclosure, the planning capacity is a memory space reserved for the memory management method of the embodiment of the present disclosure. It may be a whole memory space or a part of memory space, and may be defined by a proportion or a fixed size.

According to an embodiment of the present disclosure, the projected capacity may be dynamically determined according to a load of the server. For example, the projected capacity may be increased when the server load is high to cache more data with a higher frequency of use, and may be decreased when the server load is low to allow the server sufficient available memory space when handling various tasks that are difficult to predict.

In operation S420, a portion of data to be written into the memory is determined as a data set to be cached, based on the usage frequency of the multiple sets of data and the planning capacity. According to the embodiment of the disclosure, all data can be sorted according to the use frequency and sequentially written into the memory until the space occupied by the written data reaches the planning capacity.

In operation S430, part or all of the data in the data set to be cached is written into a memory.

The method of the embodiment of the disclosure can be periodically executed to realize dynamic management of the memory.

According to the method, the data are preloaded by predicting the use frequency of the data, the query is accelerated, and the possibility of memory waste or overflow is reduced.

Based on the same inventive concept, the present disclosure also provides a memory management device, and the memory management device according to the embodiment of the present disclosure is described below with reference to fig. 5 to 7.

Fig. 5 schematically shows a block diagram of a memory management device 500 according to an embodiment of the present disclosure.

As shown in fig. 5, the memory management device 500 includes an obtaining module 510, a predicting module 520, and a writing module 530. The memory management device 500 may perform the various methods described above.

The obtaining module 510, for example, performs the operation S210 described with reference to fig. 2 above, for obtaining data characteristics of multiple sets of data.

The prediction module 520, for example, performs operation S220 described with reference to fig. 2 above, for predicting the frequency of use of the plurality of sets of data based on the data characteristics.

The writing module 530, for example, performs the operation S230 described with reference to fig. 2 above, and is configured to write a part of the data in the plurality of sets of data into the memory based on the usage frequency of the plurality of sets of data.

According to an embodiment of the present disclosure, in a case where the data includes data in a data table, the partial data includes data in a data slice of the data table.

Fig. 6 schematically shows a block diagram of a memory management device 600 according to another embodiment of the present disclosure.

As shown in fig. 6, the memory management apparatus 600 further includes a determining module 610 and a removing module 620 based on the illustration in fig. 5.

The determining module 610, for example, performs operation S310 described with reference to fig. 3 above, and is configured to determine, based on the usage frequency of the multiple sets of data, a portion of data to be written into the memory as a set of data to be cached.

The removing module 620, for example, executes the operation S320 described with reference to fig. 3 above, to remove the data not belonging to the data set to be cached from the memory.

According to an embodiment of the present disclosure, the writing module 530, for example, performs the operation S330 described with reference to fig. 3 above, to write the part of the data, which is not stored in the memory, in the data set to be cached into the memory.

According to an embodiment of the present disclosure, the

apparatus

500 or 600 may further include a training module for training the machine learning model based on the data features and the query behavior. The prediction module 520 is configured to predict a frequency of use of a plurality of sets of data corresponding to the data features based on the machine learning model.

FIG. 7 schematically shows a block diagram of a write module 530 according to an embodiment of the disclosure.

As shown in fig. 7, the write module 530 includes an obtaining sub-module 710, a determining sub-module 720, and a writing sub-module 730.

The obtaining sub-module 710, for example, performs the operation S410 described with reference to fig. 4 above, for obtaining the planned capacity.

The determining submodule 720, for example, executes the operation S420 described with reference to fig. 4 above, and is configured to determine, based on the usage frequency of the multiple sets of data and the planning capacity, a portion of data to be written into the memory as a set of data to be cached.

The writing sub-module 730, for example, executes the operation S430 described with reference to fig. 4 above, and is configured to write part or all of the data in the data set to be cached into the memory.

Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

For example, any number of the obtaining module 510, the predicting module 520, the writing module 530, the determining module 610, the removing module 620, the training module, the obtaining sub-module 710, the determining sub-module 720, and the writing sub-module 730 may be combined in one module to be implemented, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the disclosure, at least one of the obtaining module 510, the predicting module 520, the writing module 530, the determining module 610, the removing module 620, the training module, the obtaining sub-module 710, the determining sub-module 720, and the writing sub-module 730 may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware by any other reasonable manner of integrating or packaging a circuit, or may be implemented in any one of three implementations of software, hardware, and firmware, or in a suitable combination of any of them. Alternatively, at least one of the obtaining module 510, the predicting module 520, the writing module 530, the determining module 610, the removing module 620, the training module, the obtaining sub-module 710, the determining sub-module 720 and the writing sub-module 730 may be at least partially implemented as a computer program module which, when executed, may perform a corresponding function.

Fig. 8 schematically shows a block diagram of a server cluster 800 according to an embodiment of the disclosure. The computer system illustrated in FIG. 8 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure.

As shown in fig. 8, a server cluster 800 includes at least one processor 810 and at least one memory, including a computer-readable storage medium 820. The electronic device 800 may perform a method according to an embodiment of the disclosure.

In particular, processor 810 may include, for example, a general purpose microprocessor, an instruction set processor and/or related chip set and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), and/or the like. The processor 810 may also include on-board memory for caching purposes. Processor 810 may be a single processing unit or a plurality of processing units for performing different actions of a method flow according to embodiments of the disclosure.

Computer-readable storage medium 820, for example, may be a non-volatile computer-readable storage medium, specific examples including, but not limited to: magnetic storage devices, such as magnetic tape or Hard Disk Drives (HDDs); optical storage devices, such as compact disks (CD-ROMs); a memory, such as a Random Access Memory (RAM) or a flash memory; and so on.

The computer-readable storage medium 820 may include a computer program 821, which computer program 821 may include code/computer-executable instructions that, when executed by the processor 810, cause the processor 810 to perform a method according to an embodiment of the present disclosure, or any variation thereof.

The computer program 821 may be configured with, for example, computer program code comprising computer program modules. For example, in an example embodiment, code in computer program 821 may include one or more program modules, including for example 821A, modules 821B, … …. It should be noted that the division and number of modules are not fixed, and those skilled in the art may use suitable program modules or program module combinations according to actual situations, and when the program modules are executed by the processor 810, the processor 810 may execute the method according to the embodiment of the present disclosure or any variation thereof.

According to an embodiment of the present invention, at least one of the obtaining module 510, the predicting module 520, the writing module 530, the determining module 610, the removing module 620, the training module, the obtaining sub-module 710, the determining sub-module 720 and the writing sub-module 730 may be implemented as a computer program module as described with reference to fig. 8, which when executed by the processor 810 may implement the corresponding operations described above.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

While the disclosure has been shown and described with reference to certain exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the appended claims and their equivalents. Accordingly, the scope of the present disclosure should not be limited to the above-described embodiments, but should be defined not only by the appended claims, but also by equivalents thereof.

Claims

1. A memory management method includes:

acquiring data characteristics and query behaviors of a plurality of groups of data;

taking data characteristics as input, inquiring the behavior as an output label, and training a machine learning model;

predicting, based on the machine learning model, a frequency of use of a plurality of sets of data corresponding to the data features;

and writing partial data in the multiple groups of data into a memory based on the use frequency of the multiple groups of data.

2. The method of claim 1, further comprising:

determining partial data to be written into the memory as a data set to be cached based on the use frequency of the multiple groups of data;

and removing the data which do not belong to the data set to be cached from the memory.

3. The method of claim 2, wherein the writing a portion of the plurality of sets of data to memory based on the frequency of use of the plurality of sets of data comprises:

and writing part of data which is not stored in the memory in the data set to be cached into the memory.

4. The method of claim 1, wherein the writing partial data of the plurality of sets of data to memory based on the frequency of use of the plurality of sets of data comprises:

obtaining a planned capacity;

determining partial data to be written into the memory as a data set to be cached based on the use frequency of the multiple groups of data and the planning capacity;

and writing part or all of the data in the data set to be cached into a memory.

5. The method of claim 1, wherein the partial data comprises data in a data slice of a data table where the data comprises data in the data table.

6. A memory management device, comprising:

the acquisition module is used for acquiring data characteristics and query behaviors of a plurality of groups of data;

the prediction module is used for training a machine learning model by taking the data characteristics as input and the query behavior as output labels; predicting, based on the machine learning model, a frequency of use of a plurality of sets of data corresponding to the data features;

and the writing module is used for writing part of data in the multiple groups of data into a memory based on the use frequency of the multiple groups of data.

7. The apparatus of claim 6, further comprising:

the determining module is used for determining partial data to be written into the memory as a data set to be cached based on the use frequency of the multiple groups of data;

a removing module, configured to remove, from the memory, data that does not belong to the data set to be cached,

the writing module is used for writing part of data which is not stored in the memory in the data set to be cached into the memory.

8. A cluster of servers, comprising:

at least one processor;

at least one memory having a computer program stored thereon, the computer program, when executed by the processor, causes the processor to:

9. A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, causes the processor to: