CN112783656B

CN112783656B - Memory management method, medium, device and computing equipment

Info

Publication number: CN112783656B
Application number: CN202110127352.3A
Authority: CN
Inventors: 陈涛; 高民; 朱浩齐
Original assignee: Hangzhou Netease Zhiqi Technology Co Ltd
Current assignee: Hangzhou Netease Zhiqi Technology Co Ltd
Priority date: 2021-01-29
Filing date: 2021-01-29
Publication date: 2024-04-30
Anticipated expiration: 2041-01-29
Also published as: CN112783656A

Abstract

The embodiment of the disclosure provides a memory management method, medium, device and computing equipment. The memory includes at least a first storage space for storing first data and a second storage space for storing second data, the method comprising: acquiring garbage collection time consumption, wherein the garbage collection time consumption represents the time spent for cleaning garbage objects in the memory; and adjusting a first ratio of the capacity of the first storage space to the capacity of the memory according to the garbage collection time consumption and a preset garbage collection time consumption threshold. The embodiment of the disclosure can improve the utilization rate of the memory and avoid memory overflow.

Description

Memory management method, medium, device and computing equipment

Technical Field

Embodiments of the present disclosure relate to the field of computer technology, and more particularly, to a memory management method, medium, apparatus, and computing device.

Background

This section is intended to provide a background or context to the embodiments of the disclosure recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

The ratio of the amount of data stored in the memory to the memory capacity is referred to as the memory utilization. If the memory capacity allocated for the data is too large, the memory may not be fully utilized, that is, the memory utilization is too high; if the memory capacity allocated for data is too small, there is a risk of memory overflow due to the data volume being greater than the memory capacity. Therefore, the setting of the memory is key to improve the memory utilization rate and avoid the memory overflow, and the related technology cannot reasonably set the memory.

Disclosure of Invention

The present disclosure desirably provides a memory management method and apparatus.

In a first aspect of the embodiments of the present disclosure, there is provided a memory management method, the memory including at least a first storage space for storing first data and a second storage space for storing second data, the method including:

acquiring garbage collection time consumption, wherein the garbage collection time consumption represents the time spent for cleaning garbage objects in the memory;

and adjusting a first ratio of the capacity of the first storage space to the capacity of the memory according to the garbage collection time consumption and a preset garbage collection time consumption threshold.

In one embodiment of the present disclosure, the first storage space is a strongly referenced memory space and the second storage space is a soft referenced memory space.

In one embodiment of the present disclosure, further comprising:

determining the capacity of the first storage space according to the first ratio;

under the condition that the data volume in the memory is larger than the capacity of the first storage space, migrating the first data exceeding the capacity part of the first storage space to the second storage space; or alternatively

And in the case that the data amount in the first storage space is smaller than the capacity of the first storage space and the second data exists in the second storage space, migrating at least part of the second data in the second storage space to the first storage space.

In one embodiment of the present disclosure, migrating first data that exceeds a capacity portion of the first storage space to the second storage space includes:

And screening third data to be migrated from the first data based on a preset data migration strategy, and migrating the third data from the first storage space to the second storage space.

In one embodiment of the present disclosure, obtaining garbage collection is time consuming, including:

Searching a preset first corresponding relation according to the capacity of the memory to obtain a corresponding first ratio suggested value and a garbage collection time-consuming suggested value; the first corresponding relation characterizes a first ratio recommended value and a garbage collection time-consuming recommended value corresponding to different memory capacities;

And determining the garbage collection time according to the garbage collection time recommended value.

In one embodiment of the present disclosure, adjusting the first ratio of the capacity of the first storage space to the capacity of the memory according to the garbage collection time and a preset garbage collection time threshold value includes:

and adjusting the first ratio to be equal to the first ratio recommended value under the condition that the garbage collection time is not more than the garbage collection time threshold.

determining the first ratio as a value less than the first ratio recommended value if the waste recycling time is greater than the waste recycling time threshold;

Determining the capacity of the first storage space according to the first ratio, and transferring the first data exceeding the capacity part of the first storage space to the second storage space under the condition that the data amount in the memory is larger than the capacity of the first storage space;

Acquiring a current actual garbage collection time consumption value from a garbage collection log, reducing the value of the first ratio under the condition that the actual garbage collection time consumption value is larger than the garbage collection time consumption threshold value, and executing the step of determining the capacity of the first storage space again based on the reduced first ratio; and ending the adjustment process until the actual garbage collection time consumption value is not greater than the garbage collection time consumption threshold value.

In one embodiment of the present disclosure, determining the first ratio as a value less than the first ratio suggestion value includes: determining the first ratio as half of the first ratio proposal value;

Reducing the value of the first ratio, comprising: and adjusting the first ratio to be half of the original value of the first ratio.

In one embodiment of the present disclosure, ending the adjustment process until the actual value of the garbage collection time is not greater than the garbage collection time threshold comprises:

And ending the adjustment process until the actual garbage collection time consumption value is not greater than the garbage collection time consumption threshold value and the difference between the actual garbage collection time consumption value and the garbage collection time consumption threshold value is not greater than a preset threshold value.

In one embodiment of the present disclosure, further comprising:

Under the condition of different memory capacities, different values of the first ratio and corresponding garbage collection time consumption statistic values are counted;

According to the result of statistics, establishing a mapping relation between the first ratio and the garbage collection time-consuming statistic under the condition of different memory capacities;

and determining the first corresponding relation according to the mapping relation.

In one embodiment of the present disclosure, the memory further includes an index area for storing an index of the first data and the second data.

In one embodiment of the present disclosure, further comprising:

determining the cleared second data in the second storage space if the second data is cleared; the index is updated based on the purged second data.

In one embodiment of the present disclosure, further comprising:

storing the cleared second data in the second storage space into a database;

Reading the cleared second data from the database;

when new data is stored in the memory, clustering the new data and the cleared second data, and if the clustering is successful, storing the clustered data into the memory;

And updating the index according to the clustered data.

In one embodiment of the present disclosure, the first data and the second data comprise public opinion text data;

The method further comprises the steps of: clustering public opinion text data in the memory;

and carrying out at least one of propagation path analysis, emotion data analysis and public opinion trend analysis on the clustered data.

In a second aspect of the embodiments of the present disclosure, there is provided a memory management apparatus, the memory including at least a first storage space for storing first data and a second storage space for storing second data, the apparatus comprising:

the data monitoring and counting module is used for acquiring the garbage collection time consumption, and the garbage collection time consumption characterizes the time spent for cleaning the garbage objects in the memory;

And the proportion dynamic adjustment module is used for adjusting a first ratio of the capacity of the first storage space to the capacity of the memory according to the garbage collection time consumption and a preset garbage collection time consumption threshold.

In one embodiment of the present disclosure, the apparatus further comprises:

The data migration module is used for determining the capacity of the first storage space according to the first ratio; under the condition that the data volume in the memory is larger than the capacity of the first storage space, migrating the first data exceeding the capacity part of the first storage space to the second storage space; or in case the amount of data in the first storage space is smaller than the capacity of the first storage space and there is second data in the second storage space, migrating at least part of the second data in the second storage space to the first storage space.

In one embodiment of the present disclosure, the data migration module is to:

In one embodiment of the present disclosure, the data monitoring statistics module is to:

Searching a preset first corresponding relation according to the capacity of the memory to obtain a corresponding first ratio suggested value and a garbage collection time-consuming suggested value; the first corresponding relation characterizes a first ratio recommended value and a garbage collection time-consuming recommended value corresponding to different memory capacities; and determining the garbage collection time according to the garbage collection time recommended value.

In one embodiment of the present disclosure, the scaling dynamic adjustment module is to:

In one embodiment of the present disclosure, the scaling dynamic adjustment module determines the first ratio as half of the first ratio suggested value when determining the first ratio as a value less than the first ratio suggested value;

when the value of the first ratio is reduced, the proportion dynamic adjustment module adjusts the first ratio to be half of the original value of the first ratio.

In one embodiment of the disclosure, the scaling dynamic adjustment module ends the adjustment process if the actual garbage collection time value is not greater than the garbage collection time threshold and the actual garbage collection time value is not greater than a preset threshold.

In one embodiment of the present disclosure, the data monitoring statistics module is further configured to:

In one embodiment of the present disclosure, further comprising:

The data synchronization module is used for determining the cleared second data in the second storage space under the condition that the second data in the second storage space is cleared; the index is updated based on the purged second data.

In one embodiment of the present disclosure, the method further comprises an incremental offline clustering module for:

storing the cleared second data in the second storage space into a database;

Reading the purged second data from the database;

And updating the index according to the clustered data.

the apparatus further comprises:

The memory area clustering module is used for clustering public opinion text data in the memory;

and the analysis module is used for carrying out at least one of propagation path analysis, emotion data analysis and public opinion trend analysis on the clustered data.

In a third aspect of the disclosed embodiments, a computer-readable medium is provided, on which a computer program is stored, which program, when being executed by a processor, implements the steps of the memory management method described above.

In a fourth aspect of embodiments of the present disclosure, there is provided a computing device comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the memory management method when the program is executed.

According to the memory management method and device, according to the garbage collection time consumption and the preset garbage collection time consumption threshold, the ratio of the capacity of the first storage space in the memory to the capacity of the memory is adjusted, so that the capacity ratio of different storage spaces in the memory is reasonably adjusted, the memory utilization rate is improved, and memory overflow is avoided.

Drawings

The above, as well as additional purposes, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description when read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which:

FIG. 1 is a schematic frame diagram of one exemplary application scenario of an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow diagram of a memory management method implementation according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates an implementation flow diagram for migrating a first storage space and/or data stored in the first storage space according to one embodiment of the present disclosure;

FIG. 4 schematically illustrates a memory structure and a memory management scheme according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a structure of a memory management module and a manner of adjusting a capacity ratio according to an embodiment of the present disclosure

FIG. 6 schematically illustrates a flow diagram for implementing a time-consuming acquisition garbage collection according to an embodiment of the present disclosure;

FIG. 7 schematically illustrates a schematic diagram of correspondence of strongly referenced memory duty cycle to garbage collection time consumption, according to an embodiment of the present disclosure;

FIG. 8 schematically illustrates an implementation flow diagram for adjusting a first ratio of capacity of a first storage space to memory capacity in accordance with an embodiment of the present disclosure;

FIG. 9 schematically illustrates an implementation of the incremental offline clustering module performing secondary clustering on eliminated data in a soft-reference memory region, according to one embodiment of the disclosure;

FIG. 10 schematically illustrates a flowchart of an implementation of the incremental offline clustering module for secondary clustering of eliminated data, in accordance with an embodiment of the present disclosure;

FIG. 11 schematically illustrates a flow diagram of data processed by an incremental offline clustering module according to one embodiment of the present disclosure;

FIG. 12 schematically illustrates a media schematic for a memory management method according to an embodiment of the present disclosure;

FIG. 13 schematically illustrates a schematic structure of a memory management device according to an embodiment of the present disclosure;

fig. 14 schematically illustrates a structural schematic diagram of a computing device according to an embodiment of the present disclosure.

In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

Detailed Description

The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are presented merely to enable one skilled in the art to better understand and practice the present disclosure and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Those skilled in the art will appreciate that embodiments of the present disclosure may be implemented as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the following forms, namely: complete hardware, complete software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.

According to an embodiment of the disclosure, a memory management method, medium, device and computing equipment are provided.

Any number of elements in the figures are for illustration and not limitation, and any naming is used for distinction only, and not for any limiting sense.

The principles and spirit of the present disclosure are explained in detail below with reference to several representative embodiments thereof.

Summary of The Invention

The inventor discovers that the related technology cannot reasonably set the memory, so that the memory cannot be fully utilized or the risk of memory overflow exists.

In view of this, the present disclosure provides a memory management method and apparatus, which divides a memory into at least a first storage space for storing first data and a second storage space for storing second data. One of the memory spaces (the first memory space or the second memory space) after the memory division can be recycled, and the memory space can be recycled when the memory space is insufficient, so that the problem of memory overflow is avoided; because the memory space can be recovered in real time, the excessive memory capacity is not required to be allocated in advance, and the memory utilization rate can be improved.

Having described the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are specifically described below.

Application scene overview

Referring first to fig. 1, fig. 1 is a schematic diagram of a framework of one exemplary application scenario of an embodiment of the present disclosure. Wherein a user interacts with a server 102 for public opinion text clustering and analysis through a client 101 on a user device. Those skilled in the art will appreciate that the frame diagram shown in fig. 1 is but one example in which embodiments of the present disclosure may be implemented. The scope of applicability of the embodiments of the present disclosure is not limited in any way by the framework.

It should be noted that the clients herein may be any user device, existing, under development or developed in the future, capable of interacting with the server 102 via any form of limited and/or wireless connection (e.g., wi-Fi, LAN, cellular mobile communication, coaxial cable, etc.); including but not limited to: existing, developing or future developing smartphones, non-smartphones, tablet computers, laptop personal computers, desktop personal computers, minicomputers, midrange computers, mainframe computers, and the like.

It is also noted that the server 102 herein is merely one example of an existing, developing or future developed device capable of providing public opinion text clustering and analysis services to users. Embodiments of the present disclosure are not limited in this respect.

Based on the framework shown in fig. 1, the server 102 may store the public opinion text data in a memory in response to receiving the public opinion text data, which may be divided into at least a strongly referenced memory space for storing the strongly referenced data and a soft referenced memory space for storing the soft referenced data. The server can adjust the ratio of the capacity of the strongly referenced memory space to the memory capacity according to the garbage collection time consumption and a preset garbage collection time consumption threshold. Under the condition that the strong reference memory space is free, the server can store the received public opinion text data in the strong reference memory space preferentially. Because the data stored in the soft reference memory space is allowed to be cleared by garbage collection operation, when the memory space is insufficient, the data overflow can be avoided by recovering the memory of the corresponding soft reference memory space; in addition, the memory space can be recovered in real time, and excessive memory capacity is not required to be allocated in advance, so that the memory utilization rate can be improved.

Exemplary method

A memory management method according to an exemplary embodiment of the present disclosure is described below with reference to fig. 2. The memory management method of the present disclosure includes the following steps, as shown in fig. 2:

S21: acquiring garbage collection time consumption, wherein the garbage collection time consumption represents the time spent for cleaning garbage objects in the memory;

s22: and adjusting a first ratio of the capacity of the first storage space to the capacity of the memory according to the garbage collection time consumption and a preset garbage collection time consumption threshold.

In this way, the memory is divided into at least the first storage control and the second storage space according to the embodiment of the present disclosure, so that one of the storage spaces may be set to be capable of being garbage collected, and data that is allowed to be deleted is stored in the storage space; the other memory space is set so that garbage collection cannot be performed, and data stored in the memory space cannot be cleared by the garbage collection operation. Because garbage collection is allowed to be carried out on one storage space, when the memory space is insufficient, data overflow can be avoided in a mode of collecting the memory of the corresponding storage space; in addition, the memory space can be recovered in real time, and excessive memory capacity is not required to be allocated in advance, so that the memory utilization rate can be improved.

In one embodiment, the first storage space may be a strongly referenced memory space, and the second storage space may be a soft referenced memory space. Accordingly, the first data stored in the first memory space is a strongly referenced object and the second data stored in the second memory space is a soft referenced object. In JAVA virtual machine technology, strongly referenced objects cannot be garbage collected and soft referenced objects can be garbage collected. Garbage collection is a memory management function, and can actively find objects which are no longer used in a program, and clean the objects which are no longer used, so that more memory is released.

The memory is divided into at least a strongly referenced memory space and a soft referenced memory space according to the garbage collection time consumption and a preset garbage collection time consumption threshold value, and the ratio of the capacity of the strongly referenced memory space to the capacity of the memory is adjusted. Because the soft reference objects stored in the soft reference memory space are allowed to be cleared by garbage collection operation, data overflow can be avoided by a mode of collecting the memory of the corresponding memory space when the memory space is insufficient; in addition, the memory space can be recovered in real time, and excessive memory capacity is not required to be allocated in advance, so that the memory utilization rate can be improved.

In another embodiment, the first storage space may be a soft-reference memory space, and the second storage space may be a strong-reference memory space; accordingly, the first data stored in the first memory space is a soft reference object and the second data stored in the second memory space is a strong reference object.

In the following, the first storage space is taken as a strongly referenced memory space, and the second storage space is taken as a soft referenced memory space as an example.

From the standpoint of the memory use effect, the larger the space ratio of the strongly referenced memory is, the better, but the larger the space ratio of the strongly referenced memory is, the more time-consuming garbage collection is caused. Therefore, in order to give consideration to both the memory use effect and the garbage collection time consumption performance, the capacity ratio of the first storage space is adjusted according to the garbage collection time consumption monitored in real time and the garbage collection time consumption threshold value set in advance (since the memory mainly comprises the first storage space and the second storage space, the capacity ratio of the first storage space is adjusted, that means, the capacity ratio of the second storage space is adjusted at the same time), so that the capacities of all storage spaces in the memory are reasonably distributed, and under the condition that the garbage collection time consumption threshold value is met, the occupation ratio of the memory space with strong reference is larger as much as possible, and the use effect of the memory and the garbage collection time consumption performance are balanced.

In a possible implementation manner, after the first ratio of the capacity of the first storage space to the capacity of the memory is adjusted, the first storage space and/or the data stored in the first storage space may be further migrated, and fig. 3 schematically illustrates a flowchart for implementing migration of the first storage space and/or the data stored in the first storage space according to an embodiment of the present disclosure, where the flowchart includes:

S31: determining the capacity of the first storage space according to the first ratio;

S32: under the condition that the data volume in the memory is larger than the capacity of the first storage space, migrating the first data exceeding the capacity part of the first storage space to the second storage space; or in case the amount of data in the first storage space is smaller than the capacity of the first storage space and there is second data in the second storage space, migrating at least part of the second data in the second storage space to the first storage space.

In the process, the capacity of the first storage space is determined according to the first ratio, and when the data in the memory is larger than the capacity of the first storage space, the excess part is migrated to the second storage space, so that the condition that the first storage space always has no data overflow can be ensured; when the data volume in the first storage space is smaller than the capacity of the first storage space, the data existing in the first storage space is migrated to the first storage space, so that the data can be ensured to be stored in the first storage space preferentially.

Taking the example that the first data and the second data stored in the memory include public opinion text data, the disclosure may further cluster the public opinion text data in the memory. In order to realize the clustering of the public opinion text data, an inverted index and a positive sequence index are required to be constructed, the main key stored in the inverted index is the code of the word, and the value is a corresponding text document code set; the positive index storage primary key is a document code and the value is a corresponding text vector. Through the construction of the inverted index, candidate similar texts meeting the conditions are found first, and global matching is avoided. And the text vector to be matched is subjected to reverse index to find all texts containing the words through the word coding sequence numbers contained in the text vector. Therefore, candidate similar texts can be quickly searched, global matching of all the texts is avoided, and clustering efficiency is improved.

Fig. 4 schematically illustrates a schematic diagram of a seed memory structure and a memory management manner according to an embodiment of the present disclosure. As shown in fig. 4, the memory area includes a strong reference memory area and a soft reference memory area, wherein the strong reference memory area and the soft reference memory area store public opinion text data; the memory region may also include an index region for storing an index of first data (e.g., data stored in a strongly referenced memory region) and second data (e.g., data stored in a soft referenced memory region).

In FIG. 4, the data stored in the strongly referenced memory region includes strongly referenced cluster category information, such as the HashMap (HashMap) of FIG. 4; the data stored in the soft-reference memory region includes cluster category information for the soft reference, such as soft hash map (SoftHashMap) in fig. 4. The strongly referenced cluster category information cannot be subjected to garbage collection and can only be manually deleted; the cluster category information of the soft references may be performed garbage collection. The index area stores an inverted index, specifically, an index map (IndexMap) in fig. 4, which cannot be garbage-reclaimed.

As shown in fig. 4, a part of data is stored in the strongly referenced memory area, and the data to be eliminated is placed in the soft referenced memory area. The soft reference memory area can be recovered when the memory of the JAVA virtual machine is insufficient, so that the problem of memory overflow can be avoided. Due to the existence of the soft reference memory area, the memory data size of the clustering information can be automatically matched with the memory size of the JAVA virtual machine, the memory utilization rate is improved, and the risk of memory overflow is avoided, so that the memory management problem in the text data clustering process in the public opinion analysis field is solved.

In some embodiments, migrating first data that exceeds a capacity portion of a first storage space to a second storage space includes:

Among these, the above-mentioned predetermined data migration policies include, but are not limited to, first-in first-Out (FIFO, first In First Out first-in first-Out), least recently Used (LRU, least Recently Used), least frequently Used (LFU, least Frequently Used) policies.

As shown in fig. 4, in order to be able to reasonably use the strongly referenced memory region and the soft referenced memory region, a memory management module is introduced in some embodiments. The module can be divided into three sub-modules, namely a data migration module, a data monitoring and counting module and a dynamic proportion adjusting module (short for dynamic proportion adjusting module) of the strong soft memory area.

The data monitoring and counting module monitors the waste recycling time in real time. The proportion dynamic adjustment module dynamically adjusts a first ratio of the capacity of the strongly referenced memory area to the capacity of the whole memory according to the garbage collection time and the garbage collection time threshold value, and achieves full utilization of the memory on the premise that the garbage collection time monitored in real time does not exceed the garbage collection time threshold value. And the data migration module performs data migration according to the first ratio.

Since the data in the soft reference memory area is cleaned by the JAVA virtual machine, consistency of the inverted index memory data and the clustering information memory data needs to be maintained. To ensure the foregoing consistency, the present disclosure may further include: determining the cleared second data in the second storage space if the second data is cleared; the index is updated based on the purged second data. As shown in fig. 4, a data synchronization module is introduced in some embodiments. The data synchronization module is responsible for ensuring the data synchronization of the inverted index and the clustering information, and after the data in the soft reference memory area is cleared by the JAVA virtual machine, the callback method is called through the clustering information KEY to synchronously clear the data in the inverted index.

Further, since some data in the soft reference memory area is cleared, there is a case that clustering of the public opinion text data fails for a long period of public opinion text data. To avoid the case of cluster failure, the present disclosure may further include: storing the cleared second data in the second storage space into a database; reading the cleared second data from the database; when new data is stored in the memory, clustering the new data and the cleared second data, and if the clustering is successful, storing the clustered data in the memory; and updating the index according to the clustered data. As shown in fig. 4, in some embodiments, an incremental offline clustering module is introduced, and the above-mentioned data synchronization module records the time range of the memory obsolete data, and actively triggers the incremental offline clustering module to perform secondary clustering on the data in a specific time window.

The foregoing describes steps of the memory management method proposed in the present disclosure, and modules for performing the steps described above, in conjunction with fig. 4. The above modules are only examples, and the present disclosure may also use other related modules to perform the above steps. The steps and corresponding modules are described in detail below.

The first part, soft referenced memory region:

In some embodiments, the soft-reference memory region is used to store public opinion text clustering information. Specifically, the inverted index data is stored in a strongly referenced memory area, and the data in the memory area can only be manually deleted and cannot be actively cleaned by the JAVA virtual machine. The clustering information is stored in the strong reference memory area and the soft reference memory area, for example, data in the latest time range is stored in the strong reference memory area, the expired data is eliminated according to the LRU policy, and the expired input is stored in the soft reference memory area. The soft reference memory area can be recovered when the memory of the JAVA virtual machine is insufficient, so that the problem of memory overflow can be avoided. Due to the existence of the soft reference memory area, the memory data size of the clustering information can be automatically matched with the memory size of the JAVA virtual machine, and meanwhile, the risk of memory overflow is avoided, so that the problem of long-period text clustering memory management in the public opinion analysis field is solved.

The present disclosure may utilize a soft reference (SoftReference) object in JAVA to transfer a obsolete soft reference object in a soft reference memory region into a memory queue, and then call a get function to obtain the obsolete soft reference object in the memory queue, and then delete the obsolete soft reference object.

Since the strong reference memory area and the soft reference memory area both store public opinion text clustering information, when reading the public opinion text clustering information, the method can firstly inquire from the strong reference memory area, and if the public opinion text clustering information can be inquired, the public opinion text clustering information is directly returned; if the query is not completed, the query is performed in the soft reference memory area, and the queried public opinion text clustering information is returned. It can be seen that the partitioning of the soft-reference memory region does not prevent the reading of the public opinion text clustering information.

The second part, the memory management and control module:

The memory management and control module is used for adjusting the capacity ratio of the strong reference memory area and the soft reference memory area in the memory. Fig. 5 schematically illustrates a structure of a memory management module and a manner of adjusting a capacity ratio according to an embodiment of the disclosure. As shown in fig. 5, the memory management and control module is divided into three sub-modules, which are respectively a data migration module, a data monitoring and statistics module, and a dynamic proportion adjustment module (abbreviated as a dynamic proportion adjustment module) of the strong soft memory area. And the data migration module adopts a preset strategy to move the strong reference data with the data volume larger than the capacity of the strong reference memory area to the soft reference memory area. The soft referenced memory region may continue to access data but may be cleaned up by the JAVA virtual machine when memory usage reaches 100%. The data monitoring and counting module monitors the statistic data of capacity proportion distribution and garbage recovery time consumption of the strong and soft memory areas. The proportion dynamic adjustment module dynamically adjusts the proportion of the capacity of the memory areas with strong references and soft references through the statistical data, and can quickly adjust the memory allocation proportion on the basis of fully utilizing the memory, so that the garbage collection time is kept in a reasonable range. The aforementioned predetermined policies include, but are not limited to, first-in first-Out (FIFO, first In First Out first-in first-Out), least recently Used (LRU, least Recently Used), least frequently Used (LFU, least Frequently Used) policies.

The proportion dynamic adjustment module obtains the waste recycling time consumption, and adjusts a first ratio of the capacity of the first storage space to the capacity of the memory according to the waste recycling time consumption and a preset waste recycling time consumption threshold. FIG. 6 schematically illustrates a flow diagram for obtaining garbage collection time-consuming implementations according to an embodiment of the present disclosure, including:

S61: searching a preset first corresponding relation according to the capacity of the memory to obtain a corresponding first ratio suggested value and a garbage collection time-consuming suggested value; the first corresponding relation characterizes a first ratio recommended value and a garbage collection time-consuming recommended value corresponding to different memory capacities;

s62: and determining the garbage collection time according to the garbage collection time recommended value.

Table 1 is one example of the first correspondence relationship described above.

TABLE 1

Memory capacity

1G

2G

4G

8G

16G

32G

First ratio suggested value

50％

65％

75％

85％

90％

95％

Suggested value for waste recovery time

45ms

75ms

100ms

200ms

250ms

300ms

Taking table 1 as an example, if the memory capacity is 4G, according to the first correspondence shown in table 1, it can be found that the first recommended value of the ratio corresponding to the memory capacity of 4G is 75%, and the recommended value of the garbage collection time is 100ms.

In table 1, as the memory size increases, the first recommended value of the ratio increases from 50% to about 95%. The reason is that as the memory increases, the memory size mainly reserved for the soft reference area remains within a reasonable range under the condition that the data volume is unchanged, so that the time consumption can be stabilized within a certain threshold.

Accordingly, the present disclosure may further include a process for establishing the first correspondence, including:

and determining a first corresponding relation according to the mapping relation.

The method and the device can adopt the data monitoring statistics module to count different values of the first ratio and corresponding garbage collection time consumption statistics values under the condition of different capacity memories.

The larger the proportion of the strongly referenced memory, the longer the time it takes to recycle the full amount of garbage at a time. In short, the larger the strong reference duty ratio is, the more objects remain after one full garbage collection, so the larger the probability of scanning the root node and fragmentation of the memory is, and the time required for memory arrangement in the garbage collection process is also increased. Even more serious, the strong references occupy a large number of long-period objects, and are used up in a short time, so that the frequency of full garbage collection is significantly increased. FIG. 7 schematically illustrates a schematic diagram of the correspondence of strongly referenced memory duty cycle to garbage collection time consumption according to an embodiment of the present disclosure. As shown in fig. 7, the abscissa is the duty ratio of the strongly referenced memory, and the ordinate is the product of the actual t of single garbage collection and the frequency F of garbage collection per unit time, indicating the total time consumed for garbage collection over a period of time. As shown in fig. 7, in the case of memory size 8G, when the strongly referenced memory is more than 85%, the product of the single time consumption and the frequency of occurrence of the full garbage collection increases approximately exponentially. Therefore, in the case where the memory capacity is 8G, the 85% ratio can be regarded as the first recommended ratio value in table 1 described above. It can be seen that the first proportion suggestion value can be regarded as a data inflection point, or can be understood as a risk critical point of the system, and when the first proportion suggestion value is larger than the critical point, the garbage collection time and frequency are increased sharply, so that data backlog is finally caused, and when the garbage collection time is longer, the system avalanche can be caused in severe cases.

Thus, according to the garbage collection time consumption and the preset garbage collection time consumption threshold, the method for adjusting the first ratio of the capacity of the first storage space to the capacity of the memory may include: and under the condition that the garbage collection time is not more than the garbage collection time threshold, adjusting the first ratio to be equal to the first ratio recommended value.

Optionally, according to the garbage collection time consumption and the preset garbage collection time consumption threshold, the method for adjusting the first ratio of the capacity of the first storage space to the capacity of the memory may further include:

determining the first ratio as a value smaller than the recommended value of the first ratio under the condition that the garbage collection time is greater than the garbage collection time threshold;

determining the capacity of a first storage space according to a first ratio, and under the condition that the data volume in a memory is larger than the capacity of the first storage space, migrating first data exceeding the capacity part of the first storage space to a second storage space;

Acquiring a current actual garbage collection time consumption value from a garbage collection log, reducing the value of the first ratio under the condition that the actual garbage collection time consumption value is larger than a garbage collection time consumption threshold value, and executing the step of determining the capacity of the first storage space again based on the reduced first ratio; and ending the adjustment process until the actual value of the garbage collection time is not more than the threshold value of the garbage collection time.

In some embodiments, the above-mentioned process of determining the first ratio may use a binary search method, for example, in the above-mentioned process, determining the first ratio as a value smaller than the recommended value of the first ratio includes: determining the first ratio as half of the first ratio recommended value;

The garbage collection time consumption threshold can be determined according to the text clustering real-time requirement, if the real-time requirement is high, the garbage collection time consumption threshold can be set smaller, and if the clustering real-time requirement is not high, the threshold can be properly set larger, so that higher memory utilization rate is obtained as much as possible, and the clustering effect is improved.

In addition, in the adjustment process, the condition for ending adjustment can further comprise that the difference between the actual garbage collection time consumption value and the garbage collection time consumption threshold value is not greater than a preset threshold, so that on the premise that the actual garbage collection time consumption value is not greater than the garbage collection time consumption threshold value, a larger memory duty ratio with strong reference is set as much as possible, and the memory use effect is improved.

Optionally, ending the adjustment process includes: and ending the adjustment process until the actual value of the garbage collection time is not greater than the threshold of the garbage collection time and the difference between the actual value of the garbage collection time and the threshold of the garbage collection time is not greater than a preset threshold.

Fig. 8 schematically illustrates an implementation flowchart for adjusting a first ratio of a capacity of a first storage space to a memory capacity, in the example illustrated in fig. 8, the first storage space being a strongly referenced memory, the first ratio being referred to as a strongly referenced memory duty cycle, according to an embodiment of the present disclosure. The process shown in FIG. 8 includes the following steps

S81: the memory capacity is read.

S82: searching a first corresponding relation (as shown in table 1) according to the memory capacity, and determining a garbage collection time-consuming recommended value corresponding to the memory capacity; and determining the garbage collection time consumption (marked as t) corresponding to the current memory capacity to be equal to the garbage collection time consumption recommended value.

S83: and judging whether T is not more than a garbage collection time consumption threshold (marked as T). If T is less than or equal to T, determining the duty ratio (r) of the strongly referenced memory to be equal to a first recommended value corresponding to the memory capacity in the table 1, and outputting the duty ratio of the strongly referenced memory. If T > T, step S84 is performed.

Wherein, the garbage collection time consumption threshold (i.e. T) can be determined according to text clustering real-time requirements. For example, when different types of public opinion text data are clustered, the corresponding text clustering real-time requirements are different; therefore, the type of public opinion text data stored in the memory can be determined first, the corresponding text clustering real-time requirement is determined according to the type, and then the garbage collection time consumption threshold (namely T) is determined according to the text clustering real-time requirement.

S84: the strongly referenced memory duty cycle (i.e., r) is determined to be equal to half of the first ratio proposal value.

S85: monitoring the current waste recovery time (i.e., t).

S86: and judging whether T is not more than a garbage collection time consumption threshold (namely T). If T is less than or equal to T, executing step S87; if T > T, the strongly referenced memory duty cycle (i.e., r) is determined to be equal to half the current strongly referenced memory duty cycle, and the process returns to step S85.

S87: and judging whether the difference between T and T is not more than a preset threshold (marked as C). If the T-T is less than or equal to C, outputting the current memory duty ratio with strong reference; if T-T > C, step S88 is performed.

S88: the determination of the strongly referenced memory duty cycle (i.e., r) is determined to be equal to the intermediate value of the current strongly referenced memory duty cycle and the first ratio advice value, and the process returns to step S85.

For a specific example, when the memory capacity is 8G and the garbage collection time consumption threshold t=100 ms, the corresponding garbage collection time consumption recommended value is 200ms and the corresponding first ratio recommended value is 85% according to the memory capacity lookup table 1. The garbage collection time-consuming recommended value is greater than T, so the value of the strongly referenced memory duty cycle r is determined to be half of the first ratio recommended value, i.e., 42.5%. The actual value of the garbage collection time consumption is monitored in real time, the current actual value of the garbage collection time consumption is assumed to be 70ms, the current actual value of the garbage collection time consumption is smaller than T, but the difference (namely 30 ms) between the current actual value of the garbage collection time consumption and T is larger than a preset threshold (the preset threshold is assumed to be 10 ms). Then, the value of the strongly referenced memory duty cycle r is again determined to be equal to the median of the current strongly referenced memory duty cycle (i.e., 42.5%) and the first recommended value of the ratio (i.e., 85%), i.e., 63.75%. After the occupancy rate of the memory with the strong reference is determined to be 63.75%, the actual value of the time consuming for garbage collection is continuously monitored in real time, and whether the current actual value of the time consuming for garbage collection is larger than T is judged. And repeating the steps until the set occupancy ratio of the strongly referenced memory enables the actual value of the garbage recovery time to be smaller than T and the difference between the actual value and T is not larger than a preset threshold, ending the adjustment of the occupancy ratio of the strongly referenced memory, and outputting the finally determined occupancy ratio of the strongly referenced memory.

For another example, when the memory capacity is 8G and the garbage collection time consumption threshold t=300 ms, the corresponding garbage collection time consumption recommended value is 200ms and the corresponding first ratio recommended value is 85% according to the memory capacity lookup table 1. The garbage collection time-consuming recommended value is smaller than T, so that the value of the duty ratio r of the strongly-referenced memory is determined to be equal to the first-ratio recommended value, namely, equal to 85%, and the duty ratio of the strongly-referenced memory is output.

One reason for the above manner of setting the strongly referenced memory duty cycle is: the garbage collection time consumption is positively correlated with the duty ratio of the strongly referenced memory, and the larger the duty ratio of the strongly referenced memory is, the longer the garbage collection time consumption is. From the system performance point of view, the shorter and better the garbage recovery time is; from the standpoint of algorithm effect, the larger the memory ratio of the strong reference is, the better. Thus, the strongly referenced memory duty cycle is set essentially to balance both the algorithm effect and the system performance. As can be seen from the schematic diagram of the correspondence between the duty ratio of the strongly referenced memory and the garbage collection time consumption shown in fig. 7, in the range that the duty ratio of the strongly referenced memory is smaller than the first recommended value of the ratio, the garbage collection time consumption increases slowly with the increase of the duty ratio of the strongly referenced memory; in the range that the duty ratio of the strongly referenced memory exceeds the first proportion recommended value, the garbage collection time consumption is increased sharply along with the increase of the duty ratio of the strongly referenced memory; the first proportional advice value may thus be considered a data inflection point.

It can be seen that, in the case that the duty ratio of the strongly referenced memory is greater than the first recommended value, the increase of the duty ratio of the strongly referenced memory by a very small amount will bring about a very large increase of the actual value of the garbage collection consumption; accordingly, with the weak improvement of the algorithm effect caused by the small increase of the duty ratio of the strongly referenced memory, the system performance is greatly reduced, which is obviously inexpensiveness. Therefore, in the case that the garbage collection time-consuming recommended value is not greater than the garbage collection time-consuming threshold, the strongly referenced memory duty cycle may be set equal to the first proportional recommended value corresponding to the garbage collection time-consuming recommended value; under the condition that the garbage collection time consumption recommended value is larger than the garbage collection time consumption threshold, the duty ratio of the strong reference memory can be set to be smaller than a first proportional recommended value corresponding to the garbage collection time consumption recommended value, and the real-time monitored actual garbage collection time consumption value is enabled to be as close to the garbage collection time consumption threshold as possible.

The data synchronization module is responsible for guaranteeing data synchronization of the inverted index and the clustering information, and when data of a soft reference memory area is cleared by a JAVA virtual machine, a callback method is called through a clustering information Keyword (KEY) to synchronously clear the data in the inverted index. Specifically, embodiments of the present disclosure may use a Soft Reference (Soft Reference) object of a JAVA virtual machine, implement Soft hash mapping (SoftHashMap) by using the Soft Reference object, register the Soft Reference object cleaned by the JAVA virtual machine into a data queue, and the data synchronization module acquires the cleaned KEY by pulling the data queue. The implementation scheme of the module is based on hash mapping (HashMap) of soft reference, the embodiment of the disclosure can expand the callback function of the hash mapping (HashMap), design a callback interface, and call the callback interface according to the KEY when the soft reference object is cleared.

FIG. 9 schematically illustrates an implementation of the incremental offline clustering module in secondary clustering of eliminated data in soft-reference memory regions, according to one embodiment of the disclosure. Since some data in the soft-reference memory area is eliminated, clustering failure may occur for long-period public opinion text data. In order to solve this problem, as shown in fig. 9, the present disclosure introduces an incremental offline clustering module, records the actual range of the memory obsolete data by the data synchronization module, and actively triggers the offline clustering module to perform secondary clustering on the obsolete data of a specific time window by the data synchronization module.

Compared with the common offline clustering, the method records the data time range eliminated by the memory through the data synchronization module, and actively triggers the incremental offline clustering module to perform offline clustering on the data in the specific time range. Because the data does not need to be stored in full quantity, only the part of the data eliminated by the memory needs to be aggregated offline, the offline clustering module needs less memory, and therefore the problem of long-period text clustering effect is solved.

FIG. 10 schematically illustrates a flowchart of an implementation of the incremental offline clustering module for secondary clustering of eliminated data, according to one embodiment of the disclosure, including:

S101: the data synchronization module triggers the incremental offline clustering module to cluster.

S102: the incremental offline clustering module reads the obsolete data.

S103: when new added data is added into the memory, the incremental offline clustering module reads the new added data.

S104: and the incremental offline clustering module matches and clusters the eliminated data with the newly added data, and judges whether the eliminated data and the newly added data are clustered successfully or not. If yes, step S105 is performed; otherwise, the process returns to step S103.

S105: and storing the clustered data into a memory, and deleting information in the incremental offline clustering module.

Because the eliminated data read by the incremental offline clustering module can be deleted, the incremental offline clustering module can be ensured to only store information which is not clustered successfully, and the memory consumption is greatly reduced.

FIG. 11 schematically illustrates a flow diagram of data processed by an incremental offline clustering module according to one embodiment of the present disclosure. As shown in fig. 11, the incremental offline clustering module reads the eliminated memory data from the database, and when new data is added into the memory, the incremental offline clustering module reads the new data; and clustering the eliminated memory data and the newly added data, and if the clustering is successful, updating the clustering information into the memory data.

In summary, the invention provides four core modules, including a soft reference memory area, a memory management and control module, a data synchronization module and an incremental offline clustering module.

And the soft reference memory storage area is used for storing the clustering information. The soft reference memory area can be recovered when the memory of the JAVA virtual machine is insufficient, so that the problem of memory overflow can be avoided. Due to the existence of the soft reference memory area, the memory data size of the clustering information can be automatically matched with the memory size of the JAVA virtual machine, and meanwhile, the risk of memory overflow is avoided, so that the problem of long-period text clustering memory management in the public opinion analysis field is solved.

The memory management and control module is used for reasonably using the two memory areas with strong and soft references. The proportion of the size of the memory area with strong reference and soft reference can be quickly and dynamically adjusted by using a dichotomy through statistical data, so that the garbage collection time can be kept in a reasonable range on the basis of full utilization of the memory.

The data synchronization module is responsible for ensuring the data synchronization of the inverted index and the clustering information, and when the data of the soft reference memory area is cleared by the JAVA virtual machine, the callback method is called through the clustering information KEY to synchronously clear the data in the inverted index.

The method also introduces an increment offline clustering module, records the actual range of the memory obsolete data through the data synchronization module, and actively triggers the offline clustering module to perform secondary clustering on the data of the specific time window. Compared with the common offline clustering, the method records the data time range eliminated by the memory through the data synchronization module, and actively triggers the incremental offline clustering module to perform offline clustering on the data mirror images in the specific time range. Because the data does not need to be stored in full quantity, only the part of the data eliminated by the memory needs to be aggregated offline, the offline clustering module needs less memory, and therefore the problem of long-period text clustering effect is solved.

Exemplary Medium

Having described the method of an exemplary embodiment of the present disclosure, next, a medium of an exemplary embodiment of the present disclosure will be described with reference to fig. 12.

In some possible implementations, aspects of the present disclosure may also be implemented as a computer-readable medium having a program stored thereon, which when executed by a processor, is configured to implement the steps in a memory management method according to various exemplary embodiments of the present disclosure described in the "exemplary methods" section of the present specification.

Specifically, the processor is configured to implement the following steps when executing the program:

acquiring garbage collection time consumption, wherein the garbage collection time consumption represents the time spent for cleaning garbage objects in a memory;

and adjusting a first ratio of the capacity of the first storage space to the capacity of the memory according to the garbage collection time and a preset garbage collection time threshold.

It should be noted that: the medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

As shown in fig. 12, a medium 120 is depicted that may employ a portable compact disc read-only memory (CD-ROM) and that includes a program and that may run on a device, in accordance with an embodiment of the present disclosure. However, the disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take many forms, including, but not limited to: electromagnetic signals, optical signals, or any suitable combination of the preceding. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the context of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN).

Exemplary apparatus

Having described the media of the exemplary embodiments of the present disclosure, next, an apparatus of the exemplary embodiments of the present disclosure will be described with reference to fig. 13.

The present disclosure proposes a memory management device, where a memory managed by the device includes at least a first storage space for storing first data and a second storage space for storing second data; as shown in fig. 13, a memory management device according to an embodiment of the present disclosure may include:

The data monitoring and counting module 1310 is configured to obtain a time consumption for garbage collection, where the time consumption represents a time period spent for cleaning the garbage objects in the memory;

the dynamic proportion adjustment module 1320 is configured to adjust a first ratio of the capacity of the first storage space to the capacity of the memory according to the garbage collection time and a preset garbage collection time threshold.

In one possible implementation manner, the first storage space is a strongly referenced memory space, and the second storage space is a soft referenced memory space.

In one possible embodiment, the apparatus further includes:

A data migration module 1330, configured to determine a capacity of the first storage space according to the first ratio; under the condition that the data volume in the memory is larger than the capacity of the first storage space, migrating the first data exceeding the capacity part of the first storage space to the second storage space; or in case the amount of data in the first storage space is smaller than the capacity of the first storage space and there is second data in the second storage space, migrating at least part of the second data in the second storage space to the first storage space.

In one possible implementation, the data migration module 1330 is configured to:

In one possible implementation, the data monitoring statistics module 1310 is configured to:

In one possible implementation, the scaling dynamic adjustment module 1320 is configured to:

And under the condition that the garbage collection time is not more than the garbage collection time threshold, adjusting the first ratio to be equal to the first ratio recommended value.

Determining the capacity of the first storage space according to the first ratio, and under the condition that the data volume in the memory is larger than the capacity of the first storage space, migrating the first data exceeding the capacity part of the first storage space to the second storage space;

Acquiring a current actual garbage collection time consumption value from a garbage collection log, reducing the value of a first ratio under the condition that the actual garbage collection time consumption value is larger than a garbage collection time consumption threshold value, and executing the step of determining the capacity of the first storage space again based on the reduced first ratio; and ending the adjustment process until the actual value of the garbage collection time is not more than the threshold value of the garbage collection time.

In one possible implementation, the scaling module 1320 may determine the first ratio as half of the first ratio recommended value when determining the first ratio as a value less than the first ratio recommended value;

The dynamic scaling module 1320 adjusts the first ratio to half the original value of the first ratio when the value of the first ratio is reduced.

In one possible implementation, the above-mentioned dynamic adjustment module 1320 ends the adjustment process when the actual value of the waste recycling time is not greater than the threshold value of the waste recycling time and the difference between the actual value of the waste recycling time and the threshold value of the waste recycling time is not greater than a preset threshold.

In one possible implementation, the data monitoring statistics module 1310 is further configured to:

Under the condition of different memory capacities, different values of the first ratio and corresponding garbage collection time consumption statistics values are counted;

according to the statistical result, establishing a mapping relation between the first ratio and the garbage collection time-consuming statistical value under the condition of different memory capacities;

In one possible implementation manner, the memory further includes an index area for storing indexes of the first data and the second data.

In one possible embodiment, the apparatus further includes:

a data synchronization module 1340 for determining the purged second data in the second storage space if the second data is purged; the index is updated based on the purged second data.

In one possible implementation, the foregoing apparatus further includes an incremental offline clustering module 1350 configured to:

Storing the cleared second data in the second storage space into a database;

Reading the cleared second data from the database;

When new data is stored in the memory, clustering the new data and the cleared second data, and if the clustering is successful, storing the clustered data in the memory;

And updating the index according to the clustered data.

In one possible implementation manner, the first data and the second data include public opinion text data;

in one possible embodiment, the apparatus further includes:

the memory region clustering module 1360 is used for clustering public opinion text data in the memory;

And an analysis module 1370, configured to perform at least one of propagation path analysis, emotion data analysis, and public opinion trend analysis on the clustered data.

Exemplary computing device

Having described the methods, media, and apparatus of exemplary embodiments of the present disclosure, a computing device of exemplary embodiments of the present disclosure is next described with reference to fig. 14.

Those skilled in the art will appreciate that the various aspects of the present disclosure may be implemented as a system, method, or program product. Accordingly, various aspects of the disclosure may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.

In some possible implementations, a computing device according to embodiments of the present disclosure may include at least one processing unit and at least one storage unit. Wherein the storage unit stores program code that, when executed by the processing unit, causes the processing unit to perform steps in a memory management method according to various exemplary embodiments of the present disclosure described in the section "exemplary method" above in the present specification.

A computing device 140 according to such an implementation of the present disclosure is described below with reference to fig. 14. The computing device 140 shown in fig. 14 is merely an example and should not be taken as limiting the functionality and scope of use of embodiments of the present disclosure.

As shown in fig. 14, computing device 140 is in the form of a general purpose computing device. Components of computing device 140 may include, but are not limited to: the at least one processing unit 1401, the at least one storage unit 1402, and a bus 1403 connecting different system components (including the processing unit 1401 and the storage unit 1402).

Bus 1403 includes a data bus, a control bus, and an address bus.

The storage unit 1402 may include readable media in the form of volatile memory, such as Random Access Memory (RAM) 14021 and/or cache memory 14022, and may further include readable media in the form of non-volatile memory, such as Read Only Memory (ROM) 14023.

The storage unit 1402 may also include a program/utility 14025 having a set (at least one) of program modules 14024, such program modules 14024 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

Computing device 140 can also communicate with one or more external devices 1404 (e.g., keyboard, pointing device, etc.). Such communication may occur through an input/output (I/O) interface 1405. Moreover, computing device 140 may also communicate with one or more networks, such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 1406. As shown in FIG. 14, the network adapter 1406 communicates with other modules of the computing device 140 over a bus 1403. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with computing device 140, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

It should be noted that although in the above detailed description several units/modules or sub-units/sub-modules of the memory management device are mentioned, such a division is only exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module in accordance with embodiments of the present disclosure. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.

Furthermore, although the operations of the methods of the present disclosure are depicted in the drawings in a particular order, this is not required or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.

While the spirit and principles of the present disclosure have been described with reference to several particular embodiments, it is to be understood that this disclosure is not limited to the particular embodiments disclosed nor does it imply that features in these aspects are not to be combined to benefit from this division, which is done for convenience of description only. The disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A memory management method, wherein the memory includes at least a first storage space for storing first data and a second storage space for storing second data, the method comprising:

acquiring garbage collection time consumption, wherein the garbage collection time consumption characterizes the time spent for cleaning garbage objects in the memory;

according to the garbage collection time consumption and a preset garbage collection time consumption threshold, a first ratio of the capacity of the first storage space to the capacity of the memory is adjusted;

Wherein, it is time consuming to acquire rubbish recovery, includes: searching a preset first corresponding relation according to the capacity of the memory to obtain a corresponding first ratio suggested value and a garbage collection time-consuming suggested value; the first corresponding relation characterizes a first ratio recommended value and a garbage collection time-consuming recommended value corresponding to different memory capacities; and determining the garbage collection time consumption according to the garbage collection time consumption recommended value.

2. The method of claim 1, wherein the first storage space is a strongly referenced memory space and the second storage space is a soft referenced memory space.

3. The method according to claim 1 or 2, further comprising:

Migrating the first data exceeding the capacity part of the first storage space to the second storage space under the condition that the data amount in the memory is larger than the capacity of the first storage space; or alternatively

4. The method of claim 3, wherein the migrating the first data that exceeds the capacity portion of the first storage space to the second storage space comprises:

5. The method according to claim 1, wherein the capacity and the first storage space are adjusted according to the garbage collection time and a preset garbage collection time threshold

The first ratio of the capacity of the memory includes:

6. The method of claim 1, wherein adjusting the first ratio of the capacity of the first storage space to the capacity of the memory according to the garbage collection time and a preset garbage collection time threshold comprises:

Determining the capacity of the first storage space according to the first ratio, and migrating the first data exceeding the capacity part of the first storage space to the second storage space under the condition that the data volume in the memory is larger than the capacity of the first storage space;

7. The method of claim 6, wherein the determining the first ratio as a value less than the first ratio recommendation value comprises: determining the first ratio as half of the first ratio suggestion value;

The reducing the value of the first ratio includes: and adjusting the first ratio to be half of the original value of the first ratio.

8. The method of claim 6, wherein ending the adjustment process until the actual value of garbage collection time is not greater than the garbage collection time consumption threshold comprises:

9. The method as recited in claim 1, further comprising:

under the condition of different memory capacities, different values of the first ratio and corresponding pulls are counted

Waste recovery time-consuming statistics;

10. The method of claim 1 or 2, the memory further comprising an index area for storing an index of the first data and the second data.

11. The method as recited in claim 10, further comprising:

determining the cleared second data in the second storage space if the second data is cleared; updating the index according to the cleared second data.

12. The method as recited in claim 11, further comprising:

Storing the cleared second data in the second storage space into a database;

Reading the purged second data from the database;

And updating the index according to the clustered data.

13. The method of claim 1 or 2, wherein the first data and the second data comprise public opinion text data;

14. A memory management device, wherein the memory comprises at least a first storage space for storing first data and a second storage space for storing second data, the device comprising:

The proportion dynamic adjustment module is used for adjusting a first ratio of the capacity of the first storage space to the capacity of the memory according to the garbage collection time consumption and a preset garbage collection time consumption threshold;

The data monitoring and statistics module is further used for: searching preset according to the capacity of the memory

The first corresponding relation is used for obtaining a corresponding first comparison value suggested value and a corresponding garbage recycling time-consuming suggested value; the first corresponding relation characterizes a first ratio recommended value and a garbage collection time-consuming recommended value corresponding to different memory capacities; and determining the garbage collection time consumption according to the garbage collection time consumption recommended value.

15. The apparatus of claim 14, wherein the first storage space is a strongly referenced memory space and the second storage space is a soft referenced memory space.

16. The apparatus according to claim 14 or 15, characterized in that the apparatus further comprises:

The data migration module is used for determining the capacity of the first storage space according to the first ratio; migrating the first data exceeding the capacity part of the first storage space to the second storage space under the condition that the data amount in the memory is larger than the capacity of the first storage space; or in case the amount of data in the first storage space is smaller than the capacity of the first storage space and the second data is present in the second storage space, migrating at least part of the second data in the second storage space to the first storage space.

17. The apparatus of claim 16, wherein the data migration module is configured to:

18. The apparatus of claim 14, wherein the scaling module is configured to:

19. The apparatus of claim 18, wherein the scaling module is configured to:

acquiring a current actual garbage collection time consumption value from a garbage collection log, and under the condition that the actual garbage collection time consumption value is larger than the garbage collection time consumption threshold value, taking the value of the first ratio

Reducing, and performing the step of determining the capacity of the first storage space again based on the reduced first ratio; and ending the adjustment process until the actual garbage collection time consumption value is not greater than the garbage collection time consumption threshold value.

20. The apparatus of claim 19, wherein the scaling module determines the first ratio as half of the first ratio proposal value when determining the first ratio as a value less than the first ratio proposal value;

And when the value of the first ratio is reduced, the proportion dynamic adjustment module adjusts the first ratio to be half of the original value of the first ratio.

21. The apparatus of claim 19, wherein the dynamic scaling module ends the scaling process if the actual value of garbage collection time is not greater than the threshold of garbage collection time and the actual value of garbage collection time is not greater than a preset threshold.

22. The apparatus of claim 14, wherein the data monitoring statistics module is further configured to:

23. The apparatus of claim 14 or 15, the memory further comprising an index area for storing an index of the first data and the second data.

24. The apparatus as recited in claim 23, further comprising:

the data synchronization module is used for determining the cleared second data in the second storage space under the condition that the second data in the second storage space are cleared; updating the index according to the cleared second data.

25. The apparatus of claim 24, further comprising an incremental offline clustering module to:

Storing the cleared second data in the second storage space into a database;

Reading the purged second data from the database;

when new data is stored in the memory, the new data and the cleared second data are stored in the memory

Clustering the data, and if the clustering is successful, storing the clustered data into the memory;

And updating the index according to the clustered data.

26. The apparatus of claim 14 or 15, wherein the first data and the second data comprise public opinion text data;

The apparatus further comprises:

The memory area clustering module is used for clustering the public opinion text data in the memory;

27. A medium storing a computer program, which when executed by a processor performs the method of any one of claims 1-13.

28. A computing device, comprising:

one or more processors;

a storage means for storing one or more programs;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-13.