WO2016078388A1 - 一种数据老化方法及装置 - Google Patents
一种数据老化方法及装置 Download PDFInfo
- Publication number
- WO2016078388A1 WO2016078388A1 PCT/CN2015/080504 CN2015080504W WO2016078388A1 WO 2016078388 A1 WO2016078388 A1 WO 2016078388A1 CN 2015080504 W CN2015080504 W CN 2015080504W WO 2016078388 A1 WO2016078388 A1 WO 2016078388A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- memory data
- data
- activity
- access
- indicator table
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
Definitions
- This paper relates to the field of computer technology, and in particular to a data aging method and apparatus.
- Memory Database System is a commonly used resource management method in real-time communication devices.
- a certain process or thread accesses a certain data in the in-memory database, the data is in use, and other processes and threads cannot be used. This data is manipulated.
- the in-memory database itself does not provide a data aging function that releases unusable data.
- the aging of data generally creates an independent aging thread according to its own business logic, and calls the query interface of the database to periodically scan each data table in the database to delete the data that reaches the aging condition. Recycle the corresponding memory space.
- the above method requires a separate CPU thread to implement resource aging, which not only imposes higher requirements on CPU performance, but also easily causes lock conflict between the aging daemon thread and the service data processing thread, that is, aging thread and service data processing.
- a thread whicheverever the two, locks the memory data it is accessing, preventing another access to that memory data, affecting the normal operation of the other. Therefore, the lock conflict on the one hand reduces the system performance, on the other hand, it also causes data loneliness, that is, when the aging thread wants to access a certain memory data, but the memory data is frequently locked by the business data processing thread and cannot be accessed. The memory data cannot be released for a long time, which makes resource recovery difficult.
- this aging method must also scan the entire data table, with a long scan period and low aging efficiency.
- the embodiment of the invention provides a data aging method and device, which solves the problem that the in-memory database data in the related art has low aging efficiency, is easy to conflict, and may generate data loneliness.
- An embodiment of the present invention provides a data aging method, including: one or more program running units respectively use a memory data real-time maintenance activity indicator table, and the activity indicator table setting Arranging the index of the in-memory data into a series of data nodes according to the access activity of the in-memory data; the program running unit releases the memory whose access activity is lower than a preset access threshold according to the corresponding activity indicator table data.
- the program running unit includes a thread and/or a process.
- the one or more program running units are respectively a real-time maintenance activity indicator table of the memory data used by the program, and the method includes: when the one or more program running units first access the preset type of memory data, Creating a corresponding activity indicator table by using the preset type of memory data, and placing the accessed index of the memory data as a data node at the end of the activity indicator table; each of the one or more program running units When accessing the preset type of memory data: when the index of the accessed memory data already exists in the corresponding activity indicator table, updating the data node corresponding to the accessed memory data to correspond The activity indicator indicates the end of the table; in the case that the index of the accessed memory data is not in the corresponding activity indicator table, the data node corresponding to the accessed memory data is created, and the created data node is placed At the end of the corresponding activity indicator table.
- the activity indicator table is configured to: arrange the index of the in-memory data into a series of data nodes in descending order of access activity of the in-memory data.
- the program running unit releases the memory data whose access activity is lower than the preset access threshold according to the corresponding activity indicator table, including: each program running unit starts an aging operation separately; each program running unit Determining whether the access activity of the memory data corresponding to the head node of the corresponding activity indicator is lower than the preset access threshold: the access activity of the memory data corresponding to the head node is lower than the preset access
- the threshold the head node and its corresponding memory data are released, and it is determined whether the access activity of the memory data corresponding to the next node of the activity indicator table is lower than the preset access threshold until determining The access activity of the memory data corresponding to the node of the activity indicator is higher than or equal to the preset access threshold, and the aging operation is exited; the access to the memory data corresponding to the head node is high. If the preset access threshold is equal to or equal to the preset access threshold, the aging operation is exited.
- the embodiment of the invention further provides a data aging device, comprising: a maintenance unit, which is set to be a real-time maintenance activity indicator table of memory data used by one or more program running units, respectively.
- the activity indicator table is configured to arrange the index of the in-memory data into a series of data nodes according to the access activity of the in-memory data; and a release unit, configured to respectively respectively according to the activity indicator table corresponding to each program running unit , release memory data with access activity below the preset access threshold.
- the program running unit includes a thread and/or a process.
- the maintenance unit is configured to: when the one or more program running units first access the preset type of memory data, create a corresponding activity indicator table for the preset type of memory data, and Locating the index of the accessed memory data as a data node at the end of the activity indicator table; each time the one or more program running units access the preset type of memory data: being accessed Updating the data node corresponding to the accessed memory data to the end of the corresponding activity indicator table in the case where the index of the memory data is already present in the corresponding activity indicator table; in the memory data being accessed When the index is not in the corresponding activity indicator table, the data node corresponding to the accessed memory data is created, and the created data node is placed at the end of the corresponding activity indicator table.
- the activity indicator table is configured to arrange the indexes of the memory data into a series of data nodes in descending order of access vitality of the memory data.
- the releasing unit is configured to: start an aging operation for each program running unit separately; determine, for each program running unit, whether the access activity level of the memory data corresponding to the head node of the corresponding activity indicator table is If the access activity level of the memory data corresponding to the head node is lower than the preset access threshold, the head node and its corresponding memory data are released, and the determination continues to be performed.
- the access activity of the memory data corresponding to the next node of the activity indicator table is lower than the preset access threshold, until it is determined that the access activity of the memory data corresponding to one node of the activity indicator table is higher than or And the aging operation is exited when the access activity of the memory data corresponding to the head node is higher than or equal to the preset access threshold.
- the embodiment of the invention further provides a computer readable storage medium storing program instructions, which can be implemented when the program instructions are executed.
- the data aging method and device provided by the embodiments of the present invention the program running units are respectively made by themselves
- the memory data is used to maintain the activity indicator in real time, and then the memory data whose activity is lower than the preset access threshold is released according to the corresponding activity indicator table, so that the aging program is executed inside each program running unit on the one hand.
- the data nodes in the activity indicator table are arranged according to the access activity of their corresponding memory data, it can be easily Determining the first data node that does not need to be released and demarcating the node to determine all the data nodes that need not be released, thereby directly exiting the aging process, so that the aging process can be completed without scanning the entire in-memory database. Greatly improved the aging efficiency.
- FIG. 1 is a flowchart of a data aging method according to an embodiment of the present invention
- FIG. 2 is a schematic diagram of a resource organization manner of an activity indicator table according to an embodiment of the present invention
- FIG. 3 is a flowchart of creating an activity indicator table in an embodiment of the present invention.
- FIG. 5 is a flowchart of fast aging data by using an activity indicator table according to an embodiment of the present invention
- FIG. 6 is a schematic structural diagram of a data aging apparatus according to an embodiment of the present invention.
- an embodiment of the present invention provides a data aging method, including:
- the one or more program running units are respectively a memory data real-time maintenance activity indicator table used by the user, and the activity indicator table is set to arrange the index of the memory data according to the access activity of the memory data into a series of Data node
- the program running unit releases the memory data whose access activity is lower than a preset access threshold according to the corresponding activity indicator table.
- the activity indicator table maintained by the program running unit is the activity level corresponding to the running unit of the program. Instructions table.
- Each program running unit may correspond to one activity indicator table, or may correspond to multiple activity indicator tables.
- the program running unit is a real-time maintenance activity indicator table of the memory data used by the program, and then releasing the memory data whose access activity is lower than the preset access threshold according to the corresponding activity indicator table.
- the program running unit is a real-time maintenance activity indicator table of the memory data used by the program, and then releasing the memory data whose access activity is lower than the preset access threshold according to the corresponding activity indicator table.
- the program running unit refers to a unit that can complete a certain function relatively independently, and may include a process or a thread.
- a program can include one or more program execution units, and each program execution unit may need to call multiple data during the run.
- the user needs to query the phone bill detail list, which may include a user thread for querying the user and a cost thread for querying the fee.
- the user thread needs to access the user identity database, and the fee thread needs to access the SMS billing database, the telephone billing database, and the network flow meter.
- Fee database and other value-added business databases which can reside in memory to form an in-memory database.
- an in-memory database can be either an internal component or an external component combined with an internal component.
- the frequency of each memory data item in each type of in-memory data can be expressed by the access activity. The higher the frequency of the call, the more active the memory data. According to the user's frequent attention to different frequency ranges, the access activity also has multiple expressions. After an in-memory data is accessed, its corresponding access activity will be updated accordingly.
- each program running unit is a real-time maintenance activity indicator table for the memory data used by the user, that is, the user thread maintains an activity indicator table for the user identity database, and the query thread respectively corresponds to the corresponding SMS billing database.
- Telephone billing database, network traffic billing database and other value-added service database maintenance activity indicator can be as shown in FIG. 2.
- the jump indicator can include:
- the one or more program running units When the one or more program running units first access the preset type of memory data, create a corresponding activity indicator table for the preset type of memory data, and use the index of the accessed memory data as a data node. Placed at the end of the activity indicator table;
- the end of the activity indicator table is the index of the memory data with the highest access activity.
- the program of the user querying the credit is still taken as an example.
- a corresponding activity indicator table is created for the database, and the index of the user identity data to be accessed is obtained (for example, the index number is 22). ) is placed as a data node at the end of the activity indicator.
- the user thread may also access other user identity data, and the corresponding index number is, for example, 8, 14 in sequence, then, when accessing the data corresponding to index numbers 8 and 14, it will also be 8 and The corresponding data nodes are placed in turn at the end of the activity indicator.
- the data node corresponding to the sequence number 22 is placed at the end of the activity indicator table, so as to reciprocate so that the data nodes arranged at the end of the activity indicator table correspond to Memory data is always the most active. That is to say, the activity indicator table arranges the indexes of the memory data into a series of data nodes in descending order of the access activity of the memory data.
- the query thread maintains one or more such activity indicator tables in a similar manner.
- the activity indicator table may also arrange the indexes of the memory data into a series of data nodes in descending order of the access activity of the memory data, as long as It is relatively easy to determine the first data node that does not need to be released and use the node as a boundary to determine all data nodes that do not need to be released.
- the program running unit releases the memory data whose access activity is lower than the preset access threshold according to the corresponding activity indicator table, and may include:
- Each of the program running units respectively initiates an aging operation
- Each of the program running units determines whether the access activity of the memory data corresponding to the head node of the corresponding activity indicator table is lower than the preset access threshold:
- the aging operation is exited.
- the user thread and the query thread may respectively start the aging operation after performing the corresponding operation, and determine whether the head node access activity of the corresponding activity indicator table is lower than the The access threshold is preset to determine if the corresponding memory data needs to be released.
- the access activity of the memory data corresponding to the head node is 0 times in the last 1 minute, and once in the last 2 minutes, the head node corresponds to The access activity of the memory data is lower than the preset access threshold, thereby releasing the head node and its corresponding memory data, and using the next data node of the head node as a new head node, and continuing to determine that the new head node corresponds to Whether the access activity of the memory data is lower than the preset access threshold. If the access activity of the memory data corresponding to the new head node is accessed once in the last 30 seconds, and the access activity is higher than the preset access threshold, the new head node may be determined later.
- the access activity of the memory data corresponding to the data node is higher than the preset access threshold, so it is not necessary to examine the remaining data nodes and their corresponding memory data one by one, and directly exit the aging process.
- the access activity of the memory data corresponding to the data node is measured by the reciprocal of the current time interval and the time interval of the last access to the memory data, and other indicators that reflect the access activity of the data or resources, such as the number of historical visits, average The access interval, average access frequency, etc. can also be included in the visit activity survey system after different weighting.
- the activity indicator table can create a data node by the following steps.
- Step 201 Create a data node by using a database interface.
- Step 202 the thread activity indicator table is empty, then the current data node is the first node of the table;
- step 203 if the activity indicator of the thread is not empty, the newly created data node has the highest activity according to the activity definition, and moves it to the end of the activity indicator.
- the activity indicator table can be updated as follows.
- Step 301 querying a data node by using a database interface
- Step 302 According to the activity definition, the newly queried data node has the highest activity and moves it to the end of the activity indicator.
- the position in the activity indicator table is adjusted, which is equivalent to dynamically implementing the activity ordering of the data nodes. Nodes with low access frequencies will slowly focus on the header location of the indicator table.
- a thread or process can perform rapid aging of resources as follows.
- Step 401 Acquire an activity indicator head node on the thread, start scanning from the head node, and define, according to the activity indicator table, the head node is the least active node;
- Step 402 the current node is not empty and reaches the aging condition, if yes, the database interface is called to delete the current node, if not, then jump to step 404;
- step 403 the head node of the current indication table is updated to be the next node, and then jumps to step 401 to continue scanning of the next node.
- Step 404 If the current header node is empty, the current indication table is empty, and the current aging process is exited.
- the header node is the least active node according to the activity indicator table. Therefore, the subsequent nodes will not reach the aging condition, and the current aging process can be directly exited.
- data activity indicator table in the foregoing embodiment may instantiate multiple copies according to service requirements in each thread, and each data table type may correspond to one activity indicator table.
- the embodiment of the present invention further provides a data aging device, including:
- the maintenance unit 60 is configured to respectively be a memory data real-time maintenance activity indicator table used by one or more program running units, and the activity indicator table is set to actively register the index of the memory data according to the memory data. Degrees are arranged into a series of data nodes;
- the release unit 62 is configured to release the memory data whose access activity is lower than the preset access threshold according to the activity indicator table corresponding to each program running unit maintained by the maintenance unit 60.
- the maintenance unit 60 can respectively maintain a live activity indication table for the memory data used by one or more program running units, and the release unit 62 can release the access activity according to the activity indicator table.
- Memory data below the preset access threshold thus, on the one hand, by performing the aging procedure inside each program running unit, the lock conflict and data orphan caused by running a separate aging thread are effectively avoided; on the other hand, active
- the data nodes in the degree indicator table are arranged according to the access activity of the corresponding memory data, so that the first data node that does not need to be released can be easily determined and the node is used as a boundary to determine that all the information is not needed to be released.
- the data node directly exits the aging program, so that the aging process can be completed without scanning the entire in-memory database, which greatly improves the aging efficiency.
- the program running unit refers to a unit that can complete a certain function relatively independently, and may include a process or a thread.
- a program can include one or more program execution units, and each program execution unit may need to call multiple data during the run.
- the maintenance unit 60 can be configured to:
- the one or more program running units When the one or more program running units first access the preset type of memory data, create a corresponding activity indicator table for the preset type of memory data, and access the accessed memory data.
- the index is placed as a data node at the end of the activity indicator table;
- the activity indicator table is configured to arrange the indexes of the memory data into a series of data nodes in descending order of access vitality of the memory data.
- the release unit 62 can be configured to:
- the aging operation is exited.
- the program running unit is a real-time maintenance activity indicator table of the memory data used by the program, and then releasing the access activity level lower than the preset access threshold according to the corresponding activity indicator table.
- Memory data so on the one hand, by performing the aging program inside each program running unit, the lock conflict and data orphan point caused by running a separate aging thread are effectively avoided; on the other hand, due to the data node in the activity indicator table It is arranged according to the access activity of its corresponding memory data, so it can be easily determined that the first data node that does not need to be released is demarcated by the node, and all the data nodes that do not need to be released are determined, thereby directly exiting The aging process, so that the aging process can be completed without scanning the entire in-memory database, greatly improving the aging efficiency.
Abstract
一种数据老化方法及装置，所述方法包括：一个或多个程序运行单位分别为自身使用的内存数据实时维护活跃度指示表，所述活跃度指示表设置为将所述内存数据的索引按照所述内存数据的访问活跃度排列成一系列数据节点；所述程序运行单位根据对应的所述活跃度指示表，释放访问活跃度低于预设访问阈值的内存数据。
Description
本文涉及计算机技术领域，特别是涉及一种数据老化方法及装置。
内存数据库(Memory Database System)是目前实时通信设备中常用的资源管理方式，当有某个进程或线程在访问内存数据库中的某个数据时，该数据处于使用状态，其他的进程和线程不能对该数据进行操作。但内存数据库本身并不提供将不常用的数据释放的数据老化功能。
相关技术中，数据(或资源)的老化一般由各个产品根据自身的业务逻辑创建一个独立的老化线程，通过调用数据库的查询接口，循环扫描数据库中的各个数据表，删除达到老化条件的数据，回收相应的内存空间。
然而，上述方法由于需要占用独立的CPU线程来实现资源老化，不但对CPU性能提出了更高的要求，还容易使老化守护线程与业务数据处理线程发生锁冲突，即，老化线程与业务数据处理线程，二者无论哪个，都会对其正在访问的内存数据加锁，从而阻止另一个对该内存数据的访问，影响另一个的正常运行。因此，锁冲突一方面降低了系统性能，另一方面还会引起数据孤点，即，当老化线程要访问某个内存数据，但该内存数据却被业务数据处理线程频繁加锁而无法访问时，该内存数据就长期无法被释放，导致资源回收困难。另外，这种老化方法还必须扫描整个数据表，扫描周期长，老化效率低。
发明内容
本发明实施例提供一种数据老化方法及装置，解决相关技术中内存数据库数据老化效率低、容易冲突且可能会产生数据孤点的问题。
本发明实施例提供一种数据老化方法，包括：一个或多个程序运行单位分别为自身使用的内存数据实时维护活跃度指示表，所述活跃度指示表设置
为将所述内存数据的索引按照所述内存数据的访问活跃度排列成一系列数据节点；所述程序运行单位根据对应的所述活跃度指示表，释放访问活跃度低于预设访问阈值的内存数据。
可选的，所述程序运行单位包括线程和/或进程。
可选的，所述一个或多个程序运行单位分别为自身使用的内存数据实时维护活跃度指示表，包括：所述一个或多个程序运行单位初次访问预设种类的内存数据时，为所述预设种类的内存数据创建对应的活跃度指示表，并将访问的所述内存数据的索引作为一个数据节点放置在所述活跃度指示表的末尾；所述一个或多个程序运行单位每次访问所述预设种类的内存数据时：在被访问的内存数据的索引已经存在于所述对应的活跃度指示表的情况下，将所述被访问的内存数据对应的数据节点更新到对应的活跃度指示表的末尾；在被访问的内存数据的索引不在所述对应的活跃度指示表的情况下，创建所述被访问的内存数据对应的数据节点，将创建的所述数据节点放置在所述对应的活跃度指示表的末尾。
可选的，所述活跃度指示表是设置为：将所述内存数据的索引按照所述内存数据的访问活跃度由低到高的顺序排列成一系列数据节点。
可选的，所述程序运行单位根据对应的所述活跃度指示表，释放访问活跃度低于预设访问阈值的内存数据，包括：每个程序运行单位分别启动老化操作；每个程序运行单位确定对应的所述活跃度指示表的头节点对应的内存数据的访问活跃度是否低于所述预设访问阈值：在所述头节点对应的内存数据的访问活跃度低于所述预设访问阈值的情况下，释放所述头节点及其对应的内存数据，并继续确定所述活跃度指示表的下一个节点对应的内存数据的访问活跃度是否低于所述预设访问阈值，直至确定出所述活跃度指示表的一个节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值，退出所述老化操作为止；在所述头节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值的情况下，退出所述老化操作。
本发明实施例还提供一种数据老化装置，包括：维护单元，设置为分别为一个或多个程序运行单位自身使用的内存数据实时维护活跃度指示表，所
述活跃度指示表设置为将所述内存数据的索引按照所述内存数据的访问活跃度排列成一系列数据节点；以及释放单元，设置为分别根据每个程序运行单位对应的所述活跃度指示表，释放访问活跃度低于预设访问阈值的内存数据。
可选的，所述程序运行单位包括线程和/或进程。
可选的，所述维护单元是设置为：在所述一个或多个程序运行单位初次访问预设种类的内存数据时，为所述预设种类的内存数据创建对应的活跃度指示表，并将访问的所述内存数据的索引作为一个数据节点放置在所述活跃度指示表的末尾；在所述一个或多个程序运行单位每次访问所述预设种类的内存数据时：在被访问的内存数据的索引已经存在于所述对应的活跃度指示表的情况下，将所述被访问的内存数据对应的数据节点更新到对应的活跃度指示表的末尾；在被访问的内存数据的索引不在所述对应的活跃度指示表的情况下，创建所述被访问的内存数据对应的数据节点，将创建的所述数据节点放置在所述对应的活跃度指示表的末尾。
可选的，所述活跃度指示表，是设置为将所述内存数据的索引按照所述内存数据的访问活跃度由低到高的顺序排列成一系列数据节点。
可选的，所述释放单元是设置为：为每个程序运行单位分别启动老化操作；为每个程序运行单位确定对应的所述活跃度指示表的头节点对应的内存数据的访问活跃度是否低于所述预设访问阈值：在所述头节点对应的内存数据的访问活跃度低于所述预设访问阈值的情况下，释放所述头节点及其对应的内存数据，并继续确定所述活跃度指示表的下一个节点对应的内存数据的访问活跃度是否低于所述预设访问阈值，直至确定出所述活跃度指示表的一个节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值，退出所述老化操作为止；在所述头节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值的情况下，退出所述老化操作。
本发明实施例还提供一种计算机可读存储介质，存储有程序指令，当该程序指令被执行时可实现上述方法。
本发明实施例提供的数据老化方法及装置，程序运行单位分别为自身使
用的内存数据实时维护活跃度指示表，然后根据对应的所述活跃度指示表释放访问活跃度低于预设访问阈值的内存数据，这样，一方面通过在每个程序运行单位内部执行老化程序，有效避免了运行单独的老化线程时引起的锁冲突和数据孤点；另一方面，由于活跃度指示表中的数据节点是按照其对应的内存数据的访问活跃度排列的，因此可以很容易地确定出第一个无需进行释放的数据节点并以该节点为分界，确定出所有不需释放的数据节点，从而直接退出老化程序，这样，无需对整个内存数据库进行扫描即可完成老化程序，大大提高了老化效率。
附图概述
图1是本发明实施例提供的数据老化方法的一种流程图；
图2是本发明实施例中活跃度指示表的资源组织方式的一种示意图；
图3是本发明实施例中活跃度指示表的创建流程图；
图4是本发明实施例中活跃度指示表的更新流程图；
图5是本发明实施例中利用活跃度指示表进行数据快速老化流程图；
图6是本发明实施例提供的数据老化装置的一种结构示意图。
需要说明的是，在不冲突的情况下，本文中实施例以及实施例中的特征可以相互任意组合。
如图1所示，本发明实施例提供一种数据老化方法，包括：
S11，一个或多个程序运行单位分别为自身使用的内存数据实时维护活跃度指示表，所述活跃度指示表设置为将所述内存数据的索引按照所述内存数据的访问活跃度排列成一系列数据节点；
S12，所述程序运行单位根据对应的所述活跃度指示表，释放访问活跃度低于预设访问阈值的内存数据。
程序运行单位所维护的活跃度指示表为该程序运行单位所对应的活跃度
指示表。每个程序运行单位可以对应一个活跃度指示表，也可以对应多个活跃度指示表。
本发明实施例提供的数据老化方法，程序运行单位分别为自身使用的内存数据实时维护活跃度指示表，然后根据对应的所述活跃度指示表释放访问活跃度低于预设访问阈值的内存数据，这样，一方面通过在每个程序运行单位内部执行老化程序，有效避免了运行单独的老化线程时引起的锁冲突和数据孤点；另一方面，由于活跃度指示表中的数据节点是按照其对应的内存数据的访问活跃度排列的，因此可以很容易地确定出第一个无需进行释放的数据节点并以该节点为分界，确定出所有不需释放的数据节点，从而直接退出老化程序，这样，无需对整个内存数据库进行扫描即可完成老化程序，大大提高了老化效率。
需要说明的是，本发明实施例中，程序运行单位是指可以相对独立的完成一定功能的单位，可以包括进程或线程。一个程序可以包括一个或多个程序运行单位，每个程序运行单位在运行过程中可能需要调用多种数据。例如用户需要查询话费详单，可能包括一个查询用户的用户线程和一个查询费用的费用线程，用户线程需要访问用户身份数据库，而费用线程需要访问短信计费数据库，电话计费数据库，网络流量计费数据库和其他增值业务数据库，这些数据库都可以驻留内存从而形成内存数据库。内存数据库既可以为内部组件的方式也可以为外部组件与内部组件组合的方式。其中，每一类内存数据中的每一个内存数据项被调用的频繁程度都可以用访问活跃度来表示，被调用频繁程度越高，说明该项内存数据越活跃。根据用户对不同时间范围内的频繁程度关注的不同，访问活跃度也具有多种表达方式，一项内存数据被访问后，其对应的访问活跃度至也会相应更新。
步骤S11中，每个程序运行单位分别为自身使用的内存数据实时维护活跃度指示表，也就是用户线程为用户身份数据库维护一个活跃度指示表，查询线程分别为其对应的短信计费数据库、电话计费数据库、网络流量计费数据库和其他增值业务数据库维护活跃度指示表。数据活跃度指示表的资源组织方式可如图2所示。
其中，一个或多个程序运行单位分别为自身使用的内存数据实时维护活
跃度指示表可包括：
所述一个或多个程序运行单位初次访问预设种类的内存数据时，为所述预设种类的内存数据创建对应的活跃度指示表，并将访问的所述内存数据的索引作为一个数据节点放置在所述活跃度指示表的末尾；
所述一个或多个程序运行单位每次访问所述预设种类的内存数据时：
在被访问的内存数据的索引已经存在于所述对应的活跃度指示表的情况下，将所述被访问的内存数据对应的数据节点更新到对应的活跃度指示表的末尾；
在被访问的内存数据的索引不在所述对应的活跃度指示表的情况下，创建所述被访问的内存数据对应的数据节点，将创建的所述数据节点放置在所述对应的活跃度指示表的末尾。
如果活跃度指示表中所述内存数据的索引按照所述内存数据的访问活跃度由低到高的顺序排列，则在活跃度指示表的末尾是访问活跃度最高的内存数据的索引。
仍以用户查询话费的程序为例进行说明，当用户线程初次访问用户身份数据库时，为该数据库创建一个对应的活跃度指示表，并且将访问到的用户身份数据的索引(如索引序号为22)作为一个数据节点放置在该活跃度指示表的末尾。随着程序的运行，该用户线程还可能会访问其他的用户身份数据，其对应的索引序号例如依次为8、14，那么，访问到索引序号8和14对应的数据时，也会将8和14对应的数据节点依次放置到活跃度指示表的末尾。当线程再次访问到序号为22的用户身份数据后，又会将该序号22对应的数据节点放置在活跃度指示表的末尾，如此往复，以使排列在活跃度指示表末尾的数据节点对应的内存数据总是最活跃的。也就是说，活跃度指示表将内存数据的索引按照所述内存数据的访问活跃度由低到高的顺序排列成一系列数据节点。同样道理，查询线程也以类似的方式维护着一个或多个这样的活跃度指示表。
当然，在其他实施例中，活跃度指示表也可以将内存数据的索引按照所述内存数据的访问活跃度由高到低的顺序排列成一系列数据节点，只要能够
比较容易地确定出第一个无需进行释放的数据节点并以该节点为分界，确定出所有不需释放的数据节点即可。
可选的，在步骤S12中，所述程序运行单位根据对应的所述活跃度指示表，释放访问活跃度低于预设访问阈值的内存数据，可包括：
每个所述程序运行单位分别启动老化操作；
每个所述程序运行单位确定对应的所述活跃度指示表的头节点对应的内存数据的访问活跃度是否低于所述预设访问阈值：
在所述头节点对应的内存数据的访问活跃度低于所述预设访问阈值的情况下，释放所述头节点及其对应的内存数据，并继续确定所述活跃度指示表的下一个节点对应的内存数据的访问活跃度是否低于所述预设访问阈值，直至确定出所述活跃度指示表的一个节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值，退出所述老化操作为止；
在所述头节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值的情况下，退出所述老化操作。
在活跃度指示表中或者其他位置记录访问每个内存数据时的时间戳信息(即访问的时间)，并且使用相应的内存数据的索引可以查询到对应的内存数据的被访问的时间戳信息，在判断访问活跃度是否低于预设访问阈值时，根据该时间戳信息进行判断。
例如，对于上述实施例中的话费查询程序，用户线程和查询线程在执行完相应的操作后，可以分别启动老化操作，确定自己对应的活跃度指示表的头节点访问活跃度是否低于所述预设访问阈值，从而确定是否需要释放相应的内存数据。例如，如果预设访问阈值为最近1分钟内访问过1次，头节点对应的内存数据的访问活跃度为最近1分钟内访问过0次，最近2分钟内访问过1次，头节点对应的内存数据的访问活跃度低于预设访问阈值，从而释放头节点及其对应的内存数据，并将所述头节点的下一个数据节点作为新的头节点，继续确定所述新的头节点对应的内存数据的访问活跃度是否低于所述预设访问阈值。若新的头节点对应的内存数据的访问活跃度为最近30秒内访问过一次，访问活跃度高于预设访问阈值，则可以确定该新的头节点以后
的数据节点对应的内存数据的访问活跃度都高于预设访问阈值，因此无需再逐个考察剩余的数据节点及其对应的内存数据，直接退出老化程序即可。本实施例中，数据节点对应的内存数据的访问活跃度采用当前时刻与最近一次访问该内存数据的时间间隔的倒数来计量，其他体现数据或资源的访问活跃度的指标如历史访问次数、平均访问间隔、平均访问频率等也可以经过不同的加权后纳入访问活跃度考察体系。
下面对上述实施例中的数据老化方法进行详细说明。
如图3所示，活跃度指示表可以按照以下步骤创建数据节点。
步骤201，使用数据库接口创建一个数据节点；
步骤202，线程的活跃度指示表为空，那么将当前数据节点作为表的第一个节点；
步骤203，线程的活跃度指示表不为空，则根据活跃度定义，新创建的数据节点的活跃度最高，将其移动到活跃度指示表尾部。
如图4所示，活跃度指示表可以按照以下步骤更新。
步骤301，使用数据库接口查询到一个数据节点；
步骤302，根据活跃度定义，新查询的数据节点活跃度最高，将其移动到活跃度指示表尾部。
本实施例中，数据节点查询后即调整其在活跃度指示表中的位置，相当于动态实现了数据节点的活跃度排序。访问频率低的节点则会慢慢集中于指示表的表头位置。
如图5所示，线程或进程可以按照如下步骤进行资源的快速老化。
步骤401，获取线程上活跃度指示表头节点，从头节点开始扫描，根据活跃度指示表定义，头节点是最不活跃的节点；
步骤402，当前节点不为空且达到老化条件，如果是，则调用数据库接口删除当前节点，如果否，则跳转到步骤404；
步骤403，更新当前指示表的头节点为下一个节点，然后跳转到步骤401继续下一个节点的扫描。
步骤404，如果当前表头节点为空，说明当前指示表为空，退出当前老化流程即可。
如果当前表头节点没有达到老化条件，根据活跃度指示表定义，表头节点为最不活跃节点，因此后续节点必然均不会达到老化条件，则可直接退出当前老化流程。
需要说明的是，上述实施例中的数据活跃度指示表，在每个线程中可以根据业务需要实例化多份，每一个数据表种类可以对应一个活跃度指示表。
相应的，如图6所示，本发明实施例还提供一种数据老化装置，包括：
维护单元60，设置为分别为一个或多个程序运行单位自身使用的内存数据实时维护活跃度指示表，所述活跃度指示表设置为将所述内存数据的索引按照所述内存数据的访问活跃度排列成一系列数据节点；以及
释放单元62，设置为分别根据维护单元60维护的每个程序运行单位对应的所述活跃度指示表，释放访问活跃度低于预设访问阈值的内存数据。
本发明实施例提供的数据老化装置，维护单元60能够分别为一个或多个程序运行单位自身使用的内存数据实时维护活跃度指示表，释放单元62能够根据所述活跃度指示表释放访问活跃度低于预设访问阈值的内存数据，这样，一方面通过在每个程序运行单位内部执行老化程序，有效避免了运行单独的老化线程时引起的锁冲突和数据孤点；另一方面，由于活跃度指示表中的数据节点是按照其对应的内存数据的访问活跃度排列的，因此可以很容易地确定出第一个无需进行释放的数据节点并以该节点为分界，确定出所有不需释放的数据节点，从而直接退出老化程序，这样，无需对整个内存数据库进行扫描即可完成老化程序，大大提高了老化效率。
需要说明的是，本实施例中，程序运行单位是指可以相对独立的完成一定功能的单位，可以包括进程或线程。一个程序可以包括一个或多个程序运行单位，每个程序运行单位在运行过程中可能需要调用多种数据。
可选的，维护单元60可设置为：
在所述一个或多个程序运行单位初次访问预设种类的内存数据时，为所述预设种类的内存数据创建对应的活跃度指示表，并将访问的所述内存数据
的索引作为一个数据节点放置在所述活跃度指示表的末尾；
在所述一个或多个程序运行单位每次访问所述预设种类的内存数据时：
在被访问的内存数据的索引已经存在于所述对应的活跃度指示表的情况下，将所述被访问的内存数据对应的数据节点更新到对应的活跃度指示表的末尾；
在被访问的内存数据的索引不在所述对应的活跃度指示表的情况下，创建所述被访问的内存数据对应的数据节点，将创建的所述数据节点放置在所述对应的活跃度指示表的末尾。
可选的，所述活跃度指示表，是设置为将所述内存数据的索引按照所述内存数据的访问活跃度由低到高的顺序排列成一系列数据节点。
可选的，释放单元62可设置为：
为每个程序运行单位分别启动老化操作；
为每个程序运行单位确定对应的所述活跃度指示表的头节点访问活跃度是否低于所述预设访问阈值：
在所述头节点对应的内存数据的访问活跃度低于所述预设访问阈值的情况下，释放所述头节点及其对应的内存数据，并继续确定所述活跃度指示表的下一个节点对应的内存数据的访问活跃度是否低于所述预设访问阈值，直至确定出所述活跃度指示表的一个节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值，退出所述老化操作为止；
在所述头节点访问活跃度高于或等于所述预设访问阈值的情况下，退出所述老化操作。
本领域普通技术人员可以理解上述方法中的全部或部分步骤可通过程序来指令相关硬件完成，上述程序可以存储于计算机可读存储介质中，如只读存储器、磁盘或光盘等。可选地，上述实施例的全部或部分步骤也可以使用一个或多个集成电路来实现。相应地，上述实施例中的各模块/单元可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。本发明实施例不限制于任何特定形式的硬件和软件的结合。
本发明实施例提供的数据老化方法及装置，程序运行单位分别为自身使用的内存数据实时维护活跃度指示表，然后根据对应的所述活跃度指示表释放访问活跃度低于预设访问阈值的内存数据，这样，一方面通过在每个程序运行单位内部执行老化程序，有效避免了运行单独的老化线程时引起的锁冲突和数据孤点；另一方面，由于活跃度指示表中的数据节点是按照其对应的内存数据的访问活跃度排列的，因此可以很容易地确定出第一个无需进行释放的数据节点并以该节点为分界，确定出所有不需释放的数据节点，从而直接退出老化程序，这样，无需对整个内存数据库进行扫描即可完成老化程序，大大提高了老化效率。
Claims (11)
- 一种数据老化方法，包括：一个或多个程序运行单位分别为自身使用的内存数据实时维护活跃度指示表，所述活跃度指示表设置为将所述内存数据的索引按照所述内存数据的访问活跃度排列成一系列数据节点；所述程序运行单位根据对应的所述活跃度指示表，释放访问活跃度低于预设访问阈值的内存数据。
- 根据权利要求1所述的方法，其中，所述程序运行单位包括线程和/或进程。
- 根据权利要求1或2所述的方法，其中，所述一个或多个程序运行单位分别为自身使用的内存数据实时维护活跃度指示表，包括：所述一个或多个程序运行单位初次访问预设种类的内存数据时，为所述预设种类的内存数据创建对应的活跃度指示表，并将访问的所述内存数据的索引作为一个数据节点放置在所述活跃度指示表的末尾；所述一个或多个程序运行单位每次访问所述预设种类的内存数据时：在被访问的内存数据的索引已经存在于所述对应的活跃度指示表的情况下，将所述被访问的内存数据对应的数据节点更新到对应的活跃度指示表的末尾；在被访问的内存数据的索引不在所述对应的活跃度指示表的情况下，创建所述被访问的内存数据对应的数据节点，将创建的所述数据节点放置在所述对应的活跃度指示表的末尾。
- 根据权利要求1或2或3所述的方法，其中，所述活跃度指示表是设置为：将所述内存数据的索引按照所述内存数据的访问活跃度由低到高的顺序排列成一系列数据节点。
- 根据权利要求4所述的方法，其中，所述程序运行单位根据对应的所述活跃度指示表，释放访问活跃度低于预设访问阈值的内存数据，包括：每个程序运行单位分别启动老化操作；每个程序运行单位确定对应的所述活跃度指示表的头节点对应的内存数据的访问活跃度是否低于所述预设访问阈值：在所述头节点对应的内存数据的访问活跃度低于所述预设访问阈值的情况下，释放所述头节点及其对应的内存数据，并继续确定所述活跃度指示表的下一个节点对应的内存数据的访问活跃度是否低于所述预设访问阈值，直至确定出所述活跃度指示表的一个节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值，退出所述老化操作为止；在所述头节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值的情况下，退出所述老化操作。
- 一种数据老化装置，包括：维护单元，设置为分别为一个或多个程序运行单位自身使用的内存数据实时维护活跃度指示表，所述活跃度指示表设置为将所述内存数据的索引按照所述内存数据的访问活跃度排列成一系列数据节点；以及释放单元，设置为分别根据每个程序运行单位对应的所述活跃度指示表，释放访问活跃度低于预设访问阈值的内存数据。
- 根据权利要求6所述的装置，其中，所述程序运行单位包括线程和/或进程。
- 根据权利要求6或7所述的装置，所述维护单元是设置为：在所述一个或多个程序运行单位初次访问预设种类的内存数据时，为所述预设种类的内存数据创建对应的活跃度指示表，并将访问的所述内存数据的索引作为一个数据节点放置在所述活跃度指示表的末尾；在所述一个或多个程序运行单位每次访问所述预设种类的内存数据时：在被访问的内存数据的索引已经存在于所述对应的活跃度指示表的情况下，将所述被访问的内存数据对应的数据节点更新到对应的活跃度指示表的末尾；在被访问的内存数据的索引不在所述对应的活跃度指示表的情况下，创建所述被访问的内存数据对应的数据节点，将创建的所述数据节点放置在所 述对应的活跃度指示表的末尾。
- 根据权利要求6或7或8所述的装置，其中，所述活跃度指示表，是设置为将所述内存数据的索引按照所述内存数据的访问活跃度由低到高的顺序排列成一系列数据节点。
- 根据权利要求9所述的装置，所述释放单元是设置为：为每个程序运行单位分别启动老化操作；为每个程序运行单位确定对应的所述活跃度指示表的头节点对应的内存数据的访问活跃度是否低于所述预设访问阈值：在所述头节点对应的内存数据的访问活跃度低于所述预设访问阈值的情况下，释放所述头节点及其对应的内存数据，并继续确定所述活跃度指示表的下一个节点对应的内存数据的访问活跃度是否低于所述预设访问阈值，直至确定出所述活跃度指示表的一个节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值，退出所述老化操作为止；在所述头节点对应的内存数据的访问活跃度高于或等于所述预设访问阈值的情况下，退出所述老化操作。
- 一种计算机可读存储介质，存储有程序指令，当该程序指令被执行时可实现权利要求1-5任一项所述的方法。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410674791.6 | 2014-11-21 | ||
| CN201410674791.6A CN105653556B (zh) | 2014-11-21 | 2014-11-21 | 一种数据老化方法及装置 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2016078388A1 true WO2016078388A1 (zh) | 2016-05-26 |
Family
ID=56013223
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2015/080504 WO2016078388A1 (zh) | 2014-11-21 | 2015-06-01 | 一种数据老化方法及装置 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN105653556B (zh) |
| WO (1) | WO2016078388A1 (zh) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110162958A (zh) * | 2018-10-18 | 2019-08-23 | 腾讯科技（深圳）有限公司 | 用于计算设备的综合信用分的方法、装置和记录介质 |
| CN110389958A (zh) * | 2019-07-26 | 2019-10-29 | 迈普通信技术股份有限公司 | 动态调整硬件表项老化频率的方法、装置及计算机存储介质 |
| CN111752951A (zh) * | 2020-06-28 | 2020-10-09 | 中国银行股份有限公司 | 一种数据库表的处理方法及装置 |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106657249B (zh) * | 2016-10-25 | 2019-07-09 | 杭州迪普科技股份有限公司 | 更新缓存资源的方法及装置 |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070067575A1 (en) * | 2005-09-20 | 2007-03-22 | Morris John M | Method of managing cache memory based on data temperature |
| CN101620618A (zh) * | 2009-07-24 | 2010-01-06 | 中兴通讯股份有限公司 | 内存存储数据的维护方法与装置 |
| US20120089784A1 (en) * | 2010-10-12 | 2012-04-12 | Red Hat, Inc. | Lock Amortization In A Data Counter |
| CN103150259A (zh) * | 2013-03-22 | 2013-06-12 | 华为技术有限公司 | 一种内存回收方法和装置 |
| CN103440207A (zh) * | 2013-07-31 | 2013-12-11 | 北京智谷睿拓技术服务有限公司 | 缓存方法及装置 |
| CN103810115A (zh) * | 2012-11-15 | 2014-05-21 | 深圳市腾讯计算机系统有限公司 | 一种内存池的管理方法和装置 |
| CN103914265A (zh) * | 2014-04-09 | 2014-07-09 | 江苏物联网研究发展中心 | 集群细粒度内存管理方法 |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8086804B2 (en) * | 2008-09-24 | 2011-12-27 | Oracle America, Inc. | Method and system for optimizing processor performance by regulating issue of pre-fetches to hot cache sets |
-
2014
- 2014-11-21 CN CN201410674791.6A patent/CN105653556B/zh active Active
-
2015
- 2015-06-01 WO PCT/CN2015/080504 patent/WO2016078388A1/zh active Application Filing
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070067575A1 (en) * | 2005-09-20 | 2007-03-22 | Morris John M | Method of managing cache memory based on data temperature |
| CN101620618A (zh) * | 2009-07-24 | 2010-01-06 | 中兴通讯股份有限公司 | 内存存储数据的维护方法与装置 |
| US20120089784A1 (en) * | 2010-10-12 | 2012-04-12 | Red Hat, Inc. | Lock Amortization In A Data Counter |
| CN103810115A (zh) * | 2012-11-15 | 2014-05-21 | 深圳市腾讯计算机系统有限公司 | 一种内存池的管理方法和装置 |
| CN103150259A (zh) * | 2013-03-22 | 2013-06-12 | 华为技术有限公司 | 一种内存回收方法和装置 |
| CN103440207A (zh) * | 2013-07-31 | 2013-12-11 | 北京智谷睿拓技术服务有限公司 | 缓存方法及装置 |
| CN103914265A (zh) * | 2014-04-09 | 2014-07-09 | 江苏物联网研究发展中心 | 集群细粒度内存管理方法 |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110162958A (zh) * | 2018-10-18 | 2019-08-23 | 腾讯科技（深圳）有限公司 | 用于计算设备的综合信用分的方法、装置和记录介质 |
| CN110162958B (zh) * | 2018-10-18 | 2023-04-18 | 腾讯科技（深圳）有限公司 | 用于计算设备的综合信用分的方法、装置和记录介质 |
| CN110389958A (zh) * | 2019-07-26 | 2019-10-29 | 迈普通信技术股份有限公司 | 动态调整硬件表项老化频率的方法、装置及计算机存储介质 |
| CN110389958B (zh) * | 2019-07-26 | 2022-05-17 | 迈普通信技术股份有限公司 | 动态调整硬件表项老化频率的方法、装置及计算机存储介质 |
| CN111752951A (zh) * | 2020-06-28 | 2020-10-09 | 中国银行股份有限公司 | 一种数据库表的处理方法及装置 |
| CN111752951B (zh) * | 2020-06-28 | 2023-09-08 | 中国银行股份有限公司 | 一种数据库表的处理方法及装置 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105653556B (zh) | 2021-03-12 |
| CN105653556A (zh) | 2016-06-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11580107B2 (en) | Bucket data distribution for exporting data to worker nodes | |
| US11593377B2 (en) | Assigning processing tasks in a data intake and query system | |
| US11494380B2 (en) | Management of distributed computing framework components in a data fabric service system | |
| US11599541B2 (en) | Determining records generated by a processing task of a query | |
| US11586692B2 (en) | Streaming data processing | |
| US20200364223A1 (en) | Search time estimate in a data intake and query system | |
| US20210294801A1 (en) | Generating a subquery for an external data system using a configuration file | |
| US20220327149A1 (en) | Dynamic partition allocation for query execution | |
| US20200065303A1 (en) | Addressing memory limits for partition tracking among worker nodes | |
| US20200050612A1 (en) | Supporting additional query languages through distributed execution of query engines | |
| US10795884B2 (en) | Dynamic resource allocation for common storage query | |
| US11163758B2 (en) | External dataset capability compensation | |
| Huang et al. | Research on architecture and query performance based on distributed graph database Neo4j | |
| US20190138642A1 (en) | Execution of a query received from a data intake and query system | |
| US20190138641A1 (en) | Subquery generation based on a data ingest estimate of an external data system | |
| US20190138639A1 (en) | Generating a subquery for a distinct data intake and query system | |
| US10698897B2 (en) | Executing a distributed execution model with untrusted commands | |
| US20180089258A1 (en) | Resource allocation for multiple datasets | |
| EP2477355B1 (en) | Method and device for managing association of network resources | |
| US10140313B2 (en) | Parallel processing of large data files on distributed file systems with dynamic workload balancing | |
| WO2016078388A1 (zh) | 一种数据老化方法及装置 | |
| US10331499B2 (en) | Method, apparatus, and chip for implementing mutually-exclusive operation of multiple threads | |
| CN105468619B (zh) | 用于数据库连接池的资源分配方法和装置 | |
| CN108369550B (zh) | 实时更改来自不同源的数据 | |
| EP3690669A1 (en) | Method, apparatus, device and storage medium for managing index technical field |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 15860772 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 15860772 Country of ref document: EP Kind code of ref document: A1 |