CN108228679B - Time series data metering method and time series data metering device - Google Patents

Time series data metering method and time series data metering device Download PDF

Info

Publication number
CN108228679B
CN108228679B CN201611201060.5A CN201611201060A CN108228679B CN 108228679 B CN108228679 B CN 108228679B CN 201611201060 A CN201611201060 A CN 201611201060A CN 108228679 B CN108228679 B CN 108228679B
Authority
CN
China
Prior art keywords
time
time interval
metering
index information
creation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201611201060.5A
Other languages
Chinese (zh)
Other versions
CN108228679A (en
Inventor
杨世泉
付哲
杨恋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201611201060.5A priority Critical patent/CN108228679B/en
Publication of CN108228679A publication Critical patent/CN108228679A/en
Application granted granted Critical
Publication of CN108228679B publication Critical patent/CN108228679B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/1727Details of free space management performed by the file system
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/11File system administration, e.g. details of archiving or snapshots

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Recording Measured Values (AREA)
  • Medical Treatment And Welfare Office Work (AREA)

Abstract

The application discloses time sequence data metering method and device, and the time sequence data metering method comprises the following steps: dividing the creation time period of the cells in the memory file into a plurality of time intervals; establishing index information of each time interval and writing the index information into a disk file; when a metering request of time sequence data is received, determining a time interval corresponding to the metering time point by using the index information according to the metering time point in the metering request; and calculating the data quantity sum of the cells created in the time interval corresponding to the metering time point and the subsequent time interval to obtain the time sequence data quantity. For the calculation of unexpired data, the invention does not need to regularly arrange the data, avoids the conflict between background data arrangement and online service for the IOPS of the disk, ensures the service capability of the disk, can quickly count the unexpired data volume in real time and ensures the accuracy of unexpired data statistics.

Description

Time series data metering method and time series data metering device
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a time series data metering method and a time series data metering device.
Background
With the rise of the internet of things, time series data begins to be generated on a large scale. The time-series data is data that is continuously generated as time goes forward, such as program log data, status data that is periodically generated by the device. The time series data is characterized in that the time series data is generated regularly, the data is rarely modified after being generated, and the frequency of accessing the data is lower when the storage time is longer. Many memory systems currently available begin to provide customized functionality for time series data, focusing on how to efficiently store time series data.
Since each piece of time series data has an expiration time, a common metering scheme is to rewrite the data at regular intervals (e.g., 24 hours), delete the expired data, and then calculate the size of the new unexpired data. The conventional method for processing time series data roughly comprises the following steps:
1. the memory file is written into a disk file, and the data metering program directly calculates the size of real-time sequence data according to the effective data volume in the disk file;
2. as time goes on, more and more disk files are obtained, partial data in some files are expired, at the moment, a plurality of disk files and the like need to be scanned again, expired data are filtered out, unexpired data are written into a new disk file, and then the previous disk files are deleted;
3. counting the real-time sequence data volume of the new disk file;
the above steps 1-3 are repeated with the passage of time.
It can be seen that the prior art relies on periodic full volume data consolidation to calculate unexpired data volume for meter billing or other statistical purposes, the above method has the following disadvantages:
1. background data arrangement and online service compete for disk IOPS (read-write operation times per second), and the service capacity of the stored software is influenced;
2. failure to obtain unexpired data volume in real time may lead to statistical errors.
With the increasing size of machine hard disks and the increasing amount of single-machine stored data, it is increasingly impractical to regularly arrange the data once, and a new method is needed to quickly obtain the unexpired data volume.
Disclosure of Invention
In view of the foregoing problems, an embodiment of the present invention provides a time series data metering method and a time series data metering device to solve the problems in the prior art.
In order to solve the above problem, an embodiment of the present application discloses a time series data metering method, including:
dividing the creation time period of the cells in the memory file into a plurality of time intervals;
establishing index information of each time interval and writing the index information into a disk file, wherein the index information comprises the starting time and the ending time of the time interval and the data volume of the created cells in the time interval;
according to the metering time point in the metering request, determining a time interval corresponding to the metering time point by using the index information;
and calculating the data quantity sum of the cells created in the time interval corresponding to the metering time point and the subsequent time interval to obtain the real-time sequence data quantity.
In an embodiment of the time-series data metering method, in the step of dividing the creation time period of the cell in the memory file into a plurality of time intervals, the creation time period is divided into a plurality of time intervals according to a specified time interval.
In an embodiment of the above method for measuring time series data, the step of dividing the creation time period of the cell in the memory file into a plurality of time intervals includes:
acquiring a creation time period of the cells according to the earliest creation time and the latest creation time of the cells;
and dividing the creation time period into a plurality of time intervals according to a specified time interval.
In an embodiment of the above time series data metering method, before the step of dividing the creation time period of the cell in the memory file into a plurality of time intervals, the method further includes:
and acquiring the creation time of each unit cell from the version number of the unit cell.
In an embodiment of the above-mentioned time series data metering method, the step of determining, according to the metering time point in the metering request, the time interval corresponding to the metering time point by using the index information includes:
judging whether an initial time node corresponding to the metering time point exists in the index information or not;
if the starting time node exists, taking the time interval where the starting time node is located as the time interval corresponding to the metering time point;
and if the starting time node does not exist, determining the starting time node closest to the current time in the index information, and taking the time interval in which the starting time node is located as the time interval corresponding to the metering time point.
To solve the above problem, an embodiment of the present invention further provides a time series data metering device, including:
the time interval dividing module is used for dividing the creation time period of the cell in the memory file into a plurality of time intervals;
the index information establishing module is used for establishing index information of each time interval and writing the index information into a disk file, wherein the index information comprises the starting time and the ending time of the time interval and the data volume of the cells established in the time interval;
the time sequence data quantity acquisition module is used for determining a time interval corresponding to a metering time point by utilizing the index information according to the metering time point in a metering request when the metering request of the time sequence data is received;
and the time sequence quantity acquisition module is used for calculating the data quantity sum of the cells created in the time interval corresponding to the metering time point and the subsequent time interval to acquire the real-time sequence data quantity.
In an embodiment of the above time-series data metering device, the time interval dividing module divides the creation time period into a plurality of time intervals according to a specified time interval.
In an embodiment of the above-mentioned time-series data metering device, the time interval dividing module includes:
the boundary acquisition submodule is used for acquiring the creation time period of the cells according to the earliest creation time and the latest creation time of the cells;
and the time interval division submodule is used for dividing the creation time period into a plurality of time intervals according to the specified time interval.
In an embodiment of the above time-series data metering device, the device further includes:
and the creation time acquisition module is used for acquiring the creation time of each cell from the version number of the cell.
In an embodiment of the above-mentioned time-series data measurement apparatus, the time interval determining module includes:
the judging unit is used for judging whether an initial time node corresponding to the metering time point exists in the index information or not;
the first determining unit is used for taking the time interval where the starting time node is located as the time interval corresponding to the metering time point when the starting time node exists;
and the second determining unit is used for determining the starting time node closest to the current time in the index information when the starting time node does not exist, and taking the time interval where the starting time node is located as the time interval corresponding to the metering time point.
An embodiment of the present application further provides an electronic device, including:
one or more processors; and
a memory having one or more computer instructions stored thereon that, when executed by the one or more processors, cause the electronic device to perform:
dividing the creation time period of the cells in the memory file into a plurality of time intervals;
establishing index information of each time interval and writing the index information into a disk file, wherein the index information comprises the starting time and the ending time of the time interval and the data volume of the cells created in the time interval;
when a metering request of time sequence data is received, determining the data volume sum of the cells created in the time interval corresponding to the time point in the metering request and the subsequent time interval according to the index information, and obtaining the real-time sequence data volume.
In an embodiment of the electronic apparatus, the dividing the creation time period of the cell in the memory file into a plurality of time intervals specifically includes:
the creation time period is divided into a plurality of time intervals according to a specified time interval.
In an embodiment of the electronic apparatus, the step of dividing the creation time period of the cell in the memory file into a plurality of time intervals includes:
acquiring a creation time period of the cells according to the earliest creation time and the latest creation time of the cells;
and dividing the creation time period into a plurality of time intervals according to a specified time interval.
In an embodiment of the electronic apparatus, before the step of dividing the creation time period of the cell in the memory file into a plurality of time intervals, the electronic apparatus further performs:
and acquiring the creation time of each unit cell from the version number of the unit cell.
In an embodiment of the electronic apparatus, the determining, according to the metering time point in the metering request, the time interval corresponding to the metering time point by using the index information includes:
judging whether an initial time node corresponding to the metering time point exists in the index information or not;
if the starting time node exists, taking the time interval where the starting time node is located as the time interval corresponding to the metering time point;
and if the starting time node does not exist, determining the starting time node closest to the current time in the index information, and taking the time interval in which the starting time node is located as the time interval corresponding to the metering time point.
As can be seen from the above, the time series data metering method, the metering device and the electronic device provided in the embodiments of the present application at least include the following advantages:
in the time series data metering method, the time series data metering device and the electronic device provided by the embodiment, the time interval is divided by the creation time of the cell, and the time interval is indexed to obtain the unexpired data volume in real time. For the calculation of unexpired data, the invention does not need to regularly arrange the data, avoids the conflict between background data arrangement and online service for the IOPS of the disk, ensures the service capability of the disk, avoids the time spent on arranging all the data in the disk, can quickly count the unexpired data volume in real time, and ensures the accuracy of unexpired data counting.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a flowchart of a time series data metering method according to a first embodiment of the present application.
Fig. 2 is a schematic diagram illustrating a statistical process of real-time data amount when a memory file is written into a disk file according to an embodiment of the present invention.
Fig. 3 is a flowchart of a time series data metering method according to a second embodiment of the present application.
Fig. 4 is a block diagram of a time-series data metering device according to a third embodiment of the present application.
Fig. 5 is a block diagram of a time-series data metering device according to a fourth embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments that can be derived from the embodiments given herein by a person of ordinary skill in the art are intended to be within the scope of the present disclosure.
One of the core ideas of the application is to provide a time sequence data metering method and a time sequence data metering device, wherein the time period for creating the cell in the memory file is divided into a plurality of time intervals; and establishing index information of each time interval and writing the index information into a disk file, so that when the size of time sequence data needs to be obtained in real time, the real-time data volume sum can be obtained only by obtaining the data volume sum of the cells created in the corresponding time interval and the subsequent time interval according to the index information, the data does not need to be regularly sorted, the disk IOPS (read-write operation times per second) contended by background data sorting and online service is avoided, and the speed of obtaining the data volume sum is improved.
The time series data metering method and the time series data metering device of the invention are described below with reference to a plurality of embodiments.
First embodiment
A first embodiment of the present invention provides a time series data metering method. Fig. 1 is a flowchart illustrating a timing data metering method according to a first embodiment of the present invention. As shown in fig. 1, the time series data metering method of the embodiment of the present invention includes the following steps:
s101, dividing a cell creation time period in a memory file into a plurality of time intervals;
in the present invention, each data record recorded in the storage end is called a row, and any column in each row of data can be a cell (cell). In the embodiment of the invention, unexpired time sequence data needs to be counted. The unexpired time series data is, for example, time series data generated within 24 hours, and time series data exceeding 24 hours are automatically deleted.
In this step, the creation time of each cell in the memory file may be obtained, and the creation time period of the cell in the memory file may be obtained, where the cell in the memory file is created in the creation time period and the creation time period. The creation time period is then divided into a plurality of time intervals. Generally, the creation time of a cell may be recorded in the version number of the cell, and thus the creation time of the cell may be obtained from the version number of each cell. However, the present invention is not particularly limited. It is within the scope of the present invention to obtain the creation time of the plurality of cells and divide the plurality of cells into a plurality of time intervals according to the creation time.
Fig. 2 is a schematic diagram illustrating a statistical process of real-time data amount when a memory file is written into a disk file according to an embodiment of the present invention. As shown in fig. 2, in a specific example, the earliest creation time is denoted by min, the latest creation time (e.g., the latest time that can be counted) is denoted by max, and the plurality of cells may be divided into 7 time intervals from [ min, 10] to [60, max ] according to the creation time and a specified time interval (e.g., 10 minutes). The total data amount of the cells in each time interval can be obtained through statistics, for example, the total data amount of the cells [ min, 10] is 110M, and the total data amount of [60, max ] is 400M.
S102, establishing index information of each time interval and writing the index information into a disk file, wherein the index information comprises the starting time and the ending time of the time interval and the data volume of the created cells in the time interval;
in this step, when writing the plurality of cells of the memory file into the disk file, the index information of the plurality of time intervals may also be written into the disk file, so as to facilitate subsequent indexing. The index information may include, for example, a start-stop time for each time interval and a total data amount for cells created within the time interval. The total data amount of the cells is obtained by accumulating the sizes of the cells in the time interval.
Optionally, the data amount of the cells created in each time interval may be counted while the memory file is written into the disk file, and the counting is completed after all the cells are written into the disk file, and then the index information may be written into the tail portion of the disk file along with the cells in the memory file.
In a specific example, as shown in fig. 2, after writing a plurality of cells of a memory file into a disk file, in addition to the plurality of cells, index information of an entire data block is stored in the disk file, where the index information includes a start time of each time interval from [ min, 10] to [60, max ] and a total data amount of the cells created in the time interval.
S103, when a metering request of time sequence data is received, according to the metering time point in the metering request, the time interval corresponding to the metering time point is determined by using the index information.
In this step, a corresponding time interval may be determined according to the metering time point in the metering request, so as to calculate the data amount according to the time interval in the following step.
For example, if the measurement time point included in the measurement request is 50, the time point corresponds to a time interval of [50,60] in the penultimate column of fig. 2 according to the index information, and the data amount in the time interval and the time interval after the time interval are counted in the subsequent steps.
If the measurement request is the statistical data amount according to the current time, the current time can be acquired as the measurement time point, for example, 50, and then the corresponding time interval is determined according to the current time, and the corresponding time interval is taken as the starting time interval corresponding to the current time.
And S104, counting the data quantity sum of the cells corresponding to the time interval and the time interval after the time interval according to the index information of the established time interval, and obtaining the real-time sequence data quantity.
In this step, when it is necessary to acquire an effective data amount in real time, the sum of the data amounts of the cells created in the time interval and later is calculated using the time interval corresponding to the metering time point determined in step S103, thereby obtaining the size of the real-time series data amount.
For example, if the time interval corresponding to the metering time point obtained in step 103 is [50,60], if the size of the valid data needs to be obtained in real time, the values of the data amount corresponding to the last two columns [50,60], [60, max ] in fig. 2 may be added as the size of the valid data, that is, 20M +400M ═ 420M.
In step S101, the specified time interval may be set by a developer at the time of development, or may be set by the developer according to conditions such as the level of the user and the accuracy of the real-time data to be acquired. For example, when the user is a senior user, the time interval may be, for example, in the order of minutes, i.e., may be divided into one time interval per minute; when the user is a normal user, the time interval may be set to an hour level. For another example, when the precision of the real-time data to be acquired is high precision, the time interval may be set to a minute level, and when the precision of the real-time data to be counted is low precision, the time interval may be set to an hour level.
The specified time interval described above may also be defined in terms of an expiration time. For example, the expiration time is 36 hours, then the interval may be set to 3 hours; if the expiration time is 365 days, the interval may be 10 days. The present invention is not particularly limited to the manner and principle of setting the designated time intervals.
In view of the above, a first embodiment of the present invention provides a time sequence data metering method, which divides a time interval into creation times of cells, and indexes the time interval to obtain an unexpired data amount in real time, so as to divide a creation time period of a cell in a memory file into a plurality of time intervals. For the calculation of unexpired data, the invention does not need to regularly arrange the data, avoids the conflict between background data arrangement and online service for the IOPS of the disk, ensures the service capability of the disk, avoids the time spent on arranging all the data in the disk, can quickly count the data amount of unexpired time sequence data in real time, and ensures the accuracy of unexpired data statistics.
Second embodiment
A second embodiment of the present invention provides a time series data metering method. FIG. 3 is a flowchart illustrating a timing data metering method according to a second embodiment of the present invention. As shown in fig. 3, the time series data metering method of the embodiment of the present invention includes the following steps:
s201, dividing a cell creation time period in a memory file into a plurality of time intervals;
in this step, the creation time of each cell in the memory file may be obtained, and the creation time period of the cell in the memory file may be obtained, where the cell in the memory file is created in the creation time period and the creation time period. The creation time period is then divided into a plurality of time intervals. Generally, the creation time of a cell may be recorded in the version number of the cell, and thus the creation time of the cell may be obtained from the version number of each cell. However, the present invention is not particularly limited. It is within the scope of the present invention to obtain the creation time of the plurality of cells and divide the plurality of cells into a plurality of time intervals according to the creation time.
S202, establishing index information of each time interval and writing the index information into a disk file, wherein the index information comprises the starting time and the ending time of the time interval and the data volume of the created cells in the time interval;
in this step, when writing the plurality of cells of the memory file into the disk file, the index information of the plurality of time intervals may also be written into the disk file, so as to facilitate subsequent indexing. The index information may include, for example, a start-stop time for each time interval and a total data amount for cells created within the time interval. The total data amount is obtained by accumulating the sizes of the plurality of cells in the time interval.
And S203, when a metering request of time sequence data is received, determining a time interval corresponding to the metering time point by using the index information according to the metering time point in the metering request.
In this step, a corresponding time interval may be determined according to the metering time point in the metering request, so as to calculate the data amount according to the time interval in the following step. If the metering request is the statistical data quantity according to the current time, the current time can be acquired as a metering time point, a corresponding time interval is determined according to the current time, and the corresponding time interval is taken as an initial time interval corresponding to the current time. .
And S204, counting the data quantity sum of the cells corresponding to the time interval and the time interval after the time interval according to the index information of the established time interval, and obtaining the real-time sequence data quantity.
In this step, when it is necessary to acquire an effective data amount in real time, the sum of the data amounts of the cells created in the time interval and later is calculated using the time interval corresponding to the metering time point determined in step S103, thereby obtaining the size of the real-time series data amount.
The steps S201 to S204 are the same as or similar to the steps S101 to S104 of the previous embodiment, and are not repeated herein. The relevant content may refer to steps S101 to S104 of the previous embodiment. This embodiment focuses on differences from the previous embodiment.
In the present embodiment, the time interval is divided according to a specified time interval. Therefore, step 201 may be preceded by:
and S200, acquiring a specified time interval.
In this step, the specified time interval may be set by a developer at the time of development, or may be set by the developer according to the user level, the accuracy of the real-time data to be acquired, and the like.
Step S201 is:
s201', dividing the creation time period of the cell in the memory file into a plurality of time intervals according to the designated time interval;
in this embodiment, optionally, step S201 specifically includes:
s201a, acquiring the creation time period of the cells according to the earliest creation time and the latest creation time of the cells;
s201b, dividing the creating time period into a plurality of time intervals according to the designated time interval.
With continuing reference to FIG. 2, the creation time period of the acquired cell is the time period [ max-min ] bounded by the earliest creation time min and the latest creation time max. If the specified time period is 10 minutes, the divided time intervals are 7 intervals of [ min, 10] to [60, max ]. The time intervals may be equal time intervals or unequal time intervals.
Optionally, in step S203, that is, when a metering request of time series data is received, the step of determining a time interval corresponding to the metering time point by using the index information according to the metering time point in the metering request specifically includes:
s203a, judging whether the index information has a start time node corresponding to the current time;
the start time node is, for example, a start time of a certain time interval in the index information, such as min, 10, 20, etc. shown in fig. 2.
S203b, if the starting time node exists, taking the time interval of the starting time node as the corresponding time interval;
as shown in fig. 2, in this step, if the current time is 20, it may be determined that 20 starting time nodes exist in the index information, and therefore, the time interval [20, 30] is taken as the time interval corresponding to the current time.
S203c, if the start time node does not exist, determining the start time node closest to the current time in the index information, and taking the time interval in which the start time node is located as the corresponding time interval.
For another example, if the current time is 45, but there is no 45 time node in the divided time intervals, an expiration time period may be advanced, a starting time node closest to the current time in the index information, i.e., 40, is determined, and a time interval [40,50, in which the starting time node is 40, is taken as the time interval corresponding to the current time.
In view of the foregoing, in a second embodiment of the present invention, a time sequence data metering method is provided, in which a time interval is divided by creating time of a cell, and an unexpired data amount is obtained in real time by indexing the time interval, so as to divide a creating time period of the cell in a memory file into a plurality of time intervals. For the calculation of unexpired data, the invention does not need to regularly arrange the data, avoids the conflict between background data arrangement and online service for the IOPS of the disk, ensures the service capability of the disk, avoids the time spent on arranging all the data in the disk, can quickly count the unexpired data volume in real time, and ensures the accuracy of unexpired data counting.
In addition, in the time series data metering method of the embodiment, the user can be allowed to obtain the size of the valid data after a certain time point, and as long as the user gives a date, the system can return the size of the valid data after the date, so that an interface can be opened for the user, the user is allowed to obtain the size of the unexpired data volume at a certain time point in the future, and the user can optimize the storage cost.
In addition, by modifying the time interval, the invention can balance the data metering precision and the index resource occupation by controlling the size of the index time interval.
Third embodiment
A third embodiment of the present invention provides a time series data metering device, and fig. 4 is a block diagram of the time series data metering device according to the third embodiment of the present invention. As shown in fig. 4, the time-series data metering device 300 includes:
a time interval dividing module 301, configured to divide a creation time period of a cell in a memory file into a plurality of time intervals;
in this embodiment, the time interval dividing module 301 may divide the creation time periods of the multiple cells in the memory file into multiple time intervals; generally, the creation time of a cell is recorded in the version number of the cell, and the creation time of the cell may be obtained from the version number of each cell, and the creation time periods of all cells in the memory file are obtained, and then the creation time periods are divided into a plurality of time intervals.
An index information creating module 302, configured to create index information for each time interval and write the index information into a disk file, where the index information includes start and end times of the time interval and a data amount of a cell in the time interval;
in this embodiment, the index information creating module 302 may write the index information of the multiple time intervals into the disk file when writing the multiple cells of the memory file into the disk file, so as to facilitate subsequent indexing. The index information may include, for example, a start-stop time for each time interval and a total data amount for cells created within the time interval. The total data amount is obtained by accumulating the sizes of the plurality of cells in the time interval.
Optionally, the data amount of the cells created in each time interval may be counted while the memory file is written into the disk file, and the counting is completed after all the cells are written into the disk file, and then the index information may be written into the tail portion of the disk file along with the cells in the memory file.
A time interval determining module 303, configured to determine, when a metering request of time series data is received, a time interval corresponding to a metering time point by using the index information according to the metering time point in the metering request;
in this embodiment, when the effective data amount needs to be obtained in real time, the time interval determining module 303 may be configured to determine an initial time interval according to the metering time point in the metering request by using the index information established by the index information establishing module 302, for subsequent calculation.
The time sequence quantity obtaining module 304 is configured to calculate a data quantity sum of the cell created in the time interval corresponding to the metering time point and the time interval thereafter, so as to obtain a real-time sequence data quantity.
When the effective data amount needs to be acquired in real time, the time sequence number acquisition module 304 may calculate the sum of the data amounts of the cells created in the time interval and the subsequent time interval by using the time interval corresponding to the metering time point determined in the time interval determination module 303, so as to obtain the size of the real-time sequence data amount.
In view of the foregoing, in a third embodiment of the present invention, a time sequence data metering device is provided, which divides a time interval into creation time periods of cells, and divides the creation time periods of the cells in a memory file into a plurality of time intervals by obtaining an unexpired data amount in real time by indexing the time interval. For the calculation of unexpired data, the invention does not need to regularly arrange the data, avoids the conflict between background data arrangement and online service for the IOPS of the disk, ensures the service capability of the disk, avoids the time spent on arranging all the data in the disk, can quickly count the unexpired data volume in real time, and ensures the accuracy of unexpired data counting.
Fourth embodiment
A time series data metering device according to a fourth embodiment of the present invention is provided, and fig. 5 is a block diagram of the time series data metering device according to the fourth embodiment of the present invention. As shown in fig. 5, the time-series data metering device 400 includes:
a time interval dividing module 401, configured to divide a creation time period of a cell in a memory file into a plurality of time intervals;
in this embodiment, the time interval dividing module 401 may divide the creation time periods of the multiple cells in the memory file into multiple time intervals; generally, the creation time of a cell is recorded in the version number of the cell, and the creation time of the cell may be obtained from the version number of each cell, and the creation time periods of all cells in the memory file are obtained, and then the creation time periods are divided into a plurality of time intervals.
An index information creating module 402, configured to create index information for each time interval and write the index information into a disk file, where the index information includes start and end times of the time interval and a data amount of a cell in the time interval;
in this embodiment, the index information creating module 402 may write the index information of the multiple time intervals into the disk file when writing the multiple cells of the memory file into the disk file, so as to facilitate subsequent indexing. The index information may include, for example, a start-stop time for each time interval and a total data amount for cells created within the time interval. The total data amount is obtained by accumulating the sizes of the plurality of cells in the time interval.
Optionally, the data amount of the cells created in each time interval may be counted while the memory file is written into the disk file, and the counting is completed after all the cells are written into the disk file, and then the index information may be written into the tail portion of the disk file along with the cells in the memory file.
A time interval determining module 403, configured to determine, when a metering request of time series data is received, a time interval corresponding to a metering time point by using the index information according to the metering time point in the metering request;
in this embodiment, when the effective data amount needs to be obtained in real time, the time interval determining module 403 may be configured to determine an initial time interval according to the metering time point in the metering request by using the index information established by the index information establishing module 402, for subsequent calculation.
And a time sequence quantity obtaining module 404, configured to calculate a data quantity sum of the cell created in the time interval corresponding to the metering time point and the time interval thereafter, so as to obtain a real-time sequence data quantity.
When the effective data amount needs to be acquired in real time, the time series quantity acquisition module 304 may calculate the sum of the data amounts of the cells created in the time interval and the subsequent time interval by using the time interval corresponding to the metering time point determined in the time interval determination module 403, so as to obtain the size of the real-time series data amount.
The above modules 401 to 404 are the same as or similar to the steps 301 to 304 in the previous embodiment, and are not described again here. Relevant content reference may be made to steps 301 to 304 of the previous embodiment. The present embodiment focuses on differences from the previous embodiment.
In this embodiment, the time interval dividing module 401 may divide the creation time period into a plurality of time intervals according to a specified time interval.
In this embodiment, the apparatus further includes:
a time interval obtaining module 404, configured to obtain the specified time interval.
In this embodiment, the time interval dividing module 401 includes:
the boundary obtaining submodule 401a is configured to obtain a creation time period of the cell according to the earliest creation time and the latest creation time of the plurality of cells;
the time interval dividing submodule 401b is configured to divide the time period between the earliest creation time and the latest creation time into a plurality of time intervals according to a specified time interval.
In this embodiment, the apparatus further includes:
a creation time obtaining module 400' for obtaining the creation time of each cell from the version number of the cell.
In this embodiment, the time interval determining module 403 includes:
the judging unit is used for judging whether an initial time node corresponding to the current time exists in the index information or not;
a first determining unit, configured to use, when the start time node exists, a time interval in which the start time node is located as a corresponding time interval;
and the second determining unit is used for determining the starting time node closest to the current time in the index information when the starting time node does not exist, and taking the time interval in which the starting time node is located as the corresponding time interval.
In view of the foregoing, in a fourth embodiment of the present invention, a time sequence data metering device is provided, which divides a creation time of a cell into time intervals, and indexes the time intervals to obtain an unexpired data amount in real time, so as to divide the creation time period of the cell in a memory file into a plurality of time intervals. For the calculation of unexpired data, the invention does not need to regularly arrange the data, avoids the conflict between background data arrangement and online service for the IOPS of the disk, ensures the service capability of the disk, avoids the time spent on arranging all the data in the disk, can quickly count the unexpired data volume in real time, and ensures the accuracy of unexpired data counting.
In addition, in the time series data metering method of the embodiment, the user can be allowed to obtain the size of the valid data after a certain time point, and as long as the user gives a date, the system can return the size of the valid data after the date, so that an interface can be opened for the user, the user is allowed to obtain the size of the unexpired data volume at a certain time point in the future, and the user can optimize the storage cost.
In addition, by modifying the time interval, the invention can balance the data metering precision and the index resource occupation by controlling the size of the index time interval.
For the apparatus embodiment, since it is basically similar to the method embodiment, it is described relatively simply, and for the relevant points, refer to the partial description of the method embodiment.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
As will be appreciated by one of skill in the art, embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
In a typical configuration, the computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory. The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium. Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (fransitory media), such as modulated data signals and carrier waves.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the true scope of the embodiments of the application.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The time series data metering method and the time series data metering device provided by the application are introduced in detail, specific examples are applied in the description to explain the principle and the implementation mode of the application, and the description of the embodiments is only used for helping to understand the method and the core idea of the application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims (15)

1. A method for metering time series data, comprising:
dividing the creation time period of the cells in the memory file into a plurality of time intervals;
establishing index information of each time interval and writing the index information into a disk file, wherein the index information comprises the starting time and the ending time of the time interval and the data volume of the cells created in the time interval;
when a metering request of time sequence data is received, determining a time interval corresponding to the metering time point by using the index information according to the metering time point in the metering request;
and calculating the data quantity sum of the cells created in the time interval corresponding to the metering time point and the subsequent time interval to obtain the time sequence data quantity.
2. The method of claim 1, wherein the dividing the time period for creating the cells in the memory file into a plurality of time intervals is specifically:
the creation time period is divided into a plurality of time intervals according to a specified time interval.
3. The method of claim 1, wherein the step of dividing the time period for creating the cells in the memory file into a plurality of time intervals comprises:
acquiring a creation time period of the cells according to the earliest creation time and the latest creation time of the cells;
and dividing the creation time period into a plurality of time intervals according to a specified time interval.
4. The method of claim 1, wherein prior to the step of dividing the time period for creating the cells in the memory file into the plurality of time intervals, the method further comprises:
and acquiring the creation time of each unit cell from the version number of the unit cell.
5. The method as claimed in claim 1, wherein the step of determining the time interval corresponding to the metering time point by using the index information according to the metering time point in the metering request comprises:
judging whether an initial time node corresponding to the metering time point exists in the index information or not;
if the starting time node exists, taking the time interval where the starting time node is located as the time interval corresponding to the metering time point;
and if the starting time node does not exist, determining the starting time node closest to the current time in the index information, and taking the time interval in which the starting time node is located as the time interval corresponding to the metering time point.
6. A time series data metering device, comprising:
the time interval dividing module is used for dividing the creation time period of the cell in the memory file into a plurality of time intervals;
the index information establishing module is used for establishing index information of each time interval and writing the index information into a disk file, wherein the index information comprises the starting time and the ending time of the time interval and the data volume of the cells established in the time interval;
the time interval determining module is used for determining a time interval corresponding to a metering time point by using the index information according to the metering time point in the metering request when the metering request of the time sequence data is received;
and the time sequence quantity acquisition module is used for calculating the data quantity sum of the cells created in the time interval corresponding to the metering time point and the subsequent time interval to acquire the time sequence data quantity.
7. The apparatus of claim 6, wherein the time interval dividing module divides the creation time period into a plurality of time intervals according to a specified time interval.
8. The apparatus of claim 6, wherein the time interval division module comprises:
the boundary acquisition submodule is used for acquiring the creation time period of the cells according to the earliest creation time and the latest creation time of the cells;
and the time interval division submodule is used for dividing the creation time period into a plurality of time intervals according to the specified time interval.
9. The apparatus of claim 6, wherein the apparatus further comprises:
and the creation time acquisition module is used for acquiring the creation time of each cell from the version number of the cell.
10. The apparatus of claim 6, wherein the time interval determination module comprises:
the judging unit is used for judging whether an initial time node corresponding to the metering time point exists in the index information or not;
the first determining unit is used for taking the time interval where the starting time node is located as the time interval corresponding to the metering time point when the starting time node exists;
and the second determining unit is used for determining the starting time node closest to the current time in the index information when the starting time node does not exist, and taking the time interval where the starting time node is located as the time interval corresponding to the metering time point.
11. An electronic device, comprising:
one or more processors; and
a memory having one or more computer instructions stored thereon that, when executed by the one or more processors, cause the electronic device to perform:
dividing the creation time period of the cells in the memory file into a plurality of time intervals;
establishing index information of each time interval and writing the index information into a disk file, wherein the index information comprises the starting time and the ending time of the time interval and the data volume of the cells created in the time interval;
when a metering request of time sequence data is received, determining the data volume sum of the cells created in the time interval corresponding to the time point in the metering request and the subsequent time interval according to the index information, and obtaining the time sequence data volume.
12. The electronic device according to claim 11, wherein the dividing of the cell creation time period in the memory file into a plurality of time intervals is specifically:
the creation time period is divided into a plurality of time intervals according to a specified time interval.
13. The electronic device of claim 11, wherein the step of dividing the time period for creating the cells in the memory file into a plurality of time intervals comprises:
acquiring a creation time period of the cells according to the earliest creation time and the latest creation time of the cells;
and dividing the creation time period into a plurality of time intervals according to a specified time interval.
14. The electronic device according to claim 11, wherein before the step of dividing the creation time period of the cell in the memory file into a plurality of time intervals, the electronic device further performs:
and acquiring the creation time of each unit cell from the version number of the unit cell.
15. The electronic device according to claim 11, wherein the determining, according to the metering time point in the metering request, the time interval corresponding to the metering time point by using the index information comprises:
judging whether an initial time node corresponding to the metering time point exists in the index information or not;
if the starting time node exists, taking the time interval where the starting time node is located as the time interval corresponding to the metering time point;
and if the starting time node does not exist, determining the starting time node closest to the current time in the index information, and taking the time interval in which the starting time node is located as the time interval corresponding to the metering time point.
CN201611201060.5A 2016-12-22 2016-12-22 Time series data metering method and time series data metering device Active CN108228679B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201611201060.5A CN108228679B (en) 2016-12-22 2016-12-22 Time series data metering method and time series data metering device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201611201060.5A CN108228679B (en) 2016-12-22 2016-12-22 Time series data metering method and time series data metering device

Publications (2)

Publication Number Publication Date
CN108228679A CN108228679A (en) 2018-06-29
CN108228679B true CN108228679B (en) 2022-02-18

Family

ID=62657083

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201611201060.5A Active CN108228679B (en) 2016-12-22 2016-12-22 Time series data metering method and time series data metering device

Country Status (1)

Country Link
CN (1) CN108228679B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110196851B (en) * 2019-05-09 2024-05-10 腾讯科技(深圳)有限公司 Data storage method, device, equipment and storage medium
CN110619001B (en) * 2019-08-30 2022-03-11 安徽耐科装备科技股份有限公司 Discontinuous interval time interval calculation method
CN113360334B (en) * 2020-03-05 2023-09-22 北京京东振世信息技术有限公司 Early warning method and device for memory resources and computer readable storage medium
CN112214503A (en) * 2020-10-10 2021-01-12 深圳壹账通智能科技有限公司 Data processing method and device, electronic equipment and storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101894153A (en) * 2010-06-29 2010-11-24 武汉绿色网络信息服务有限责任公司 Annular storage list-based dynamic data compressing and storing method
CN102354406A (en) * 2011-06-07 2012-02-15 同济大学 Real time statistical system and method for rail traffic network passenger flow based on RFID technology
CN102890722A (en) * 2012-10-25 2013-01-23 国家电网公司 Indexing method applied to time sequence historical database
CN103139272A (en) * 2011-12-02 2013-06-05 北大方正集团有限公司 Method of obtaining online time within selected time period and device using the same
CN103714708A (en) * 2013-12-18 2014-04-09 福建工程学院 Optimal path planning method based on split-time experience path of taxi
CN105468728A (en) * 2015-11-20 2016-04-06 北京先进数通信息技术股份公司 Cross-section data acquisition method and system
CN105512199A (en) * 2015-11-27 2016-04-20 广州神马移动信息科技有限公司 Search method, search device and search server

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5908174B2 (en) * 2013-07-09 2016-04-26 株式会社日立国際電気 Image processing apparatus and image processing method
CN103605805B (en) * 2013-12-09 2016-10-26 冶金自动化研究设计院 A kind of storage method of magnanimity time series data
CN105242882B (en) * 2015-10-13 2018-09-21 东方网力科技股份有限公司 The frame storage method and device of time series data, time series data querying method and device

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101894153A (en) * 2010-06-29 2010-11-24 武汉绿色网络信息服务有限责任公司 Annular storage list-based dynamic data compressing and storing method
CN102354406A (en) * 2011-06-07 2012-02-15 同济大学 Real time statistical system and method for rail traffic network passenger flow based on RFID technology
CN103139272A (en) * 2011-12-02 2013-06-05 北大方正集团有限公司 Method of obtaining online time within selected time period and device using the same
CN102890722A (en) * 2012-10-25 2013-01-23 国家电网公司 Indexing method applied to time sequence historical database
CN103714708A (en) * 2013-12-18 2014-04-09 福建工程学院 Optimal path planning method based on split-time experience path of taxi
CN105468728A (en) * 2015-11-20 2016-04-06 北京先进数通信息技术股份公司 Cross-section data acquisition method and system
CN105512199A (en) * 2015-11-27 2016-04-20 广州神马移动信息科技有限公司 Search method, search device and search server

Also Published As

Publication number Publication date
CN108228679A (en) 2018-06-29

Similar Documents

Publication Publication Date Title
CN108228679B (en) Time series data metering method and time series data metering device
US20180157438A1 (en) Slow-disk detection method and apparatus
CN109388550B (en) Cache hit rate determination method, device, equipment and readable storage medium
CN108243032B (en) Method, device and equipment for acquiring service level information
CN110490635B (en) Commercial tenant dish transaction prediction and meal preparation method and device
CN110795035B (en) Migration time determination method, device and equipment and readable storage medium
CN112149011A (en) Method and device for updating popularity ranking list, server and computer storage medium
CN110908587A (en) Method and device for storing time sequence data
CN108809751B (en) Policy testing method and device
CN115002187B (en) Binding relation processing method and related equipment
CN108984572B (en) Website information pushing method and device
CN113568754A (en) Resource allocation method and device, computing equipment and storage medium
CN108958658B (en) Target data acquisition method and device
CN109587548B (en) Video playing data calculation method and device
CN113468398A (en) Book level determination method, computing device and computer storage medium
CN110008269B (en) Data reflow method, device, equipment and system
CN110874268B (en) Data processing method, device and equipment
CN111078122B (en) Data processing method, device and equipment
CN111666535A (en) Method and device for determining user activity duration, electronic equipment and storage medium
CN112445835A (en) Business data processing method and device, network management server and storage medium
US11714737B2 (en) Time clock quality determination
CN112965828B (en) Multithreading data processing method, device, equipment and storage medium
CN109615465A (en) Processing method, device and the electronic equipment of service order
CN115309809A (en) Data batch insertion method and device and computer storage medium
CN109729393B (en) Data processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 1257268

Country of ref document: HK

GR01 Patent grant
GR01 Patent grant