CN110990449A - Time series processing method, device, storage medium and processor - Google Patents
Time series processing method, device, storage medium and processor Download PDFInfo
- Publication number
- CN110990449A CN110990449A CN201911037564.1A CN201911037564A CN110990449A CN 110990449 A CN110990449 A CN 110990449A CN 201911037564 A CN201911037564 A CN 201911037564A CN 110990449 A CN110990449 A CN 110990449A
- Authority
- CN
- China
- Prior art keywords
- time
- subsequences
- data
- sql
- subsequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 47
- 238000000034 method Methods 0.000 claims abstract description 36
- 238000001914 filtration Methods 0.000 claims abstract description 32
- 230000002159 abnormal effect Effects 0.000 claims description 24
- 230000008569 process Effects 0.000 claims description 19
- 238000012937 correction Methods 0.000 claims description 17
- 238000012216 screening Methods 0.000 claims description 11
- 230000002123 temporal effect Effects 0.000 claims description 8
- 238000010586 diagram Methods 0.000 description 14
- 238000004590 computer program Methods 0.000 description 8
- 238000007726 management method Methods 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 230000004048 modification Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000012731 temporal analysis Methods 0.000 description 3
- 238000000700 time series analysis Methods 0.000 description 3
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000012300 Sequence Analysis Methods 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000010006 flight Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a time sequence processing method and device, a storage medium and a processor, and belongs to the technical field of data processing. The method comprises the following steps: splitting a time sequence to be processed into a plurality of sections of continuous time subsequences; respectively filtering each time subsequence according to a preset filtering condition to obtain a plurality of corresponding first time subsequences; respectively carrying out data adjustment processing on each segment of first time subsequence according to a preset data adjustment condition to obtain a plurality of corresponding segments of second time subsequences, wherein the data adjustment processing comprises correcting a designated field on the first time subsequence according to a preset rule; and splicing the second time subsequences of all the sections to obtain a new time sequence. According to the embodiment of the invention, the integrity of the time sequence is ensured by splitting, adjusting and splicing the time sequence data, and the problems of weak timeliness, high labor cost and high profit and loss rate of a traditional time sequence data single data processing mode are solved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a time series processing method and apparatus, a storage medium, and a processor.
Background
Time series (or dynamic number series) refers to a number series formed by arranging numerical values of the same statistical index according to the sequence of the occurrence time of the numerical values, and many fields at present relate to giving data in a time series form. The time points in the time series can be years, quarters, months or any other time form according to the occurrence time, so the time series can also be understood as the numerical value series formed at each time point, and the time series analysis can be understood as predicting the future data by observing the historical data.
Based on the meaning of time series analysis, the method is generally applied to computer software systems (especially systems with characteristics of flow-oriented and continuous data) in various industries at present by means of time management data or giving time attributes to the data, and aims to obtain time series data. For this type of system, the consistency of the time series data is crucial. The break of the time series can cause operation accidents caused by system reasons, for example, in the traffic management industry, the break of the time series related to vehicle operation is caused by the influence of irresistible factors such as heavy rain, heavy snow, debris flow, typhoon and the like, and further the wrong estimation of scheduled flights, trains and the like which are expected to arrive at a station is caused, and the operation and user experience are influenced.
However, in the prior art, in the process of acquiring time-series data, a single data mode is generally adopted to process the time-series data, and the inventor of the present application finds that in the process of implementing the embodiment of the present invention, the single data mode has at least the following three disadvantages:
1. the timeliness is weak. For a system related to actual production, if the time series data of the system is adjusted due to an irresistible factor, the processing mode of single data is weak in timeliness. For example, 5800 pieces of maintained data are processed, and the way of processing single data is long in time and cannot respond in time.
2. The labor cost is high. In a certain time, a certain amount of maintenance data is adjusted, and the processing mode of single data consumes high human resources. For example, 5800 pieces of data maintained in 15 minutes are processed, and the single data processing mode needs 15-20 manpower.
3. The profit and loss rate is high. In the face of the external burst irresistible factor, the existing single data processing mode has a relatively higher possibility of causing the loss of the event subject.
Disclosure of Invention
The embodiment of the invention aims to provide a time sequence processing method, a time sequence processing device, a storage medium and a processor, which are used for solving the problems of weak timeliness, high labor cost and high profit and loss rate of a traditional time sequence data single data processing mode.
In order to achieve the above object, an embodiment of the present invention provides a time series processing method, including: splitting a time sequence to be processed into a plurality of sections of continuous time subsequences; respectively filtering each time subsequence according to a preset filtering condition to obtain a plurality of corresponding first time subsequences; respectively carrying out data adjustment processing on each segment of first time subsequence according to a preset data adjustment condition to obtain a plurality of corresponding segments of second time subsequences, wherein the data adjustment processing comprises correcting a designated field on the first time subsequence according to a preset rule; and splicing the second time subsequences of all the sections to obtain a new time sequence.
Optionally, the splitting the time-series data to be processed into a plurality of continuous time subsequences includes: determining a target time period; and according to the target time period, splitting the time sequence data into a plurality of continuous time subsequences, wherein one of the split plurality of continuous time subsequences corresponds to the target time period.
Optionally, the preset filtering condition comprises a sequence screening rule for screening out the first time subsequence; and/or the preset data adjustment condition comprises an abnormal point correction rule for performing data adjustment on the abnormal point on the time series.
Optionally, the respectively performing data adjustment processing on each segment of the first time subsequence according to a preset data adjustment condition includes: determining outliers on the first temporal subsequence; and correcting the field corresponding to the abnormal point according to the abnormal point correction rule.
Optionally, the concatenating each second time subsequence includes: generating a structured description language (SQL) of each second time subsequence; and formally executing the SQL to splice the second time subsequences into the new time sequence.
Optionally, before the formally executing the SQL, the method further includes: and pre-executing the SQL, and calculating the influence data information of the operation of formally executing the SQL on the time sequence according to the pre-executed result.
Optionally, the pre-executing the SQL comprises: inserting a temporary table for storing the SQL to be executed; and executing the SQL based on the temporary table, detecting and updating the cascading fields of the SQL in the executing process and checking data.
On the other hand, an embodiment of the present invention further provides a time-series processing apparatus, including: the splitting unit is used for splitting the time sequence to be processed into a plurality of sections of continuous time subsequences; the filtering unit is used for respectively filtering each time subsequence according to a preset filtering condition to obtain a plurality of corresponding first time subsequences; the adjusting unit is used for respectively carrying out data adjustment processing on each segment of the first time subsequence according to a preset data adjustment condition to obtain a plurality of corresponding segments of the second time subsequence, wherein the data adjustment processing comprises correcting a designated field on the first time subsequence according to a preset rule; and the splicing unit is used for splicing the second time subsequences to obtain a new time sequence.
Optionally, the splitting unit includes: the target time period determining module is used for determining a target time period; and the sequence splitting module is used for splitting the time sequence data into a plurality of continuous time subsequences according to the target time period, wherein one of the split plurality of continuous time subsequences corresponds to the target time period.
Optionally, the preset filtering condition comprises a sequence screening rule for screening out the first time subsequence; and/or the preset data adjustment condition comprises an abnormal point correction rule for performing data adjustment on the abnormal point on the time series.
Optionally, the adjusting unit includes: an outlier determination module configured to determine outliers on the first temporal subsequence; and the field correction module is used for correcting the field corresponding to the abnormal point according to the abnormal point correction rule.
Optionally, the splicing unit comprises: the language generation module is used for generating a structured description language SQL of each section of the second time subsequence; and the first execution module is used for formally executing the SQL so as to splice the second time subsequences into the new time sequence.
Optionally, the splicing unit further includes: and the second execution module is used for pre-executing the SQL before the first execution module formally executes the SQL, and calculating the influence data information of the operation of formally executing the SQL on the time sequence according to the pre-executed result.
Optionally, the second execution module includes: the inserting sub-module is used for inserting a temporary table to store the SQL to be executed; and the execution submodule is used for executing the SQL based on the temporary table, detecting and updating the cascading field of the SQL in the execution process and carrying out data verification.
In another aspect, the embodiment of the present invention further provides a machine-readable storage medium, where instructions are stored on the machine-readable storage medium, and the instructions are used to enable a machine to execute the above time-series processing method.
In another aspect, an embodiment of the present invention further provides a processor, configured to execute a program, where the program is executed to perform: such a time-series processing method is described above.
Through the technical scheme, the embodiment of the invention ensures the integrity of the time sequence by splitting, adjusting and splicing the time sequence data in batches, and solves the problems of weak timeliness, high labor cost and high profit and loss rate of a traditional time sequence data single data processing mode.
Additional features and advantages of embodiments of the invention will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the embodiments of the invention without limiting the embodiments of the invention. In the drawings:
FIG. 1 is a schematic flow chart diagram of a time series processing method according to an embodiment of the invention;
FIG. 2 is a flow diagram illustrating time series processing in an example of an embodiment of the present invention;
FIG. 3 is a schematic diagram of a current time series and a split of the time series in an example of an embodiment of the invention; and
fig. 4 is a schematic structural diagram of a time-series processing apparatus according to an embodiment of the present invention.
Description of the reference numerals
410. A splitting unit; 420. a filtration unit; 430. an adjustment unit; 440. a splicing unit;
411. a target time period determination module; 412. a sequence splitting module;
431. an anomaly determination module; 432. a field correction module;
441. a language generation module; 442. a first execution module; 443. and a second execution module.
Detailed Description
The following detailed description of embodiments of the invention refers to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating embodiments of the invention, are given by way of illustration and explanation only, not limitation.
Fig. 1 is a schematic flow chart of a time-series processing method according to an embodiment of the present invention, where the time-series may be a series related to traffic data, enterprise business data, and the like, and the embodiment of the present invention is not limited thereto. As shown in fig. 1, the time-series processing method according to the embodiment of the present invention may include the following steps:
step S110, the time sequence to be processed is divided into a plurality of consecutive time subsequences.
The time sequence can be divided according to the requirement of the actual application scene, for example, the train-to-point time sequence of 2018 years in a certain city is divided into 12 time subsequences according to the requirement of the city for analyzing the monthly punctuation condition of the train.
Further, in many application scenarios, it is often necessary to analyze data of a time series of a specific time period, for example, for a certain city, it is necessary to specifically analyze train arrival conditions of the city in a specific time period such as a spring time period, a typhoon time period, and the like.
In this regard, in a preferred embodiment, the step S110 may include: determining a target time period; and according to the target time period, splitting the time sequence data into a plurality of continuous time subsequences, wherein one of the split plurality of continuous time subsequences corresponds to the target time period.
For example, if the target time period is divided into the time subsequences from beginning of the first year to end of the year, when the train arrival time sequences of each year are split, it is ensured that one of the obtained time subsequences exactly corresponds to the time subsequences divided into the time subsequences from beginning of the year to end of the year, and the remaining time subsequences are still split according to the requirements of the actual application scene, for example, split into the time subsequences corresponding to different months.
And step S120, respectively filtering each time subsequence according to preset filtering conditions to obtain a plurality of corresponding first time subsequences.
Wherein, the filtering condition can be configured according to the actual requirement, and is aimed at filtering out the data with obvious error or the data without subsequent data adjustment processing, so as to screen out the first time subsequence with data adjustment processing. Preferably, the preset filtering condition may include a sequence screening rule for screening the first temporal subsequence, so as to directly screen the satisfactory first temporal subsequence by the sequence screening rule. For example, a sequence filtering rule can be "equal to" or "not equal to" a preset value, "greater than" or "less than" a preset value, "contain" specified data, and/or require field data "between specified ranges of values," etc.
Here, by setting the filtering condition, the data amount for subsequent data adjustment can be effectively reduced, thereby facilitating the improvement of the data processing efficiency.
Step S130, respectively performing data adjustment processing on each segment of the first time subsequence according to a preset data adjustment condition, to obtain a plurality of corresponding segments of the second time subsequence.
Wherein the data adjustment process includes modifying the designated field on the first time sub-sequence according to a preset rule, and the modification includes modifying the field content, for example, increasing the arrival time of a train in a sequence by 30 minutes. In other embodiments, the data adjustment process may further include deleting specified fields in the time sequence, but the embodiment of the present invention is preferably modified because the filtering process in step S120 can already filter out obviously wrong data, and the remaining data can be modified to be suitable for the operation requirement, and instead, the direct deletion may affect the normal operation, for example, the train is 30 minutes later on a certain day due to an irresistible factor, but belongs to a special case, and the same situation does not necessarily occur later, so that 30 minutes may be added to ensure the normal monitoring of the arrival time of the train, and the direct deletion of the data about the arrival time may cause the system to no longer predict the arrival time of the train on the following days.
In this regard, in a preferred embodiment, the preset data adjustment condition may include an abnormal point correction rule for performing data adjustment on an abnormal point on the time series, where the abnormal point may be, for example, an abnormal arrival time generated in the above-mentioned train late situation, and the abnormal point correction rule may be, for example, "add 30 minutes" as described above.
Based on the outlier correction rule, in a more preferred embodiment, the step S130 may include: determining outliers on the first temporal subsequence; and correcting the field corresponding to the abnormal point according to the abnormal point correction rule.
The anomaly point is determined by configuring a detector to detect according to a preset detection rule, for example, the detector marks a field showing that the arrival time exceeds a preset threshold as the anomaly point. The abnormal point correction rule requires, for example, modification to a set value, an increase in a specified value, and/or a decrease in a specified value.
And step S140, splicing the second time subsequences to obtain a new time sequence.
In a preferred embodiment, the step S140 may further include: generating a Structured description Language (SQL) of each second time subsequence; and formally executing the SQL to splice the second time subsequences into the new time sequence.
The specific generation method is conventional for those skilled in the art, and therefore, details are not repeated herein. In addition, formally executing SQL includes querying temporary data, updating a formal table, and the like, and the execution process thereof is also conventional for those skilled in the art, and therefore, the detailed description thereof is omitted. The structural feature of SQL makes it possible to directly obtain a new time sequence with consistent structure by executing the corresponding SQL on each second time subsequence, and the new time sequence has more integrity for the field with the exception point adjusted.
However, executing SQL may affect field data on the time series, for example, affect some concatenated fields, thereby affecting the concatenation of the time sub-series. In this regard, in a preferred embodiment, before formally executing the SQL in step S140, the method may further include: and pre-executing the SQL, and calculating the influence data information of the operation of formally executing the SQL on the time sequence according to the pre-executed result.
The execution result of the pre-executed SQL includes, for example, how many pieces of data are affected, differences between new data and expected data and original data, and the like, and the affected data information calculated according to the execution results includes, for example, factors showing the degree of influence of the operation of the formal execution SQL on the time sequence, so that the influence on the splicing time sequence in the process of the formal execution of the SQL can be avoided, the integrity of the time sequence is further affected, and even the computer software system for performing the subsequent execution time sequence analysis is affected.
In a more preferred embodiment, the pre-executing the SQL may include: inserting a temporary table for storing the SQL to be executed; and executing the SQL based on the temporary table, detecting and updating the cascading fields of the SQL in the executing process and checking data. The purpose of using the temporary table is to make the pre-execution SQL not affect the online data.
In addition, it should be noted that data adjustment is not required for each second time sub-sequence, so that the new time sequence obtained by splicing in step S140 includes time sub-sequences not subjected to data adjustment, in addition to the time sub-sequences subjected to data adjustment.
The following specifically describes an application of the time-series processing method to which the embodiment of the present invention is applied, by way of example. Fig. 2 is a schematic flow chart of time-series processing in an example of an embodiment of the present invention, where the example is implemented by using typhoon No. 4 in 2018, and aging degradation is required to be performed on all lines currently validated in the cantonese region (2018-01-2002: 00-9999-12-3101: 00), where a filtering condition is set to that destination nodes in the line management belong to cantonese, and a data adjusting condition is set to that departure time is increased by 12 hours (+12 hours).
As shown in fig. 2, the following steps may be included:
step S210, obtain the current time series.
Step S220, splitting the current time sequence.
Fig. 3 is a schematic diagram of a current time sequence and a time sequence splitting performed in an example of the embodiment of the present invention. As shown in fig. 3. The current time sequence is a sequence with the effective time of 2018-01-2002: 00 and the failure time of 9999-12-3101:00, and is divided into the following three subsequences:
1) time series 1: the effective time is 2018-01-2002: 00, and the failure time is 2018-07-0501: 00;
2) time series 2: the effective time is 2018-07-0602: 00, and the failure time is 2018-07-0807: 00;
3) time series 3: the effective time is 2018-07-0902: 00, and the failure time is 9999-12-3101: 00.
Wherein time series 2 corresponds to No. 4 typhoon period of 2018, belonging to the target time period.
In step S230, filtering conditions are set.
In this example, the filtering condition is set to "select guangzhou for a destination city in the land line management".
In step S240, a data adjustment condition is set.
The data adjustment condition may also be referred to as an execution condition, i.e. the modification of the fields of each temporal subsequence is performed according to a condition indication such that the content of the fields switches from satisfying one rule to satisfying another rule.
In this example, the data adjustment condition is set to "the departure time of the vehicles meeting the filtering condition is increased by 12 hours", so that the time sequence of the vehicles at the late departure time due to typhoon is corrected, and the normal monitoring of the departure time of the vehicles by the land transportation line management system in the following process is favorably ensured, so as to ensure the timeliness of the system response.
In step S250, SQL is generated for each time subsequence.
Step S260, SQL is executed in advance.
If the result of pre-executing SQL shows that the operation does not affect the time series data or the difference from the original data is within a preset reasonable range, step S270 may be executed, otherwise, step S230 or step S240 may be returned to re-perform the condition setting.
Step S270, formally executing SQL.
If the SQL is formally executed successfully, a new time sequence formed by splicing the time sequence 1, the time sequence 2 (corrected field) and the time sequence 3 can be obtained, and the integrity of the time sequence is ensured; if the execution of the SQL is not successful, the procedure may return to step S230 or step S240 to reset the condition setting.
In this example, there are nearly 5800 pieces of data to be processed in time for the land line management system caused by typhoon, and if the past method (i.e., one modification on the system) is adopted, 4 persons are needed to continuously operate for two days, while the tool configured by the time series processing method of the embodiment of the present invention is used, only 1 person is needed to operate for 15 minutes.
By the example, it can be known that by using the time series processing method of the embodiment of the present invention, when a problem caused by an irresistible factor is encountered, the field correction of the time series can be performed quickly in a short time, so as to ensure the quick response of the correlation system when performing subsequent time series analysis (such as sequence query).
It should be noted that, although the above example is directed to traffic data, the time series processing method according to the embodiment of the present invention is applicable to any industry that needs to acquire time series data. For example, it can be used for enterprise business data, such as a certain enterprise establishes a client consultation system to hope to obtain business opportunities from the client online stay, the system requires that the business personnel must process the client leave messages on the same day, but because some external factors (network connection disconnection, etc.), there may be some message delay, etc., so that the client consultation system loses its monitoring.
To sum up, corresponding to the disadvantages of weak timeliness, high labor cost and high profit and loss rate of the prior art in which a single data mode is adopted to process time series data, the time series processing method based on batch splitting and splicing of continuous time series in the embodiment of the present invention has at least the following three advantages:
1. the timeliness is strong. If the problems caused by the nonreactive factors are met, the system can quickly respond in a short time, and the reasonability of the acquisition time sequence is ensured.
2. The labor cost is reduced. Referring to the above example, the time series processing method according to the embodiment of the present invention can be implemented in a system tool manner, which can save a lot of human resources.
3. And the loss is reduced. Referring to the above example, while labor costs are reduced, losses due to non-reactable factors are also reduced because the time to deal with the problem is greatly reduced.
Fig. 4 is a schematic structural diagram of a time-series processing apparatus according to an embodiment of the present invention, which is based on the same inventive concept as the time-series processing method according to the above-described embodiment. As shown in fig. 4, the time-series processing means may include: a splitting unit 410, configured to split the time sequence to be processed into multiple consecutive time subsequences; the filtering unit 420 is configured to filter each time subsequence according to a preset filtering condition, so as to obtain a plurality of corresponding first time subsequences; an adjusting unit 430, configured to perform data adjustment processing on each segment of the first time subsequence according to a preset data adjustment condition, to obtain a corresponding plurality of segments of the second time subsequence, where the data adjustment processing includes correcting a specified field on the first time subsequence according to a preset rule; and a splicing unit 440, configured to splice the second time subsequences to obtain a new time sequence
In a preferred embodiment, the splitting unit 410 further comprises: a target time period determination module 411, configured to determine a target time period; and a sequence splitting module 412, configured to split the time series data into multiple continuous time subsequences according to the target time period, where one of the split multiple continuous time subsequences corresponds to the target time period.
Wherein the preset filtering condition comprises a sequence screening rule for screening the first time subsequence; and/or the preset data adjustment condition comprises an abnormal point correction rule for performing data adjustment on the abnormal point on the time series. Based on this, in a preferred embodiment, the adjusting unit 430 may include: an outlier determination module 431 configured to determine outliers on the first temporal subsequence; and a field correction module 432, configured to correct a field corresponding to the abnormal point according to the abnormal point correction rule.
In a preferred embodiment, the splicing unit 440 may include: the language generation module 441 is used for generating a structured description language SQL of each segment of the second time subsequence; and a first executing module 442, configured to formally execute the SQL, so as to splice the second time sub-sequences into the new time sequence.
In a more preferred embodiment, the splicing unit 440 may further include: a second execution module 443, configured to pre-execute the SQL before the first execution module formally executes the SQL, and calculate, according to a result of the pre-execution, data information about an influence of an operation of formally executing the SQL on the time series.
Wherein the second executing module 443 preferably includes: an insertion sub-module (not shown in the figure) for inserting a temporary table for storing the SQL to be executed; and an execution submodule (not shown in the figure) for executing the SQL based on the temporary table, and detecting and updating the concatenated field of the SQL and performing data check during the execution.
It should be noted that, for more details and effects of the time-series processing apparatus according to the embodiment of the present invention, reference may be made to the above-mentioned embodiments related to the time-series processing method, and details are not repeated herein.
In addition, it should be further noted that the time-series processing apparatus may include a processor and a memory, where the splitting unit, the filtering unit, the adjusting unit, the splicing unit, and the like are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.
The processor comprises a kernel, and the kernel calls a corresponding program unit from the memory. One or more than one kernel can be set, and the time sequence processing of the embodiment of the invention is realized by adjusting the kernel parameters. The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
The embodiment of the invention also provides a machine-readable storage medium, which stores instructions for causing a machine to perform the time-series processing method of the above embodiment.
The embodiment of the invention also provides a processor, wherein the processor is used for running the program, and the time sequence processing method of the embodiment is executed when the program runs.
The embodiment of the present invention further provides an apparatus, where the apparatus includes a processor, a memory, and a program stored in the memory and capable of being executed on the processor, and when the processor executes the program, the steps of the time-series processing method according to the following embodiments are implemented. The device in the embodiment of the invention can be a server, a PC, a PAD, a mobile phone and the like.
An embodiment of the present invention further provides a computer program product, which, when executed on a data processing apparatus, is adapted to execute a program that initializes the following method steps: the time-series processing method of the above embodiment.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A time-series processing method, characterized by comprising:
splitting a time sequence to be processed into a plurality of sections of continuous time subsequences;
respectively filtering each time subsequence according to a preset filtering condition to obtain a plurality of corresponding first time subsequences;
respectively carrying out data adjustment processing on each segment of first time subsequence according to a preset data adjustment condition to obtain a plurality of corresponding segments of second time subsequences, wherein the data adjustment processing comprises correcting a designated field on the first time subsequence according to a preset rule; and
and splicing the second time subsequences of all the sections to obtain a new time sequence.
2. The time series processing method according to claim 1, wherein the splitting the time series data to be processed into a plurality of continuous time subsequences comprises:
determining a target time period; and
according to the target time period, the time sequence data is divided into a plurality of continuous time subsequences, wherein one of the divided continuous time subsequences corresponds to the target time period.
3. The time-series processing method according to claim 1,
the preset filtering condition comprises a sequence screening rule for screening the first time subsequence; and/or
The preset data adjustment condition comprises an abnormal point correction rule used for performing data adjustment on the abnormal point on the time sequence.
4. The method according to claim 3, wherein the performing the data adjustment processing on each of the first time subsequences according to the preset data adjustment condition comprises:
determining outliers on the first temporal subsequence; and
and correcting the field corresponding to the abnormal point according to the abnormal point correction rule.
5. The time series processing method according to claim 1, wherein said concatenating each second time subsequence comprises:
generating a structured description language (SQL) of each second time subsequence; and
formally executing the SQL to splice the second time subsequences into the new time sequence.
6. The method of time-series processing according to claim 5, further comprising, before said formally executing said SQL:
and pre-executing the SQL, and calculating the data information of the influence of the operation of formally executing the SQL on the time sequence according to the pre-executed result.
7. The method of time series processing according to claim 6, wherein said pre-executing said SQL comprises:
inserting a temporary table for storing the SQL to be executed; and
and executing the SQL based on the temporary table, and detecting and updating the cascading fields of the SQL and checking data in the executing process.
8. A time-series processing apparatus, characterized in that the time-series processing apparatus comprises:
the splitting unit is used for splitting the time sequence to be processed into a plurality of sections of continuous time subsequences;
the filtering unit is used for respectively filtering each time subsequence according to a preset filtering condition to obtain a plurality of corresponding first time subsequences;
the adjusting unit is used for respectively carrying out data adjustment processing on each segment of the first time subsequence according to a preset data adjustment condition to obtain a plurality of corresponding segments of the second time subsequence, wherein the data adjustment processing comprises correcting a designated field on the first time subsequence according to a preset rule; and
and the splicing unit is used for splicing the second time subsequences of all the sections to obtain a new time sequence.
9. A machine-readable storage medium having stored thereon instructions for causing a machine to execute the time-series processing method of any one of claims 1 to 7.
10. A processor configured to execute a program, wherein the program is configured to perform: the time-series processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911037564.1A CN110990449A (en) | 2019-10-29 | 2019-10-29 | Time series processing method, device, storage medium and processor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911037564.1A CN110990449A (en) | 2019-10-29 | 2019-10-29 | Time series processing method, device, storage medium and processor |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110990449A true CN110990449A (en) | 2020-04-10 |
Family
ID=70082717
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911037564.1A Pending CN110990449A (en) | 2019-10-29 | 2019-10-29 | Time series processing method, device, storage medium and processor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110990449A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114363062A (en) * | 2021-12-31 | 2022-04-15 | 深信服科技股份有限公司 | Domain name detection method, system, equipment and computer readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107885869A (en) * | 2017-11-24 | 2018-04-06 | 北京农信互联科技有限公司 | A kind of method and system for changing database |
| CN108595528A (en) * | 2018-03-29 | 2018-09-28 | 重庆大学 | A kind of multivariate time series are based on Fourier coefficient symbolism classification set creation method |
| CN109491289A (en) * | 2018-11-15 | 2019-03-19 | 国家计算机网络与信息安全管理中心 | A kind of dynamic early-warning method and device for data center's dynamic environment monitoring |
| CN109871401A (en) * | 2018-12-26 | 2019-06-11 | 北京奇安信科技有限公司 | A kind of time series method for detecting abnormality and device |
-
2019
- 2019-10-29 CN CN201911037564.1A patent/CN110990449A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107885869A (en) * | 2017-11-24 | 2018-04-06 | 北京农信互联科技有限公司 | A kind of method and system for changing database |
| CN108595528A (en) * | 2018-03-29 | 2018-09-28 | 重庆大学 | A kind of multivariate time series are based on Fourier coefficient symbolism classification set creation method |
| CN109491289A (en) * | 2018-11-15 | 2019-03-19 | 国家计算机网络与信息安全管理中心 | A kind of dynamic early-warning method and device for data center's dynamic environment monitoring |
| CN109871401A (en) * | 2018-12-26 | 2019-06-11 | 北京奇安信科技有限公司 | A kind of time series method for detecting abnormality and device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114363062A (en) * | 2021-12-31 | 2022-04-15 | 深信服科技股份有限公司 | Domain name detection method, system, equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11968106B2 (en) | Method and apparatus for monitoring state of blockchain consensus node | |
| CN110209560B (en) | Data anomaly detection method and detection device | |
| CN110650036A (en) | Alarm processing method and device and electronic equipment | |
| CN107153593B (en) | Method and device for determining Internet service monitoring threshold | |
| CN108492150B (en) | Method and system for determining entity heat degree | |
| CN111090502A (en) | Streaming data task scheduling method and device | |
| CN114911800A (en) | Fault prediction method and device for power system and electronic equipment | |
| CN110990449A (en) | Time series processing method, device, storage medium and processor | |
| CN111309696A (en) | Log processing method and device, electronic equipment and readable medium | |
| CN109992470B (en) | Threshold value adjusting method and device | |
| CN113742036B (en) | Index processing method and device and electronic equipment | |
| CN115658441B (en) | Method, equipment and medium for monitoring abnormality of household service system based on log | |
| CN111680293A (en) | Information monitoring management method and device based on Internet and server | |
| CN108667893B (en) | Data recommendation method and device and electronic equipment | |
| CN111523844A (en) | Storage management method, device, server and readable storage medium | |
| CN110704223A (en) | Recovery system and method for single-node abnormity of database | |
| CN114338435B (en) | Network change monitoring method, device, computer equipment and storage medium | |
| CN110991945B (en) | Information processing method and device for equipment spare part, storage medium and processor | |
| CN111369373A (en) | Vehicle interior damage determination method and device | |
| CN113067713A (en) | Method and device for adding engineering identification to alarm network element | |
| CN114124758B (en) | Flow monitoring method and device | |
| CN110908823A (en) | Operation maintenance method and device for big data cluster | |
| CN113657720B (en) | Method, system, medium and equipment for automatically planning CDN server disk | |
| CN115794195B (en) | Banking business management and maintenance method and device based on Viper framework | |
| CN113010424B (en) | Interface automatic test processing method, system, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200428 Address after: 100085 Floor 102-1, Building No. 35, West Second Banner Road, Haidian District, Beijing Applicant after: Seashell Housing (Beijing) Technology Co.,Ltd. Address before: 300280 unit 05, room 112, floor 1, building C, comprehensive service area, Nangang Industrial Zone, Binhai New Area, Tianjin Applicant before: BEIKE TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200410 |