CN107798009A - Data aggregation method, apparatus and system - Google Patents

Data aggregation method, apparatus and system Download PDF

Info

Publication number
CN107798009A
CN107798009A CN201610795892.8A CN201610795892A CN107798009A CN 107798009 A CN107798009 A CN 107798009A CN 201610795892 A CN201610795892 A CN 201610795892A CN 107798009 A CN107798009 A CN 107798009A
Authority
CN
China
Prior art keywords
business datum
data
timestamp
polymerization
aggregate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610795892.8A
Other languages
Chinese (zh)
Inventor
李巨雷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201610795892.8A priority Critical patent/CN107798009A/en
Publication of CN107798009A publication Critical patent/CN107798009A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2282Tablespace storage structures; Management thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/215Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2458Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2477Temporal data queries

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Fuzzy Systems (AREA)
  • Quality & Reliability (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention discloses a kind of data aggregation method, apparatus and system, it is related to areas of information technology, the efficiency of data aggregate can be improved, can also solves the problems, such as that data aggregate operation has delay, and then ensure the real-time of data aggregate.The present invention main technical schemes be:When the magnanimity business datum stored in needing to cloud environment polymerize according to some cycles, the business datum being currently received first from the data collector acquisition data collector, then judge whether the timestamp is more than polymerization triggered time point corresponding to the first during cycle described in when being currently needed for carrying out data aggregate, if being more than, business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.Present invention is mainly used for the magnanimity business datum in cloud environment is polymerize.

Description

Data aggregation method, apparatus and system
Technical field
The present invention relates to areas of information technology, more particularly to a kind of data aggregation method, apparatus and system.
Background technology
With the continuous development of internet, the storage of business datum is carried out under cloud environment with counting also more and more common. For example, different classes of to QPS (Query Per Second, query rate per second), PV (Page View, page browsing amount) etc. Business datum is stored and counted, and detailed process can be as shown in figure 11, first by the mass data of streaming be stored in file or In other storage mediums, file is then read at regular intervals, and file content is counted by the way of MapReduce Calculate.Because the data volume stored under cloud environment is very big, therefore the magnanimity business datum stored in cloud environment counts When, it is necessary to polymerize to the business datum of magnanimity.
At present, when the magnanimity business datum stored in cloud environment polymerize, different time sections are received first Business datum be written in file, the whole business datums of a certain classification for needing to polymerize then are read from file, finally Data aggregate is carried out after the business datum of reading is ranked up according to the time.However, due to when carrying out data aggregate, it is necessary to Operation is written and read to file, and in order to solve the problems, such as to carry out existing file lock conflict during file read-write operations, meeting The file reproduction process of data to be polymerized is once preserved in data aggregate, so as to cause data calculate delay compared with It is high;And due to also needing to the business datum of reading carrying out a minor sort according to the time, so as to cause the process of data aggregate Relatively complicated, data aggregate operates less efficient, and data aggregate operating delay is higher.
The content of the invention
In view of this, the embodiment of the present invention provides a kind of data aggregation method, apparatus and system, and main purpose is that solve The problem of data aggregate operating delay is higher be present when polymerizeing to the magnanimity business datum stored in cloud environment.
To reach above-mentioned purpose, the present invention provides following technical scheme:
On the one hand, the present invention provides a kind of data aggregation method, including:
When carrying out data aggregate according to the first during cycle, obtain the data collector from data collector and currently connect The business datum received, the business datum carry timestamp, and the timestamp is used to identify the data collector reception To the time of the business datum;
Judge whether the timestamp is more than corresponding to the first during cycle described in when being currently needed for carrying out data aggregate to gather Triggered time point is closed, first during cycle corresponds to multiple polymerization triggered time points, the multiple polymerization triggered time point minute The not corresponding time is incremented by one by one;
If so, gather described in during then by the timestamp of preservation to be currently needed for carrying out data aggregate corresponding to the first during cycle The business datum for closing triggered time point is polymerize.
On the other hand, the present invention provides a kind of method of data capture, including:
Receive business datum to be polymerized;
Stabbed for the business datum setup time, the timestamp receives the industry for identifying the data collector The time for data of being engaged in,
Obtain the business datum so as to which aggregate server must be analyzed and line number is entered according to the timestamp of the business datum According to polymerization.
Another further aspect, the present invention provide a kind of aggregated data querying method, including:
Data are sent to analysis aggregate server and report instruction, and the data report in instruction and carry carry out data in need The business datum of polymerization, so that must analyze aggregate server receives time of the business datum and pre- according to data collector Polymerization triggered time point is put, according to during cycle to needing the business datum for carrying out data aggregate to carry out data aggregate;
The instruction of data aggregate result queries is sent to the analysis aggregate server, is carried in the data query instruction The temporal information of business datum to be checked;
Receive data aggregate result corresponding with the temporal information and shown.
Another aspect, the present invention provide a kind of data aggregate device, including:
Acquiring unit, for when carrying out data aggregate according to the first during cycle, the number to be obtained from data collector The business datum being currently received according to collector, the business datum carry timestamp, and the timestamp is described for identifying Data collector receives the time of the business datum;
Judging unit, whether the timestamp for judging the acquiring unit acquisition, which is more than, is currently needed for carrying out data Polymerization triggered time point corresponding to first during cycle described in during polymerization, when first during cycle corresponds to multiple polymerization triggerings Between point, it is the multiple polymerization the triggered time point respectively corresponding to the time one by one be incremented by;
Polymerized unit, if determining to be more than for the judging unit first preset described in when being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to cycle, then business datum of the timestamp of preservation for the polymerization triggered time point is gathered Close.
Another aspect, the present invention provide a kind of transacter, including:
Receiving unit, for receiving business datum to be polymerized;
Dispensing unit, for being stabbed for the business datum setup time, the timestamp is used to identify the Data Collection Device receives the time of the business datum,
Obtain the business datum so as to which aggregate server must be analyzed and line number is entered according to the timestamp of the business datum According to polymerization.
Another aspect, the present invention provide a kind of aggregated data inquiry unit, including:
Transmitting element, instruction is reported for sending data to analysis aggregate server, the data report to be carried in instruction The business datum in need for carrying out data aggregate, so that must analyze aggregate server receives the business according to data collector The time of data and preset polymerization triggered time point, enter line number according to during cycle to the business datum for needing to carry out data aggregate According to polymerization;
The transmitting element, it is additionally operable to send the instruction of data aggregate result queries to the analysis aggregate server, it is described The temporal information of business datum to be checked is carried in data query instruction;
Receiving unit, for receiving data aggregate result corresponding with the temporal information and being shown.
Another aspect, the present invention provide a kind of data aggregate system, including:Data collector and analysis aggregate server;
The data collector, during for receiving business datum, stabbed for the business datum setup time, and will described in The timestamp of business datum and the business datum is preserved;
The analysis aggregate server, for when according to the first during cycle carry out data aggregate when, from data collector The business datum that the data collector is currently received is obtained, the business datum carries timestamp;
The analysis aggregate server, when being additionally operable to whether judge the timestamp more than being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to first during cycle, first during cycle correspond to multiple polymerization triggered time points, Time corresponding to the multiple polymerization triggered time point difference is incremented by one by one;
If so, then business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.
By above-mentioned technical proposal, technical scheme provided in an embodiment of the present invention at least has following advantages:
A kind of data aggregation method provided in an embodiment of the present invention, apparatus and system, are stored in needing to cloud environment When magnanimity business datum is polymerize according to some cycles, obtain the data collector from data collector first and currently receive The business datum arrived, then judges whether the timestamp is more than the first during cycle described in when being currently needed for carrying out data aggregate Corresponding polymerization triggered time point, if being more than, business datum of the timestamp of preservation for the polymerization triggered time point is entered Row polymerization.With when the magnanimity business datum stored in cloud environment polymerize, being required at present to for storage service number According to file be written and read operation, and it also requires by the business datum of reading according to the time carry out a minor sort compare, this hair It is bright the described in when being currently needed for carrying out data aggregate due to only needing whether to be more than by the timestamp that detects current service data Polymerization triggered time point corresponding to one during cycle, you can realize and polymerize to the business datum of different time sections, improve number According to the efficiency of polymerization, and because detection operation is carried out in real time, the delay of data aggregate operation can be reduced, while can be with Improve the real-time of data aggregate.
Described above is only the general introduction of technical solution of the present invention, in order to better understand the technological means of the present invention, And can be practiced according to the content of specification, and in order to allow above and other objects of the present invention, feature and advantage can Become apparent, below especially exemplified by the embodiment of the present invention.
Brief description of the drawings
By reading the detailed description of hereafter preferred embodiment, it is various other the advantages of and benefit it is common for this area Technical staff will be clear understanding.Accompanying drawing is only used for showing the purpose of preferred embodiment, and is not considered as to the present invention Limitation.And in whole accompanying drawing, identical part is denoted by the same reference numerals.In the accompanying drawings:
Fig. 1 shows a kind of flow chart of data aggregation method provided in an embodiment of the present invention;
Fig. 2 shows the flow chart of another data aggregation method provided in an embodiment of the present invention;
Fig. 3 shows a kind of flow chart of aggregated data querying method provided in an embodiment of the present invention;
Fig. 4 shows a kind of flow chart of method of data capture provided in an embodiment of the present invention;
Fig. 5 shows a kind of data aggregate apparatus structure schematic diagram provided in an embodiment of the present invention;
Fig. 6 shows another data aggregate apparatus structure schematic diagram provided in an embodiment of the present invention;
Fig. 7 shows a kind of aggregated data inquiry unit structural representation provided in an embodiment of the present invention;
Fig. 8 shows a kind of aggregated data statistic device structural representation provided in an embodiment of the present invention;
Fig. 9 shows a kind of data aggregate system structure diagram provided in an embodiment of the present invention;
Figure 10 shows business datum circulation schematic diagram provided in an embodiment of the present invention;
Figure 11 shows the schematic diagram of data statistical approach framework in the prior art;
Figure 12 shows aggregated data querying method logical architecture schematic diagram provided in an embodiment of the present invention;
Figure 13 shows the logical construction schematic diagram of portal server provided in an embodiment of the present invention.
Embodiment
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although the disclosure is shown in accompanying drawing Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here Limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure Completely it is communicated to those skilled in the art.
A kind of data aggregation method provided in an embodiment of the present invention, as shown in figure 1, methods described includes:
101st, when carrying out data aggregate according to the first during cycle, obtain the data collector from data collector and work as Before the business datum that receives.
Wherein, first during cycle is used to react the time granularity for carrying out data aggregate, is gathered according to different data Conjunction demand can configure different time granularities., can be by first such as when needing to carry out the less data aggregate of time granularity During cycle configuration it is relatively smaller, polymerization such as per minute once, polymerization in every five minutes it is one inferior, such as when needing to carry out the time During the larger data aggregate of granularity, the first during cycle can be configured relatively larger, such as polymerize per hour once, per half Its polymerization one is inferior.For the embodiment of the present invention, different polymerization time granularities is configured according to different demands, can be further Improve the precision of data aggregate.The business datum carries timestamp, and the timestamp is used to identify the data collector Receive the time that the business datum triggers after the business datum.For example, data collector receives the industry after business datum The time for data-triggered of being engaged in is 10:01, then the timestamp currently carried from the business datum of data collector acquisition is 10: 01.The business datum can be QPS (Query Per Second, query rate per second), PV (Page View, page browsing Amount) etc. different classes of business datum, or the business datum of same category different application, such as using A and application B PV Data, the embodiment of the present invention do not limit.
102nd, it is corresponding to judge whether the timestamp is more than the first during cycle described in when being currently needed for carrying out data aggregate Polymerization triggered time point.
Wherein, first during cycle corresponds to multiple polymerization triggered time points, the multiple polymerization triggered time point minute The not corresponding time is incremented by one by one, and it is upper one polymerization triggered time to timestamp that each polymerization triggered time point, which is used to trigger, Whole business datums of point carry out converging operation.For example, the first during cycle polymerize once to be per minute, then from 10:00 starts, It is respectively 10 that first during cycle, which corresponds to multiple polymerization triggered time points,:01、10:02、10:03 grade by that analogy, works as timestamp For 10:02 is more than polymerization triggered time point 10 corresponding to the first during cycle described in when currently carrying out data aggregate:When 01, triggering It is 10 to carry out timestamp:The polymerization processing of 01 whole business datums.
It should be noted that the multiple polymerization triggered time point can be sequentially in time with the side such as list, time shaft Formula is shown, and the embodiment of the present invention does not limit, for example, when multiple polymerization triggered time points are sequentially in time with time shaft When being shown etc. mode, time on time shaft can continuous rolls forward, to ensure the time on time shaft as current time. For the embodiment of the present invention, by detect the timestamp whether be more than be currently needed for carrying out data aggregate when described in it is first preset Polymerization triggered time point, that is, detect the time for the business datum being currently received and the last time receive business number corresponding to cycle According to time whether change, determine whether trigger business datum converging operation, business datum is entered so as to ensure that The real-time of row converging operation.
For example, the first during cycle polymerize once to be per minute, if the industry being currently received obtained from data collector The timestamp that data of being engaged in carry is 10:01, polymerization corresponding to the first during cycle described in when being currently needed for carrying out data aggregate is touched It is all 10 to send out time point:01, it is determined that do not trigger business datum converging operation;If it is currently received from what data collector obtained Business datum carry timestamp be 10:02, it is corresponding more than the first during cycle described in when being currently needed for carrying out data aggregate Polymerization triggered time point 10:01, it is determined that triggering business datum converging operation.
If the 103, being more than polymerization triggered time point corresponding to the first during cycle described in when being currently needed for carrying out data aggregate, Then business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.
Wherein, the timestamp is stored in preset storage location for the business datum of the polymerization triggered time point.It is described Preset storage location can be preset mapping table, preset queue etc., and the embodiment of the present invention does not limit.For of the invention real Example is applied, if the timestamp is more than the polymerization triggered time corresponding to the first during cycle described in when being currently needed for carrying out data aggregate Point, such as by 10:01 becomes for 10:02 or by 10:01:01 becomes for 10:01:02, illustrate current service data carry when Between stab and changed forward, i.e. triggering converging operation in real time so as to ensure that the real-time of business datum converging operation, improves The converging operation efficiency of business datum.
A kind of data aggregation method provided in an embodiment of the present invention, the magnanimity business datum stored in needing to cloud environment When being polymerize according to some cycles, the business number that the data collector is currently received is obtained from data collector first According to then judging whether the timestamp is more than described in when being currently needed for carrying out data aggregate and polymerize corresponding to the first during cycle Triggered time point, if being more than, business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.With mesh It is preceding when the magnanimity business datum stored in cloud environment polymerize, be required for for storage service data file progress Read-write operation, and it also requires by the business datum of reading according to the time carry out a minor sort compare, the embodiment of the present invention due to Only need whether to be more than by the timestamp for detecting current service data first preset described in when being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to cycle, you can realize and polymerize to the business datum of different time sections, improve data aggregate Efficiency, and because detection operation is carried out in real time, the delay of data aggregate operation can be reduced, while number can be improved According to the real-time of polymerization.
Further, the embodiment of the present invention provides another data aggregation method, as shown in Fig. 2 methods described includes:
201st, when carrying out data aggregate according to the first during cycle, obtain the data collector from data collector and work as Before the business datum that receives.
Wherein, the business datum carries timestamp, and the timestamp receives for identifying the service server The time of the business datum.The corresponding description of first during cycle and the business datum is referred to real described in Fig. 1 The corresponding description of example is applied, will not be repeated here.
For the embodiment of the present invention, step 201 can specifically include:When according to the first during cycle progress data aggregate When, from when obtaining the business datum in the corresponding preset queue of pre-polymerization task, the data collection server is each A preset queue is respectively configured in the business datum of individual classification.Specifically can be as shown in Figure 10, it can be configured in data collector The transponders such as ONS (messenger service), Metaq (message-oriented middleware), the transponder are used to carry out data collector and polymerization analysis Stream compression between server.Specifically, data collector can be by preset routing rule by different classes of business number According to different dummy nodes 0,1,2 etc. are corresponded to respectively, then every a kind of business datum is saved in respectively by dummy node In ONS different queue Queue1, Queue2 etc., then for each queue a task, such as task is respectively configured ServerNode1, so as to ensure that category carries out business datum circulation and follow-up data polymerization processing, further improve data and gather Close precision.
202nd, it is corresponding to judge whether the timestamp is more than the first during cycle described in when being currently needed for carrying out data aggregate Polymerization triggered time point.
Wherein, first during cycle corresponds to multiple polymerization triggered time points, the multiple polymerization triggered time point minute The not corresponding time is incremented by one by one.The specific descriptions of the configuration mode of first during cycle are referred to implementation described in Fig. 1 The corresponding description of example, will not be repeated here.
For the embodiment of the present invention, by detect the timestamp whether be more than be currently needed for carrying out data aggregate when described in Polymerization triggered time point corresponding to first during cycle, that is, the time for detecting the business datum being currently received receive with last Whether the time to business datum changes forward, determines whether to trigger business datum converging operation, so as to ensure that The real-time of converging operation is carried out to business datum.
Further, can also include after step 202:If the timestamp is identical with the polymerization triggered time point, Then the business datum being currently received is saved in the preset queue corresponding with working as pre-polymerization task.Wherein, institute State and whole business datums of the timestamp for polymerization triggered time point are cached with preset queue.Different preset queues are cached with respectively It polymerize the different business datum of triggered time point, due to being cached to queue and the delay of read operation is relatively low, therefore, by industry Data buffer storage be engaged in into queue, can further reduce the delay of data aggregate operation, improves the real-time of data aggregate operation. If the timestamp is less than the polymerization triggered time point, the business datum being currently received is deleted.Such as it polymerize tactile It is 10 to send out time point:01, timestamp 09:50, then it is 09 by the timestamp being currently received:50 business datum is deleted, from And ensure the accuracy of data aggregate.
If the 203, being more than polymerization triggered time point corresponding to the first during cycle described in when being currently needed for carrying out data aggregate, Judge to be currently received in the first preset time period after the business datum, if receive timestamp and touch for the polymerization Send out the business datum at time point.
It should be noted that after the business datum of some timestamp is polymerize, due to originals such as transmission data delays Cause, data collector are possible to receive the business datum of the timestamp again, now, if the business not received to delay Data are handled, and can cause the loss of business datum, and then make it that the accuracy of business datum polymerization is relatively low, therefore, are being entered Before row data aggregate, tolerated the phase by setup time, i.e. the first preset time period, the business number that delay can be avoided to receive According to loss, improve business datum polymerization accuracy.
Wherein, first preset time period can be configured according to the first during cycle.If for example, the first preset week Phase is smaller, and such as polymerization each second once, then the first preset time period can be 1 second, 2 seconds etc., if the first during cycle is more suitable In, once, then the first preset time period can be 10 seconds, 20 seconds etc., such as every if the first during cycle is larger for polymerization such as per minute Once, then the first preset time period can be 10 minutes, 20 minutes etc. for hour polymerization.By according to the first during cycle configuration the One preset time period, the configuration accuracy of tolerance phase time can be further improved, and then further improve business datum polymerization Accuracy.
If 204a, not receiving, business datum of the timestamp of preservation for the polymerization triggered time point is gathered Close.
If the step 204b arranged side by side with step 204a, receiving, by what is received within first preset time period The business datum and the timestamp preserved are polymerize for the business datum of the polymerization triggered time point.
For example, the first during cycle polymerize once to be per minute, then now preset buffer time section is configured to 5 minutes. If the timestamp that the business datum currently obtained from data collector carries is 10:02, it is currently needed for carrying out data aggregate when institute It is 10 to state polymerization triggered time point corresponding to the first during cycle:01, then it is being 10 to timestamp:01 business datum is gathered Before conjunction, the timestamp of carrying whether is received in five minutes after judging since current time as 10:01 business datum, if Receive, be then 10 by the timestamp received in five minutes:01 business datum and the timestamp received before are 10: 01 business datum carries out data aggregate;If not receiving, illustrate the business datum that delay is now not present, be by timestamp 10:01 business datum carries out data aggregate, so as to ensure that the accuracy of data aggregate.
205th, timestamp corresponding to the business datum after polymerization and the business datum is stored in into the first during cycle to gather Close in table.
Wherein, the first during cycle Aggregation Table is used to preserve the data after according to the first during cycle being polymerize.It is described First during cycle Aggregation Table can be stored in the storages such as OTS (form storage), HBase (the distributed database towards row) and be situated between In matter.If for example, the first during cycle for polymerization per minute once, preserved in the first during cycle Aggregation Table to be per minute Data after polymerization.The first during cycle Aggregation Table can be with as follows:
For the embodiment of the present invention, in order to realize while carry out the data aggregate of different time granularity, gone back after step 205 It can include:When needing to carry out data aggregate according to the second during cycle simultaneously, the number is obtained from the data collector The timestamp for the business datum being currently received according to collector, second during cycle are more than first during cycle, and And second during cycle is the integral multiple of first during cycle;Then judge whether the timestamp reaches described Polymerization triggered time point corresponding to two during cycles, second during cycle corresponds to multiple polymerization triggered time points, described more Time corresponding to individual polymerization triggered time point difference is incremented by one by one;If so, then obtained from the first during cycle Aggregation Table Timestamp is located at a upper polymerization triggered time point corresponding to second during cycle to when in pre-polymerization triggered time point Business datum is polymerize, and the business datum after polymerization is stored in the second during cycle Aggregation Table.
For example, the first during cycle is once polymerize to be per minute, the second during cycle is to carry out once for every five minutes Polymerization, the timestamp for the business datum being currently received is 10:06, more than polymerization triggered time point corresponding to the second during cycle 10:05, then timestamp is obtained from 1 minute Aggregation Table and be located at 10:01 to 10:05 business datum is polymerize, and will polymerization Business datum afterwards is stored in 5 minutes Aggregation Tables.
For the embodiment of the present invention, the business datum by after polymerization is stored in the step in the second during cycle Aggregation Table It can also include before rapid:It is the business datum after the polymerization to polymerize the triggered time according to corresponding to current second during cycle Setup time stabs, and is now 10 for the business datum setup time stamp after the polymerization based on the example above:05.Now, will be poly- Timestamp corresponding to business datum and the business datum after conjunction is stored in the second during cycle Aggregation Table.
Further, can also include after step 205:The data query instruction that data query server is sent is received, Temporal information is carried in the data query instruction;From the first during cycle Aggregation Table obtain timestamp with it is described when Between business datum corresponding to information and be sent to the inquiry server, to cause the inquiry server to carry out data display. For example, when the temporal information is 10:01-10:05, acquisition timestamp is respectively from the first during cycle Aggregation Table 10:01、10:02、10:03、10:04、10:05 aggregated data.
Wherein, the inquiry server can be Portal (portal server), and the Portal can provide all kinds Query interface, so as to client by call Portal provide data-interface polymerize after business datum inquiry. Meanwhile Portal can also provide the registration interface of business datum, so that data collector is according to the registration preserved in Portal Information carries out log-on message checking to business datum.
For the embodiment of the present invention, will can also polymerize corresponding to first during cycle and current first during cycle Triggered time point is stored in preset storage location.By will gather corresponding to first during cycle and current first during cycle Close triggered time point and be stored in preset storage location, it is ensured that when system reboot or system break down recovery, reduction To restart or failure before time state, so as to prevent loss of data and data are computed repeatedly, further improve data The precision of polymerization.
Another data aggregation method provided in an embodiment of the present invention, the magnanimity business number stored in needing to cloud environment During according to being polymerize according to some cycles, the business number that the data collector is currently received is obtained from data collector first According to then judging whether the timestamp is more than described in when being currently needed for carrying out data aggregate and polymerize corresponding to the first during cycle Triggered time point, if being more than, business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.With mesh It is preceding when the magnanimity business datum stored in cloud environment polymerize, be required for for storage service data file progress Read-write operation, and it also requires by the business datum of reading according to the time carry out a minor sort compare, the embodiment of the present invention due to Only need whether to be more than by the timestamp for detecting current service data first preset described in when being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to cycle, you can realize and polymerize to the business datum of different time sections, improve data aggregate Efficiency, and because detection operation is carried out in real time, the delay of data aggregate operation can be reduced, while number can be improved According to the real-time of polymerization.
Further, the embodiment of the present invention provides a kind of aggregated data querying method, as shown in figure 3, methods described includes:
301st, data are sent to analysis aggregate server and reports instruction.
Wherein, the data report the business datum that progress data aggregate in need is carried in instruction, so as to must analyze poly- Hop server receives the time of the business datum and preset polymerization triggered time point according to data collector, according to preset week Phase is to needing the business datum for carrying out data aggregate to carry out data aggregate.For the embodiment of the present invention, step 301 can specifically wrap Include:Instruction is reported by calling Portal interfaces to send data to data aggregating server.
For the embodiment of the present invention, Portal building-block of logic can be as shown in figure 13, in embodiments of the present invention Portal has two parts effect, and first is the inquiry for carrying out aggregated data, has provided the user all kinds business datum Query interface, so as to inquire about polymerization result from the storage server OTS in paradigmatic system by query interface, and return to visitor Family end carries out aggregated data displaying.Second is the registration for carrying out reported data metamessage, then writes log-on message in RDS, For carrying out data check when collector receives data.
302nd, the instruction of data aggregate result queries is sent to the analysis aggregate server.
Wherein, the temporal information of business datum to be checked is carried in the data query instruction.Implement for the present invention Example, step 302 can specifically include:Taken by the data-query interfaces that portal server Portal is configured to the analysis polymerization Device transmission data aggregate result queries of being engaged in instruct.
303rd, receive data aggregate result corresponding with the temporal information and shown.
For the embodiment of the present invention, concrete application scene can be as shown in figure 12, including:It is as follows to manage flow:Call first Portal interfaces need the monitored item of reported data to paradigmatic system registration, then call collector to report interface, in system Report specifies the data of monitored item, and now the data received are sent to middleware ONS by collector, and then analyzer will be from ONS Middle pulling data carries out data analysis and polymerization, and is stored in after data are carried out into polymerization calculating in OTS (storage server).When need When carrying out aggregated data inquiry, the query interface that user can be provided by Portal inquires about the result of aggregated data, and leads to Cross the statistical result of data display page display data.
For the embodiment of the present invention, provided the user by portal server Portal and report and inquire about connecing for aggregated data Mouthful, i.e., user can conveniently carry out the inquiry reported with polymerization result of data to be polymerized, and then can improve and wait to gather That closes data reports efficiency and the search efficiency to polymerization result.
Further, the embodiment of the present invention provides a kind of method of data capture, as shown in figure 4, methods described includes:
401st, business datum to be polymerized is received.402nd, stabbed for the business datum setup time.
Wherein, the timestamp is used to identify the time that the data collector receives the business datum, to cause Aggregate server is analyzed to obtain the business datum and carry out data aggregate according to the timestamp of the business datum.
For the embodiment of the present invention, by the business datum setup time stamp to receive, it is ensured that triggering in time The polymerization of business datum, so as to realize the real time aggregation to business datum.
Further, the specific implementation as method shown in Fig. 1, the embodiment of the present invention provide a kind of data aggregate device, As shown in figure 5, described device can include:Acquiring unit 51, judging unit 52, polymerized unit 53.
Acquiring unit 51, for when according to the first during cycle carry out data aggregate when, from data collector obtain described in The business datum that data collector is currently received, the business datum carry timestamp, and the timestamp is used to identify institute State the time that data collector receives the business datum;
Judging unit 52, whether the timestamp for judging the acquisition of acquiring unit 51, which is more than to work as, needs into line number According to triggered time point is polymerize corresponding to the first during cycle described in during polymerization, first during cycle correspond to multiple polymerizations and triggered Time point, time corresponding to the multiple polymerization triggered time point difference are incremented by one by one;
Polymerized unit 53, if determining to be more than for the judging unit 52 first described in when being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to during cycle, then business datum of the timestamp of preservation for the polymerization triggered time point is entered Row polymerization.
It should be noted that the device embodiment is corresponding with preceding method embodiment, and for ease of reading, present apparatus embodiment No longer the detail content in preceding method embodiment is repeated one by one, it should be understood that the device in the present embodiment can The corresponding full content realized in preceding method embodiment.
A kind of data aggregate device provided in an embodiment of the present invention, the magnanimity business datum stored in needing to cloud environment When being polymerize according to some cycles, the business number that the data collector is currently received is obtained from data collector first According to then judging whether the timestamp is more than described in when being currently needed for carrying out data aggregate and polymerize corresponding to the first during cycle Triggered time point, if being more than, business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.With mesh It is preceding when the magnanimity business datum stored in cloud environment polymerize, be required for for storage service data file progress Read-write operation, and it also requires by the business datum of reading according to the time carry out a minor sort compare, the embodiment of the present invention due to Only need whether to be more than by the timestamp for detecting current service data first preset described in when being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to cycle, you can realize and polymerize to the business datum of different time sections, improve data aggregate Efficiency, and because detection operation is carried out in real time, the delay of data aggregate operation can be reduced, while number can be improved According to the real-time of polymerization.
Further, the specific implementation as method shown in Fig. 2, the embodiment of the present invention provide a kind of data aggregate device, As shown in fig. 6, described device can include:Acquiring unit 61, judging unit 62, polymerized unit 63.
Acquiring unit 61, for when according to the first during cycle carry out data aggregate when, from data collector obtain described in The business datum that data collector is currently received, the business datum carry timestamp, and the timestamp is used to identify institute State the time that data collector receives the business datum;
Judging unit 62, whether the timestamp for judging the acquisition of acquiring unit 61, which is more than, is currently needed for carrying out Polymerization triggered time point corresponding to first during cycle described in during data aggregate, first during cycle correspond to multiple polymerizations and touched Time point is sent out, the time corresponding to the multiple polymerization triggered time point difference is incremented by one by one;
Polymerized unit 63, if determining to be more than for the judging unit 62 first described in when being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to during cycle, then business datum of the timestamp of preservation for the polymerization triggered time point is entered Row polymerization.
Further, described device also includes:Storage unit 64.
Storage unit 64, during for corresponding to the business datum after the polymerized unit 63 is polymerize and the business datum Between stab and be stored in the first during cycle Aggregation Table.
Further, the acquiring unit 61, it is additionally operable to need to carry out data aggregate according to the second during cycle simultaneously When, the timestamp for the business datum that the data collector is currently received is obtained from the data collector, described second is pre- Putting the cycle is more than first during cycle;
The judging unit 62, it is additionally operable to judge whether the timestamp reaches polymerization corresponding to second during cycle Triggered time point, second during cycle correspond to multiple polymerization triggered time points, the multiple polymerization triggered time point difference The corresponding time is incremented by one by one;
The polymerized unit 63, if being additionally operable to reach polymerization triggered time point corresponding to second during cycle, from When acquisition timestamp is located at upper polymerization triggering corresponding to second during cycle in the first during cycle Aggregation Table Between point to work as pre-polymerization triggered time point in business datum polymerize;
The storage unit 64, it is additionally operable to the business datum after polymerization being stored in the second during cycle Aggregation Table.
Further, described device also includes:Dispensing unit 65.
The dispensing unit 65, for polymerizeing the triggered time according to corresponding to current second during cycle for after the polymerization Business datum setup time stamp;
The storage unit 64, specifically for timestamp corresponding to the business datum after polymerization and the business datum is protected In the presence of in the second during cycle Aggregation Table.
Further, the acquiring unit 61, specifically for when according to the first during cycle carry out data aggregate when, from When obtaining the business datum in preset queue corresponding to pre-polymerization task, the data collection server is each classification A preset queue is respectively configured in business datum.
Further, described device also includes:Receiving unit 66.
The receiving unit 66, for receiving the data query instruction of data query server transmission, the data query Temporal information is carried in instruction;
The acquiring unit 61, it is additionally operable to obtain timestamp and time letter from the first during cycle Aggregation Table Business datum corresponding to breath is simultaneously sent to the inquiry server, to cause the inquiry server to carry out data display.
Further, the storage unit 64, it is additionally operable to first during cycle and current first during cycle pair The polymerization triggered time point answered is stored in preset storage location.
Further, the judging unit 62, it is additionally operable to judge to be currently received first pre- after the business datum Put in the period, if receive business datum of the timestamp for the polymerization triggered time point;
The polymerized unit 63, it is the polymerization triggered time by the timestamp of preservation if specifically for not receiving The business datum of point is polymerize;Or
If receiving, by the business datum received within first preset time period and the timestamp preserved Business datum for the polymerization triggered time point is polymerize.
Further, described device also includes:Delete unit 67.
The storage unit 64, if be additionally operable to the timestamp with it is described polymerization triggered time point it is identical, will currently connect The business datum received is saved in the preset queue corresponding with working as pre-polymerization task;
The deletion unit 67, if being less than the polymerization triggered time point for the timestamp, it will be currently received The business datum delete.
It should be noted that the device embodiment is corresponding with preceding method embodiment, and for ease of reading, present apparatus embodiment No longer the detail content in preceding method embodiment is repeated one by one, it should be understood that the device in the present embodiment can The corresponding full content realized in preceding method embodiment.
Another data aggregate device provided in an embodiment of the present invention, the magnanimity business number stored in needing to cloud environment During according to being polymerize according to some cycles, the business number that the data collector is currently received is obtained from data collector first According to then judging whether the timestamp is more than described in when being currently needed for carrying out data aggregate and polymerize corresponding to the first during cycle Triggered time point, if being more than, business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.With mesh It is preceding when the magnanimity business datum stored in cloud environment polymerize, be required for for storage service data file progress Read-write operation, and it also requires by the business datum of reading according to the time carry out a minor sort compare, the embodiment of the present invention due to Only need whether to be more than by the timestamp for detecting current service data first preset described in when being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to cycle, you can realize and polymerize to the business datum of different time sections, improve data aggregate Efficiency, and because detection operation is carried out in real time, the delay of data aggregate operation can be reduced, while number can be improved According to the real-time of polymerization.
Yet further, the specific implementation as method shown in Fig. 3, the embodiment of the present invention provide a kind of aggregated data inquiry Device, as shown in fig. 7, described device can include:Transmitting element 71, receiving unit 72.
Transmitting element 71, instruction is reported for sending data to analysis aggregate server, the data report to be taken in instruction Business datum with progress data aggregate in need, to cause the analysis aggregate server to receive institute according to data collector State business datum time and it is preset polymerization triggered time point, according to the first during cycle to need carry out data aggregate business Data carry out data aggregate;
The transmitting element 71, it is additionally operable to send the instruction of data aggregate result queries, institute to the analysis aggregate server State the temporal information that business datum to be checked is carried in data query instruction;
Receiving unit 72, for receiving data aggregate result corresponding with the temporal information and being shown.
The transmitting element 71, specifically for by the data-query interfaces that portal server Portal is configured to described point Analyse aggregate server and send the instruction of data aggregate result queries.
It should be noted that the device embodiment is corresponding with preceding method embodiment, and for ease of reading, present apparatus embodiment No longer the detail content in preceding method embodiment is repeated one by one, it should be understood that the device in the present embodiment can The corresponding full content realized in preceding method embodiment.
A kind of aggregated data inquiry unit provided in an embodiment of the present invention, is provided the user by portal server Portal The interface of aggregated data is reported and inquires about, i.e. user can conveniently carry out reporting and polymerization result for data to be polymerized Inquiry, and then can improve data to be polymerized reports efficiency and the search efficiency to polymerization result.
Yet further, the specific implementation as method shown in Fig. 4, the embodiment of the present invention provide a kind of Data Collection dress Put, as shown in figure 8, described device can include:Receiving unit 81, dispensing unit 82.
Receiving unit 81, for receiving business datum to be polymerized;
Dispensing unit 82, for being stabbed for the business datum setup time, the timestamp is used to identify the data receipts Storage receives the time of the business datum, so that must analyze aggregate server obtains the business datum and according to the industry The timestamp for data of being engaged in carries out data aggregate.
A kind of transacter provided in an embodiment of the present invention, by the business datum setup time to receive Stamp, it is ensured that the polymerization of triggering business datum in time, so as to realize the real time aggregation to business datum.
Yet further, the specific implementation as method shown in Fig. 1 or Fig. 2, the embodiment of the present invention provide a kind of data Paradigmatic system, as shown in figure 9, the system can include:Data collector 91, analysis aggregate server 92.
The data collector 91, during for receiving business datum, stabbed for the business datum setup time;
It is described analysis aggregate server 92, for when according to the first during cycle carry out data aggregate when, from Data Collection Device obtains the business datum that the data collector 91 is currently received, and the business datum carries timestamp;
The analysis aggregate server 92, it is additionally operable to judge whether the timestamp is more than and is currently needed for carrying out data aggregate Polymerization triggered time point, first during cycle correspond to multiple polymerization triggered times corresponding to the during cycles of Shi Suoshu first Point, time corresponding to the multiple polymerization triggered time point difference are incremented by one by one;
If so, it is then the business datum for polymerizeing triggered time point by the timestamp preserved in analysis aggregate server 92 It is polymerize.
Further, the system also includes:Inquire about server 93.
The inquiry server 93, for sending data query instruction, the data to the analysis aggregate server 92 Temporal information is carried in query statement;
It is described analysis aggregate server 92, be additionally operable to from the first during cycle Aggregation Table obtain timestamp with it is described Business datum corresponding to temporal information is simultaneously sent to the inquiry server;
The inquiry server 93, is additionally operable to export the business datum.
It should be noted that the system embodiment is corresponding with preceding method embodiment, and for ease of reading, the system embodiment No longer the detail content in preceding method embodiment is repeated one by one, it should be understood that the system in the present embodiment can The corresponding full content realized in preceding method embodiment.
A kind of data aggregate system provided in an embodiment of the present invention, the magnanimity business datum stored in needing to cloud environment When being polymerize according to some cycles, aggregate server is analyzed first and is currently connect from the data collector acquisition data collector The business datum received, when then whether analysis aggregate server judges the timestamp more than being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to first during cycle, if being more than, when the timestamp of preservation is triggered for the polymerization Between the business datum put polymerize.With when the magnanimity business datum stored in cloud environment polymerize, being required at present To being written and read operation for the file of storage service data, and it also requires the business datum of reading is carried out into one according to the time Minor sort is compared, and the embodiment of the present invention is currently needed for due to only needing by the way that whether the timestamp for detecting current service data is more than It polymerize triggered time point corresponding to first during cycle described in when carrying out data aggregate, you can realize the business to different time sections Data are polymerize, and improve the efficiency of data aggregate, and because detection operation is carried out in real time, can reduce data aggregate The delay of operation, while the real-time of data aggregate can be improved.
Data aggregate device includes processor and memory in the chart, and above-mentioned acquiring unit, judging unit, polymerization are single Member etc. stores in memory as program unit, by the said procedure unit of computing device storage in memory Lai real Now corresponding function.
Kernel is included in processor, is gone in memory to transfer corresponding program unit by kernel.Kernel can set one Or more, solve the asking in the presence of display interim card when the figure being selected in chart is shown by adjusting kernel parameter Topic.
Memory may include computer-readable medium in volatile memory, random access memory (RAM) and/ Or the form such as Nonvolatile memory, such as read-only storage (ROM) or flash memory (flash RAM), memory includes at least one deposit Store up chip.
Present invention also provides a kind of computer program product, when being performed on data processing equipment, is adapted for carrying out just The program code of beginningization there are as below methods step:When carrying out data aggregate according to the first during cycle, obtained from data collector The business datum for taking the data collector to be currently received, the business datum carry timestamp, and the timestamp is used for Identify the time that the data collector receives the business datum;Then judge whether the timestamp is more than to be currently needed for It polymerize triggered time point corresponding to first during cycle described in when carrying out data aggregate, first during cycle corresponds to multiple poly- Triggered time point is closed, the time corresponding to the multiple polymerization triggered time point difference is incremented by one by one;If so, then the data are received The timestamp preserved in storage is polymerize for the business datum of the polymerization triggered time point.
It should be understood by those skilled in the art that, embodiments herein can be provided as method, system or computer program Product.Therefore, the application can use the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware Apply the form of example.Moreover, the application can use the computer for wherein including computer usable program code in one or more The computer program production that usable storage medium is implemented on (including but is not limited to magnetic disk storage, CD-ROM, optical memory etc.) The form of product.
The application is with reference to the stream according to the data aggregation method of the embodiment of the present application, device and computer program product Journey figure and/or block diagram describe.It should be understood that can be by each in computer program instructions implementation process figure and/or block diagram Flow and/or the flow in square frame and flow chart and/or block diagram and/or the combination of square frame.These computer journeys can be provided Sequence instruction to all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices processor with Produce a machine so that produce and be used for by the instruction of computer or the computing device of other programmable data processing devices Realize the dress for the function of being specified in one flow of flow chart or multiple flows and/or one square frame of block diagram or multiple square frames Put.
These computer program instructions, which may be alternatively stored in, can guide computer or other programmable data processing devices with spy Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which produces, to be included referring to Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that counted Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, so as in computer or The instruction performed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one The step of function of being specified in individual square frame or multiple square frames.
In a typical configuration, computing device includes one or more processors (CPU), input/output interface, net Network interface and internal memory.
Memory may include computer-readable medium in volatile memory, random access memory (RAM) and/ Or the form such as Nonvolatile memory, such as read-only storage (ROM) or flash memory (flash RAM).Memory is computer-readable Jie The example of matter.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by any method Or technology come realize information store.Information can be computer-readable instruction, data structure, the module of program or other data. The example of the storage medium of computer includes, but are not limited to phase transition internal memory (PRAM), static RAM (SRAM), moved State random access memory (DRAM), other kinds of random access memory (RAM), read-only storage (ROM), electric erasable Programmable read only memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc read-only storage (CD-ROM), Digital versatile disc (DVD) or other optical storages, magnetic cassette tape, the storage of tape magnetic rigid disk or other magnetic storage apparatus Or any other non-transmission medium, the information that can be accessed by a computing device available for storage.Define, calculate according to herein Machine computer-readable recording medium does not include temporary computer readable media (transitory media), such as data-signal and carrier wave of modulation.
Embodiments herein is these are only, is not limited to the application.To those skilled in the art, The application can have various modifications and variations.All any modifications made within spirit herein and principle, equivalent substitution, Improve etc., it should be included within the scope of claims hereof.

Claims (17)

  1. A kind of 1. data aggregate system, it is characterised in that including:Data collector and analysis aggregate server;
    The data collector, during for receiving business datum, stabbed for the business datum setup time;
    The analysis aggregate server, for when carrying out data aggregate according to the first during cycle, being obtained from data collector The business datum that the data collector is currently received, the business datum carry timestamp;
    The analysis aggregate server, it is additionally operable to judge whether the timestamp is more than described in when being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to first during cycle, first during cycle corresponds to multiple polymerization triggered time points, described Time corresponding to multiple polymerization triggered time points difference is incremented by one by one;
    If so, then business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.
  2. 2. data aggregate system according to claim 1, it is characterised in that the system also includes:Inquire about server;
    The inquiry server, for sending data query instruction, the data query instruction to the analysis aggregate server In carry temporal information;
    The analysis aggregate server, it is additionally operable to obtain timestamp and time letter from the first during cycle Aggregation Table Business datum corresponding to breath is simultaneously sent to the inquiry server;
    The inquiry server, it is additionally operable to export the business datum.
  3. A kind of 3. data aggregation method, it is characterised in that including:
    When carrying out data aggregate according to the first during cycle, obtain the data collector from data collector and be currently received Business datum, the business datum carries timestamp, and the timestamp, which is used to identifying the data collector, receives institute State the time of business datum;
    Judge whether the timestamp is more than polymerization corresponding to the first during cycle described in when being currently needed for carrying out data aggregate and touches Time point is sent out, first during cycle corresponds to multiple polymerization triggered time points, and the multiple polymerization triggered time point is right respectively The time answered is incremented by one by one;
    If so, polymerization corresponding to the first during cycle described in during then by the timestamp of preservation to be currently needed for carrying out data aggregate is touched The business datum at hair time point is polymerize.
  4. 4. data aggregation method according to claim 3, it is characterised in that described to preserve in the data collector After timestamp is polymerize for the business datum of the polymerization triggered time point, methods described also includes:
    Timestamp corresponding to business datum after polymerization and the business datum is stored in the first during cycle Aggregation Table.
  5. 5. data aggregation method according to claim 4, it is characterised in that the business datum by after polymerization and described Timestamp corresponding to business datum is stored in after the step in the first during cycle Aggregation Table, and methods described also includes:
    When needing to carry out data aggregate according to the second during cycle simultaneously, the Data Collection is obtained from the data collector The timestamp for the business datum that device is currently received, second during cycle are more than first during cycle, and described Second during cycle is the integral multiple of first during cycle;
    Judge whether the timestamp reaches polymerization triggered time point, the second preset week corresponding to second during cycle Phase corresponds to multiple polymerization triggered time points, and the time corresponding to the multiple polymerization triggered time point difference is incremented by one by one;
    It is located at if so, then obtaining timestamp from the first during cycle Aggregation Table upper one corresponding to second during cycle Individual polymerization triggered time point is polymerize to the business datum worked as in pre-polymerization triggered time point, and by the business datum after polymerization It is stored in the second during cycle Aggregation Table.
  6. 6. data aggregation method according to claim 5, it is characterised in that the business datum by after polymerization is stored in Before step in second during cycle Aggregation Table, methods described also includes:
    It is the business datum setup time stamp after the polymerization to polymerize the triggered time according to corresponding to current second during cycle;
    The business datum by after polymerization, which is stored in the second during cycle Aggregation Table, to be included:
    Timestamp corresponding to business datum after polymerization and the business datum is stored in the second during cycle Aggregation Table.
  7. 7. data aggregation method according to claim 3, it is characterised in that described to enter line number according to the first during cycle During according to polymerization, the business datum being currently received from the data collector acquisition data collector includes:
    When carrying out data aggregate according to the first during cycle, from described in obtained in the corresponding preset queue of pre-polymerization task A preset queue is respectively configured for the business datum of each classification in business datum, the data collection server.
  8. 8. data aggregation method according to claim 4, it is characterised in that methods described also includes:
    The data query instruction that data query server is sent is received, temporal information is carried in the data query instruction;
    Timestamp business datum corresponding with the temporal information is obtained from the first during cycle Aggregation Table and is sent to The inquiry server, to cause the inquiry server to carry out data display.
  9. 9. data aggregation method according to claim 3, it is characterised in that methods described also includes:
    Preset storage position is stored in by triggered time point is polymerize corresponding to first during cycle and current first during cycle Put.
  10. 10. data aggregation method according to claim 3, it is characterised in that the timestamp by preservation is described poly- Before the business datum of conjunction triggered time point is polymerize, methods described also includes:
    Judge to be currently received in the preset buffer time section after the business datum, if receive timestamp to be described poly- Close the business datum of triggered time point;
    The timestamp by preservation carries out polymerization for the business datum of the polymerization triggered time point to be included:
    If not receiving, business datum of the timestamp of preservation for the polymerization triggered time point is polymerize;
    It is institute by the business datum received in the preset buffer time section and the timestamp preserved if receiving The business datum for stating polymerization triggered time point is polymerize.
  11. 11. data aggregation method according to claim 7, it is characterised in that described to judge whether the timestamp is more than It polymerize corresponding to first during cycle described in when being currently needed for carrying out data aggregate after triggered time point, methods described is also wrapped Include:
    If the timestamp is identical with the polymerization triggered time point, the business datum being currently received is saved in institute State in preset queue corresponding with working as pre-polymerization task;
    If the timestamp is less than the polymerization triggered time point, the business datum being currently received is deleted.
  12. A kind of 12. method of data capture, it is characterised in that including:
    Receive business datum to be polymerized;
    Stabbed for the business datum setup time, the timestamp receives the business number for identifying the data collector According to time,
    Gather so as to which aggregate server must be analyzed and obtain the business datum and carry out data according to the timestamp of the business datum Close.
  13. A kind of 13. aggregated data querying method, it is characterised in that including:
    Data are sent to analysis aggregate server and report instruction, and the data report in instruction and carry carry out data aggregate in need Business datum so that must analyze aggregate server receives time of the business datum and preset poly- according to data collector Triggered time point is closed, according to during cycle to needing the business datum for carrying out data aggregate to carry out data aggregate;
    The instruction of data aggregate result queries is sent to the analysis aggregate server, is carried in the data query instruction to be checked Ask the temporal information of business datum;
    Receive data aggregate result corresponding with the temporal information and shown.
  14. 14. aggregated data querying method according to claim 13, it is characterised in that described to the analysis aggregated service Device, which sends the instruction of data aggregate result queries, to be included:
    By the data-query interfaces that portal server Portal is configured data aggregate knot is sent to the analysis aggregate server Fruit query statement.
  15. A kind of 15. data aggregate device, it is characterised in that including:
    Acquiring unit, for when carrying out data aggregate according to the first during cycle, obtaining the data from data collector and receiving The business datum that storage is currently received, the business datum carry timestamp, and the timestamp is used to identify the data Collector receives the time of the business datum;
    Judging unit, whether the timestamp for judging the acquiring unit acquisition, which is more than, is currently needed for carrying out data aggregate Polymerization triggered time point, first during cycle correspond to multiple polymerization triggered times corresponding to the during cycles of Shi Suoshu first Point, time corresponding to the multiple polymerization triggered time point difference are incremented by one by one;
    Polymerized unit, if determining to be more than the first during cycle described in when being currently needed for carrying out data aggregate for the judging unit Corresponding polymerization triggered time point, then business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.
  16. A kind of 16. transacter, it is characterised in that including:
    Receiving unit, for receiving business datum to be polymerized;
    Dispensing unit, for being stabbed for the business datum setup time, the timestamp connects for identifying the data collector The time of the business datum is received,
    Gather so as to which aggregate server must be analyzed and obtain the business datum and carry out data according to the timestamp of the business datum Close.
  17. A kind of 17. aggregated data inquiry unit, it is characterised in that including:
    Transmitting element, instruction is reported for sending data to analysis aggregate server, the data are reported to carry in instruction and needed The business datum of data aggregate is carried out, so that must analyze aggregate server receives the business datum according to data collector Time and it is preset polymerization triggered time point, according to during cycle to need carry out data aggregate business datum carry out data gather Close;
    The transmitting element, it is additionally operable to send the instruction of data aggregate result queries, the data to the analysis aggregate server The temporal information of business datum to be checked is carried in query statement;
    Receiving unit, for receiving data aggregate result corresponding with the temporal information and being shown.
CN201610795892.8A 2016-08-31 2016-08-31 Data aggregation method, apparatus and system Pending CN107798009A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610795892.8A CN107798009A (en) 2016-08-31 2016-08-31 Data aggregation method, apparatus and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610795892.8A CN107798009A (en) 2016-08-31 2016-08-31 Data aggregation method, apparatus and system

Publications (1)

Publication Number Publication Date
CN107798009A true CN107798009A (en) 2018-03-13

Family

ID=61529581

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610795892.8A Pending CN107798009A (en) 2016-08-31 2016-08-31 Data aggregation method, apparatus and system

Country Status (1)

Country Link
CN (1) CN107798009A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109408544A (en) * 2018-09-27 2019-03-01 平安科技(深圳)有限公司 Data aggregation method, device and storage medium, server based on engine
CN110502543A (en) * 2019-08-07 2019-11-26 京信通信系统(中国)有限公司 Device performance data storage method, device, equipment and storage medium
WO2021179122A1 (en) * 2020-03-09 2021-09-16 深圳市欢太科技有限公司 Data processing method and related products

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101063973A (en) * 2006-04-27 2007-10-31 国际商业机器公司 Method and system for the aggregation of data in multiple processing environments
CN102447589A (en) * 2010-09-30 2012-05-09 杭州华三通信技术有限公司 Method and device for aggregating records
US20130132681A1 (en) * 2011-11-22 2013-05-23 Microsoft Corporation Temporal standby list
CN105005274A (en) * 2014-01-31 2015-10-28 费希尔-罗斯蒙特系统公司 Managing big data in process control systems

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101063973A (en) * 2006-04-27 2007-10-31 国际商业机器公司 Method and system for the aggregation of data in multiple processing environments
CN102447589A (en) * 2010-09-30 2012-05-09 杭州华三通信技术有限公司 Method and device for aggregating records
US20130132681A1 (en) * 2011-11-22 2013-05-23 Microsoft Corporation Temporal standby list
CN105005274A (en) * 2014-01-31 2015-10-28 费希尔-罗斯蒙特系统公司 Managing big data in process control systems

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
翁溪 等: "openflow网络测量分析系统的设计实现", 《通信学报》 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109408544A (en) * 2018-09-27 2019-03-01 平安科技(深圳)有限公司 Data aggregation method, device and storage medium, server based on engine
CN109408544B (en) * 2018-09-27 2023-11-03 平安科技(深圳)有限公司 Data aggregation method and device based on engine, storage medium and server
CN110502543A (en) * 2019-08-07 2019-11-26 京信通信系统(中国)有限公司 Device performance data storage method, device, equipment and storage medium
WO2021179122A1 (en) * 2020-03-09 2021-09-16 深圳市欢太科技有限公司 Data processing method and related products

Similar Documents

Publication Publication Date Title
US11947556B1 (en) Computerized monitoring of a metric through execution of a search query, determining a root cause of the behavior, and providing a notification thereof
US11620300B2 (en) Real-time measurement and system monitoring based on generated dependency graph models of system components
US11934417B2 (en) Dynamically monitoring an information technology networked entity
US20200257680A1 (en) Analyzing tags associated with high-latency and error spans for instrumented software
US20210176136A1 (en) Continuous data sensing of functional states of networked computing devices to determine efficiency metrics for servicing electronic messages asynchronously
CN103403699B (en) Load in estimation flow database application cuts down the system and method for data
US9237164B2 (en) Systems and methods for capturing, replaying, or analyzing time-series data
CN107145489A (en) A kind of information statistical method and device of the client application based on cloud platform
CN104281684B (en) Massive logs are stored and querying method and system
CN109684052A (en) Transaction analysis method, apparatus, equipment and storage medium
WO2020043267A1 (en) Device and method for anomaly detection on an input stream of events
US9929930B2 (en) Reducing an amount of captured network traffic data to analyze
US11755531B1 (en) System and method for storage of data utilizing a persistent queue
US11663219B1 (en) Determining a set of parameter values for a processing pipeline
CN110647447B (en) Abnormal instance detection method, device, equipment and medium for distributed system
CN110262951A (en) A kind of business second grade monitoring method and system, storage medium and client
CN107798009A (en) Data aggregation method, apparatus and system
CN111813573A (en) Communication method of management platform and robot software and related equipment thereof
TWI748989B (en) Data aggregation method and device
CN110266555A (en) Method for analyzing web site service request
CN105094810B (en) Data processing method and device based on CGI(Common gateway interface) plug-in unit
US10489485B2 (en) Analytic system for streaming quantile computation
CN112491650A (en) Method for dynamically analyzing call loop condition between services and related equipment
CN114546817A (en) Information processing method, device, storage medium and computer program product
CN109086116A (en) Communication tool speech method of real-time and device based on miscellaneous function service

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 1252392

Country of ref document: HK

RJ01 Rejection of invention patent application after publication

Application publication date: 20180313

RJ01 Rejection of invention patent application after publication