CN102902813A - Log collection system - Google Patents
Log collection system Download PDFInfo
- Publication number
- CN102902813A CN102902813A CN201210404697XA CN201210404697A CN102902813A CN 102902813 A CN102902813 A CN 102902813A CN 201210404697X A CN201210404697X A CN 201210404697XA CN 201210404697 A CN201210404697 A CN 201210404697A CN 102902813 A CN102902813 A CN 102902813A
- Authority
- CN
- China
- Prior art keywords
- data
- field
- value
- merger
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a log collection system and belongs to the technical field of internets. The log collection system comprises a server and a plurality of clients which are deployed on different production servers, wherein the clients are used for acquiring data which is generated by the production servers and corresponds to different types of services, merging the data with specific identifiers into a data and forwarding the data to the server; and the server is used for receiving the data from each client and storing or forwarding the data. According to the technical scheme, data comprising any digital section can be transmitted, so that the data transmission is not limited, and the data is merged at the clients; and therefore, the problem of network congestion and delay because of transmission of lots of identical or similar data is solved.
Description
Technical field
The present invention relates to Internet technical field, be specifically related to a kind of result collection system.
Background technology
The Internet era back-end data extremely important and huge, such as daily record data and statistics.These back-end datas may be the firsthand information of backstage slip-stick artist's routine analyzer operation conditions, also may be the first references that the service operation decision-making relies on.Yet the website of large flow generally has up to ten million to produce server, and is distributed in each different machine room.On the production server that journal file or statistics will leave these network isomeries in and distribute scattered, give daily record collection, transmit, gather and analyze and bring very large difficulty.There are at present some softwares of increasing income to be used for collecting these daily records, but also exist a lot of indeterminable situations.
Some open source softwares of comparatively commonly using at present are Scribe for example, can reach the purpose of simple collection daily record data.
Scribe is the result collection system of increasing income of a current large-scale social networking service website, gets a lot of applications in this large-scale inside, social networking service website.It can store in the centralized storage system (can be NFS, distributed file system HDFS etc.) from collector journal on the various Log Sources, so that concentrate statistical study to process.It provides a scheme extendible, that height is fault-tolerant for " distributed collection, the unified processing " of daily record.When the network of central storage system or machine broke down, scribe can dump to daily record this locality or another position, and after central storage system was recovered, scribe can be transferred to the centralized storage system again with the daily record of unloading.It is combined with Hadoop usually, and scribe is used for to HDFS push daily record, and Hadoop regularly processes by the MapReduce operation.
Fig. 1 is the synoptic diagram of existing Scribe collector journal.As shown in Figure 1, Scribe is put in the shared queue from collecting data as each application of planting data source, and then push is on the centralized storage system of rear end.When central storage system broke down, scribe can temporarily write daily record in the local file, and after centralized storage system restorability, scribe resumes local daily record in the centralized storage system.
Each data source must be by THRIFT(owing to have adopted THRIFT, client can adopt various language compilation to the scribe the transmission of data, and every data record comprises a category and a message).The THRIFT Thread Count (being defaulted as 3) that can be used in the scribe configuration listening port.In the rear end, scribe can be with the deposit data of different category in different directories, so that process respectively.The log store mode of rear end can be various store, comprising: the file(file), the double-deck storage of buffer(, main a storage, a secondary storage), another scribe server of network() etc.
But there is following shortcoming in scribe:
(1) scribe shortcoming is that the front group organization data is dumb, can only use two fields, be catagory and message, in the application program of producing server, if want to send data with scribe, then every data can only have catagory and two fields of message, if want to transmit a plurality of fields, then must own organising data, a plurality of data that will transmit merge to the message the inside.When the post analysis data, also want oneself to resolve message, obtain original a plurality of fields.This has caused many restrictions and inconvenience to data transfer.
(2) another shortcoming is, scribe can receive each bar data, and they verily are recorded in local cache, with certain frequency Batch sending data, even if the category of two data and message are living.This is very large in volume of transmitted data, when transmission frequency is very high, cause easily serious network blockage and delay.
Summary of the invention
In view of the above problems, the present invention has been proposed in order to a kind of result collection system that overcomes the problems referred to above or address the above problem at least in part is provided.
According to the present invention, a kind of result collection system is provided, this system comprises: server end be deployed in different production servers on a plurality of clients,
Described client is suitable for obtaining the data of producing the corresponding different classes of business that server produces, and is to send to described server end after the data with the aggregation of data of specific identifier;
Described server end is suitable for from each client data, and stores or transmit.
Alternatively, this client comprises: data capture unit, merger processing unit and a plurality of storage unit, and described a plurality of storage unit are the different classes of business of correspondence respectively, and each storage unit has the timing cycle of a correspondence;
Described data capture unit is suitable for obtaining from producing server the data of corresponding different classes of business, and the data of obtaining are preserved to the storage unit of correspondence according to the different classes of distribution of services of correspondence; Wherein, every data comprise more than one field, and different fields has different types, and at least one field identification of every data has key;
Each storage unit is suitable for preserving the data from data capture unit;
Described merger processing unit is suitable for when timing cycle corresponding to each storage unit finishes, and it is to send to server end after the data that the sign in the data that this storage unit is preserved has the identical aggregation of data of value of the field of key.
Alternatively, described merger processing unit, be further adapted for when the professional corresponding timing cycle of every kind finishes, when having the identical aggregation of data of value of the field of key to be data the sign in the data of such various-service of preserving, to not identifying the field of key, carry out different merger according to different types and process.
Alternatively, the merger processing unit is further adapted for when according to different types the field that does not identify key being carried out different merger processing one or more combination below adopting:
For the field of sum-type, sign is had the numerical value addition on this fields of each identical data of the value of field of key, itself and as the value of this field after the merger;
For the field that is averaging type, sign is had the numerical value on this fields of each identical data of the value of field of key be averaging, its average as merger after the value of this field;
For the field of maximal value type, from having value on this fields of each identical data of the value of field of key, sign finds out maximal value, as the value of this field after the merger;
For the field of normal character types, from sign value on this field of getting article one data each identical data of the value of field of key is arranged, as the value of this field after the merger;
For the field of cumulative character types, sign is had character on this fields of each identical data of the value of field of key by the specified order serial connection after, as the value of this field after the merger.
Alternatively, described server end is suitable for the data retransmission that will receive to other server, or is forwarded to database facility, or retain costs ground file.
According to of the present invention this at different production server difference deploying clients, each client is issued server end with the data of collecting, wherein client stores classifiedly the data the obtained different classes of business according to correspondence, every data comprise more than one dissimilar field, when the professional corresponding timing cycle of every kind finishes, with in the data of such various-service of preserving, it is the technical scheme that sends to server end after the data that sign has the identical aggregation of data of value of the field of key, can transmit the arbitrarily data of a field, and just carried out the aggregation of data processing in client, having solved thus existing scribe only allows every data that catagory and two fields of message can only be arranged, thereby so that the transmission of data has the problem of many restrictions, and solved existing scribe at front end record data verily just, not carrying out merger processes, cause volume of transmitted data large, transmission frequency is high, causes easily the problem of network blockage and delay.
Above-mentioned explanation only is the general introduction of technical solution of the present invention, for can clearer understanding technological means of the present invention, and can be implemented according to the content of instructions, and for above and other objects of the present invention, feature and advantage can be become apparent, below especially exemplified by the specific embodiment of the present invention.
Description of drawings
By reading hereinafter detailed description of the preferred embodiment, various other advantage and benefits will become cheer and bright for those of ordinary skills.Accompanying drawing only is used for the purpose of preferred implementation is shown, and does not think limitation of the present invention.And in whole accompanying drawing, represent identical parts with identical reference symbol.In the accompanying drawings:
Fig. 1 is the synoptic diagram of existing Scribe collector journal;
Fig. 2 shows according to an embodiment of the invention a kind of block diagram of result collection system;
Fig. 3 shows according to an embodiment of the invention a kind of structural drawing of client of result collection system;
Fig. 4 shows a kind of according to an embodiment of the invention process flow diagram of collecting the method for data.
Embodiment
Exemplary embodiment of the present disclosure is described below with reference to accompanying drawings in more detail.Although shown exemplary embodiment of the present disclosure in the accompanying drawing, yet should be appreciated that and to realize the disclosure and the embodiment that should do not set forth limits here with various forms.On the contrary, it is in order to understand the disclosure more thoroughly that these embodiment are provided, and can with the scope of the present disclosure complete convey to those skilled in the art.
Fig. 2 shows according to an embodiment of the invention a kind of block diagram of result collection system.As shown in Figure 2, this system comprises: server end 202 and a plurality of client 201.A plurality of clients 201 are deployed in respectively different needs and collect on the production server of various data.The Data Concurrent that each client 201 collection self place production server produces is given server end 202, and server end 202 receives the data that each clients 201 are beamed back, and carries out the local server of storing or being transmitted to other.Specifically:
Each client 201 is suitable for obtaining the data of producing the corresponding different classes of business that server produces, and the data the obtained different classes of business according to correspondence is stored classifiedly.Wherein, every data comprise more than one field, and different fields has different types, and at least one field identification of every data has key; Every kind business has the timing cycle of a correspondence;
Each client 201, when the professional corresponding timing cycle of every kind finished, with in the data of such various-service of preserving, it was to send to described server end 202 after the data that sign has the identical aggregation of data of value of the field of key;
Here, the data layout of the data of one species various-service is identical, and namely the type of the field number that comprises of data and each field is all identical.The form that can define according to the actual requirements data of business of all categories, the field number that namely can comprise according to data of practical business requirement definition and the type of each field.For example, can be defined as follows the field of type: sum-type (SUM_INT), be averaging type (AVG_INT), maximal value type (MAX_INT), normal character types (CONST_STRING) and cumulative character types (CONST_STRING) etc.
The purpose that these fields are set is in order to do various optimization processes for the data of various different kinds of business, so that data occupy little space, speed is faster when analyzing and processing data, and committed memory still less, the implication of easier each field of identification, and be that the merger of back is ready.
Each client 201 is according to configuring maintenance a plurality of " boxes ", and each box is deposited the data of same format.That is to say professional corresponding one " box " of a kind, the deposit data of such various-service is in this corresponding box.Client 201 determines to leave in which box the data of collecting according to its data layout.When the one-period of certain box finished, client 201 was done a merger with the data in this box and is processed, and then sends to server end 202.
In the present invention, as the foundation of merger, identify key (" Key " attribute) in some fields of data, when doing aggregation of data, sign in the meeting comparing data has the field of key, and only having sign that the identical data of value of the field of key are arranged can merger be data.
When client 201 finishes at the professional corresponding timing cycle of every kind, with in the data of such various-service of preserving, when sign has the identical aggregation of data of value of the field of key to be data, to not identifying the field of key, carry out different merger according to different types and process.Be that field type is different, its merger mode is also different.
(1) for the field of sum-type: when merger, sign is had the numerical value addition on the sum-type field of each identical data of the value of field of key, itself and as the value of the sum-type field of data after the merger;
(2) for the field that is averaging type: when merger, sign is had the numerical value that is averaging on the type field of each identical data of the value of field of key be averaging, its average as merger after the value that is averaging type field of data;
(3) for the field of maximal value type: when merger, from sign has value on the maximal value type field of each identical data of the value of field of key, find out maximal value, as the value of the maximal value type field of data after the merger;
(4) for the field of normal character types: when merger, from sign value on the normal character types field of getting article one data each identical data of the value of field of key is arranged, as the value of the normal character types field of data after the merger;
(5) for the field of cumulative character types: when merger, sign is had character on the cumulative character types field of each identical data of the value of field of key by the specified order serial connection after, as the value of the cumulative character types field of data after the merger.
More than given an example 5 kinds of field types with and corresponding merger mode separately.But the field type among the present invention is not limited to above 5 kinds, can according to the more eurypalynous field of practical business requirement definition with and the merger mode.For example can also define floating number and be averaging type (AVG_FLOAT), minimum value type (MIN_INT) and floating number sum-type (SUM_FLOAT) etc. describe in detail here no longer one by one.
The below provides an object lesson that data is carried out the merger processing.
Define one and log in professional data layout, this data layout is used for logging in business---and the data of " user accesses the number of times of a page " are carried out record, safeguard one " box " in client accordingly, be " Login ", cycle is 300 seconds, and then data layout is specially: Login (300): user_id KEY_STR, script KEY_STR, number SUM_INT, datetime TIME_FLOOR;
This data layout comprises 4 fields, the first two field user_id and the relevant key KEY_STR of the upper sign of script, and the type of latter two field number and datetime is respectively sum-type (SUM_INT) and floor time type (TIME_FLOOR).
After definition is finished, just can send the data that meet each field type at the production server, the client that is deployed on this production server is collected the data that send.Such as shown in table 1 to the data of client collection between the 2012-09-2100:04:59 at 2012-09-2100:00:00:
| ZK | Index.php | 1 | 2012-09-2100:00:00 |
| ZK | Index.php | 1 | 2012-09-2100:01:03 |
| ZK | Index.php | 5 | 2012-09-2100:01:23 |
| ZK | Login.php | 2 | 2012-09-2100:02:14 |
| HGL | Login.php | 2 | 2012-09-2100:02:14 |
| ZK | Index.php | 3 | 2012-09-2100:03:19 |
| HGL | Index.php | 7 | 2012-09-2100:04:10 |
| HGL | Index.php | 10 | 2012-09-2100:04:34 |
Table 1
Data shown in the table 1 are to belong to log in professional data, and its form is identical, are therefore put into " Login " box by client.After 300 seconds cycle had arrived, client can be done a merger to the data in " Login " this box, and the merger result is as shown in table 2:
| ZK | Index.php | 10 | 2012-09-2100:00:00 | Article the 1st, 2,3,6, merger result |
| ZK | Login.php | 2 | 2012-09-2100:00:00 | Article 4, merger result |
| HGL | Index.php | 17 | 2012-09-2100:00:00 | Article 7,8, merger result |
| HGL | Login.php | 2 | 2012-09-2100:00:00 | Article 5, merger result |
Table 2
Last row of table 2 are the explanations to merger.As seen, because the 1st, 2,3 and the sign of 6 data in the table 1 have the content of the first two field of key identical, therefore can be merged into data, the data after the merging: the first two field still is original value; The 3rd field is sum-type, thus its value for the data in the 3rd field of the 1st, 2,3 and 6 data in the table 1 and, be specially 10; The 4th field is floor time type, so its value is the zero-time in this cycle.By that analogy, the 4th data merging in the table 1, the 7th and the 8th data in the table 1 can merge, and the 5th data in the table 1 merge.Amalgamation result is referring to table 2.
Like this, the data clauses and subclauses of input " Login " box are 8 data in one-period (2012-09-2100:00:00 is to 2012-09-2100:04:59), have only sent 4 data when sending to server end 202.
As seen, server end 202 receives the data that each clients are beamed back, and server end 202 receives server or the database facility that can also be transmitted to other after the data, namely plays the part of " agency " role.Go for like this network environment or the machine room of isomery.
As seen by above-mentioned, the system of this collection data of the present invention owing in client data have been carried out flexibly processing and merger, so can realize the collection to daily record, can be used for using again getting statistics ready.
The below introduces the composition structure of client 201.
Fig. 3 shows the according to an embodiment of the invention structural drawing of the client of log collection.As shown in Figure 3, this client comprises: data capture unit 301, merger processing unit 303 and a plurality of storage unit 302, a plurality of storage unit 302 are the different classes of business of correspondence respectively, and each storage unit 302 has the timing cycle of a correspondence.Wherein:
Each storage unit 302 is suitable for preserving the data from data capture unit 301;
Here, the data layout of the data of one species various-service is identical, and namely the type of the field number that comprises of data and each field is all identical.The form that can define according to the actual requirements data of business of all categories, the field number that namely can comprise according to data of practical business requirement definition and the type of each field.
In one embodiment of the invention, merger processing unit 303 is further adapted for when the professional corresponding timing cycle of every kind finishes, when having the identical aggregation of data of value of the field of key to be data the sign in the data of such various-service of preserving, to not identifying the field of key, carry out different merger according to different types and process.
In one embodiment of the invention, merger processing unit 303 is further adapted for when according to different types the field that does not identify key being carried out different merger processing, one or more combination below adopting:
For the field of sum-type, sign is had the numerical value addition on this fields of each identical data of the value of field of key, itself and as the value of this field after the merger;
For the field that is averaging type, sign is had the numerical value on this fields of each identical data of the value of field of key be averaging, its average as merger after the value of this field;
For the field of maximal value type, from having value on this fields of each identical data of the value of field of key, sign finds out maximal value, as the value of this field after the merger;
For the field of normal character types, from sign value on this field of getting article one data each identical data of the value of field of key is arranged, as the value of this field after the merger;
For the field of cumulative character types, sign is had character on this fields of each identical data of the value of field of key by the specified order serial connection after, as the value of this field after the merger.
Fig. 4 shows a kind of according to an embodiment of the invention process flow diagram of collecting the method for data.As shown in Figure 4, the method comprises:
Step S410 is deployed in the data that the client of producing on the server is obtained the corresponding different classes of business that this production server produces; Wherein, every data comprise more than one field, and different fields has different types, and at least one field identification of every data has key;
Here, the data layout of the data of one species various-service is identical, and namely the type of the field number that comprises of data and each field is all identical.
Step S420, client stores classifiedly the data the obtained different classes of business according to correspondence; Wherein, every kind business has the timing cycle of a correspondence;
Step S430, professional for every kind, when client finished at corresponding timing cycle, it was to send to server end after the data that the sign in the data of such various-service of preserving is had the identical aggregation of data of value of the field of key.
Wherein, in step S430, the identical aggregation of data of value that sign is had the field of key is that data comprise: for the field that does not identify key, carry out different merger according to different types and process.This field for not identifying key, carry out different merger according to different field types and process and comprise following one or more combination:
For the field of sum-type, sign is had the numerical value addition on this fields of each identical data of the value of field of key, itself and as the value of this field after the merger;
For the field that is averaging type, sign is had the numerical value on this fields of each identical data of the value of field of key be averaging, its average as merger after the value of this field;
For the field of maximal value type, from having value on this fields of each identical data of the value of field of key, sign finds out maximal value, as the value of this field after the merger;
For the field of normal character types, from sign value on this field of getting article one data each identical data of the value of field of key is arranged, as the value of this field after the merger;
For the field of cumulative character types, sign is had character on this fields of each identical data of the value of field of key by the specified order serial connection after, as the value of this field after the merger.
In sum, of the present invention this at different production server difference deploying clients, each client is issued server end with the data of collecting, wherein client stores classifiedly the data the obtained different classes of business according to correspondence, every data comprise more than one dissimilar field, when the professional corresponding timing cycle of every kind finishes, with in the data of such various-service of preserving, it is the technical scheme that sends to server end after the data that sign has the identical aggregation of data of value of the field of key, can transmit the arbitrarily data of a field, and just carried out the aggregation of data processing in client, having solved thus existing scribe only allows every data that catagory and two fields of message can only be arranged, thereby so that the transmission of data has the problem of many restrictions, and solved existing scribe at front end record data verily just, not carrying out merger processes, cause volume of transmitted data large, transmission frequency is high, causes easily the problem of network blockage and delay.Technical scheme of the present invention, can save bandwidth, dispose simply, safeguard easily and performance efficient, when technical scheme of the present invention has satisfied network data transmission to a greater extent, to the flexible and changeable demand of log transmission.
Need to prove:
Intrinsic not relevant with any certain computer, virtual system or miscellaneous equipment with demonstration at this algorithm that provides.Various general-purpose systems also can be with using based on the teaching at this.According to top description, it is apparent constructing the desired structure of this type systematic.In addition, the present invention is not also for any certain programmed language.Should be understood that and to utilize various programming languages to realize content of the present invention described here, and the top description that language-specific is done is in order to disclose preferred forms of the present invention.
In the instructions that provides herein, a large amount of details have been described.Yet, can understand, embodiments of the invention can be put into practice in the situation of these details not having.In some instances, be not shown specifically known method, structure and technology, so that not fuzzy understanding of this description.
Similarly, be to be understood that, in order to simplify the disclosure and to help to understand one or more in each inventive aspect, in the description to exemplary embodiment of the present invention, each feature of the present invention is grouped together in single embodiment, figure or the description to it sometimes in the above.Yet the method for the disclosure should be construed to the following intention of reflection: namely the present invention for required protection requires the more feature of feature clearly put down in writing than institute in each claim.Or rather, as following claims reflected, inventive aspect was to be less than all features of the disclosed single embodiment in front.Therefore, follow claims of embodiment and incorporate clearly thus this embodiment into, wherein each claim itself is as independent embodiment of the present invention.
Those skilled in the art are appreciated that and can adaptively change and they are arranged in one or more equipment different from this embodiment the module in the equipment among the embodiment.Can be combined into a module or unit or assembly to the module among the embodiment or unit or assembly, and can be divided into a plurality of submodules or subelement or sub-component to them in addition.In such feature and/or process or unit at least some are mutually repelling, and can adopt any combination to disclosed all features in this instructions (comprising claim, summary and the accompanying drawing followed) and so all processes or the unit of disclosed any method or equipment make up.Unless in addition clearly statement, disclosed each feature can be by providing identical, being equal to or the alternative features of similar purpose replaces in this instructions (comprising claim, summary and the accompanying drawing followed).
In addition, those skilled in the art can understand, although embodiment more described herein comprise some feature rather than further feature included among other embodiment, the combination of the feature of different embodiment means and is within the scope of the present invention and forms different embodiment.For example, in the following claims, the one of any of embodiment required for protection can be used with array mode arbitrarily.
All parts embodiment of the present invention can realize with hardware, perhaps realizes with the software module of moving at one or more processor, and perhaps the combination with them realizes.It will be understood by those of skill in the art that and to use in practice microprocessor or digital signal processor (DSP) to realize according to some of the client and server end in the data gathering system of the embodiment of the invention or all some or repertoire of parts.The present invention can also be embodied as be used to part or all equipment or the device program (for example, computer program and computer program) of carrying out method as described herein.Such realization program of the present invention can be stored on the computer-readable medium, perhaps can have the form of one or more signal.Such signal can be downloaded from internet website and obtain, and perhaps provides at carrier signal, perhaps provides with any other form.
It should be noted above-described embodiment the present invention will be described rather than limit the invention, and those skilled in the art can design alternative embodiment in the situation of the scope that does not break away from claims.In the claims, any reference symbol between bracket should be configured to limitations on claims.Word " comprises " not to be got rid of existence and is not listed in element or step in the claim.Being positioned at word " " before the element or " one " does not get rid of and has a plurality of such elements.The present invention can realize by means of the hardware that includes some different elements and by means of the computing machine of suitably programming.In having enumerated the unit claim of some devices, several in these devices can be to come imbody by same hardware branch.The use of word first, second and C grade does not represent any order.Can be title with these word explanations.
Claims (5)
1. result collection system comprises: server end be deployed in different production servers on a plurality of clients,
Described client is suitable for obtaining the data of producing the corresponding different classes of business that server produces, and is to send to described server end after the data with the aggregation of data of specific identifier;
Described server end is suitable for from each client data, and stores or transmit.
2. the system as claimed in claim 1, wherein, described client comprises: data capture unit, merger processing unit and a plurality of storage unit, and described a plurality of storage unit are the different classes of business of correspondence respectively, and each storage unit has the timing cycle of a correspondence;
Described data capture unit is suitable for obtaining from producing server the data of corresponding different classes of business, and the data of obtaining are preserved to the storage unit of correspondence according to the different classes of distribution of services of correspondence; Wherein, every data comprise more than one field, and different fields has different types, and at least one field identification of every data has key;
Each storage unit is suitable for preserving the data from data capture unit;
Described merger processing unit is suitable for when timing cycle corresponding to each storage unit finishes, and it is to send to server end after the data that the sign in the data that this storage unit is preserved has the identical aggregation of data of value of the field of key.
3. system as claimed in claim 2, wherein,
Described merger processing unit, be further adapted for when the professional corresponding timing cycle of every kind finishes, when having the identical aggregation of data of value of the field of key to be data the sign in the data of such various-service of preserving, to not identifying the field of key, carry out different merger according to different types and process.
4. system as claimed in claim 3, wherein,
The merger processing unit is further adapted for when according to different types the field that does not identify key being carried out different merger processing one or more combination below adopting:
For the field of sum-type, sign is had the numerical value addition on this fields of each identical data of the value of field of key, itself and as the value of this field after the merger;
For the field that is averaging type, sign is had the numerical value on this fields of each identical data of the value of field of key be averaging, its average as merger after the value of this field;
For the field of maximal value type, from having value on this fields of each identical data of the value of field of key, sign finds out maximal value, as the value of this field after the merger;
For the field of normal character types, from sign value on this field of getting article one data each identical data of the value of field of key is arranged, as the value of this field after the merger;
For the field of cumulative character types, sign is had character on this fields of each identical data of the value of field of key by the specified order serial connection after, as the value of this field after the merger.
5. such as each described system in the claim 1 to 4, wherein,
Described server end is suitable for the data retransmission that will receive to other server, or is forwarded to database facility, or retain costs ground file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210404697.XA CN102902813B (en) | 2012-10-22 | 2012-10-22 | Result collection system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210404697.XA CN102902813B (en) | 2012-10-22 | 2012-10-22 | Result collection system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102902813A true CN102902813A (en) | 2013-01-30 |
| CN102902813B CN102902813B (en) | 2016-08-24 |
Family
ID=47575045
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210404697.XA Active CN102902813B (en) | 2012-10-22 | 2012-10-22 | Result collection system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102902813B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103414693A (en) * | 2013-07-15 | 2013-11-27 | 北京奇虎科技有限公司 | Dotting method and dotting device |
| CN104935528A (en) * | 2015-06-10 | 2015-09-23 | 柳州市智融科技有限公司 | Internet big data processing platform |
| CN105007236A (en) * | 2015-06-10 | 2015-10-28 | 柳州市智融科技有限公司 | Network big data processing system |
| CN105007232A (en) * | 2015-06-10 | 2015-10-28 | 柳州市智融科技有限公司 | Network big data processing platform |
| CN105007237A (en) * | 2015-06-10 | 2015-10-28 | 柳州市智融科技有限公司 | Network information processing platform |
| CN105049371A (en) * | 2015-06-10 | 2015-11-11 | 柳州市智融科技有限公司 | Network information processing system |
| CN105357549A (en) * | 2015-11-09 | 2016-02-24 | 天津网络广播电视台有限公司 | Data collection system and data collection method for set-top box |
| CN105721179A (en) * | 2014-12-02 | 2016-06-29 | 北京奇虎科技有限公司 | Log collection system and data transmission method and local server therein |
| CN107180060A (en) * | 2016-03-11 | 2017-09-19 | 北京京东尚科信息技术有限公司 | The output intent and daily record output device of log information |
| CN108322350A (en) * | 2018-02-27 | 2018-07-24 | 阿里巴巴集团控股有限公司 | Business monitoring method and device and electronic equipment |
| CN110968561A (en) * | 2018-09-30 | 2020-04-07 | 北京国双科技有限公司 | Log storage method and distributed system |
| CN111159129A (en) * | 2019-12-31 | 2020-05-15 | 北京神州绿盟信息安全科技股份有限公司 | Statistical method and device for log report |
| CN112003743A (en) * | 2014-11-14 | 2020-11-27 | 北京通达无限科技有限公司 | Service data processing method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100088354A1 (en) * | 2006-11-30 | 2010-04-08 | Alibaba Group Holding Limited | Method and System for Log File Analysis Based on Distributed Computing Network |
| CN102523106A (en) * | 2011-12-04 | 2012-06-27 | 东华大学 | Video website user behavior analysis system based on Flex RIA (Rich Internet Applications) technology |
-
2012
- 2012-10-22 CN CN201210404697.XA patent/CN102902813B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100088354A1 (en) * | 2006-11-30 | 2010-04-08 | Alibaba Group Holding Limited | Method and System for Log File Analysis Based on Distributed Computing Network |
| CN102523106A (en) * | 2011-12-04 | 2012-06-27 | 东华大学 | Video website user behavior analysis system based on Flex RIA (Rich Internet Applications) technology |
Non-Patent Citations (1)
| Title |
|---|
| 许小明: "多源异构日志的数据归并和预处理技术", 《中国优秀硕士学位论文全文数据库信息科技辑》, 15 June 2009 (2009-06-15) * |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103414693A (en) * | 2013-07-15 | 2013-11-27 | 北京奇虎科技有限公司 | Dotting method and dotting device |
| CN112003743A (en) * | 2014-11-14 | 2020-11-27 | 北京通达无限科技有限公司 | Service data processing method and device |
| CN112003743B (en) * | 2014-11-14 | 2023-04-18 | 北京通达无限科技有限公司 | Service data processing method and device |
| CN105721179A (en) * | 2014-12-02 | 2016-06-29 | 北京奇虎科技有限公司 | Log collection system and data transmission method and local server therein |
| CN104935528A (en) * | 2015-06-10 | 2015-09-23 | 柳州市智融科技有限公司 | Internet big data processing platform |
| CN105007236A (en) * | 2015-06-10 | 2015-10-28 | 柳州市智融科技有限公司 | Network big data processing system |
| CN105007232A (en) * | 2015-06-10 | 2015-10-28 | 柳州市智融科技有限公司 | Network big data processing platform |
| CN105007237A (en) * | 2015-06-10 | 2015-10-28 | 柳州市智融科技有限公司 | Network information processing platform |
| CN105049371A (en) * | 2015-06-10 | 2015-11-11 | 柳州市智融科技有限公司 | Network information processing system |
| CN105357549A (en) * | 2015-11-09 | 2016-02-24 | 天津网络广播电视台有限公司 | Data collection system and data collection method for set-top box |
| CN107180060A (en) * | 2016-03-11 | 2017-09-19 | 北京京东尚科信息技术有限公司 | The output intent and daily record output device of log information |
| CN108322350A (en) * | 2018-02-27 | 2018-07-24 | 阿里巴巴集团控股有限公司 | Business monitoring method and device and electronic equipment |
| CN110968561A (en) * | 2018-09-30 | 2020-04-07 | 北京国双科技有限公司 | Log storage method and distributed system |
| CN110968561B (en) * | 2018-09-30 | 2024-02-13 | 北京国双科技有限公司 | Log storage method and distributed system |
| CN111159129A (en) * | 2019-12-31 | 2020-05-15 | 北京神州绿盟信息安全科技股份有限公司 | Statistical method and device for log report |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102902813B (en) | 2016-08-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102902813A (en) | Log collection system | |
| US11182098B2 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| CN106611046B (en) | Spatial data storage processing middleware system based on big data technology | |
| Shah et al. | A framework for social media data analytics using Elasticsearch and Kibana | |
| CN106934014B (en) | Hadoop-based network data mining and analyzing platform and method thereof | |
| CN106708993B (en) | Method for realizing space data storage processing middleware framework based on big data technology | |
| CN105677844B (en) | A kind of orientation of moving advertising big data pushes and user is across screen recognition methodss | |
| CN102937984B (en) | A kind of collect the system of data, client and method | |
| CN106815338A (en) | A kind of real-time storage of big data, treatment and inquiry system | |
| CN103984755A (en) | Multidimensional model based oil and gas resource data key system implementation method and system | |
| US20210279265A1 (en) | Optimization for Real-Time, Parallel Execution of Models for Extracting High-Value Information from Data Streams | |
| CN110633186A (en) | Log monitoring system for electric power metering micro-service architecture and implementation method | |
| CN109388637A (en) | Data warehouse information processing method, device, system, medium | |
| CN103970902A (en) | Method and system for reliable and instant retrieval on situation of large quantities of data | |
| CN109063196A (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN107103064A (en) | Data statistical approach and device | |
| CN109977125A (en) | A kind of big data safety analysis plateform system based on network security | |
| US10127617B2 (en) | System for analyzing social media data and method of analyzing social media data using the same | |
| CN105956932A (en) | Distribution and utilization data fusion method and system | |
| CN109446167A (en) | A kind of storage of daily record data, extracting method and device | |
| CN114443599A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN110134688B (en) | Hot event data storage management method and system in online social network | |
| Suciu et al. | Big data technology for scientific applications | |
| EP3380906A1 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| CN113111244A (en) | Multisource heterogeneous big data fusion system based on traditional Chinese medicine knowledge large-scale popularization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220725 Address after: Room 801, 8th floor, No. 104, floors 1-19, building 2, yard 6, Jiuxianqiao Road, Chaoyang District, Beijing 100015 Patentee after: BEIJING QIHOO TECHNOLOGY Co.,Ltd. Address before: 100088 room 112, block D, 28 new street, new street, Xicheng District, Beijing (Desheng Park) Patentee before: BEIJING QIHOO TECHNOLOGY Co.,Ltd. Patentee before: Qizhi software (Beijing) Co.,Ltd. |
|
| TR01 | Transfer of patent right |