CN110083600A - A kind of method, apparatus, calculating equipment and the storage medium of log collection processing - Google Patents
A kind of method, apparatus, calculating equipment and the storage medium of log collection processing Download PDFInfo
- Publication number
- CN110083600A CN110083600A CN201910268683.1A CN201910268683A CN110083600A CN 110083600 A CN110083600 A CN 110083600A CN 201910268683 A CN201910268683 A CN 201910268683A CN 110083600 A CN110083600 A CN 110083600A
- Authority
- CN
- China
- Prior art keywords
- log
- database
- dimension
- preset period
- journal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Abstract
The invention discloses a kind of method, apparatus of log collection processing, calculate equipment and storage medium, wherein, method includes: the first log for receiving terminal and sending, first log is handled, by treated, the first log is stored to first database, and the concordance list of first database is updated, according to the concordance list of first database, first database, determine catalogue file.The technical solution is used to collect effective log of user's needs, improves collection efficiency, reduces compiling costs and terminal pressure.
Description
Technical field
The present embodiments relate to the method, apparatus of log collection field more particularly to a kind of processing of log collection, calculate
Equipment and storage medium.
Background technique
The collection of terminal APP (Application, application program) log and processing technique reach its maturity at present, and develop
Diversification.Current log collection technology is mostly whole logs of collection terminal, and is handled and shown.But the prior art
In, the partial log being collected into not is that is, invalid log required for user, collects these invalid logs and not only increases receipts
Collect cost, while bringing unnecessary pressure to terminal.
Summary of the invention
The embodiment of the present invention provides a kind of method, apparatus of log collection processing, calculates equipment and storage medium, for receiving
Collect effective log that user needs, improves collection efficiency, reduce compiling costs and terminal pressure.
A kind of method of log collection processing provided in an embodiment of the present invention, comprising:
Receive the first log that terminal is sent；First log is the terminal after receiving configuration information, according to
It is determined in the configuration information to the dimension of collector journal；
First log is handled；
By treated, the first log is stored to first database, and updates the concordance list of the first database；
According to the concordance list of the first database, the first database, catalogue file is determined.
In above-mentioned technical proposal, terminal receives configuration information, determines the first log according to configuration information, and by the first log
It is sent to server, the first log that server receiving terminal is sent, effective day in first log, that is, log collection technique
Will, server are not necessarily to whole logs of collection terminal, filter out invalid log during collector journal, only collect what user needed
Effective log improves the efficiency of log collection, and reduces compiling costs and terminal pressure.Further, server receives
After effective log, effective log is handled, improves the efficiency of log processing, log is stored to first by treated
Database, and then catalogue file is determined according to the concordance list of first database, first database, wherein the catalogue file is straight
It sees, visual file, user can both inquire the catalogue file without learning Relational database language, reduce user's
Learning cost, and high-volume data can be showed into user, improve the efficiency of inquiry log.
Optionally, before the first log that the reception terminal is sent, further includes:
Configuration information is obtained, the configuration information is sent to the terminal.
In above-mentioned technical proposal, user can choose the dimension to collector journal, and issue to server include this to
The configuration information of the dimension of collector journal, user can be selected the dimension to collector journal according to their needs, be improved
User experience.And configuration information can be sent to terminal after receiving the configuration information by server so that terminal according to
The dimension to collector journal in configuration information sends the first log to server, and then server only collects user's needs
Effective log improves the efficiency of log collection, and reduces compiling costs and terminal pressure.
Optionally, the concordance list according to the first database, the first database, determines catalogue file, packet
It includes:
Obtain the first database treated first log described in preset period of time storage and first number
According to library the preset period of time concordance list；
According to the first database in the concordance list of the preset period of time, the first database is deposited in preset period of time
Treated that the first log is determined as multiple journal files for storage described, and generates the catalogue of each journal file；
The catalogue of the multiple journal file and each journal file is formed into the catalogue file.
In above-mentioned technical proposal, server is directed to each preset period of time, generates the catalogue file of this preset period of time, i.e., pre-
If after the period, according to first database in the concordance list of this preset period of time, will be deposited in first database in this preset period of time
The data of storage generate the catalogue file of this preset period of time, i.e., by the data for needing database language to inquire be converted into it is intuitive,
Visual catalogue file facilitates user in the case that without treating inquiry log and being looked into when learning Relational database language
It askes, and keeps the consistency of first database data in the data and catalogue file that this preset period of time stores, it is ensured that data are not
It loses.Catalogue file is stored into local data base, is supported the direct local search of user, is improved the efficiency of inquiry log.
Optionally, further includes:
Receive log query instruction；It include preset period of time where log to be checked in log query instruction and to be checked
Ask the dimension of log；
If the preset period of time where the log to be checked is the current preset period, according to the dimension of the log to be checked
Degree, inquires the first database in the concordance list of the current preset period, and institute is determined from the first database
State log to be checked；
The corresponding catalogue file of the log to be checked is generated according to the log to be checked, so that the user looks into
It askes.
In above-mentioned technical proposal, server log inquiry instruction, when log query instruction can be inquiry current preset
The log of section, is also possible to inquire the log (before the current preset period) of other preset period of time, is instructed according to log query
Log to be checked where preset period of time, it is determined whether the log of inquiry current preset period needs if so, determining the
It is inquired in one database, at this time, it may be necessary to the concordance list according to first database in the current preset period, from first database really
Make log to be checked, and the log to be checked determined generated into corresponding catalogue file, store to local, for user into
Row inquiry, this programme support the log of user query current preset period, and provide local search and intuitive visual mesh
Record inquiry.
Correspondingly, the embodiment of the invention also provides a kind of methods of log collection processing, comprising:
The configuration information that server is sent is received, the configuration information includes the dimension to collector journal；
According to the dimension to collector journal, the first log is determined；
First log is sent to the server, so that the server is after receiving first log,
First log is handled.
In above-mentioned technical proposal, terminal receives the configuration information that server is sent, wherein configuration information includes due-in market day
The dimension of will, terminal determine that the first log is reported according to the dimension to collector journal, i.e., terminal is not necessarily to whole logs
File reports, and reduces the pressure that terminal is given the correct time in log.
Optionally, it is described according to the dimension of collector journal, determine the first log, comprising:
According to the dimension to collector journal, the dimension to collector journal is determined from the log being locally stored
Corresponding log information；
The corresponding log information of the dimension to collector journal is determined as first log.
In above-mentioned technical proposal, terminal determines the corresponding log of dimension to collector journal from the log being locally stored
Information, i.e., the effective log for only determining that user needs reduce the pressure of terminal for reporting to server.
Correspondingly, the embodiment of the invention also provides a kind of devices of log collection processing, comprising:
Receiving unit, for receiving the first log of terminal transmission；First log, which is the terminal, matches receiving
After confidence breath, according to what is determined in the configuration information to the dimension of collector journal；
Processing unit, for handling first log；By treated, the first log is stored to the first data
Library, and update the concordance list of the first database；According to the concordance list of the first database, the first database, really
Determine catalogue file.
Optionally, the processing unit is also used to:
Before the first log that the reception terminal is sent, configuration information is obtained, the configuration information is sent to institute
State terminal.
Optionally, the processing unit is specifically used for:
Obtain the first database treated first log described in preset period of time storage and first number
According to library the preset period of time concordance list；
According to the first database in the concordance list of the preset period of time, the first database is deposited in preset period of time
Treated that the first log is determined as multiple journal files for storage described, and generates the catalogue of each journal file；
The catalogue of the multiple journal file and each journal file is formed into the catalogue file.
Optionally, the processing unit is also used to:
Receive log query instruction；It include preset period of time where log to be checked in log query instruction and to be checked
Ask the dimension of log；
If the preset period of time where the log to be checked is the current preset period, according to the dimension of the log to be checked
Degree, inquires the first database in the concordance list of the current preset period, and institute is determined from the first database
State log to be checked；
The corresponding catalogue file of the log to be checked is generated according to the log to be checked, so that the user looks into
It askes.
Correspondingly, the embodiment of the invention also provides a kind of devices of log collection processing, comprising:
Receiving unit, for receiving the configuration information of server transmission, the configuration information includes the dimension to collector journal
Degree；
Processing unit, for determining the first log according to the dimension to collector journal；First log is sent
To the server, so that the server after receiving first log, is handled first log.
Optionally, the processing unit is specifically used for:
According to the dimension to collector journal, the dimension to collector journal is determined from the log being locally stored
Corresponding log information；
The corresponding log information of the dimension to collector journal is determined as first log.
Correspondingly, the embodiment of the invention also provides a kind of calculating equipment, comprising:
Memory, for storing program instruction；
Processor executes above-mentioned log according to the program of acquisition for calling the program instruction stored in the memory
The method for collecting processing.
Correspondingly, the embodiment of the invention also provides a kind of computer-readable non-volatile memory medium, including computer
Readable instruction, when computer is read and executes the computer-readable instruction, so that computer executes at above-mentioned log collection
The method of reason.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly introduced, it should be apparent that, drawings in the following description are only some embodiments of the invention, for this
For the those of ordinary skill in field, without creative efforts, it can also be obtained according to these attached drawings other
Attached drawing.
Fig. 1 is a kind of schematic diagram of system architecture provided in an embodiment of the present invention；
Fig. 2 is the schematic diagram of another system architecture provided in an embodiment of the present invention；
Fig. 3 is a kind of flow diagram of the method for log collection processing provided in an embodiment of the present invention；
Fig. 4 is a kind of structural schematic diagram of the device of log collection processing provided in an embodiment of the present invention；
Fig. 5 is a kind of structural schematic diagram of the device of log collection processing provided in an embodiment of the present invention.
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction with attached drawing to the present invention make into
It is described in detail to one step, it is clear that described embodiments are only a part of the embodiments of the present invention, rather than whole implementation
Example.Based on the embodiments of the present invention, obtained by those of ordinary skill in the art without making creative efforts
All other embodiment, shall fall within the protection scope of the present invention.
Fig. 1 illustratively shows the system architecture that the method that the embodiment of the present invention provides log collection processing is applicable in,
The system architecture may include server 100 and terminal 200, and server 100 is connect with multiple terminals 200, and server 100 is used for
The log that acquisition terminal 200 reports, and log is handled and stored.
In a kind of achievable mode, server 100 can be multiple, and configuration can be divided into according to different function
Server 110, gateway 120, processing server 130, HDFS (Hadoop Distributed File System, distributed text
Part system) 140 and log server 150, be referred to Fig. 2 shows system architecture.Wherein, configuration server 110 is for connecing
The configuration information that user issues is received, and configuration information is sent to terminal 200, so that terminal 200 is reported according to configuration information
Log is sent to processing server 130 by log to gateway 120, gateway 120, and processing server 130 carries out ETL to log
(Extract-Transform-Load is extracted, conversion, loaded) processing, wherein preset period of time can be set, when each default
A grouping in the corresponding HDFS 140 of section can will treated day after log is carried out ETL processing by processing server 130
Will data are written according to preset period of time in real time in the grouping of the corresponding HDFS 140 of the preset period of time, if certain preset period of time terminates
Afterwards, the data in the grouping of the corresponding HDFS 140 of the preset period of time may synchronize to log server 150, log server
150 provide local log query for user.
Based on foregoing description, Fig. 3 illustratively shows a kind of side of log collection processing provided in an embodiment of the present invention
The process of method, the process can by log collection handle device execute, hereafter mainly with server and terminal interact for
Example is described.As shown in figure 3, the process specifically includes:
Step 301, server sends configuration information to terminal.
In the embodiment of the present invention, configuration information can be sent to the SDK (Software of terminal APP by server
Development Kit, Software Development Kit) it is inner, wherein it may include the dimension to collector journal in configuration information, i.e.,
Terminal reports the rule of log information, such as sampling prescription (every ten are sampled one), reporting types regular (crash log, access
Log, accelerates log at identification log) etc..
Configuration information can be according to rule it is preconfigured, be also possible to user and be handed down to server.User issues
It, can be at the interface of server, as the interface portal selects the dimension to collector journal, and by the due-in of selection when configuration information
The dimension of collection log is input to server, so that server generates configuration information, and the configuration information of generation is sent to end
End.In addition, the configuration information can be what user was configured in the server for the first time, existing due-in market day can also be directed to
After the dimension of will is modified, that is to say, that user can according to the needs of the dimension to collector journal, to configuration information into
Row is modified at any time.
Step 302, terminal determines the first log according to the dimension to collector journal.
Specifically, terminal stores the log of generation to local, it, can be according to wait collect when terminal reports log information
The dimension of log determines the corresponding log information of dimension to collector journal from the log being locally stored, and this is due-in
The corresponding log information of dimension of collection log is determined as the first log.Illustratively, if the dimension to collector journal is type rule
Then, and the type rule includes the crash log of reporting terminal and identifies log, then configuration information is sent to terminal by server
Afterwards, terminal determines crash log and identification log according to typing rule from the log being locally stored.
Seen from the above description, the first log is terminal after receiving configuration information, according in configuration information wait collect
What the dimension of log determined, which typing rule, sampling prescription or other rules can determine according to.It needs
Supplement, the dimension to collector journal can also connect for area rule, server with multiple terminals, and multiple terminals can divide
It Wei Yu multiple areas, for example, server and 100 terminals connections have 20 terminals to be located at Fujian in 100 terminals
Area, 80 terminals are located at Beijing area, then server can send configuration information to 100 terminals, wherein configuration information
In the dimension to collector journal can be regional rule, for example, the terminal of Beijing area reports log information, at this point, being located at
80 terminals of Beijing area report respective log information, and 20 terminals positioned at In Fujian Province do not report respective log to believe
Breath.
Step 303, first log is sent to the server by terminal.
A kind of optional implementation can be, terminal can call gateway interface and will after determining the first log
First log reports to server, and the first log is distributed in different message queues by server according to different Log Types,
The different topic such as kafka.
Step 304, server handles first log.
Real-time handler carries out real-time ETL processing according to the data in message queue in server, i.e. terminal reports
After first log to server, server handles first log in real time.Wherein, which can be
The real-time processing frame of Flink, Flink are the distributed process engines for being directed to flow data and batch data.It mainly by
Java code realizes there is that infinite horizontal extension, handling capacity is big, retardance is low.
Step 305, by treated, the first log is stored to first database server, and updates the first database
Concordance list.
In the embodiment of the present invention, first database can be understood as a distributed file system, the first database
In may include the grouping of multiple databases, the journal file of preset period of time, preset period of time, that is, pre- are stored in each database grouping
The period of the log collection first set, preset period of time can rule of thumb be set as a few houres, one day or several days.It can be such as table 1
For, each preset period of time is one day, corresponds to every day and is grouped in a database, and a corresponding concordance list.
After server is handled the first log, according to being currently located the period, will treated the first log stores to
In first database, and the concordance list of first database is updated, for example, being currently located the period is Day2, then server is by first
Log is stored into the corresponding second packet of Day2, and updates corresponding second concordance list of Day2.
In the embodiment of the present invention, the concordance list of first database can be divided into corresponding with preset period of time according to preset period of time
Concordance list, such as corresponding concordance list of each preset period of time.In addition, the concordance list of first database can also be an index
Table, i.e., multiple preset period of time correspond to a concordance lists, may include the keyword for identifying preset period of time in the concordance list.
Table 1
| Preset period of time | Grouping in first database | Concordance list |
| Day1 | First grouping | First concordance list |
| Day2 | Second packet | Second concordance list |
| Day3 | Third grouping | Third concordance list |
| Day4 | 4th grouping | 4th concordance list |
| Day5 | 5th grouping | 5th concordance list |
In the concrete realization, first database can be HDFS, and preset period of time is 1 day (0:00-24.00), and server will
On the day of the first log collected stored after handling in real time to HDFS in corresponding grouping, and according to log field and HDFS mesh
Record maps out HIVE appearance and partition table daily.In addition, HIVE table generates next day partition table in daily 23:00.
Step 306, server determines catalogue file according to the concordance list of the first database, the first database.
Server is after a certain preset period of time, the data that available first database is stored in the preset period of time,
The first log that i.e. treated and the first database the preset period of time concordance list, and according to first database at this
First database is determined as multiple journal files and each day in the data that the preset period of time stores by the concordance list of preset period of time
The catalogue of will file.For table 1, if after Day2, the data of the corresponding second packet of the available Day2 of server with
And second concordance list, and according to the second concordance list, the data of second packet are determined as multiple journal files and each log text
The catalogue of part.It is construed to, the data in first database is synchronized in local data base, storing in first database is place
The first log and concordance list after reason, in synchronizing process, can according to concordance list will treated the first log according to intuitive
Visual mode is synchronous be multiple journal files and each journal file catalogue, and by multiple journal files and each log
After the catalogue composition catalogue file of file, store to local data base, so that user intuitively selects and checks.
In the concrete realization, the data on the day before HDFS can be synchronized to log server, the log in morning 2:00
Server can provide a user intuitive visual catalogue file, and user when in use, can direct log server phase
Journal file is checked under the catalogue of pass.
In addition, user can check journal file in the server, specifically, user selects inquiry dimension according to demand
Afterwards, log query instruction is issued to server, server instructs according to log query and generates SQL (Structured Query
Language, structured query language) go log server to inquire.Log query instruction is including default where log to be checked
The dimension of period and log to be checked, wherein the preset period of time where log to be checked can be current preset period of time, can also
Preset period of time before thinking, for example, user issues log query instruction in Day2, to be checked in log query instruction
The log that log can collect for the Day2 period is ask, the log that the Day1 period collects is also possible to.
Server is different for the preset period of time where log to be checked, can carry out different inquiry modes, specifically,
Server is after receiving log query instruction, when whether the preset period of time where first judging log to be checked is current preset
Section, if it is not, then server can determine to be stored with the relevant catalogue file of the log to be checked in local data base,
Server can feed back to user or server after directly inquiring in the local database according to the dimension of log to be checked
It can guide user that log server is gone to check.If server determines that the preset period of time where log to be checked is current preset
Section, then can determine in local data base it is also not stored have a relevant catalogue file of the log to be checked, server according to
The dimension of inquiry log inquires concordance list of the first database in the current preset period, and determine from first database to
Inquiry log, and the corresponding catalogue file of log to be checked is generated according to log to be checked, and feed back to user.
During specific implementation, user can issue instruction to be checked by portal, and portal is accessed by interface
HIVE appearance in HDFS, and the corresponding daily record data of instruction to be checked is inquired by HIVE appearance, portal passes through outside HIVE
After table is inquired, interface can recorde whether query result succeeds, and portal accesses recording in the interface as a result, if this connects
The result recorded in mouthful determines day to be checked by HIVE appearance as a result, then representing portal for successful inquiring from HDFS
Will, and log to be checked is integrated into log server, user can check the day to be checked by log server later
Will.
In above-mentioned technical proposal, terminal receives configuration information, determines the first log according to configuration information, and by the first log
It is sent to server, the first log that server receiving terminal is sent, effective day in first log, that is, log collection technique
Will, server are not necessarily to whole logs of collection terminal, filter out invalid log during collector journal, only collect what user needed
Effective log improves the efficiency of log collection, and reduces compiling costs and terminal pressure.Further, server receives
After effective log, effective log is handled, improves the efficiency of log processing, log is stored to first by treated
Database, and then catalogue file is determined according to the concordance list of first database, first database, wherein the catalogue file is straight
It sees, visual file, user can both inquire the catalogue file without learning Relational database language, reduce user's
Learning cost, and high-volume data can be showed into user, improve the efficiency of inquiry log.
Based on the same inventive concept, Fig. 4 illustratively shows a kind of log collection processing provided in an embodiment of the present invention
Device structure, the device can with execution journal collect processing method process.
Receiving unit 401, for receiving the first log of terminal transmission；First log is that the terminal is receiving
After configuration information, according to what is determined in the configuration information to the dimension of collector journal；
Processing unit 402, for handling first log；By treated, the first log is stored to the first number
According to library, and update the concordance list of the first database；According to the concordance list of the first database, the first database,
Determine catalogue file.
Optionally, the processing unit 402 is also used to:
Before the first log that the reception terminal is sent, configuration information is obtained, the configuration information is sent to institute
State terminal.
Optionally, the processing unit 402 is specifically used for:
Obtain the first database treated first log described in preset period of time storage and first number
According to library the preset period of time concordance list；
According to the first database in the concordance list of the preset period of time, the first database is deposited in preset period of time
Treated that the first log is determined as multiple journal files for storage described, and generates the catalogue of each journal file；
The catalogue of the multiple journal file and each journal file is formed into the catalogue file.
Optionally, the processing unit 402 is also used to:
Receive log query instruction；It include preset period of time where log to be checked in log query instruction and to be checked
Ask the dimension of log；
If the preset period of time where the log to be checked is the current preset period, according to the dimension of the log to be checked
Degree, inquires the first database in the concordance list of the current preset period, and institute is determined from the first database
State log to be checked；
The corresponding catalogue file of the log to be checked is generated according to the log to be checked, so that the user looks into
It askes.
Based on the same inventive concept, Fig. 5 illustratively shows a kind of log collection processing provided in an embodiment of the present invention
Device structure, the device can with execution journal collect processing method process.
Receiving unit 501, for receiving the configuration information of server transmission, the configuration information includes to collector journal
Dimension；
Processing unit 502, for determining the first log according to the dimension to collector journal；By first log
It is sent to the server, so that the server after receiving first log, is handled first log.
Optionally, the processing unit 502 is specifically used for:
According to the dimension to collector journal, the dimension to collector journal is determined from the log being locally stored
Corresponding log information；
The corresponding log information of the dimension to collector journal is determined as first log.
Based on the same inventive concept, the embodiment of the invention also provides a kind of calculating equipment, comprising:
Memory, for storing program instruction；
Processor executes above-mentioned log according to the program of acquisition for calling the program instruction stored in the memory
The method for collecting processing.
Based on the same inventive concept, the embodiment of the invention also provides a kind of computer-readable non-volatile memory medium,
Including computer-readable instruction, when computer is read and executes the computer-readable instruction, so that computer execution is above-mentioned
The method of log collection processing.
The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Although preferred embodiments of the present invention have been described, it is created once a person skilled in the art knows basic
Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as
It selects embodiment and falls into all change and modification of the scope of the invention.
Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art
Mind and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to include these modifications and variations.
Claims (14)
1. a kind of method of log collection processing characterized by comprising
Receive the first log that terminal is sent；First log is the terminal after receiving configuration information, according to described
It is determined in configuration information to the dimension of collector journal；
First log is handled；
By treated, the first log is stored to first database, and updates the concordance list of the first database；
According to the concordance list of the first database, the first database, catalogue file is determined.
2. the method as described in claim 1, which is characterized in that before the first log that the reception terminal is sent, also wrap
It includes:
Configuration information is obtained, the configuration information is sent to the terminal.
3. the method as described in claim 1, which is characterized in that described according to the first database, the first database
Concordance list, determine catalogue file, comprising:
Obtain the first database treated the first log and first database described in preset period of time storage
In the concordance list of the preset period of time；
According to the first database in the concordance list of the preset period of time, the first database is stored in preset period of time
Described treated that the first log is determined as multiple journal files, and generates the catalogue of each journal file；
The catalogue of the multiple journal file and each journal file is formed into the catalogue file.
4. the method as described in claim 1, which is characterized in that further include:
Receive log query instruction；It include the preset period of time where log to be checked and day to be checked in log query instruction
The dimension of will；
If preset period of time where the log to be checked is the current preset period, according to the dimension of the log to be checked,
The first database is inquired in the concordance list of the current preset period, and determine from the first database it is described to
Inquiry log；
The corresponding catalogue file of the log to be checked is generated according to the log to be checked, so that the user inquires.
5. a kind of method of log collection processing characterized by comprising
The configuration information that server is sent is received, the configuration information includes the dimension to collector journal；
According to the dimension to collector journal, the first log is determined；
First log is sent to the server, so that the server is after receiving first log, to institute
The first log is stated to be handled.
6. method as claimed in claim 5, which is characterized in that it is described according to the dimension of collector journal, determine first
Log, comprising:
According to the dimension to collector journal, determine that the dimension to collector journal is corresponding from the log being locally stored
Log information；
The corresponding log information of the dimension to collector journal is determined as first log.
7. a kind of device of log collection processing characterized by comprising
Receiving unit, for receiving the first log of terminal transmission；First log is that the terminal is being received with confidence
After breath, according to what is determined in the configuration information to the dimension of collector journal；
Processing unit, for handling first log；By treated, the first log is stored to first database, and
Update the concordance list of the first database；According to the concordance list of the first database, the first database, catalogue is determined
File.
8. device as claimed in claim 7, which is characterized in that the processing unit is also used to:
Before the first log that the reception terminal is sent, configuration information is obtained, the configuration information is sent to the end
End.
9. device as claimed in claim 7, which is characterized in that the processing unit is specifically used for:
Obtain the first database treated the first log and first database described in preset period of time storage
In the concordance list of the preset period of time；
According to the first database in the concordance list of the preset period of time, the first database is stored in preset period of time
Described treated that the first log is determined as multiple journal files, and generates the catalogue of each journal file；
The catalogue of the multiple journal file and each journal file is formed into the catalogue file.
10. device as claimed in claim 7, which is characterized in that the processing unit is also used to:
Receive log query instruction；It include the preset period of time where log to be checked and day to be checked in log query instruction
The dimension of will；
If preset period of time where the log to be checked is the current preset period, according to the dimension of the log to be checked,
The first database is inquired in the concordance list of the current preset period, and determine from the first database it is described to
Inquiry log；
The corresponding catalogue file of the log to be checked is generated according to the log to be checked, so that the user inquires.
11. a kind of device of log collection processing characterized by comprising
Receiving unit, for receiving the configuration information of server transmission, the configuration information includes the dimension to collector journal；
Processing unit, for determining the first log according to the dimension to collector journal；First log is sent to institute
Server is stated, so that the server after receiving first log, is handled first log.
12. device as claimed in claim 11, which is characterized in that the processing unit is specifically used for:
According to the dimension to collector journal, determine that the dimension to collector journal is corresponding from the log being locally stored
Log information；
The corresponding log information of the dimension to collector journal is determined as first log.
13. a kind of calculating equipment characterized by comprising
Memory, for storing program instruction；
Processor requires 1 to 6 according to the program execution benefit of acquisition for calling the program instruction stored in the memory
Described in any item methods.
14. a kind of computer-readable non-volatile memory medium, which is characterized in that including computer-readable instruction, work as computer
When reading and executing the computer-readable instruction, so that computer executes such as method as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910268683.1A CN110083600A (en) | 2019-04-04 | 2019-04-04 | A kind of method, apparatus, calculating equipment and the storage medium of log collection processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910268683.1A CN110083600A (en) | 2019-04-04 | 2019-04-04 | A kind of method, apparatus, calculating equipment and the storage medium of log collection processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110083600A true CN110083600A (en) | 2019-08-02 |
Family
ID=67414225
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910268683.1A Pending CN110083600A (en) | 2019-04-04 | 2019-04-04 | A kind of method, apparatus, calculating equipment and the storage medium of log collection processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110083600A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110851514A (en) * | 2019-10-17 | 2020-02-28 | 杭州安恒信息技术股份有限公司 | ETL (extract transform and load) processing method based on FLINK (Linear rotation index) |
| CN111177193A (en) * | 2019-12-13 | 2020-05-19 | 航天信息股份有限公司 | Flink-based log streaming processing method and system |
| CN112084265A (en) * | 2020-09-11 | 2020-12-15 | 北京金山云网络技术有限公司 | Log data synchronization method and device, server and storage medium |
| CN113791946A (en) * | 2021-08-31 | 2021-12-14 | 北京达佳互联信息技术有限公司 | Log processing method and device, electronic equipment and storage medium |
| CN114006916A (en) * | 2020-07-16 | 2022-02-01 | 京东方科技集团股份有限公司 | Method, device and system for collecting device logs |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105279276A (en) * | 2015-11-11 | 2016-01-27 | 浪潮(北京)电子信息产业有限公司 | Database index optimization system |
| CN105468737A (en) * | 2015-11-24 | 2016-04-06 | 湖北大学 | Web service big data analysis method, cloud computing platform and mining system |
| CN105530299A (en) * | 2015-12-12 | 2016-04-27 | 天津南大通用数据技术股份有限公司 | Directory service log realization method |
-
2019
- 2019-04-04 CN CN201910268683.1A patent/CN110083600A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105279276A (en) * | 2015-11-11 | 2016-01-27 | 浪潮(北京)电子信息产业有限公司 | Database index optimization system |
| CN105468737A (en) * | 2015-11-24 | 2016-04-06 | 湖北大学 | Web service big data analysis method, cloud computing platform and mining system |
| CN105530299A (en) * | 2015-12-12 | 2016-04-27 | 天津南大通用数据技术股份有限公司 | Directory service log realization method |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110851514A (en) * | 2019-10-17 | 2020-02-28 | 杭州安恒信息技术股份有限公司 | ETL (extract transform and load) processing method based on FLINK (Linear rotation index) |
| CN111177193A (en) * | 2019-12-13 | 2020-05-19 | 航天信息股份有限公司 | Flink-based log streaming processing method and system |
| CN114006916A (en) * | 2020-07-16 | 2022-02-01 | 京东方科技集团股份有限公司 | Method, device and system for collecting device logs |
| CN112084265A (en) * | 2020-09-11 | 2020-12-15 | 北京金山云网络技术有限公司 | Log data synchronization method and device, server and storage medium |
| CN113791946A (en) * | 2021-08-31 | 2021-12-14 | 北京达佳互联信息技术有限公司 | Log processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110083600A (en) | A kind of method, apparatus, calculating equipment and the storage medium of log collection processing | |
| CN106126641B (en) | A kind of real-time recommendation system and method based on Spark | |
| CN104424229B (en) | A kind of calculation method and system that various dimensions are split | |
| CN104408159B (en) | A kind of data correlation, loading, querying method and device | |
| CN107070890A (en) | Flow data processing device and communication network major clique system in a kind of communication network major clique system | |
| CN110175154A (en) | A kind of processing method of log recording, server and storage medium | |
| CN105989076A (en) | Data statistical method and device | |
| CN107679192A (en) | More cluster synergistic data processing method, system, storage medium and equipment | |
| CN102054000B (en) | Data querying method, device and system | |
| CN106649828A (en) | Data query method and system | |
| CN110321383A (en) | Big data platform method of data synchronization, device, computer equipment and storage medium | |
| CN104111958A (en) | Data query method and device | |
| CN110704400A (en) | Real-time data synchronization method and device and server | |
| CN103793493A (en) | Method and system for processing car-mounted terminal mass data | |
| CN108536808A (en) | A kind of data capture method and device based on Spark Computational frames | |
| CN111666344B (en) | Heterogeneous data synchronization method and device | |
| CN111723161A (en) | Data processing method, device and equipment | |
| CN114416868B (en) | Data synchronization method, device, equipment and storage medium | |
| CN108763323A (en) | Meteorological lattice point file application process based on resource set and big data technology | |
| CN109033173A (en) | It is a kind of for generating the data processing method and device of multidimensional index data | |
| CN106649530B (en) | Cloud detail query management system and method | |
| CN105550351B (en) | The extemporaneous inquiry system of passenger's run-length data and method | |
| CN102982034A (en) | Internet website information search method and search system | |
| CN103778223B (en) | Pervasive word-reciting system based on cloud platform and construction method thereof | |
| WO2016206395A1 (en) | Weekly report information processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190802 |