CN110362548A - Log extracting method and relevant device - Google Patents
Log extracting method and relevant device Download PDFInfo
- Publication number
- CN110362548A CN110362548A CN201910505732.9A CN201910505732A CN110362548A CN 110362548 A CN110362548 A CN 110362548A CN 201910505732 A CN201910505732 A CN 201910505732A CN 110362548 A CN110362548 A CN 110362548A
- Authority
- CN
- China
- Prior art keywords
- file
- log
- slice
- time
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000000605 extraction Methods 0.000 claims abstract description 19
- 230000005540 biological transmission Effects 0.000 claims abstract description 11
- 239000000284 extract Substances 0.000 claims description 8
- 238000004891 communication Methods 0.000 claims description 6
- 230000006835 compression Effects 0.000 claims description 3
- 238000007906 compression Methods 0.000 claims description 3
- 238000012856 packing Methods 0.000 claims description 2
- 238000012216 screening Methods 0.000 abstract description 8
- 238000011835 investigation Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 7
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000004321 preservation Methods 0.000 description 2
- 239000012141 concentrate Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Abstract
The invention discloses a kind of log extracting method and relevant device, which includes: that terminal log is carried out slice storage, wherein the attribute of each slice file includes: the file generated time；After receiving the log request of some period of extraction of server transmission, at least one slice file corresponding with the period is extracted；At least one slice file generated journal file based on this；The journal file is uploaded to the server.The log extracting method automatic operation, without manual crawl；Also, since the daily record data of terminal is saved with multiple slice document forms, screening range is reduced, is conducive to reduction screening logging time, the present invention is conducive to improve log crawl efficiency, shortens the time of investigation terminal problem.The present invention is widely used in the long-range extraction of terminal log.
Description
Technical field
The present invention relates to field of computer technology, more particularly, to a kind of log extracting method and relevant device.
Background technique
When terminal breaks down, usually by the log recording of crawl terminal, carry out the reason of positioning terminal breaks down.
Traditional way is: after the whole journal files for obtaining terminal, then the crawl and screening of looking for professional to carry out log.
The shortcomings that conventional method, is: since the crawl and screening of log need to link up with corresponding developer, leading
It causes to increase and links up cost, to influence the speed of investigation problem.
Summary of the invention
The present invention is directed to solve at least some of the technical problems in related technologies.For this purpose, of the invention
One purpose is to provide a kind of log extracting method and relevant device, can facilitate crawl and screening log, be conducive to the row of raising
Interrogate the efficiency of topic.
The technical scheme adopted by the invention is that:
In a first aspect, the present invention provides a kind of log extracting method, which includes: to carry out terminal log
Slice storage, wherein the attribute of each slice file includes: the file generated time；In the extraction for receiving server transmission
After the log request of a period, at least one slice file corresponding with the period is extracted；Based on this, at least one is cut
Piece file generated journal file；The journal file is uploaded to the server.
Wherein, each slice file includes that file settling time and file finally revise the time；It extracts and the period
At least one corresponding slice file, comprising: finally revised the time based on this document settling time and this document, when obtaining with this
Between section it is corresponding at least one slice file.
Wherein, based at least one slice file generated journal file, comprising: by this, at least one slice file is carried out
It is packaged compression, generates the journal file.
Wherein, which further comprises that a plurality of record generates the time；At least one slice file generated based on this
Journal file, comprising: the time is generated according to the record, interception meets respectively in each file of at least one slice file from this
The daily record data of the period, and the daily record data intercepted in each file is individually saved as into a file；It will interception
The All Files arrived concentrate compression, generate the journal file.
Wherein, the size of each slice file is identical；Before terminal log is carried out slice storage, including pass through the clothes
The file size of the slice file is arranged to the terminal for business device.
Second aspect, the present invention provide a kind of log extraction element, which includes: log memory module,
For terminal log to be carried out slice storage, wherein the attribute of each slice file includes: the file generated time；Slice crawl
Module, for extracting corresponding with the period after receiving the log request of some period of extraction of server transmission
At least one slice file；Log generation module, for based at least one slice file generated journal file；And day
Will sending module, for the journal file to be uploaded to the server.
The third aspect, the present invention provide a kind of terminal, which includes: at least one processor；And with this at least one
The memory of a processor communication connection；Wherein, which is stored with the instruction that can be executed by least one processor, should
Instruction is able to carry out such as above-mentioned method.
Fourth aspect, the present invention provide a kind of server, wherein the server is used to carry out log from such as above-mentioned terminal
It extracts.
Wherein, after which is also used to obtain the journal file, it is corresponding which is stored in the server
Storage location, if the journal file number that the storage location saves exceeds scheduled quantity, at first by journal file replacement
The file of preservation.
5th aspect, the present invention provides a kind of log extraction system, the log extraction system include such as above-mentioned terminal and
Above-mentioned server.
The beneficial effects of the present invention are:
The present invention carries out slice storage by using by daily record data, the period then issued according to server, screening
At least one slice file corresponding with the period out, and based at least one slice file generated journal file, most
The journal file is uploaded to server afterwards.The log extracting method automatic operation, without manual crawl；Also, due to end
The daily record data at end is saved with multiple slice document forms, reduces screening range, is conducive to reduction screening logging time；Therefore,
The date extracting method of the embodiment of the present invention is conducive to improve log crawl efficiency, shortens the time of investigation terminal problem.
Further, each slice file of terminal of the present invention includes that file settling time and file finally revise the time,
Each slice file is equivalent to a time interval so that it is convenient to determine slice file corresponding with query time section.
Further, there is each slice file of terminal of the present invention a plurality of record to generate the time, can reject slice in this way
Unnecessary daily record data in file, reduces the scale of construction of journal file.
Further, it is also possible in the size of server end remote adjustment terminal slice file.
The remote automation that the present invention is widely used in terminal log extracts.
Detailed description of the invention
Fig. 1 is the flow diagram of an embodiment of log extracting method of the present invention；
Fig. 2 is the flow diagram of an embodiment of the step S13 of Fig. 1；
Fig. 3 is the structural schematic diagram of an embodiment of log extraction element of the present invention；
Fig. 4 is the structural schematic diagram of an embodiment of log extraction system of the present invention.
Specific embodiment
It should be noted that in the absence of conflict, the features in the embodiments and the embodiments of the present application can phase
Mutually combination.
Embodiment one:
Referring to Fig. 1, Fig. 1 is the flow diagram of an embodiment of log extracting method of the present invention.As shown in Figure 1, should
Method includes the following steps:
S11: terminal log is stored according to slice file mode, wherein the attribute of each slice file includes: text
Part generates the time；
In step s 11, after terminal or the cloud card server communicated with terminal are started, terminal log is generated.
The data of the terminal log include: terminal WIFI information, terminal SIM card information, terminal network link information, end cell information
And cloud card running state information.
Preferably, the size of each slice file is identical.For example, the size of each slice file is 20M, it is sliced when one
After the log recording size that file saves is more than 20M, a new slice file is regenerated.It is of course also possible to according to pre-
Fixed chronomere carries out saving slice file, that is, sets the one slice file of generation in every 5 minutes or 10 minutes.In step S11
Before, the file size of the slice file is set by the server to the terminal.
Preferably, each slice file includes file settling time and final revision time, for example, each slice file is pressed
It is named according to the initial creation time of file, saves its final revision time in the end of file position of each slice file.
Wherein, each slice file includes that a plurality of record generates the time.Optionally, each slice file includes a plurality of log
Record, every log recording include that a record generates the time.Also (including several logs of one section of log recording can be set
Record) it include: home record time and termination record time.
S12: it after receiving the log request of some period of extraction of server transmission, extracts and the period pair
At least one the slice file answered；
In step s 12, include the following steps:
(1) the period field is obtained.After terminal receives the acquisition log request of server transmission, from solicited message
Middle acquisition query time section, the query time section include initial time and termination time.If when the starting of the query time section
Between be greater than terminal current time, then exit execute this step S12.
(2) located slices file.Since each slice file includes that file initially sets up the time and when file is finally revised
Between, i.e., each corresponding time interval of slice file.Initial time and the termination time for searching the query time section are right respectively
The time interval of which or which two slice files is answered, then the slice file extracted is the corresponding slice file or this is corresponding
All Files between two slice files.For example, terminal currently saves 100 slice files, the name of this 100 slice files
Claim to arrange according to file settling time sequence, respectively " time 1 ", " time 2 ", " time 3 " ... " time 100 ".Through searching,
If the initial time of the query time section and termination time are respectively positioned in the time interval of slice file " time 3 ", extract
It is sliced file " time 3 "；It, should if the initial time of the query time section is located in the time interval of slice file " time 3 "
The termination time of query time section be located at slice file " time 7 " time interval in, then extract 5 slice files " time 3 ",
" time 4 ", " time 5 ", " time 6 " and " time 7 ".
S13: based at least one slice file generated journal file；
It in step s 13, is the transmission rate for improving journal file, by this, at least one slice file carries out packing pressure
Contracting, generates the journal file.
Preferably due to unnecessary there are still that may preserve at least one slice file of step S12 crawl
Log recording can further reject the log recording of non-query time section.
Specifically, as shown in Fig. 2, step S13 includes following sub-step:
S131: generating the time according to record, and from this, interception meets this respectively in each file of at least one slice file
The daily record data of period, and the daily record data intercepted in each file is individually saved as into a file.
In step S131, if the period, across multiple slice files, first slice file is cut with the last one
Piece file then generates the time according to the record, from this first slice file there may be unnecessary log recording is preserved
Interception meets the daily record data of the period, and the log that will be intercepted in each file respectively in the last one slice file with this
Data individually save as a file.
That is, the quantity of documents of step S12 crawl still maintains constant after step S131 processing.
S132: the All Files being truncated to are concentrated and are compressed, the journal file is generated.
In step S132, journal file is compressed file packet, is conducive to the efficiency of transmission for improving file.
S14: the journal file is uploaded to the server.
In step S14, by specific file transport protocol, for example pass through ftp file transfer protocol for the journal file
It is uploaded to the server, can further improve the efficiency of transmission of file.
Embodiment two:
Referring to Fig. 3, Fig. 3 is the structural schematic diagram of an embodiment of log extraction element of the present invention.As shown in figure 3, should
Device includes log memory module 31, slice handling module 32, log generation module 33 and log sending module 34.
Log memory module 31, for terminal log to be carried out slice storage, wherein the attribute packet of each slice file
It includes: the file generated time.
Be sliced handling module 32, for receive server transmission extraction some period log request after,
Extract at least one slice file corresponding with the period.
Log generation module 33, for based at least one slice file generated journal file.
Log sending module 34, for the journal file to be uploaded to the server.Specifically, the work of each module of the device
It is identical as method described in embodiment one to make method, details are not described herein.
Embodiment three:
Referring to Fig. 4, Fig. 4 is the structural schematic diagram of an embodiment of log extraction system of the present invention.As shown in figure 4, should
Log extraction system includes multiple terminals 41 and server 42.
Terminal 41 includes: at least one processor；And the memory being connect at least one processor communication；Its
In, which is stored with the instruction that can be executed by least one processor, which is able to carry out as described in embodiment one
Method.
Server 42 with terminal 41 for being communicated.Server 42 and 41 communication mode of terminal are for wire communication or wirelessly
Communication.Wherein, which is stored in the corresponding storage position of the server for the journal file after obtaining the journal file
It sets, if the journal file number that the storage location saves exceeds scheduled quantity, journal file replacement is saved at first
File.For example, the journal file quantity for setting storage location preservation is up to 50, if the log that the storage location saves
The journal file is then deposited into the storage location by quantity of documents < 50；Conversely, the file that the storage location is saved at first
It is deleted, is then restored again into the journal file into the storage location.In this way, being beneficial to prevent the log of the storage of server 42
Quantity of documents is excessive, simplifies server space.
Wherein, server 42 can be remote server.One application scenarios of the log extraction system can be with are as follows: multiple ends
End 41 is distributed in user location, and the setting of server 42 excludes any terminal 41 at 42 end of server in centralized control center, staff
Technical failure problem.
It is to be illustrated to preferable implementation of the invention, but the invention is not limited to the implementation above
Example, those skilled in the art can also make various equivalent variations on the premise of without prejudice to spirit of the invention or replace
It changes, these equivalent deformations or replacement are all included in the scope defined by the claims of the present application.
Claims (10)
1. a kind of log extracting method characterized by comprising
Terminal log is subjected to slice storage, wherein the attribute of each slice file includes: the file generated time；
After receiving the log request of some period of extraction of server transmission, it is based on the file generated time, is mentioned
Take at least one slice file corresponding with the period；
Based at least one described slice file generated journal file；
The journal file is uploaded to the server.
2. log extracting method according to claim 1, which is characterized in that the file of each slice file is raw
It include: that file settling time and file finally revise the time at the time；
Extract at least one slice file corresponding with the period, comprising:
Finally revised the time based on the file settling time and the file, obtain it is corresponding with the period at least one
It is sliced file.
3. log extracting method according to claim 2, which is characterized in that based at least one described slice file generated
Journal file, comprising:
At least one described slice file is subjected to packing compression, generates the journal file.
4. log extracting method according to claim 2, which is characterized in that the slice file further comprises a plurality of note
Record generates the time；
Based at least one described slice file generated journal file, comprising:
The time is generated according to the record, when interception meets described respectively from each file of at least one slice file
Between section daily record data, and the daily record data intercepted in each file is individually saved as into a file；
The All Files being truncated to are concentrated and are compressed, the journal file is generated.
5. log extracting method according to any one of claims 1 to 4, which is characterized in that each slice file
Size it is identical；
Before terminal log is carried out slice storage, comprising:
The file size of the slice file is set by the server to the terminal.
6. a kind of log extraction element characterized by comprising
Log memory module, for terminal log to be carried out slice storage, wherein the attribute of each slice file includes: file
Generate the time；
It is sliced handling module, for being based on institute after receiving the log request of some period of extraction of server transmission
The file generated time is stated, at least one slice file corresponding with the period is extracted；
Log generation module, for based at least one described slice file generated journal file；And
Log sending module, for the journal file to be uploaded to the server.
7. a kind of terminal characterized by comprising
At least one processor；And
The memory being connect at least one described processor communication；Wherein,
The memory is stored with the instruction that can be executed by least one described processor, and described instruction is able to carry out right such as and wants
Seek 1 to 5 described in any item methods.
8. a kind of server, which is characterized in that for extracting log from terminal as claimed in claim 7.
9. server according to claim 8, which is characterized in that after the server is used to obtain the journal file,
The journal file is stored in the corresponding storage location of the server, if the journal file that the storage location saves
Number exceeds scheduled quantity, and the journal file is replaced to the file saved at first.
10. a kind of log extraction system characterized by comprising
Terminal as claimed in claim 7；With
Server as claimed in claim 8 or 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910505732.9A CN110362548B (en) | 2019-06-12 | 2019-06-12 | Log extraction method and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910505732.9A CN110362548B (en) | 2019-06-12 | 2019-06-12 | Log extraction method and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110362548A true CN110362548A (en) | 2019-10-22 |
| CN110362548B CN110362548B (en) | 2023-11-14 |
Family
ID=68215744
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910505732.9A Active CN110362548B (en) | 2019-06-12 | 2019-06-12 | Log extraction method and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110362548B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113055455A (en) * | 2021-03-09 | 2021-06-29 | 中国联合网络通信集团有限公司 | File uploading method and equipment |
| WO2023279900A1 (en) * | 2021-07-05 | 2023-01-12 | 北京字节跳动网络技术有限公司 | Log file sending method and apparatus, and client |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180018210A1 (en) * | 2011-07-27 | 2018-01-18 | International Business Machines Corporation | Unified logs and device statistics |
| CN108011966A (en) * | 2017-12-14 | 2018-05-08 | 广东金赋科技股份有限公司 | The optimization method that a kind of self-aided terminal log compression uploads |
| CN108737549A (en) * | 2018-05-25 | 2018-11-02 | 江苏联盟信息工程有限公司 | A kind of log analysis method and device of big data quantity |
-
2019
- 2019-06-12 CN CN201910505732.9A patent/CN110362548B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180018210A1 (en) * | 2011-07-27 | 2018-01-18 | International Business Machines Corporation | Unified logs and device statistics |
| CN108011966A (en) * | 2017-12-14 | 2018-05-08 | 广东金赋科技股份有限公司 | The optimization method that a kind of self-aided terminal log compression uploads |
| CN108737549A (en) * | 2018-05-25 | 2018-11-02 | 江苏联盟信息工程有限公司 | A kind of log analysis method and device of big data quantity |
Non-Patent Citations (2)
| Title |
|---|
| 席景科等: "基于数据仓库的Web日志挖掘技术研究", 《计算机工程与设计》 * |
| 席景科等: "基于数据仓库的Web日志挖掘技术研究", 《计算机工程与设计》, no. 24, 31 December 2007 (2007-12-31) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113055455A (en) * | 2021-03-09 | 2021-06-29 | 中国联合网络通信集团有限公司 | File uploading method and equipment |
| WO2023279900A1 (en) * | 2021-07-05 | 2023-01-12 | 北京字节跳动网络技术有限公司 | Log file sending method and apparatus, and client |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110362548B (en) | 2023-11-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112507029B (en) | Data processing system and data real-time processing method | |
| CN109918349A (en) | Log processing method, device, storage medium and electronic device | |
| CN103095819A (en) | Data information pushing method and data information pushing system | |
| CN106897159A (en) | A kind of system and method for gathering Data of Automatic Weather | |
| CN105512201A (en) | Data collection and processing method and device | |
| CN109656999A (en) | Method of data synchronization, equipment, storage medium and the device of big data quantity | |
| CN110704400A (en) | Real-time data synchronization method and device and server | |
| CN110362548A (en) | Log extracting method and relevant device | |
| CN110990482A (en) | Data synchronization method and device between asynchronous databases | |
| CN110858192A (en) | Log query method and system, log checking system and query terminal | |
| CN104378234A (en) | Cross-data-center data transmission processing method and system | |
| CN106407078A (en) | An information interaction-based client performance monitoring device and method | |
| CN110309021A (en) | A kind of backup of Python database cross platform and restoration methods | |
| CN110019539A (en) | A kind of method and apparatus that the data of data warehouse are synchronous | |
| CN107870982A (en) | Data processing method, system and computer-readable recording medium | |
| CN104750814B (en) | The automatic storage method of polynary heterogeneous data flow based on multisensor | |
| CN202268912U (en) | Data management system based on cloud computation structure | |
| EP2690560A2 (en) | Method of benchmarking the behaviour of a replacement information system with the old system | |
| CN109783571A (en) | Data processing method, device, computer equipment and the storage medium of isolation environment | |
| US9742832B2 (en) | Transmission apparatus, transmission method, computer-readable storage medium storing transmission program, and relay system | |
| CN113918436A (en) | Log processing method and device | |
| CN107330089B (en) | Cross-network structured data collection system | |
| CN110555077A (en) | automatic question bank synchronization method and device | |
| CN110515955A (en) | Storage, querying method, system, electronic equipment and the storage medium of data | |
| CN109992573A (en) | A kind of method and system realizing HDFS file occupancy and monitoring automatically |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 518000 401, building B, No. 2, lanshui Industrial Zone, Longxin community, Baolong street, Longgang District, Shenzhen, Guangdong Province Applicant after: Ocdop Ltd. Address before: 518000 room 602, 6 / F, Langfeng building, No. 2 Kefa Road, Yuehai street, Nanshan District, Shenzhen, Guangdong Applicant before: SHENZHEN OCTOPUS TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230831 Address after: 518000 401, building B, No. 2, lanshui Industrial Zone, Longxin community, Baolong street, Longgang District, Shenzhen, Guangdong Province Applicant after: Shenzhen Shenghui Technology Co.,Ltd. Address before: 518000 401, building B, No. 2, lanshui Industrial Zone, Longxin community, Baolong street, Longgang District, Shenzhen, Guangdong Province Applicant before: Ocdop Ltd. |
|
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 518000 401, building 1, nantaiyun chuanggu Park, Southeast of the intersection of Guangming Avenue and Dongchang Road, Tangwei community, Fenghuang street, Guangming District, Shenzhen City, Guangdong Province Applicant after: Shenzhen Shenghui Technology Co.,Ltd. Address before: 518000 401, building B, No. 2, lanshui Industrial Zone, Longxin community, Baolong street, Longgang District, Shenzhen, Guangdong Province Applicant before: Shenzhen Shenghui Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |