CN110750587B - Method for processing large amount of high-speed logs of debugging equipment - Google Patents
Method for processing large amount of high-speed logs of debugging equipment Download PDFInfo
- Publication number
- CN110750587B CN110750587B CN201910980171.8A CN201910980171A CN110750587B CN 110750587 B CN110750587 B CN 110750587B CN 201910980171 A CN201910980171 A CN 201910980171A CN 110750587 B CN110750587 B CN 110750587B
- Authority
- CN
- China
- Prior art keywords
- log
- cache block
- read
- cache
- journal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method for processing a large number of high-speed logs of debugging equipment, which comprises the following steps: starting a virtual mode of a windows list function, and acquiring a starting number of a log file through a callback function of binding dynamic loading data; setting a log file reading cache group; obtaining the total line number of the log file arranged in the Windows list box control according to the callback function, and determining a cache block corresponding to the log file to be read and a number in the cache block according to the total line number and the cache block storage log line number; judging whether the journal number to be read is in the current hit cache group, if not, extracting a unit corresponding to the journal to be read according to the intersection position of the rows and the columns of the journal file, and if so, reading the history record block again according to the journal number to cover the current hit cache block; the scroll bar function is activated to display the read log content. The method and the device ensure that a large number of high-speed log files are read without being blocked, and a scroll bar is dragged smoothly in the process of retrieving the log files.
Description
Technical Field
The invention relates to the field of data processing, in particular to a method for processing a large number of high-speed logs of debugging equipment.
Background
The program directly receives and displays the high-speed Log, when Log data is large, a display window is blocked, and the memory is not used enough, so that the system is crashed.
The Log is imported into the database, and the database is additionally added according to condition retrieval, so that the cost is high. If the number of logs meeting the condition is too large, the problem of blocking or paging faults still exists, so that analysis is discontinuous and operation is complex when Log record files are read at high speed.
Disclosure of Invention
The invention provides a method for processing a large number of high-speed logs of debugging equipment, which aims to overcome the technical problems.
The invention discloses a method for processing a large number of high-speed logs of debugging equipment, which comprises the following steps:
starting a virtual mode of a windows list function, and acquiring a starting number of a log file through a callback function of binding dynamic loading data;
setting a log file reading cache set, wherein the cache set comprises: a current hit cache block, at least one current hit cache block forward predicted cache block, and at least one current hit cache block backward predicted cache block;
obtaining the total line number of the log file arranged in the Windows list box control according to the callback function, and determining a cache block corresponding to the log file to be read and a number in the cache block according to the total line number and the line number of the cache block storage log;
judging whether the journal number to be read is in the current hit cache group, if not, extracting a unit corresponding to the journal to be read according to the intersection position of the rows and the columns of the journal file, and if so, re-reading the history record block according to the journal number to cover the current hit cache block;
the scroll bar function is activated to display the read log content.
Further, the starting roller function displays the read log content, including:
a refreshing mode, namely updating the total number of the real-time log records to the total number of lines of the virtual mode list, and simultaneously moving the scroll bar to the last log entry;
still mode, the real-time total number of log records of the virtual mode list is not updated.
Further, after the log file is set to read the cache set, the method further includes:
the number of log entries stored per cache block is set according to the maximum resolution.
Further, the setting the number of log entries stored in each cache block according to the maximum resolution includes:
using the formula
Calculating the number of displayed log entries, wherein LN is the number of displayed log entries, the resolution is L x W, L is long, W is the width, and the display screen size is A x B mm, n x The number of logs between corresponding stripes defined for using a resolution test card having a certain interval, b is a correction value, and is floated with the number of headers.
Further, after the scroll bar function is started to display the read log content, the method further comprises:
setting a dynamically-growing log information structure list;
and finishing the latest received log files into a data structure, and adding the latest received log files to the tail of the dynamic list piece by piece.
The invention ensures that a large number of high-speed log files are read without being blocked, and a scroll bar is dragged smoothly in the process of retrieving the log files, thereby greatly improving the working efficiency and reducing the debugging cost.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it will be obvious that the drawings in the following description are some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort to a person skilled in the art.
FIG. 1 is a flow chart of a method of high-speed log processing of a debugging device of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
FIG. 1 is a flow chart of a method for processing a large amount of high-speed logs of a debugging device according to the present invention, and the method in this embodiment includes:
starting a virtual mode of a windows list function, and acquiring a starting number of a log file through a callback function of binding dynamic loading data;
setting a log file reading cache set, wherein the cache set comprises: a current hit cache block, at least one current hit cache block forward predicted cache block, and at least one current hit cache block backward predicted cache block;
obtaining the total line number of the log file arranged in the Windows list box control according to the callback function, and determining a cache block corresponding to the log file to be read and a number in the cache block according to the total line number and the line number of the cache block storage log;
judging whether the journal number to be read is in the current hit cache group, if not, extracting a unit corresponding to the journal to be read according to the intersection position of the rows and the columns of the journal file, and if so, re-reading the history record block according to the journal number to cover the current hit cache block;
the scroll bar function is activated to display the read log content.
Specifically, long-term test is required in the research and development of new automobiles, and the operation effect of the vehicle-mounted module is required to be known according to various road conditions and conditions. The bus module can push the running effect of the program inside to the outside (about 1 MByte/s) through the network at high speed in real time in the running process. In this embodiment, the log of the running condition of the vehicle-mounted module of the automobile is read, firstly, a virtual mode of a windows list function is started, and the starting number of the log file is obtained through a callback function of binding dynamic loading data. For example, the log file of the on-board module starts with a number of 0-9999. The log file reading cache group is set to comprise three cache blocks: a current hit cache block, a previous cache block and a subsequent cache block to the current hit cache block. Every 1000 logs are stored in one cache block, the line number of the log file to be read is 4587, the ratio of the line number to 1000 is +1, the cache block required for positioning the log file is a fifth cache block, and the number in the cache block is 587 according to the remainder 587 obtained by the line number/1000, namely the indicated log number. And if the cache block corresponding to the current hit cache group is the seventh cache block, the eighth cache block and the ninth cache block. And extracting the fifth cache block, the fourth cache block and the sixth cache block according to the row-column crossing position of the log file. If the cache blocks corresponding to the current hit cache group are the third cache block, the fourth cache block and the fifth cache block, the fifth cache block corresponding to the read number 587 covers the current hit cache block. And updating the current hit cache block, the forward prediction cache block and the backward prediction cache block, extracting the pre-judgment result from the disk, and updating the current cache block list. Since the display position center has been moved by the user, the pre-determination result should be updated accordingly.
In the Windows list box control, each content to be displayed is a unit content, the Windows list box control can only display tens of lines of logs currently, and the next operation of the user is to look at other log contents forwards or backwards from the current display position with high probability. The processing is to pre-judge the user operation in advance and import the pre-judging result into the memory buffer area.
Further, the starting roller function displays the read log content, including:
a refreshing mode, namely updating the total number of the real-time log records to the total number of lines of the virtual mode list, and simultaneously moving the scroll bar to the last log entry;
still mode, the real-time total number of log records of the virtual mode list is not updated.
Specifically, the device to be tested can push logs to the monitoring device at a high speed in real time: refresh function: and displaying the latest log content in real time, automatically scrolling the scroll bar to the lowest position, and simultaneously counting the log pushing speed. Stationary mode: the monitoring background receiving device pushes the log, but the Windows list control does not display the latest result and only displays the current user viewing position.
Further, after the log file is set to read the cache set, the method further includes:
the number of log entries stored per cache block is set according to the maximum resolution.
Further, the setting the number of log entries stored in each cache block according to the maximum resolution includes:
using the formula
Calculating the number of displayed log entries, wherein LN is the number of displayed log entries, the resolution is L x W, L is long, W is the width, and the display screen size is A x B mm, n x The number of logs between corresponding stripes defined for using a resolution test card having a certain interval, b is a correction value, and is floated with the number of headers. The number of the display log entries calculated by the formula can ensure that the display content is completely covered on the corresponding screen. For example, the resolution test card displays 3 logs, the computer display screen size is 28 inches, and the display resolution is 1920 x 1200. Log 247 bars may be displayed in a full screen condition. The computer display screen size was 14 inches and the display resolution was 1024 x 768. The log 72 bars may be displayed in a full screen condition. The computer display screen size was 20 inches and the display resolution was 1920 x 1200. The log 176 pieces may be displayed in a full screen condition. The number of the header is adjusted according to the resolution, and the maximum number of the header is 10.
Further, after the scroll bar function is started to display the read log content, the method further comprises:
setting a dynamically-growing log information structure list;
and finishing the latest received log files into a data structure, and adding the latest received log files to the tail of the dynamic list piece by piece.
Specifically, the total number of elements contained in the storage linked list after the log file is added is counted, and if the total number is equal to the number of the logs stored in one buffer block, the storage linked list is directly stored in a computer disk in a serialization mode. The latest received log files can be used as a set to be uniformly imported into one cache block in the cache list, and log entry information with the same number of the serialized storage logs is deleted from the head of the dynamic linked list. So as to better release the memory occupied by the already stored disk logs.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention.
Claims (5)
1. A method for processing a plurality of high-speed logs of a debugging device, comprising:
starting a virtual mode of a windows list function, and acquiring a starting number of a log file through a callback function of binding dynamic loading data;
setting a log file reading cache set, wherein the cache set comprises: a current hit cache block, at least one current hit cache block forward predicted cache block, and at least one current hit cache block backward predicted cache block;
obtaining the total line number of the log file arranged in the Windows list box control according to the callback function, and determining a cache block corresponding to the log file to be read and a number in the cache block according to the total line number and the line number of the cache block storage log;
judging whether the journal number to be read is in the current hit cache group, if not, extracting a unit corresponding to the journal to be read according to the intersection position of the rows and the columns of the journal file, and if so, re-reading the history record block according to the journal number to cover the current hit cache block;
the scroll bar function is activated to display the read log content.
2. The method of claim 1, wherein the enabling the scrollbar function to display the read log content comprises:
a refreshing mode, namely updating the total number of the real-time log records to the total number of lines of the virtual mode list, and simultaneously moving the scroll bar to the last log entry;
still mode, the real-time total number of log records of the virtual mode list is not updated.
3. The method of claim 1, wherein after the log file is set to read the cache set, further comprising:
and calculating the number of log entries stored in each cache block according to the maximum resolution.
4. A method according to claim 3, wherein said calculating the number of log entries stored per cache block based on the maximum resolution comprises:
using the formula
Calculating the number of displayed log entries, wherein LN is the number of displayed log entries, the resolution is L x W, L is long, W is the width, and the display screen size is A x B mm, n x The number of logs between corresponding stripes defined for using a resolution test card with a certain spacing, b is the correctionThe value, which is floating with the number of headers.
5. The method of claim 1, wherein after the enabling the scroll bar function to display the read log content, further comprising:
setting a dynamically-growing log information structure list;
and sorting the latest received log files into a data structure, and adding the latest received log files to the tail part of the dynamic list one by one.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910980171.8A CN110750587B (en) | 2019-10-15 | 2019-10-15 | Method for processing large amount of high-speed logs of debugging equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910980171.8A CN110750587B (en) | 2019-10-15 | 2019-10-15 | Method for processing large amount of high-speed logs of debugging equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110750587A CN110750587A (en) | 2020-02-04 |
| CN110750587B true CN110750587B (en) | 2023-04-25 |
Family
ID=69278448
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910980171.8A Active CN110750587B (en) | 2019-10-15 | 2019-10-15 | Method for processing large amount of high-speed logs of debugging equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110750587B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113741410B (en) * | 2021-11-08 | 2022-02-15 | 西安热工研究院有限公司 | Distributed control system debugging data refreshing method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101639848A (en) * | 2009-06-01 | 2010-02-03 | 北京四维图新科技股份有限公司 | Spatial data engine and method applying management spatial data thereof |
| CN101819577A (en) * | 2009-01-08 | 2010-09-01 | 国际商业机器公司 | Method, system and apparatus for maintaining file system client directory caches |
| CN102760161A (en) * | 2012-06-12 | 2012-10-31 | 天津神舟通用数据技术有限公司 | Log organization structure clustered based on transaction aggregation and method for realizing corresponding recovery protocol thereof |
| CN103914485A (en) * | 2013-01-07 | 2014-07-09 | 上海宝信软件股份有限公司 | System and method for remotely collecting, retrieving and displaying application system logs |
| CN107733863A (en) * | 2017-09-07 | 2018-02-23 | 济南双瑞软件有限公司 | Daily record adjustment method and device under a kind of distributed hadoop environment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2003281021A (en) * | 2002-03-25 | 2003-10-03 | Alps System Integration Co Ltd | Url information sharing system using proxy cache or proxy log |
| US8074027B2 (en) * | 2006-12-04 | 2011-12-06 | Microsoft Corporation | Multi-level read caching for multiplexed transactional logging |
-
2019
- 2019-10-15 CN CN201910980171.8A patent/CN110750587B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101819577A (en) * | 2009-01-08 | 2010-09-01 | 国际商业机器公司 | Method, system and apparatus for maintaining file system client directory caches |
| CN101639848A (en) * | 2009-06-01 | 2010-02-03 | 北京四维图新科技股份有限公司 | Spatial data engine and method applying management spatial data thereof |
| CN102760161A (en) * | 2012-06-12 | 2012-10-31 | 天津神舟通用数据技术有限公司 | Log organization structure clustered based on transaction aggregation and method for realizing corresponding recovery protocol thereof |
| CN103914485A (en) * | 2013-01-07 | 2014-07-09 | 上海宝信软件股份有限公司 | System and method for remotely collecting, retrieving and displaying application system logs |
| CN107733863A (en) * | 2017-09-07 | 2018-02-23 | 济南双瑞软件有限公司 | Daily record adjustment method and device under a kind of distributed hadoop environment |
Non-Patent Citations (2)
| Title |
|---|
| 游理通 ; 王振杰 ; 黄林鹏 ; .一个基于日志结构的非易失性内存键值存储系统.计算机研究与发展.2018,(第09期),全文. * |
| 范定国,贺硕,段富.指导网络缓存协议模拟程序参数调整过程的新手段――理想命中率.光电子技术与信息.2005,(第01期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110750587A (en) | 2020-02-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021120789A1 (en) | Data writing method and apparatus, and storage server and computer-readable storage medium | |
| CN109873904B (en) | User message state reporting processing method, device and storage medium for message pushing | |
| CN103853729A (en) | Page loading method and system | |
| CN107526631B (en) | Task monitoring method, device, equipment and medium | |
| US20040080511A1 (en) | Waveform drawing system and method | |
| WO2016141735A1 (en) | Cache data determination method and device | |
| US20180329921A1 (en) | Method and apparatus for storing time series data | |
| US20140237343A1 (en) | Method and system for optimizing rendering of data tables | |
| CN109960554B (en) | Method, device and computer storage medium for displaying reading content | |
| WO2012024142A2 (en) | Inferring that a message has been read | |
| CN103425765A (en) | Method and device for extracting webpage text and method and system for webpage preview | |
| CN110750587B (en) | Method for processing large amount of high-speed logs of debugging equipment | |
| CN112835776A (en) | Page event reproduction method, page event acquisition method, page event reproduction device and electronic equipment | |
| CN113360324B (en) | Data backup device based on distributed file data | |
| CN111259270A (en) | Weather reminding method, equipment and computer readable medium | |
| CN111125591A (en) | Statistical method, device, terminal and storage medium of exposure information | |
| CN112100036B (en) | Page performance monitoring method and system based on PaaS front-end engine | |
| CN110659280B (en) | Road blocking abnormity detection method and device, computer equipment and storage medium | |
| US8718978B2 (en) | Performance logging framework | |
| CN1740967A (en) | Method for reducing holding of internal memory in using table | |
| CN111538705B (en) | Video thumbnail preview method, control server and medium | |
| CN115629695A (en) | Application page display method, device, equipment and storage medium | |
| CN112395260B (en) | Data storage method and medium | |
| CN114036431A (en) | Page cache skipping method and device, electronic equipment and storage medium | |
| CN114169451A (en) | Behavior data classification processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |