CN110750587A - Method for processing large amount of high-speed logs of debugging equipment - Google Patents
Method for processing large amount of high-speed logs of debugging equipment Download PDFInfo
- Publication number
- CN110750587A CN110750587A CN201910980171.8A CN201910980171A CN110750587A CN 110750587 A CN110750587 A CN 110750587A CN 201910980171 A CN201910980171 A CN 201910980171A CN 110750587 A CN110750587 A CN 110750587A
- Authority
- CN
- China
- Prior art keywords
- log
- cache block
- read
- cache
- log file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method for processing a large number of high-speed logs of debugging equipment, which comprises the following steps: starting a virtual mode of a windows list function, and acquiring an initial number of a log file by binding a callback function of dynamically loaded data; setting a log file reading cache group; acquiring the total line number of the log file arranged in the Windows list box control according to the callback function, and determining a cache block corresponding to the log file to be read and the number in the cache block according to the total line number and the number of the stored log lines of the cache block; judging whether the number of the log to be read is in the current hit cache group, if not, extracting a unit corresponding to the log to be read according to the line and column crossing position of the log file, and if so, reading the historical record block to cover the current hit cache block according to the log number; and starting a scroll bar function to display the read log content. The invention ensures that a large number of high-speed log files are read without blockage, and the dragging of the scroll bar is smooth in the process of retrieving the log files.
Description
Technical Field
The invention relates to the field of data processing, in particular to a method for processing a large number of high-speed logs of debugging equipment.
Background
And (3) directly receiving and displaying the high-speed Log by the program, and when Log data is large, the display window is blocked and the memory is not enough, so that the system is crashed.
And the Log is imported into a database, and the database needs to be additionally added according to condition retrieval, so that the cost is high. If the number of logs meeting the conditions is too large, the problem of blockage or paging fault still exists, so that when Log record files are read at high speed, analysis is discontinuous, and operation is complex.
Disclosure of Invention
The invention provides a method for processing a large number of high-speed logs of debugging equipment, which aims to overcome the technical problem.
The invention discloses a method for processing a large number of high-speed logs of debugging equipment, which comprises the following steps:
starting a virtual mode of a windows list function, and acquiring an initial number of a log file by binding a callback function of dynamically loaded data;
setting a log file reading cache group, wherein the cache group comprises: a currently hitting cache block, at least one currently hitting cache block forward prediction cache block and at least one currently hitting cache block backward prediction cache block;
acquiring the total line number of the log file arranged in the Windows list box control according to the callback function, and determining a cache block corresponding to the log file to be read and the number in the cache block according to the total line number and the number of the stored log line number of the cache block;
judging whether the number of the log to be read is in the current hit cache group, if not, extracting a unit corresponding to the log to be read according to the line and column crossing position of the log file, and if so, reading a historical record block to cover the current hit cache block according to the log number;
and starting a scroll bar function to display the read log content.
Further, the initiating roller function displays the read log content, including:
updating the real-time total log record number to the total line number of the virtual mode list and moving the scroll bar to the last log entry in the refreshing mode;
and in the static mode, the real-time total log record number of the virtual mode list is not updated.
Further, after the setting the log file to read the cache group, the method further includes:
and setting the number of the log entries stored in each cache block according to the maximum resolution.
Further, the setting the number of log entries stored in each cache block according to the maximum resolution includes:
using a formula
Calculating the number of the displayed journal entries, wherein LN is the number of the displayed journal entries, the resolution is L x W, L is the length, W represents the width, the size of the display screen is A x B mm, nxThe number of logs between corresponding stripes defined by using a resolution test card with a certain interval, and b is a correction value and floats with the number of the heads.
Further, after the starting of the scroll bar function displays the read log content, the method further includes:
setting a dynamically increased log information structure list;
and arranging the latest received log files into a data structure, and adding the latest received log files to the tail part of the dynamic list one by one.
The invention ensures that a large number of high-speed log files are read without blockage, and the dragging of the scroll bar is smooth in the process of retrieving the log files, thereby greatly improving the working efficiency and reducing the debugging cost.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow chart of a method for processing a large number of high-speed logs of a debugging device of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a flowchart of a method for processing a large number of high-speed logs of a debugging device of the present invention, where the method of this embodiment includes:
starting a virtual mode of a windows list function, and acquiring an initial number of a log file by binding a callback function of dynamically loaded data;
setting a log file reading cache group, wherein the cache group comprises: a currently hitting cache block, at least one currently hitting cache block forward prediction cache block and at least one currently hitting cache block backward prediction cache block;
acquiring the total line number of the log file arranged in the Windows list box control according to the callback function, and determining a cache block corresponding to the log file to be read and the number in the cache block according to the total line number and the number of the stored log line number of the cache block;
judging whether the number of the log to be read is in the current hit cache group, if not, extracting a unit corresponding to the log to be read according to the line and column crossing position of the log file, and if so, reading a historical record block to cover the current hit cache block according to the log number;
and starting a scroll bar function to display the read log content.
Particularly, long-term tests are needed in the research and development of new automobiles, and the running effect of the vehicle-mounted module needs to be known according to various road conditions and conditions. The automobile bus module can push the internal operation effect of the program to the outside (1MByte/s) in real time at a high speed in the driving process. In this embodiment, a log of an operation condition of a vehicle-mounted module of an automobile is read, first, a virtual mode of a windows list function is started, and a start number of a log file is obtained by binding a callback function of dynamically loaded data. For example, the log file of the vehicle-mounted module has a starting number of 0-9999. The set log file read cache set comprises three cache blocks: a currently hitting cache block, a previous cache block and a next cache block of the currently hitting cache block. Every 1000 logs are stored in a cache block, the row number of a log file to be read is 4587, the ratio of the row number to 1000 is +1, the cache block required by the log file positioning is a fifth cache block, and the number in the cache block is determined to be 587 according to the remainder 587 obtained by the row number/1000, namely the designated log number. And if the cache block corresponding to the current hit cache group is the seventh cache block, the eighth cache block and the ninth cache block. And extracting a fifth cache block, a fourth cache block and a sixth cache block according to the row-column crossing position of the log file. If the cache block corresponding to the currently hit cache group is the third cache block, the fourth cache block, and the fifth cache block, the fifth cache block corresponding to the reading number 587 covers the currently hit cache block. And updating the current hit cache block, the forward prediction cache block and the backward prediction cache block, extracting the pre-judgment result from the disk, and updating the current cache block list. Since the center of the display position has been moved by the user, the prediction result should be updated accordingly.
In the Windows list box control, each content to be displayed is a unit content, the Windows list box control can only display dozens of rows of logs currently, and the next operation of a user has high probability that other log contents are viewed from the current display position forwards or backwards. The processing is to pre-judge the user operation in advance and introduce the pre-judged result into the memory buffer area.
Further, the initiating roller function displays the read log content, including:
updating the real-time total log record number to the total line number of the virtual mode list and moving the scroll bar to the last log entry in the refreshing mode;
and in the static mode, the real-time total log record number of the virtual mode list is not updated.
Specifically, the device to be tested can push the log to the monitoring device at a high speed in real time: the refreshing function is as follows: and displaying the latest log content in real time, automatically scrolling the scroll bar to the lowest position, and simultaneously counting the log pushing speed. A static mode: and monitoring the push log of the background receiving equipment, but the Windows list control does not display the latest result and only displays the current user viewing position.
Further, after the setting the log file to read the cache group, the method further includes:
and setting the number of the log entries stored in each cache block according to the maximum resolution.
Further, the setting the number of log entries stored in each cache block according to the maximum resolution includes:
using a formula
Calculating the number of the displayed journal entries, wherein LN is the number of the displayed journal entries, the resolution is L x W, L is the length, W represents the width, the size of the display screen is A x B mm, nxThe number of logs between corresponding stripes defined by using a resolution test card with a certain interval, and b is a correction value and floats with the number of the heads. The number of the display log entries calculated by the formula can ensure that the display contents are completely covered on the corresponding screen. By way of example, areThe resolution test card displays 3 logs, the size of a computer display screen is 28 inches, and the display resolution is 1920 multiplied by 1200. The log 247 bar may be displayed in a full screen condition. The size of the computer display screen is 14 inches, and the display resolution is 1024 multiplied by 768. The log 72 bar may be displayed in a full screen condition. The size of the computer display screen is 20 inches, and the display resolution is 1920 multiplied by 1200. The log 176 bar may be displayed in a full screen condition. The number of the gauge heads is adjusted according to the resolution, and the gauge heads occupy 10 bars at most.
Further, after the starting of the scroll bar function displays the read log content, the method further includes:
setting a dynamically increased log information structure list;
and arranging the latest received log files into a data structure, and adding the latest received log files to the tail part of the dynamic list one by one.
Specifically, the total number of elements contained in the storage linked list after the log file is added is counted, and if the total number is equal to the number of the logs stored in one buffer block, the storage linked list is directly stored in a computer disk in a serialization mode. The latest received log files can be used as a set and uniformly introduced into a cache block in a cache list, and simultaneously, log entry information with the same quantity of serialized storage logs is deleted from the head of a dynamic linked list. So as to better release the memory occupied by the stored log.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (5)
1. A method for processing a large number of high-speed logs of a debugging device, comprising:
starting a virtual mode of a windows list function, and acquiring an initial number of a log file by binding a callback function of dynamically loaded data;
setting a log file reading cache group, wherein the cache group comprises: a currently hitting cache block, at least one currently hitting cache block forward prediction cache block and at least one currently hitting cache block backward prediction cache block;
acquiring the total line number of the log file arranged in the Windows list box control according to the callback function, and determining a cache block corresponding to the log file to be read and the number in the cache block according to the total line number and the number of the stored log line number of the cache block;
judging whether the number of the log to be read is in the current hit cache group, if not, extracting a unit corresponding to the log to be read according to the line and column crossing position of the log file, and if so, reading a historical record block to cover the current hit cache block according to the log number;
and starting a scroll bar function to display the read log content.
2. The method of claim 1, wherein the initiating a scroll wheel function displays the read log content, comprising:
updating the real-time total log record number to the total line number of the virtual mode list and moving the scroll bar to the last log entry in the refreshing mode;
and in the static mode, the real-time total log record number of the virtual mode list is not updated.
3. The method of claim 1, wherein after setting the log file to read the cache set, further comprising:
and setting the number of the log entries stored in each cache block according to the maximum resolution.
4. The method of claim 3, wherein setting the number of log entries stored per cache block according to a maximum resolution comprises:
using a formula
Calculating the number of the displayed journal entries, wherein LN is the number of the displayed journal entries, the resolution is L x W, L is the length, W represents the width, the size of the display screen is A x B mm, nxThe number of logs between corresponding stripes defined by using a resolution test card with a certain interval, and b is a correction value and floats with the number of the heads.
5. The method of claim 1, wherein after the initiating the scroll bar function to display the read log content, further comprising:
setting a dynamically increased log information structure list;
and arranging the latest received log files into a data structure, and adding the latest received log files to the tail part of the dynamic list one by one.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910980171.8A CN110750587B (en) | 2019-10-15 | 2019-10-15 | Method for processing large amount of high-speed logs of debugging equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910980171.8A CN110750587B (en) | 2019-10-15 | 2019-10-15 | Method for processing large amount of high-speed logs of debugging equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110750587A true CN110750587A (en) | 2020-02-04 |
| CN110750587B CN110750587B (en) | 2023-04-25 |
Family
ID=69278448
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910980171.8A Active CN110750587B (en) | 2019-10-15 | 2019-10-15 | Method for processing large amount of high-speed logs of debugging equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110750587B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113741410A (en) * | 2021-11-08 | 2021-12-03 | 西安热工研究院有限公司 | Distributed control system debugging data refreshing method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030182401A1 (en) * | 2002-03-25 | 2003-09-25 | Alps System Integration Co., Ltd. | URL information sharing system using proxy cache of proxy log |
| US20080133615A1 (en) * | 2006-12-04 | 2008-06-05 | Microsoft Corporation | Multi-level read caching for multiplexed transactional logging |
| CN101639848A (en) * | 2009-06-01 | 2010-02-03 | 北京四维图新科技股份有限公司 | Spatial data engine and method applying management spatial data thereof |
| CN101819577A (en) * | 2009-01-08 | 2010-09-01 | 国际商业机器公司 | Method, system and apparatus for maintaining file system client directory caches |
| CN102760161A (en) * | 2012-06-12 | 2012-10-31 | 天津神舟通用数据技术有限公司 | Log organization structure clustered based on transaction aggregation and method for realizing corresponding recovery protocol thereof |
| CN103914485A (en) * | 2013-01-07 | 2014-07-09 | 上海宝信软件股份有限公司 | System and method for remotely collecting, retrieving and displaying application system logs |
| CN107733863A (en) * | 2017-09-07 | 2018-02-23 | 济南双瑞软件有限公司 | Daily record adjustment method and device under a kind of distributed hadoop environment |
-
2019
- 2019-10-15 CN CN201910980171.8A patent/CN110750587B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030182401A1 (en) * | 2002-03-25 | 2003-09-25 | Alps System Integration Co., Ltd. | URL information sharing system using proxy cache of proxy log |
| US20080133615A1 (en) * | 2006-12-04 | 2008-06-05 | Microsoft Corporation | Multi-level read caching for multiplexed transactional logging |
| CN101819577A (en) * | 2009-01-08 | 2010-09-01 | 国际商业机器公司 | Method, system and apparatus for maintaining file system client directory caches |
| CN101639848A (en) * | 2009-06-01 | 2010-02-03 | 北京四维图新科技股份有限公司 | Spatial data engine and method applying management spatial data thereof |
| CN102760161A (en) * | 2012-06-12 | 2012-10-31 | 天津神舟通用数据技术有限公司 | Log organization structure clustered based on transaction aggregation and method for realizing corresponding recovery protocol thereof |
| CN103914485A (en) * | 2013-01-07 | 2014-07-09 | 上海宝信软件股份有限公司 | System and method for remotely collecting, retrieving and displaying application system logs |
| CN107733863A (en) * | 2017-09-07 | 2018-02-23 | 济南双瑞软件有限公司 | Daily record adjustment method and device under a kind of distributed hadoop environment |
Non-Patent Citations (2)
| Title |
|---|
| 游理通;王振杰;黄林鹏;: "一个基于日志结构的非易失性内存键值存储系统" * |
| 范定国,贺硕,段富: "指导网络缓存协议模拟程序参数调整过程的新手段――理想命中率" * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113741410A (en) * | 2021-11-08 | 2021-12-03 | 西安热工研究院有限公司 | Distributed control system debugging data refreshing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110750587B (en) | 2023-04-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107526631B (en) | Task monitoring method, device, equipment and medium | |
| US20040080511A1 (en) | Waveform drawing system and method | |
| CN111694341A (en) | Fault data storage method and device, vehicle-mounted equipment and storage medium | |
| CN103425765A (en) | Method and device for extracting webpage text and method and system for webpage preview | |
| CN109344044B (en) | Method and system for calculating rendering time of first screen of page | |
| CN110750587B (en) | Method for processing large amount of high-speed logs of debugging equipment | |
| US10198147B2 (en) | Object management and visualization using a computing device | |
| CN111125591A (en) | Statistical method, device, terminal and storage medium of exposure information | |
| CN112100036B (en) | Page performance monitoring method and system based on PaaS front-end engine | |
| CN102722456B (en) | Flash memory device and data writing method thereof | |
| CN1311338C (en) | Method for reducing holding of internal memory in using table | |
| US8718978B2 (en) | Performance logging framework | |
| CN111259270A (en) | Weather reminding method, equipment and computer readable medium | |
| CN115629695A (en) | Application page display method, device, equipment and storage medium | |
| US11675536B2 (en) | Intelligent scheduling for garbage collection | |
| CN115421691A (en) | Data rolling method and device based on virtual list and computer equipment | |
| CN114036431A (en) | Page cache skipping method and device, electronic equipment and storage medium | |
| CN113779449A (en) | Page loading method and device, electronic equipment and storage medium | |
| CN111723052B (en) | Editing method and device for large file data | |
| CN110673910B (en) | Control method and control device for controlling popup window display in app system | |
| CN108563605B (en) | Method and device for adjusting refresh rate of electronic reader and computer storage medium | |
| CN112256996A (en) | Web-end-based page loading method | |
| CN111538705A (en) | Video thumbnail previewing method, control server and medium | |
| CN112016021B (en) | Browsing information browsing state display method and device, storage medium and electronic equipment | |
| CN111459373A (en) | Electronic book idea data display method, computing device and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |