CN114936195A - Interface system based on Spring Boot framework - Google Patents

Interface system based on Spring Boot framework Download PDF

Info

Publication number
CN114936195A
CN114936195A CN202210675742.9A CN202210675742A CN114936195A CN 114936195 A CN114936195 A CN 114936195A CN 202210675742 A CN202210675742 A CN 202210675742A CN 114936195 A CN114936195 A CN 114936195A
Authority
CN
China
Prior art keywords
data
log data
user
spring boot
log
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210675742.9A
Other languages
Chinese (zh)
Inventor
吕晓钢
王鹏飞
王通宇
李一帆
武欣桐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tianjin Optical Electrical Communication Technology Co Ltd
Original Assignee
Tianjin Optical Electrical Communication Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tianjin Optical Electrical Communication Technology Co Ltd filed Critical Tianjin Optical Electrical Communication Technology Co Ltd
Publication of CN114936195A publication Critical patent/CN114936195A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/30—Creation or generation of source code
    • G06F8/36—Software reuse
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention relates to the technical field of computer science, in particular to an interface system based on a Spring Boot framework, which comprises: the data acquisition module is used for acquiring various log data; the data processing module is used for processing various log data according to a rule set by a user; the data storage module is used for storing the processed various log data and the user set rule; and the Spring Boot framework is connected with the data storage module, acquires a corresponding script of the user set rule from the data storage module when a request is received, then performs corresponding explanation and execution, and returns corresponding content. The interface system can collect, process and store all log data, and enables a user to effectively obtain the log data according to requirements.

Description

Interface system based on Spring Boot framework
Technical Field
The invention relates to the technical field of computer science, in particular to an interface system based on a Spring Boot architecture, a log data processing method, electronic equipment and a readable storage medium.
Background
Under the background of a large data platform, a large amount of distributed log information which is easy to lose is generated in production application, such as an operation log, a login log, an information modification log and the like of an equipment information platform, the data is large in quantity and specific in structure, and how to properly store log data is a problem to be solved. When a large amount of data is processed by a traditional database, the problems of low flexibility, insufficient horizontal expansion force and poor processing performance often exist. The large number of distributed logs generated in the large data platform are not independent and exist singly, and the analysis and the processing of the logs need to be carried out comprehensively.
According to specific and application requirements of the log under the environment of the big data platform, aiming at the defects of a conventional log processing and analyzing means, a novel log processing scheme is necessary to be researched, a large amount of log data applied to the big data are collected, stored and analyzed, and a log data collecting, processing and storing mode is obtained, so that a user can effectively obtain the log data according to the requirements.
Disclosure of Invention
Solves the technical problem
In view of the above disadvantages in the prior art, the present invention provides an interface system based on a Spring Boot architecture, which can collect, process and store log data, and enable a user to effectively obtain log data according to requirements.
Technical scheme
In order to achieve the purpose, the invention is realized by the following technical scheme:
the invention provides an interface system based on Spring Boot architecture, comprising:
the data acquisition module is used for acquiring various log data;
the data processing module is used for processing various log data according to a rule set by a user;
the data storage module is used for storing the processed various log data and the user set rule;
and the Spring Boot framework is connected with the data storage module, acquires the corresponding script of the user setting rule from the data storage module when the request is received, then performs corresponding explanation execution, and returns the corresponding log data.
Further, the data acquisition module acquires various log data by using a File beat service.
Further, the data acquisition module uses Kafka to transfer various log data streams acquired by File beat service to the data processing module.
Further, the data acquisition module checks the consumption state of the log data through Kafka to verify whether the log data is collected successfully.
Further, the data processing module performs data processing on the various Log data by configuring a Log push pipeline, and a configuration rule of the Log push pipeline sets a rule for the user.
Further, the data storage module comprises an Elastic Search and a mysql database, wherein the Elastic Search is used for storing the Log data processed by the Log hash pipeline, and the mysql database is used for storing the user-set rule.
Further, the Spring Boot architecture further comprises a Spring Boot web layer, and the Spring Boot web layer is used for displaying and user-defined configuring the user setting rule.
Further, the user configures the user setting rule in a customized manner on the Spring boot web layer by means of a regular expression.
Based on the same invention concept, the invention also provides a log data processing method based on the Spring Boot architecture, and the method uses any one of the interface systems and comprises the following steps:
s1, acquiring various log data;
s2, processing the various log data according to the user set rules;
s3, storing the processed various log data and the user set rules;
and S4, receiving the user request, acquiring the corresponding script of the user setting rule, then performing corresponding interpretation execution, and returning the corresponding log data.
Based on the same inventive concept, the present invention also provides an electronic device, which includes a processor and a memory, where the memory stores a computer program, and when the computer program is executed by the processor, the electronic device implements the method described above.
Based on the same inventive concept, the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method.
Advantageous effects
According to the technical scheme provided by the invention, the collected log data is stored in the non-relational database in a regularized manner through the user-defined design rule, and the user-defined rule information is correspondingly stored in the relational database, so that the user data can be conveniently inquired and acquired in the later period, the huge log information generated by a large data platform can be processed and analyzed through an interface, the management efficiency of the platform log data is effectively improved, and the problems of complexity and time consumption of manual management are solved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 is a schematic diagram of an interface system based on a Spring Boot architecture according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a processing flow of log data by an interface system based on a Spring Boot architecture according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, an embodiment of the present invention provides an interface system based on a Spring Boot architecture, including:
the data acquisition module is used for acquiring various log data;
the data processing module is used for processing various log data according to a rule set by a user;
the data storage module is used for storing the processed various log data and the user set rule;
and the Spring Boot framework is connected with the data storage module, acquires the corresponding script of the user setting rule from the data storage module when the request is received, then performs corresponding explanation execution, and returns the corresponding log data.
In this embodiment, as shown in fig. 2, the data obtaining module generally obtains various log data by using File beat service. The data acquisition module generally uses Kafka to transfer various log data streams acquired by File beat service to the data processing module. The data acquisition module also generally checks the consumption status of the log data through Kafka to verify whether the log data is successfully collected. The data processing module generally performs data processing on the various Log data by configuring a Log push pipeline, wherein a configuration rule of the Log push pipeline sets a rule for the user. The data storage module generally comprises an Elastic Search and a mysql database, wherein the Elastic Search is used for storing the Log data processed by the Log hash pipeline, and the mysql database is used for storing the user setting rule. The Spring Boot architecture generally further comprises a Spring Boot web layer, and the Spring Boot web layer is used for displaying and configuring the user setting rule in a user-defined mode. And the user generally configures the user setting rule in a self-defined manner by a regular expression on the Spring boot web layer.
Therefore, the invention designs a Log monitoring interface service based on a Spring Boot framework, the system regularly stores the collected Log data into an Elastic Search non-relational database by calling File bead, Kafka, Log stack and Elastic Search services and by user-defined design rules, and correspondingly stores the user-defined rule information into a MySql relational database, thereby facilitating the later-stage user data query and acquisition, realizing the processing and analysis of huge Log information generated by a large data platform by an interface, effectively improving the management efficiency of the platform Log data and solving the problem of complicated manual management and time consumption.
In particular, as shown in FIG. 2, the present invention relates to File bead, Kafka, Log stack and Elastic search services. The File Beat is a lightweight tool for forwarding and concentrating the Log data and is used as an agent installation on the server, and the File Beat monitors the Log File or the position specified by the user, collects the Log data and forwards the Log data to Elastic Search or Log stack for indexing. Since log stack is relatively better than fuel consumption, File beat is used for data collection.
The Kafka service is a high throughput distributed publish-subscribe messaging system that can handle all action flow data in a consumer-scale website, with Log stack as an open-source data collection engine with real-time data transfer capability. It can filter data from different sources uniformly and output to the destination according to the specification set by the developer. The Log stash collection data object is Log data. Because the log data sources are many (such as system logs, server logs and the like), and the contents are messy, the user can not observe the log data conveniently. Therefore, Log files can be collected and uniformly filtered by using the Log stash, and the Log files become contents with high readability, so that developers or operation and maintenance personnel can conveniently observe the contents, the running performance of a system/project can be effectively analyzed, and preparation work of monitoring and early warning can be well carried out. Elastic Search is a Lucene-based Search server and can also be considered a non-relational database. It provides a distributed multi-user capable full-text search engine based on RESTful web interface. According to the interface provided by the invention, the collected log data are sent to the Elasticissearch, the log data are effectively filtered through the user-defined matching rule, the data are visually displayed and analyzed in a browser, and the user-defined rule information can be stored in the mysql database, so that the user can conveniently search and the front-end intelligent output is realized.
Specifically, as shown in fig. 2, after all services are started, the Spring Boot architecture can implement log obtaining and analyzing functions, and the Spring Boot architecture is connected with a Mysql database, so that a user can view a log rule set before the user views the log rule in a web layer of the Spring Boot architecture; and then configuring the user setting rule on a web layer of the Spring Boot framework by a user according to requirements, and realizing the process in a regular expression mode.
Based on the same invention concept, the invention also provides a log data processing method based on the Spring Boot architecture, and the method uses any one of the interface systems and comprises the following steps:
s1, acquiring various log data;
s2, processing the various log data according to the user setting rules;
s3, storing the processed various log data and the user set rules;
and S4, receiving the user request, acquiring the corresponding script of the user setting rule, then performing corresponding explanation execution, and returning the corresponding content.
Based on the same invention idea, the invention further provides an electronic device, which includes a processor and a memory, where the memory stores a computer program, and when the computer program is executed by the processor, the method for processing log data based on the Spring Boot architecture is implemented.
The processor may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor (e.g., GPU), or other data Processing chip in some embodiments. The processor is typically used to control the overall operation of the electronic device. In this embodiment, the processor is configured to run a program code stored in the memory or process data, for example, run a program code of the log data processing method based on the Spring Boot architecture.
The memory includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage may be an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device. In other embodiments, the memory may also be an external storage device of the electronic device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the electronic device. Of course, the memory may also include both internal and external memory units of the electronic device. In this embodiment, the memory is generally used for storing an operation method installed in the electronic device and various types of application software, such as program codes of the log data processing method based on the Spring Boot architecture. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.
Based on the same idea, the present invention further provides a readable storage medium, where a computer program is stored in the readable storage medium, and when the computer program is executed by a processor, the method for processing log data based on the Spring Boot architecture is implemented.
The invention has the advantages that the collected log data are regularly stored in the non-relational database through the user-defined design rule, and the user-defined rule information is correspondingly stored in the relational database, so that the user data can be conveniently inquired and acquired in the later period, the huge log information generated by a large data platform can be processed and analyzed by an interface, the management efficiency of the platform log data is effectively improved, and the problems of complexity and time consumption of manual management are solved.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not cause the essence of the corresponding technical solutions to depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. An interface system based on Spring Boot architecture, comprising:
the data acquisition module is used for acquiring various log data;
the data processing module is used for processing various log data according to a rule set by a user;
the data storage module is used for storing the processed various log data and the user set rule;
and the Spring Boot framework is connected with the data storage module, acquires the corresponding script of the user setting rule from the data storage module when the request is received, then performs corresponding explanation execution, and returns the corresponding log data.
2. The Spring Boot architecture-based interface system according to claim 1, wherein the data obtaining module obtains various log data by using File beat service.
3. The Spring Boot architecture based interface system according to claim 2, wherein the data obtaining module uses Kafka to transfer various log data streams obtained by File eat service to the data processing module.
4. The Spring Boot architecture-based interface system according to claim 3, wherein the data obtaining module further checks the consumption status of the log data through Kafka to verify whether the log data is successfully collected.
5. The interface system according to claim 3, wherein the data processing module performs data processing on the various Log data by configuring a Log hash pipeline, and a rule of the Log hash pipeline is set for the user.
6. The Spring Boot architecture based interface system according to claim 5, wherein the data storage module comprises an Elastic Search and a mysql database, the Elastic Search is used for storing the Log data processed by the Log hash pipeline, and the mysql database is used for storing the user setting rule.
7. The Spring Boot architecture-based interface system according to claim 1, wherein the Spring Boot architecture further comprises a Spring Boot web layer, and the Spring Boot web layer is used for displaying and user-defined configuring the user-set rule.
8. A log data processing method based on a Spring Boot architecture uses any one of the interface systems, and is characterized by comprising the following steps:
s1, acquiring various log data;
s2, processing the various log data according to the user setting rules;
s3, storing the processed various log data and the user set rules;
and S4, receiving the user request, acquiring the corresponding script of the user setting rule, then performing corresponding explanation execution, and returning the corresponding log data.
9. An electronic device, comprising a processor and a memory, the memory having stored thereon a computer program which, when executed by the processor, implements the method of claim 8.
10. A readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of claim 9.
CN202210675742.9A 2022-03-08 2022-06-15 Interface system based on Spring Boot framework Pending CN114936195A (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210227324 2022-03-08
CN2022102273243 2022-03-08

Publications (1)

Publication Number Publication Date
CN114936195A true CN114936195A (en) 2022-08-23

Family

ID=82867018

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210675742.9A Pending CN114936195A (en) 2022-03-08 2022-06-15 Interface system based on Spring Boot framework

Country Status (1)

Country Link
CN (1) CN114936195A (en)

Similar Documents

Publication Publication Date Title
CN108694195B (en) Management method and system of distributed data warehouse
CN102200979A (en) Distributed parallel information retrieval system and distributed parallel information retrieval method
CN106354817B (en) Log processing method and device
US20160283479A1 (en) Method and system for manipulating data
CN111046011A (en) Log collection method, system, node, electronic device and readable storage medium
CN109992569A (en) Cluster log feature extracting method, device and storage medium
CN112182043A (en) Log data query method, device, equipment and storage medium
CN111813774B (en) Method for monitoring and acquiring traceability information based on sysdig system
CN113656673A (en) Master-slave distributed content crawling robot for advertisement delivery
EP3139290A1 (en) Method and system for merging data
CN102055780A (en) System and method for testing disk array
CN112667873A (en) Crawler system and method suitable for general data acquisition of most websites
US20200380037A1 (en) Information Retrieval Precision Evaluation Method, System and Device and Computer-Readable Storage Medium
CN110888909B (en) Data statistical processing method and device for evaluation content
CN114936195A (en) Interface system based on Spring Boot framework
CN112579552A (en) Log storage and calling method, device and system
CN110881030A (en) Logstack-based method and device for recording operation logs of web service administrator
CN110011845A (en) Log collection method and system
CN111376255A (en) Robot data acquisition method and device and terminal equipment
CN115809226A (en) Data docking method and system for intelligent management system of coal washery
CN112685370B (en) Log collection method, device, equipment and medium
WO2022261249A1 (en) Distributed task assignment, distributed alerts and supression management, and artifact life tracking storage in a cluster computing system
CN116628042A (en) Data processing method, device, equipment and medium
CN102446131A (en) Positioning method and system of computing node and management node
CN103106366A (en) Dynamic maintenance method of sample database based on cloud

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination