CN109872135A - A kind of government affairs service data analysis method based on ELK - Google Patents
A kind of government affairs service data analysis method based on ELK Download PDFInfo
- Publication number
- CN109872135A CN109872135A CN201910160306.6A CN201910160306A CN109872135A CN 109872135 A CN109872135 A CN 109872135A CN 201910160306 A CN201910160306 A CN 201910160306A CN 109872135 A CN109872135 A CN 109872135A
- Authority
- CN
- China
- Prior art keywords
- business
- government affairs
- search engine
- elk
- service data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The government affairs service data analysis method based on ELK that the present invention is more particularly directed to a kind of.The government affairs service data analysis method based on ELK develops government affairs service data analytical equipment on the basis of search engine ES, the statistical model that can be supported more scenes, can be used interchangeably is established, to complete function of statistic analysis；The government affairs service data analytical equipment built-in timer, is timed each statistical model, by synchronizing traffic data in database into search engine ES；The business datum synchronized is counted using search engine ES.The government affairs service data analysis method based on ELK, the Statistic analysis models of dimension are drawn by establishing adaptation, logic complexity, multimode, multi-region, model can also interact independently of each other with model, the business datum of statistical analysis is put into search engine ES from taking out in database using timer, the statistics to each model is completed by search engine ES again, to achieve the effect that high-efficient, real-time, accurate, small to database pressure.
Description
Technical field
The present invention relates to government data analysis technical field, in particular to a kind of government affairs service data analysis based on ELK
Method.
Background technique
With popularizing for E-Government, the social activities public is continuously increased government affairs demand for services Internet-based, more
It handles affairs on the net come more working masses by mobile phone, computer, from equipment such as master terminals, passes through interconnection in society and the public
While convenient, efficient, high quality that net is enjoyed, the service of the government affairs of personalization, the service logic of government affairs service also becomes increasingly
The portfolio of complexity, government affairs service is more and more, while statistics to business datum and analysis become more difficult.
For a large amount of data and the service logic of complexity, realize that statistical analysis has become more next by database
It is more out of strength.The not high statistics of sometimes one complexity requires one minute to respond, this is for the user for using statistics
Very bad experience, at the same it is also bigger to the pressure of database.
Government affairs service business has following three features: first is that service logic is complex, entire operation flow is very long, such as
Preliminary hearing accepts, audits, examining, addressee etc..Second is that service condition is more, such as to preliminary hearing, wait accept, polishing correction, cancel
Deng.Third is that zoning level is more, such as city-level, area's grade, street, hall, the multiple dimensions of window.By to some of database
Optimization and increases index etc. the efficiency that can improve of operation is cup water for the complex statistics that government affairs service big data
Che Xin.In order to solve the function of statistic analysis that government affairs service business data volume is huge and service logic is complicated, just have to use
New technology counts to realize, cannot completely depend on database, just can guarantee the efficiency and accuracy of statistics in this way.
In this context, it has to abandon the common mode for looking into database, but select at some new data
Reason technology completes the function, and ELK technology is then a relatively good selection in numerous data processing engines.ELK is
The collector journal for the complete set that elastic company provides and the product for analyzing displaying, wherein E refers to ElasticSearch,
It is a real-time distributed search and analysis engine；L refers to Logstash, is the number with real-time Transmission ability
According to collection engine；K refers to Kibana, it provides an analysis and visual web platform for ElasticSearch, passes through
Kibana can inquire the data of the index of Elasticsearch, polymerization, the various operations such as deletion.
Based on the above situation, the government affairs service data analysis method based on ELK that the invention proposes a kind of.
Summary of the invention
In order to compensate for the shortcomings of the prior art, the present invention provides a kind of government affairs based on ELK being simple and efficient to service number
According to analysis method.
The present invention is achieved through the following technical solutions:
A kind of government affairs service data analysis method based on ELK, which comprises the following steps:
(1) the search engine ES based on Lucene is used, the analysis of government affairs service data is developed on the basis of search engine ES
Device establishes the statistical model that can be supported more scenes, can be used interchangeably, to complete function of statistic analysis；
(2) the government affairs service data analytical equipment built-in timer, is timed each statistical model, by database
Middle synchronizing traffic data is into search engine ES；
(3) business datum synchronized is counted using search engine ES.
In the step (1), statistical model is the SQL model that can find business information and the information containing zoning.
When constructing statistical model, first constructs a government affairs service business data visualization by database and show not same district grade
And the statistics menu of different conditions, the statistics menu include the main table of business of business details, record each link of the business
The zoning table of record sheet and storage zoning information；By the main table of business of the business details, the note of each link of the business is recorded
Record table and store zoning information zoning table it is interrelated, can assemble can find business information and the information containing zoning
SQL model.
The statistical model includes time statistical model, window statistical model, unit statistical model and hall statistical model.
In the step (2), the data in database are synchronized to search using data collection engine Logstash and are drawn
It holds up in ES；Specify a field as each synchrodata in the starting configuration file of data collection engine Logstash
Then timestamp needs the time interval of synchrodata using timer setting；For the first time synchrodata when can by database when
The data generated before the preceding time synchronize into entirely and current synchronization time are recorded in file；According to timer setting
Time interval will take out this hyposynchronous time as the time started, when subsynchronous below when next synchrodata
Between business datum new in this period of time is synchronized as the end time.
The dependence packet path of configuration database, statistical model road in data collection engine Logstash starting configuration file
Diameter, timer, this synchronization time record file path and timestamp field information.
Also to refer to the output position of fixed number evidence in data collection engine Logstash starting configuration file simultaneously, specifying will
Data are synchronized in configured search engine ES and specify the index name generated in search engine ES.
In the step (3), stores each business datum of the search engine in ES and all contain statistical function needs
All field informations, polymerize according to zoning coding and different service conditions be added as querying condition and obtain different zonings
Under each service condition business sum, by zoning coding and status of service field inquire to obtain every kind as querying condition
The details list of business sum, and visualize and come out after data are come out from search engine ES.
The beneficial effects of the present invention are: it is somebody's turn to do the government affairs service data analysis method based on ELK, it is multiple by establishing adaptation, logic
Miscellaneous, multimode, multi-region draw the Statistic analysis models of dimension, and model can also interact independently of each other with model, utilize timing
The business datum of statistical analysis is put into search engine ES from taking out in database by device, then is completed pair by search engine ES
The statistics of each model, to achieve the effect that high-efficient, real-time, accurate, small to database pressure.
Detailed description of the invention
Attached drawing 1 is that the present invention is based on the government affairs service data analysis method schematic diagrames of ELK.
Attached drawing 2 is the interrelated relation schematic diagram between record sheet and zoning table the present invention is based on business main table.
Attached drawing 3 is that the present invention utilizes timer synchronization data method schematic diagram.
Attached drawing 4 is that the present invention utilizes search engine ES statistical data method schematic diagram.
Specific embodiment
In order to which technical problems, technical solutions and advantages to be solved are more clearly understood, tie below
Drawings and examples are closed, the present invention will be described in detail.It should be noted that specific embodiment described herein is only
To explain the present invention, it is not intended to limit the present invention.
The government affairs service data analysis method based on ELK, comprising the following steps:
(1) the search engine ES (ElasticSearch) based on Lucene is used, is developed on the basis of search engine ES
Government affairs service data analytical equipment establishes the statistical model that can be supported more scenes, can be used interchangeably, to complete statistical analysis function
Energy；
(2) the government affairs service data analytical equipment built-in timer, is timed each statistical model, by database
Middle synchronizing traffic data is into search engine ES；
(3) business datum synchronized is counted using search engine ES.
In the step (1), statistical model is the SQL model that can find business information and the information containing zoning.
When constructing statistical model, first constructs a government affairs service business data visualization by database and show not same district grade
And the statistics menu of different conditions, the statistics menu include the main table of business of business details, record each link of the business
The zoning table of record sheet and storage zoning information；By the main table of business of the business details, the note of each link of the business is recorded
Record table and store zoning information zoning table it is interrelated, can assemble can find business information and the information containing zoning
SQL model.
The statistical model includes time statistical model, window statistical model, unit statistical model and hall statistical model.
In the step (2), the data in database are synchronized to search using data collection engine Logstash and are drawn
It holds up in ES；Specify a field as each synchrodata in the starting configuration file of data collection engine Logstash
Then timestamp needs the time interval of synchrodata using timer setting；For the first time synchrodata when can by database when
The data generated before the preceding time synchronize into entirely and current synchronization time are recorded in file；According to timer setting
Time interval will take out this hyposynchronous time as the time started, when subsynchronous below when next synchrodata
Between business datum new in this period of time is synchronized as the end time.
The dependence packet path of configuration database, statistical model road in data collection engine Logstash starting configuration file
Diameter, timer, this synchronization time record file path and timestamp field information.
Also to refer to the output position of fixed number evidence in data collection engine Logstash starting configuration file simultaneously, specifying will
Data are synchronized in configured search engine ES and specify the index name generated in search engine ES.
In the step (3), stores each business datum of the search engine in ES and all contain statistical function needs
All field informations, polymerize according to zoning coding and different service conditions be added as querying condition and obtain different zonings
Under each service condition business sum, by zoning coding and status of service field inquire to obtain every kind as querying condition
The details list of business sum, and visualize and come out after data are come out from search engine ES.
Compared with prior art, it is somebody's turn to do the government affairs service data analysis method based on ELK, is had the following characteristics that
1, the efficiency of big data statistics
By using the characteristic of search engine, can quickly count on it is desired as a result, the data volume of millions and compared with
It can be realized the effect that the second looks into for complicated business logic.
2, the pressure to database is reduced
It is counted by using the mode of ELK, reduces the direct pressure to database, do not needed in real time for some
The statistical model of update, our timings synchrodata when formal library frequency of use is lower at noon or at night.And it needs
The portfolio of each Timing Synchronization of the data of real-time synchronization is fewer, to the pressure very little of database.
3, the reusability of statistical model
Designed statistical model, which can be used alone, to be used interchangeably, and the reuse rate of statistical model is very high, and one
Model might have the statistics of several scenes.
Claims (8)
1. a kind of government affairs service data analysis method based on ELK, which comprises the following steps:
(1) the search engine ES based on Lucene is used, government affairs service data analytical equipment is developed on the basis of search engine ES,
The statistical model that can be supported more scenes, can be used interchangeably is established, to complete function of statistic analysis；
(2) the government affairs service data analytical equipment built-in timer, is timed each statistical model, by industry in database
Business data are synchronized in search engine ES；
(3) business datum synchronized is counted using search engine ES.
2. the government affairs service data analysis method according to claim 1 based on ELK, it is characterised in that: the step (1)
In, statistical model is the SQL model that can find business information and the information containing zoning.
3. the government affairs service data analysis method according to claim 2 based on ELK, it is characterised in that: building statistics mould
When type, first constructs a government affairs service business data visualization by database and show the not statistics dish of same district grade and different conditions
Single, the statistics menu includes the main table of business of business details, records the record sheet and storage zoning letter of each link of the business
The zoning table of breath；By the main table of business of the business details, the record sheet and storage zoning information of each link of the business are recorded
Zoning table it is interrelated, the SQL model that can find business information and the information containing zoning can be assembled.
4. the government affairs service data analysis method according to claim 3 based on ELK, it is characterised in that: the statistics mould
Type includes time statistical model, window statistical model, unit statistical model and hall statistical model.
5. the government affairs service data analysis method according to claim 4 based on ELK, it is characterised in that: the step (2)
In, the data in database are synchronized in search engine ES using data collection engine Logstash；Draw in data collection
It holds up and specifies timestamp of the field as each synchrodata in the starting configuration file of Logstash, then use timer
The time interval for needing synchrodata is set；It for the first time can be complete by the data generated before current time in database when synchrodata
It synchronizes into and current synchronization time is recorded in file；According to the time interval that timer is set, in next synchrodata
When, this hyposynchronous time will be taken out as the time started, the subsynchronous time synchronizes this as the end time below
New business datum in the section time.
6. the government affairs service data analysis method according to claim 5 based on ELK, it is characterised in that: in data collection
Engine Logstash starts the dependence packet path of configuration database in configuration file, and statistical model path, timer, this is subsynchronous
Time records file path and timestamp field information.
7. the government affairs service data analysis method according to claim 6 based on ELK, it is characterised in that: while will also be
Data collection engine Logstash starts the output position of configuration file middle finger fixed number evidence, and specified be synchronized to data has been configured
In good search engine ES and specify the index name generated in search engine ES.
8. the government affairs service data analysis method according to claim 7 based on ELK, it is characterised in that: the step (3)
In, it stores each business datum of the search engine in ES and all contains all field informations of statistical function needs, according to
Zoning coding obtains the industry of each service condition under different zonings to polymerize and be added different service conditions as querying condition
Business sum is arranged by the details that zoning coding and status of service field inquire to obtain as querying condition every kind of business sum
Table, and visualize and come out after data are come out from search engine ES.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910160306.6A CN109872135A (en) | 2019-03-04 | 2019-03-04 | A kind of government affairs service data analysis method based on ELK |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910160306.6A CN109872135A (en) | 2019-03-04 | 2019-03-04 | A kind of government affairs service data analysis method based on ELK |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109872135A true CN109872135A (en) | 2019-06-11 |
Family
ID=66919725
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910160306.6A Pending CN109872135A (en) | 2019-03-04 | 2019-03-04 | A kind of government affairs service data analysis method based on ELK |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109872135A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111339147A (en) * | 2020-03-06 | 2020-06-26 | 杭州依图医疗技术有限公司 | Medical data processing method, dimension reduction query method and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110137932A1 (en) * | 2009-12-08 | 2011-06-09 | Akhil Wable | Second-Order Connection Search in a Social Networking System |
| CN103399887A (en) * | 2013-07-19 | 2013-11-20 | 蓝盾信息安全技术股份有限公司 | Query and statistical analysis system for mass logs |
-
2019
- 2019-03-04 CN CN201910160306.6A patent/CN109872135A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110137932A1 (en) * | 2009-12-08 | 2011-06-09 | Akhil Wable | Second-Order Connection Search in a Social Networking System |
| CN103399887A (en) * | 2013-07-19 | 2013-11-20 | 蓝盾信息安全技术股份有限公司 | Query and statistical analysis system for mass logs |
Non-Patent Citations (3)
| Title |
|---|
| 山东梅长苏: "ELK日志监控分析系统Logstash详解之——input模块", 《CSDN》 * |
| 曾恒: "基于ELK的网络安全日志管理分析系统的设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 王裕辰: "基于ELK Stack的实时日志分析系统的设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111339147A (en) * | 2020-03-06 | 2020-06-26 | 杭州依图医疗技术有限公司 | Medical data processing method, dimension reduction query method and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109408347A (en) | A kind of index real-time analyzer and index real-time computing technique | |
| CN101754253A (en) | General packet radio service (GPRS) end-to-end performance analysis method and system | |
| CN110413599A (en) | Generating date and storage system and method | |
| CN103902537A (en) | Multi-service log data storage processing and inquiring system and method thereof | |
| CN108197237A (en) | Visualization data, which collect, shows system | |
| CN101141754B (en) | Value-added service analysis system and method thereof | |
| CN104239304A (en) | Data processing method, device and equipment | |
| CN112559513A (en) | Link data access method, device, storage medium, processor and electronic device | |
| CN105049298A (en) | Method and system for monitoring cloud resource | |
| CN102026228B (en) | Statistical method and equipment for communication network performance data | |
| CN105338208B (en) | Unified bill charging method and system | |
| CN106934057A (en) | A kind of data cached update method of paging and device | |
| CN109271561A (en) | A kind of profile information search method and device | |
| CN109872135A (en) | A kind of government affairs service data analysis method based on ELK | |
| CN115314499A (en) | Multi-terminal cooperative working method and system suitable for electric power field | |
| CN103312540B (en) | Customer service demand parameter is determined method and apparatus | |
| CN104182470B (en) | A kind of mobile terminal application class system and method based on SVM | |
| CN102075896A (en) | Price previewing method and system | |
| CN100571267C (en) | A kind of method for associating general multiple protocols | |
| CN112463527A (en) | Data processing method, device, equipment, system and storage medium | |
| CN116644136A (en) | Data acquisition method, device, equipment and medium for increment and full data | |
| CN101610459A (en) | The automatically acquiring MMS content system and method | |
| CN115033646A (en) | Method for constructing real-time warehouse system based on Flink and Doris | |
| CN103067934A (en) | Core network multiple interfaces signal flow connection method | |
| CN100488219C (en) | Method for traffic data gathering and analysis statistic |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190611 |