CN110865986B - Enterprise retail system data summarization synchronization scheme - Google Patents
Enterprise retail system data summarization synchronization scheme Download PDFInfo
- Publication number
- CN110865986B CN110865986B CN201911103228.2A CN201911103228A CN110865986B CN 110865986 B CN110865986 B CN 110865986B CN 201911103228 A CN201911103228 A CN 201911103228A CN 110865986 B CN110865986 B CN 110865986B
- Authority
- CN
- China
- Prior art keywords
- log
- request
- information
- nginx
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses an enterprise retail system data summarization synchronization scheme, which initiates a request to log information through a browser; firstly, when the client requests the nginx of the server, we put a tag on each request in the nginx; step S2, recording a request marked with a tag, and recording tag information to a log storage system; then, the nginx forwards the service request agent to the application system together with the tag information; the application system takes tag information as a part of log information and writes the tag information and the system log into a log storage system on the server; the logs on each machine are pushed back to the intranet asynchronously through kafuka. The invention real-time log monitoring comprises the steps of monitoring log file generation abnormality in real time; monitoring abnormal log content and early warning in real time; data synchronization across machine rooms comprises the steps of completely and uninterruptedly pulling the environmental logs on each line to the local; streaming avoids log files that are too large to be transmitted.
Description
Technical Field
The invention relates to the technical field of KOM systems, in particular to an enterprise retail system data summarization synchronization scheme.
Background
In the middle of the traditional synchronization scheme, most of logs of the independent deployment applications of the system are directly written on a disk of a server, are directly logged on a corresponding machine when needing to be checked, are downloaded, are manually analyzed, and if data synchronization is met, the logs are logged on the server to lead the data to csv and then are led into a local new database, namely, in the traditional mode, the logs are sequentially and manually segmented into files, manually packaged and downloaded, and the local reduction summary is converted into the database for storage.
The existing defects and problems are as follows:
1. because the application is oriented to a plurality of overseas countries, data are stored in a plurality of servers overseas, when the logs need to be checked, the problem of crossing machine rooms exists, and particularly in a distributed system, the latest logs need to be searched in a plurality of machines on line.
2. The local log data has delay and depends on the pulling of timing tasks, so that log monitoring and on-line problems can not be found in time.
3. Traditional logs are archived according to pull time, log interruption occurs, and the record service integrity is not good.
Disclosure of Invention
The invention aims to provide an enterprise retail system data summarization synchronization scheme which solves the problem of the defects existing in the prior art.
In order to achieve the above purpose, the present invention provides the following technical solutions: an enterprise retail system data summary synchronization scheme comprising the steps of:
step S1: initiating a request for log information through a browser;
step S2: firstly, when the client requests the nginx of the server, we put a tag on each request in the nginx;
step S3: step S2, recording a request marked with a tag, and recording tag information to a log storage system;
step S4: then, the nginx forwards the service request agent to the application system together with the tag information;
step S5: the application system takes tag information as a part of log information and writes the tag information and the system log into a log storage system on the server;
step S6: the logs on each machine are asynchronously pushed back to the intranet through kafuka;
step S7: order analysis system: the judgment basis is that the corresponding url of the order is called, and the request of the HTTP 200 state code with the normal result is returned; when we find that there is this corresponding request; the specific situation of the request order can be known by querying the database in the intranet in the reverse direction; the bought sku, quantity and amount;
step S8: internal network pressure measurement and mirror image flow: the log of the nginx is utilized, the parameter of the request is stored in the middle of the ginx log, and the real request sample of the request can be spliced by the method; then request the test environment: if there is a problem, feedback will be provided immediately in the middle of the test procedure.
Preferably, in step S1, the request may also be initiated through a mobile phone or an IOS.
Preferably, the communication protocol in step S1 is an http protocol or a threft protocol.
Preferably, the cross-machine room assembly of kafka in step S6; the Kakfa MirrorMake consumes information in each application server, and then generates data in the intranet, so that the data is ensured not to be accumulated and lost.
Preferably, log-stack real-time monitoring nginx and log files output by applications are deployed on each server, information is asynchronously collected and formatted, and the information is transmitted to an intranet server through kafaka to be summarized uniformly.
Compared with the prior art, the invention has the following beneficial effects:
1. real-time log monitoring, including monitoring log file generation abnormality in real time; monitoring abnormal log content and early warning in real time.
2. Data synchronization across machine rooms comprises the steps of completely and uninterruptedly pulling the environmental logs on each line to the local; streaming avoids log files that are too large to be transmitted.
3. And the system service complete operation is restored by utilizing the local summarized information, so that the subsequent operation of optimizing the system and analyzing the service by the big data is facilitated.
Drawings
FIG. 1 is a schematic flow chart of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1
Referring to fig. 1, the present invention provides a technical scheme of an enterprise retail system data summarization synchronization scheme: an enterprise retail system data summary synchronization scheme comprising the steps of:
step S1: initiating a request for log information through a browser; the communication protocol is an http protocol or a thraft protocol.
Step S2: first, when a request is sent to the server's nginx, we put a tag on each request in the nginx.
Step S3: in step S2, the request marked with the tag is recorded, and tag information is recorded to a log storage system.
Step S4: then, the ngix forwards the service request agent to the application system together with the tag information.
Step S5: the application system takes tag information as a part of log information and writes the tag information and the system log into a log storage system on the server; and (3) deploying log-mesh real-time monitoring nginx and log files output by the application on each server, asynchronously collecting information, formatting, transmitting to an intranet server through kafaka, and carrying out unified summarization.
Step S6: the logs on each machine are asynchronously pushed back to the intranet through kafuka; kafka's cross-machine room component; the Kakfa MirrorMake generates data in the internal network after consuming information of each application server, ensures that the data cannot be accumulated and lost, and can not only realize data loss across machine rooms but also support formatting customization of consumed data and original data compared with the traditional binlog master-slave.
Step S7: order analysis system: the judgment basis is that the corresponding url of the order is called, and the request of the HTTP 200 state code with the normal result is returned; when we find that there is this corresponding request; the specific situation of the request order can be known by querying the database in the intranet in the reverse direction; sku, quantity, and amount purchased.
Step S8: internal network pressure measurement and mirror image flow: the log of the nginx is utilized, the parameter of the request is stored in the middle of the ginx log, and the real request sample of the request can be spliced by the method; then request the test environment: if there is a problem, feedback will be provided immediately in the middle of the test procedure.
Example 2
Referring to fig. 1, the present invention provides a technical scheme of an enterprise retail system data summarization synchronization scheme: an enterprise retail system data summary synchronization scheme comprising the steps of:
step S1: initiating a request for log information through a mobile phone; the communication protocol is an http protocol or a thraft protocol.
Step S2: first, when a request is sent to the server's nginx, we put a tag on each request in the nginx.
Step S3: in step S2, the request marked with the tag is recorded, and tag information is recorded to a log storage system.
Step S4: then, the ngix forwards the service request agent to the application system together with the tag information.
Step S5: the application system takes tag information as a part of log information and writes the tag information and the system log into a log storage system on the server; and (3) deploying log-mesh real-time monitoring nginx and log files output by the application on each server, asynchronously collecting information, formatting, transmitting to an intranet server through kafaka, and carrying out unified summarization.
Step S6: the logs on each machine are asynchronously pushed back to the intranet through kafuka; kafka's cross-machine room component; the Kakfa MirrorMake generates data in the internal network after consuming information of each application server, ensures that the data cannot be accumulated and lost, and can not only realize data loss across machine rooms but also support formatting customization of consumed data and original data compared with the traditional binlog master-slave.
Step S7: order analysis system: the judgment basis is that the corresponding url of the order is called, and the request of the HTTP 200 state code with the normal result is returned; when we find that there is this corresponding request; the specific situation of the request order can be known by querying the database in the intranet in the reverse direction; sku, quantity, and amount purchased.
Step S8: internal network pressure measurement and mirror image flow: the log of the nginx is utilized, the parameter of the request is stored in the middle of the ginx log, and the real request sample of the request can be spliced by the method; then request the test environment: if there is a problem, feedback will be provided immediately in the middle of the test procedure.
Example 3
Referring to fig. 1, the present invention provides a technical scheme of an enterprise retail system data summarization synchronization scheme: an enterprise retail system data summary synchronization scheme comprising the steps of:
step S1: initiating a request for log information through an IOS; the communication protocol is an http protocol or a thraft protocol.
Step S2: first, when a request is sent to the server's nginx, we put a tag on each request in the nginx.
Step S3: in step S2, the request marked with the tag is recorded, and tag information is recorded to a log storage system.
Step S4: then, the ngix forwards the service request agent to the application system together with the tag information.
Step S5: the application system takes tag information as a part of log information and writes the tag information and the system log into a log storage system on the server; and (3) deploying log-mesh real-time monitoring nginx and log files output by the application on each server, asynchronously collecting information, formatting, transmitting to an intranet server through kafaka, and carrying out unified summarization.
Step S6: the logs on each machine are asynchronously pushed back to the intranet through kafuka; kafka's cross-machine room component; the Kakfa MirrorMake generates data in the internal network after consuming information of each application server, ensures that the data cannot be accumulated and lost, and can not only realize data loss across machine rooms but also support formatting customization of consumed data and original data compared with the traditional binlog master-slave.
Step S7: order analysis system: the judgment basis is that the corresponding url of the order is called, and the request of the HTTP 200 state code with the normal result is returned; when we find that there is this corresponding request; the specific situation of the request order can be known by querying the database in the intranet in the reverse direction; sku, quantity, and amount purchased.
Step S8: internal network pressure measurement and mirror image flow: the log of the nginx is utilized, the parameter of the request is stored in the middle of the ginx log, and the real request sample of the request can be spliced by the method; then request the test environment: if there is a problem, feedback will be provided immediately in the middle of the test procedure.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (5)
1. An enterprise retail system data summarization synchronization scheme, characterized in that: the method comprises the following steps:
step S1: initiating a request for log information through a browser;
step S2: firstly, when the client requests the nginx of the server, we put a tag on each request in the nginx;
step S3: step S2, recording a request marked with a tag, and recording tag information to a log storage system;
step S4: then, the nginx forwards the service request agent to the application system together with the tag information;
step S5: the application system takes tag information as a part of log information and writes the tag information and the system log into a log storage system on the server;
step S6: the logs on each machine are asynchronously pushed back to the intranet through kafuka;
step S7: order analysis system: the judgment basis is that the corresponding url of the order is called, and the request of the HTTP 200 state code with the normal result is returned; when we find that there is this corresponding request; the specific situation of the request order can be known by querying the database in the intranet in the reverse direction; the bought sku, quantity and amount;
step S8: internal network pressure measurement and mirror image flow: the log of the nginx is utilized, the parameter of the request is stored in the middle of the ginx log, and the real request sample of the request can be spliced by the method; then request the test environment: if there is a problem, feedback will be provided immediately in the middle of the test procedure.
2. An enterprise retail system data aggregation synchronization scheme as claimed in claim 1, wherein: in step S1, the request may also be initiated through a mobile phone or an IOS.
3. An enterprise retail system data aggregation synchronization scheme as claimed in claim 1, wherein: the communication protocol in step S1 is an http protocol or a threft protocol.
4. An enterprise retail system data aggregation synchronization scheme as claimed in claim 1, wherein: the cross-machine room assembly of kafka in step S6; the Kakfa MirrorMake consumes information in each application server, and then generates data in the intranet, so that the data is ensured not to be accumulated and lost.
5. An enterprise retail system data aggregation synchronization scheme as claimed in claim 1, wherein: and (3) deploying log-mesh real-time monitoring nginx and log files output by the application on each server, asynchronously collecting information, formatting, transmitting to an intranet server through kafaka, and carrying out unified summarization.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911103228.2A CN110865986B (en) | 2019-11-12 | 2019-11-12 | Enterprise retail system data summarization synchronization scheme |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911103228.2A CN110865986B (en) | 2019-11-12 | 2019-11-12 | Enterprise retail system data summarization synchronization scheme |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110865986A CN110865986A (en) | 2020-03-06 |
| CN110865986B true CN110865986B (en) | 2023-05-12 |
Family
ID=69654608
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911103228.2A Active CN110865986B (en) | 2019-11-12 | 2019-11-12 | Enterprise retail system data summarization synchronization scheme |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110865986B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104902432A (en) * | 2015-06-04 | 2015-09-09 | 北京京东尚科信息技术有限公司 | Method and device for generating application operation log of terminal mobile |
| CN105260452A (en) * | 2015-10-12 | 2016-01-20 | 成都视达科信息技术有限公司 | System and method for collecting, searching, and analyzing offline log |
| CN109062762A (en) * | 2018-07-26 | 2018-12-21 | 浙江数链科技有限公司 | Formatted log method for pushing and device |
| CN110019239A (en) * | 2017-12-29 | 2019-07-16 | 百度在线网络技术（北京）有限公司 | Storage method, device, electronic equipment and the storage medium of reported data |
-
2019
- 2019-11-12 CN CN201911103228.2A patent/CN110865986B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104902432A (en) * | 2015-06-04 | 2015-09-09 | 北京京东尚科信息技术有限公司 | Method and device for generating application operation log of terminal mobile |
| CN105260452A (en) * | 2015-10-12 | 2016-01-20 | 成都视达科信息技术有限公司 | System and method for collecting, searching, and analyzing offline log |
| CN110019239A (en) * | 2017-12-29 | 2019-07-16 | 百度在线网络技术（北京）有限公司 | Storage method, device, electronic equipment and the storage medium of reported data |
| CN109062762A (en) * | 2018-07-26 | 2018-12-21 | 浙江数链科技有限公司 | Formatted log method for pushing and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110865986A (en) | 2020-03-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107832196B (en) | Monitoring device and monitoring method for abnormal content of real-time log | |
| US7685270B1 (en) | Method and apparatus for measuring latency in web services | |
| CN107341258B (en) | Log data acquisition method and system | |
| CN112559475B (en) | Data real-time capturing and transmitting method and system | |
| CN111416821A (en) | Internet of things equipment information acquisition method, system and device | |
| EP2902908A1 (en) | System operation trace method in distributed system | |
| CN103713981A (en) | Database server performance detection and early warning method | |
| CN110727560A (en) | Cloud service alarm method and device | |
| CN105471932A (en) | Front-end application monitoring method, front-end application and front-end application monitoring system | |
| CN111522703A (en) | Method, apparatus and computer program product for monitoring access requests | |
| CN110515799B (en) | MySQL monitoring system based on python language and implementation method | |
| CN110659186A (en) | Alarm information reporting method and device | |
| CN108288997A (en) | A kind of transmission network luminous power automated collection systems | |
| CN114253806A (en) | Access stratum log collection, analysis and early warning system | |
| CN110865986B (en) | Enterprise retail system data summarization synchronization scheme | |
| CN114090529A (en) | Log management method, device, system and storage medium | |
| CN108712306A (en) | A kind of information system automation inspection platform and method for inspecting | |
| CN112380254A (en) | Data caching system and method of automation equipment | |
| CN114238018B (en) | Method, system and device for detecting integrity of log collection file and storage medium | |
| CN113395332B (en) | Data assembling method and related equipment | |
| CN113726808A (en) | Website monitoring method, device, equipment and storage medium | |
| CN111639249A (en) | Automatic monitoring method, device and equipment for user feedback error reporting | |
| CN116074388B (en) | Flow forwarding method and system based on log queue | |
| CN112217866B (en) | System and method for forwarding market quotation data of internal and external networks | |
| CN116860580A (en) | Log acquisition state generation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |