CN115643166A - Method and device for returning CDN log with high reliability - Google Patents
Method and device for returning CDN log with high reliability Download PDFInfo
- Publication number
- CN115643166A CN115643166A CN202211569705.6A CN202211569705A CN115643166A CN 115643166 A CN115643166 A CN 115643166A CN 202211569705 A CN202211569705 A CN 202211569705A CN 115643166 A CN115643166 A CN 115643166A
- Authority
- CN
- China
- Prior art keywords
- access log
- log
- data
- edge node
- cdn
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Information Transfer Between Computers (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides a method and a device for highly reliably returning CDN logs, which comprises the following steps: collecting an access log of the edge node; forwarding the access log on the edge node to a local message queue through a Nginx component arranged on the edge node; according to the forwarding service of the edge node, acquiring the access log from the local message queue and forwarding the access log to a data center; sending the access log in the data center to a persistent storage in a log storage device. According to the method and the device, the access log is forwarded through the CDN node, when the public network is temporarily unavailable, the access log can be cached locally at the edge node, the log can not be lost due to network instability, and the analyzability of the service running condition is realized when the service is abnormal.
Description
Technical Field
The application relates to the field of log management, in particular to a method for returning CDN logs with high reliability. The application also relates to a device for highly reliably returning the CDN log.
Background
The edge node is a logic abstraction of basic common capability of a plurality of product forms of the edge side such as an edge gateway, an edge controller and an edge server, and the product forms have common capability of real-time data analysis, local data storage, real-time network connection and the like of the edge side.
At present, the development of mobile internet technology makes a great deal of various multi-type mobile internet terminals such as handheld devices and wearable devices used, and massive information is generated.
A content distribution network service provider provides accelerated services to customers through a large number of edge node servers. The number of log files generated by each node server in the CDN network is huge. In the face of massive log file information, the method can efficiently and quickly analyze, screen and refine massive log files, and becomes a necessary means for solving a plurality of problems.
However, when the caching time of the collected log files reaches the preset time or the size of the collected log files meets the preset quantity threshold, the collected log files are uploaded to the corresponding central collection server, the central collection server uploads the received log files of the edge node servers to the data center, and the data center performs rapid analysis, screening and refinement on the log files.
When the log file is uploaded to the central collection server, when the central controller receives a log file uploading request of the edge node server, the central controller needs to determine a log uploading route according to the service quality of the central controller and the service quality of the edge node server and the central controller or the service quality of the uploading server, and finally uploads data.
The edge node (CDN) is used for caching data acquired by the user from the server and improving the access speed of the user. The method comprises the steps of obtaining part of information related to a client through an access record of a user CDN resource, wherein the part of information includes a client address, a resource request address, a request state code, a requested file size, a requested edge node address and the like, so that the collection of a CDN edge log is realized, and the resource accessed by the CDN can be analyzed through the collected information. For example: the request state or whether the request node is abnormal or not, whether attack exists or not, whether the node distribution is distributed according to the nearby distribution or not, in addition, the file size of the resource is recorded in the log field, and the flow can be counted through the resource size condition and the user charging condition can be calculated through the flow.
Application No. 202110834285.9, patent name: chinese patent mentioning a monitoring data processing method and system, the monitoring data processing system includes: the data acquisition module is used for monitoring the content distribution network and generating monitoring log data; the data storage module is used for storing the monitoring log data; the data stream state analysis module is used for counting the storage state information of the monitoring log data, obtaining the monitoring result of the monitoring log data and sending the monitoring result to the data storage module; and the data application module is used for sending out alarm information when the monitoring result in the data storage module is an abnormal result. The invention can find abnormal monitoring log data in time.
However, if the edge logs are not collected normally, the operation condition of the whole service cannot be judged, and when the CDN resource request is abnormal, the reason for the abnormality cannot be quickly judged, and the user charging is also affected.
Disclosure of Invention
The invention aims to overcome the defect that the running condition of the whole service cannot be judged if the logs are not normally collected in the prior art, and provides a method for highly reliably returning the CDN logs. The application also relates to a device for highly reliably returning the CDN logs.
The application provides a method for returning CDN logs with high reliability, which comprises the following steps:
collecting an access log of the edge node;
forwarding the access log on the edge node to a local message queue through a Nginx component arranged on the edge node;
according to the forwarding service of the edge node, obtaining the access log from the local message queue and forwarding the access log to a data center;
sending the access log in the data center to a persistent storage in a log storage device.
Optionally, the obtaining the access log and forwarding the access log to the data center includes:
and judging whether the access log is sent or not, and if not, retrying according to an exponential backoff mode.
Optionally, the access log is used to record resource request conditions.
Optionally, a data interface is disposed in the data center, and is configured to send the access log to the outside according to a request.
Optionally, the method further includes: before the persistent storage, the access log is visualized.
The present application further provides a device for highly reliably returning a CDN log, including:
the collection module is used for collecting the access logs of the edge nodes;
a sending module, configured to forward the access log on the edge node to a local message queue through a Nginx component set on the edge node;
the forwarding module is used for acquiring the access log from the local message queue according to the forwarding service of the edge node and forwarding the access log to a data center;
and the storage module is used for sending the access log in the data center to log storage equipment for persistent storage.
Optionally, the forwarding module obtains the access log and forwards the access log to the data center, where the forwarding module includes:
and judging whether the access log is sent or not, and if not, retrying according to an exponential backoff method.
Optionally, the access log is used to record resource request conditions.
Optionally, a data interface is disposed in the data center, and is configured to send the access log to the outside according to a request.
Optionally, the method further includes: before the storage module stores the access log in a persistent mode, the access log is visualized.
The application has the advantages and beneficial effects that:
the application provides a method for returning CDN logs with high reliability, which comprises the following steps: collecting an access log of the edge node; forwarding the access log on the edge node to a local message queue through a Nginx component arranged on the edge node; according to the forwarding service of the edge node, obtaining the access log from the local message queue and forwarding the access log to a data center; sending the access log in the data center to a persistent storage in a log storage device. According to the method and the device, the access log is forwarded through the CDN node, when the public network is temporarily unavailable, the access log can be cached locally at the edge node, the log can not be lost due to network instability, and the analyzability of the service running condition is realized when the service is abnormal.
Drawings
Fig. 1 is a schematic flow chart of highly reliable CDN log return in the present application.
Fig. 2 is a schematic view of a first display interface of visualized data in the present application.
Fig. 3 is a schematic diagram of a second display interface for visualization data in the present application.
Fig. 4 is a schematic diagram of an apparatus for highly reliable returning CDN logs in the present application.
Detailed Description
The present invention is further described below in conjunction with the appended drawings and specific embodiments so that those skilled in the art may better understand and practice the present invention.
The following is an example of a specific implementation process provided for explaining the technical solutions to be protected in the present application in detail, but the present application may also be implemented in other ways than those described herein, and a person skilled in the art may implement the present application by using different technical means under the guidance of the idea of the present application, so that the present application is not limited by the following specific embodiments.
The application provides a method for returning CDN logs with high reliability, which comprises the following steps:
collecting an access log of the edge node; forwarding the access log on the edge node to a local message queue through a Nginx component arranged on the edge node;
according to the forwarding service of the edge node, acquiring the access log from the local message queue and forwarding the access log to a data center;
sending the access log in the data center to a persistent storage in a log storage device.
According to the method and the device, the access logs are forwarded through the CDN nodes, when the public network is temporarily unavailable, the access logs can be cached locally at the edge nodes, the logs cannot be lost due to network instability, and the analyzability of the service running condition is achieved when the service is abnormal.
Fig. 1 is a schematic flow chart of highly reliable CDN log return in the present application.
Referring to fig. 1, S101 collects an access log of an edge node.
Because the server or the distance between the server and the user end and the speed of processing the request of the client end by the server cause delay, jamming and the like of the data request from the server by the user, an edge node (CDN) is generally arranged at a position close to the user end to transfer and process the data.
When the client requests data to a server or a data center for data service, the client first enters the edge node, the edge node searches the requested data, and if the requested data is not found, the client forwards the request to the data center or the server for acquisition.
And the edge node receives the data fed back to the client request by the server or the data center, and forwards the data to the client and stores the data at the same time, so that the data does not need to be acquired from the server or the data center when the client requests again.
On the other hand, after the edge node acquires the data, the data may be further processed according to client information in the client request information to adapt to the specification of the client, for example, to perform cropping or size conversion on a picture.
The client requests data, an access log can be generated at the edge node, and the access log comprises: access time, access content size, etc.
The method comprises the steps of obtaining part of information related to a client through access records of CDN resources of a user, wherein the information comprises a client address, a resource request address, a request state code, a requested file size, a requested edge node address and the like, so that the CDN edge logs are collected, and the resources accessed by the CDN can be analyzed through the collected information.
According to the method, access logs of a client accessing the edge nodes are collected from the edge nodes, and after the client sends a data request to the CDN nodes, the access logs are generated according to the data request of the client.
The collection access log may be collected by directly reading log information. Specifically, the path of the log file obtained by file.yml configuration and the configuration output to the ES can be found in the installation directory of the file; and checking the log information in the es index through the elastic search-head plug-in to realize data acquisition.
Further, after the log is obtained, the log is compressed and packaged, and is listed in a message sending queue.
The method and the device have the advantages that log acquisition can be carried out in real time, further, the log acquisition is carried out by setting the user-defined small interval, the real-time performance is not influenced by the small interval, and meanwhile, a data processing peak execution program can be avoided.
Specifically, the custom interval realizes data processing amount judgment according to setting, determines whether to execute log acquisition according to the data processing amount, and determines the size of the custom interval according to the duration of the data processing peak. Wherein the data processing amount judgment is realized by setting a threshold value.
Referring to fig. 1, S102 forwards the access log on the edge node to a local message queue through a Nginx component disposed on the edge node.
Specifically, the message is sent to a local message queue through a nginnx component, where the local message queue is the edge node (CDN).
The Nginx component depends on an ngx _ http _ upstream _ module, and supported proxy modes comprise: proxy _ pass, fastcgi _ pass, memcached _ pass, etc. The scheduling algorithms can be divided into two categories, namely a static scheduling algorithm and a dynamic scheduling algorithm.
And (3) a static scheduling algorithm:
the allocation is performed according to the rule set by the user, without considering the situation of a back-end node server, such as: rr poll (default schedule): distributing the data to different configured back-end node servers one by one according to the request of the client; wrr weight poll: adding weight on the basis of rr polling, wherein the larger the weight value is, the more requests are forwarded; ip _ hash: the method comprises the steps that the distribution is carried out according to the hash value of the IP of a client, and the IP addresses of the clients are the same and can be distributed to the same host; url _ hash: and distributing according to the hash value of the accessed URL.
And (3) a dynamic scheduling algorithm:
the decision whether to distribute the request is based on the current state of the backend node, e.g.: fair, allocating requests according to the response time of a back-end node server, preferentially allocating least _ conn with short response time, and allocating the requests according to the number of the connections of the back-end node, wherein the requests are distributed when the number of the connections is small.
The Nginx assembly mainly comprises: and the http _ proxy _ module is used for forwarding the request to another server, matching the specified URI through the location function, and pointing the URI request which is matched to the defined upstream module through proxy _ pass.
The access log firstly generates a text character string of the access log according to log content and then caches the text character string in the message queue. Before this, the number of messages to be sent in the message queue may be determined, and whether to join the message queue may be selected based on the determination.
Specifically, after the log content is collected, the log content is stored at the edge node, and the log content is deleted after being correctly sent out.
The storage may be a cache of the log content, and in this application, the storage also refers to storing the log content in a nonvolatile memory. The non-volatile memory is a storage device arranged on the edge node to store the content requested by the client to the server or the data center.
In the present application, the nonvolatile memory is divided into an independent memory area for storing the log content, and the size of the memory area can be freely set.
The selection means that the access log cannot affect the speed of the client for requesting data, and therefore, selection needs to be performed based on whether the sending of the access log will cause pressure on data transmission between the client and the CDN. At this time, when the pressure of the data transmission is large, the access log of the edge node may be stored according to the nonvolatile memory. Further, the pressure may be higher when the public network is temporarily unavailable, and the access log may be directly read from the non-volatile memory. In addition, the message queue may continuously read the access log from the non-easy-to-implement memory, add the access log to the message queue, and repeatedly send the access log until the sending is successful.
When the client requests, the log is firstly cached in a process internal cache region, and then response content is directly returned to the client. The present application makes the following settings: when the asynchronous thread waits for the buffer to be full or exceed a certain time (e.g., 30 seconds), all the contents of the buffer are sent to the native message queue. Because the deployment adopted when the data is sent to the local message queue is the loop network card of the local, the public network access of the client side cannot be influenced, and meanwhile, the data is asynchronously sent to the message queue, and the response time of the ongoing request cannot be increased.
In the present application, the logs are divided into access logs and error logs. The error log comprises monitoring data such as node states, request data and connection numbers, has higher priority than other data, and can be sent preferentially, so that the working condition of the edge node can be fed back to the data center in time.
Furthermore, after the log data is sent to the local message queue, a reading program is arranged to read the data in the local message queue, compress the data through a snappy algorithm and then send the data to the data center. The Snappy algorithm takes up less CPU and has a higher compression ratio. When the sending speed has a certain limit, the dynamic adjustment can be carried out according to the utilization rate of the current network card. If the network card load is always high, the access log will be sent with a delay, but the error log will not be sent with a delay.
In this application, whether the access log enters the message queue of the client may be determined by the following expression:
wherein P is the data transmission time of an existing message queue, the
Is the first in the message queueiThe data size of the message, n is the message queue message number, S is the current network speed, M is the transmission interval time, and G is a preset threshold.
And when the P is smaller than the G, adding the P into the message queue, otherwise, performing enqueue waiting.
Specifically, when P is greater than G, the access log will not be immediately added to the message queue, and at this time, the P is repeatedly calculated according to a preset time interval, and until P is less than G, the access log is added to the message queue.
In addition, the method is also provided with an enqueue waiting time threshold, and when the enqueue waiting time is greater than the waiting time threshold, the enqueue waiting time is added into the message queue.
Specifically, the queue-insertion operation is added to the message queue, and based on the judgment of the data volume corresponding to each message queue message in the message queue, the expression is as follows:
wherein
Representing the amount of dataThe message with the smallest data size, number c, in the message queue is the position for selective insertion.
Then, the message of the access log is inserted into the message queue.
Referring to fig. 1, in step S103, according to the forwarding service of the edge node, the access log is obtained from the local message queue and forwarded to the data center.
And the edge node forwards the access log to a data center, and the access log is cached by the edge node, acquired from the message queue and sent to the data center.
The data center is provided with an edge router which is used for sending and receiving data information, and the access log information is obtained through the edge router and sent to the data center. And after receiving the access log, the data center analyzes the access log according to a preset analysis rule to obtain an analysis result.
Specifically, the analysis is performed based on a preset starting condition, that is, when receiving the access data and triggering a preset rule, the access log is called and analyzed.
In this application, the predetermined rule includes starting when the time interval for receiving the access log is greater than a predetermined maximum interval, and performing a preliminary analysis.
Based on the preliminary analysis result, the data center can send a control instruction to the edge node and wait for replying to the continuous transmission of the access log.
If the edge node has network problems in the forwarding process, the message forwarding service retries according to an exponential backoff method until the log is successfully sent.
And then analyzing the CDN request condition based on the access log, recording and counting the user service flow condition, collecting edge CDN node logs, when the user encounters CDN request abnormity, quickly analyzing through the logs, counting the user request flow through the logs, calculating the CDN charging service condition, autonomously collecting and analyzing the edge CDN logs, ensuring high reliability of collected information, providing an analysis result for the user, and enabling the user to more visually check the state of acceleration service.
Referring to fig. 1, S104 sends the access log in the data center to a persistent storage in a log storage device.
After the data center acquires the log data, the log data is visualized and includes attribute category statistical data based on the log data, as shown in fig. 2 and 3.
And finally, sending the log data to a storage device for permanent storage.
In this application, the storage of the log data includes storage and backup storage, the storage refers to storage in a server or a data center, and the backup storage refers to storage of the data of the access log in a backup storage.
Before the storage, the log data is processed based on a preset format and is visualized. The visualization of the access log may be set based on a display format and display items that are set based on actual conditions.
The data center is provided with a data interface, and is used for sending the data of the access log to the user when the user requests the data of the access log.
The access log data is visualized, when a user selects a display interface, a request command of the access log is automatically generated based on the data of the display interface, and a data request is made to the data center. And the data center screens the access log according to the request command, obtains data to be sent and sends the data to a client.
The present application further provides a device for returning CDN logs with high reliability, including: the system comprises a collection module 201, a sending module 202, a forwarding module 203 and a storage module 204.
Fig. 4 is a schematic diagram of an apparatus for highly reliable returning CDN logs in the present application.
Referring to fig. 4, a collecting module 201 is used for collecting an access log of an edge node.
Because the server or the distance between the server and the user end and the speed of the server for processing the request of the client end enable the user to delay, jam and the like when requesting data from the server, an edge node (CDN) is usually arranged at a position close to the user end to transfer and process data.
When the client requests data to a server or a data center to request the data when providing service, firstly the request enters the edge node, the edge node searches the requested data, and if the requested data is not found, the request is forwarded to the data center or the server to be acquired.
And the edge node receives the data fed back to the client request by the server or the data center, and forwards the data to the client and stores the data at the same time, so that the data does not need to be acquired from the server or the data center when the client requests again.
On the other hand, after the edge node acquires the data, the data may be further processed according to client information in the client request information to adapt to the specification of the client, for example, to perform cropping or size conversion on a picture.
The client requests data, an access log can be generated at the edge node, and the access log comprises: access time, access content size, etc.
The method comprises the steps of obtaining part of information related to a client through access records of CDN resources of a user, wherein the information comprises a client address, a resource request address, a request state code, a requested file size, a requested edge node address and the like, so that the CDN edge logs are collected, and the resources accessed by the CDN can be analyzed through the collected information.
According to the method, firstly, access logs of a client for accessing the edge nodes are collected from the edge nodes, and after the client sends a data request to the CDN nodes, the access logs are generated according to the data request of the client.
The collection access log may be collected by directly reading log information. Specifically, the path of the log file and the configuration output to the ES can be obtained by finding the filebeat-yml configuration in the installation directory of the filebeat; and checking the log information in the es index through the elastic search-head plug-in to realize data acquisition.
Further, after the log is obtained, the log is compressed and packaged, and is listed in a message sending queue.
The method and the device have the advantages that log acquisition can be carried out in real time, further, the log acquisition is carried out by setting the user-defined small interval, the real-time performance is not influenced by the small interval, and meanwhile, a data processing peak execution program can be avoided.
Specifically, the custom interval realizes data processing amount judgment according to setting, determines whether to execute log acquisition according to the data processing amount, and determines the size of the custom interval according to the duration of the data processing peak. Wherein the data processing amount judgment is realized by setting a threshold value.
Referring to fig. 4, the sending module 202 is configured to forward the access log on the edge node to a local message queue through a Nginx component disposed on the edge node.
Specifically, the message is sent to a local message queue through a nginnx component, where the local message queue is the edge node (CDN).
The Nginx component depends on an ngx _ http _ upstream _ module, and the supported proxy mode comprises the following steps: proxy _ pass, fastcgi _ pass, memcached _ pass, etc. The scheduling algorithms can be divided into two categories, namely a static scheduling algorithm and a dynamic scheduling algorithm.
And (3) a static scheduling algorithm:
the allocation is performed according to the rule set by the user, without considering the situation of a back-end node server, such as: rr poll (default schedule): distributing the data to different configured back-end node servers one by one according to the request of the client; wrr weight poll: adding weight on the basis of rr polling, wherein the larger the weight value is, the more the forwarded requests are; ip _ hash: the method comprises the steps that distribution is carried out according to the hash value of the IP of a client, and the IP addresses of the clients are the same and can be distributed to the same host; url _ hash: and distributing according to the hash value of the accessed URL.
And (3) dynamic scheduling algorithm:
the decision whether to distribute the request is based on the current state of the backend node, e.g.: the fair allocates the request according to the response time of the back-end node server, preferentially allocates the least _ conn with short response time according to the number of the connections of the back-end node, and distributes the requests when the number of the connections is small.
The Nginx assembly mainly comprises: and the http _ proxy _ module is used for forwarding the request to another server, matching the specified URI through the location function, and pointing the URI request which is matched to the defined upstream module through proxy _ pass.
Specifically, the message is sent to a local message queue through a nginnx component, where the local message queue is the edge node (CDN).
The access log firstly generates a text character string of the access log according to the log content and then caches the text character string in the message queue. Before this, the number of messages to be sent in the message queue may be determined, and whether to join the message queue may be selected based on the determination.
Specifically, after the log content is collected, the log content is stored in the edge node, and the log content is deleted after being correctly sent out.
The storage may be a cache of the log content, and in this application, the storage also refers to storing the log content in a nonvolatile memory. The non-volatile memory is a storage device arranged at the edge node to store the content requested by the client to the server or the data center.
In the present application, the nonvolatile memory is divided into a separate storage area for storing the log content, and the size of the storage area can be set by itself.
The selection means that the access log cannot influence the speed of the client for requesting data, and therefore, whether the sending of the access log will cause pressure on data transmission between the client and the CDN needs to be selected.
In this application, whether the access log enters the message queue of the client may be determined by the following expression:
wherein P is the data transmission time of an existing message queue, the
Is the first in the message queueiThe data size of each message, n is the message queue message number, S is the current network speed, M is the transmission interval time, and G is a preset threshold.
And when the P is smaller than the G, adding the P into the message queue, otherwise, performing enqueue waiting.
In addition, the method is also provided with an enqueue waiting time threshold, and when the enqueue waiting time is greater than the waiting time threshold, the enqueue waiting time is added into the message queue.
Specifically, the queue-insertion operation is added to the message queue, and based on the judgment of the data volume corresponding to each message queue message in the message queue, the expression is as follows:
wherein
The message with the number c is the position for selective insertion.
Then, the message of the access log is inserted into the message queue.
Specifically, when P is greater than G, the access log will not be immediately added to the message queue, and at this time, the P is repeatedly calculated according to a preset time interval, and until P is less than G, the access log is added to the message queue.
Referring to fig. 3, the forwarding module 203 is configured to obtain the access log from the local message queue according to the forwarding service of the edge node, and forward the access log to the data center.
And the edge node forwards the access log to a data center, and comprises the steps of caching the access log through the edge node, acquiring the access log from the message queue and sending the access log to the data center.
The data center is provided with an edge router which is used for sending and receiving data information, and the access log information is obtained through the edge router and sent to the data center. And after receiving the access log, the data center analyzes the access log according to a preset analysis rule to obtain an analysis result.
Specifically, the analysis is performed based on a preset starting condition, that is, when the access data is received and a preset rule is triggered, the access log is called and analyzed.
In this application, the predetermined rule includes starting when the time interval for receiving the access log is greater than a predetermined maximum interval, and performing a preliminary analysis.
Based on the preliminary analysis result, the data center can send a control instruction to the edge node and wait for replying to the continuous transmission of the access log.
If the edge node has network problems in the forwarding process, the message forwarding service retries according to an exponential back-off method until the log is successfully sent.
And then analyzing the CDN request condition based on the access log, recording and counting the user service flow condition, collecting edge CDN node logs, when the user encounters CDN request abnormity, quickly analyzing through the logs, counting the user request flow through the logs, calculating the CDN charging service condition, autonomously collecting and analyzing the edge CDN logs, ensuring high reliability of collected information, providing an analysis result for the user, and enabling the user to more visually check the state of acceleration service.
Referring to fig. 4, the storage module 204 is configured to send the access log in the data center to a persistent storage in a log storage device.
After the data center acquires the log data, the log data is visualized and includes attribute category statistical data based on the log data, as shown in fig. 2 and 3.
And finally, sending the log data to a storage device for permanent storage.
In this application, the storage of the log data includes storage and backup storage, the storage refers to storage in a server or a data center, and the backup storage refers to storage of the data of the access log in a backup storage.
And before the storage, the log data is processed based on a preset format and is visualized. The visualization of the access log may be set based on a display format and display items that are set based on actual conditions.
The data center is provided with a data interface, and is used for sending the data of the access log to the user when the user requests the data of the access log.
The access log data are visualized, when a user selects a display interface, a request command of the access log is automatically generated based on the data of the display interface, and a data request is made to the data center. And the data center screens the access log according to the request command, obtains data to be sent and sends the data to a client.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The scheme in the embodiment of the application can be implemented by adopting various computer languages, such as object-oriented programming language Java and transliterated scripting language JavaScript.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. Instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. A method for returning CDN logs with high reliability is characterized by comprising the following steps:
collecting an access log of the edge node;
forwarding the access log on the edge node to a local message queue through a Nginx component arranged on the edge node;
according to the forwarding service of the edge node, obtaining the access log from the local message queue and forwarding the access log to a data center;
sending the access log in the data center to a persistent storage in a log storage device.
2. The method for highly reliable back transmission of CDN logs as recited in claim 1, wherein the obtaining the access log and forwarding to the data center comprises:
and judging whether the access log is sent or not, and if not, retrying according to an exponential backoff mode.
3. The method for highly reliable returning of CDN logs as claimed in claim 1, wherein the access log is used for recording resource request condition.
4. The method for highly reliable back transmission of CDN logs as recited in claim 1, wherein a data interface is disposed in the data center, and is configured to send the access log to the outside according to a request.
5. The method for highly reliable back transmission of CDN logs according to any one of claims 1 to 4, further comprising: before the persistent storage, the access log is visualized.
6. An apparatus for returning CDN logs with high reliability, comprising:
the collection module is used for collecting the access logs of the edge nodes;
a sending module, configured to forward the access log on the edge node to a local message queue through a Nginx component set on the edge node;
the forwarding module is used for acquiring the access log from the local message queue according to the forwarding service of the edge node and forwarding the access log to a data center;
and the storage module is used for sending the access log in the data center to log storage equipment for persistent storage.
7. The apparatus for highly reliable backtransmitting CDN log according to claim 6, wherein the forwarding module obtains the access log and forwards the access log to the data center, including:
and judging whether the access log is sent or not, and if not, retrying according to an exponential backoff method.
8. The apparatus for highly reliable backtransmitting CDN log according to claim 6 wherein said access log is used to record resource request condition.
9. The device for highly reliable back transmission of CDN logs as claimed in claim 6, wherein a data interface is disposed in the data center, and is configured to send the access log to outside according to a request.
10. The device for highly reliably returning the CDN log according to any one of claims 6 to 9, further comprising: before the storage module persistently stores the access log, the access log is visualized.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211569705.6A CN115643166B (en) | 2022-12-08 | 2022-12-08 | Method and device for returning CDN log with high reliability |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211569705.6A CN115643166B (en) | 2022-12-08 | 2022-12-08 | Method and device for returning CDN log with high reliability |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115643166A true CN115643166A (en) | 2023-01-24 |
| CN115643166B CN115643166B (en) | 2023-04-07 |
Family
ID=84949116
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211569705.6A Active CN115643166B (en) | 2022-12-08 | 2022-12-08 | Method and device for returning CDN log with high reliability |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115643166B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116095180A (en) * | 2023-03-07 | 2023-05-09 | 天翼云科技有限公司 | Log return routing method, device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112087490A (en) * | 2020-08-07 | 2020-12-15 | 上海绊糖信息科技有限公司 | High-performance mobile terminal application software log collection system |
| CN113190522A (en) * | 2021-06-29 | 2021-07-30 | 阿里云计算有限公司 | Log processing method and device, computing equipment and storage medium |
| CN113934683A (en) * | 2020-07-13 | 2022-01-14 | 北京金山云网络技术有限公司 | Log transmission method and device, CDN system and electronic equipment |
| CN113986862A (en) * | 2021-10-27 | 2022-01-28 | 上海星融汽车科技有限公司 | Client log collection method |
-
2022
- 2022-12-08 CN CN202211569705.6A patent/CN115643166B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113934683A (en) * | 2020-07-13 | 2022-01-14 | 北京金山云网络技术有限公司 | Log transmission method and device, CDN system and electronic equipment |
| CN112087490A (en) * | 2020-08-07 | 2020-12-15 | 上海绊糖信息科技有限公司 | High-performance mobile terminal application software log collection system |
| CN113190522A (en) * | 2021-06-29 | 2021-07-30 | 阿里云计算有限公司 | Log processing method and device, computing equipment and storage medium |
| CN113986862A (en) * | 2021-10-27 | 2022-01-28 | 上海星融汽车科技有限公司 | Client log collection method |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116095180A (en) * | 2023-03-07 | 2023-05-09 | 天翼云科技有限公司 | Log return routing method, device and storage medium |
| CN116095180B (en) * | 2023-03-07 | 2023-06-23 | 天翼云科技有限公司 | Log return routing method, device and storage medium |
| WO2024183402A1 (en) * | 2023-03-07 | 2024-09-12 | 天翼云科技有限公司 | Log backhaul routing method and apparatus, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115643166B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9804981B2 (en) | Method, controller, and system for service flow control in object-based storage system | |
| CN109542361B (en) | Distributed storage system file reading method, system and related device | |
| CN113687974B (en) | Client log processing method and device and computer equipment | |
| CN106027595A (en) | Access log processing method and system for CDN node | |
| CN115643166B (en) | Method and device for returning CDN log with high reliability | |
| US20190363961A1 (en) | Content Delivery Network Server Testing | |
| CN105302801A (en) | Resource caching method and apparatus | |
| CN110661673B (en) | Heartbeat detection method and device | |
| CN111988387B (en) | Interface request processing method, device, equipment and storage medium | |
| CN111901396B (en) | Resource request response method, redirection server and decision distribution server | |
| CN112054923B (en) | Service request detection method, equipment and medium | |
| CN110855786B (en) | Processing method and device for network data request and computer storage medium | |
| CN116418893A (en) | Data management method and corresponding device | |
| CN107707373B (en) | A kind of dynamic resource access accelerated method based on API request | |
| CN113778810A (en) | Log collection method, device and system | |
| CN113055493A (en) | Data packet processing method, device, system, scheduling device and storage medium | |
| US20170223136A1 (en) | Any Web Page Reporting and Capture | |
| CN110798751B (en) | Data transmission method and equipment | |
| CN116723154A (en) | Route distribution method and system based on load balancing | |
| CN108063713B (en) | Network request processing method and device | |
| CN112131198B (en) | Log analysis method and device and electronic equipment | |
| CN114866488B (en) | Information flow identification method, network chip and network equipment | |
| EP3745680B1 (en) | Apparatus and method for transmitting content | |
| CN115118782A (en) | Data control method and device, electronic equipment and computer readable storage medium | |
| CN114301848A (en) | CDN-based communication method, system, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |