CN113497723A

CN113497723A - Log processing method, log gateway and log processing system

Info

Publication number: CN113497723A
Application number: CN202010202716.5A
Authority: CN
Inventors: 黄若儒; 梁小一
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2020-03-20
Filing date: 2020-03-20
Publication date: 2021-10-12
Anticipated expiration: 2040-03-20
Also published as: CN113497723B

Abstract

The embodiment of the specification discloses a log processing method, a log gateway and cluster, a log processing system, an electronic commerce information processing method, an electronic commerce platform, a terminal device and a computer readable storage medium. The method comprises the following steps: receiving configuration updating information pushed by a configuration service platform through first message middleware; receiving a log request sent by an application client according to locally stored log reporting configuration information, wherein the log request is used for reporting log data; and sending response information to the log request to an application client, wherein the response information comprises the configuration updating information so that the application client can update the log reporting configuration information according to the configuration updating information. The log processing scheme is suitable for large-flow high-concurrency log requests.

Description

Log processing method, log gateway and log processing system

Technical Field

The present specification relates to a network technology, and more particularly, to a log processing method, a log gateway cluster, a log processing system, a method of processing electronic commerce information, an electronic commerce platform, a terminal device, and a computer-readable storage medium.

Background

In large data items, there is often a need to handle large flows of highly concurrent data requests. This is especially true for large data items that are log-oriented.

The log service process is: the application client reports logs (such as abnormal logs and crash logs) generated in the running process to the log processing system, the log processing system analyzes the logs to obtain a log analysis result, and the log analysis result can be used for multiple purposes, such as searching crash reasons of the application client, evaluating load and health condition of the server, analyzing behaviors of users and the like.

In the process, the process that the application client reports the log to the log processing system, namely the process that the application client submits the log request to the log processing system, has the characteristics of large flow and high concurrency. Firstly, the data volume of the log message is relatively large and much larger than that of a general Web request. Moreover, the log request has higher concurrency, for example, the log processing system may face a plurality of applications, provide log analysis service for the plurality of applications, and one application may be installed in hundreds of millions of terminal devices, so that the log processing system needs to process a large number of log requests simultaneously and in parallel. Daily traffic that the log processing system needs to process can reach the level of TB (Trillionbyte) and even PB (Petabyte).

In addition, electronic merchandise platforms are also exposed to high volume, high concurrency data requests in some specific business scenarios, such as ticket polling during the spring festival.

Therefore, it is necessary to provide a processing scheme for high-traffic and high-concurrency data requests, which can well cope with the high-traffic and high-concurrency data requests.

Disclosure of Invention

The embodiment of the specification provides a data request processing scheme, which is suitable for a large-flow high-concurrency data request scene.

According to a first aspect of the embodiments of the present invention, there is provided a log processing method, including:

receiving configuration updating information pushed by a configuration service platform through first message middleware;

receiving a log request sent by an application client according to locally stored log reporting configuration information, wherein the log request is used for reporting log data;

and sending response information to the log request to an application client, wherein the response information comprises the configuration updating information so that the application client can update the log reporting configuration information according to the configuration updating information.

The log processing method provided in the first aspect of the embodiments of the present invention may be implemented by a log gateway.

According to a second aspect of the embodiments of the present invention, there is provided a log processing method, including:

sending a log request to a log gateway according to locally stored log reporting configuration information, wherein the log request is used for reporting log data;

receiving response information returned by the log gateway, wherein the response information comprises configuration updating information;

and updating the log reporting configuration information according to the configuration change information.

The log processing method provided by the second aspect of the embodiment of the present invention may be implemented by an application client.

According to a third aspect of the embodiments of the present invention, there is provided a log processing method, including the steps of:

the log gateway receives configuration updating information pushed by a configuration service platform through first message middleware;

an application client sends a log request to a log gateway according to locally stored log reporting configuration information, wherein the log request is used for reporting log data;

the log gateway sends response information to the log request to an application client, wherein the response information comprises the configuration updating information;

and the application client updates the log reporting configuration information according to the configuration change information.

According to a fourth aspect of embodiments of the present invention, there is provided a logging gateway having a memory and a processor; the memory has stored therein computer readable instructions which, when executed by the processor, implement the method provided by the first aspect of the invention.

According to a fifth aspect of the embodiments of the present invention, there is provided a log gateway cluster including a plurality of log gateways provided in the fourth aspect of the present invention.

According to a sixth aspect of the embodiments of the present invention, there is provided a log processing system, including the log gateway cluster, the configuration service platform, and the log analysis processing platform provided in the fifth aspect of the present invention.

According to a seventh aspect of the embodiments of the present invention, there is provided a terminal device equipped with an application client, where the application client is provided with a program compilation file; the application client is configured to perform the method provided by the second aspect of the invention.

According to an eighth aspect of embodiments of the present invention, there is provided a computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method provided by the first aspect of the present invention.

According to a ninth aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method provided by the second aspect of the present invention.

According to a tenth aspect of the embodiments of the present invention, there is provided a method for processing electronic commerce information, the method including:

receiving a commodity query request sent by an application client according to locally stored commodity query configuration information;

and sending response information to the commodity query request to an application client, wherein the response information comprises the configuration updating information, so that the application client can update the commodity query configuration information according to the configuration updating information.

The method for processing e-commerce information provided by the tenth aspect of the embodiment of the present invention may be implemented by a commodity inquiry server.

According to an eleventh aspect of the embodiments of the present invention, there is provided an electronic commerce information processing method, including:

sending a commodity query request to a commodity query server according to the locally stored commodity query configuration information;

receiving response information returned by the commodity inquiry server, wherein the response information comprises configuration updating information;

and updating the commodity inquiry configuration information according to the configuration change information.

The method for processing e-commerce information provided by the eleventh aspect of the embodiment of the present invention may be implemented by an application client.

According to a twelfth aspect of the embodiment of the present invention, an electronic commerce platform is provided, which includes a commodity inquiry server and a configuration service platform; the commodity inquiry server is provided with a memory and a processor; the memory has stored therein computer readable instructions which, when executed by the processor, implement the method provided by the tenth aspect of the invention.

According to a thirteenth aspect of the embodiments of the present invention, there is provided a terminal device loaded with an application client configured to execute the method provided by the eleventh aspect of the present invention.

According to a fourteenth aspect of the embodiments of the present invention, there is provided a computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method provided by the tenth aspect of the present invention.

According to a fifteenth aspect of embodiments of the present invention, there is provided a computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method provided by the eleventh aspect of the present invention.

According to the log processing scheme provided by the embodiment of the invention, the application client sends the log request according to the locally stored log reporting configuration information, when a technician updates the configuration information according to the condition of receiving the log request by the system, the configuration update information is pushed to the log gateway in real time by using a message broadcasting mechanism, and the log gateway provides the configuration update information to the application client when responding to the log request, so that the application client can adjust the log reporting strategy according to the configuration update information in time. The log processing scheme provided by the embodiment of the invention can adjust the log reporting strategy of the application client in real time, so that the log processing system can cope with large-flow and high-concurrency log requests.

According to the processing scheme of the electronic commerce information provided by the embodiment of the invention, the application client sends the commodity query request to the commodity query server according to the commodity query configuration information stored locally, when the configuration information is updated according to the condition that the system receives the commodity query request, the configuration update information is pushed to the commodity query server in real time by using a message broadcasting mechanism, and the commodity query server provides the configuration update information to the application client when responding to the commodity query request, so that the application client can adjust the commodity query request sending strategy according to the configuration update information in time. The processing scheme of the electronic commerce information provided by the embodiment of the invention can adjust the commodity query request sending strategy of the application client in real time, so that the commodity query server can cope with the commodity query request with high flow rate and high concurrency.

Features of embodiments of the present specification and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the specification and together with the description, serve to explain the principles of the embodiments of the specification.

Fig. 1 is a block diagram of a log processing system provided in one embodiment of the present specification.

FIG. 2 is a flow diagram of a log processing method provided by one embodiment of the present specification;

FIG. 3 is a flow diagram of a log processing method provided by one embodiment of the present specification;

FIG. 4 is a flow diagram of a log processing method provided by one embodiment of the present specification;

FIG. 5 is a flow diagram of a log processing method provided by one embodiment of the present specification;

FIG. 6 is a schematic diagram of a log gateway provided by one embodiment of the present description;

FIG. 7 is a block diagram of an e-commerce platform provided by one embodiment of the present specification.

Fig. 8 is a flowchart of a method for processing e-commerce information according to an embodiment of the present disclosure.

Detailed Description

Various exemplary embodiments of the present specification will now be described in detail with reference to the accompanying drawings.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the embodiments, their application, or uses.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

With the rapid development of network population and network technology, more and more log-oriented big data services appear. For example, a shopping application may be installed in hundreds of millions of terminal devices, and log data generated during a peak shopping period is massive, and in order to ensure that a user purchases successfully, the massive log data needs to be processed and analyzed in real time. For example, a service provider dedicated to providing a log service simultaneously serves a plurality of applications, each of which is installed in a large number of terminal devices, and the log service provider may need to receive and analyze a large amount of log data of the plurality of applications at the same time.

In these cases, the log gateway needs to handle large flows of highly concurrent log requests. And, because the log gateway is the only data entry, the availability of the log gateway determines the availability of the whole subsequent link, and the log gateway is required to have high guarantee. Furthermore, since the related logging service is exposed on the internet, it is particularly important to secure the log data and the log gateway itself.

The embodiment of the specification aims to provide a log processing scheme, which can deal with large-flow and high-concurrency log requests and realize an extensible big data log gateway platform with certain universality.

Before describing the log processing scheme of the embodiment of the present specification, the following related terms are explained:

OpenResty: the high-performance Web platform based on Nginx and Lua is a piece of opening source software.

Nginx: is a high-performance HTTP and reverse proxy Web server, and also provides IMAP/POP3/SMTP service. Nginx has the property of handling HTTP/HTTPs requests in stages. The process of processing the HTTP/HTTPs request by the Nginx is totally divided into 11 stages, and the stages are a post-read stage (read request content stage), a server-write stage (server request address rewrite stage), a find-configuration stage (configuration search stage), a write stage (Location request address rewrite stage), a post-write stage (request address rewrite submit stage), a pre-access stage (access right check preparation stage), an access stage (access right check stage), a post-access stage (access right check submit stage), a try-files stage (configuration item processing stage), a content stage (content generation stage), and a log stage (log module processing stage) in sequence according to the execution order.

Lua: is a small script language designed to provide flexible extension and customization functions for an application by being flexibly embedded in the application.

Http (hypertext Transfer protocol): hypertext transfer protocol.

Https (secure Hypertext Transfer protocol): secure hypertext transfer protocol.

IMAP (Post Office Protocol 3): version 3 of the post office protocol.

POP3(Simple Mail Transfer Protocol): simple mail transfer protocol.

SMTP (Internet Mail Access protocol): interactive mail access protocol.

Html (hypertext Markup language): hypertext markup language.

Remote dictionary service (remote dictionary service): the system is an open-source high-performance and high-availability Key-Value Key Value type storage system, is realized based on a memory, and can be used as a database, a cache and message middleware.

Kafka: a high-throughput distributed publish-subscribe message system is an open-source high-performance stream processing platform.

SDK (Software Development Kit ): a kit for implementing a function of a software product.

Fig. 1 is a block diagram of a log processing system provided in an embodiment of the present specification. As shown in fig. 1, the log processing system includes a log gateway cluster, a service room for providing service for the log gateway cluster, a configuration service platform, and a log analysis processing platform. The log gateway cluster comprises a plurality of log gateways.

The log service provider provides log service by using a log processing system, the log processing system is used for providing log service for one or more applications, specifically, the log processing system receives logs from a terminal device carrying an application client, processes and analyzes the logs, and a log analysis result can be used for multiple purposes, such as finding a cause of a breakdown of the application client, evaluating load and health condition of a server corresponding to the application client, analyzing behaviors of a user, and the like.

Fig. 1 only schematically shows one terminal device, and in practice, the number of terminal devices may be large, and one terminal device may carry one or more application clients. The user implements application-related functions by accessing the corresponding server using the application client. The application may be, for example, a browser, shopping, payment, video, mail, instant messaging, etc. application. The application client may be, for example, an android application client, an IOS application client, an HTML application client, and the like, which is not limited in this embodiment of the present invention.

The application client side can be integrated with a program compiling file for sending the log request, and sends the log request to the log processing system by calling the program compiling file, wherein the log request is used for reporting log data. The application client sends the log request based on the HTTP/HTTPS protocol, namely the log request is the HTTP/HTTPS request. The program compilation file may be in the form of a log SDK.

The program compilation file can be provided for an application service provider by a log service provider, and the application service provider integrates the program compilation file into an application client for a user to download the application client integrated with the program compilation file.

The service room may include servers, routers, and the like. And the service machine room receives the log request sent by the terminal equipment and provides services such as HTTPS gateway service, web routing service, load balancing service and the like for the log gateway cluster. The HTTPS gateway service may include HTTPS certificate checking, etc. The Web routing service distributes the log request to the gateways in the log gateway cluster, and the load balancing service distributes the flow of the log request processed by the log gateways.

And the configuration service platform is mainly used for technicians to configure and manage the log gateway. The log gateway cluster comprises a plurality of log gateways, and the log gateways are used for preprocessing the log requests and providing the log data obtained through preprocessing to the log analysis processing platform. And the log analysis processing platform further processes and analyzes the log data to obtain a log analysis result. The log analysis processing platform can comprise a real-time computing platform to perform statistical updating on log data in real time.

The terminal device may be, for example, an electronic device installed with an intelligent operating system (e.g., android system, apple IOS system, microsoft Windows system, Linux, etc.), including but not limited to a laptop, a desktop computer, a mobile phone, a tablet computer, etc. One or more application clients can be loaded in the terminal equipment. The hardware configuration of the terminal device includes, but is not limited to, a processor, a memory, an interface device, a communication device, an input device, a display device, a speaker, a microphone, and a camera. The processor may include, but is not limited to, a central processing unit CPU, a microprocessor MCU, or the like. The memory may include, but is not limited to, ROM (read only memory), RAM (random access memory), non-volatile memory such as a hard disk, and the like. The interface means may include, but is not limited to, a USB interface, a serial interface, a parallel interface, etc. The communication means is capable of wired or wireless communication, for example, and may specifically include WiFi communication, bluetooth communication, 2G/3G/4G/5G communication, and the like. Input devices may include, but are not limited to, a keyboard, a mouse, a touch screen, and the like. The display device may include, but is not limited to, a liquid crystal screen, a touch screen, and the like. The hardware configuration of the terminal device may include only some of the above devices.

The configuration service platform, the log gateway and the log analysis processing platform can be built by adopting a server. The configuration of the server involved in the embodiment of the present invention may include, but is not limited to: processor, memory, interface device, communication device, input device, output device. The processor may include, but is not limited to, a central processing unit CPU, a microprocessor MCU, or the like. The memory may include, but is not limited to, ROM, RAM, non-volatile memory such as a hard disk, and the like. The interface means may include, but is not limited to, a USB interface, a serial interface, a parallel interface, etc. The communication means is capable of wired or wireless communication, for example, and may specifically include WiFi communication, bluetooth communication, 2G/3G/4G/5G communication, and the like. Input devices may include, but are not limited to, a keyboard, a mouse, and the like. The output device may include, but is not limited to, a display screen or the like. The server may be configured to include only some of the above devices.

The log processing system shown in FIG. 1 is illustrative only and is not intended to suggest any limitation as to the scope of use or use of embodiments of the specification. It should be understood by those skilled in the art that although the foregoing describes a plurality of apparatuses of the terminal device, the configuration service platform, the log gateway, and the log analysis processing platform, the embodiments of the present specification may only refer to some of the apparatuses. Those skilled in the art can design instructions based on the disclosed embodiments of the present specification. How the instructions control the operation of the processor is well known in the art and will not be described in detail herein.

Referring to fig. 1 and fig. 2, a log processing method according to an embodiment of the present invention is described below:

1002. and the application client sends a log request to the log processing system through the network according to the locally stored log reporting configuration information, wherein the log request is used for reporting log data.

In a specific example, a program compiling file for sending the log request is integrated in the application client, and the application client calls the program compiling file to configure and send the log request according to the log reporting configuration information. The program compiling file is also used for managing and maintaining log reporting configuration information. The program compilation file may be in the form of a log SDK.

The log reporting configuration information may include: request header configuration information, key configuration information, service flow limitation configuration information, log retransmission configuration information, log compression configuration information and log merging configuration information.

The request header configuration information is used for configuring the content in the request header of the log request, for example, the request header is required to include a signature, a service identifier, a timestamp, a service parameter, a domain name, and the like. The request body of the log request includes specific log data, and in the embodiment of the invention, all log data included in one log request is called one log data.

The signature is an authorized signature of the application client or the compiled file of the program for proving the identity of the log request.

The service identification is used for indicating the identity of the service, and each service has a unique service identification. A service is a service that an application can provide, and one application may contain a plurality of services. For example, a shopping application having a plurality of services such as account login, search for merchandise, collection, shopping cart, payment, etc. For example, an audio application with multiple services of account logging, playing music, searching for music, collecting music, purchasing music, etc.

The timestamp indicates the time at which the log request was generated.

The service parameters are key information of the log data related to the service. For example, key information of log data of a login service includes a user name; for example, the key information of the log data of the search commodity service includes a search keyword.

The domain name is the domain name to be accessed by the access activity that generated the log data.

The key configuration information includes a key for encrypting log data.

The service current limiting configuration information is used for limiting the reporting of log data of certain services. For example, the reporting of log data of the service B1 is prohibited; for example, the frequency of reporting log data of the service B2 is required to be not more than 5 times/second.

The log retransmission configuration information is used to restrict the repeated transmission of the log request. Generally, after the application client sends the request, if the response message is not received within a preset time, the application client will usually send the request again. In the embodiment of the invention, the configuration information is used for configuring the retransmission of the log request through the log retransmission. For example, an application client is prohibited from repeatedly sending log requests; for example, the application client is allowed to repeatedly send log requests; for example; prohibit the log request of service B1 from being sent repeatedly; for example, the log request of the service B2 is allowed to be repeatedly sent.

The log compression configuration information is used for configuring whether log data is compressed or not and an algorithm used for compression.

The log merging configuration information is used for configuring whether the log data are merged or not and the specific mode of merging.

In the embodiment of the invention, the application client can realize log compression reporting and log merging reporting, thereby greatly reducing the flow consumption (the bandwidth can be saved by more than 2 times), simultaneously reducing unnecessary requests and reducing the service pressure of a log processing system.

1004. And after receiving the log request, the service room provides services such as HTTPS gateway service, web routing service, load balancing service and the like.

The HTTPS gateway service may include HTTPS certificate checking on HTTP requests, and the like.

The Web routing service distributes the log request to the gateways in the log gateway cluster. In one specific example, the log gateway is pre-configured with the domain name it serves. For example, a logging gateway cluster includes a logging gateway P₁，P₂，…，P_N，P_N+1，…，P_M，P_M+1，…P_L，P_L+1…. The domain name corresponding to the application A1 comprises C1, the domain name corresponding to the application A2 comprises C2 and C3, and the log gateway P₁，P₂，…，P_NFor providing a logging service for a domain name C1, a logging gateway P_N+1，…，P_MFor providing a logging service for a domain name C2, a logging gateway P_M+1，…，P_LFor providing a logging service for domain name C3. The service room can obtain the domain name from the request header of the log request, and determine the log gateway providing service for the service room according to the domain name.

The load balancing service is used for distributing the flow of the log gateway processing the log request. For example, the log gateway P₁，P₂，…，P_NFor providing a logging service for domain name C1, a load balancing service evenly distributes log requests corresponding to domain name C1 to log gateways P₁，P₂，…，P_NTherefore, the comprehensive response speed of the log gateway cluster is improved, and the failure of the log gateway caused by overlarge data flow is avoided.

In the embodiment of the invention, the monitoring node can be arranged on the layer of the service room to carry out operation and maintenance monitoring on the work of the service room so as to ensure the availability of the log processing system.

1006. And the log gateway receives the log request and preprocesses the log request.

Referring to fig. 2, the pre-treatment in an embodiment of the present invention may include two stages. First, the first stage of the pretreatment is explained:

after receiving the log request, the log gateway acquires information such as a signature, a service identifier, a timestamp, a service parameter, a domain name and the like from a request header of the log request.

The log gateway checks the service identifier, and in a specific example, the checking of the service identifier includes checking whether the data type, the length, and the like of the service identifier meet preset rules. For example, according to a preset rule, the service identifier should adopt numbers, cannot adopt character strings or floating point numbers, and the length of the service identifier should be 10 bits. If the service identifier acquired by the log gateway from the request header conforms to a preset rule, the service identifier passes the inspection; if the service identification acquired by the log gateway from the request header does not accord with the preset rule, the service identification does not pass the inspection, the log gateway stops the subsequent processing steps, and the log request is abandoned.

In one embodiment, the configuration service platform includes a database, service verification information corresponding to the service is stored in the database, and technicians can add, modify, delete and the like the service verification information in the database through the configuration service platform. And after the service identification passes the verification, the log gateway acquires service verification information corresponding to the service identification through the first message middleware according to the service identification. The first message middleware can be Redis middleware, and the database is the weakest link of concurrent access in a high concurrency scene. The first message middleware may also employ kafka middleware.

The service verification information may include signature verification information, timestamp verification information, service parameter verification information, a key required for decrypting log data, an algorithm used for unpacking, and the like.

The time stamp in the request header of the log request is used to indicate the time at which the log request was generated, and is a type of complete verifiable data that can indicate that a copy of the data already exists at a particular point in time. In the embodiment of the invention, the log gateway verifies the signature acquired from the request header of the log request by using the signature verification information, verifies the timestamp acquired from the request header of the log request by using the timestamp verification information so as to judge whether the identity of the log request is legal, whether the log request is tampered and the like, and stops the subsequent processing steps and abandons the log request as long as any one of the verifications fails. In addition, if the time between the time indicated by the timestamp acquired from the request header of the log request and the time when the log gateway receives the log request is too long, the log request is considered invalid, and the timestamp check is failed, so that the same log request which is repeatedly sent by the application client for multiple times is prevented from being repeatedly processed by the log gateway, and the waste of log gateway resources is avoided.

The service parameter checking information indicates which service parameters should exist, the log gateway checks the service parameters acquired from the request header of the log request by using the service parameter checking information, if the service parameters acquired from the request header of the log request are matched with the service parameters indicated by the service checking parameters, the service parameter checking is passed, otherwise, the service parameter checking is not passed. For example, the service parameter indicating information in the service parameter check information includes a user name, and if the service parameter obtained from the request header of the log request does not include the user name, the service parameter check fails.

In this embodiment, the log gateway performs the multiple verifications on the log request, and if any one of the multiple verifications fails, that is, it indicates that the log request fails to pass the verification, stops the subsequent processing steps, and discards the log request.

In this embodiment, the log gateway has a message restriction function, and refuses to receive log data reported by the log request when the log request does not pass the verification. The HTTP/HTTPs request is actually sent in multiple portions, and the log gateway in the embodiment of the present invention is implemented using an OpenResty platform, where OpenResty is a high-performance Web platform based on Nginx and Lua, so that the log gateway has a capability of the Nginx to perform staged processing on the HTTP/HTTPs request. In this embodiment, by using the feature of processing in one stage, the log gateway performs operations of reading and analyzing the request header, verifies the log request by using information in the request header, and if the log request fails to pass the verification, the log gateway refuses to receive log data reported by the log request. In a specific example, the log gateway is configured to complete the aforementioned checking operation in an access phase (access permission checking phase), and if the log request fails to pass the checking operation, the log gateway terminates the processing in the subsequent phase and refuses to receive the log data reported by the log request, so as to reduce traffic loss and improve processing efficiency.

In an embodiment, the database of the configuration service platform further stores log reporting configuration information, and a technician can update the log reporting configuration information in the database through the configuration service platform when needed. For example, if the current flow of the log request is too large, the log gateway cluster is difficult to load, a technician may change the log retransmission configuration information from allowing retransmission to prohibiting retransmission, may modify the service flow limiting configuration information to limit the flow of the service log request of some services, and the like. For example, the technician may make modifications to the request header configuration information. For example, a technician may modify the key configuration information to update the key that encrypted the log data.

And after the configuration service platform acquires the specific configuration updating information, the configuration updating information is pushed to the log gateway in the cluster in real time through the first message middleware. That is, in the embodiment of the present invention, a technician does not need to write new log reporting configuration information to the log gateways one by one, and the log gateways do not need to pull the latest log reporting configuration information from the database at regular time, but the message broadcasting mechanism of the first message middleware is used to actively push the configuration update information to the log gateways in the cluster in real time. The method can reduce the workload of technicians, realize the cache real-time refreshing capability of an application layer, and improve the real-time performance of configuration change response.

In the embodiment of the present invention, through the above message broadcasting mechanism, the log gateway may obtain the latest service current limiting configuration information, and after the log gateway obtains the service identifier from the request header of the log request, the log gateway may determine, according to the latest service current limiting configuration information, whether the service corresponding to the service identifier belongs to the service for which the log reporting is prohibited, before the service identifier is checked, or after the service identifier passes the check, and if the service corresponding to the service identifier belongs to the service for which the log reporting is prohibited, refuse to receive the log data reported by the log request, terminate the subsequent processing, so as to reduce the flow loss and improve the processing efficiency.

Under the condition that the log request passes the verification, a plurality of pieces of log data corresponding to the same service identifier can be sampled, and part of the log data can be reserved. The technical personnel can set the situation according to the actual scene requirement, for example, hundreds of millions of log requests can be generated in a short time by a service, the log analysis processing platform wants to analyze and count the probability of occurrence of the service, in this case, the log gateway can sample the log data of the service, only part of the log data is reserved after sampling, and the statistical analysis can be performed by using the reserved log data without influencing the final result. In a specific example, what traffic log data needs to be sampled specifically and in a specific sampling manner may also be set by a technician through the configuration service platform and pushed to the log gateway through the first message middleware. The log gateway in the embodiment of the invention has the data sampling capability, can reduce the data inflow while ensuring that the business data statistics is not influenced, and saves the computing and storing resources.

After passing through the first stage of pre-processing, the retained log data proceeds to the second stage of pre-processing:

the log gateway decrypts the log data by using the key and unpacks the decrypted log data by using an unpacking algorithm. And performing data processing on the unpacked log data according to a preset data processing rule, for example, deleting sensitive information in the log data corresponding to a certain service identifier. According to the service identification, the log data after data processing are merged, so that each piece of log data does not need to be processed respectively in the following process, the execution times of the following steps are saved, and the data volume is further reduced. Then, the log data subjected to the aforementioned processing is subjected to Serialization (Serialization), which is a process of converting the log data into a storable form.

In the embodiment of the invention, after the preprocessing, the log data volume is reduced, which is beneficial to saving the calculated amount of the log analysis processing platform.

In the embodiment of the invention, the work of the log gateway cluster can be monitored so as to ensure the availability of the log processing system.

1008. And the log gateway sends the log data obtained through the preprocessing to a log analysis processing platform.

The log gateway caches the preprocessed log data by using the second message middleware, and forwards the log obtained after preprocessing to the log analysis processing platform, so that the log gateway does not need to store locally, and the security of the log data is improved.

In a specific example, a plurality of analysis processing units are preset in the log analysis processing platform, the analysis processing units subscribe the service identifier of the log data that the log gateway wants to analyze to the second message middleware, and the log gateway sends the log data to the analysis processing units subscribed with the service identifier through the second message middleware according to the service identifier.

In this embodiment of the present invention, the Second message middleware may be Kafka message middleware, and the distributed publish-subscribe message middleware with high throughput, such as Kafka, is used to cache and relay log data to the log analysis processing platform, so that the response time of the log gateway to the log analysis processing platform can be reduced, and the query rate Per Second (QPS) and the transaction number Per Second (TPS) of the system can be improved.

1010. After passing through step 1006 or after passing through step 1008, the log gateway platform sends response information to the log request to the application client.

In the embodiment of the invention, the response information comprises configuration updating information. And after receiving the configuration updating information, the application client updates the log reporting configuration information of the application client according to the configuration updating information. For example, the service current limiting configuration information and the log retransmission configuration information in the log reporting configuration information are updated according to the configuration update information, the application client is prohibited from repeatedly sending log requests, the log requests of a certain service are prohibited from being sent, and the like, and the latest requirements of the configuration service platform for log reporting are timely responded. For example, the key for encrypting the log data is updated according to the configuration updating information, so that the security of log data transmission is further improved.

The application client can integrate a program compiling file for sending the log request, and updates the log reporting configuration information by calling the program compiling file. The program compilation file may be in the form of a log SDK.

According to the embodiment of the invention, the log request is submitted at the application client according to the locally stored log reporting configuration information, the log reporting configuration information of the application client is timely updated by utilizing the response information of the log gateway to the log request in combination with the real-time update of the reporting configuration information by the configuration service platform through the first message middleware, so that the log processing system has higher processing performance, better safety protection and lower deployment operation cost.

In the embodiment of the invention, the log gateway can realize linkage with the application client by utilizing the response information, thereby improving the real-time performance of configuration change without influencing the overall service performance.

1012. And the log analysis processing platform further processes and analyzes the log data to obtain a log analysis result.

In a specific example, the log analysis processing platform may include a real-time computing platform to perform statistical updates on the log data in real time, such as counting the concurrency amount of the log data. The analysis processing unit of the log analysis processing platform may use log data analysis to search for the crash reason of the application client, evaluate the load and health condition of the server corresponding to the application client, and analyze the behavior of the user, and so on, which is not described herein again.

The log processing method provided by the embodiment of the invention can be realized based on an open source free technology, wherein the log gateway is realized by using an OpenResty technology, Redis middleware is used between the log gateway and the configuration service platform for information intercommunication, and the log gateway distributes log data to the log analysis processing platform through Kafka message middleware.

According to the log processing method provided by the embodiment of the invention, the OpenResty technology is selected as the core foundation of the log gateway, and the performance is very excellent in the input/output intensive request scene, so that the processing capacity of the log gateway cluster on large-flow high-concurrency log requests is realized. The log processing method provided by the embodiment of the invention reduces the construction and maintenance costs of the log gateway cluster by improving the processing capacity of the single log gateway node,

the log processing system provided by the embodiment of the invention is provided with the configuration service platform, so that the log gateway system has the configurable and expandable capability, and other variant services can be easily realized according to the service form.

According to the log processing method provided by the embodiment of the invention, the security level of log service is improved by combining various strategies including log request signature, log data encryption, no local storage and the like.

Referring to fig. 3, according to an embodiment of the present invention, there is provided a log processing method implemented by a log gateway, including the following steps:

s102, receiving configuration updating information pushed by a configuration service platform through first message middleware.

And S104, receiving a log request sent by the application client according to the locally stored log reporting configuration information, wherein the log request is used for reporting log data.

And S106, sending response information to the log request to the application client, wherein the response information comprises configuration updating information so that the application client can update the log according to the configuration updating information and report the configuration information.

Referring to fig. 4, according to an embodiment of the present invention, there is provided a log processing method implemented by an application client, including the following steps:

s202, sending a log request to the log gateway according to the log reporting configuration information stored locally, wherein the log request is used for reporting log data.

And S204, receiving response information returned by the log gateway, wherein the response information comprises configuration updating information.

And S206, updating the log reporting configuration information according to the configuration change information.

Referring to fig. 5, according to an embodiment of the present invention, there is provided a log processing method, including the steps of:

s302, the log gateway receives configuration updating information pushed by the configuration service platform through the first message middleware.

S304, the application client sends a log request to the log gateway according to the locally stored log reporting configuration information, wherein the log request is used for reporting log data.

S306, the log gateway sends response information to the log request to the application client, wherein the response information comprises configuration updating information.

And S308, the application client updates the log to report the configuration information according to the configuration change information.

Referring to fig. 6, according to an embodiment of the present invention, there is provided a logging gateway having a memory 12 and a processor 11; the memory 12 has stored therein computer readable instructions which, when executed by the processor 11, implement any of the foregoing methods performed by the logging gateway.

According to an embodiment of the present invention, a log gateway cluster is provided, including any one of the log gateways described above.

According to the embodiment of the invention, the log processing system is provided and comprises the log gateway cluster, the configuration service platform and the log analysis processing platform.

According to an embodiment of the present invention, there is provided a terminal device loaded with an application client configured to execute any one of the foregoing methods executed by the application client.

According to an embodiment of the present invention, there is provided a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement any of the foregoing methods performed by a logging gateway.

According to an embodiment of the present invention, there is provided a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement any of the foregoing methods performed by an application client.

Similar to the log request processing scheme, the embodiment of the invention also provides an electronic commerce information processing scheme, which can be suitable for a large-flow high-concurrency commodity query request scene.

FIG. 7 is a block diagram of an e-commerce platform provided by embodiments of the present specification. As shown in FIG. 7, the e-commerce platform includes a cluster of query servers and a configuration service platform. The query server cluster comprises one or more commodity query servers.

Fig. 7 only schematically shows one terminal device, and in practice, the number of terminal devices may be large, and one terminal device may carry one or more application clients. The user accesses the commodity query server by using the application client to realize the corresponding commodity query function. The application client may be, for example, an android application client, an IOS application client, an HTML application client, and the like, which is not limited in this embodiment of the present invention.

The commodity providing party provides commodity inquiring service by utilizing a commodity inquiring server, the commodity inquiring server is used for providing the commodity inquiring service for one or more applications, specifically, the commodity inquiring server receives a commodity inquiring request from a terminal device carrying an application client, inquires according to the commodity inquiring request to obtain an inquiring result, and responds to the inquiring request to feed back the inquiring result to the application client. For example, the goods provider is a train ticket official sales center for providing ticket selling services and remaining ticket inquiring services to a variety of different applications, and the application may be, for example, an application dedicated to providing a ticket reservation service, or an application providing a travel integrated service, and the like.

The service room may include servers, routers, and the like. And the service machine room receives the commodity query request sent by the terminal equipment and provides services such as HTTPS gateway service, load balancing service and the like for the query server cluster. The HTTPS gateway service may include HTTPS certificate verification and the like, and the load balancing service distributes traffic of the commodity query server processing the commodity query request.

And the configuration service platform is mainly used for the operation of configuring and managing the commodity query server by operators.

The application client side can integrate a program compiling file used for sending the commodity query request, the application client side sends the commodity query request to the commodity query server by calling the program compiling file, the program compiling file can be provided for a specific application service provider by a commodity provider, and the application service provider integrates the program compiling file in the application client side so that a user can download the application client side integrated with the program compiling file. The program compilation file may be in the form of a commodity query SDK.

The commodity query server and the configuration service platform can be built by adopting the server. The configuration of the server involved in the embodiment of the present invention may include, but is not limited to: processor, memory, interface device, communication device, input device, output device. The processor may include, but is not limited to, a central processing unit CPU, a microprocessor MCU, or the like. The memory may include, but is not limited to, ROM, RAM, non-volatile memory such as a hard disk, and the like. The interface means may include, but is not limited to, a USB interface, a serial interface, a parallel interface, etc. The communication means is capable of wired or wireless communication, for example, and may specifically include WiFi communication, bluetooth communication, 2G/3G/4G/5G communication, and the like. Input devices may include, but are not limited to, a keyboard, a mouse, and the like. The output device may include, but is not limited to, a display screen or the like. The server may be configured to include only some of the above devices.

The e-commerce platform shown in FIG. 7 is merely illustrative and is in no way intended to suggest any limitation as to the scope of embodiments of the specification, their application, or use. It should be understood by those skilled in the art that although the foregoing describes a plurality of devices of the terminal equipment, the configuration service platform and the goods inquiry server, the embodiments of the present specification may only refer to some of the devices. Those skilled in the art can design instructions based on the disclosed embodiments of the present specification. How the instructions control the operation of the processor is well known in the art and will not be described in detail herein.

Referring to fig. 7 and 8, a method for processing e-commerce information according to an embodiment of the present invention is described below:

2002. and the application client sends a commodity inquiry request to the electronic commerce platform through the network according to the locally stored commodity inquiry configuration information.

In a specific example, a program compiling file for sending the commodity query request is integrated in the application client, and the application client calls the program compiling file to configure and send the commodity query request according to the commodity query configuration information. The program compiling file is also used for managing and maintaining commodity inquiry configuration information. The program compilation file may be in the form of a commodity query SDK.

The goods inquiry configuration information may include: request header configuration information, service flow limitation configuration information, query request retransmission configuration information, and query request compression configuration information.

The request header configuration information is used for configuring the content in the request header of the commodity query request, for example, the request header is required to include a signature, a service identifier, a timestamp, a service parameter, and the like. The request body of the commodity query request comprises specific query request content.

The signature is an authorized signature of the application client or the program compiled file and is used for proving the identity of the commodity inquiry request.

The service identification is used for indicating the identity of the service, and each service has a unique service identification. The e-commerce platform may provide a variety of different services, such as a vehicle number inquiry service, a station-to-station inquiry service.

The time stamp indicates the time at which the merchandise inquiry request was generated.

Service parameters are key information related to a service. For example, key information of the train number inquiry service should include a train number.

The service flow restriction configuration information is used to restrict the flow of inquiry requests for certain services. For example, the frequency of inquiry requests of the service B2 is required to be not more than 30 times/hour.

The query request retransmission configuration information is used to restrict repeated transmission of the commodity query request. Generally, after the application client sends the request, if the response message is not received within a preset time, the application client will usually send the request again. In the embodiment of the invention, the configuration information is retransmitted through the inquiry request to configure retransmission of the commodity inquiry request. For example, the application client is prohibited from repeatedly sending commodity query requests; for example, the application client is allowed to repeatedly send commodity query requests; for example; prohibiting the commodity inquiry request of the service B1 from being repeatedly sent; for example, the commodity inquiry request of the service B2 is allowed to be repeatedly sent.

The query request compression configuration information is used for configuring whether the query request is compressed or not and an algorithm used by compression.

In the embodiment of the invention, the application client can compress and send the commodity inquiry request, thereby greatly reducing the flow consumption (the bandwidth can be saved by more than 2 times).

2004. And after receiving the commodity query request, the service room provides services such as HTTPS gateway service, web routing service, load balancing service and the like.

The load balancing service is used for distributing the flow of the commodity query request processed by the commodity query server so as to improve the comprehensive response speed of the query server cluster and avoid the failure of the query server caused by overlarge data flow.

2006. And the commodity query server receives the commodity query request and preprocesses the commodity query request.

Referring to fig. 8, after receiving the product query request, the product query server obtains information such as a signature, a service identifier, a timestamp, and a service parameter from a request header of the product query request.

The commodity query server verifies the service identifier, and in a specific example, the verification of the service identifier includes verifying whether the data type, the length and the like of the service identifier meet preset rules. For example, according to a preset rule, the service identifier should adopt numbers, cannot adopt character strings or floating point numbers, and the length of the service identifier should be 10 bits. If the service identification acquired by the commodity inquiry server from the request head accords with the preset rule, the service identification passes the inspection; if the service identification acquired by the commodity inquiry server from the request header does not accord with the preset rule, the service identification is not checked to pass, and the commodity inquiry server stops the subsequent processing steps and abandons the commodity inquiry request.

In one embodiment, the configuration service platform includes a database, service verification information corresponding to the service is stored in the database, and technicians can add, modify, delete and the like the service verification information in the database through the configuration service platform. And after the service identification passes the verification, the commodity inquiry server acquires service verification information corresponding to the service identification through the first message middleware according to the service identification. The first message middleware can be Redis middleware, and the database is the weakest link for concurrent access in a high concurrency scene. The first message middleware may also employ kafka middleware.

The service verification information may include signature verification information, timestamp verification information, service parameter verification information, and the like.

The time stamp in the request header of the goods inquiry request is used for indicating the time when the goods inquiry request is generated, and the time stamp is complete verifiable data which can indicate that a piece of data exists at a specific time point. In the embodiment of the invention, the commodity inquiry server verifies the signature acquired from the request header of the commodity inquiry request by using the signature verification information, verifies the timestamp acquired from the request header of the commodity inquiry request by using the timestamp verification information so as to judge whether the identity of the commodity inquiry request is legal, whether the commodity inquiry request is falsified and the like, and stops subsequent processing steps and abandons the commodity inquiry request as long as any one of the verifications is not passed. In addition, if the time between the time indicated by the timestamp acquired from the request header of the commodity query request and the time when the commodity query server receives the commodity query request is too long, the commodity query request is considered invalid, and the timestamp check is failed, so that the same commodity query request which is repeatedly sent by the application client side for many times is prevented from being repeatedly processed by the commodity query server, and the waste of resources of the commodity query server is avoided.

The business parameter checking information indicates which business parameters should exist, the commodity inquiry server uses the business parameter checking information to check the business parameters acquired from the request head of the commodity inquiry request, if the business parameters acquired from the request head of the commodity inquiry request are matched with the business parameters indicated by the business checking parameters, the business parameter checking is passed, otherwise, the business parameter checking is not passed. For example, if the service parameter indicating information in the service parameter checking information includes a vehicle number, and the service parameter indicating information in the service parameter checking information includes a vehicle number, the service parameter checking fails.

In this embodiment, the commodity inquiry server performs the above-mentioned multiple verifications on the commodity inquiry request, and if any one of the multiple verifications fails, that is, it indicates that the commodity inquiry request fails to be verified, the subsequent processing step is stopped, and the commodity inquiry request is discarded.

In one embodiment, the database of the configuration service platform further stores commodity query configuration information, and a technician can update the commodity query configuration information in the database through the configuration service platform when needed. For example, if the flow of the current commodity query request is too large, the commodity query server cluster is difficult to load, a technician may change the query request retransmission configuration information from retransmission permission to retransmission prohibition, may modify the service flow limiting configuration information to limit the flow of the commodity query requests of certain services, and the like.

And after the configuration service platform acquires the specific configuration updating information, the configuration updating information is pushed to the commodity query server in the cluster in real time through the first message middleware. That is, in the embodiment of the present invention, it is not necessary for a technician to write new commodity query configuration information into the commodity query server one by one, and it is not necessary for the commodity query server to pull the latest commodity query configuration information from the database at regular time, but the configuration update information is actively pushed to the commodity query servers in the cluster in real time by using the message broadcasting mechanism of the first message middleware. The method can reduce the workload of technicians, realize the cache real-time refreshing capability of an application layer, and improve the real-time performance of configuration change response.

2008. And the commodity query server platform sends response information to the commodity query request to the application client.

After the verification in step 2006 is passed, the commodity query server performs query work according to the commodity query request to obtain a query result, and sends response information to the application client, where the response information includes the query result.

In the embodiment of the present invention, the response information may further include configuration update information. And after the application client receives the configuration updating information, updating the commodity query configuration information of the application client according to the configuration updating information. For example, the query request retransmission configuration information in the commodity query configuration information is updated according to the configuration update information, the application client is prohibited from repeatedly sending the commodity query request, and the latest requirement of the configuration service platform for the query request is responded.

The application client can integrate a program compiling file for sending the commodity query request, and updates the commodity query configuration information by calling the program compiling file.

According to the embodiment of the invention, the commodity query request is submitted at the application client according to the locally stored commodity query configuration information, the commodity query configuration information of the application client is updated in real time by combining the configuration service platform through the first message middleware, and the commodity query configuration information of the application client is updated in time by utilizing the response information of the commodity query server to the commodity query request, so that the query server cluster has higher processing performance, better safety protection and lower deployment operation cost.

In the embodiment of the invention, the commodity query server can realize linkage with the application client by utilizing the response information, so that the real-time property of configuration change is improved, and the overall service performance is not influenced.

The method for processing the electronic commerce information provided by the embodiment of the invention can be realized based on an open source free technology, wherein the commodity query server is realized by using an OpenResty technology, and Redis middleware is used between the commodity query server and the configuration service platform for information intercommunication.

The method for processing the electronic commerce information selects the OpenResty technology as the core foundation of the commodity query server, and has excellent performance in the input/output intensive request scene, so that the processing capacity of a commodity query server cluster on large-flow high-concurrency commodity query requests is realized. According to the method provided by the embodiment of the invention, the processing capacity of the single commodity query server node is improved, and the construction and maintenance cost of the commodity query server cluster is reduced.

The electronic commerce platform provided by the embodiment of the invention is provided with a configuration service platform and a commodity query server cluster, and the commodity query server cluster has configurable and expandable capability and can easily realize other variant services according to business forms.

According to the embodiment of the invention, the method for processing the electronic commerce information is implemented by a commodity inquiry server and comprises the following steps:

and sending response information to the commodity query request to the application client, wherein the response information comprises configuration updating information so that the application client can update the commodity query configuration information according to the configuration updating information.

According to the embodiment of the invention, the method for processing the electronic commerce information is implemented by an application client, and comprises the following steps:

According to an embodiment of the present invention, there is provided a method for processing electronic commerce information, the method including:

the commodity query server receives configuration update information pushed by the configuration service platform through first message middleware;

the application client sends a commodity query request to a commodity query server according to the locally stored commodity query configuration information;

the commodity query server sends response information to the commodity query request to the application client, wherein the response information comprises configuration updating information so that the application client can update the commodity query configuration information according to the configuration updating information;

and the application client updates the commodity query configuration information according to the configuration change information.

According to an embodiment of the present invention, there is provided a commodity inquiry server having a memory and a processor; the memory has stored therein computer readable instructions that, when executed by the processor, implement any of the foregoing methods performed by the merchandise query server.

According to an embodiment of the present invention, a commodity query server cluster is provided, including any one of the commodity query servers described above.

According to the embodiment of the invention, an electronic commerce platform is provided, which comprises the commodity inquiry server cluster and the configuration service platform.

According to an embodiment of the present invention, there is provided a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement any of the foregoing methods performed by an article query server.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device and apparatus embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for relevant points.

The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

Embodiments of the present description may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement aspects of embodiments of the specification.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

Computer program instructions for carrying out operations for embodiments of the present description may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), can execute computer-readable program instructions to implement various aspects of embodiments of the present specification by utilizing state information of the computer-readable program instructions to personalize the electronic circuit.

Aspects of embodiments of the present specification are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present description. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, by software, and by a combination of software and hardware are equivalent.

The foregoing description of the embodiments of the present specification has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method of log processing, the method comprising:

2. The method of claim 1, wherein a program compilation file is integrated into the application client, and the application client sends the log request and updates the log reporting configuration information by calling the program compilation file.

3. The method of claim 1, wherein the log reporting configuration information at least comprises one of the following configuration information:

service flow limiting configuration information, log retransmission configuration information, log compression configuration information, log merging configuration information, key configuration information and request header configuration information.

4. The method according to any one of claims 1-3, further comprising:

acquiring a service identifier from a request header of the log request, and loading service verification information corresponding to the service identifier through a first message middleware according to the service identifier;

and checking the log request according to the service checking information.

5. The method of claim 4, the request header further comprising a signature, a timestamp, a traffic parameter;

the service verification information includes: signature verification information, timestamp verification information and service parameter verification information.

6. The method of claim 4, the traffic verification information further comprising a key and a unpacking algorithm required to decrypt log data.

7. The method of claim 4, further comprising:

refusing to receive the log data reported by the log request under the condition that the log request does not pass the verification; and/or the presence of a gas in the gas,

and judging whether the service identifier is the service identifier of the service for which the log is forbidden to be reported according to the service current limiting configuration information, and if so, refusing to receive the log data requested to be reported by the log.

8. The method of claim 4, further comprising:

and sampling a plurality of pieces of log data corresponding to the same service identifier, and reserving part of the log data.

9. The method of claim 4, further comprising:

and merging the plurality of pieces of log data corresponding to the same service identifier.

10. The method of claim 1, further comprising:

and according to the service identifier, distributing the log data to a log analysis processing platform subscribed with the service identifier through a second message middleware.

11. The method of claim 10, the second message middleware being kafka middleware.

12. The method of claim 1, the first message middleware being Redis middleware.

13. The method of claim 1, wherein the log gateway is implemented using an OpenResty platform.

14. A method of log processing, the method comprising:

15. The method of claim 14, wherein a program compilation file is integrated into the application client, and the application client sends the log request and updates the log reporting configuration information by calling the program compilation file.

16. The method of claim 14, wherein the log reporting configuration information comprises at least one of the following configuration information:

17. The method of claim 14, wherein a request header of the log request comprises a service identifier, a signature, a timestamp, and a service parameter.

18. The method of claim 14, the application client being any one of: android application client, IOS application client and HTML application client.

19. A log processing method, comprising the steps of:

20. The method of claim 19, wherein a program compilation file is integrated into the application client, and the application client sends the log request and updates the log reporting configuration information by calling the program compilation file.

21. A logging gateway having a memory and a processor; the memory has stored therein computer readable instructions which, when executed by the processor, implement the method of any of claims 1-13.

22. A logging gateway cluster comprising a plurality of logging gateways of claim 21.

23. A log processing system comprising the log gateway cluster of claim 22, a configuration service platform, and a log analytics processing platform.

24. A terminal device carrying an application client is provided with a program compiling file; the application client is configured to perform the method of any of claims 14-18.

25. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method of any one of claims 1-13.

26. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method of any one of claims 14-18.

27. A method for processing e-commerce information, the method comprising:

28. The method of claim 27, the merchandise inquiry request comprising an inquiry request for a ticket.

29. The method of claim 27, wherein the merchandise inquiry configuration information comprises at least one of the following configuration information:

service flow limitation configuration information, query request retransmission configuration information, query request compression configuration information and request header configuration information.

30. The method according to any one of claims 27-29, further comprising:

acquiring a service identifier from a request header of the query request, and loading service verification information corresponding to the service identifier according to the service identifier through a first message middleware;

checking the query request according to the service checking information;

and sending the response information to the application client under the condition that the query request passes the verification.

31. The method of claim 30, the request header further comprising a signature, a timestamp, a traffic parameter;

32. A method for processing e-commerce information, the method comprising:

33. The method of claim 32, wherein the merchandise inquiry configuration information comprises at least one of the following configuration information:

34. The method as claimed in claim 32, wherein the request header of the goods inquiry request includes service identifier, signature, timestamp, and service parameter.

35. An electronic commerce platform comprises a commodity inquiry server and a configuration service platform; the commodity inquiry server is provided with a memory and a processor; the memory has stored therein computer-readable instructions that, when executed by the processor, implement the method of any of claims 27-31.

36. A terminal device hosting an application client configured to perform the method of any of claims 32-34.

37. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method of any of claims 27-31.

38. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method of any of claims 32-34.