CN110198333B - Data acquisition method and device, storage medium and electronic device - Google Patents
Data acquisition method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN110198333B CN110198333B CN201810349630.8A CN201810349630A CN110198333B CN 110198333 B CN110198333 B CN 110198333B CN 201810349630 A CN201810349630 A CN 201810349630A CN 110198333 B CN110198333 B CN 110198333B
- Authority
- CN
- China
- Prior art keywords
- address
- content server
- target data
- data
- url
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
Abstract
The invention discloses a data acquisition method and device, a storage medium and an electronic device. The method comprises the following steps: receiving a data request carrying a first URL (uniform resource locator) address, wherein the data request is used for requesting to acquire target data, the first URL address comprises a first address parameter, and the first address parameter is used for indicating an address of a content server for storing the target data; under the condition that the target data is not cached in the cache server, replacing the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address; and acquiring target data from the content server according to the second URL address. The invention solves the technical problem that the operation and maintenance cost of the configuration list is higher because the varnish reverse proxy needs to pre-configure the address list of the content server in the related technology.
Description
Technical Field
The invention relates to the field of computers, in particular to a data acquisition method and device, a storage medium and an electronic device.
Background
Currently, a varnish-based reverse proxy needs to pre-configure the address list of the back-end content server. After the http request of the front end arrives, varnish may select one from a preconfigured address list according to a load balancing algorithm or other matching rules and forward the request to the content server. Since varnish caches data with URLs as identifiers, basically no changes are made to the requesting URL, at most only some unnecessary parameters are removed. From the perspective of the user, the content server of the reverse proxy needs to be defined in the address list in advance, and the client request URL is the real request address, and varnish simply forwards the request URL to the selected content server.
However, since there are many clients using the video cloud service, and the backend domain names and parameters of different clients are different from each other, if the varnish reverse proxy method is used, there are two problems as follows:
(1) the backend content server domain names of all clients need to be configured in full. When a plurality of clients or domain names are updated, the full varnish configuration of all cache nodes needs to be modified, so that the operation and maintenance cost and risk are extremely high.
(2) When the client requests the on-demand content, the unique client backend domain name needs to be determined according to the request information, and a default load balancing algorithm cannot be used. If all back-ends are configured with different groups, it is theoretically feasible to parse requests and select corresponding content servers in the varnish configuration file, but the problem that the configuration file is huge and difficult to maintain is brought.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a data acquisition method and device, a storage medium and an electronic device, which are used for at least solving the technical problem that the operation and maintenance cost of a configuration list is higher because a varnish reverse proxy needs to configure an address list of a content server in advance in the related art.
According to an aspect of an embodiment of the present invention, there is provided a data acquisition method, including: receiving a data request carrying a first URL (uniform resource locator) address, wherein the data request is used for requesting to acquire target data, the first URL address comprises a first address and a first address parameter, the first address is used for indicating a cache server for caching the target data, and the first address parameter is used for indicating an address of a content server for storing the target data; under the condition that the target data is not cached in the cache server, replacing the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address; and acquiring the target data from the content server according to the second URL address.
According to another aspect of the embodiments of the present invention, there is also provided a data acquisition apparatus, including: a receiving unit, configured to receive a data request carrying a first URL address, where the data request is used to request to acquire target data, the first URL address includes a first address and a first address parameter, the first address is used to indicate a cache server that caches the target data, and the first address parameter is used to indicate an address of a content server that stores the target data; a replacing unit, configured to replace the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address when the target data is not cached in the cache server; and the acquisition unit is used for acquiring the target data from the content server according to the second URL address.
According to another aspect of the embodiments of the present invention, there is also provided a storage medium, in which a computer program is stored, wherein the computer program is configured to execute any one of the data acquisition methods in the embodiments of the present invention when the computer program runs.
According to another aspect of the embodiments of the present invention, there is also provided an electronic apparatus, including a memory and a processor, where the memory stores therein a computer program, and the processor is configured to execute any one of the data acquisition methods in the embodiments of the present invention through the computer program.
In the embodiment of the invention, after a data request carrying a first URL (uniform resource locator) address is received, the data request is used for requesting to acquire target data, the first URL address comprises a first address and a first address parameter, the first address is used for indicating a cache server for caching the target data, and the first address parameter is used for indicating an address of a content server for storing the target data; under the condition that target data are not cached in the cache server, replacing the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address; and acquiring the target data from the content server according to the second URL address. According to the embodiment of the invention, the first URL address carried in the data request is replaced by the address of the content server indicated by the first address parameter to obtain the second URL address, and then the target data is obtained from the content server according to the second URL address, so that the purpose of obtaining the requested target data can be achieved without pre-configuring the address list of the content server, and the technical problem that the operation and maintenance cost of the configuration list is high due to the fact that the address list of the content server needs to be pre-configured in the varnish reverse proxy in the related technology is solved, so that the operation and maintenance cost in the reverse proxy is reduced, and the technical effect of improving the efficiency of the reverse proxy is achieved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a schematic diagram of a hardware environment for a data acquisition method according to an embodiment of the invention;
FIG. 2 is a flow diagram of an alternative data acquisition method according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating the complete process of an alternative one-time varnish request, according to an embodiment of the invention;
FIG. 4 is a flow diagram of an alternative varnish-based dynamic reverse proxy, according to an embodiment of the invention;
FIG. 5 is a schematic diagram of an alternative varnish-based dynamic reverse proxy implementation code, according to an embodiment of the present invention;
FIG. 6 is an architecture diagram of an alternative ngnix reverse proxy, varnish cache, in accordance with embodiments of the present invention;
FIG. 7 is a schematic diagram of an alternative data acquisition device according to an embodiment of the present invention;
FIG. 8 is a schematic view of an alternative data acquisition device according to an embodiment of the present invention;
FIG. 9 is a schematic view of an alternative data acquisition device according to an embodiment of the present invention;
FIG. 10 is a schematic view of an alternative data acquisition device according to an embodiment of the present invention;
FIG. 11 is a schematic view of an alternative data acquisition device according to an embodiment of the present invention;
FIG. 12 is a schematic view of an alternative data acquisition device according to an embodiment of the present invention; and
fig. 13 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
According to an aspect of an embodiment of the present invention, a data acquisition method is provided.
Alternatively, in the present embodiment, the data acquisition method may be applied to a hardware environment formed by the server 102 and the terminal 104 as shown in fig. 1. As shown in fig. 1, a server 102 is connected to a terminal 104 via a network including, but not limited to: a wide area network, a metropolitan area network, or a local area network. The terminal 104 is not limited to a PC, a mobile phone, a tablet computer, and the like. The server 102 may be varnish. The data acquisition method according to the embodiment of the present invention may be executed by the server 102, or may be executed by both the server 102 and the terminal 104. The data acquisition method of the embodiment of the present invention executed by the terminal 104 may also be executed by a client installed thereon.
Alternatively, the process of the server 102 executing the data obtaining method according to the embodiment of the present invention may be described as follows: the server 102 receives a data request carrying a first URL address, wherein the data request is used for requesting to acquire target data, the first URL address comprises a first address and a first address parameter, the first address is used for indicating a cache server for caching the target data, and the first address parameter is used for indicating an address of a content server for storing the target data; the server 102, under the condition that the target data is not cached in the cache server, replaces the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address; the server 102 acquires the target data from the content server according to the second URL address.
Alternatively, the process of the server 102 and the terminal 104 jointly executing the data acquisition method according to the embodiment of the present invention may be described as follows: the method comprises the steps that a server 102 receives a data request which is sent by a terminal 104 and carries a first URL (uniform resource locator) address, wherein the data request is used for requesting to acquire target data, the first URL address comprises a first address and a first address parameter, the first address is used for indicating a cache server for caching the target data, and the first address parameter is used for indicating an address of a content server for storing the target data; the server 102, under the condition that the target data is not cached in the cache server, replaces the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address; the server 102 acquires target data from the content server according to the second URL address; the server 102 transmits the acquired target data to the terminal 104.
The following describes the data acquisition method according to the embodiment of the present invention in detail by taking the server 102 as an execution subject.
Fig. 2 is a flow chart of an alternative data acquisition method according to an embodiment of the present invention, which may include the following steps, as shown in fig. 2:
step S202, receiving a data request carrying a first URL (uniform resource locator) address, wherein the data request is used for requesting to acquire target data, the first URL address comprises a first address and a first address parameter, the first address is used for indicating a cache server for caching the target data, and the first address parameter is used for indicating an address of a content server for storing the target data;
step S204, under the condition that the target data is not cached in the cache server, replacing the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address;
and step S206, acquiring target data from the content server according to the second URL address.
Through the steps S202 to S206, the first URL address carried in the data request is replaced by the address of the content server indicated by the first address parameter to obtain the second URL address, and then the target data is obtained from the content server according to the second URL address, so that the purpose of obtaining the requested target data can be achieved without pre-configuring the address list of the content server, and the technical problem that the operation and maintenance cost of the configuration list is high because the varnish reverse proxy needs to pre-configure the address list of the content server in the related art is solved, thereby achieving the technical effects of reducing the operation and maintenance cost in the reverse proxy and improving the efficiency of the reverse proxy.
In the technical solution provided in step S202, the data request in the embodiment of the present invention may be used to request target data, where the type of the target data is not specifically limited in the embodiment of the present invention, and the target data may be, for example, video, audio, pictures, texts, and the like.
Optionally, the triggering manner of the data request is not specifically limited in the embodiments of the present invention, for example, the data request may be generated by being triggered by a touch operation performed by a user on a client, where the touch operation performed by the user on the client may include, but is not limited to, a click, a long press, a slide, a gesture, and the like. After the client detects the touch operation, a data request can be generated by triggering and then sent to the server.
Optionally, the data request may carry a first URL address, where the first URL address may include a first address and a first address parameter, where the first address may be used to indicate a cache server that caches the target data, and the first address parameter may be used to indicate an address of a content server that stores the target data. It should be noted here that, in the reverse proxy process, if the target data is stored in the cache server, the target data in the cache server may be directly sent to the client after receiving the data request; if the target data is not cached in the cache server, the data request needs to be forwarded to the corresponding content server after the data request is received, the requested target data is obtained from the content server, and then the requested target data is returned to the client.
As an optional example, the first URL address carried in the data request may be: http:// test.vodflowxy.myqcloud.com/live/test.mp 4sourceUrl ═ http://6000.li vector.myqcloud.com/live/68 l6U8ck8Hg.mp4, wherein the first address can be http:// test.vodflowxy.myqcloud.com/live/test.mp 4, and the first address parameter can be sourceUrl:// http://6000. live.myqcloud.com/live/68 lU8Hg.mp 4.
Optionally, the first address may include a domain name of the cache server and identification information of the target data cached in the cache server. In the above example, http:// test. vodflowproxy. myqcloud.com is the domain name of the cache server, and/live/test. mp4 is the identification information of the target data cached in the cache server.
Optionally, the first address parameter may include a domain name of the content server and an address of the target data in the content server. In the above example, http://6000.live play.myqcloud.com is the domain name of the content server, and/live/68 l6U8ck8Hg.mp4 is the identification information of the target data cached in the cache server.
It should be noted here that the cache server may be a server executing the data acquisition method according to the embodiment of the present invention.
In the technical solution provided in step S204, after the cache server receives the data request, if the cache server has the requested target data, the target data may be directly returned to the client; if the requested target data does not exist in the cache server, the cache server needs to acquire the address of the content server storing the target data, then forwards the data request to the content server to acquire the requested target data, and then returns the acquired target data to the client.
Alternatively, the cache server may obtain the address of the content server storing the target data by replacing the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address, where the second URL address may indicate the content server storing the target data.
In the above example, the first URL address is replaced by the address of the content server indicated by the first address parameter, and the obtained second URL address is: http://6000. liveplant.myqcloud.com/live/68 l6U8ck8Hg.mp4.
Optionally, the step S204 of replacing the address of the content server indicated by the first address parameter in the first URL address to obtain the second URL address may include the following steps:
step S2042, obtaining the domain name and port number of the content server from the first address parameter;
step S2044, obtain the IP address of the content server corresponding to domain name of the content server;
step S2046, the first URL address is replaced with the IP address and the port number of the content server, and a second URL address is obtained.
When the first URL address is replaced with the address of the content server indicated by the first address parameter, the first URL address may be replaced with the URL address format shown in the above example, or the first URL address may be replaced with the IP address and the port number, and in any format, the second URL address obtained after replacement may indicate the content server storing the target data.
Optionally, the step S2046 of replacing the first URL address with the IP address and the port number of the content server to obtain the second URL address may include the following steps:
step S20462, generating a target structure by using the IP address and the port number of the content server, where the target structure stores the IP address and the port number of the content server;
step S20464, replace the first URL address with the IP address and port number of the content server stored in the target structure, to obtain a second URL address.
It should be noted that, in the process of the varnish reverse proxy, after receiving a data request, data is requested from the back-end content server according to a configured address list, where the address list exists in a structural form. Therefore, in order to implement the reverse proxy, in the embodiment of the present invention, a target structure may be generated by using the IP address and the port number of the content server, and the target structure is assigned to a structure for storing an address list of a backend content server of the varnish reverse proxy, and then the IP address and the port number of the content server in the structure are used to replace the first URL address, so as to obtain the address of the content server storing the target data, thereby implementing the purpose of requesting the target data.
In the technical solution provided in step S206, after the second URL address is obtained, the embodiment of the present invention may obtain the target data from the content server according to the second URL address, and return the obtained target data to the client.
Optionally, after obtaining the target data from the content server according to the second URL address, the embodiment of the present invention may further include the following steps:
step S208, the memory occupied by the IP address and the port number of the content server stored in the target structure is released.
It should be noted that, with the data acquisition method according to the embodiment of the present invention, the purpose of implementing reverse proxy may be achieved without configuring an address list of a content server in advance, and after receiving a data request, a target structure is created to store an IP address and a port number of the content server, and then the IP address and the port number of the content server are used to replace a first URL address to obtain a second URL address, thereby implementing acquisition of target data from the content server according to the second URL address. In the embodiment of the invention, after the target data is acquired, the memory occupied by the target structure body can be released, so that the purpose of preventing the memory leakage is achieved.
As an alternative embodiment, after obtaining the target data from the content server according to the second URL address, the embodiment of the present invention may further include: caching target data and sending the target data to a client; or sending the target data to a target server for caching, and sending the target data to the client by using the target server.
It should be noted that, in the embodiment of the present invention, one server (e.g., varnish) may be used to obtain the target data and cache the target data, or one server (e.g., nginx) may be used to obtain the target data and another server (e.g., varnish) may be used to cache the target data. Both the two modes can realize reverse proxy, and one of the two modes can be selected according to actual requirements in an actual application scene.
As an optional embodiment, the data request in the embodiment of the present invention may be used to request traffic-free data, that is, the request data does not consume surf. Correspondingly, the step S202 of receiving the data request carrying the first URL address may include: receiving a data request carrying a first URL address, wherein the data request is used for requesting to acquire flow-free data; the step S206 of acquiring the target data from the content server according to the second URL address may include: and acquiring the traffic-free data from the content server according to the second URL address.
It should be noted that, in addition to requesting traffic-free data, such as traffic-free video, the embodiments of the present invention may also request other types of data, such as video on demand, VIP video, and the like.
The data acquisition method provided by the embodiment of the invention can be applied to multiple fields, such as the video field for requesting to play video, the game field for requesting game equipment, the financial field for requesting description information of financial products and the like.
The following preferred embodiments will be described in detail by taking as an example the application of the present invention to the field of video.
The invention also provides a preferred embodiment, which provides a dynamic reverse proxy method based on varnish.
Before the present preferred embodiment is described, the technical terms involved therein are first explained as follows:
varnish: the scheme is used for realizing dynamic direction proxy and caching data.
back _ hit: one structure used to represent the backend content server in the varnish profile includes IP and ports.
backend: a back-end content server representing a varnish reverse proxy.
struct director: and the varnish module is used for neutralizing the C language structure corresponding to the background _ hit.
Flow-free: the user requests a specific traffic using the mobile cellular network and the operator does not charge a traffic fee.
Returning a source: when the reverse proxy server has no data requested by the user locally, the reverse proxy server forwards the user request to a back-end content server to pull back the requested data and forwards the data to the user.
The method realizes a dynamic reverse proxy service based on an open source varnish project. Generally, varnish is used for acting on one or a group of fixed content servers, and when a user request arrives, the request is forwarded to a corresponding content server at a back end and data returned by the back end is cached.
In the video-on-demand stream-free project, a request of a client for opening a stream-free service first reaches varnish, the varnish is mainly used as a cache server of an on-demand file, and if the requested data varnish does not exist in the cache and needs to be returned to a source, the varnish serves as a reverse proxy. The content of the reverse proxy is the on-demand content of different clients on the video cloud, and the access URL, the parameters and the like of the reverse proxy are different from client to client.
The method is based on an open source varnish project, provides an optimized reverse proxy implementation, can dynamically proxy different back-end content servers, avoids defining a large number of back-end lists in a configuration file in advance, and does not need to change the configuration file when the back-end content servers are changed. The varnish configuration and operation and maintenance risks can be effectively simplified, and the access of a third-party client is facilitated.
The method can be used for video cloud on-demand flow-free projects, the varnish deployment IP is a white list IP free of flow of a cooperative operator, and the flow operator can not charge flow fees for users through the IPs. The method enables the varnish to dynamically reverse different requests arriving at the front end, dynamically analyzes the URL according to the user request information and finds out the corresponding content server to pull the data to the local, thereby greatly simplifying the configuration, operation and maintenance of the varnish.
As shown in fig. 3, the complete process of a varnish request may include:
(1) the client clinet sends a request req to varnish.
(2) The varnish construct requests bereq to access the back end content server backend and pulls back the data beresp.
(3) varnish caches data beresp into local memory.
(4) varnish responds to client clinet requesting content resp.
The general idea of the scheme is to define the URL format requested by the third-party client, so that the request URL carries the URL of the back-end content server. And then, realizing a corresponding analysis function in the varnish configuration script, so that the configuration script can analyze the information of the back-end content server from the request URL and construct a special back-end data format of the varnish. And finally, replacing the request URL with a real content server URL at the rear end and forwarding the request, and releasing the memory resource when the request is ended.
The specific implementation flow of the method is shown in fig. 4, and may include the following steps:
step S401, searching for a real domain name and a port.
Step S402, domain name resolution is carried out to obtain an IP and a port.
In step S403, a structure struct director is constructed using a varnish structure.
Step S404, judging whether the structure is successfully constructed, and if so, executing step S405; otherwise, ending.
Step S405, assigning the struct director to the back _ hit.
In step S406, the request URL is replaced with the real URL.
In step S407, data is requested using backup _ hit to the back end.
Step S408, judging whether the request data is successful, if so, executing step S409, releasing the structure director, then executing step S410, caching and returning data to the client; if the result is failed, step S409 is executed to release the structure director.
What needs to be described with respect to the above steps is:
and defining the format of the user request URL, so that the address information of the back-end content server is carried in the request URL reaching varnish. Examples of URLs:
http://test.vodflowproxy.myqcloud.com/live/test.mp4sourceUrl＝http://6000.liveplay.myqcloud.com/live/68l6U8ck8Hg.mp4
it should be noted that irrelevant information of the scheme, such as authority verification, is omitted from the URL, and only reverse proxy and URL information relevant to the scheme are reserved. Wherein, the "test.vodflowproxy.myqcloud.com" is a flow-free domain name, the third-party client requests flow-free service through the domain name, and the request can be sent to a varnish server; mp4 is an index key used when the file is cached in varnish; the URL with sourceUrl is the real back-end request URL.
After varnish receives the URL, the default processing mode is to set the URL of the back-end request to "/live/test. mp 4? sourceUrl ═ http://6000. liveplant. myqcloud. com/live/68l6U8ck8Hg. mp4 ", then find the appropriate backup in the configuration file and forward the request. Therefore, the default processing capacity of varnish needs to be expanded, namely, "/live/test.mp 4sourceUrl ═ http://6000.live plate.myqcloud.com/live/68 l6U8ck8Hg.mp4" is analyzed to respectively obtain key (/ live/test.mp4), back-end back (6000.live plate.myqcloud.com) and back-end request URL (/ live/68l6U8ck8Hg.mp4) during caching, and then corresponding fields are rewritten and the source is returned.
The cached key and the back-end request URL have corresponding variable representations in the default configuration file and are both of character string types, so direct rewriting is only needed. And the default mode of the backend is to define a structure body in the configuration file in advance and represent the structure body as the backend _ hit, and the field of the structure body cannot be directly assigned in the configuration script. So it is next necessary to construct backup _ hit from the request URL.
The backup _ hit in the configuration file actually corresponds to the struct director of the C language code. Therefore, the scheme dynamically constructs and releases the struct director by expanding the capability of the varnish bottom layer interface. Specifically, the sourceUrl parameter value is transmitted into a custom interface; the user-defined interface analyzes the transmitted domain name (or IP) and port; if the analyzed domain name is the domain name, performing domain name analysis to obtain an IP; construct pointer and return its pointer based on varnish internal interface using IP and port.
The pointer of the struct director can be freely used in the configuration script after returning to the configuration script, but because the struct director is a dynamic application memory, an interface for releasing the struct director also needs to be realized. The implementation mode is that struct director is transmitted into a custom release interface in a configuration script, and the custom release interface deletes the memory corresponding to structdirector based on a varnish internal interface, so that memory leakage is prevented.
The scheme is equivalent to converting static configuration into dynamic analysis, thereby simplifying the complexity of the configuration file.
From the perspective of the scheme itself, the configuration that originally required the configuration of thousands of backands and if-else branches can be reduced to a code block as shown in fig. 5. Even if the client is added or the configuration of the back end of the client is changed, the configuration change is not needed by varnish, and only the URL rule in the client request is needed to modify the request URL, so that the project maintenance cost and the operation and maintenance risk are greatly reduced.
Based on the scheme, the on-demand streaming-free and acceleration functions can be quickly realized, and the popularization of a third-party client and the attraction of potential users are facilitated. Thereby helping third party clients to quickly take advantage of market position.
It should be noted that in the present solution, the varnish simultaneously plays the roles of a reverse proxy and a cache server. There are many open source implementations of reverse proxies, such as nginx, but nginx is not as flexible and as professional as varnish in its ability to implement local caching. Therefore, the scheme can also be implemented by combining nginx and varnish, namely varnish is used for caching, nginx is used for reverse proxy, and the architecture diagram is shown in fig. 6.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
According to another aspect of the embodiment of the invention, a data acquisition device for implementing the data acquisition method is also provided. Fig. 7 is a schematic diagram of an alternative data acquisition device according to an embodiment of the present invention, as shown in fig. 7, the device may include:
a receiving unit 22, configured to receive a data request carrying a first URL address, where the data request is used to request to acquire target data, the first URL address includes a first address and a first address parameter, the first address is used to indicate a cache server that caches the target data, and the first address parameter is used to indicate an address of a content server that stores the target data; a replacing unit 24, configured to, when the target data is not cached in the cache server, replace the first URL address with the address of the content server indicated by the first address parameter, so as to obtain a second URL address; an obtaining unit 26, configured to obtain the target data from the content server according to the second URL address.
It should be noted that the receiving unit 22 in this embodiment may be configured to execute step S202 in this embodiment, the replacing unit 24 in this embodiment may be configured to execute step S204 in this embodiment, and the obtaining unit 26 in this embodiment may be configured to execute step S206 in this embodiment.
It should be noted here that the modules described above are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the disclosure of the above embodiments. It should be noted that the modules described above as a part of the apparatus may operate in a hardware environment as shown in fig. 1, and may be implemented by software or hardware.
As an alternative embodiment, as shown in fig. 8, the replacement unit 24 may include: a first obtaining module 242, configured to obtain a domain name and a port number of the content server from the first address parameter; a second obtaining module 244, configured to obtain an IP address of the content server corresponding to the domain name of the content server; the replacing module 246 is configured to replace the first URL address with the IP address and the port number of the content server, so as to obtain a second URL address.
As an alternative embodiment, as shown in fig. 9, the replacement module 246 may include: a generating submodule 2462, configured to generate a target structure by using the IP address and the port number of the content server, where the target structure stores the IP address and the port number of the content server; and a replacing submodule 2464, configured to replace the first URL address with the IP address and the port number of the content server stored in the target structure, to obtain a second URL address.
As an alternative embodiment, as shown in fig. 10, the apparatus may further include: and the releasing unit 28 is configured to release the memory occupied by the target structure for storing the IP address and the port number of the content server after the target data is obtained from the content server according to the second URL address.
As an alternative embodiment, as shown in fig. 11, the receiving unit 22 may include: a receiving module 222, configured to receive a data request carrying a first URL address, where the data request is used to request to acquire traffic-free data; the acquisition unit 26 may include: and a third obtaining module 262, configured to obtain the traffic-exempt data from the content server according to the second URL address.
As an alternative embodiment, as shown in fig. 12, the apparatus may further include: a first caching unit 210, configured to cache the target data after acquiring the target data from the content server according to the second URL address, and send the target data to the client; or the second cache unit 212 is configured to send the target data to the target server for caching, and send the target data to the client by using the target server.
It should be noted here that the modules described above are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the disclosure of the above embodiments. It should be noted that the modules described above as a part of the apparatus may operate in a hardware environment as shown in fig. 1, and may be implemented by software or hardware.
Through the module, the purpose of acquiring the requested target data can be achieved without pre-configuring the address list of the content server, and the technical problem that the operation and maintenance cost of the configuration list is high due to the fact that the address list of the content server needs to be pre-configured by the varnish reverse proxy in the related technology is solved, so that the operation and maintenance cost during reverse proxy is reduced, and the technical effect of improving the efficiency of the reverse proxy is achieved.
According to another aspect of the embodiment of the present invention, there is also provided an electronic device for implementing the data acquisition method.
Fig. 13 is a block diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 13, the electronic device may include: one or more processors 201 (only one is shown in the figure), and a memory 203, wherein the memory 203 may store a computer program, and the processor 201 may be configured to execute the computer program to perform the data acquisition method according to the embodiment of the present invention.
The memory 203 may be used to store computer programs and modules, such as program instructions/modules corresponding to the data acquisition method and apparatus in the embodiments of the present invention, and the processor 201 executes various functional applications and data processing by running the computer programs and modules stored in the memory 203, that is, implements the data acquisition method. The memory 203 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 203 may further include memory located remotely from the processor 201, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Optionally, as shown in fig. 13, the electronic device may further include: a transmission device 205 and an input-output device 207. Wherein the transmission means 205 is used for receiving or transmitting data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 205 includes a network adapter (NIC) that can be connected to a router via a network cable and other network devices to communicate with the internet or a local area network. In one example, the transmission device 205 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
It can be understood by those skilled in the art that the structure shown in fig. 13 is only an illustration, and the electronic device may be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a Mobile Internet Device (MID), a PAD, etc. Fig. 13 is a diagram illustrating a structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in FIG. 13, or have a different configuration than shown in FIG. 13.
Alternatively, in this embodiment, the memory 203 may be used to store a computer program.
Optionally, in this embodiment, the processor may be configured to execute a computer program to perform the following steps: receiving a data request carrying a first URL (uniform resource locator) address, wherein the data request is used for requesting to acquire target data, the first URL address comprises a first address and a first address parameter, the first address is used for indicating a cache server for caching the target data, and the first address parameter is used for indicating an address of a content server for storing the target data; under the condition that target data are not cached in the cache server, replacing the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address; and acquiring the target data from the content server according to the second URL address.
The processor 201 is further configured to perform the steps of: acquiring a domain name and a port number of a content server from the first address parameter; acquiring an IP address of a content server corresponding to a domain name of the content server; and replacing the first URL address with the IP address and the port number of the content server to obtain a second URL address.
The processor 201 is further configured to perform the following steps: generating a target structure body by using the IP address and the port number of the content server, wherein the IP address and the port number of the content server are stored in the target structure body; and replacing the first URL address with the IP address and the port number of the content server stored in the target structure body to obtain a second URL address.
The processor 201 is further configured to perform the following steps: and after the target data is acquired from the content server according to the second URL address, releasing the memory occupied by the IP address and the port number of the content server stored in the target structure body.
The processor 201 is further configured to perform the following steps: receiving a data request carrying a first URL address, wherein the data request is used for requesting to acquire flow-free data; the obtaining the target data from the content server according to the second URL address includes: and acquiring the traffic-free data from the content server according to the second URL address.
The processor 201 is further configured to perform the following steps: after the target data are obtained from the content server according to the second URL address, caching the target data and sending the target data to the client; or sending the target data to a target server for caching, and sending the target data to the client by using the target server.
Optionally, for a specific example in this embodiment, reference may be made to the example described in the foregoing embodiment, and this embodiment is not described herein again.
By adopting the embodiment of the invention, a data acquisition scheme is provided, a first URL address carried in a data request is replaced by an address of a content server indicated by a first address parameter to obtain a second URL address, and then target data is acquired from the content server according to the second URL address, so that the purpose of acquiring the requested target data can be achieved without pre-configuring an address list of the content server, and the technical problem that the operation and maintenance cost of a configuration list is higher because a varnish reverse proxy needs to pre-configure the address list of the content server in the related technology is solved, thereby realizing the technical effects of reducing the operation and maintenance cost in the reverse proxy and improving the efficiency of the reverse proxy.
According to still another aspect of an embodiment of the present invention, there is also provided a storage medium. The storage medium has stored therein a computer program, wherein the computer program is arranged to execute the steps of the data acquisition method in the above embodiments when run.
Alternatively, in this embodiment, the storage medium may be located on at least one of the plurality of network devices in the network shown in the above-described embodiment.
Alternatively, in the present embodiment, the storage medium is configured to store a computer program for executing the steps of:
s1, receiving a data request carrying a first URL address, wherein the data request is used for requesting to acquire target data, the first URL address comprises a first address and a first address parameter, the first address is used for indicating a cache server for caching the target data, and the first address parameter is used for indicating an address of a content server for storing the target data;
s2, under the condition that the target data is not cached in the cache server, replacing the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address;
and S3, acquiring the target data from the content server according to the second URL address.
Optionally, the storage medium is further arranged to store a computer program for performing the steps of: acquiring a domain name and a port number of a content server from the first address parameter; acquiring an IP address of a content server corresponding to a domain name of the content server; and replacing the first URL address with the IP address and the port number of the content server to obtain a second URL address.
Optionally, the storage medium is further arranged to store a computer program for performing the steps of: generating a target structure body by using the IP address and the port number of the content server, wherein the IP address and the port number of the content server are stored in the target structure body; and replacing the first URL address with the IP address and the port number of the content server stored in the target structure body to obtain a second URL address.
Optionally, the storage medium is further arranged to store a computer program for performing the steps of: and after the target data is acquired from the content server according to the second URL address, releasing the memory occupied by the IP address and the port number of the content server stored in the target structure body.
Optionally, the storage medium is further arranged to store a computer program for performing the steps of: receiving a data request carrying a first URL address, wherein the data request is used for requesting to acquire flow-free data; the obtaining the target data from the content server according to the second URL address includes: and acquiring the traffic-free data from the content server according to the second URL address.
Optionally, the storage medium is further arranged to store a computer program for performing the steps of: after the target data are obtained from the content server according to the second URL address, caching the target data and sending the target data to the client; or sending the target data to a target server for caching, and sending the target data to the client by using the target server.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment is not described herein again.
Optionally, in this embodiment, a person skilled in the art may understand that all or part of the steps in the method of the foregoing embodiment may be implemented by instructing, by a program, hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages and disadvantages of the embodiments.
The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, network devices, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and amendments can be made without departing from the principle of the present invention, and these modifications and amendments should also be considered as the protection scope of the present invention.
Claims (8)
1. A data acquisition method is applied to a reverse proxy server and comprises the following steps:
receiving a data request carrying a first URL (uniform resource locator) address, wherein the data request is used for requesting to acquire target data, the first URL address comprises a first address and a first address parameter, the first address is used for indicating a cache server for caching the target data, and the first address parameter is used for indicating an address of a content server for storing the target data; the first address comprises a domain name of the cache server and identification information cached by target data in the cache server; the cache server is in communication connection with the reverse proxy server;
under the condition that the target data is not cached in the cache server, replacing the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address;
acquiring the target data from the content server according to the second URL address;
the receiving the data request carrying the first URL address includes: receiving a data request carrying the first URL address, wherein the data request is used for requesting to acquire flow-free data;
the obtaining the target data from the content server according to the second URL address includes: acquiring the traffic-free data from the content server according to the second URL address;
the replacing the first URL address with the address of the content server indicated by the first address parameter to obtain a second URL address includes:
acquiring the domain name and the port number of the content server from the first address parameter;
acquiring an IP address of the content server corresponding to the domain name of the content server;
replacing the first URL address with the IP address and the port number of the content server to obtain a second URL address;
after the obtaining the target data from the content server according to the second URL address, the method further includes:
and releasing the memory occupied by the IP address and the port number of the content server.
2. The method of claim 1, wherein the replacing the first URL address with the IP address and the port number of the content server to obtain the second URL address comprises:
generating a target structure body by using the IP address and the port number of the content server, wherein the IP address and the port number of the content server are stored in the target structure body;
and replacing the first URL address with the IP address and the port number of the content server stored in the target structure body to obtain the second URL address.
3. The method according to claim 1 or 2, wherein after the obtaining the target data from the content server according to the second URL address, the method further comprises:
caching the target data and sending the target data to a client; or
And sending the target data to a target server for caching, and sending the target data to the client by using the target server.
4. A data acquisition device, wherein the device is disposed in a reverse proxy server, comprising:
a receiving unit, configured to receive a data request carrying a first URL address, where the data request is used to request to acquire target data, the first URL address includes a first address and a first address parameter, the first address is used to indicate a cache server that caches the target data, and the first address parameter is used to indicate an address of a content server that stores the target data; the first address comprises a domain name of the cache server and identification information of target data cached in the cache server; the cache server is in communication connection with the reverse proxy server;
a replacing unit, configured to replace the first URL address with an address of the content server indicated by the first address parameter to obtain a second URL address when the target data is not cached in the cache server;
an acquisition unit configured to acquire the target data from the content server according to the second URL address;
the receiving unit includes: a receiving module, configured to receive a data request carrying the first URL address, where the data request is used to request to acquire traffic-free data;
the acquisition unit includes: a third obtaining module, configured to obtain the traffic-free data from the content server according to the second URL address;
the replacement unit includes:
a first obtaining module, configured to obtain a domain name and a port number of the content server from the first address parameter;
a second obtaining module, configured to obtain an IP address of the content server corresponding to the domain name of the content server;
the replacing module is used for replacing the first URL address with the IP address and the port number of the content server to obtain a second URL address;
and the releasing unit is used for releasing the memory occupied by the IP address and the port number of the content server after the target data is acquired from the content server according to the second URL address.
5. The apparatus of claim 4, wherein the replacement module comprises:
the generation submodule is used for generating a target structure body by using the IP address and the port number of the content server, wherein the target structure body stores the IP address and the port number of the content server;
and the replacing submodule is used for replacing the first URL address with the IP address and the port number of the content server stored in the target structure body to obtain the second URL address.
6. The apparatus of claim 4 or 5, further comprising:
the first caching unit is used for caching the target data after the target data is acquired from the content server according to the second URL address, and sending the target data to a client; or
And the second cache unit is used for sending the target data to a target server for caching and sending the target data to the client by using the target server.
7. A storage medium, in which a computer program is stored, wherein the computer program, when executed by a processor, performs the method of any one of claims 1 to 3.
8. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810349630.8A CN110198333B (en) | 2018-04-18 | 2018-04-18 | Data acquisition method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810349630.8A CN110198333B (en) | 2018-04-18 | 2018-04-18 | Data acquisition method and device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110198333A CN110198333A (en) | 2019-09-03 |
| CN110198333B true CN110198333B (en) | 2022-05-20 |
Family
ID=67751051
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810349630.8A Active CN110198333B (en) | 2018-04-18 | 2018-04-18 | Data acquisition method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110198333B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113301175B (en) * | 2020-07-14 | 2022-04-12 | 阿里巴巴集团控股有限公司 | Service calling method, data storage method, device, equipment and storage medium |
| CN113312036B (en) * | 2021-06-15 | 2023-08-18 | 曙光信息产业(北京)有限公司 | Large screen display method, device and equipment of Web page and storage medium |
| CN113973135A (en) * | 2021-10-19 | 2022-01-25 | 北京沃东天骏信息技术有限公司 | Data caching processing method and device, caching grid platform and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103248684A (en) * | 2013-04-28 | 2013-08-14 | 北京奇虎科技有限公司 | Resource acquiring method and device on Internet |
| CN103647811A (en) * | 2013-11-28 | 2014-03-19 | 北京奇虎科技有限公司 | A method and an apparatus for application's accessing backstage service |
| CN106790059A (en) * | 2016-12-20 | 2017-05-31 | Tcl集团股份有限公司 | Intelligent television Launcher cloud systems service calling method and system |
| WO2017107568A1 (en) * | 2015-12-21 | 2017-06-29 | 北京大学 | Cloud-side cooperation-based method for optimizing resource loading at mobile browser |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102780711B (en) * | 2011-05-09 | 2016-03-30 | 腾讯科技（深圳）有限公司 | A kind of SNS application data access method and device thereof and system |
-
2018
- 2018-04-18 CN CN201810349630.8A patent/CN110198333B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103248684A (en) * | 2013-04-28 | 2013-08-14 | 北京奇虎科技有限公司 | Resource acquiring method and device on Internet |
| CN103647811A (en) * | 2013-11-28 | 2014-03-19 | 北京奇虎科技有限公司 | A method and an apparatus for application's accessing backstage service |
| WO2017107568A1 (en) * | 2015-12-21 | 2017-06-29 | 北京大学 | Cloud-side cooperation-based method for optimizing resource loading at mobile browser |
| CN106790059A (en) * | 2016-12-20 | 2017-05-31 | Tcl集团股份有限公司 | Intelligent television Launcher cloud systems service calling method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110198333A (en) | 2019-09-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109561171B (en) | Configuration method and device of virtual private cloud service | |
| CN104580192B (en) | The treating method and apparatus of the network access request of application program | |
| WO2019183978A1 (en) | Technologies for content delivery network with multi-access edge computing | |
| CN110198333B (en) | Data acquisition method and device, storage medium and electronic device | |
| CN108370379A (en) | With cunicular equipment management | |
| CN108494860B (en) | WEB access system, WEB access method and device for client | |
| CN108156210A (en) | The acquisition methods and device of target resource | |
| CN105611422B (en) | Online live broadcasting method and device based on multimedia list | |
| JP2018506936A (en) | Method and system for an end-to-end solution for distributing content in a network | |
| CN110198229B (en) | Network configuration method and device, storage medium and electronic device | |
| CN110399578A (en) | Page access method and device | |
| CN109729183A (en) | Request processing method, device, equipment and storage medium | |
| CN112732572A (en) | Service testing method, device and system, storage medium and electronic device | |
| CN105095220B (en) | A kind of browser implementation method, terminal and virtualization agent device | |
| CN110351276A (en) | Data processing method, equipment and computer readable storage medium | |
| CN111367685B (en) | Interface calling method and device, computer equipment and storage medium | |
| CN108345606A (en) | The acquisition methods and device of web page resources | |
| CN109522462A (en) | A kind of cloud querying method, device, equipment and storage medium based on block chain | |
| CN112241298A (en) | Page display method and device, storage medium and electronic device | |
| CN109194706A (en) | Internet resources dial testing method and terminal | |
| CN106156048B (en) | Method and apparatus for providing short uniform resource locator service and communication system | |
| CN115242882B (en) | Method and device for accessing k8s container environment based on transport layer route | |
| CN110545453B (en) | Content distribution method, device and system of content distribution network | |
| KR20220006605A (en) | Cloud communication method and device, user device, network device | |
| US20160344838A1 (en) | Caching of tracking elements in network content |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |